Extend hadoop::repo to support Apache Bigtop based custom distributions


Apache Bigtop (http://bigtop.apache.org) is a framework for building and testing Apache big data stack distributions, initially derived from the CDH build infrastructure, now underpinning other distributions like HDP, WANDisco, etc. The killer feature of this framework is it provides users the ability to build RPM or DEB packages from arbitrary SCM repositories and versions of the respective projects, substituting custom or modified versions of the software, yet producing package names and a dependency structure that should more or less drop into hadoop::repo given the similarity to CDH and HDP.

Suggested changes:

Admit "bigtop" as a third distribution alternative in hadoop::repo.

Allow for yum_repo_url , yum_repo_key_url, apt_repo_url, and apt_repo_key_url to be supplied by the user for the "bigtop" alternative.

Provide an option for installing unsigned binaries (with a warning).

This will be an essential feature for any developer of Hadoop ecosystem software who wishes to use Coopr for provisioning test clusters to test their work. Using Bigtop isn't too onerous. For example when building test HBase packages, typically these are the steps required to stage new HBase RPM packages for upgrade or reinstallation, assuming previously a repository was initialized by a one time build of all Bigtop packages:

This is easy to automate with a dev script. It might then be easy to (re)deploy with Coopr.

Release Notes



Andrew Purtell
October 10, 2014, 5:27 PM

BTW, thanks a lot for adding support for Apache Bigtop and custom distributions based on it!

jay vyas
October 13, 2014, 1:15 AM

Ditto, thanks ! This is really cool .... !

Derek Wood
October 14, 2014, 4:54 PM

Great! Yeah, we'll just bump the default version to 0.8.0.

Chris Gianelloni
October 16, 2014, 6:15 PM

The cookbook is released to the wild, so I am going to RESOLVE this. Inclusion into Coopr is already out for review.


Chris Gianelloni
October 16, 2014, 6:15 PM

Released with hadoop cookbook 1.6.1 version...



Derek Wood


Andrew Purtell



Docs Impact


UX Impact