Name: cloudbreak
Owner: Hortonworks Inc
Description: A tool for provisioning and managing Apache Hadoop clusters in the cloud. Cloudbreak, as part of the Hortonworks Data Platform, makes it easy to provision, configure and elastically grow HDP clusters on cloud infrastructure. Cloudbreak can be used to provision Hadoop across cloud infrastructure providers including AWS, Azure, GCP and OpenStack.
Created: 2014-05-10 10:03:07.0
Updated: 2017-12-24 04:47:26.0
Pushed: 2018-01-15 19:11:03.0
Homepage: http://hortonworks.github.io/cloudbreak-docs
Size: 36279
Language: Java
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
As of now this document is focusing on setting up your development environment on OSX. You'll need brew to install certain components in case you don't have them already. To get brew please follow the install instructions on the brew homepage: https://brew.sh
As a prerequisite you need to have Java installed. You can get it with the following command:
cask install java8
You'll need a Hypervisor too. Cloudbreak-Deployer has built-in xhyve setup option, but some of us use VirtualBox instead. Cloudbreak-Deployer works with both, it's up to you which one you prefer.
To set xhyve up:
install docker-machine-driver-xhyve
For VirtualBox usage:
cask install virtualbox
The simplest way to setup the working environment to be able to start Cloudbreak on your local machine is to use the Cloudbreak Deployer.
First you need to create a sandbox directory which will store the necessary configuration files and dependencies of Cloudbreak Deployer. This directory must be created outside of the cloned Cloudbreak git repository:
r cbd-local
bd-local
The next step is to download the cloudbreak-deployer onto your machine:
-s https://raw.githubusercontent.com/hortonworks/cloudbreak-deployer/master/install-dev | sh [-s branch] && cbd --version
Use the -s branch option for sh here in case you'd like to checkout another branch than master.
The next step is to setup your docker-machine. The cbd machine create command initializes xhyve for you, you can setup you docker machine with virtualbox as well if you prefer.
To configure docker machine with xhyve:
xhyve:
machine create
$(docker-machine env cbd)
To configure docker machine with VirtualBox:
VirtualBox:
er-machine create --driver virtualbox --virtualbox-disk-size "50000" cbd
$(docker-machine env cbd)
IP settings are based on your docker-machine configuration. The 'docker-machine ip cbd' command prints the ip address of your docker machine. This is the address you should use in multiple places below. Let's refer to this address as YOUR_IP throughout this document:
_IP=$(docker-machine ip cbd)
Add the following to the file named Profile under the cbd-local directory you have just created. Please note, when a cbd command is executed you should go to the deployment's directory where your Profile file could be found (cbd-local in our example). The CB_SCHEMA_SCRIPTS_LOCATION environment variable configures the location of SQL scripts that are in the 'core/src/main/resources/schema' directory in the cloned Cloudbreak git repository. CB_LOCAL_DEV_BIND_ADDR is the address of the network interface (vboxnet0 for VirtualBox or bridge100 for xhyve, one can check it with ifconfig).
Please note that the full path needs to be configured and env variables like $USER cannot be used. You also have to set a password for your local Cloudbreak in UAA_DEFAULT_USER_PW:
xhyve:
rt PRIVATE_IP=$PUBLIC_IP
rt ULU_SUBSCRIBE_TO_NOTIFICATIONS=true
rt CB_INSTANCE_UUID=$(uuidgen | tr '[:upper:]' '[:lower:]')
rt CB_SCHEMA_SCRIPTS_LOCATION=/Users/YOUR_USERNAME/YOUR_PROJECT_DIR/cloudbreak/core/src/main/resources/schema
rt UAA_DEFAULT_USER_PW=YOUR_PASSWORD
VirtualBox:
rt PUBLIC_IP=$(docker-machine ip cbd)
rt DOCKER_MACHINE=cbd
rt PRIVATE_IP=$PUBLIC_IP
rt ULU_SUBSCRIBE_TO_NOTIFICATIONS=true
rt CB_INSTANCE_UUID=$(uuidgen | tr '[:upper:]' '[:lower:]')
rt CB_SCHEMA_SCRIPTS_LOCATION=/Users/YOUR_USERNAME/YOUR_PROJECT_DIR/cloudbreak/core/src/main/resources/schema
rt UAA_DEFAULT_USER_PW=YOUR_PASSWORD
rt CB_LOCAL_DEV_BIND_ADDR=192.168.99.1
Then run these commands:
start
logs cloudbreak
In case you see org.apache.ibatis.migration.MigrationException at the end of the logs run these commands to fix the DB and the re-run the previous section(cbd start and logs):
migrate cbdb up
migrate cbdb pending
If everything went well then Cloudbreak will be available on http://YOUR_IP. For more details and config parameters please check the documentation of Cloudbreak Deployer.
The deployer has generated a certs directory under cbd-local directory which will be needed later on to set up IDEA properly.
In order to kill Cloudbreak and Periscope containers running in docker/boot2docker and redirect the Cloudbreak and Periscope related traffic to the Cloudbreak running in IDEA, use the following command:
util local-dev
Go to https://github.com/hortonworks/cloudbreak, Either clone or download the repository, use SSH which is described here: https://help.github.com/articles/connecting-to-github-with-ssh/
In IDEA set your SDK to your Java version under: Configure -> Project Defaults -> Project Structure -> Project SDK
Cloudbreak can be imported into IDEA as gradle project by specifying the cloudbreak repo root under Import Project. Once it is done, you need to import the proper code formatter by using the File -> Import Settings… menu and selecting the idea_settings.jar located in the config/idea directory in Cloudbreak git repository.
To launch the Cloudbreak application execute the com.sequenceiq.cloudbreak.CloudbreakApplication class (Set 'Use classpath of module' to core_main) with the following VM options:
.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD
.client.id=cloudbreak
.client.secret=CB_SECRET_GENERATED_BY_CBD
.db.port.5432.tcp.addr=YOUR_IP
.db.port.5432.tcp.port=5432
.identity.server.url=http://YOUR_IP:8089
rver.port=9091
.schema.migration.auto=true
The -Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD value above needs to be replaced with the full path of certs directory generated by the Cloudbreak-Deployer e.g. /Users/YOUR_USERNAME/YOUR_PROJECT_DIR/cbd-local/certs.
The -Dcb.client.secret=CB_SECRET_GENERATED_BY_CBD value has to be replaced with the value of UAA_DEFAULT_SECRET from the cdb-local/Profile file.
The database migration scripts run automatically by Cloudbreak by default, this migration can be turned off with the -Dcb.schema.migration.auto=false VM option.
In order to be able to determine the local Cloudbreak version automatically, a Before launch task has to be configured for the project in IntelliJ Idea. The required steps are the following:
Run/Debug Configurations for the projectAdd in the Before launch panelRun Gradle Task with the following parametersGradle project: cloudbreak:coreTasks: buildInfoTo run Cloudbreak from command line, you have to list the JVM parameters from above for gradle:
adlew :core:buildInfo :core:bootRun -PjvmArgs="-Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD \
.client.id=cloudbreak \
.client.secret=CB_SECRET_GENERATED_BY_CBD \
.db.port.5432.tcp.addr=YOUR_IP \
.db.port.5432.tcp.port=5432 \
.identity.server.url=http://YOUR_IP:8089 \
rver.port=9091"
The -Dcb.cert.dir=FULL_PATH_OF_THE_CERTS_DIR_GENERATED_BY_CBD value above needs to be replaced with the full path of certs directory generated by the Cloudbreak-Deployer e.g. /Users/YOUR_USERNAME/YOUR_PROJECT_DIR/cbd-local/certs.
The -Dcb.client.secret=CB_SECRET_GENERATED_BY_CBD value has to be replaced with the value of UAA_DEFAULT_SECRET from the cdb-local/Profile file.
The database migration scripts run automatically by Cloudbreak by default, this migration can be turned off with the -Dcb.schema.migration.auto=false VM option.
If any schema change is required in Cloudbreak database (cbdb), then the developer needs to write SQL scripts to migrate the database accordingly. The schema migration is managed by MYBATIS Migrations in Cloudbreak and the cbd tool provides an easy-to-use wrapper for it. The syntax for using the migration commands is cbd migrate <database name> <command> [parameters] e.g. cbd migrate migrate status.
Create a SQL template for schema changes:
migrate cbdb new "CLOUD-123 schema change for new feature"
As as result of the above command an SQL file template is generated under the path specified in CB_SCHEMA_SCRIPTS_LOCATION environment variable, which is defined in Profile. The structure of the generated SQL template looks like the following:
/ CLOUD-123 schema change for new feature
igration SQL that makes the change goes here.
/@UNDO
QL to undo the change goes here.
Once you have implemented your SQLs then you can execute them with:
migrate cbdb up
Make sure pending SQLs to run as well:
migrate cbdb pending
If you would like to rollback the last SQL file, then just use the down command:
migrate cbdb down
On order to check the status of database
migrate cbdb status
ry script that has not been executed will be marked as ...pending... in the output of status command:
--------------------------------------------------------------------
yBatis Migrations - status
--------------------------------------------------------------------
Applied At Description
============================================================================
0421140021 2015-07-08 10:04:28 create changelog
0421150000 2015-07-08 10:04:28 CLOUD-607 create baseline schema
0507121756 2015-07-08 10:04:28 CLOUD-576 change instancegrouptype hostgroup to core
1008090632 ...pending... CLOUD-123 schema change for new feature
--------------------------------------------------------------------
Gradle is used for build and dependency management. Gradle wrapper is added to Cloudbreak git repository, therefore building can be done with:
adlew clean build