Name: cloudbreak-images
Owner: Hortonworks Inc
Description: Saltstack scripts to bake amazon/gcc/azure images suitable for Cloudbreak
Created: 2014-11-17 13:54:12.0
Updated: 2017-12-18 20:58:37.0
Pushed: 2018-01-15 16:12:29.0
Size: 5776
Language: Shell
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Table of Contents
Cloudbreak is a tool to simplify the provisioning, configuration and scaling of Hortonworks Data Platform clusters on cloud provider infrastructure. Cloudbreak can be used to provision across cloud infrastructure providers including: Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP).
Learn more about Cloudbreak here: http://hortonworks.github.io/cloudbreak-docs/
Cloudbreak launches clusters from an image that includes default configuration and default tooling for provisioning. These are considered the Standard Default images and these images are provided with each Cloudbreak version.
From bird's-eye view, images contain the following:
Important: Ambari and HDP packages are not part of the image and the desired version of Ambari and HDP packages are downloaded during provision time. This makes the images agnostic to the version of Ambari and HDP that can be installed by Cloudbreak.
The following Standard Default images and Linux versions are available for each Cloudbreak version:
In some cases, these default images might not fit the requirements of users (e.g. they need custom OS hardening, libraries, tooling, etc) and instead, the user would like to start their clusters from their own custom image. The repository includes instructions and scripts to help build those custom images. Once you have an images, refer to the Cloudbreak documentation for information on how to register and use these images with Cloudbreak: http://hortonworks.github.io/cloudbreak-docs/
You have two options for building a custom image:
package.json to start from your own base image.Our recommendation is to fork this repo to to your own GitHub account or to the account of your organization and you can make changes there and create an image from there. If you think that some of the changes you made might be useful for the Cloudbreak product as a whole, feel free to send us a pull request.
Note: After you have have forked the repository, you are responsible to keep it up to date and fetch the latest changes from the upstream repository.
This repository contains different branches for different Cloudbreak versions. Cloudbreak versions are defined as:
or>.<minor>.<patch>[-build sequence] e.g 1.16.3 or 1.16.4-rc.7
If you are creating a custom image for Cloudbreak, always make sure that you are using the correct branch from cloudbreak-images repository.
You can find the related branch based on the
Note: If you do not use the appropriate branch for creating your image then there is a chance that Cloudbreak will not be able to install the cluster successfully.
Images for Cloudbreak are created by Packer. The main entry point for creating an image is the Makefile which provides wrapper functionality around Packer scripts.
You can find more details about how it works in the Packer documentation.
Main configuration of Packer for building the Cloudbreak images is located in the packer.json file.
The following are requirements for the image building environment:
Set the following environment variables to build AWS images:
Example for environment variables:
rt AWS_ACCESS_KEY_ID=AKIAIQ**********
rt AWS_SECRET_ACCESS_KEY=XHj6bjmal***********************
Note: Since Packer is the underlaying technology used to build the AWS images, you can learn more about the environment variables at Packer > Amazon Reference and the minimal set of AWS IAM or Role policies necessary at Packer > Amazon EC2 Roles
Use the following commands to build AWS images based on the following base operating systems:
| OS | Build Command |
|—|—|
| Amazon Linux | make build-aws-amazonlinux |
| CentOS 6 | make build-aws-centos6 |
| CentOS 7 | make build-aws-centos7 |
| RHEL 7 | make build-aws-rhel7 |
If you want to start from your own base image, follow the instructions in Advanced topics to modify the
package.jsonto start from your own base image. Then use the commands above to build that image.
Set the following environment variables to build Azure images:
Example for environment variables:
rt ARM_CLIENT_ID=3234bb21-e6d0-*****-****-**********
rt ARM_CLIENT_SECRET=2c8bzH******************************
rt ARM_SUBSCRIPTION_ID=a9d4456e-349f-*****-****-**********
rt ARM_TENANT_ID=b60c9401-2154-*****-****-**********
rt ARM_GROUP_NAME=resourcegroupname
rt ARM_STORAGE_ACCOUNT=storageaccountname
rt AZURE_IMAGE_PUBLISHER=OpenLogic
rt AZURE_IMAGE_OFFER=CentOS
rt AZURE_IMAGE_SKU=7.2
Note: Since Packer is the underlaying technology used to build the Azure images, you can learn more about the environment variables at Packer > Azure Reference.
Use the following commands to build Azure images based on the following base operating systems:
| OS | Build Command |
|—|—|
| CentOS 7 | make build-azure-centos7 |
If you want to start from your own base image, follow the instructions in Advanced topics to modify the
package.jsonto start from your own base image. Then use the commands above to build that image.
Set the following environment variables to build Google Cloud Platform images:
Example for environment variables:
rt GCP_ACCOUNT_FILE=/var/lib/jenkins/.gce/siq-haas.json
rt GCP_CLIENT_SECRET=/var/lib/jenkins/.gce/client_secret.json
rt GCP_PROJECT=siq-haas
Note: Since Packer is the underlaying technology used to build the Google Cloud Platform images, you can learn more about the environment variables at Packer > Google Compute Reference.
Use the following commands to build GCP images based on the following base operating systems:
| OS | Build Command |
|—|—|
| CentOS 7 | make build-gc-centos7 |
If you want to start from your own base image, follow the instructions in Advanced topics to modify the
package.jsonto start from your own base image. Then use the commands above to build that image.
Set the following environment variables to build OpenStack images:
Example for environment variables:
rt OS_AUTH_URL=http://openstack.eng.hortonworks.com:5000/v2.0
rt OS_USERNAME=cloudbreak
rt OS_TENANT_NAME=cloudbreak
rt OS_PASSWORD=**********
Note: Since Packer is the underlaying technology used to build the OpenStack images, you can learn more about the environment variables at Packer > OpenStack Reference.
Use the following commands to build OpenStack images based on the following base operating systems:
| OS | Build Command |
|—|—|
| CentOS 7 | make build-os-centos7 |
If you want to start from your own base image, follow the instructions in Advanced topics to modify the
package.jsonto start from your own base image. Then use the commands above to build that image.
If you run Packer in debug mode then you can SSH into the VM during build phase and do additional debugging steps on the VM. This is how to start a build in debug mode:
ER_OPTS=--debug make build-aws-rhel7
In debug mode, you need to hit enter before each step is executed by Packer. Once the VM is launched by Packer you can login and do additional debug steps:
-i ec2_aws-rhel7.pem ec2-user@<address of the machine displayed by Packer>
A simple file browser is launched during image creation which can be accessed on port 9999.
User:
admin, password:secret.
To access the browser, you need to open port 9999 in the security group of the generated resource group manually on your cloud provider. The generated resource group name will be displayed at the start of the build process.
E.g. on Azure:
arm-centos7: Creating Azure Resource Manager (ARM) client ...
arm-centos7: Creating resource group ...
arm-centos7: -> ResourceGroupName : 'packer-Resource-Group-qx0lx7wkg7'
arm-centos7: -> Location : 'northeurope'
arm-centos7: -> Tags :
You can read more about postprocessors and customizing your base image with custom scripts and logic here.