Name: aws-cicd-docker-containers
Owner: Amazon Web Services - Labs
Owner: AWS Samples
Description: This repo explains how to build a continuous delivery pipeline that automatically builds and deploys your Docker images and deploys them onto an ECS cluster.
Created: 2017-01-16 18:04:53.0
Updated: 2018-01-11 07:37:38.0
Pushed: 2017-06-07 15:19:46.0
Homepage: null
Size: 81
Language: null
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This getting started guide is intended to help you set up and configure a continuous delivery pipeline for Amazon EC2 Container Service (Amazon ECS) using Jenkins, GitHub, and the Amazon EC2 Container Registry (Amazon ECR). The pipeline builds Docker images from a GitHub repository, pushes those images to an ECR registry, creates an ECS task definition, and then uses that task definition to create a service on the ECS cluster. We use Jenkins to orchestrate the different steps in the workflow.
To use this guide, you must have the following software components installed:
Note: Homebrew is the package manager for OS X. You will need homebrew to install jq.
Note: when installing Chocolatey, you might have to launch command window as Administrator. Once you have Chocolatey installed on your machine, you can use it to install the remaining prerequisites.
iam create-access-key --username <user_name>
<user_name>
is an IAM user with AdministratorAccess.
Copy and paste the output from the previous command to a text file
Note: AdministratorAccess is a managed policy that allows attached entities to perform all actions against all resources. Although we?re using it here for convenience, you should remove the AdministratorAccess policy should be removed from your IAM user when it?s no longer needed.
Create an AWS profile on your local machine. From a command prompt, type:
configure
At the prompts, paste your aws access key ID, aws secret key ID, the preferred region (us-west-2), and json as the output format.
Create an SSH key in the us-west-2 region. We will use this SSH key to login to the Jenkins server to retrieve the administrator password.
A file will be downloaded to your default download directory
(OS X only) Change the working directory to your download directory and change permission so only the current logged in user can read it. <file_name>
is the name of the .pem file you downloaded earlier:
d <file_name> 400
Clone the git repository that contains the CloudFormation templates to create the infrastructure we?ll use to build our pipeline.
Open a command prompt and clone the Github repository that has the template.
clone https://github.com/jicowan/hello-world
Change the working directory to the directory that was created when you cloned the repository. At the command prompt, type or paste the following. Where <key_name>
is the name of an SSH key in the region where you're creating the ECS cluster:
cloudformation create-stack --template-body file://ecs-cluster.template --stack-name EcsClusterStack --capabilities CAPABILITY_IAM --tags Key=Name,Value=ECS --region us-west-2 --parameters ParameterKey=KeyName,ParameterValue=<key_name> ParameterKey=EcsCluster,ParameterValue=getting-started ParameterKey=AsgMaxSize,ParameterValue=2
Note: Do not proceed to the next step until the Stack Status shows CREATE_COMPLETE. To get the status of the stack at a command prompt, type aws cloudformation describe-stacks --stack-name EcsClusterStack --query 'Stacks[*].[StackId, StackStatus]'
Jenkins is a popular server for implementing continuous integration and continuous delivery pipelines. In this example, you'll use Jenkins to build a Docker image from a Dockerfile, push that image to the Amazon ECR registry that you created earlier, and create a task definition for your container. Finally, you'll deploy and update a service running on your ECS cluster.
cloudformation create-stack --template-body file://ecs-jenkins-demo.template --stack-name JenkinsStack --capabilities CAPABILITY_IAM --tags Key=Name,Value=Jenkins --region us-west-2 --parameters ParameterKey=EcsStackName,ParameterValue=EcsClusterStack`
Note: Do not proceed to the next step until the Stack Status shows CREATE_COMPLETE. To get the status of the stack type aws cloudformation describe-stacks --stack-name JenkinsStack --query 'Stacks[*].[StackId, StackStatus]'
at a command prompt.
Retrieve the public hostname of the Jenkins server. Open a terminal window and type the following command:
ec2 describe-instances --filters "Name=tag-value","Values=JenkinsStack" | jq .Reservations[].Instances[].PublicDnsName
Copy the public hostname
SSH into the instance, and then copy the temp password from /var/lib/jenkins/secrets/initialAdminPassword
.
On OS X, use the following command:
-i <full_path_to_key_file> ec2-user@<public_hostname>
For Windows instructions, see Connecting to your Linux Instance from Windows Using PuTTY
Run the following command:
cat /var/lib/jenkins/secrets/initialAdminPassword
Copy the output and logout of the instance by typing the following command:
ut
Amazon ECR is a private Docker container registry that you'll use to store your container images. For this example, we'll create a repository named hello-world in the us-west-2 (Oregon) region.
Create a ECR registry by running the following command:
ecr create-repository --repository-name hello-world --region us-west-2
Record the value of the URL of this repository because you will need it later.
Verify that you can log in to the repository you created (optional).
Because the Docker CLI doesn't support the standard AWS authentication methods, you need to authenticate the Docker client in another way so ECR knows who is trying to push an image. Using the AWS CLI, you generate an authorization token that you pass into the Docker login command.
$(aws ecr get-login)
aws ecr get-login | cmd
Note: This command will not succeed unless you have the Docker client tools installed on your machine and the Docker Virtual Machine is running. The output should say Login Succeeded.
Paste the public hostname of the Jenkins server from step 2.3 into a browser.
Paste the password you copied from the /var/lib/jenkins/secrets
directory from Step 2: Create a Jenkins Server (step 2.4) in the password field, and then choose Next.
Choose Install suggested plugins.
Create your first admin user by providing the following information:
<username>
<password>
<password>
<full_name>
<email_address>
Choose Save and finish.
Choose Start Using Jenkins.
Install Jenkins plugins.
In this step, you install the Amazon ECR plugin and the Cloudbees Docker build and publish plugin. You use the Amazon ECR plugin to push Docker images to an Amazon ECR repository. You use the Cloudbees Docker build and publish plugin to build Docker images.
In this step, you create an SSH key and import it into GitHub so we can login into Github over SSH.
If you?re running OS X, open terminal window. If you?re running Windows open a Git Bash shell. Run the following command:
keygen -t rsa -b 4069 -C your_email@company.com
Accept the file location and type a passphrase.
Ensure ssh-agent
is enabled by running the following command:
"$(ssh-agent -s)"
Add the SSH key to agent:
add ~/.ssh/id_rsa
Note: If you already have a key called id_rsa, choose another name.
Copy the contents of the id_rsa.pub
file to the clipboard. On OS X you can use the following command:
py < ~/.ssh/id_rsa.pub
Login to Github. If you don't have a Github account, follow the instructions posted here, https://help.github.com/articles/signing-up-for-a-new-github-account/.
In the top right corner of any page, choose your profile picture, then choose settings.
In the user settings sidebar, choose SSH and GPG keys.
Choose New SSH key or Add SSH key.
Type a title for the key.
Paste your key in the key field.
Click Add SSH key.
If prompted, confirm your GitHub password.
In this step you create a repository to store your dockerfile and all its dependencies.
Create a repository
Push code to your repository.
Open a terminal window (OS X) or a Git Bash shell (Windows).
Change the working directory to the root of the hello-world
repository you cloned earlier
Delete the hidden .git directory
If you're running OSX, type rm -fR .git
. Otherwise, type del /S /F /Q .git
.
Reinitialize the repository and push the contents to your new GitHub repository using SSH by running the following command:
init
Stage your files:
add .
commit -m "First commit"
If you are using SSH, run the following command:
remote add origin 'git@github.com:<your_repo>.git'
If you are using HTTPS, run the following command:
remote add origin 'https://github.com/<your_repo>.git'
Note: If you created the SSH key for GitHub on your machine, you can use either method. The HTTPS method requires that you to enter your GitHub username and password at the prompts.
push -u origin master
This project includes a file called taskdef.json. You can view it in the GitHub interface or with a text editor on your local machine. This file is the JSON representation of your ECS task definition.
Note: You must supply values for the family and name keys. These are used used later in the Jenkins execution scripts. You have to set the value of the image key to %REPOSITORY_URI%:v_%BUILD_NUMBER%
. You will use this mechanism to add the Jenkins build number as a tag to the Docker image.
Enable webhooks on your repo so Jenkins is notified when files are pushed
Jenkins (github plugin)
FQDN/github-webhook/
of your Jenkins server in the Jenkins URL field prepended by your Jenkins username & password.Note: If your Jenkins password containers special characters, you will need to encode them using URL escape codes.
Make sure there is a trailing / at the end of the URL. For example:
://username:password@FQDN/github-webhook/
In this step you will create a Jenkins Freestyle project to automate the tasks in your pipeline.
Create a freestyle project in Jenkins
Note: Make sure the name does not include spaces.
Choose freestyle project from the list of project types.
Choose OK.
Under the source code management heading, choose the git button.
In the repository URL field, type the name of your GitHub repository,
https://github.com/<repo>.git
In Credentials, choose the GitHub credentials you create in step 1 of this procedure.
Under build triggers, choose Build when a change is pushed to Github.
Scroll to the build section, and then choose Add a build step.
Choose execute shell.
In the command field, type or paste the following text:
in/bash
ER_LOGIN=`aws ecr get-login --region us-west-2`
CKER_LOGIN}'
Choose Add a build step, and then choose Docker Build and Publish.
In the repository name field enter the name of your ECR repository.
In the tag field, enter v_$BUILD_NUMBER
.
In the Docker registry URL, type the URL of your Docker registry. Use only the fully qualified domain name (FQDN) of the ECR repository you created earlier in Step 3: Create an ECR Registry.
Click Add a build step.
Choose execute shell.
In the command field, type or paste for following text. Be sure to replace the <ECR_repo>
and <cluster_name>
with the appropriate values from your environment:
in/bash
stants
ON=us-west-2
SITORY_NAME=<ECR_repo>
TER=<cluster_name>
LY=`sed -n 's/.*"family": "\(.*\)",/\1/p' taskdef.json`
=`sed -n 's/.*"name": "\(.*\)",/\1/p' taskdef.json`
ICE_NAME=${NAME}-service
re the repositoryUri as a variable
SITORY_URI=`aws ecr describe-repositories --repository-names ${REPOSITORY_NAME} --region ${REGION} | jq .repositories[].repositoryUri | tr -d '"'`
lace the build number and respository URI placeholders with the constants above
-e "s;%BUILD_NUMBER%;${BUILD_NUMBER};g" -e "s;%REPOSITORY_URI%;${REPOSITORY_URI};g" taskdef.json > ${NAME}-v_${BUILD_NUMBER}.json
ister the task definition in the repository
ecs register-task-definition --family ${FAMILY} --cli-input-json file://${WORKSPACE}/${NAME}-v_${BUILD_NUMBER}.json --region ${REGION}
ICES=`aws ecs describe-services --services ${SERVICE_NAME} --cluster ${CLUSTER} --region ${REGION} | jq .failures[]`
latest revision
SION=`aws ecs describe-task-definition --task-definition ${NAME} --region ${REGION} | jq .taskDefinition.revision`
ate or update service
"$SERVICES" == "" ]; then
"entered existing service"
RED_COUNT=`aws ecs describe-services --services ${SERVICE_NAME} --cluster ${CLUSTER} --region ${REGION} | jq .services[].desiredCount`
${DESIRED_COUNT} = "0" ]; then
SIRED_COUNT="1"
ecs update-service --cluster ${CLUSTER} --region ${REGION} --service ${SERVICE_NAME} --task-definition ${FAMILY}:${REVISION} --desired-count ${DESIRED_COUNT}
"entered new service"
ecs create-service --service-name ${SERVICE_NAME} --desired-count 1 --task-definition ${FAMILY} --cluster ${CLUSTER} --region ${REGION}
Note: Before saving this project, make sure that the variable CLUSTER is set to the name you gave your cluster, the REPOSITORY_NAME is set to the name of your ECR registry, and the REGION is set to the region where you created your ECS cluster.
Make a change to a file in your repository, e.g. readme.md, and push it to GitHub (from step 6: Create a GitHub Repository, repeat step 2, or use the Github interface). If you've configured things correctly, Jenkins pulls the code from your Git repository into a workspace, build the container image, pushes the container image to ECR, creates a task and service definition, and starts your service.
Confirm that your service is running.
getting-started
.hello-world-service
.You should see the following image in your browser: