starkandwayne/pipeline-templates

Name: pipeline-templates

Owner: Stark & Wayne

Description: Generalized pipeline templates for various objectives, and a repipe script for compiling them.

Created: 2016-03-04 20:58:16.0

Updated: 2018-05-21 08:49:08.0

Pushed: 2018-04-25 18:55:21.0

Homepage: null

Size: 766

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Pipeline Templates

This repository collects up all that tribal wisdom we've gained from building Concourse Pipelines for various different purposes. Each template provides a base ci/pipeline.yml that structures the pipeline, ci/settings.yml to override any parameters defined by the template, a set of scripts (in ci/scripts/) that are referenced from the pipeline, and a ci/repipe utility for putting all the pieces together into a functioning Concourse pipeline.

Setup is straightforward:

clone https://github.com/starkandwayne/pipeline-templates
ipeline-templates
tup boshrelease ~/bosh/my-new-boshrelease

Alternatively, you can initialize the pipeline from inside the target repository:

ode/my-buildpack
de/pipeline-templates/setup buildpack

The first argument to setup is the template you want to use. Currently available templates:

• docker/base
• go
• boshrelease
• buildpack

Don't let the name fool you! ./setup can also be used to update an existing templated pipeline ci/ directory to pick up new changes made to the templates.

Once you've set up your repository, you'll need to fill in your ci/settings.yml file with any parameters the template required:

ode/my-project
epipe # Attempt to update the pipeline config,
      # spitting out errors for missing parameters
i/settings.yml # fill in the missing parameters
epipe # Deploy the pipeline config!

Dependencies

The ci/repipe script uses Spruce to merge ci/pipeline.yml and your bespoke ci/settings.yml.

On MacOS/Homebrew:

 install starkandwayne/cf/spruce

On Debian/Ubuntu:

 -q -O - https://raw.githubusercontent.com/starkandwayne/homebrew-cf/master/public.key | apt-key add -
 "deb http://apt.starkandwayne.com stable main" | tee /etc/apt/sources.list.d/starkandwayne.list
get update

get install spruce

And Now, The Templates!

docker/base and docker/ext-tests

For building Docker images, with tests. This template comes in two flavors: docker/base (where the unit tests are inside the Docker image) and docker/ext-tests, where the tests live outside of the image.

go

Takes a Go software project repository, runs unit tests and (when the manual shipit job is run) releases to Github.

boshrelease

Tries to create a BOSH release from the repository, upload it to a hosted BOSH-lite for viability testing, and (when the manual shipit job is run) releasing it to Github with a tarball artifact, and also uploading that release tarball to S3.

buildpack

Runs unit and integration tests on a Cloud Foundry buildpack, and can release it to Github.

ci/settings.yml

You will need to customize your pipeline with information about your CI, your Amazon AWS credentials + S3 bucket for storing assets + version file, your Slack account, etc.

You will create and maintain ci/settings.yml for this.

Try very very hard to not modify ci/pipeline.yml. Instead, use ./setup to update ci/pipeline.yml with new changes from this repo. If you do need to modify ci/pipeline.yml please feel welcome to submit PRs to this repo so that we can merge them and share them with everyone.

At Stark & Wayne we store our credentials for pipelines in Vault. We use the spruce syntax (( vault "path1" )) to dynamically fetch these values during ci/repipe. Recently Concourse CI has added native support for Vault, so we will investigate this in the future. Or you could try to use it and let us know how it goes!

Here is an example ci/settings.yml from the eden CLI project (uses the go template):

:
me: eden
rget: sw
l:     https://ci.starkandwayne.com

itial_version: 0.5.0

:
binary: eden
cmd_module: .

s:
access_key: (( vault "secret/aws/starkandwayne-s3:access" ))
secret_key: (( vault "secret/aws/starkandwayne-s3:secret" ))
region_name: eu-central-1

ack:
webhook: (( vault "secret/pipelines/eden/slack:webhook" ))
channel: "#eden" # https://openservicebrokerapi.slack.com/messages/C6Y5A2N8Z/
username: starkandwayne-ci
icon:     https://www.starkandwayne.com/assets/images/shield-blue-50x50.png

thub:
owner: starkandwayne
repo: eden
access_token: (( vault "secret/pipelines/shared/github:access_token" ))
private_key: (( vault  "secret/pipelines/shared/github:private_key" ))

Bonus, we use https://github.com/starkandwayne/safe as our CLI to interact with Vault.

For example, to populate the (( vault "secret/pipelines/eden/slack:webhook" )) value in Vault:

 set secret/pipelines/eden/slack webhook=https://hooks.slack.com/services/T2S1X7xxx/B6Y5A7xx/0nP7jxxx