kickstarter/cfn-flow

Name: cfn-flow

Owner: Kickstarter

Description: A practical workflow for AWS CloudFormation

Created: 2015-01-26 15:46:56.0

Updated: 2018-05-20 07:22:58.0

Pushed: 2016-08-25 20:54:37.0

Homepage:

Size: 112

Language: Ruby

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

cfn-flow

cfn-flow is a command-line tool for developing AWS CloudFormation templates and deploying stacks.

It provides a simple, standard, and flexible process for using CloudFormation, ideal for DevOps-style organizations.

Table of Contents

• Opinions
• Installation
• Key concepts
  • Services
  • Environments
  • Deploying
  • AWS credentials
• Configuration
• UX improvements
  • YAML > JSON
  • Embedded ruby in cfn-flow.yml
• Usage
• Working with stacks
  • Deploy (launch) a stack
  • List stacks for your service or environment
  • Inspect a stack
  • Show stack events
  • Delete a stack
• Common workflows
  • Deploying to production
• Launching a development environment
• Working with templates
  • Validate templates
  • Publish templates to S3
• License

Opinions

cfn-flow introduces a consistent, convenient workflow that encourages good template organization and deploy practices.

1. Optimize for happiness. The workflow should be easy & enjoyable to use.
2. Optimize for onboarding. The workflow should be simple to learn, understand, & debug.
3. Auditable changes. Know who changed what when & why. Leverage git history.
4. Immutable releases. The code in a release never changes. To make a change, launch a new stack.

The features & implementation of cfn-flow itself must also be simple. This follows the Unix philosophy of “worse is better“. cfn-flow values a simple design and implementation, and being composable with other workflows over handling every edge case out of the box.

See this introductory blog post for our motivation behind cfn-flow.

Installation

Via rubygems:

install cfn-flow

The git command is also needed.

Key concepts

cfn-flow works from a directory containing a cfn-flow.yml config file, and a CloudFormation template. Presumably your app code is in the same directory, but it doesn't have to be.

There are two key concepts for cfn-flow: services and environments.

Services

A service comprises a set of resources that change together. Each service has its own cfn-flow.yml config file. A service can be instantiated as several distinct environments.

For example, a WebApp service could have a CloudFormation template that creates an ELB, LaunchConfig, and AutoScalingGroup resources.

All the resources in a service change together. Deploying the WebApp service to an environment will create a new ELB, LaunchConfig, and AutoScalingGroup.

Resources that do not change across deploys are not part of the service (from cfn-flow's perspective). Say all WebApp EC2 servers connect to a long-running RDS database. That database is not part of the cfn-flow service because it is re-used across deploys. The database is a backing resource the service uses; not part of the service itself.

Environments

An environment is an particular instantiation of a service. For example, you could deploy your WebApp service to both a development and production environment.

cfn-flow is designed to support arbitrary environments like git supports arbitrary branches.

Pro tip: Use the CFN_FLOW_ENVIRONMENT environment variable in cfn-flow.yml config to use the environment in your template parameters. See Configuration for examples.

Deploying

Deployments consist of launching a new stack in a particular environment, then shutting down the old stack. For example:

flow deploy ENVIRONMENT --cleanup

This follows the red/black or blue/green deployment pattern.

After verifying the new stack is working correctly, the deployer is expected to delete the old stack.

To roll back a bad deploy, simply delete the new stack, while the old stack is running.

Although CloudFormation supports updating existing stacks, cfn-flow prefers launching immutable stacks. Stack updates are more difficult to test than new stacks; and there's less chance of a deployment error disrupting or breaking important resources.

AWS credentials

Set your AWS credentials so they can be found by the AWS SDK for Ruby (details here), e.g. using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

Configuration

cfn-flow looks for ./cfn-flow.yml for stack and template configuration. You can override this path by setting the CFN_FLOW_CONFIG_PATH environment variable to another path.

Here's a minimal cfn-flow.yml config file:

quired service name
ice: MyService

nimal configuration for launching the stack.
k:
Stack name uses embedded ruby to support dynamic values
ack_name: MyService-<%= Time.now.to_i %>
Required: *either* template_url or template_body
NB: template_body is a local path to the template
mplate_body: path/to/template.json
Alternatively:
template_url: https://MyS3Bucket.s3.amazonaws.com/MyPrefix/release/abc123/template.json

And here's a maximal config file:

ample cfn-flow.yml

ice: MyService

t the AWS region here to override or avoid setting the AWS_REGION env var
on: us-east-1


mplates

ese define where templates will get published.
$ cfn-flow publish --release my-cfn-template.json
Published url: https://MyS3Bucket.s3.amazonaws.com/My/S3/Prefix/<git sha>/my-cfn-template.json
lates:
cket: MyS3Bucket
_prefix: 'My/S3/Prefix'


acks

ese are the arguments passed when launching a new stack.
's nearly identical to the create_stack args in the ruby sdk, except
rameters and tags are hashes. See http://amzn.to/1M0nBuq

k:
Use the CFN_FLOW_ENVIRONMENT var & git sha in stack name
ack_name: MyService-<%= ENV['CFN_FLOW_ENVIRONMENT'] %>-<%= `git rev-parse --short HEAD`.chomp %>
# NB: template_body is a local path to the template
template_body: path/to/template.yml
template_url: http://...
parameters:
  # Your parameters, e.g.:
  vpcid: vpc-1234
  ami: ami-abcd

  ##
  # Use outputs from other stacks

  # This set the `load_balancer` parameter to the value of the
  # `elbname` output of `my-elb-stack`
  load_balancer:
    stack: my-elb-stack
    output: elbname

  # If you don't specify the output name, it's assumed to be same
  # as the parameter key:
  ssh_security_group:
    stack: my-bastion-stack

disable_rollback: true,
timeout_in_minutes: 1,
notification_arns: ["NotificationARN"],
capabilities: ["CAPABILITY_IAM"], # This stack does IAM stuff
on_failure: "DO_NOTHING", # either DO_NOTHING, ROLLBACK, DELETE
stack_policy_body: "StackPolicyBody",
stack_policy_url: "StackPolicyURL",
tags:
  # Whatever you want.
  # Note that `cfn-flow` automatically adds two tags: `CfnFlowService` and `CfnFlowEnvironment`
  TagKey: TagValue
  # Who launched this stack
  Deployer: <%= ENV['USER'] %>
  # Tag production and development environments for accounting
  BillingType: <%= ENV['CFN_FLOW_ENVIRONMENT'] == 'production' ?  'production' : 'development' %>

UX improvements

cfn-flow includes a few developer-friendly features:

YAML > JSON

cfn-flow lets you write templates in either JSON or YAML. YAML is a superset of JSON that allows a terser, less cluttered syntax, inline comments, and code re-use with anchors (like variables). YAML templates are transparently converted to JSON when uploaded to S3 for use in CloudFormation stacks.

Note that you can use JSON snippets inside YAML templates. JSON is always valid YAML.

Embedded ruby in cfn-flow.yml

To allow dynamic/programmatic attributes, use ERB in cfn-flow.yml. For example:

k:
me: my-stack-<%= Time.now.to_i %>
.
rameters:
git_sha: <%= `git rev-parse --verify HEAD`.chomp %>

Use stack outputs as parameters

cfn-flow lets you easily reference stack outputs as parameters for new stacks.

n-flow.yml
k:
rameters:
# Set my-param to the `my-param` output of `another-stack`
my-param:
  stack: another-stack

# Set my-param to the `my-output` output of `another-stack`
my-param:
  stack: another-stack
  output: my-output

Usage

Getting help:

t help
flow help

flow help COMMAND
g.:
flow help deploy

Launch a CloudFormation stack:

flow deploy production

Working with stacks

cfn-flow automatically sets two tags on any stack it launches:

Name | Example value — | — CfnFlowService | myapp CfnFlowEnvironment | production

These tags let cfn-flow associate stacks back to services & environments.

Deploy (launch) a stack

flow deploy ENVIRONMENT

Launches a stack in ENVIRONMENT. E.g. cfn-flow deploy production

Add the --cleanup option to be prompted to shut down other stacks in the environment.

List stacks for your service or environment

flow list [ENVIRONMENT]

Show all stacks running in your service, or just in an ENVIRONMENT.

r example:
n-flow list production

p-production-aaa (CREATE_COMPLETE)
p-production-bbb (CREATE_FAILED)

Inspect a stack

flow show STACK

Show the status of STACK.

Show stack events

flow events STACK

List events for STACK

Use the --poll option to poll for new events until the stack status is no longer *_IN_PROGRESS

Delete a stack

flow delete STACK

Deletes a stack.

r example:
n-flow delete myapp-production-aaa

Common workflows

Deploying to production

unch a new stack for the current git commit
n-flow deploy production
ching stack myapp-production-abc123
. wait for it to be ready

e the other stacks
n-deploy list production

p-production-abc123 CREATE_COMPLETE
p-production-xyz987 CREATE_COMPLETE

ut down the old stack
n-flow delete myapp-production-xyz987

Launching a development environment

Launch a new stack for myenv environment

flow deploy myenv

Working with templates

Validate templates

flow validate TEMPLATE [...]

Validates CloudFormation templates; does not persist to S3.

r example:
n-flow validate path/to/template.yml

Publish templates to S3

flow publish TEMPLATE [...]

Publish templates to S3 with immutable release names, or overwrite “dev names” for quicker testing.

Note: Publishing to S3 is only needed if you want to use nested stack resources, (that is, stacks that include other stacks).

r example:
n-flow publish path/to/template.yml
lidates & uploads templates to dev path
v var CFN_FLOW_DEV_NAME=aaron
g. https://mybucket.s3.amazonaws.com/myprefix/dev/aaron/mytemplate.yml

n-flow upload --release
lidates & uploads templates for current git sha
g. https://mybucket.s3.amazonaws.com/myprefix/deadbeef/mytemplate.yml

n-flow upload --release=v1.0.0
load templates for an arbitrary release name
g. https://mybucket.s3.amazonaws.com/myprefix/v1.0.0/mytemplate.yml

License

Copyright Kickstarter, Inc.

Released under an MIT License.