buildkite/lifecycled

Name: lifecycled

Owner: Buildkite

Description: A daemon for responding to AWS AutoScaling Lifecycle Hooks

Created: 2016-04-07 06:29:31.0

Updated: 2018-04-30 00:59:25.0

Pushed: 2018-04-30 00:59:24.0

Homepage:

Size: 1405

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Lifecycled - Gracefully handle EC2 scaling events

Lifecycled is designed to run on an AWS EC2 instance and listen for various state change mechanisms:

• AWS AutoScaling
• Spot Instance Termination Notices

A handler script is provided to gracefully handle these actions before shutdown.

Installing

Either install with go get -u github.com/buildkite/lifecycled or download a binary release for Linux or Windows.

Running

Check out the Cloudformation Template for an example of how to setup an autoscaling group with the right permissions and an SNS topic. Once you have the SNS topic, run lifecycled on your aws instance:

cycled --sns-topic arn:aws:sns:us-east-1:11111111:lifecycled-test-1501806648-LifecycleTopic-UTAZ7PQOA32Q

Autoscaling Hooks

Lifecycle events are consumed from an SQS queue and a corresponding hook is executed. Whilst the hook is executing lifecycled sends heartbeats to the Autoscaling group to stall further action. When the hook completes executing, if it completes successfully the lifecycle action is completed successfully, if the hook returns a non-zero exit code then the lifecycle action is abandoned.

The handler script gets passed the event and the instance id, e.g: autoscaling:EC2_INSTANCE_TERMINATING i-001405f0fc67e3b12

Spot Termination

These notices are consumed by polling the local metadata url every 5 seconds. They are passed to the handler script as a custom event, the instance id and the timestamp, e.g ec2:SPOT_INSTANCE_TERMINATION i-001405f0fc67e3b12 2015-01-05T18:02:00Z

Cleaning up Leftover SQS Queues

The lifecycled daemon should clean up the per-instance SQS queues that are created when it shuts down, but there has been a bug where this does not happen (See https://github.com/buildkite/lifecycled/issues/12). To mitigate this, you can run a cleanup tool:

et -u github.com/buildkite/lifecycled/tools/lifecycled-queue-cleaner
cycled-queue-cleaner