histograph/aws

Name: aws

Owner: Histograph

Description: AWS scripts

Created: 2015-09-11 14:31:56.0

Updated: 2016-10-26 02:32:18.0

Pushed: 2016-03-17 12:57:42.0

Homepage: null

Size: 86

Language: Shell

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

What is this

A tool to assist in creation of the histograph.io stack on AWS.

We create a cloud-init configuration file (see cloudinit.py for implementation details). This configuration is read by the machine on startup and it will

(More info on cloud-init here)

This configuration is then gzipped, base64 encoded and passed to EC2 when launching an instance. Launching, tagging, waiting and other AWS functionality is found in aws.py which implements this using boto3.

Finally, setting up the various nodes in done using shell scripts, see scripts/

Amazon Linux (CentOS) scripts:

Node processes are kept running using forever, which I find slightly irritating. Some functions to create init scripts for it are found in scripts/library-functions.sh

Finally, after starting a machine the systems tries to login with SSH and when sucessful it will tail /var/log/cloud-init-output.log.

Config

Common settings are stored in cluster.yaml, it looks like this:

s:
er-name: "~/.ssh/id_rsa.pub ssh key contents"

-conf:
default image, amazon linux
chine-image: ami-a6b0b7bb

which VPC and subnet you want the instances to start in
c: vpc-6865cc01
bnet: subnet-71b36f0a
gion: eu-central-1

associated security group
curity-group: sg-baac24d3

default instance type
stance-type: t2.micro

s:
i:
# when set, this will create a system user with this name/group
app-user: histograph

# machines are assigned a static IP (for now),
# make sure this corresponds with your VPC/subnet settings
ip-address: 10.0.0.51

# override the default instance-type
instance-type: t2.small

# each of these scripts is saved to /root/ with permissions 0700
# and then executed, in the order specified here
scripts:
  # this scripts registers the node repo
  - install-nodejs-repository.sh

  # some functions, to install node, forever, create init scripts
  - library-functions.sh

  # setup histograph API
  - install-histograph-api.sh
re:
ip-address: 10.0.0.52
instance-type: t2.micro
app-user: histograph
scripts:
  - install-nodejs-repository.sh
  - library-functions.sh
  - install-histograph-core.sh
dis:
ip-address: 10.0.0.53
instance-type: t2.micro
scripts:
  - install-redis.sh
o4j:
ip-address: 10.0.0.54
instance-type: m3.large
# we run Neo4J on Debian 8 (jessy)
machine-image: ami-b092aaad
scripts:
  - install-neo4j.sh

For each user you want to grant access to the instance: Replace 'name' with the username you wish to grant access. Replace 'key' with the public key (from the pem file you created) for the user (on aws).

The configuration and scripts work together, so beware.

Installation

This requires python3,

brew install python3

You should prefer to run python in a jail called virtualenv.

pip3 install virtualenv

Create jail in subdir venv/, ensure Python version 3.

virtualenv -p python3 venv

Enter the jail through shell magic.

source venv/bin/activate

(or source venv/bin/activate.fish if you are using fish)

And now install the requirements

pip3 install -r requirements.txt

Ensure that you have AWS credentials setup

# ~/.aws/credentials
[default]
aws_access_key_id = YOUR_KEY
aws_secret_access_key = YOUR_SECRET

Also set up a default region (in e.g. ~/.aws/config):

# ~/.aws/config
[default]
region = eu-central-1

Now you can run the scripts.

If you have AWS environment variables set, these will be picked up by boto3. Beware they might conflict with ~/.aws/credentials. In bash you can clear them like this:

unset AWS_SECRET_ACCESS_KEY
unset AWS_ACCESS_KEY_ID

Workflow

Creating a machine from the config above:

./aws-tool create cluster.yaml redis

By default a 'dry run' is enabled. Change the DryRun=True value to False or comment out the option in aws.py to do a proper run.

Then wait… You will hopefully see the logs. Check if all went fine, then create next instance.

./aws-tool create cluster.yaml neo4j
./aws-tool create cluster.yaml api
./aws-tool create cluster.yaml core

This is not ideal, but good enough for now.

Finding base images

We use two kinds of images, Debian (Neo4J) and Amazon Linux (everything else).

To find all official Debian owned images, run this command. You should prefer HVM over paravirtual and please note that image identifiers are region dependent.

aws --region eu-central-1 ec2 describe-images --owners 379101102735 \
--filters "Name=name,Values=debian-*" \
--query "Images[*].[Architecture,ImageId,VirtualizationType,Name]" \
--output text

Set up and tear down staging environment

Run the script ./setup_staging.sh. This script will clone ElasticSearch's repository and create a new cluster pointing at it.

Then it will clone the disk of Neo4j's production instance and create and launch an instance based on the cloned disk. The id of the instance to clone is specified in the config file cluster_staging.yml, see example below.

Finally, it will launch and install the following machines:

s-staging
-staging
staging

To tear down the staging environment, run the script teardown_staging.sh. Tearing down is not fully automated, you need to perform some actions as specified in the script's output.

Config

The setting for the staging are stored in cluster_staging.yaml, it looks like this:

s:
just place your ~/.ssh/id_*.pub key contents here
if you dont have such a public key, run `ssh-keygen`
efano: "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxIMaALlvQooxnPj9NiDMyhMap7IX0j4Yq/LHEZc+c4sj/KQXjovM845F/H8yj9r5Ibw4YWzkKROB9fRW+ZYcR43dlbDmlf9hZO/QYtWuk3ZC5DOvqBQ2+/Ume2sU70nyhR3u+Y60cXUhpSrr5sf1yTiQweNk8VhfezjXFCpcEUhEFtBYHiVrGY4wCIsx9IZ63Pr41A+pYiqMINXgxw/cB9s4uMIyNBG8NIiaPJC3MJgpfaX3FXLKT9BefrJP3kWBh1jTMfYyDiKCgerMh/2d5YPSiWDt1R3SPh9jQ0WgckCQsbMgl8df9Um+8iEO63iI88PIw11sEAvlE/wlUN03kQ== stefano@waag"

-conf:
default image, amazon linux
chine-image: ami-a6b0b7bb
bnet: subnet-1a960e73 #staging subnet
curity-group: sg-b9ed94d0
c: vpc-6865cc01
gion: eu-central-1
stance-type: t2.micro
p-user: histograph

s:
i-staging:
ip-address: 10.0.1.51
instance-type: t2.small
app-user: histograph
scripts:
  - install-nodejs-repository.sh
  - library-functions.sh
  - staging/install-histograph-api.sh
re-staging:
ip-address: 10.0.1.52
instance-type: t2.micro
app-user: histograph
scripts:
  - install-nodejs-repository.sh
  - library-functions.sh
  - staging/install-histograph-core.sh
dis-staging:
ip-address: 10.0.1.53
instance-type: t2.micro
scripts:
  - install-redis.sh

ese two are cloned from production i/o created from scratch
j:
-address: 10.0.1.54
stanceId: i-af0db813
nstance-type: m3.large
achine-image: ami-b092aaad # debian 8 (jessy)
cripts:
 - install-neo4j.sh
lasticsearch:
 ip-address: 10.0.1.55
 instance-type: m3.xlarge
 machine-image: ami-b092aaad # debian 8 (jessy)
 scripts:
   - install-elasticsearch.sh

Copyright © 2015 Waag Society.

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.