citusdata/test-automation

Name: test-automation

Owner: Citus Data

Description: Tools for making our tests easier to run

Created: 2016-12-13 08:02:28.0

Updated: 2018-05-24 20:29:06.0

Pushed: 2018-05-24 08:25:04.0

Homepage: null

Size: 952

Language: C

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

test-automation

Tools for making our tests easier to run. Automates setting up a cluster with CloudFormation and installs a script which automates setting up citus and everything required for testing citus.

Table of Contents

Getting Started

You can find more information about every step below in other categories. This list of commands show how to get started quickly. Please see other items below to understand details and solve any problems you face.

Basic Cluster Setup

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

ickly start a cluster of (1 + 2) c3.xlarge nodes at availability zone us-east-1b
dformation/create-stack.sh -k metin-keypair -s FormationMetin -n 2 -i c3.xlarge -a us-east-1b

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

tup your test cluster with PostgreSQL 10.1 and Citus master branch
use.postgres:10.2 use.citus:master setup.basic_testing

ts change some conf values 
pg.set_config:max_wal_size,"'5GB'"
pg.set_config:max_connections,1000

d restart the cluster
pg.restart

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "FormationMetin"
Running PgBench Tests

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

d your Github ssh key for enterprise (private) repo
add

ickly start a cluster of (1 + 3) c3.4xlarge nodes 
dformation/create-stack.sh -k metin-keypair -s PgBenchFormation -n 3 -i c3.4xlarge

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

is will run default pgBench tests with PG=10.1 and Citus Enterprise 7.1 and 7.2 release branches
d it will log results to pgbench_results_{timemark}.csv file
s, that's all :) You can change settings in fabfile/pgbench_confs/pgbench_default.ini
run.pgbench_tests

's possible to provide another configuration file for tests
ch as with this, we run the same set of default pgBench tests without transactions
run.pgbench_tests:pgbench_default_without_transaction.ini

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "PgBenchFormation"
Running Scale Tests

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

d your Github ssh key for enterprise (private) repo
add

ickly start a cluster of (1 + 3) c3.4xlarge nodes 
dformation/create-stack.sh -k metin-keypair -s ScaleFormation -n 3 -i c3.4xlarge

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

is will run scale tests with PG=10.1 and Citus Enterprise 7.1 and 7.2 release branches
d it will log results to pgbench_results_{timemark}.csv file
u can change settings in files under the fabfile/pgbench_confs/ directory
run.pgbench_tests:scale_test.ini
run.pgbench_tests:scale_test_no_index.ini
run.pgbench_tests:scale_test_prepared.ini
run.pgbench_tests:scale_test_reference.ini
run.pgbench_tests:scale_test_foreign.ini
run.pgbench_tests:scale_test_100_columns.ini

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "ScaleFormation"
Running PgBench Tests Against Citus Cloud

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

ickly start a cluster with no worker nodes
dformation/create-stack.sh -k metin-keypair -s PgBenchFormation -n 0 -i c3.4xlarge

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

e pgbench_cloud.ini config file with connection string of your Citus Cloud cluster
n't forget to escape `=` at the end of your connection string
run.pgbench_tests:pgbench_cloud.ini,connectionURI='postgres://citus:HJ3iS98CGTOBkwMgXM-RZQ@c.fs4qawhjftbgo7c4f7x3x7ifdpe.db.citusdata.com:5432/citus?sslmode\=require'

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "PgBenchFormation"
Running TPC-H Tests

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

d your Github ssh key for enterprise (private) repo
add

ickly start a cluster of (1 + 3) c3.4xlarge nodes 
dformation/create-stack.sh -k metin-keypair -s TPCHFormation -n 3 -i c3.4xlarge

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

is will run TPC-H tests with PG=10.1 and Citus Enterprise 7.1 and 7.2 release branches
d it will log results to their own files on the home directory. You can use diff to 
mpare results.
u can change settings in files under the fabfile/tpch_confs/ directory
run.tpch_automate

 you want to run only Q1 with scale factor=1 against community master,
u can use this config file. Feel free to edit conf file
run.tpch_automate:tpch_q1.ini

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "TPCHFormation"
Running TPC-H Tests Against Citus Cloud

On your local machine:

d your EC2 keypair's private key to your agent
add path_to_keypair/metin-keypair.pem

d your Github ssh key for enterprise (private) repo
add

ickly start a cluster with no worker nodes
dformation/create-stack.sh -k metin-keypair -s TPCHFormation -n 0 -i c3.4xlarge

en your cluster is ready, it will prompt you with the connection string, connect to master node
-A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com

On the coordinator node:

ovide your tpch config file or go with the default file
n't forget to escape `=` at the end of your connection string
run.tpch_automate:tpch_q1.ini,connectionURI='postgres://citus:dwVg70yBfkZ6hO1WXFyq1Q@c.fhhwxh5watzbizj3folblgbnpbu.db.citusdata.com:5432/citus?sslmode\=require'

On your local machine:

lete the formation
's a good practice to check deletion status from the cloud formation console
cloudformation delete-stack --stack-name "TPCHFormation"

Detailed Configuration

Starting a Cluster

You'll need to have installed the AWS CLI. Once that's installed you should configure it with aws configure. Once it's configured you can run something like:

cloudformation/create-stack.sh -k [your keypair name] -s MyStack

This will take some time (around 7 minutes) and emit some instructions for connecting to the master node.

The name you pass -s must be unique. There are more parameters you can pass such as -n, which changes the number of worker nodes which are launched. If you run create-stack with no parameters it will tell you more.

If you forget the name of your cluster you can get the list of active clusters by running:

aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE CREATE_IN_PROGRESS CREATE_FAILED --query "StackSummaries[*].{StackName:StackName,StackStatus:StackStatus}"

This will only list clusters which are in your default region. You can specify a region with the --region flag.

To get the hostname of the master you can run:

aws cloudformation describe-stacks --stack-name MyStack --query Stacks[0].Outputs[0].OutputValue

Connecting to the Master

  1. Make sure you have a running ssh-agent

    If you are running linux you either have a running ssh-agent or know that you don't ;). If you're using OSX you probably have a working ssh-agent but it's possible Apple has pushed an update and broken things.

    If it is running then the SSH_AUTH_SOCK environment variable will be set:

    n@rhythm:~$ echo $SSH_AUTH_SOCK
/ssh-s1OzC5ULhRKg/agent.1285

    If your SSH_AUTH_SOCK is empty then Google is your friend for getting an ssh-agent to start automatically when you login. As a temporary fix you can run exec ssh-agent bash:

    n@rhythm:~$ echo $SSH_AUTH_SOCK

n@rhythm:~$ exec ssh-agent bash
n@rhythm:~$ echo $SSH_AUTH_SOCK
/ssh-et5hwGiqPxUn/agent.31580

  2. Add your keypair's private key to your ssh agent

    When you created your EC2 keypair it gave you a .pem file for safekeeping. Add it to your agent with:

    n@rhythm:~$ ssh-add Downloads/brian-eu.pem
tity added: Downloads/brian-eu.pem (Downloads/brian-eu.pem)

    find ~ -name '*.pem' can help you find your key. Running ssh-add ~/.ssh/id_rsa will not work, you must use the keypair you received from EC2.

  3. If you plan on checking out private repos, add that private key to your agent as well

    When you check out a private repo on the master/workers, they will reach back and talk to your ssh-agent in order to authenticate as you when talking to Github. You can find your list of keys here. One of them should be added to ssh-agent with another ssh-add command. Personally, I run:

    n@rhythm:~$ ssh-add
tity added: /home/brian/.ssh/id_rsa (/home/brian/.ssh/id_rsa)

    Note: If you use 2-factor authentication that will not change these instructions in any way. For the purposes of ssh access your key counts as a factor. If you have a passphrase on your key you will be prompted for it when you run ssh-add, that's the second factor. If you do not have a passphrase on your key, well, Github has no way of knowing that.

  4. ssh into the master in a way which allows it to use your local ssh-agent

    The create-stack.sh script should have given you a connection string to use. You can also find the hostname of the master node in the cloudformation control panel in the “outputs” section. There's a command above which tells you how to get the hostname without going to the control panel.

    You'll connect using a string like:

    -A ec2-user@ec2-35-156-235-11.eu-central-1.compute.amazonaws.com

    The -A is not optional. It is required so the master node can ssh into the worker nodes and so the nodes can checkout private repos.

    That means you should not pass the -i flag into ssh:

    reful, this is wrong!
-i Downloads/brian2.pem ec2-user@[hostname]

    If you need to pass the -i flag in order to connect this means the key is not in your agent. That means the master node will not be able to ssh into the worker nodes when you later run fab.

It's unfortunate that you have no flexibility here. This restriction which will hopefully be lifted in the future.

Example fab Commands

Use fab --list to see all the tasks you can run! This is just a few examples.

Once you have a cluster you can use many different variations of the “fab” command to install Citus:

Tasks, and Ordering of Tasks

When you run a command like fab use.citus:v7.1.1 setup.basic_testing you are running two different tasks: use.citus with a v7.1.1 argument and setup.basic_testing. Those tasks are always executed from left to right, and running them is usually equivalent to running them as separate commands. For example:

is command:
setup.basic_testing add.tpch
s exactly the same effect as this series of commands:
setup.basic_testing
add.tpch

An exception is the use namespace, tasks such as use.citus and use.postgres only have an effect on the current command:

is works:
use.citus:v7.1.1 setup.basic_testing
is does not work:
use.citus:v7.1.1  # tells fabric to install v7.1.1, but only works during this command
setup.basic_testing  # will install the master branch of citus

use tasks must come before setup tasks:

is does not work!
nce the `setup` task is run before the `use` task the `use` task will have no effect
setup.basic_testing use.citus:v.7.1.1

Finally, there are tasks, such as the ones in the add namespace, which asssume a cluster is already installed and running. They must be run after a setup task!

Task Namespaces

use Tasks

These tasks configure the tasks you run after them. When run alone they have no effect. Some examples:

use.citus:v7.1.1 setup.basic_testing
use.enterprise:v7.1.1 setup.enterprise
use.debug_mode use.postgres:10.1 use.citus:v7.1.1 setup.basic_testing

use.debug_mode passes the following flags to postges' configure: --enable-debug --enable-cassert CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer"

use.asserts passes --enable-cassert, it's a subset of use.debug_mode.

add Tasks

It is possible to add extra extensions and features to a Citus cluster:

For a complete list, run fab --list.

As described above, you can run these at the same time as you run setup tasks:

pg Tasks

These tasks run commands which involve the current postgres instance.

If you want to use a literal comma in a command you must escape it (this applies to all fab tasks)

Using pg.set_config it's possible to get yourself into trouble. pg.set_config uses ALTER SYSTEM, so if you've broken your postgres instance so bad it won't boot, you won't be able to use pg.set_config to fix it.

To reset to a clean configuration run this command:

run Tasks

In order to run pgbench and tpch tests automatically, you can use run.pgbench_tests or run.tpch_automate. If you want to use default configuration files, running commands without any parameter is enough.

To change configuration file for pgbench tests, you should prepare configuration file similar to fabfile/pgbench_confs/pgbench_config.ini.

To change the configuration file for tpch tests, you should prepare configuration file similar to fabfile/tpch_confs/tpch_default.ini.

Advanced fab Usage

By default your fab commands configure the entire cluster, however you can target roles or individual machines.

You can also ask to run arbitrary commands by adding them after --.

Using Multiple Citus Installations, pg-latest

Some kinds of tests (such as TPC-H) are easier to perform if you create multiple simultanious installations of Citus and are able to switch between them. The fabric scripts allow this by maintaining a symlink called pg-latest.

Most tasks which interact with a postgres installation (such as add.cstore or pg.stop) simply use the installation in pg-latest. Tasks such as setup.basic_testing which install postgres will overwrite whatever is currently in pg-latest.

You can change where pg-latest points by running fab set_pg_latest:some-absolute-path. For example: fab set_pg_latest:/home/ec2-user/enterprise-installation. Using multiple installations is a matter of changing your prefix whenever you want to act upon or create a different installation.

Here's an example:

set_pg_latest:/home/ec2-user/pg-960-citus-600
use.postgres:9.6.0 use.citus:v6.0.0 setup.basic_testing
set_pg_latest:/home/ec2-user/pg-961-citus-601
use.postgres:9.6.1 use.citus:v6.0.1 setup.basic_testing
u now have 2 installations of Citus!
pg.stop  # stop the existing Citus instance
set_pg_latest:/home/ec2-user/pg-960-citus-600  # switch to using the new instance
pg.start  # start the new instance
w you've switched back to the first installation

e above can be abbreviated by writing the following:
pg.stop set_pg_latest:/home/ec2-user/pg-960-citus-600 pg.start
This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.