Duke-GCB/lando

Name: lando

Owner: Duke Center for Genomic and Computational Biology

Description: Lando's not a man, he's a system

Created: 2016-11-22 17:11:22.0

Updated: 2016-11-29 14:34:35.0

Pushed: 2018-01-10 15:16:27.0

Homepage:

Size: 993

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

lando

I'm the administrator of this facility

Service that runs cwl workflows on VMs in a openstack cloud.

The project is made up of 3 scripts:

• lando - server that spawns VMs and sends messages for them to run job steps
• lando_worker - program that runs inside the VMs that listens for messages to run different steps
• lando_client - program that can send lando the start/cancel message (only used for testing purposes)

The major external components are:

• Rabbitmq - a queue were messages are placed for lando and lando_worker to consume.
• bespin-api - a REST API that contains data about jobs to run and will put start_job and cancel_job in the queue for lando. https://github.com/Duke-GCB/bespin-api
• Openstack - a cloud where VMs are created and will have lando_client run in them to execute workflows.

Message Flow

Running job message flow (omitting Rabbitmq):

1. bespin-api posts a start_job message for lando

2. lando tells Openstack to creates VM that runs lando_worker

3. lando posts a stage_job message for lando_worker

4. lando_worker downloads files for the job

5. lando_worker sends stage_job_complete to lando

6. lando posts a run_job message for lando_worker

7. lando_worker runs the CWL workflow for the job

8. lando_worker sends run_job_complete to lando

9. lando posts a save_output message for lando_worker

10. lando_worker runs the CWL workflow for the job

11. lando_worker sends save_output_complete to lando

12. lando tells Openstack to terminate the lando_worker VM

Additionally lando reads and updates bespin-api job table as the job progresses.

Setup

Assumes you have installed Python 2.7, Openstack, Rabbitmq.

Install lando-messaging and lando.

install git+git://github.com/Duke-GCB/lando.git

Install Bespin-api.

Follow the instructions to install the lando_api branch: https://github.com/Duke-GCB/bespin-api/blob/master/README.md

Create job in Bespin-api

Using the bespin superuser you created in the previous step go into the admin interface and setup a job.

Create lando config files

There are two config files that are used by lando.

• /etc/lando_config.yml - this is the main configuration file used by the server program(lando).
• /etc/lando_worker_config.yml - this is the configuration file used by the worker. When using Openstack the server program creates and puts the worker's config file on the VM in the correct location.

Sample /etc/lando_config.yml file:

bbitmq settings
_queue:
st: 10.109.253.74       # ip address of the rabbitmq
ername: lando           # username for lando server
ssword: secret1         # password for lando server
sten_queue: lando       # queue that lando server should listen on  
rker_username: worker   # username for lando worker
rker_password: secret2  # password for lando worker

enstack VM settings
ettings:
rker_image_name: lando_worker    # Name of the image that has lando installed and lando_worker setup to run as a service
h_key_name: jpb67                # Name of the openstack SSH key to install on the worker
twork_name: selfservice          # Openstack network name to add the vm onto
locate_floating_ips: false       # Should each worker VM get allocated a floating IP defaults to false
oating_ip_pool_name: ext-net     # Pool of floating IP's that will have one assigned to the VM
fault_flavor_name: m1.small       # Flavor of image to use by default when creating a VM

neral Openstack settings
d_settings:
th_url: http://10.109.252.9:5000/v3
ername: jpb67                        
ssword: secret3  
er_domain_name: Default               
oject_name: jpb67               # name of the project we will add VMs to
oject_domain_name: Default

spin job API settings
in_api:
l: http://localhost:8000/api
ername: jpb67
ssword: secret4

If you are running with valid openstack credentials you will not need to create a /etc/lando_worker_config.yml file. The lando service does this for you.

Add users to Rabbitmq

itmqctl add_user lando secret1
itmqctl set_permissions -p / lando  ".*" ".*" ".*"

itmqctl add_user worker secret2
itmqctl set_permissions -p / worker  ".*" ".*" ".*"

Running with Openstack

You can start lando by simply running lando where it can see the /etc/lando_config.yml config file.

Running without Openstack

Turn on option to fake cloud service in /etc/lando_config.yml

At the end of /etc/lando_config.yml add the following:

_cloud_service: True

This will cause lando to print a message telling you to run lando_worker.

Sample /etc/lando_worker_config.yml file for fake cloud service:

: 10.109.253.74
name: worker
word: secret2
e_name: local_worker  

The queue name local_worker is always used for workers when fake_cloud_service is True in /etc/lando_config.yml.

If you are running on osx you may need to specify custom --tmpdir-prefix and --tmp-outdir-prefix flags for cwl. You can replace the default cwl-runner command by adding lines similar to these:

base_command:
"cwl-runner"
"--debug"
"--tmpdir-prefix=/Users/jpb67/Documents/work/tmp"
"--tmp-outdir-prefix=/Users/jpb67/Documents/work/tmp"

Run lando client

This command will put a job in the rabbitmq queue for the lando server to receive. This reads the config from /etc/lando_config.yml.

o_client start_job 1

This command is just meant for testing purposes. In a typical use case this message would be queued by bespin-api.

Run lando server

This will listen for messages from the 'lando' rabbitmq queue. This reads the config from /etc/lando_config.yml.

o

It should display that it has received a message to run a job. Since we have set the fake_cloud_service: True in /etc/lando_config.yml instead of trying to launch a vm it should print this message: Pretend we create vm: local_worker. Finally it should put a staging data message in the worker's queue.

Run lando worker

This reads the config from /etc/lando_worker_config.yml. It should talk back and forth with lando server staging data, running, job and storing output.

o_worker

lando_worker should terminate once it completes the job.