SciLifeLab/bcbio-nextgen-deploy

Name: bcbio-nextgen-deploy

Owner: Science For Life Laboratory

Description: Suite of scripts to automatically deploy nextgen pipeline

Created: 2012-10-19 09:11:31.0

Updated: 2013-12-17 12:26:11.0

Pushed: 2013-12-17 12:26:10.0

Homepage: null

Size: 164

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Overview

This is a set of scripts to perform an automatic deployment of the next gen sequencing analysis pipeline. You can do it in three ways:

• Configure and install the pipeline in UPPMAX (see detail below)
• Perform a local installation of the pipeline within a virtual machine.
• Install the pipeline locally in your machine

Current build status:

Installation in UPPMAX

Requirements

To use this script you need python 2.7 or higher. In UPPMAX you can just do a:

    module load python/2.7

to get python 2.7.

Install

To configure and install the pipeline in UPPMAX you just have to execute the following command (after cloning this repository):

        python deploy_non_root.py install

This will:

• Set up virtualenvwrapper creating and configuring a proper virtual environment
• Set up the config directory containing the configuration files for the pipeline
• Set up custom modules with correct software versions
• Set up bcbb pipeline code properly
• Download and install SciLifeLab utility scripts
• Run the testsuite in background:
  • This will run a job that will write the tests results in ~/opt/bcbb/nextgen/tests/tests_results.out
  • You'll find also an XML file with a summary of the test results in ~/opt/bcbb/nextgen/tests/nosetests.xml

You can also install a specific version of the pipeline using the -v option. To do so, you just have to specify a commit sha when running the installation script, like this:

    python deploy_non_root.py -v 1116 install

And the script will checkout for this commit before installing.

Notes

If you don't want to run the test suite after the installation, you can specify this:

python deploy_non_root.py --no-tests install

Uninstall

To completely uninstall the pipeline and all its configuration files/directories execute:

        python deploy_non_root.py uninstall

Local installation within a virtual machine

Requirements

All the software and dependencies needed by the pipeline are automatically downloaded and installed. However, you need to install the necessary tools for running this script:

• Vagrant - To download and install the Virtual Machine
• Python fabric - To run the main script

Furthermore, in order to correctly execute all the tests, please have in mind that 2GB of memory are reserved for the Virtual Machine when it's on.

Installing the pipeline and running the tests

To install the virtual machine, the pipeline within it and run the tests, just type:

        fab -f deploy_on_vm.py install

And the script will start installing the virtual machine and the pipeline. When the installation is finished, the tests will run.

Installing in several virtual machines

If you have a cluster of vagrant virtual machines configured in host-only networking mode, you can install the pipeline in all of them just specifying the “comma separated” list of IPs:

fab -f deploy_on_vm.py -H 10.10.10.3,10.10.10.4 install

This will install the pipeline in all of the specified machines.

Notes

For the multi-VM installation

• It is necessary that the virtual machines specified in the list are Vagrant VMs, or at least, they need to have a user vagrant (with password also vagrant), as the fabric calls are executed with this user.
• If you're using your own machines (not creating a new one with the script), please take into account that the script has been tested and prepared for Ubuntu 12.04, other operating systems are not officially supported.
• When installing in a multi-VM environment the test suite is not run. You can run the tests manually connecting to each VM and following the instructions descrived in the pipeline documentation. This is due to the fact that the tests are quite large. If we run the test suite in each machine after the installation, the tests executions on the different machines will probably overlap, leading to a very high system load.

About Vagrant in general

After the installation has finished and the tests have run, you can connect to the VM and take a look at the tests results using the command:

        vagrant ssh

After that, you can turn off the virtual machine with:

        vagrant halt

You can always turn on the virtual machine again with the command:

        vagrant up

So you'll have a Virtual Machine whith the pipeline installed to test it whenever you want without installing any software in your system but vagrant and fabric.

For more information about how vagrant works, refear to the vagrant guide

Local installation

To install the pipeline in your machine, execute the command:

        python deploy_non_root.py install

NOTE: The installation of the pipeline itself doesn't need root permissions, but in this case, you have to take care of the installation of all the software dependencies of the pipeline. Read the pipeline documentation for more information about what software do you need.

Important note about the uninstall function

This will completely remove the directories and configuration files created during the pipeline installation, including master virtualenv with all python dependencies installed, and restoring all the configuration files modified during the installation (.bashrc, etc)

After the installation

In order to be able to use and install any python module without requiring root permissions, a python virtual environment is installed. This virtual environment is called master, and before using the pipeline you have to activate it:

    workon master

Then, the pipeline can be used normally.