d3b-center/d3b-pedcbio-importer

Name: d3b-pedcbio-importer

Owner: Center for Data Driven Discovery in Biomedicine

Description: automates data import into pedcbio

Created: 2017-04-27 19:47:09.0

Updated: 2017-05-02 15:41:41.0

Pushed: 2017-05-25 22:28:58.0

Homepage:

Size: 18

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

!!!!!!! mostly rendered obsolete by d3b-pedcbimport !!!!!!!! *

to import the output of pipelines into pedcbio (see SOP document (TODO) and this google doc describing the input in greater detail - TODO: move doc to google team)

Jenkins

export aws_access_key_id=<...>
rt aws_secret_access_key=<...>

 export s3cmd_version="1.1.0~beta3-2"
   export pip_version="1.5.4-1ubuntu4"
   export git_version="1:1.9.1-1ubuntu0.4"
rt python_dev_version="2.7.5-5ubuntu3"
 export maven_version="3.0.5-1"
 export java8_version="8u111-b14-3~14.04.1"

 apt-get install -y git=1:1.9.1-1ubuntu0.4
clone https://github.com/d3b-center/d3b-pedcbio-importer

stall minimum dependencies
 d3b-pedcbio-importer/scripts/setup.sh

trieve input
pedcbio-importer/scripts/input.sh s3://test.cavatica/my_data_dir

nvert it to cbio format
pedcbio-importer/scripts/pipelines/wes.sh

lidate and load
pedcbio-importer/scripts/cbio.sh

notes:

• that by default, it creates and use ~/.pedcbio as working directory.
• mysql: initially the idea was to use a throw-away local mysql instance, and then just export all the relevant INSERT statements only (and therefore not write directly to the real db). unfortunately it seems cbio's import script also bundles some aggregations that require the entire set of data (and seems pretty hard to extract).

!!!!!!! mostly rendered obsolete by d3b-pedcbimport !!!!!!!! *