Name: nrc-ngs-downloader
Owner: Biological Informatics CoE @ Agriculture and Agri-Food Canada
Owner: Biological Informatics CoE @ Agriculture and Agri-Food Canada
Description: nrc-ngs-downloader is a software written in Python. This software explores the NRC-LIMS website, downloads all the sequence files, and keeps the meta data of all the sequences in a sqlite database.
Created: 2017-10-20 17:28:41.0
Updated: 2017-10-20 19:31:52.0
Pushed: 2017-10-31 12:56:40.0
Homepage: null
Size: 72
Language: Python
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
NRC-LIMS-Datadownloader is a software written in Python. This software explores the NRC-LIMS website, downloads all the sequence files, and keeps the meta data of all the sequences in a sqlite database.
The list of the tasks performed by the software:
Create and start the virtual enviroment
cd path/to/your/folder
virtualenv -p /path/to/python2.7 venv
source venv/bin/activate
Install the program and all the dependencies
pip install nrc_ngs_dl
Copy the sample configuration file config.ini.sample to config.ini and provide the required settings
cp venv/bin/config.ini.sample config.ini
vim config.ini
Run the program
cd path/to/your/folder
source venv/bin/activate
lims_downloader -c config.ini
Get the permission to access hcron1.science.gc.ca by opening an IT centre ticket with message:
HPC Dorval - Supercomputing - DC000131
Please register my account chz001 to use hcron on:
hcron1.science.gc.ca
Setup Passwordless Login
https://portal.science.gc.ca/confluence/display/SCIDOCS/SSH+Login+without+a+Password
mkdir -p ~/.ssh
chmod -R 700 ~/.ssh
cd ~/.ssh
ssh-keygen -q -t rsa
cat ~/.ssh/id_rsa.pub » ~/.ssh/authorized_key
Create the home for your hcron events:
bash-4.1$ mkdir -p ~/.hcron/hcron1.science.gc.ca/events
Create a hcron event file (file downloader as an example) :
bash-4.1$ cd ~/.hcron/hcron1.science.gc.ca/events
bash-4.1$ hcron-event downloader
Configure the event file to run the program nightly (file downloader as an example):
bash-4.1$ cat ~/.hcron/hcron1.science.gc.ca/events/downloader
as_user=
host=142.135.29.204
command=bash -l -c /space/project/grdi/eco/groups/mbb/chz001/dataDownloader/hcron_command.sh
notify_email=chunfang.zheng@canada.ca
notify_message=message from hcron
when_month=
when_day=
when_hour=2
when_minute=0
when_dow=*
template_name=
bash-4.1$
Command file(hcron_command.sh)
bash-4.1$ cat /space/project/grdi/eco/groups/mbb/chz001/dataDownloader/hcron_command.sh
!/bin/bash
echo “start at
date
” » $HOME/check_step
cd /space/project/grdi/eco/groups/mbb/chz001/dataDownloader
source venv/bin/activate
lims_downloader -c config.ini
echo “end atdate
” » $HOME/check_step
Getting Your Environment Right
from https://expl.info/display/HCRON/Getting+Your+Environment+Right
“The brute force way is to run a shell as a login shell, which will provide an environment almost equivalent to an interactive session”
command=bash -l -c ““
note: from my test, the trick works for host=142.135.29.204, but not for host=gpsc-in.science.gc.ca
Connect to hcron server
bash-4.1$ ssh hcron1.science.gc.ca
Load your hcron events
chz001@hcron1: hcron-reload
Three tables are maintained in this database. Tables will be updated when the program is run.