Name: amadeus_fetcher
Owner: unicef
Description: MagicBox
Created: 2016-09-13 15:29:28.0
Updated: 2017-09-28 15:52:46.0
Pushed: 2017-01-14 02:04:55.0
Homepage:
null
Size: 16
Language: JavaScript
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Email | Most Recent Commit | # Commits |
|---|
README
Amadeus Fetcher
Downloads raw mobility data from amadeus sftp and stores to Azure blob. This data will be aggregated by admin.
- This is a component of MagicBox
- Fetches names of collections on sftp (Each directory in root is a collection).
- Creates a container in storage account (noted in config.js) for each collection if it doesn't already exist.
- Compares files in blobs with files in sftp collection.
- New files in sftp are downloaded to local / or mounted cloud and then uploaded to blob.
Install
- Clone repo and cd into project
- Run
cp config-sample.js config.js; mkdir data
- Add ssh and Azure blob storage credentials to config
- Run npm install
- Run node main.js
Run
nohup node main.js nohup.out 2>&1&
TODO
- Create cron job to run weekly
