hasadna/revolving-doors-pipelines

Name: revolving-doors-pipelines

Owner: The Public Knowledge Workshop

Description: Pipelines for scraping and data research for revolving doors project

Created: 2018-02-24 10:33:45.0

Updated: 2018-02-24 10:48:53.0

Pushed: 2018-02-24 13:36:12.0

Homepage: null

Size: 21

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Revolving Doors Pipelines

Visualization and research aids to work with revolving doors data

Uses the datapackage pipelines framework

Running the pipelines locally

Most pipelines are available to run locally with minimal infrastructure dependencies.

Install some dependencies (following works for latest version of Ubuntu):

 apt-get install -y python3.6 python3-pip python3.6-dev libleveldb-dev libleveldb1v5
 pip3 install pipenv

install the pipeline dependencies

nv install

activate the virtualenv

nv shell

List the available pipelines

run a pipeline

run <PIPELINE_ID>

running using docker

er pull orihoch/knesset-data-pipelines
er run -it --entrypoint bash -v `pwd`:/pipelines orihoch/knesset-data-pipelines

Continue with Running the pipelines locally section above

You can usually fix permissions problems on the files by running inside the docker chown -R 1000:1000 .