ContentMine/visualizations

Name: visualizations

Owner: The ContentMine

Description: null

Created: 2016-05-18 13:58:19.0

Updated: 2018-03-21 11:13:07.0

Pushed: 2016-11-24 15:34:16.0

Homepage: null

Size: 94447

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

visualizations

Create and publish interactive visualizations of ContentMine-facts. We use the Bokeh library for creating interactive visualization applets that can be run locally or deployed on a server.

Installation and Requirements

The easiest setup is by using Anaconda for Python 3, opening the Anaconda Prompt and creating a new virtual environment.

a create -n contentmine3 python=3.5.2 anaconda

After that activate the virtual environment and install the packages with pip:

ce activate contentmine3
install pandas bokeh

• Other useful packages:
• pandas
• bokeh
• jupyter
• pycproject

Configuration

At the moment very basic, three options have to be set in config.py:

• rawdatapath = where the raw data in form of facts.json and metadata.json is expected
• cacheddatapath = where the intermediary dataframes can be stored for reuse
• resultspath = where the output-htmls and plots are expected

Running an interactive visualization

Fork, clone or download this repo.

Get some play data

• Download the facts and metadata json-files from Zenodo, and place them into the rawdatapath.
• rename the files into facts.json and metadata.json
• Run preprocessing from visualizations-folder:

  on3 preprocessing/preprocessing.py --raw /PATH/TO/RAWDATA --cache /PATH/TO/CACHEDDATA
  

Setting up local environments

• Active virtualenv with source activate contentmine3
• Change paths in config.py
• change working directory to visualizations
• move the preprocessed data which was output to CACHEDDATA to data - this data will be visualized
• start application with bokeh serve --show APPNAME/APPNAME.py, e.g. bokeh serve --show cooccurrences/cooccurrences.py
• This should - after a few seconds of processing - open a new browser window with the interactive visualization

Deployment to heroku

Deployment notes

• https://devcenter.heroku.com/articles/python-gunicorn
• Updating dependencies: http://stackoverflow.com/questions/9471017/how-do-i-upgrade-a-dependency-in-a-python-project-on-heroku
• Scaling out and up: https://devcenter.heroku.com/articles/dynos#isolation-and-security

Currently we have to split bokeh and flask into two apps (dynos), until the interprocess communication problem can be solved. Flask frontend-development is happening on branch flask, bokeh server development is happening on branch server. Differences are the Procfile, and in app.py, where autoload_server() looks at different urls when running on the same or two dynos. Data needs to be updated in each branch.

Deploy via pipeline

• Production Pipelines: Read https://devcenter.heroku.com/articles/pipelines and https://devcenter.heroku.com/articles/github-integration-review-apps

Add staging app and server app to git remotes

remote add staging https://git.heroku.com/contentmine-demo-staging.git
remote add server https://git.heroku.com/contentmine-demos.git

Push new flask-app code to staging

checkout flask
push staging flask:master    

Push new server-app code to server

checkout server
push server server:master    

Review functionality and then promote to production (via CLI or web interface)