Name: job-manager
Owner: Data Biosphere
Description: Job Manager API and UI for interacting with asynchronous batch jobs and workflows.
Created: 2017-08-30 18:51:22.0
Updated: 2018-01-10 20:25:04.0
Pushed: 2018-02-12 19:21:04.0
Size: 867
Language: Python
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This product is in Alpha and not yet ready for production use. We welcome all feedback!
See the development guide below.
The Job Manager is an API and UI for monitoring and managing jobs in a backend execution engine.
The Broad, Verily, and many other organizations in the life sciences execute enormous numbers of scientific workflows and need to manage those operations. Job Manager was born out of the experiences of producing data for some of the world?s largest sequencing projects such as The Cancer Genome Atlas, Baseline, and the Thousand Genomes Project.
The Job Manager aspires to bring ease and efficiency to developing and debugging workflows while seamlessly scaling to production operations management.
The current code is a work in progress towards an alpha release and as such has started with core features: connecting to both backends, visualizing workflow and task status and metadata, quick access to log files, and simple filtering.
The near-term roadmap includes improvements to failure troubleshooting, creating a robust dashboard for grouping jobs and seeing status overviews, and improving handling of widely scattered workflows.
We envision a product with user-customizable views of jobs running, insights into workflow compute cost, the ability to re-launch jobs, and the potential to make custom reports about the jobs that have been run.
The Job Manager defines an API via OpenAPI. An Angular2 UI is provided over the autogenerated Typescript bindings for this API. The UI is configurable at compilation time to support various deployment environments (see environment.ts), including auth, cloud projects, and label columns.
The UI must be deployed along with a backend implementation of the API; two such implementations are provided here:
Monitors jobs launched by the Cromwell workflow engine. The Python Flask wrapper was created using Swagger Codegen and can be configured to pull data from a specific Cromwell instance. The Job Manager currently supports Cromwell version 29.
Monitors jobs that were launched via the dsub CLI. Thin stateless wrapper around the dsub Python library. Authorization is required for deploying the UI, which is used to communicate with the Google Genomics Pipelines API. The wrapper itself is implemented in Python Flask using Swagger codegen models. A Dockerfile is provided which serves for production deployment using gunicorn.
Note that a ?task? in dsub nomenclature corresponds to a Job Manager API?s ?job?.
The following commands assume you have symbolically linked your preferred
local API backend docker compose file as docker-compose.yml
, e.g.:
sf dsub-local-compose.yml docker-compose.yml
Alternatively, use:
er-compose -f dsub-google-compose.yml CMD
For setting up development with dsub
see servers/dsub.
For setting up development with cromwell
see servers/cromwell.
docker-compose up
from the root of the repository:package.json
or requirements.txt
require a rebuild with:er-compose up --build
Alternatively, rebuild a single component:er-compose build ui
We use swagger-codegen to automatically implement the API, as defined in api/jobs.yaml
, for all
servers and the UI. Whenever the API is updated, follow these steps to
update the server implementations:
nux
http://central.maven.org/maven2/io/swagger/swagger-codegen-cli/2.2.3/swagger-codegen-cli-2.2.3.jar -O swagger-codegen-cli.jar
cOS
install swagger-codegen
i/src/app/shared/model/*
ervers/dsub/jobs/models/*
ervers/cromwell/jobs/models/*
-jar swagger-codegen-cli.jar generate \
pi/jobs.yaml \
ypescript-angular2 \
i/src/app/shared
-jar swagger-codegen-cli.jar generate \
pi/jobs.yaml \
ython-flask \
ervers/dsub \
pportPython2=true,packageName=jobs
-jar swagger-codegen-cli.jar generate \
pi/jobs.yaml \
ython-flask \
ervers/cromwell \
pportPython2=true,packageName=jobs
For UI server documentation, see ui.
dsub
ServerFor dsub
server documentation, see servers/dsub.
cromwell
ServerFor cromwell
server documentation, see servers/cromwell.