Name: patriot-workflow-scheduler
Owner: CyberAgent
Description: null
Created: 2015-06-05 01:47:46.0
Updated: 2018-05-23 07:17:53.0
Pushed: 2018-05-23 07:17:51.0
Homepage: null
Size: 1603
Language: Ruby
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Patriot-workflow-scheduler is a workflow scheduler that can manage batch jobs with complicated dependency relations. The workflow scheduler can deal with an extensible monolithic workflow. Since the various kinds of jobs can be consolidated into the monolithic workflow, various combinations of data integration can be flexibly achieved.
The scheduler offers an original DSL for batch configuration (PBC: Patriot Batch Configuration). The description of a complicated workflow can be simplified by the DSL. In addition, the DSL is designed to be extensible so that customized job types can be easily integrated.
m install patriot-workflow-scheduler
triot-init ${install_dir}
gem install won't upgrade the js for web console. The patriot upgrade should be used to replace the js with new one.
${install_dir}
bin/patriot upgrade
Command line tools are available for processing jobs defined in PBC. The tool can be triggered by the patriot script. The script takes three arguments: a tool name ('execute' for job execution), a target date of the jobs (in yyyy-MM-dd) and a path to the PBC file. In processing the PBC files, a variable '_date_' will be replaced with the target date.
${install_dir}
t batch/sample/daily/test.pbc
me "test"
mmands "echo '#{_date_}' > /tmp/test.out"
bin/patriot execute 2015-04-01 batch/sample/daily/test.pbc
t /tmp/test.out
-04-01
Dependencies between jobs are defined through products. The products produced/required by jobs are configured by produce and require, respectively. A job becomes ready to be executed when all products required by the job are available. The products become available when all jobs which produce the product are finished.
To run jobs according to dependencies, the strict option should be passed to the execute tool.
t flow_sample.pbc
quire ['product1'] # run after 'product1' is created
me "consumer"
mmands "echo 'this is a consumer'"
oduce ['product1'] # this job creates 'product1'
me "producer"
mmands "echo 'this is a producer'"
bin/patriot execute --strict 2015-04-01 flow_sample.pbc
ecute echo 'this is a producer'
ecute echo 'this is a consumer'
The Web console implemented in JSX (React.js) and compiled with Grunt. To compile the JSXs, install Nodejs and set up grunt.
install node (e.g., brew install node)
m install -g grunt-cli
m install
unt
"grunt --watch" to watchify, "grunt build" to uglify js files. )
– Build gem and upgrade installation
build patriot-workflow-scheduler.gemspec
{install_dir}
n/patriot upgrade
The gh-pages branch should be managed as workdir at docs/build/html. The document can be generatd by Sphinx.
t-new-workdir . docs/build/html gh-pages
docs
ke html
You can easily generate API documents using yard.
m install yard
rd doc lib plugins
rd server
Now you can see docs at http://localhost:8808/docs/frames/index.
For understanding how to manage dependencies in a complicated workflow, the architecture of the scheduler, other command line tools, etc, please see the Github pages.
Copyright © CyberAgent, Inc. All Rights Reserved.
This package is released under Apache License Ver. 2.0. http://www.apache.org/licenses/LICENSE-2.0.txt