Name: mustached-octo-ironman
Owner: biocore
Description: Easy dispatched compute in a Tornado environment
Created: 2014-10-17 17:28:04.0
Updated: 2016-05-27 01:19:21.0
Pushed: 2016-05-27 01:19:20.0
Size: 110
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Easy dispatched compute via in a Tornado environment using Redis and IPython. Updates are readily available to the client-side via a websocket and a customizable div. The specific goals of moi are:
This codebase originates from qiita and is heavily influenced by their dev-team, in particular @squirrelo and @josenavas.
To submit a job that can update status and publish updates via Redis but does not need to update client-side:
moi.job import submit_nouser
hello(**kwargs):
return "hi!"
it_nouser(hello)
To submit a job that is can be client-side (assumes the moi websocket handler is in place and that moi.js is loaded client-side):
moi import ctx_default
moi.job import submit
tornado.web import RequestHandler
hello(**kwargs):
kwargs['status_update']("I'm about to say hello")
return "hi!"
s Handler(RequestHandler):
def get(self):
result_handler = "/hello_result"
submit(ctx_default, self.current_user, "The hello job", result_handler,
hello)
moi supports submission of arbitrary system calls and somewhat arbitrary Python code from the command line. Below is an example of sending some Python code. There are a few things to note, first that the return allows you to store data in the result field of a moi job. Second, we're using --block which will halt until the command completes, and then dump the associated job details.
i submit --cmd "x = 5; return x + 10" --block
itted job id: 0c1ce595-4f6d-4fcb-b872-90118f6b1cd9
****** 0c1ce595-4f6d-4fcb-b872-90118f6b1cd9 **********
ext : default
_created : 2015-06-01 15:46:49.977480
_end : 2015-06-01 15:46:49.981942
_start : 2015-06-01 15:46:49.981620
: 0c1ce595-4f6d-4fcb-b872-90118f6b1cd9
: no-user-cmd-submit
nt : ac97957a-a658-47c8-bfaf-08d965b3e745
ub : 0c1ce595-4f6d-4fcb-b872-90118f6b1cd9:pubsub
us : Success
: job
: None
lt : 15
These jobs are submitted within the relevant context which can be provided on the command line. Next is an example of a system call via moi:
i submit --cmd "echo 'hello from moi'" --block --cmd-type=system
itted job id: 3cb7c548-895a-48ac-b1dc-786e89476c97
****** 3cb7c548-895a-48ac-b1dc-786e89476c97 **********
ext : default
_created : 2015-06-01 15:39:17.829820
_end : 2015-06-01 15:39:17.840456
_start : 2015-06-01 15:39:17.834566
: 3cb7c548-895a-48ac-b1dc-786e89476c97
: no-user-cmd-submit
nt : ac97957a-a658-47c8-bfaf-08d965b3e745
ub : 3cb7c548-895a-48ac-b1dc-786e89476c97:pubsub
us : Success
: job
: None
lt : [u'hello from moi\n', u'', 0]
Now, lets say we have some long running work that we want to kick off, and come back to later. The next example shows a system call that blocks for some extended period, and in the interest of showing what happens when a command fails, we're going to force the command to do something wrong. Note, we are not using --block.
i submit --cmd "sleep 10; foobar" --cmd-type=system
itted job id: 088c98fb-8612-43af-80df-1f64555531be
Control immediately returns, but let's check the job status while it's “running”.
i job --job-id=088c98fb-8612-43af-80df-1f64555531be
****** 088c98fb-8612-43af-80df-1f64555531be **********
ext : default
_created : 2015-06-01 15:51:21.979308
_end : None
_start : 2015-06-01 15:51:21.983198
: 088c98fb-8612-43af-80df-1f64555531be
: no-user-cmd-submit
nt : ac97957a-a658-47c8-bfaf-08d965b3e745
ub : 088c98fb-8612-43af-80df-1f64555531be:pubsub
us : Running
: job
: None
lt : None
We can see that it has a status of “Running” right now. If we give it a few more seconds, we can see the job finishes. But, since the command foobar doesn't exist, we get our error output.
i job --job-id=088c98fb-8612-43af-80df-1f64555531be
****** 088c98fb-8612-43af-80df-1f64555531be **********
ext : default
_created : 2015-06-01 15:51:21.979308
_end : 2015-06-01 15:51:32.004002
_start : 2015-06-01 15:51:21.983198
: 088c98fb-8612-43af-80df-1f64555531be
: no-user-cmd-submit
nt : ac97957a-a658-47c8-bfaf-08d965b3e745
ub : 088c98fb-8612-43af-80df-1f64555531be:pubsub
us : Failed
: job
: None
lt : Traceback (most recent call last):
le "/Users/mcdonadt/ResearchWork/software/mustached-octo-ironman/moi/job.py", line 140, in _redis_wrap
result = func(*args, **kwargs)
le "./moi", line 33, in _system_exec
le "/Users/mcdonadt/ResearchWork/software/mustached-octo-ironman/moi/job.py", line 52, in system_call
(cmd, stdout, stderr))
eError: Failed to execute: sleep 10; foobar
ut:
rr: /bin/sh: foobar: command not found
moi can retrieve all the information it knows about a user's jobs and dump it. In this example, we're going to submit two jobs, one that works and one that fails, and then take a look at the resulting output.
First, next start with a user that doesn't exist. Notice the argument is
--key as you can specify the user's UUID or the username.
i userjobs --key=example
own key: example
So we're starting on a clean slate. Now, let's submit some work. The first job hopefully will succeed and the second probably will crash and burn a horrible fiery death.
i submit --user=example --cmd "return 42"
itted job id: 558de7a9-fb99-4a67-8b27-62b2ddfb80ff
i submit --user=example --cmd "return float('crash and burn')"
itted job id: 44650179-deda-4f5c-9c92-5802f7b13154
Now we can examine what happened! The job information is provided below, and will be sorted by date.
moi userjobs --key=example
****** 558de7a9-fb99-4a67-8b27-62b2ddfb80ff **********
ext : default
_created : 2015-06-01 16:13:16.499326
_end : 2015-06-01 16:13:16.503163
_start : 2015-06-01 16:13:16.502835
: 558de7a9-fb99-4a67-8b27-62b2ddfb80ff
: no-user-cmd-submit
nt : effc00bd-d9be-4bfa-b788-a273e1c7d5da
ub : 558de7a9-fb99-4a67-8b27-62b2ddfb80ff:pubsub
us : Success
: job
: None
lt : 42
****** 44650179-deda-4f5c-9c92-5802f7b13154 **********
ext : default
_created : 2015-06-01 16:13:21.959556
_end : 2015-06-01 16:13:21.963721
_start : 2015-06-01 16:13:21.963070
: 44650179-deda-4f5c-9c92-5802f7b13154
: no-user-cmd-submit
nt : effc00bd-d9be-4bfa-b788-a273e1c7d5da
ub : 44650179-deda-4f5c-9c92-5802f7b13154:pubsub
us : Failed
: job
: None
lt : Traceback (most recent call last):
le "/Users/mcdonadt/ResearchWork/software/mustached-octo-ironman/moi/job.py", line 140, in _redis_wrap
result = func(*args, **kwargs)
le "./moi", line 27, in _python_exec
le "<string>", line 1, in execwrapper
eError: could not convert string to float: crash and burn
But that can be quite a bit of information if the user has a large number of jobs associated. So we can also just dump a lighter summary that is still sorted by time of run.
i userjobs --key=example --summary
: 2015-06-01 16:13:16.499326 id: 558de7a9-fb99-4a67-8b27-62b2ddfb80ff status: Success
: 2015-06-01 16:13:21.959556 id: 44650179-deda-4f5c-9c92-5802f7b13154 status: Failed
Almost function that can be sent over to an IPython client is acceptable. The two expectations are:
**kwargsGoing one step further, the code also supports system calls through a special function moi.job.system_call, where the argument being passed is the command to run.
In moi, jobs are associated with a group (e.g., self.current_user). A group can have 0 to many jobs. A group has an associated pubsub channel at <group>:pubsub that can be used to perform actions on the group.
All groups have a Redis set associated under <group>:jobs that contain the job IDs associated with the group.
All jobs are keyed in Redis by their ID. In addition, each job has a pubsub at the key <job ID>:pubsub that can be used to notify subscribers of changes to the job.
All communication over pubsub channels consists of JSON objects, where the keys are the actions to be performed and the values are communication and/or action dependent.
A group accepts the following actions via pubsub:
add : {list, set, tuple, generator} of str
Add the job IDs described by each str to the group
remove : {list, set, tuple, generator} of str
Remove the job IDs describe by each str from the group
get : {list, set, tuple, generator} of str
Get the job details for the IDs
A job can send the following actions over a pubsub:
update : {list, set, tuple, generator} of str
Notifies subscribers that the corresponding job has been updated. A job can notify that other jobs have been updated.
Jobs are described in a hierarchy to allow jobs to be associated with multiple logically related groups. For instance, a job might be associated with a user, and additionally, associated with a workflow that the user is executing (e.g., some complex analysis). The hierarchy can be thought of as a tree, where internal nodes are “groups” and the tips are actual jobs. Paths in the tree are denoted by a “:” delimited string. For instance foo is the group “foo”, while foo:ID_1:ID_2 denotes the group “foo”, which contains “ID_1”, which contains “ID_2”. Groups are described by uuid's, as are jobs.
Job and group information can be accessed by using the ID as the key in Redis. This information is a JSON object that consists of:
id : str
A str of a UUID4, the ID
name : str
The group or job name
type : str, {'job', 'group'}
What type of info object this is.
pubsub : str
The pubsub for this info object
url : str or null
The URL for group or job results. This URL is provided the corresponding ID (e.g., /foo/<uuid>).
parent : str or null
The ID of the parent. Null if the group is the root. It is not required that this be a uuid.
status : str
The group or job status
result : str or null
The result of the job. If the job has not completed, this is null. If the job errors out, this will contain a
repr'd version of the traceback. This is null if the object described a group.
date_start : str of time
Time when the job started, expected format is %Y-%m-%d %H:%M:%s. This is null if the object describes a group.
date_end : str of time
Time when the job ended, expected format is %Y-%m-%d %H:%M:%s. This is null if the object described a group.
The default status states defined by moi are {"Queued", "Running", "Success", "Failed"}.
Communication over the websocket uses JSON and the following protocols. From server to client:
add : info object
An info object to that has been added on the server.
remove : info object
An info object that has been removed on the server.
update : info object
An info object that has been upadted on the server.
From client to server:
remove : str
An ID that the client would like to remove. If a group ID, then all descending jobs are removed as well.