lanl/libcircle

Name: libcircle

Owner: Los Alamos National Laboratory

Description: An API to provide an efficient distributed queue on a cluster. Libcircle is currently used in production to quickly traverse and perform operations on a file tree which contains several hundred-million file nodes.

Created: 2015-05-27 17:21:34.0

Updated: 2015-05-27 17:21:34.0

Pushed: 2014-09-18 06:20:47.0

Homepage: http://hpc.github.io/libcircle/

Size: 1974

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

libcircle

libcircle is an API for distributing embarrassingly parallel workloads using self-stabilization. Details on the algorithms used may be found at http://dl.acm.org/citation.cfm?id=2389114.

Dependencies

• Open MPI http://www.open-mpi.org/

Compile and install

The current build status is:

nfigure
 all check
 make install

To enable output from libcircle (including fatal errors), run configure with “–enable-loglevel=number” where “number” is one of the following options:

• “1” fatal errors only.
• “2” errors and lower log levels.
• “3” warnings and lower log levels.
• “4” info messages on internal operations and lower log levels.
• “5” fine grained debug messages and lower log levels.

RPM Build and Install

To build an RPM, use the following instructions:

Developer API Information

The basic program flow when using libcircle is the following:

1. Define callbacks which enqueue or dequeue strings from the queue.
2. Execute the program.

lude <libcircle.h>

n example of a create callback defined by your program */
 my_create_some_work(CIRCLE_handle *handle)

/*
 * This is where you should generate work that needs to be processed.
 * For example, if your goal is to lstat files on a large filesystem,
 * this is where you would readdir() and and enqueue directory names.
 *
 * This should be a small amount of work. For example, only enqueue the
 * filenames from a single directory.
 */
while((data_to_process = readdir(...)) != NULL)
{
    handle->enqueue(data_to_process);
}


n example of a process callback defined by your program. */
 my_process_some_work(CIRCLE_handle *handle)

/*
 * This is where work should be processed. For example, this is where you
 * should lstat one of the files which was placed on the queue by your
 * create_some_work callback. Again, you should try to keep this short and
 * block as little as possible.
 */
handle->dequeue(&my_data);
...
finished_work = lstat(my_data, ...);
...
store_in_database(finished_work);



nitialize state required by libcircle. Arguments should be those passed in
y the launching process. argc is a pointer to the number of arguments,
rgv is the argument vector. The return value is the MPI rank of the current
rocess.

rank = CIRCLE_init(&argc, argv);


rocessing and creating work is done through callbacks. Here's how we tell
ibcircle about our function which creates the initial work. For MPI nerds,
his is your rank 0 process.

LE_cb_create(&my_create_some_work);


fter you give libcircle a way to create work, you need to tell it how that
ork should be processed.

LE_cb_process(&my_process_some_work);


ow that everything is setup, lets execute everything.

LE_begin();


inally, give libcircle a chance to clean up after itself.

LE_finalize();