wtsi-hgi/mpistat

Name: mpistat

Owner: Wellcome Trust Sanger Institute - Human Genetics Informatics

Description: Efficiently lstat every file in a filesystem tree in parallel across multiple nodes using MPI.

Created: 2014-11-07 13:24:02.0

Updated: 2016-01-18 11:18:56.0

Pushed: 2016-09-19 15:05:17.0

Homepage:

Size: 70

Language: Python

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

mpistat

Efficiently walk a tree in parallel across many nodes using MPI. Based on http://conferences.computer.org/sc/2012/papers/1000a015.pdf. Also see http://jlafon.io/parallel-file-treewalk.html.

There is a C version using libcircle (https://github.com/hpc/libcircle) which is the C implementation used by the authors of the original article.

The python version uses Guy Coates's interpretation (https://github.com/wtsi-ssg/pwalk) of the algorithm discussed in the paper.

There is no significant difference in the time taken by the C and python versions due to the time being dominated by the lstat syscall.

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.