FredHutch/storage-crawler

Name: storage-crawler

Owner: Fred Hutchinson Cancer Research Center

Description: parallel file system crawler that manages file metadata in a postgres database

Created: 2016-10-21 03:22:35.0

Updated: 2017-02-27 15:09:26.0

Pushed: 2016-11-09 00:41:35.0

Homepage: null

Size: 34

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

storage-crawler

storcrawl

A parallel file system crawler that manages file and directory metadata in a (postgres) database.

Requirements

Language:

• Python 3.5

Python Modules:

• configargparse
• psycopg2

External Dependencies:

• a postgres DB and use with create table and drop table rights

Current State

Currently the main program storcrawl.py is working to crawl a given path. It is reasonably efficient, but still has some issues with error handling and process clean-up.

Basic Workflow

The main process creates two tables in the DB based on the --tag command line argument. The --tag is turned into a schema, and the tables and report account are created in that schema:

• *tag*.files - table contains the actual file metadata and full path with filename
• *tag*.status - a status log of the crawl

Once set up, directory walking processes (walkers), file stating processes (staters) and DB writing processes (dbprocs) are spawned up to the specified number. These processes use queues to pass paths around to each other - a file queue, a directory queue, and a db queue.

Specifically, the walkers pop from the dir queue, and push onto the dir queue and the file queue. The staters pop from the file queue and push onto the db queue, and the db procs pop from the db queue and insert into the DB.

When the dir queue is empty, the walkers are stopped. Then the file queue, and finally the db queue and the crawl is complete.

Theory

The basic idea is that by treating each file atomically, the stat process will not get bogged down in a single directory or file location. No guarantees, though!

Schemas

The *tag*.files table:

             id SERIAL,
             insert_time timestamp with time zone DEFAULT now () NOT NULL,
             path bytea NOT NULL,
             extension bytea NOT NULL,
             st_mode bit(19) NOT NULL,
             st_ino bigint NOT NULL,
             st_dev text NOT NULL,
             st_nlink int NOT NULL,
             st_uid bigint NOT NULL,
             st_gid bigint NOT NULL,
             st_size bigint NOT NULL,
             st_atime int NOT NULL,
             st_mtime int NOT NULL,
             st_ctime int NOT NULL,
             owner text

The *tag*_status table:

             id SERIAL,
             time timestamp with time zone DEFAULT now() NOT NULL,
             status text NOT NULL,
             value int,
             units text