python ingest, registration and sync tool

[jasoncoposky]

We have a requirement to ingest and / or register a considerable amount of data at rest in large, sometimes parallel, file systems. A design consideration is a new iRODS user with hundreds of millions of files, and also possibly an existing user who wishes to periodically sync a large volume with their existing iRODS catalog.

Given a target local directory, an initial list of features would include:

• operates in parallel for all possible speed - recursively descend a file system and push fully qualified paths into a worker queue for ingest threads
• option to wait N seconds to ensure file is at rest - landing zone style behavior
• option to ingest files or just register in-place
• option to checksum
• option to provide regular expressions to skip
• externalized metadata extraction using a DSL, inherited interface, or other mechanism for defining the rules to generate or extract metadata from the at-rest data
• option to set iRODS ACLs after data is ingested
• option for target collection, or collections given a mapping function
• idempotent - ability to skip unchanged, properly ingested data and metadata

Other possible features:

• proxy as other iRODS users for ingest