Reign in the chaos

[brettviren]

This package has evolved into a "bag of files" with no structure. The only attempt to organize is the readme file which only partly covers the contents. The lack of organization in this package permeates into other packages that use the files. Without even a clear file naming convention, and version strings getting baked into file names, it can be difficult to know file types and which files should be used. File names can also make it less than clear which files belong to which detectors.

This issue collects a check list of ideas to make improvements.

• define an enumerated list of file type names ("wires", "response", "noise", ...)
• define an enumerated list of canonical detector names ("pdsp", "uboone", ...)
• define a usage context, eg as a pair "(type,detector)"
• maintain a machine and human readable file that associates file type and other metadata to each file and that defines the file to use for a given context.
• maintain this file as Jsonnet with functions defined to enforce data structure schema.
• generate and commit JSON version to support use in, eg, bee
• automate any processing in a wscript
• make actual tagged releases, tie those releases to WCT releases and test framework

Some less concrete ideas

• consider breaking change: directory structure, file rename and purge
• consider committing upstream files (eg, wire "text dumps") with wscript tasks to regenerate.
• consider capturing better upstream processes used to generate these precursors (eg, art job config for wire dumping, wire-cell-python jobs for post processing GARFIELD fields, WCT or WC/LS jobs for noise spectra production).

Other comments

• make coherent with the currently nascent wire cell test data repo ideas