Best practices for collaborative ML R & D: How to structure frameworks and collaboration

[klieret]

Examples of challenges/discussion points:

Technological aspects:

1. How can we cut boilerplate and standardize interfaces so that people can focus on developing models without sacrificing "hackability". Pytorch lightning is a popular option for pytorch, but IMO the way it is laid out by default has its own challenges (and might lead to duplicated code)
2. How can we share results between the collaborators and bring everyone "on the same page" (for example using weights & biases)

Social aspects:

3. How can we make sure to move in the same direction without constraining ourselves? How do we keep everyone engaged in building a common framework and avoid people "branching off forever".
4. How do we balance more technical SW development work with model development? A lot of people want to focus on developing their model; few people want to work on framework issues. A good collaboration needs both.

I originally suggested this as a subtopic for #6 (doing open source). It also overlaps with #1 (packaging), #5 (fitting), and #19 (ML workflows for analysis). However, I think the challenges are very distinct because this targets development and R & D, rather than use in production/integration with other tools (for example, backwards compatibility isn't as big of an issue as is allowing for creativity).

[lgray]

This has a large overlap in themes with #19. Usefully different scope and kinds of requirements though!

[klieret]

Yes, I was thinking about this too, but the title of #19 led me to believe that its mainly about ML Ops and facilities (?).

[lgray]

User interface necessarily must deal with collaboration and frameworks.

[klieret]

Live notes

ML R & D Breakout session (Tuesday)

Present: Philip, Kilian, Richa, Raghav, Josue, Mike

Some of the questions that were discussed:

• What frameworks do people use (lightning & friends)?
  • pytorch lightning
  • ML Flow might also do some things that lightning does
  • Onnyx for plugging in ML in other frameworks/model exchange
• Dashboards (wandb & friends)?
  • ML Flow
  • Weights & Biases

Projects that were mentioned:

• https://github.com/jet-net/JetNet: Interface for Jet datasets (combining different data sources).
• Kilian: https://github.com/gnn-tracking/gnn_tracking
• https://github.com/FAIR4HEP/cookiecutter4fair: Cookie cutter for data science

Conclusions:

• Dashboards like (W&B / ML Flow) are a good way to bring people "on the same page" and compare/review/debug performance
• Frameworks that are built around hooks and plugin/callback structure are a good way to allow extensibility without growing "Dinosaur classes. For example, lightning hooks like on_validation_epoch_ends allow you to write callbacks to do stuff at the end of an epoch rather than sublassing/modifying your class