Java bioimage.io support: design and plans

[constantinpape]

I had a discussion with @tomburke-rse and @carlosuc3m about plans and general design for the bioimageio java support.
We currently have these main issues on the java side:

• different java based consumers (DeepIJ, QuPath, icy, other Fiji plugins (e.g. N2V)) are using/developing different code bases for supporting the spec and running inference
  • this is the case because this library (core-bioimage-io-java) currently only supports reading of the spec and does not implement inference
  • inference for imglib2 is implemented in https://github.com/imagej/imagej-modelzoo, but currently limited to tensorflow1; and it's unclear if all the java dependencies can use the imglib2 interface (this is used e.g. by the N2V plugin)
• in general weight format support is mostly limited to tensorflow 1 (DeepIJ also supports torchscript)

To improve the support for bioimage.io in java we should aim to provide a common library that can be used for inference AND that supports all/most of the bioimage.io weight formats.

• To not make our lives to difficult we should make use of existing java efforts to run deep learning models from different frameworks in java, the two best candidates are:
  • deep-java, which has support for inference with most frameworks already
  • java-cpp, which is a library for compiling c++ libraries in java, which has many of the deep learning backends compiled already: https://github.com/bytedeco/javacpp-presets/
• Based on one of these two libraries (for simplicity I will assume deep-java in the following), we should then implement inference functionality for bioimage.io models. Note that @carlosuc3m has implemented some of this already and could provide it.
• Consequently, we should also implement the pre-and-post-processing operations compatible with the deep-java interface.
• Any consumer could then make use of this library and would only need to bridge to the deep-java interface using buffers; we could provide the imglib2 binding (and other bindings that may be relevant) in separate repositories

[oeway]

Most of the points above have been discussed in a java meeting

March 31, 2022

Attendees

Name	affiliation	username (github)
Wei Ouyang	KTH	@oeway
Carlos García	Pasteur	@carlosuc3m
Constantin Pape	EMBL	@constantinpape
Ekaterina Moreva	EMBL	@KateMoreva
Estibaliz Gómez	IGC	@esgomezm
Daniel Sage	EPFL	@dasv74
Pete Bankhead	UoEdinburgh	@petebankhead
Lucia Moya	UC3M	@xion16lm
Jean-Yves Tinevez	Institut Pasteur	@tinevez
Jean-Christophe	Institut Pasteur
Tom Burke	CBG+CSBD	@tomburke-rse
Christian Tischer	EMBL	@tischi
Arrate Muñoz	UC3M	@arratemunoz
Lucia Moya	UC3M	@xion16lm

Minutes

The meeting starts by an introduction to the bioimage.io spec, core by Constantin.

Then, the following key questions/points were raised for the meeting:

• A unified way for represent the nd-array?
• What library to use to run the network?
• Public API for the consumer (including pre- and post-processing)
• License for the library? MIT or BSD should be
• Modularity:
  • Pete: 2 libraries for reading the spec (current "core-java" library) and one for inference etc.
  • Pete & Tom: the current core library is a bit outdated / maintenance isu unclear; Tom agrees that some maintenance and refactoring would be good
  • Esti: modularity is important; in python it is very nice to have optional dependencies

We proceeded with an introductory presentation to deep-java by Carlos and some following discussions:

• Backends for TF2, PyTorch and ONNX (and others that are currently not relevant for us)
• Switching between backends is easy, but switching between different versions of the same backend is more difficult, needs a restart
• Backend manager for tf by Curtis & Deborah supports this (but only for tf 1 versions)
• Backend manager for deepJava library for different versions?
• NDArray class
  • bound to "manager" class, that needs to stay alive until operations with an ndarray are done
  • all operations must apply to ndarrays from the same manager
• Design proposal:
  • consumer (icy, impgplus, imglib2) -> ndarray -> preprocessing -> model
  • model -> postprocessing -> post-processing -> consumer
• TF1 support would need a custom implementation
• deepJava: does it allow training?
  • Carlos: Not sure, Tom: yes, but TF API is not complete
  • After looking up https://docs.djl.ai/engines/tensorflow/tensorflow-engine/index.html: training for TF2 currently not supported
• deepIcy: another layer on top to keep track of API versions
• Comments Tom:
  • split between rdf parsing lib and inference runner lib
  • runner lib: similar API to the core python lib
  • support as many models as possible -> use deep java !
    • deep java is the only realistic avenue of having support for all the frameworks
• Carlos: you can still use the native APIs if youre using deep java
• Tensorflow1 already supported in Carlos deep java
• Can we re-use what Carlos did?

The next steps are:

• Use deep-java and its NDArray to power the java core library
• Carlos creates private fork of the deep java based icy library in bioimage.io
  • We can make PRs etc. against this one
• We have a look into integration with the current core-bioimageio-java = spec parser library and think about proper refactoring of both libraries to work together
• the version in icy will stay private and mostly unchanged until publication, once this is done we will tackle integration with th fork in bioimageio
• refactoring of current core-bioimageio-java to make it easier maintainable
• consumer-integration of the inference runner from Carlos

[constantinpape]

Thanks for the meeting everyone!

@carlosuc3m let us know once the fork is available :).

[carlosuc3m]

I have just created a private repository called model-runner-java where the discussed code has been uploaded