configs or models for streaming

[jeremy-syn]

First, I've found NeMo well-done and very helpful, so thank you for putting this together.
Second, I'm trying to put together an ASR system for a real-time streaming application. All of the available configs and pre-trained models seem to be built for batch mode processing, using forward-looking convolutions and full-utterance context (e.g. in the average pooling for SE blocks). This translates into latency when using the models in a streaming application. Are there any configs or (even better) pre-trained models available that are causal, or at least have limited future context?

[VahidooX]

It is our plan to extend our efforts on streaming mode and train models specifically for streaming but you may still use the available pretrained models for streaming. May not give the best latency and accuracy possible but should be OK. Riva is also a framework which you can use NeMo models for production level ASR including in streaming mode. Here is also a tutorial to show how to use nemo models in streaming mode: https://github.com/NVIDIA/NeMo/blob/stable/tutorials/asr/Streaming_ASR.ipynb

I guess you may also try to train such casual models with Citrinet. @titu1994 can give more guidance on that specific model.

[titu1994]

We will be increasing focus on streamable models, but as shown, one can use offline models for buffered inference as well with some tolerable latency.

For Citrinet / QuartzNet / Jasper specifically, you can add the "forward_context" kwarg in each layer of the encoder.jasper config section, declaring in integer number of frames how much future context should be allowed. You'll need to keep track of the subsampling level and adjust the number of future context frames to obtain the specific forward context you require.

https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/jasper.py#L530-L576

[vadimkantorov]

also wonder, has anyone tried distilling offline conformers with large-right-context-convs into streaming conformers? (E.g. distilling intermediate features for faster learning)

[AnkushMalaker]

Is this notebook still up to date? Any advancements in the last 3 years relevant for streaming ASR? @VahidooX or @titu1994
Thanks!