Add batch analyze API support for recognizer

[jimmyxie-figma]

Is your feature request related to a problem? Please describe.

Currently, the BatchAnalyzerEngine works by iterating through either a list or dictionary, analyzing and anonymizing the values one by one. while this is not an issue for the predefined recognizers, and there are improvements in the built-in NLP engine to support batch inference, it does pose an efficiency problem for the transformer recognizers, causing idle resources and low inference throughput.

Describe the solution you'd like

We want to build batch inference API support for recognizers. Early testing shows that even with a small batch size of 4, a BERT-like transformer speeds up inferences by 3x without any additional resource or memory usage.

The exact implementation is still up for discussion and one potential solution would be adding a batch recognizer mix-in, where we batch analyze the the batch recognizer first, and pass the results to the regular analyze for extension. similar to the nlp_engine.process_batch

Describe alternatives you've considered
N/A

Additional context
N/A

[omri374]

Thanks for the suggestion! One of the reasons we chose to use spacy-huggingface-pipelines is its batch support for transformer based models. Have you experimented with this option? In general, I agree that exposing an analyze_batch option for recognizers is a good idea, and having the BatchAnalyzerEngine call analyze_batch instead of analyze for each recognizer.

[jimmyxie-figma]

@omri374 thanks for the quick reply. I would imagine the transformers_nlp_engine does the same thing as well. nvm, I see the comments, that's just a wrapper around the package

The problem is that we are currently using two transformer models (and not for the purpose of multi-language support), so that approach wouldn’t work. Another way forward for us would be to fine-tune our own models and consolidate to one.

[jimmyxie-figma]

Alternatively, we could support multiple NLP engines and extend the NLP artifacts. If we decide to provide better support for batching, my team is happy to help with the PR.

[omri374]

Will the option of having a batch_analyze for recognizers be useful in your case? If yes, a PR would be great. Essentially, we could add a batch_analyze method to the EntityRecognizer base class, and have it iterate through texts and pass them to the analyze method. In specific cases, like the transformers recognizer, we could override this method with a different implementation for batch mode. WDYT?