Roadmap / Future

[valencik]

What should Textmogrify be?

Currently:

• it started from the need to use Lucene Analyzers in a cats-effect app
• it lets one use Analyzers with Streams (unused by me so far)
• provides a builder interface for customizing Analyzers
• supports 8 languages

Protosearch

Is textmogrify suitable for protosearch?

Easily addressable needs:

• tokenization
• customizable collection type
• offset tracking

Hard to meet needs:

• cross platform
• no effects, everything in memory

Not using effects is at odds with leveraging Lucene.
And of course, Lucene is JVM only, so cross platform really means writing our own
pure Scala implementation, or finding equivalent implementations in JS and Native.
I think trying to leverage a different JS and Native lib is not worth the pain.

Pure Scala Implementation

If we go through the effort of writing a pure Scala implementation of a tokenizer,
what is the benefit of keeping the Lucene module around then?

• Easy testing against Lucene in both performance and correctness.
• Lucene has MANY more Analyzers to use
• Likely still a great solution if you're on the JVM