Luke cannot open ES index file with completion suggestion

[begemot63]

If I add completion suggestion to the ES mapping then Luke cannot open the index file.
Assume I add the following suggestion in ES mapping:
{
"suggestible_text_fields": {
"match": "*_st_en",
"mapping": {
"type": "text",
"analyzer": "custom_english_index_analyzer",
"search_analyzer": "custom_english_query_analyzer",
"fields": {
"bigrammed": {
"type": "text",
"analyzer": "custom_english_bigram_index_analyzer"
},
"exact": {
"type": "text",
"analyzer": "custom_exact_analyzer"
},
"suggest": {
"type": "completion"
}
}
}
}
Luke throws exception :
[2018-12-06 17:48:32,456] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]
java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)
at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:112)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:289)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:372)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:113)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:83)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:66)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:688)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at org.apache.lucene.luke.util.IndexUtils$1.preVisitDirectory(IndexUtils.java:104)
at org.apache.lucene.luke.util.IndexUtils$1.preVisitDirectory(IndexUtils.java:99)
at java.nio.file.Files.walkFileTree(Files.java:2677)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.apache.lucene.luke.util.IndexUtils.openIndex(IndexUtils.java:99)
at org.apache.lucene.luke.app.IndexHandler.open(IndexHandler.java:60)
at org.apache.lucene.luke.app.desktop.components.dialog.menubar.OpenIndexDialogFactoryImpl$ListenerFunctions.openIndexOrDirectory(OpenIndexDialogFactoryImpl.java:316)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6535)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6300)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4891)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2750)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:109)
at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:184)
at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:229)
at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:227)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:227)
at java.awt.Dialog.show(Dialog.java:1084)
at java.awt.Component.show(Component.java:1673)
at java.awt.Component.setVisible(Component.java:1625)
at java.awt.Window.setVisible(Window.java:1014)
at java.awt.Dialog.setVisible(Dialog.java:1005)
at org.apache.lucene.luke.app.desktop.util.DialogOpener.open(DialogOpener.java:43)
at org.apache.lucene.luke.app.desktop.util.DialogOpener.open(DialogOpener.java:36)
at org.apache.lucene.luke.app.desktop.components.MenuBarProvider$ListenerFunctions.showOpenIndexDialog(MenuBarProvider.java:185)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833)
at com.apple.laf.AquaMenuItemUI.doClick(AquaMenuItemUI.java:157)
at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:877)
at java.awt.Component.processMouseEvent(Component.java:6535)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6300)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4891)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2750)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Note, completion suggestion works in ES very well, but Luke cannot open index file. I am using ES 6.4.2

[mocobeta]

Thanks for sharing the stack trace.
FYI, this seems to be related to #129

[begemot63]

I did not find "Load custom codecs" button to try #129

[mocobeta]

#129 is not resolved yet - so we need a patch for it. I will work for this when I can take time for it.

It seems that elasticsearch uses their own custom codecs for the auto completion fields. I don't like to include elasticsearch jars or classes in Luke distribution, so we should support "loading custom codecs" feature to add the codecs on runtime.

2018-12-06 17:48:32,456] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]

For quick fix, you may be able to add elasticsearch jars to the classpath when launching Luke by modifying luke.sh. Though I have not tried yet that.

[begemot63]

I added commons-codec-1.10.jar to the classpath in luke.sh but still see the same error:

[2018-12-09 15:06:52,343] ERROR (IndexHandler.java:62) - An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]
java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'completion' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene50, IDVersion]
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)

Please let me know what jar needs to be included into classpath to handle it?

[mocobeta]

Elasticsearch should include the custom codec class, so you need the elasticsearch jars you are using when launching Luke.
I have not tried yet, but I'd add all jars (except for lucene jars, which are already included in Luke) in the elasticsearch lib directory.

[begemot63]

I added commons-codec-1.10.jar, elasticsearch-6.4.2.jar, elasticsearch-core-6.4.2.jar, elasticsearch-x-content-6.4.2.jar to the classpath and it does not help

[mocobeta]

Sorry, I found that the missed codec class is not elasticsearch's one. Lucene's suggester module includes the custom codec named 'completion' and Luke 7.5.0 does not include this jar. I will add this
to Luke distribution as of the next release. Thanks for the report.
https://mvnrepository.com/artifact/org.apache.lucene/lucene-suggest

For now, can you try to add the jar to the classpath?

[mocobeta]

I just added lucene-suggest to the pom.xml dependencies in the master branch. You can checkout and try a nightly build by mvn package.

[begemot63]

I confirm. The issue is solved now.
Thanks

[mocobeta]

Thanks for the confirmation!

[begemot63]

One second. Luke can open the index but show result only if ElasticSearch is down .
That seen with suggestion field.
Please review it

[begemot63]

let me know if we need to open a new issue

[mocobeta]

Luke can open the index but show result only if ElasticSearch is down .
That seen with suggestion field.

I'm not sure, but Elasticsearch could have the index in RAM only in a while to avoid disk I/O (and dump it to disk later) - in that case Luke cannot read the indexed field until it is flushed to the disk and it's not a bug.
Is the field persisted when opening it by Luke?