Index bib files outside project scope

[maxanier]

Probably duplicate of #3305, but I wanted to make it more explicit. Feel free to close.

Based on my setup in #3655, I still have the issue that bib files outside the project root are not indexed. Hence, any citation from a reference in such files is marked as unresolved by TeXiFy.

%If the file is inside the project scope, it works
%\addbibresource{C:\Temp\TexifyProblem\untitled\src\power_amp.bib}
%\addbibresource{C:/Temp/TexifyProblem/untitled/src/power_amp.bib}
%\addbibresource{power_amp.bib}

%If the file is outside, it does not
\addbibresource{C:/cloud/datashare/EPAAD/Information/Paper/power_amp.bib}

\begin{document}

\cite{Datta2014} <- Warning unresolved reference

Compilation and opening the file via the small b icon does work, but the external bib file is not indexed.
If I add the folder of the bib file as a content root in Pycharm, the file is indexed.

In #3655 it sounded like it is possible to index external files while in #3305 it sounds like it comes with a significant performance hit.

If it is easy to implement, I would appreciate a fix, otherwise I can just suppress the warning, that's fine for me.

[PHPirates]

Yes it's essentially the same issue, coincidentally I was looking at that one yesterday, and the problem is that before we start indexing we cannot use any index (of commands) to figure out if there are any absolute paths that need to be indexed, as we still need to build it, so it's a bit of a catch-22.
Again, I think using absolute paths to bibfiles outside of the project is bad practice and TEXINPUTS or TEXMFHOME should be used instead, but I will see if this is possible to support and otherwise add an inspection to warn against it.

[PHPirates]

I may have fixed this? Please try https://plugins.jetbrains.com/plugin/download?rel=true&updateId=634490
It may need an extra restart before it works, and the paths are cached in memory so after changing a path it needs a restart to take effect. But let's first see if this simple version doesn't break other things

[maxanier]

Nice, although I did a quick test and it does not work for me yet.
If I use commands in the path as discussed in #3655, they are not indexed (same behavior as before) .
If I provide an absolute path to a bib file outside the project, something happens, but the exception below is thrown.
\addbibresource{C:/cloud/datashare/EPAAD/Information/Paper/power_amp.bib}

Unfortunately, I do not have too much time to look into this further this week.

Stacktrace

``` [4ab3996c] Failed to execute task com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$ScheduledScanningTask@438ed4b0

java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.intellij.openapi.diagnostic.RuntimeExceptionWithAttachments: PSI and index do not match.
Please report the problem to JetBrains with the files attached
file=LaTeX source file, file.class=class nl.hannahsten.texifyidea.file.LatexFile, file.lang=Language: Latex, modStamp=1
tree consistent
stub debugInfo=created in getStubTree(), with AST = false; with backReference
viewProvider=com.intellij.psi.SingleRootFileViewProvider{vFile=file://C:/LocalTools/Diss/Tex/main.tex, vFileId=197, content=VirtualFileContent{size=7418}, eventSystemEnabled=true}
viewProvider stamp: 0; file stamp: 0; file modCount: 1731669630529; file length: 7418
doc saved: true; doc stamp: 0; doc size: 7160; committed: true
indexing info: indexing timestamp = 1731577420717, binary = false, byte size = 7421, char size = 7160
ref: 50cf572587cf
latestIndexedStub=StubTree{myDebugInfo='created from index; with backReference', myRoot=PsiFileStubImpl}555597391
same size=true
debugInfo=created from index; with backReference
at com.intellij.util.ExceptionUtil.rethrow(ExceptionUtil.java:136)
at com.intellij.openapi.progress.util.ProgressIndicatorUtils.awaitWithCheckCanceled(ProgressIndicatorUtils.java:399)
at com.intellij.openapi.progress.util.ProgressIndicatorUtils.awaitWithCheckCanceled(ProgressIndicatorUtils.java:362)
at com.intellij.util.indexing.UnindexedFilesScanner.waitForPreconditions(UnindexedFilesScanner.kt:507)
at com.intellij.util.indexing.UnindexedFilesScanner.access$waitForPreconditions(UnindexedFilesScanner.kt:59)
at com.intellij.util.indexing.UnindexedFilesScanner$perform$1$1.invoke(UnindexedFilesScanner.kt:451)
at com.intellij.util.indexing.UnindexedFilesScanner$perform$1$1.invoke(UnindexedFilesScanner.kt:450)
at com.intellij.util.indexing.UnindexedFilesScanner.prepareScanningHistoryAndRun(UnindexedFilesScanner.kt:488)
at com.intellij.util.indexing.UnindexedFilesScanner.perform(UnindexedFilesScanner.kt:450)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$runScanningTask$2$2.invoke(UnindexedFilesScannerExecutorImpl.kt:222)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$runScanningTask$2$2.invoke(UnindexedFilesScannerExecutorImpl.kt:221)
at com.intellij.openapi.progress.CoroutinesKt.blockingContextInner(coroutines.kt:339)
at com.intellij.openapi.progress.CoroutinesKt$blockingContext$2.invokeSuspend(coroutines.kt:232)
at com.intellij.openapi.progress.CoroutinesKt$blockingContext$2.invoke(coroutines.kt)
at com.intellij.openapi.progress.CoroutinesKt$blockingContext$2.invoke(coroutines.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:62)
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:261)
at com.intellij.openapi.progress.CoroutinesKt.blockingContext(coroutines.kt:231)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$runScanningTask$2.invokeSuspend(UnindexedFilesScannerExecutorImpl.kt:221)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$runScanningTask$2.invoke(UnindexedFilesScannerExecutorImpl.kt)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$runScanningTask$2.invoke(UnindexedFilesScannerExecutorImpl.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:62)
at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:261)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl.runScanningTask(UnindexedFilesScannerExecutorImpl.kt:209)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl.access$runScanningTask(UnindexedFilesScannerExecutorImpl.kt:37)
at com.intellij.util.indexing.UnindexedFilesScannerExecutorImpl$1$3$1.invokeSuspend(UnindexedFilesScannerExecutorImpl.kt:103)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
at kotlinx.coroutines.internal.SoftLimitedDispatcher$Worker.run(SoftLimitedDispatcher.kt:125)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:608)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:873)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:763)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:750)
Caused by: java.util.concurrent.ExecutionException: com.intellij.openapi.diagnostic.RuntimeExceptionWithAttachments: PSI and index do not match.
Please report the problem to JetBrains with the files attached
file=LaTeX source file, file.class=class nl.hannahsten.texifyidea.file.LatexFile, file.lang=Language: Latex, modStamp=1
tree consistent
stub debugInfo=created in getStubTree(), with AST = false; with backReference
viewProvider=com.intellij.psi.SingleRootFileViewProvider{vFile=file://C:/LocalTools/Diss/Tex/main.tex, vFileId=197, content=VirtualFileContent{size=7418}, eventSystemEnabled=true}
viewProvider stamp: 0; file stamp: 0; file modCount: 1731669630529; file length: 7418
doc saved: true; doc stamp: 0; doc size: 7160; committed: true
indexing info: indexing timestamp = 1731577420717, binary = false, byte size = 7421, char size = 7160
ref: 50cf572587cf
latestIndexedStub=StubTree{myDebugInfo='created from index; with backReference', myRoot=PsiFileStubImpl}555597391
same size=true
debugInfo=created from index; with backReference
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at com.intellij.openapi.progress.util.ProgressIndicatorUtils.awaitWithCheckCanceled(ProgressIndicatorUtils.java:370)
... 32 more
Caused by: com.intellij.openapi.diagnostic.RuntimeExceptionWithAttachments: PSI and index do not match.
Please report the problem to JetBrains with the files attached
file=LaTeX source file, file.class=class nl.hannahsten.texifyidea.file.LatexFile, file.lang=Language: Latex, modStamp=1
tree consistent
stub debugInfo=created in getStubTree(), with AST = false; with backReference
viewProvider=com.intellij.psi.SingleRootFileViewProvider{vFile=file://C:/LocalTools/Diss/Tex/main.tex, vFileId=197, content=VirtualFileContent{size=7418}, eventSystemEnabled=true}
viewProvider stamp: 0; file stamp: 0; file modCount: 1731669630529; file length: 7418
doc saved: true; doc stamp: 0; doc size: 7160; committed: true
indexing info: indexing timestamp = 1731577420717, binary = false, byte size = 7421, char size = 7160
ref: 50cf572587cf
latestIndexedStub=StubTree{myDebugInfo='created from index; with backReference', myRoot=PsiFileStubImpl}555597391
same size=true
debugInfo=created from index; with backReference
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.doCreateCompleteException(StubTreeLoader.java:219)
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.lambda$createCompleteException$0(StubTreeLoader.java:194)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:735)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:691)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$computeInNonCancelableSection$4(CoreProgressManager.java:277)
at com.intellij.openapi.progress.Cancellation.computeInNonCancelableSection(Cancellation.java:62)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeInNonCancelableSection(CoreProgressManager.java:277)
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.createCompleteException(StubTreeLoader.java:193)
at com.intellij.psi.impl.source.PsiFileImpl.loadTreeElement(PsiFileImpl.java:230)
at com.intellij.psi.impl.source.PsiFileImpl.calcTreeElement(PsiFileImpl.java:812)
at com.intellij.extapi.psi.StubBasedPsiElementBase.getNode(StubBasedPsiElementBase.java:126)
at com.intellij.extapi.psi.ASTDelegatePsiElement.getFirstChild(ASTDelegatePsiElement.java:87)
at com.intellij.psi.util.PsiTreeUtil.getChildrenOfTypeAsList(PsiTreeUtil.java:413)
at nl.hannahsten.texifyidea.psi.impl.LatexCommandsImpl.getParameterList(LatexCommandsImpl.java:38)
at nl.hannahsten.texifyidea.psi.impl.LatexCommandsImplMixin.getRequiredParameters(LatexCommandsImplMixin.kt:128)
at nl.hannahsten.texifyidea.util.parser.LatexCommandsUtilKt.requiredParameter(LatexCommandsUtil.kt:168)
at nl.hannahsten.texifyidea.index.file.LatexIndexableSetContributor.getAdditionalProjectRootsToIndex(LatexIndexableSetContributor.kt:83)
at com.intellij.util.indexing.IndexableSetContributor.getProjectRootsToIndex(IndexableSetContributor.java:33)
at com.intellij.util.indexing.AdditionalIndexableFileSet.collectFilesAndDirectories(AdditionalIndexableFileSet.java:55)
at com.intellij.util.indexing.AdditionalIndexableFileSet.lambda$new$3(AdditionalIndexableFileSet.java:37)
at com.intellij.util.CachedValueImpl.doCompute(CachedValueImpl.java:39)
at com.intellij.util.CachedValueBase.lambda$getValueWithLock$3(CachedValueBase.java:236)
at com.intellij.util.CachedValueBase.computeData(CachedValueBase.java:43)
at com.intellij.util.CachedValueBase.lambda$getValueWithLock$4(CachedValueBase.java:236)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:111)
at com.intellij.openapi.util.RecursionGuard.doPreventingRecursion(RecursionGuard.java:27)
at com.intellij.openapi.util.RecursionManager.doPreventingRecursion(RecursionManager.java:66)
at com.intellij.util.CachedValueBase.getValueWithLock(CachedValueBase.java:237)
at com.intellij.util.CachedValueImpl.getValue(CachedValueImpl.java:49)
at com.intellij.util.indexing.AdditionalIndexableFileSet.isInSet(AdditionalIndexableFileSet.java:71)
at com.intellij.util.indexing.roots.IndexableFilesIndexImpl.shouldBeIndexed(IndexableFilesIndexImpl.java:63)
at com.intellij.util.indexing.FileBasedIndexImpl$5.isInSet(FileBasedIndexImpl.java:296)
at com.intellij.util.indexing.FileBasedIndexImpl.lambda$belongsToProjectIndexableFiles$26(FileBasedIndexImpl.java:1799)
at com.intellij.util.containers.ContainerUtil.find(ContainerUtil.java:872)
at com.intellij.util.containers.ContainerUtil.find(ContainerUtil.java:856)
at com.intellij.util.indexing.FileBasedIndexImpl.belongsToProjectIndexableFiles(FileBasedIndexImpl.java:1799)
at com.intellij.util.indexing.UnindexedFilesScannerStartupKt$findProjectFiles$2$1.invoke(unindexedFilesScannerStartup.kt:251)
at com.intellij.util.indexing.UnindexedFilesScannerStartupKt$findProjectFiles$2$1.invoke(unindexedFilesScannerStartup.kt:248)
at kotlin.sequences.TransformingSequence$iterator$1.next(Sequences.kt:210)
at kotlin.sequences.FilteringSequence$iterator$1.calcNext(Sequences.kt:170)
at kotlin.sequences.FilteringSequence$iterator$1.hasNext(Sequences.kt:194)
at kotlin.sequences.SequencesKt___SequencesKt.toList(_Sequences.kt:809)
at com.intellij.util.indexing.UnindexedFilesScannerStartupKt$findProjectFiles$2.invoke(unindexedFilesScannerStartup.kt:265)
at com.intellij.util.indexing.UnindexedFilesScannerStartupKt$findProjectFiles$2.invoke(unindexedFilesScannerStartup.kt:243)
at com.intellij.openapi.application.rw.InternalReadAction.insideReadAction(InternalReadAction.kt:104)
at com.intellij.openapi.application.rw.InternalReadAction.access$insideReadAction(InternalReadAction.kt:15)
at com.intellij.openapi.application.rw.InternalReadAction$tryReadCancellable$2.invoke(InternalReadAction.kt:94)
at com.intellij.openapi.application.rw.InternalReadAction$tryReadCancellable$2.invoke(InternalReadAction.kt:93)
at com.intellij.openapi.application.rw.CancellableReadActionKt.cancellableReadActionInternal$lambda$2$lambda$1$lambda$0(cancellableReadAction.kt:31)
at com.intellij.openapi.application.impl.AnyThreadWriteThreadingSupport.tryRunReadAction(AnyThreadWriteThreadingSupport.kt:291)
at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:965)
at com.intellij.openapi.application.rw.CancellableReadActionKt.cancellableReadActionInternal$lambda$2$lambda$1(cancellableReadAction.kt:29)
at com.intellij.openapi.progress.util.ProgressIndicatorUtilService.runActionAndCancelBeforeWrite(ProgressIndicatorUtilService.java:66)
at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:155)
at com.intellij.openapi.application.rw.CancellableReadActionKt.cancellableReadActionInternal(cancellableReadAction.kt:27)
at com.intellij.openapi.application.rw.InternalReadAction.tryReadCancellable(InternalReadAction.kt:93)
at com.intellij.openapi.application.rw.InternalReadAction.tryReadAction(InternalReadAction.kt:77)
at com.intellij.openapi.application.rw.InternalReadAction.readLoop(InternalReadAction.kt:64)
at com.intellij.openapi.application.rw.InternalReadAction.access$readLoop(InternalReadAction.kt:15)
at com.intellij.openapi.application.rw.InternalReadAction$readLoop$1.invokeSuspend(InternalReadAction.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
... 4 more
Caused by: java.lang.AssertionError: Stub count (132) doesn't match stubbed node length (131)
at com.intellij.psi.impl.source.FileTrees.lambda$reconcilePsi$4(FileTrees.java:188)
at com.intellij.psi.impl.DebugUtil.performPsiModification(DebugUtil.java:548)
at com.intellij.psi.impl.source.FileTrees.reconcilePsi(FileTrees.java:176)
at com.intellij.psi.impl.source.FileTrees.withAst(FileTrees.java:142)
at com.intellij.psi.impl.source.PsiFileImpl.loadTreeElement(PsiFileImpl.java:216)
... 57 more

</details>

[PHPirates]

Yes, this will never work with commands in the path.

Thanks for the exception, could you try File > Invalidate Cache and Restart and see if the exception is gone?

[maxanier]

I tried restarting without invalidation, restart with invalidation of cache and typing the commands again.
The exception has not occurred again, but the reference is still unresolved (with absolute path to bib file).
(0.9.9-alpha.5)

I don't have enough time to debug this further right now, but I will come back to you once I have tried it on my Linux system.

Thank you for your effort already!

[PHPirates]

Thanks for checking, I can reproduce the problem to some extent, but when I do

• File > Invalidate caches & restart (If you already did this, probably don't need to do this again)
• Wait until indexing is done
• Restart intellij
• File > Cache Recovery > Rescan project indexes

Then it usually started working for me. Once the things are in the index, at least it should keep working.
I don't understand why this last step is needed, I'll try to figure it out.

[maxanier]

I can confirm that it also works for me when executing all 4 steps :)

[maxanier]

However, if you think the solution is to hacky/unstable and you also think that having bib files outside the project root is a bad idea, I can also live without this feature. Maybe I can add a run configuration to copy the external files to the project and run it manually if the external files have been updated.

[PHPirates]

Thanks for checking! I think I'll first try out this simple implementation a bit in alpha releases for the next month and see if there are unexpected consequences regarding performance. If it works I can finetune it later on.

No, having bib files outside the project root is completely fine, it's just that supporting them automatically is not easy.
I clarified the options you have in the documentation: https://hannah-sten.github.io/TeXiFy-IDEA/bibtex.html#working-with-bibtex-files-outside-the-project