Performance expectations of TM4E #560

jonahgraham · 2023-06-16T14:41:41Z

jonahgraham
Jun 16, 2023

I am working on the CDT LSP plug-ins and we are currently using TM4E to support syntax highlighting.

I am wondering what the performance expectations of TM4E are? Is there a plan on how to support large files? For example, in VSCode if I open the attached file it only colorizes the visible lines, but in TM4E I think it is trying to colorize the whole file.

The behaviour I see today is that on larger files this is very slowly and runs substantially on the UI thread, causing the entire Eclipse to become unresponsive. It took approximately 15 minutes before the UI returned to normal use.

Here are some stack traces - the UI thread usage may be dominated by waiting for a lock?

"main" #1 prio=6 os_prio=0 cpu=19432.94ms elapsed=82.09s tid=0x00007fd198028e90 nid=0x13a2e waiting for monitor entry  [0x00007fd1a00f4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.eclipse.tm4e.core.model.AbstractModelLines.getOrNull(AbstractModelLines.java:147)
        - waiting to lock <0x0000000098580240> (a java.util.LinkedList)
        at org.eclipse.tm4e.core.model.TMModel.getLineTokens(TMModel.java:286)
        at org.eclipse.tm4e.ui.text.TMPresentationReconciler.colorize(TMPresentationReconciler.java:540)
        at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.colorize(TMPresentationReconciler.java:410)
        at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.lambda$0(TMPresentationReconciler.java:386)
        at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener$$Lambda$1335/0x000000010096d970.run(Unknown Source)
        at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
        at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:132)

---

"org.eclipse.tm4e.ui.internal.model.TMDocumentModel" #103 daemon prio=1 os_prio=0 cpu=44439.56ms elapsed=47.43s tid=0x00007fd198795200 nid=0x13b48 runnable  [0x00007fd068fae000]
   java.lang.Thread.State: RUNNABLE
        at java.util.LinkedList.node(java.base@17.0.7/LinkedList.java:580)
        at java.util.LinkedList.get(java.base@17.0.7/LinkedList.java:481)
        at org.eclipse.tm4e.core.model.AbstractModelLines.getOrNull(AbstractModelLines.java:149)
        - locked <0x0000000098580240> (a java.util.LinkedList)
        at org.eclipse.tm4e.core.model.TMModel.getLineTokens(TMModel.java:286)
        at org.eclipse.tm4e.ui.internal.utils.MarkerUtils.updateTextMarkers(MarkerUtils.java:119)
        at org.eclipse.tm4e.ui.internal.utils.MarkerUtils.updateTextMarkers(MarkerUtils.java:66)
        at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.modelTokensChanged(TMPresentationReconciler.java:390)
        at org.eclipse.tm4e.core.model.TMModel.emit(TMModel.java:279)
        at org.eclipse.tm4e.core.model.TMModel.buildAndEmitEvent(TMModel.java:273)
        at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.revalidateTokens(TMModel.java:121)
        at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.run(TMModel.java:101)

Here is a gzipped file that experiences the problem (part of LLVM Project) - tested with Eclipse IDE for C/C++ 2023-06 with TM4E language pack 0.2.0.202305071450 installed and opening file in generic editor.
AArch64GenAsmMatcher.cc.gz

Opening the same file in the C/C++ Editor turns on scalability mode which disables some of the more advanced features (like full semantic highlighting) but the syntax highlighting works seemingly instantaneously.

mickaelistria · 2023-06-16T14:49:49Z

mickaelistria
Jun 16, 2023
Maintainer

Is there a plan on how to support large files?

No clear plan as I don't think anyone deeply "plans" for TM4E, most active contributors are reactive to issues, updates and so on, but no one has taken an initiative to do a clear performance audit and improvements. Am I right @sebthom or do you have some secret plan?
In any case, TM4E is really open to any contributor willing to improve the performance, from "micro" optimizations to entire re-design of the parser. The current way of doing the work is not set in stone and is almost fully internal, so it's very open to change.

For example, in VSCode if I open the attached file it only colorizes the visible lines, but in TM4E I think it is trying to colorize the whole file.

I believe that TM4E colorizes the visible lines with higher priority, and then it will try to colorize other invisible lines in a background work. But it is assumed that coloration of background is "cheap" and not locking the UI Thread.

The UI thread usage may be dominated by waiting for a lock?

You can report an issue about it as it should be enough information with the stack traces to act on that particular issue, the lock conditions can probably be relaxed (ie only lock if we don't already have data for the line or if the line is marked dirty, but return immediately otherwise).

1 reply

jonahgraham Jun 16, 2023
Author

You can report an issue

I created an issue about the lock as that seems to make the situation worse - #561

sebthom · 2023-06-16T14:58:54Z

sebthom
Jun 16, 2023
Maintainer

After #540 I implemented a considerable amount of performance optimizations. Please give the latest snapshot a try.

The main issue I currently see is that when opening very large JSON files the LSP4E freezes the UI while trying to populate the outline.
Haven't had time to open a respective issue against LSP4E yet. I am currently working on the issue where reformatting the source code breaks syntax highlighting.

3 replies

jonahgraham Jun 16, 2023
Author

I tried without matching LSP4E language server to try to isolate just to TM4E's contribution.

Perhaps other languages are not as badly affected, I note that the CPP language is ~16,000 lines, while dart is ~500 lines

jonahgraham Jun 16, 2023
Author

After #540 I implemented a considerable amount of performance optimizations. Please give the latest snapshot a try.

You sure did - thanks! The time to colorize the whole file went from ~15 minutes wall time to ~2 minutes and it does not lock the main thread anymore.

I believe that TM4E colorizes the visible lines with higher priority,

I don't see this happening in practice. If I scroll to the end of the file immediately after opening, it still takes ~2 minutes for the last line to be colorized.

sebthom Jun 16, 2023
Maintainer

I tried the AArch64GenAsmMatcher.cc file with the latest snapshot. I cannot reproduce the locks. However, I see that the org.eclipse.ui.internal.genericeditor.DefaultWordHighlightStrategy freezes the UI doing:

   java.lang.Thread.State: RUNNABLE
        at java.util.regex.Pattern$CharPropertyGreedy.match([email protected]/Pattern.java:4299)
        at java.util.regex.Pattern$StartS.match([email protected]/Pattern.java:3641)
        at java.util.regex.Matcher.search([email protected]/Matcher.java:1728)
        at java.util.regex.Matcher.find([email protected]/Matcher.java:745)
        at org.eclipse.ui.internal.genericeditor.DefaultWordHighlightStrategy.findCurrentWord(DefaultWordHighlightStrategy.java:133)
        at org.eclipse.ui.internal.genericeditor.DefaultWordHighlightStrategy.applyHighlights(DefaultWordHighlightStrategy.java:88)
        at org.eclipse.ui.internal.genericeditor.DefaultWordHighlightStrategy.lambda$0(DefaultWordHighlightStrategy.java:70)
        at org.eclipse.ui.internal.genericeditor.DefaultWordHighlightStrategy$$Lambda$1122/0x00000008009cb328.selectionChanged(Unknown Source)
        at org.eclipse.jface.text.TextViewer.firePostSelectionChanged(TextViewer.java:2718)
        at org.eclipse.jface.text.TextViewer.firePostSelectionChanged(TextViewer.java:2648)
        at org.eclipse.jface.text.TextViewer$2.run(TextViewer.java:2627)
        at org.eclipse.swt.widgets.Display.runTimer(Display.java:4152)
        at org.eclipse.swt.widgets.Display.messageProc(Display.java:3272)
        at org.eclipse.swt.internal.win32.OS.DispatchMessage(Native Method)
        at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3640)
        at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1155)
        at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
        at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1046)
        at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:155)
        at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:643)
        at org.eclipse.ui.internal.Workbench$$Lambda$183/0x00000008002a2798.run(Unknown Source)
        at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
        at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:550)
        at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:171)
        at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:152)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:77)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke([email protected]/Method.java:568)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1440)

This happens on every click inside the editor. That is another issue I also saw when opening large JSON files.

jonahgraham · 2023-06-16T15:12:15Z

jonahgraham
Jun 16, 2023
Author

I am going to close this discussion for now as with @sebthom's recent improvements the bulk of the problem has gone away.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance expectations of TM4E #560

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Performance expectations of TM4E #560

jonahgraham Jun 16, 2023

Replies: 3 comments · 4 replies

mickaelistria Jun 16, 2023 Maintainer

jonahgraham Jun 16, 2023 Author

sebthom Jun 16, 2023 Maintainer

jonahgraham Jun 16, 2023 Author

jonahgraham Jun 16, 2023 Author

sebthom Jun 16, 2023 Maintainer

jonahgraham Jun 16, 2023 Author

jonahgraham
Jun 16, 2023

Replies: 3 comments 4 replies

mickaelistria
Jun 16, 2023
Maintainer

jonahgraham Jun 16, 2023
Author

sebthom
Jun 16, 2023
Maintainer

jonahgraham Jun 16, 2023
Author

jonahgraham Jun 16, 2023
Author

sebthom Jun 16, 2023
Maintainer

jonahgraham
Jun 16, 2023
Author