Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(core): lighter token explanation with scopeName #739

Merged
merged 3 commits into from
Aug 14, 2024
Merged

feat(core): lighter token explanation with scopeName #739

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1d818e7
feat: explanation `scopeName`
antfu Aug 12, 2024
165a7de
chore: lint
antfu Aug 12, 2024
5d62299
perf: update
antfu Aug 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions packages/core/src/code-to-tokens-base.ts
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if starting on line 129 can be changed to

if (options.includeExplanation && options.includeExplanation !== 'scopeName') {

since the themeSettingsSelectors do not end up being used in that case

Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,11 @@ function _tokenizeWithTheme(
offset += tokenWithScopesText.length
token.explanation.push({
content: tokenWithScopesText,
scopes: explainThemeScopes(themeSettingsSelectors, tokenWithScopes.scopes),
scopes: explainThemeScopes(
themeSettingsSelectors,
tokenWithScopes.scopes,
options.includeExplanation,
),
})

tokensWithScopesIndex! += 1
Expand All @@ -236,14 +240,17 @@ function _tokenizeWithTheme(
function explainThemeScopes(
themeSelectors: ThemeSettingsSelectors[],
scopes: string[],
includeExplanation: TokenizeWithThemeOptions['includeExplanation'],
): ThemedTokenScopeExplanation[] {
const result: ThemedTokenScopeExplanation[] = []
for (let i = 0, len = scopes.length; i < len; i++) {
const parentScopes = scopes.slice(0, i)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parentScopes is not needed if includeExplanation === 'scopeName', so the slice could be avoided. This doesn't have a big impact though; only a 1~2% savings in my tests, so not sure if it's worth the tradeoff in readability

const scope = scopes[i]
result[i] = {
scopeName: scope,
themeMatches: explainThemeScope(themeSelectors, scope, parentScopes),
themeMatches: includeExplanation === 'scopeName'
? undefined
: explainThemeScope(themeSelectors, scope, parentScopes),
}
}
return result
Expand Down
8 changes: 6 additions & 2 deletions packages/core/src/types/tokens.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type { GrammarState } from '../grammar-state'
import type { IRawThemeSetting } from '../textmate'
import type { SpecialLanguage } from './langs'
import type { SpecialTheme, ThemeRegistrationAny } from './themes'
import type { CodeOptionsThemes } from './options'
Expand Down Expand Up @@ -36,7 +37,7 @@ export interface CodeToTokensWithThemesOptions<Languages = string, Themes = stri

export interface ThemedTokenScopeExplanation {
scopeName: string
themeMatches: any[]
themeMatches?: IRawThemeSetting[]
}

export interface ThemedTokenExplanation {
Expand Down Expand Up @@ -149,9 +150,12 @@ export interface TokenizeWithThemeOptions {
/**
* Include explanation of why a token is given a color.
*
* You can optionally pass `scopeName` to only include explanation for scopes,
* which is more performant than full explanation.
*
* @default false
*/
includeExplanation?: boolean
includeExplanation?: boolean | 'scopeName'

/**
* A map of color names to new color values.
Expand Down
Loading
Loading