What would it take to support UTF16 encoding?

[kentookura]

@TOTBWF

As reported by @Trebor-Huang in Trebor-Huang/vscode-forester#11 (comment), it seems that the VSCode LSP client refuses to work with UTF16 encoded positions. I was wondering if you could say a few words about the trouble you mention here?

Thanks!

[favonia]

@kentookura To support UTF-16 efficiently, we need to avoid the recalculation of the byte offset of a UTF-16 unit when files change. This can be done by... (1) inefficient recalculation (oops) or (2) some smart data structure maintaining the mapping.

[favonia]

For ASCII printable characters, I believe byte offsets and UTF-16 units coincide.

[TOTBWF]

Ugh, VSCode being a bad citizen yet again...

As for the data structure, I think some sort of rope segmented at points where UTF-16 and UTF-8 offsets disagree ought to work. Nodes further up the tree could then store both the UTF-8 and UTF-16 range that the subtree covers, so we'd get efficient queries in both directions.