does disabling Unicode mean I can't search for \u2603?

[d3lm]

Hey there 👋

Is it correct that if I disable all unicode features that a regex won't match any unicode at all? I am a bit confused when I read the documentation.

In my use case, I want to apply a regex on a byte slice that represents some file content and that slice could potentially also contain unicode characters. If I were to disable unicode and search for say \u2603 in a file that contains those bytes (☃), does that mean it wouldn't find it?

Basically I am always operating on &[u8] so I guess I can use the bytes submodule. But I am wondering if I can disable unicode because binary size is very important.

[BurntSushi]

I am a bit confused when I read the documentation.

To be clear, and to make sure we're talking about the same thing, these are the docs that I think should answer this question: https://docs.rs/regex/1.5.4/regex/#crate-features

In particular, the last couple sentences of the second paragraph I believe are crucial here. Do they answer your question? If not, there might be some other misunderstanding (perhaps on my part) going on here:

Stated differently, enabling or disabling any of the features below can only add or subtract from the total set of valid regular expressions. Enabling or disabling a feature will never modify the match semantics of a regular expression.

So in other words, if compiling a regex like \u2603 works with all of the Unicode features disabled, then it is guaranteed to have the same semantics as it would if any of those features were enabled.

Now maybe you're talking about the u regex flag, in which case, if you try matching \u2603 with u disabled, then you should get a regex compilation error. But the u regex flag is unrelated to binary size since it's a runtime option, so I'm guessing this is not what you meant.

Also,

so I guess I can use the bytes submodule

When you say, "so I guess I can use...," what do you mean here? If you're matching on &[u8], then the bytes sub-module API is exactly the right thing you want to be using, and this is (or should be) orthogonal from the rest of your question. The bytes API has all of the Unicode features of the main top-level module API.

Also, you speak of "disabling Unicode," but I think what you mean is, "disable all crate features starting with unicode." If you take a look at those features, none of them correspond to whether you can search for a Unicode codepoint. And in particular, searching for a Unicode codepoint doesn't require any Unicode tables, and thus, there is no binary size impact nor is there any reason to disable such functionality at compile time (AFAIK).

binary size is very important

You'll likely also want to disable the perf features.

[d3lm]

I could use https://docs.rs/glob/0.2.11/glob/struct.Pattern.html but then I'd have to execute each glob individually and I think the optimization of globset is pretty cool and if you have a lot of globs, which I would most likely have, then it does make a difference.

[BurntSushi]

No, I don't. Especially given that I don't know what problem you're trying to solve or why binary size is "very important." The only place I can conceive of binary size being important is potentially in WASM or in certain embedded contexts. If you're not in either of those, I'd probably urge you to re-evaluate whether binary size is actually important or not.

At that point, you'll need to consider which is more important: the use of globs, performance or binary size. In theory, this is a false choice, but only in theory. In practice, you'll have to choose between those three things (or carving your own path, e.g., by implementing your own globbing facilities in a way that meets your demands).

[d3lm]

Yea, binary size is important because I am compiling for the Web therefore WASM. What I am trying to do is to build a simplified version of ripgrep used by VSCode. So I need to be able to glob on file paths (strings) and then search for patterns in the bytes (the file content). It doesn't have to be as optimised as ripgrep, also because it would significantly increase the size of the binary. Also, as mentioned earlier, I am operating on an in memory file system and not a real file system.

[BurntSushi]

More practically, I think these are your choices:

1. Recognize that binary size is important, but mark it as a v2 feature. Move forward without caring too much about it.
2. Fork globset and trim it down.
3. Use the glob crate and be okay with the glob perf loss. You might be able to get farther here than you think. e.g., I think this is how ag implements gitignore logic. ag does not have anything like globset.
4. Build your own globbing implementation. This is, after all, what I did.

[d3lm]

Yea that makes sense. Thanks a lot!