why does \b differ in behavior from other regex engines such as Swift's?

[TOETOE55]

"\b" represents word boundaries in regex. But it seems to behave differently in different implementations:

In regex crate, which "can't" was separated

fn main() {
    let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?";
    
    let re = Regex::new(r"\b").unwrap();
    let res = re.split(s).collect::<Vec<&str>>();
    // ["", "The", " ", "quick", " (\"", "brown", "\") ", "fox", " ", "can", "'", "t", " ", "jump", " ", "32", ".", "3", " ", "feet", ", ", "right", "?"]
    println!("{res:?}");
}

in swift 5.7

let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?"

let words = s.split(separator: /\b/)
// ["The", " ", "quick", " ", "(", "\"", "brown", "\"", ")", " ", "fox", " ", "can\'t", " ", "jump", " ", "32.3", " ", "feet", ",", " ", "right", "?"]
print(words)
// 

In unicode-segmentation crate, it's same as regex in swift, according to the [Unicode Standard Annex #29(http://www.unicode.org/reports/tr29/) rules.

fn main() {
    let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?";
    
    let res = s.split_word_bounds().collect::<Vec<&str>>();
    // ["The", " ", "quick", " ", "(", "\"", "brown", "\"", ")", " ", "fox", " ", "can't", " ", "jump", " ", "32.3", " ", "feet", ",", " ", "right", "?"]
    println!("{res:?}");
}

[BurntSushi]

It's unclear exactly what you're asking, but I took a stab at it by renaming the issue title to the question I think you're asking.

This crate defines \b as a Unicode word boundary. It is documented in the section on syntax. In short, it matches any position where there is a \w on one side and \W (or the beginning/end of a string) on the other. This is further documented in UNICODE.md, which specifically links to UTS #18 RL1.4. So in short, this crate defines \b as a "simple Unicode word boundary" according to the Unicode Technical Standard on regular expressions.

As for Swift, I went to its documentation page for regex and... found absolutely nothing about its supported syntax. Looking a little closer, I did find wordBoundaryKind, and that led me to RegexWordBoundaryKind. That in turn seems to have two possible values: default and simple. The wording for default suggests that it implements UTS #18 RL2.3. That in turn essentially says, "define \b in a way that is consistent with word boundary segmentation in UAX #29."

Indeed, after some digging into Swift's regex design documents, it is specifically documented to use UAX #29 word segmentation as its definition for \b.

Swift's Unicode support in its regex engine is impressive. Basically, this library implements level 1 support of UTS #18, where as Swift's library seems to implement level 2 support of UTS #18. This crate will never touch level 2 because it's not clear how to implement much of it in the context of a finite automata engine. For example, Swift regexes likely take worst case exponential time in the size of the haystack (I could not find a single mention of that in any of their documentation) but this library takes worst case linear time in the size of the haystack (with the regex held constant). This runtime guarantee is only achieved by restricting the scope of the library and limiting what features it supports to actual regular expressions. In contrast, Swift regexes appear to be more like PEGs:

The closest formal analogue to what we are proposing are Parsing Expression Grammars ("PEGs"), which describe a recursive descent parser. Our alternation is ordered choice and we support possessive quantification, recursive subpattern calls, and lookahead. However, we are first and foremost providing a regexy presentation: quantification, by default, is non-possessive.

(To see the above, you'll need to expand the "details" at the end of the Introduction section.)

So, bottom line:

• This crate is working as intended. \b is a crude approximation of a word boundary, but it is at least Unicode aware. (Many regex engines only provide ASCII word boundaries.)
• This crate limits its expressive power in order to provide time and space complexity guarantees.
• Swift's regex engine appears to be designed to enable a lot of expressive power. In this context, it is easier to implement things that require significant look-around (as default word boundaries undoubtedly do).

[TOETOE55]

Thank you so much!