Lifetime confusion around replace_all closures.

[tpoliaw]

I am trying to pass a closure to replace_all to replace all matching tokens. There are cases depending on external state where some matches don't need to be replaced so I was hoping to return the original string to save allocating a new one with the same contents (If this is misguided let me know but I don't think it affects the question).

My initial solution was to return a Cow<str> with a lifetime matching that of the Captures (which in turn is the lifetime of the matched text). The docs suggest that anything that is AsRef<str> is ok to return which (I think?) Cow<str> is.

use std::borrow::Cow;
use regex::{Captures, Regex};

fn replace<'a>(input: &'a str) -> Cow<'a, str> {
    let pattern = Regex::new("foo(bar)?").unwrap();
    pattern.replace_all(input, |cap: &Captures| expand(cap))
}

fn expand<'a>(cap: &Captures<'a>) -> Cow<'a, str> {
    Cow::Borrowed(cap.get(0).unwrap().as_str())
}

The compiler is happy with the expand function so I'm assuming that the lifetimes are valid there but the closure in the replace function is not accepted

   |
15 |     pattern.replace_all(input, |cap: &Captures| expand(cap))
   |                                 ---           - ^^^^^^^^^^^ returning this value requires that `'1` must outlive `'2`
   |                                 |             |
   |                                 |             return type of closure is Cow<'2, str>
   |                                 has type `&regex::Captures<'1>`

As far as I can see, the specifiers in the expand method say that the lifetime of the returned Cow is the same as the lifetime of the Captures passed in but the compiler sees them as different, '1 and '2.

Where is my intuition going wrong here? And is there a way to do this so that the compiler is happy without returning a new String from the expand method every time?

From discussion on reddit, this can be worked around by passing the original &str to the expand method and returning a slice of that instead. I'd assumed that the lifetimes of the input &str and the matched &str would be the same.

fn replace<'a>(input: &'a str) -> Cow<'a, str> {
    let pattern = Regex::new("foo(bar)?").unwrap();
    pattern.replace_all(input, |cap: &Captures| expand(input, cap))
}

fn expand<'a>(input: &'a str, cap: &Captures) -> Cow<'a, str> {
    Cow::Borrowed(&input[cap.get(0).unwrap().range()])
}

[BurntSushi]

NOTE: See @sgrif's comments below and their PR in #776 for the actual answer. I also created #777 to track this particular issue, since it can't be fixed until a hypothetical regex 2.0.0 release.

I believe the issue rests in how Replacer is implemented for closure types:

impl<F, T> Replacer for F
where
    F: FnMut(&Captures<'_>) -> T,
    T: AsRef<str>,
{
    fn replace_append(&mut self, caps: &Captures<'_>, dst: &mut String) {
        dst.push_str((*self)(caps).as_ref());
    }
}

I think the key part is this:

    F: FnMut(&Captures<'_>) -> T,
    T: AsRef<str>,

That is, there is nothing connecting the lifetime of '_ in &Captures<'_> with T. So in a generic context, I guess that the compiler can't reason about a specific closure that returns something whose lifetime is connected with the lifetime of the text stored in Captures. Maybe? That doesn't feel quite right to me, but it does feel like the problem is in the definition of the Replacer trait.

I'm not particularly confident in this answer. I'll broadcast this and see if someone smarter than me can figure this out and whether there's a work-around.

[sgrif]

It would be a breaking change if Replacer is part of the public API, but if it's internal this change is purely internal in scope

[sgrif]

I'll open the PR, we can discuss there

[sgrif]

#776

[tpoliaw]

Thank you both for the explanation of the problem and also the quick turnaround of the PR. Is there a rough timeline for regex 2.0 or is it still a distant "whenever there are enough breaking changes waiting to be merged" thing?

[BurntSushi]

Is there a rough timeline for regex 2.0 or is it still a distant "whenever there are enough breaking changes waiting to be merged" thing?

Much closer to the latter. It's not just about number either, but substance. The answer could be "never."

[JanBeh]

See this post on URLO for a possible workaround using a newtype wrapper and unstable/experimental closure lifetime binders.