[serve] Reduce cardinality for the route metric tag

[edoakes]

Why are these changes needed?

Modifies the route attribute used for logs and metrics to limit its cardinality (see linked issue).

After this change, the behavior will be:

• If the user is not using FastAPI, the route will always be the route_prefix.
• If the user is using FastAPI, the route will be the route_prefix + the matched route string.

Take the following example:

@serve.deployment
@serve.ingress(fastapi_app)
class A:
    @fastapi_app.get("/{user_id}")
    def dynamic_subpath(self, user_id: str) -> str:
        return ...

serve.run(A.bind(), route_prefix="/prefix")

Previously, this could have infinite cardinality in the metrics tags (a unique tag value for each user). With this change, the tag would instead be: "/prefix/{user_id}".

The "not found" response has also been modified to not include a route tag.

TODO:

• Add unit tests for get_asgi_route_name
• Add integration tests to test_metrics.py
• Add tests for the replica request context

Related issue number

Closes #47999

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[GeneDer]

Some nits, but overall LGTM!

[GeneDer]

Non-blocker: We can probably create a named tuple for state object to ensure those required keys exist and accessible.

[GeneDer]

Non-blocker: We can probably refactor this into ProxyRequest and implement them in ASGIProxyRequest and gRPCProxyRequest differently

[edoakes]

yeah I was highly confused at the fact that our "route" was just an app name for gRPC. the "method" field also seems to be getting overwritten to something else that actually looks like a route..

[GeneDer]

Yea, I think it depends on the context. HTTP we have methods such as "GET", "POST"...etc. gRPC has concept of service methods so we used in those in metrics/ logs. Also, in gRPC we use the app name to route so that's why the equivalent route_path was set to app names.