aws-sdk instrumentation: can direct lambda calls propagate context?

[johnbley]

Context: One lambda (A) calling another (B) directly via the aws sdk; both are instrumented with
a lambda wrapper which installs the library form of the aws-sdk instrumentation.

InvokeRequest req = InvokeRequest.builder().functionName("LambdaB").build();
lambdaClient.invoke(req);

Finding 0: Our instrumentation produces a server span from A, a client span for A calling B,
and a server span from B, as expected. However, the otel context is not propagated. X-Ray sees this
as one trace, but otel breaks and sees it as two traces.

Desired outcome: The configured propagators (e.g., the default tracecontext,baggage) pass context
appropriately to the next tier; one otel trace is kept intact across the call.

Finding 1: TracingExecutionInterceptor is hardcoded to inject context with the AwsXrayPropagator
(when configrued to use xray), and not any others.

Finding 2: When I change that code to inject using all configured propagators, and
print out the resulting headers, I see that the instrumentation sets headers appropriately. However:

Finding 3: Inbound instrumentation on B fails to see this context (still get two separate traces).

Finding 4: Inbound lambda instrumentation attempts context extraction only if the input object is
instanceof APIGatewayProxyRequestEvent. In this direct call, B's input is an empty LinkedList as no api
gateway sits between the two components.

Finding 5: Based on further debug-printing, nothing in the instrumentable surface of the aws lamba
RequestHandler (neither the input object nor the Context) seem to have access to the headers set
by the caller. It appears to my untrained eye that the lambda runtime seems to treat a direct lambda call
through the sdk as though it did not actually happen over http?

I hesitate to raise this as an Issue since I don't know if it's actually possible with the
current access that otel instrumentation has. So:

Question: who can I talk to who has expertise in this to either (a) figure
out how to propagate context in a way that inbound instrumentation can see it, or
(b) lobby amazon to give otel the same instrumentable surface as xray to make this work?

[laurit]

@rapphil could you help with this

[trask]

(also cc @tylerbenson)

[tylerbenson]

@johnbley Any chance you could make it to one of the Lambda SIG meetings? (We have one today.) I don't have experience with direct lambda invocation like this and would like to get a better understanding.

[johnbley]

@johnbley Any chance you could make it to one of the Lambda SIG meetings? (We have one today.) I don't have experience with direct lambda invocation like this and would like to get a better understanding.

Yes, though I will be a few minutes late.

[johnbley]

We had a discussion at the FaaS SIG meeting where it was suggested that my intuition was right - that the direct call to another lambda through the aws-sdk was being treated as "not through http", and otel propagation headers were being stripped off (but xray was still allowed to work correctly).

Based on a discussion there, I tried tunneling the otlp context through xray, which involved modifying the inbound lambda instrumentation to use a modified copy of the AwsXrayEnvSpanLinksExtractor (to treat as the parent rather than a span link). This "worked" in the sense that I got a coherent "A calls B" otel trace out of it, but this also kinda broke the XRay UI - the xray trace had both the A and B calls in them but they were two disconnected roots rather than the proper shape.

It seems that our options are:

• (A) adjust the spec so that, for inbound lambda, if otel headers are available they are used as parent and xray context is added as a link, or if otel headers are not available the xray context is treated as parent (possibly with configuration to get this behavior). Or:
• (B) petition aws to allow otel headers the same treatment as xray (e.g., by making "which headers propagate in direct lambda calls" a config option).

Thoughts?

[tylerbenson]

Thanks for the summary of the discussion.

I think some of this issue will be resolved when we've implemented this new change:
open-telemetry/semantic-conventions#354

That said, I think it would still be great if AWS could expand their context propagation support, but I'm not holding my breath.

[mariojonke]

For aws-sdk invoke (non-async) API there is actually another option to propagate the trace context by injecting into LambdaContext.ClientContext. the js aws-sdk instrumentation has this already implemented, see here.

[johnbley]

Thank you so much @mariojonke! This is a promising lead! Within the existing TracingExecutionInterceptor I was able to get a context propagated that was visible by the inbound instrumentation on the other side. In particular, by setting

    builder.putHeader("X-Amz-Client-Context", /* base64 of* "{ \"custom\" : { \"traceparent\" :  \"<otel trace context>\"}  }");

this was visible to context.getClientContext().getCustom() instrumentation on the B (receiving) lambda. And it doesn't affect the payload of the app or require config changes to either side. Nor does it break X-Ray behavior by tunneling through it. It's a great design approach ("look for the traceparent http header when available, and when it's not look for it in a custom field in the client context).

Unfortunately it looks like all the apis available to the instrumentable surface of the java aws-sdk operate on the entirety of the this client context header, meaning that a proper implementation of this (taking into account that some other piece of code may also be using the header, and that it's not additive like a comma-separated http header) would have to play the detect -> parse/unpack -> alter/insert -> repack/emit game through the base64 json structure. That's doable but requires a fair bit of extra code and infrastructure. Still seems like the right approach to me, though, unless somebody sees something I didn't and has a cleaner way to approach the java-specific details here?

[johnbley]

For cross-referencing purposes, see #11675