Enrich traces during compaction/ingestion

[ovidiubuligan]

If we have a lot of microservices in different languages , I want to add the userId or some well known dimension to all microservices since when it first appeared . Otel supports this via baggage and modifying all services to update their spans and add userId to each span (https://opentelemetry.io/docs/concepts/signals/baggage/) .

Let's say we have a nginx gateway that emits traces but doesn't have the option to enhance the traces with userId (taken from JWT token for example) . During compaction this attribute can be added automatically to all spans of a trace , even to 3'rd party tools that generate trace telemetry, by enriching all spans from a trace if certain spans contain some attribute.

Can this be done with tempo compactor ? (PS: I don't think OTEL collector transformprocessor is able to do this)

[joe-elliott]

Post processing on traces is possible, but not deterministically. Meaning that some of your traces will have the information correctly propagated and some won't. This is mostly dependent on how they are received by Tempo. If the spans for a trace are all received by Tempo in a short time period then the enrichment will occur as you're wanting.

RF1 rearchitecture will also change this b/c there will be only 1 copy of a trace in your backend. Currently 3 are flushed to the backend.

This method:

https://github.com/grafana/tempo/blob/main/tempodb/encoding/vparquet4/nested_set_model.go#L25

is currently called every time a trace is brought together as a whole. We currently use it to calculate the nested set model and summarize service information, but it could be extended to perform other tasks like the one you've highlighted.