Disable all metrics by default and then enable specific library metrics

[srinusanduri]

The agent configuration properties -Dotel.instrumentation.common.default-enabled=false and -Dotel.instrumentation.[name].enabled=true mentioned here allows us to disable all instrumentations by default and enable specific library instrumentations. But this applies for both traces and metrics instrumentation.

Our requirement is to enable traces by default and enable metrics for specific libraries only. We want metrics instrumentation to be disabled by default and then enable specific libraries for metrics instrumentation without affecting trace instrumentation. Please suggest the configurations required to achive this behavior.

Thank you

[trask]

Hi @srinusanduri! Which metrics are you wanting to disable?

[srinusanduri]

Hi @trask, I want to disable all metrics except for agent monitoring metrics. The thing I am trying to achieve is monitoring the opentelemetry-java-agent itself. For that I would like to export auto-instrumented agent metrics like otel_scope_name="io.opentelemetry.sdk.trace", otel_scope_name="io.opentelemetry.exporters.otlp-grpc" etc that helps us monitor the agent.

We are using opentelemetry-java-agent for tracing alone, not for metrics and logs. We have a separate prometheus scaping system setup for collecting metrics. However, to collect opentelemetry-java-agent performance metrics alone, we want to enable metrics export (OTEL_METRICS_EXPORTER=prometheus) in the agent with agent metrics alone enabled, all other metrics disabled.

For example, we would like to specifically enable below agent performance metrics

otel_scope_name="io.opentelemetry.sdk.trace"
otel_scope_name="io.opentelemetry.exporters.otlp-grpc"

And disable all other auto-instrumented metrics like otel_scope_name="io.opentelemetry.micrometer-1.5" otel_scope_name="io.opentelemetry.grpc-1.6" otel_scope_name="io.opentelemetry.tomcat-7.0" etc.

Here are some of the agent metrics we are interested in collecting to monitor the agent itself.

# HELP processedSpans_ratio_total The number of spans processed by the BatchSpanProcessor. [dropped=true if they were dropped due to high throughput]
# TYPE processedSpans_ratio_total counter
processedSpans_ratio_total{dropped="false",otel_scope_name="io.opentelemetry.sdk.trace",processorType="BatchSpanProcessor"} 6.0
# HELP queueSize_ratio The number of items queued
# TYPE queueSize_ratio gauge
queueSize_ratio{otel_scope_name="io.opentelemetry.sdk.trace",processorType="BatchSpanProcessor"} 0.0


# TYPE otlp_exporter_exported_total counter
otlp_exporter_exported_total{otel_scope_name="io.opentelemetry.exporters.otlp-grpc",success="true",type="span"} 6.0
# TYPE otlp_exporter_seen_total counter
otlp_exporter_seen_total{otel_scope_name="io.opentelemetry.exporters.otlp-grpc",type="span"} 6.0

We want the agent to export only these agent metrics, and disable all other metrics (for example, micrometer). Could you suggest the configurations required to achieve this?

[trask]

you could try to use metric view configuration file to disable the ones you don't want: https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/incubator#view-file-configuration

[umakarnati]

Hi @trask

I have tried your suggestion using below view config to drop all metrics except agent specific one.. but some reason its not working as expected and droping all metrics , I think it may be due to conflicting selector config at the end with * .. but as per this it looks DropAggregation should not result in conflicting metric identities. could you check and suggest correct view config?

`

• selector:
  instrument_name: otlp_exporter_exported_total
  instrument_type: COUNTER
  view:
  aggregation: default
• selector:
  instrument_name: otlp_exporter_seen_total
  instrument_type: COUNTER
  view:
  aggregation: default
• selector:
  instrument_name: queueSize_ratio
  instrument_type: OBSERVABLE_GAUGE
  view:
  aggregation: default
• selector:
  instrument_name: "*"
  view:
  aggregation: drop

`

[umakarnati]

it looks selector instrument_name config doesn't support all regex pattern charatcters except "*" and "." ,otherwise I could have simplified my view config, some thing like below to exclude all metrics except agent specific metrics

-selector:
instrument_name: ^(otlp_exporter_exported_total| otlp_exporter_seen_total queueSize_ratio)*
view:
aggregation: drop

[trask]

yeah, this is per specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#instrument-selection-criteria

[umakarnati]

thanks @trask
is there any other way you think we can use to keep specific metric and drop rest ? and do you know if there any specific reason for not adding full regex support in the specification ?

[trask]

I think you should be able to have one view to drop *, and additional views to keep the ones you want

check out the example at https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#view-examples

# Only Counter X will be exported, with the default configuration (match-all drop aggregation does not result in
# conflicting metric identities)
meter_provider
    .add_view("X")
    .add_view("*", aggregation=DropAggregation()) # a wildcard view to disable all instruments
    .add_metric_reader(PeriodicExportingMetricReader(ConsoleExporter()))

[umakarnati]

yeah after seeing that example .. I tried it here #10284 (comment) .. but its not working as mentioned

[trask]

can you try to create a reproducer for this using opentelemetry-sdk-extension-incubator directly (instead of indirectly via the java agent)?

check out https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/incubator#view-file-configuration

it may just be a bug in that module, in which case you can open a bug report in https://github.com/open-telemetry/opentelemetry-java

[jack-berg]

Hi @umakarnati - it looks live you've got some typos in your view config. You're using _ as the delimeter, which is how the metric names look once they're translated to prometheus format, but in the SDK everything is . delimeted. Also, your metric names include the _total suffix, which is only appended as a part of translating to prometheus format.

I would recommend more simply selecting the meter names you want to enable, instead of the instrument names. I believe the following reflects the config you're going for:

- selector:
    meter_name: io.opentelemetry.exporters.otlp-http
  view:
    aggregation: default
- selector:
    meter_name: io.opentelemetry.sdk.logs
  view:
    aggregation: default
- selector:
    meter_name: io.opentelemetry.sdk.trace
  view:
    aggregation: default
- selector:
    instrument_name: "*"
  view:
    aggregation: drop

It captures internal metrics from the OTLP span / log exporters and batch span / log processors. It worked for me.

[umakarnati]

Thanks @jack-berg .. it worked after fixing the names.. But I liked the idea of using meter_name to cover all of them ..thanks for your suggestion