Export to file, receive from file - lossless?

[perry-mitchell]

Hello!

I have a situation where I have a disconnect in between to parts of an application stack I'm building. The source is the application stack generating logs and metrics etc., and the destination is where I plan to consume the logs - processing them so they're usable in tools such as OpenSearch, Prometheus (potentually) and others, by means of receiving using OpenTelemetry.

The disconnect comes in the form of an inability to "stream" logs and metrics from source to destination. We can ship files, however - so I'm trying to find a setup using OpenTelemetry that gives me this, ideally:

1. Use a collector to receive logs, on OTLP/HTTP, and export them to a file.
2. Later move the files (not a part of this problem)
3. Use a collector, with a receiver, to pull in the logs, metrics and traces and feed them out to OpenSearch etc.

While I've seen that I can do 1, I'm not sure how feasible 3 is.. I don't know if the exporter I'm using (file) outputs in a format (JSON) that is somewhat lossless in the way that I'd expect - for the collector to read from and put these logs and metrics back into a form that the collector can somewhat seamlessly pass on to the different applications I mentioned. Ideally I'd want it to function as if there was no file exports, but just piped through straight to the relevant application.

I'll include my current collector configurations as I have them now.

Source collector:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed
  file/rotated_logs:
    path: /data/collector/otlp_logs
    rotation:
      max_megabytes: 10
      max_days: 14
      max_backups: 500
      localtime: false # Use UTC
    format: json
    flush_interval: 5s
  file/rotated_metrics:
    path: /data/collector/otlp_metrics
    rotation:
      max_megabytes: 10
      max_days: 14
      max_backups: 500
      localtime: false # Use UTC
    format: json
    flush_interval: 5s
  file/rotated_traces:
    path: /data/collector/otlp_traces
    rotation:
      max_megabytes: 10
      max_days: 14
      max_backups: 500
      localtime: false # Use UTC
    format: json
    flush_interval: 5s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: []
      exporters: [debug, file/rotated_logs]
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [debug, file/rotated_metrics]
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug, file/rotated_traces]

Destination collector:

receivers:
  filelog:
    include: [ /data/collector/otlp_logs* ]
    start_at: beginning
    operators:
      - type: json_parser

exporters:
  debug:
    verbosity: detailed
  opensearch/trace:
    dataset: ace-dev-trace
    namespace: ace
    http:
      endpoint: http://opensearch-data:9200
      tls:
        insecure: true
  opensearch/log:
    dataset: ace-dev-log
    namespace: ace
    http:
      endpoint: http://opensearch-data:9200
      tls:
        insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
    tls:
      insecure: true
    external_labels:
      data_source: otel
    resource_to_telemetry_conversion:
      enabled: true # Convert resource attributes to metric labels

processors:
  batch:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [debug, opensearch/log]
    # metrics:
    #   receivers: [otlp]
    #   processors: [batch]
    #   exporters: [debug, file/rotated_metrics]
    # traces:
    #   receivers: [otlp]
    #   processors: [batch]
    #   exporters: [debug, file/rotated_traces]

At the moment I'm able to "rehydrate" logs, using this method, so that they appear in OpenSearch. The parsing is obviously broken though, yielding deeply nested fields like so:

[trask]

(transferring to https://github.com/open-telemetry/opentelemetry-collector-contrib)

[perry-mitchell]

I got an answer on this from a very helpful Mastodon user, who linked the otlpjsonfile receiver.. after testing it I've found it works perfectly, pulling in all of my exported logs/metrics/traces that were output by the file exporter. This is what my rehydration side receivers look like now:

receivers:
  otlpjsonfile/logs:
    include:
      - /data/collector/otlp_logs*
  otlpjsonfile/metrics:
    include:
      - /data/collector/otlp_metrics*
  otlpjsonfile/traces:
    include:
      - /data/collector/otlp_traces*