Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow beats/agent to override the creation of event.original in ecs_compatibilty mode #15638

Open
P1llus opened this issue Nov 30, 2023 · 0 comments

Comments

@P1llus
Copy link
Member

P1llus commented Nov 30, 2023

I would like for the agent/beat to be able to communicate to logstash that it should not create the event.original field when running in ecs_compatibility mode.
There is already a manual setting that can prevent this, using enrich.

input {
  elastic_agent {
    port => 5044
    enrich => none
  }
}

Instead of putting the requirement on the user side, I would like integration developers to provide some sort of flag, metadata etc, that is passed with the document that is sent from beats/agent to logstash, the exact name, value or location of the data could be up to the Logstash team.

With that, integration developers and teams can choose which scenarios it want to prohibit the creation of the field from the integration side instead.

Background:
Almost all our log based Agent integrations utilizes event.original in one way or another. After the automatic creation of event.original with the introduction of the ECS compatibility mode, there has been certain scenarios that causes integrations to fail ingestion due to event.original already existing when the data hits the Elasticsearch Ingest Pipeline.

We introduced some overhaul of most of our pipelines in our integrations to simply not populate event.original, and reuse the data already there, however there are still more niche usecases that creates conflict, for example this ended up so that both event.original and the message field is populated with the same data, causing issues later down the line in our data transformation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant