You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like for the agent/beat to be able to communicate to logstash that it should not create the event.original field when running in ecs_compatibility mode.
There is already a manual setting that can prevent this, using enrich.
Instead of putting the requirement on the user side, I would like integration developers to provide some sort of flag, metadata etc, that is passed with the document that is sent from beats/agent to logstash, the exact name, value or location of the data could be up to the Logstash team.
With that, integration developers and teams can choose which scenarios it want to prohibit the creation of the field from the integration side instead.
Background:
Almost all our log based Agent integrations utilizes event.original in one way or another. After the automatic creation of event.original with the introduction of the ECS compatibility mode, there has been certain scenarios that causes integrations to fail ingestion due to event.original already existing when the data hits the Elasticsearch Ingest Pipeline.
We introduced some overhaul of most of our pipelines in our integrations to simply not populate event.original, and reuse the data already there, however there are still more niche usecases that creates conflict, for example this ended up so that both event.original and the message field is populated with the same data, causing issues later down the line in our data transformation.
The text was updated successfully, but these errors were encountered:
I would like for the agent/beat to be able to communicate to logstash that it should not create the event.original field when running in ecs_compatibility mode.
There is already a manual setting that can prevent this, using
enrich
.Instead of putting the requirement on the user side, I would like integration developers to provide some sort of flag, metadata etc, that is passed with the document that is sent from beats/agent to logstash, the exact name, value or location of the data could be up to the Logstash team.
With that, integration developers and teams can choose which scenarios it want to prohibit the creation of the field from the integration side instead.
Background:
Almost all our log based Agent integrations utilizes event.original in one way or another. After the automatic creation of event.original with the introduction of the ECS compatibility mode, there has been certain scenarios that causes integrations to fail ingestion due to event.original already existing when the data hits the Elasticsearch Ingest Pipeline.
We introduced some overhaul of most of our pipelines in our integrations to simply not populate event.original, and reuse the data already there, however there are still more niche usecases that creates conflict, for example this ended up so that both event.original and the message field is populated with the same data, causing issues later down the line in our data transformation.
The text was updated successfully, but these errors were encountered: