Event missing the message key; dropping event

[adam-mrozik]

Hey,

We are trying to coordinate 3 vectors. Flow goes as follow:

(socket_source)Vector 1  -> (http_source) Vector2 --> HTTP source Vector3

Vector1 Source and sink look like this:

sinks:
...
http_out:
    auth:
      password: ${VECTOR_REMOTE_PASSWORD}
      strategy: basic
      user: username
    batch:
      max_bytes: 1049000
      timeout_secs: 5
    buffer:
      max_events: 10000
      type: memory
      when_full: drop_newest
    compression: gzip
    encoding:
      codec: text
    healthcheck: true
    inputs:
    - socket_in
    type: http
    uri: https://our_external_url_for_vector2/data
...
sources:
  socket_in:
    address: 0.0.0.0:8001
    max_length: 262144
    mode: tcp
    type: socket

Our Vector 2 configuration looks like this:

sources:
  http_in:
    address: 0.0.0.0:8001
    auth:
      password: password
      username:username
    encoding: json
    path: /data
    type: http
sinks:
 http_out:
    auth:
      password: ${PASSWORD}
      strategy: basic
      user: superuser
    batch:
      max_bytes: 1049000
      timeout_secs: 5
    buffer:
      max_events: 10000
      type: memory
      when_full: drop_newest
    compression: gzip
    encoding:
      codec: text
    healthcheck: true
    inputs:
    - http_in
    type: http
    uri: https://our_external_url_for_vector3/data

Unfortunately, between vector2 and vector3 we get an error: Event missing the message key; dropping event

And indeed, if I stdout Vector1 message, it has keys message, timestamp etc...
But, stdout of Vector2 message are simply keys that were previously inside message key. Do you know what might be the issue here?

We have some transforms of http_in source in Vector2, but used for other sinks. I am not sure it is the issue, because http_out uses http_in as input directly.

One suspicion I have is that Vector1 is Agent and Vector2 and Vector3 are Aggregators. May it be that aggregator is dropping this structure?

Any ideas would be greatly appreciated!

[jszwedko]

Hi @adam-mrozik !

I think you will want to set:

encoding:
  codec: json

On your http sinks to match the configuration you have on the http source (which is expecting JSON). Alternatively, you can configure the http source with:

encoding: text

Note that the text encoding just passes along the raw message without any of the additional metadata fields. That might be what you want in this case though, I'm not 100% sure. It depends on if you need the source metadata fields in the aggregators.