Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

config max_message_size 10M in v0.10.0, heka didn't send data to ES when input message about 1.4M #1991

Open
chwma opened this issue Aug 7, 2017 · 1 comment

Comments

@chwma
Copy link

chwma commented Aug 7, 2017

RT
need any other config items ?

follow is my part config
[hekad]
maxprocs = 16
base_dir = "/export/home/hekad"
max_message_size = 10485760

#attack log default
[nginx_udp_551]
type = "UdpInput"
address = "172.18.182.162:551"
decoder = "JsonDecoder"
send_decode_failures = true
log_decode_failures = true

[JsonDecoder]
type = "SandboxDecoder"
filename = "lua_decoders/json.lua"

[JsonDecoder.config]
payload_keep = false
map_fields = true
Timestamp = "time_stamp"
Type = "log_type"
#type = "ngx_log"

@sampointer
Copy link

If this is anything like fluentd, which uses msgpack, then small messages can blow ElasticSearch request limits if they pack very efficiently. In fluentd the message size limits are enforced after serialization to the transport format, which when unrolled at the other end can balloon to into huge HTTP POSTs, which are rejected. This is especially prevalent with JSON and logging information, which msgpack compresses very well.

The fluentd ecosystem is a tire fire, so don't be tempted to jump ship in that direction 😃

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants