Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use protobuf codec with http input #34

Open
PhilippHomann opened this issue Feb 13, 2019 · 3 comments
Open

Cannot use protobuf codec with http input #34

PhilippHomann opened this issue Feb 13, 2019 · 3 comments

Comments

@PhilippHomann
Copy link

When using http input combined with protobuf codec I receive following error:
[WARN ][logstash.codecs.protobuf ] Couldn't decode protobuf: #<ProtocolBuffers::DecodeError: too many bytes when decoding varint>.
with other proto
Couldn't decode protobuf: #<ProtocolBuffers::DecodeError: invalid message>

Tested with the very basic unicorn.proto from here, I receive error message above when using the protobuf codec with http input.
With tcp input everything works fine.

tcp.conf:

input {
	tcp {
		port => 5000
		codec => protobuf
		{
		    class_name => "Animal::Unicorn"
                    include_path => ["/unicorn.pb.pb.rb"]
		}
	}
}
output { stdout { } }

http.conf:

input {
	http {
		port => 5000
		codec => protobuf
		{
		    class_name => "Animal::Unicorn"
                    include_path => ["/unicorn.pb.pb.rb"]
		}
	}
}
output { stdout { } }

unicorn.pb.pb.rb:

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

module Animal
  # forward declarations
  class Unicorn < ::ProtocolBuffers::Message; end

  class Unicorn < ::ProtocolBuffers::Message
    set_fully_qualified_name "Animal.Unicorn"

    optional :string, :colour, 1
    optional :int32, :horn_length, 2
    optional :int64, :last_seen, 3
  end

end

unicorn.bin (base64):

Cg5yZWQtYmx1ZS13aGl0ZRD///////////8BGAA=

Input to logstash is done by:

curl -XPOST --data-binary @unicorn.bin http://127.0.0.1:5000
nc 127.0.0.1 5000 < unicorn.bin
@jvmlet
Copy link

jvmlet commented Apr 3, 2019

Don't you need to base64-decode first ?

@PhilippHomann
Copy link
Author

I base64 encoded the binary file to paste it here ;-)
Not send the base64 encoded data to logstash.

@jvmlet
Copy link

jvmlet commented Apr 3, 2019

How do you send then byte[] over http ? Charset ,Content-Type ?
Can you add the file output to your pipeline without codec and compare the binaries ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants