Skip to content

Commit

Permalink
Automatically choose most correct codec for bytestream inputs.
Browse files Browse the repository at this point in the history
Now, if you try to use `plain` or `json` codecs with either `stdin` or
`tcp` inputs, it will automatically select `line` or `json_lines` codec
respectively.

Right now, many users try to use json codec with tcp input. This
*should* work, but due to technical limitations[1], it does not.
To compensate, we have always provided a json_lines codec to help
parse "streams of line-delimited json" which is what most folks
seem to be handling these days.

[1] I haven't found a JSON library capable of processing a byte stream.
Jackson has one, but it requires an input stream model which logstash
does not currently expose. Future work will make this possible.

Related: elastic#1391
  • Loading branch information
jordansissel committed May 20, 2014
1 parent ab9985c commit 4d64691
Show file tree
Hide file tree
Showing 3 changed files with 18 additions and 0 deletions.
16 changes: 16 additions & 0 deletions lib/logstash/inputs/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -118,4 +118,20 @@ def decorate(event)
event[field] = value
end
end

protected
def fix_streaming_codecs
require "logstash/codecs/plain"
require "logstash/codecs/line"
require "logstash/codecs/json"
require "logstash/codecs/json_lines"
case @codec
when LogStash::Codecs::Plain
@logger.info("Automatically switching from #{@codec.class.config_name} to line codec", :plugin => self.class.config_name)
@codec = LogStash::Codecs::Line.new
when LogStash::Codecs::JSON
@logger.info("Automatically switching from #{@codec.class.config_name} to json_lines codec", :plugin => self.class.config_name)
@codec = LogStash::Codecs::JSONLines.new
end
end
end # class LogStash::Inputs::Base
1 change: 1 addition & 0 deletions lib/logstash/inputs/stdin.rb
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ class LogStash::Inputs::Stdin < LogStash::Inputs::Base
public
def register
@host = Socket.gethostname
fix_streaming_codecs
end # def register

def run(queue)
Expand Down
1 change: 1 addition & 0 deletions lib/logstash/inputs/tcp.rb
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ def initialize(*args)

public
def register
fix_streaming_codecs
require "socket"
require "timeout"
require "openssl"
Expand Down

0 comments on commit 4d64691

Please sign in to comment.