CHANGELOG

1.1.2 (???, 2012)
 ## Overview of this release:

 ## IMPORTANT CHANGES FOR UPGRADES FROM 1.1.1

 ## general
 - Stopped using 'Gemfile' for dependencies, the logstash.gemspec has returned.
   (Patch by Grant Rogers)
 - New 'logstash-event.gemspec' for generating logstash events in your own
   ruby programs (Patch by Garry Shutler)
 - Wildcard config files are now sorted properly (agent -f
   /etc/logstash/*.conf)
 - The old '-vvv' setting ruby's internal $DEBUG is now gone. It was causing
   too much confusion for users due to noise.
 - Improved 'logstash event' creation speed by 3.5x

 ## inputs
 - bugfix: redis: [LOGSTASH-526] fix bug with password passing
 - new input: lumberjack: for use with the lumberjack log shipper 
   (https://github.com/jordansissel/lumberjack)
 - file input: sincedb path is now automatically generated if not specified.
   This helps work around a problem where two file inputs don't specify a
   sincedb_path would clobber eachother (LOGSTASH-554)
 - file input: no longer crashes if HOME is not set in env (LOGSTASH-458)

 ## filters
 - new: kv: useful for parsing log formats taht use 'foo=bar baz=fizz' and
   similar key-value-like things.
 - improvement: zeromq: an empty reply is now considered as a 'cancel this
   event' operation (LOGSTASH-574)
 - bugfix: mutate: fix bug in uppercase and lowercase feature that would
   prevent it from actually doing the uppercasing/lowercasing.
 - improvement: mutate: do the 'remove' action last (LOGSTASH-543)
 - feature: grok: new 'singles' config option which, when true, stores
   single-value fields simply as a single value rather than as an array, like
   [value]. (LOGSTASH-185)
 - grok patterns: the URIPARAM pattern now includes pipe '|' as a valid
   character. (Patch by Chris Mague)
 - grok patterns: improve haproxy log patterns (Patch by Kevin Nuckolls)
 - grok patterns: include 'FATAL' as a valid LOGLEVEL match 
   (patch by Corry Haines)
 - grok patterns: 'ZONE' is no longer captured by name in the HTTPDATE pattern
 - new: alter: adds some conditional field modification as well as a 
   'coalesce' feature which sets the value of a field to the first non-null
   value given in a list. (Patch by Francesco Salbaroli)
 - improvement: date: add TAI64N support

 ## outputs
 - new: exec: run arbitrary commands based on an event.
 - bugfix: redis: [LOGSTASH-526] fix bug with password passing
 - improvement: redis: [LOGSTASH-573] retry on failure even in batch-mode. This
   also fixes a prior bug where an exception in batch mode would cause logstash
   to crash. (Patch by Alex Dean)
 - improvement: riemann: metric and ttl values in riemann_event now support
   sprintf %{foo} values. (pull #174)
 - improvement: stdout: new 'dots' debug_format value emits one dot per event
   useful for tracking event rates.
 - gelf output: correct severity level mappings (patch by Jason Koppe)

1.1.1 (July 14, 2012)
 ## Overview of this release:
  * New input plugins: generator, heroku, pipe, ganglia, irc
  * New output plugins: juggernaut, metricscatcher, nagios_ncsa, pipe,
                        opentsdb, pagerduty, irc
  * New filter plugins: zeromq, environment, xml, csv, syslog_pri
  * Fixes for gelf output
  * Support for more than 1 filter worker (agent argument "-w")

 ## IMPORTANT CHANGES FOR UPGRADES FROM 1.1.0
  - zeromq input and output rewritten
      The previous zeromq support was an MVP. It has now been rewritten into
      something more flexible. The configuration options have changed entirely.
      While this is still listed as `experimental`, we don't predict any more
      configuration syntax changes. The next release will bump this to beta.
  - unix_timestamp
      Previously, several plugins did not work as expected on MRI due to the
      usage of the JRuby-only Jodatime library. We now have a contributed fix
      for a slower parser on MRI/CRuby!
  - elasticsearch version is now 0.19.8
      This means your elasticsearch cluster must be running 0.19.x for
      compatibility reasons.
  - grok pattern %{POSINT} used to match '0' -- now it does not. If you want
    to match non-negative integers, there is now a %{NONNEGINT} pattern.
  - bug in file input fixed that led to an extra leading slash in @source_path.
    Previously, file input would have @source = 'file://host//var/log/foo' and
    @source_path = '//var/log/foo'; now @source = 'file://host/var/log/foo'
    and @source_path = '/var/log/foo'. [LOGSTASH-501]
  - file input now rejects relative paths. [LOGSTASH-503]
  - event sprintf can now look inside structured field data. %{foo.bar} will
    look in the event field "foo" (if it is a hash) for "bar".  To preserve
    compatibility, we first look for a top-level key that matches exactly
    (so %{foo.bar} will first look for a field named "foo.bar", then look for
    "bar" under "foo").

  ## general
  - NOTE: gemspec removed; deploying logstash as a gem hasn't been supported
    for a while.
  - feature: logstash sub-commands "irb" and "pry" for an interactive debug
    console, useful to debug jruby when running from the monolithic jar
  - misc: newer cabin gem for logging
  - misc: initial support for reporting internal metrics (currently outputs
    to INFO log; eventually will be an internal event type)
  - misc: added a "thread watchdog" to detect hanging filter workers, and
    crash logstash w/an informational message
  - misc: jar is built with jruby 1.6.7.2
  - misc: better shutdown behavior when there are no inputs/plugins running
  - feature: logstash web now uses relative URLs; useful if you want to
    reverseproxy with a path other than "/"

  ## inputs
  - bugfix: stdin: exit plugin gracefully on EOF
  - feature: [LOGSTASH-410] - inputs can now be duplicated with the
    'threads' parameter (where supported)
  - bugfix: [LOGSTASH-490] - include cacert.pem in jar for twitter input
  - feature: [LOGSTASH-139] - support for IRC

  ## filters
  - feature: all filters support 'remove_tag' (remove tags on success)
  - feature: all filters support 'exclude_tags' (inverse of 'tags')
  - bugfix: [LOGSTASH-300] - bump grok pattern replace limit to 1000,
    fixes "deep recursion pattern compilation" problems
  - bugfix: [LOGSTASH-375] - fix bug in grep: don't drop when field is nil
    and negate is true
  - bugfix: [LOGSTASH-386] - fix some grok patterns for haproxy
  - bugfix: [LOGSTASH-446] - fix grok %{QUOTEDSTRING} pattern, should fix
    some grok filter hangs
  - bugfix: some enhancements to grok pattern %{COMBINEDAPACHELOG}
  - bugfix: grok: %{URIPATH} and %{URIPARAM} enhancements
  - feature: grok: add %{UUID} pattern
  - bugfix: grok: better error message when expanding unknown %{pattern}
  - feature: mutate: now supports a 'gsub' operation for applying a regexp
    substitution on event fields

  ## outputs
  - bugfix: [LOGSTASH-351] - fix file input on windows
  - feature: [LOGSTASH-356] - make file output flush intervals configurable
  - feature: [LOGSTASH-392] - add 'field' attribute to restrict which fields
    get sent to an output
  - feature: [LOGSTASH-374] - add gzip support to file output
  - bugfix: elastic search river now respects exchange_type and queue_name
  - bugfix: ganglia plugin now respects metric_type
  - bugfix: GELF output facility fixes; now defaults to 'logstash-gelf'
  - feature: [LOGSTASH-139] - support for IRC
  - bugfix: es_river: check river status after creation to verify status
  - feature: es: allow setting node_name
  - feature: redis: output batching for list mode

1.1.0.1 (January 30, 2012)
  ## Overview of this release:
    * date filter bugfix: [LOGSTASH-438] - update joda-time to properly
      handle leap days

1.1.0 (January 30, 2012)
  ## Overview of this release:
    * New input plugins: zeromq, gelf
    * New filter plugins: mutate, dns, json
    * New output plugins: zeromq, file
    * The logstash agent now runs also in MRI 1.9.2 and above

    This is a large release due to the longevity of the 1.1.0 betas.
    We don't like long releases and will try to avoid this in the future.

  ## IMPORTANT CHANGES FOR UPGRADES FROM 1.0.x
    - grok filter: named_captures_only now defaults to true
        This means simple patterns %{NUMBER} without any other name will 
        now not be included in the field set. You can revert to the old
        behavior by setting 'named_captures_only => false' in your grok
        filter config.
    - grok filter: now uses Ruby's regular expression engine
        The previous engine was PCRE. It is now Oniguruma (Ruby). Their
        syntaxes are quite similar, but it is something to be aware of.
    - elasticsearch library upgraded to 0.18.7
        This means you will need to upgrade your elasticsearch servers,
        if any, to the this version: 0.18.7
    - AMQP parameters and usage have changed for the better. You might
      find that your old (1.0.x) AMQP logstash configs do not work.
      If so, please consult the documentation for that plugin to find
      the new names of the parameters.

  ## general
  - feature: [LOGSTASH-158] - MRI-1.9 compatible (except for some
    plugins/functions which will throw a compatibility exception) This means
    you can use most of the logstash agent under standard ruby.
  - feature: [LOGSTASH-118] - logstash version output (--version or -V for
    agent)
  - feature: all plugins now have a 'plugin status' indicating the expectation
    of stability, successful deployment, and rate of code change. If you
    use an unstable plugin, you will now see a warning message on startup.
  - bugfix: AMQP overhaul (input & output), please see docs for updated
    config parameters.
  - bugfix: [LOGSTASH-162,177,196] make sure plugin-contained global actions
    happen serially across all plugins (with a mutex)
  - bugfix: [LOGSTASH-286] - logstash agent should not truncate logfile on
    startup
  - misc: [LOGSTASH-160] - now use gnu make instead of rake.
  - misc: now using cabin library for all internal logging
  - test: use minitest
  - upgrade: now using jruby in 1.9 mode

  ## inputs
  - feature: zeromq input. Requires you have libzmq installed on your system.
  - feature, bugfix: [LOGSTASH-40,65,234,296]: much smarter file watching for
    file inputs. now supports globs, keeps state between runs, can handle
    truncate, log rotation, etc. no more inotify is required, either (file
    input now works on all platforms)
  - feature: [LOGSTASH-172,201] - syslog input accepts ISO8601 timestamps
  - feature: [LOGSTASH-159] - TCP input lets you configure what identifies
    an input stream to the multiline filter (unique per host, or connection)
  - feature: [LOGSTASH-168] - add new GELF input plugin
  - bugfix: [LOGSTASH-8,233] - fix stomp input
  - bugfix: [LOGSTASH-136,142] - file input should behave better with log rotations
  - bugfix: [LOGSTASH-249] - Input syslog force facility type to be an integer
  - bugfix: [LOGSTASH-317] - fix file input not to crash when a file
    is unreadable

  ## filters
  - feature: [LOGSTASH-66,150]: libgrok re-written in pure ruby (no more
    FFI / external libgrok.so dependency!)
  - feature: [LOGSTASH-292,316] - Filters should run on all events if no condition
    is applied (type, etc).
  - feature: [LOGSTASH-292,316] - Filters can now act on specific tags (or sets
    of tags).
  - bugfix: [LOGSTASH-285] - for grok, add 'keep_empty_captures' setting to
    allow dropping of empty captures. This is true by default.
  - feature: [LOGSTASH-219] - support parsing unix epoch times
  - feature: [LOGSTASH-207] - new filter to parse a field as json merging it
    into the event.
  - feature: [LOGSTASH-267,254] - add DNS filter for doing forward or
    reverse DNS on an event field
  - feature: [LOGSTASH-57] - add mutate filter to help with manipulating
    event field content and type

  ## outputs
  - feature: zeromq output. Requires you have libzmq installed on your system.
  - feature: new file output plugin
  - bugfix: [LOGSTASH-307] embedded elasticsearch now acts as a full ES server;
    previously embedded was only accessible from within the logstash process.
  - bugfix: [LOGSTASH-302] - logstash's log level (-v, -vv flags) now control
    the log output from the elasticsearch client via log4j.
  - bugfix: many gelf output enhancements and bugfixes
  - feature: [LOGSTASH-281] - add https support to loggly output
  - bugfix: [LOGSTASH-167] - limit number of in-flight requests to the
    elasticsearch node to avoid creating too many threads (one thread per
    pending write request)
  - bugfix: [LOGSTASH-181] - output/statsd: set sender properly
  - bugfix: [LOGSTASH-173] - GELF output can throw an exception during gelf notify
  - bugfix: [LOGSTASH-182] - grep filter should act on all events if no type is
    specified.
  - bugfix: [LOGSTASH-309] - file output can now write to named pipes (fifo)


1.0.17 (Aug 12, 2011)
  - Bugs fixed
    - [LOGSTASH-147] - grok filter incorrectly adding fields when a match failed
    - [LOGSTASH-151] - Fix bug in routing keys on AMQP 
    - [LOGSTASH-156] - amqp issue with 1.0.16?

  - Improvement
    - [LOGSTASH-148] - AMQP input should allow queue name to be specified separately from exchange name
    - [LOGSTASH-157] - Plugin doc generator should make regexp config names more readable

  - New Feature
    - [LOGSTASH-153] - syslog input: make timestamp an optional field
    - [LOGSTASH-154] - Make error reporting show up in the web UI

1.0.16 (Aug 18, 2011)
  - Fix elasticsearch client problem with 1.0.15 - jruby-elasticsearch gem
    version required is now 0.0.10 (to work with elasticsearch 0.17.6)

1.0.15 (Aug 18, 2011)
  - IMPORTANT: Upgraded to ElasticSearch 0.17.6 - this brings a number of bug
    fixes including an OOM error caused during high index rates in some
    conditions.
    NOTE: You *must* use same main version of elasticsearch as logstash does,
    so if you are still using elasticsearch server 0.16.x - you need to upgrade
    your server before the elasticsearch output will work. If you are using
    the 'embedded' elasticsearch feature of logstash, you do not need to make
    any changes.
  - feature: tcp input and output plugins can now operate in either client
    (connect) or server (listen) modes.
  - feature: new output plugin "statsd" which lets you increment or record
    timings from your logs to a statsd agent
  - feature: new redis 'pattern_channel' input support for PSUBSCRIBE
  - feature: new output plugin "graphite" for taking metrics from events and
    shipping them off to your graphite/carbon server.
  - feature: new output plugin "ganglia" for shipping metrics to ganglia
    gmond server.
  - feature: new output plugin "xmpp" for shipping events over jabber/xmpp
  - feature: new input plugin "xmpp" for receiving events over jabber/xmpp
  - feature: amqp input now supports routing keys.
    https://logstash.jira.com/browse/LOGSTASH-122
  - feature: amqp output now supports setting routing key dynamically.
    https://logstash.jira.com/browse/LOGSTASH-122
  - feature: amqp input/output both now support SSL.
    https://logstash.jira.com/browse/LOGSTASH-131
  - feature: new input plugin "exec" for taking events from executed commands
    like shell scripts or other tools.
  - feature: new filter plugin "split" for splitting one event into multiple.
    It was written primarily for the new "exec" input to allow you to split
    the output of a single command run by line into multiple events.
  - misc: upgraded jar releases to use JRuby 1.6.3
  - bugfix: syslog input shouldn't crash anymore on weird network behaviors
    like portscanning, etc.
    https://logstash.jira.com/browse/LOGSTASH-130

1.0.14 (Jul 1, 2011)
  - feature: new output plugin "loggly" which lets you ship logs to loggly.com
  - feature: new output plugin "zabbix" - similar to the nagios output, but
    works with the Zabbix monitoring system. Contributed by Johan at
    Mach Technology.
  - feature: New agent '-e' flag which lets you specify a config in a string.
    If you specify no 'input' plugins, default is stdin { type => stdin }
    If you specify no 'output' plugins, default is stdout { debug => true }
    This is intended to be used for hacking with or debugging filters, but
    you can specify an entire config here if you choose.
  - feature: Agent '-f' flag now supports directories and globs. If you specify
    a directory, all files in that directory will be loaded as a single config.
    If you specify a glob, all files matching that glob will be loaded as a
    single config.
  - feature: gelf output now allows you to override the 'sender'. This defaults
    to the source host originating the event, but can be set to anything now.
    It supports dynamic values, so you can use fields from your event as the
    sender. Contributed by John Vincent
    Issue: https://github.com/logstash/logstash/pull/30
  - feature: added new feature to libgrok that allows you to define patterns
    in-line, like "%{FOO=\d+}" defines 'FOO' match \d+ and captures as such.
    To use this new feature, you must upgrade libgrok to at least 1.20110630
    Issue: https://logstash.jira.com/browse/LOGSTASH-94
  - feature: grok filter now supports 'break_on_match' defaulting to true
    (this was the original behavior). If you set it to false, it will attempt
    to match all patterns and create new fields as normal. If left default
    (true), it will break after the first successful match.
  - feature: grok filter now supports parsing any field. You can do either of
    these: grok { match => [ "fieldname", "pattern" ] }
    or this: grok { fieldname => "pattern" }
    The older 'pattern' attribute still means the same thing, and is equivalent
    to this: grok { match => [ "@message", "pattern" ] }
    Issue: https://logstash.jira.com/browse/LOGSTASH-101
  - feature: elasticsearch - when embedded is true, you can now set the
    'embedded_http_port' to configure which port the embedded elasticsearch
    server listens on. This is only valid for the embedded elasticsearch
    configuration. https://logstash.jira.com/browse/LOGSTASH-117
  - bugfix: amqp input now reconnects properly when the amqp broker restarts.
  - bugfix: Fix bug in gelf output when a fields were not arrays but numbers.
    Issue: https://logstash.jira.com/browse/LOGSTASH-113
  - bugfix: Fix a bug in syslog udp input due to misfeatures in Ruby's URI 
    class. https://logstash.jira.com/browse/LOGSTASH-115
  - misc: jquery and jquery ui now ship with logstash; previously they were
    loaded externally
  - testing: fixed some bugs in the elasticsearch test itself, all green now.
  - testing: fixed logstash-test to now run properly

1.0.12 (Jun 9, 2011)
  - misc: clean up some excess debugging output 
  - feature: for tcp input, allow 'data_timeout => -1' to mean "never time out"

1.0.11 (Jun 9, 2011)
  - deprecated: The redis 'name' and 'queue' options for both input and output
    are now deprecated. They will be removed in a future version.
  - feature: The redis input and output now supports both lists and channels.
  - feature: Refactor runner to allow you to run multiple things in a single
    process.  You can end each instance with '--' flag. For example, to run one
    agent and one web instance:
      % java -jar logstash-blah.jar agent -f myconfig -- web 
  - feature: Add 'embedded' option to the elasticsearch output:
      elasticsearch { embedded => true }
    Default is false. If true, logstash will run an elasticsearch server
    in the same process as logstash. This is really useful if you are just
    starting out or only need one one elasticsearch server.
  - feature: Added a logstash web backend feature for elasticsearch that tells
    logstash to use the 'local' (in process) elasticsearch:
      --backend elasticsearch:///?local
  - feature: Added 'named_captures_only' option to grok filter. This will have
    logstash only keep the captures you give names to - for example %{NUMBER}
    won't be kept, but %{NUMBER:bytes} will be.
  - feature: Add 'bind_host' option to elasticsearch output. This lets you choose the
    address ElasticSearch client uses to bind to - useful if you have a
    multihomed server.
  - feature: The mongodb output now supports authentication
  - bugfix: Fix bug in GELF output that caused the gelf short_message to be set as an
    array if it came from a grok value. The short_message field should only 
    now be a string properly.
  - bugfix: Fix bug in grep filter that would drop/cancel events if you had
    more than one event type flowing through filters and didn't have a grep
    filter defined for each type.
  - misc: Updated gem dependencies (tests still pass)
  - misc: With the above two points, you can now run a single logstash process
    that includes elasticsearch server, logstash agent, and logstash web.

1.0.10 (May 23, 2011)
  - Fix tcp input bug (LOGSTASH-88) that would drop connections.
  - Grok patterns_dir (filter config) and --grok-patterns-dir (cmdline opt)
    are now working.
  - GELF output now properly sends extra fields from the log event (prefixed
    with a "_") and sets timestamp to seconds-since-epoch (millisecond
    precision and time zone information is lost, but this is the format GELF
    asks for).
  - Inputs support specifying the format of input data (see "format" and
    "message_format" input config parameters).
  - Grok filter no longer incorrectly tags _grokparsefailure when more than
    one grok filter is enabled (for multiple types) or when an event has
    no grok configuration for it's type.
  - Fix bug where an invalid HTTP Referer: would break grok parsing of the
    log line (used to expect %{URI}). Since Referer: is not sanitized in
    the HTTP layer, we cannot assume it will be a well formed %{URI}.

1.0.9 (May 18, 2011)
  - Fix crash bug caused by refactoring that left 'break' calls in code
    that no longer used loops.

1.0.8 (May 17, 2011)
  - Remove beanstalk support because the library (beanstalk-client) is GPL3. I
    am not a lawyer, but I'm not waiting around to have someone complain about
    license incompatibilities.
  - fix bug in jar build

1.0.7 (May 16, 2011)
  - logstash 'web' now allows you to specify the elasticsearch clustername;
    --backend elasticsearch://[host[:port]]/[clustername]
  - GELF output now supports dynamic strings for level and facility
    https://logstash.jira.com/browse/LOGSTASH-83
  - 'amqp' output supports persistent messages over AMQP, now. Tunable.
    https://logstash.jira.com/browse/LOGSTASH-81
  - Redis input and output are now supported. (Contributed by dokipen)
  - Add shutdown processing. Shutdown starts when all inputs finish (like
    stdin) The sequence progresses using the same pipeline as the
    inputs/filters/outputs, so all in-flight events should finish getting
    processed before the final shutdown event makes it's way to the outputs.  
  - Add retries to unhandled input exceptions (LOGSTASH-84)

1.0.6 (May 11, 2011)
  * Remove 'sigar' from monolithic jar packaging. This removes a boatload of
    unnecessary warning messages on startup whenever you use elasticsearch
    output or logstash-web.
    Issue: https://logstash.jira.com/browse/LOGSTASH-79

1.0.5 (May 10, 2011)
  * fix queues when durable is set to true

1.0.4 (May 9, 2011)
  * Fix bugs in syslog input

1.0.2 (May 8, 2011)
  * Fix default-value handling for configs when the validation type is
    'password'

1.0.1 (May 7, 2011)
  * Fix password auth for amqp and stomp (Reported by Luke Macken)
  * Fix default elasticsearch target for logstash-web (Reported by Donald Gordon)

1.0.0 (May 6, 2011)
  * First major release.