Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add _pipeline support to log event #23

Open
AlexKovalevich opened this issue Nov 20, 2016 · 1 comment
Open

Add _pipeline support to log event #23

AlexKovalevich opened this issue Nov 20, 2016 · 1 comment

Comments

@AlexKovalevich
Copy link

Add ability to use ES 5 ingest pipelines

_pipeline : "pipelineName" to the indexed data. Thais can be done either by making serializeEvents() or serializeIndexString() protected, or by providing additional configuration like:

			String pipeline = settings.getPipeline();
			if (pipeline != null) {
				gen.writeObjectField("_pipeline ", pipeline );
			}
@w39hh
Copy link

w39hh commented Jul 13, 2017

Although this is an old request, the following may still be of some use to someone.
We already make use of pipelines with the appender to convert strings to numbers, as we want some of the attributes in our log to be numeric.

We created a simple pipeline we called to_number. The processor converts certain fields we use in our logging (duration, count) from text to number.

{
  "processors": [
    {
      "convert": {
        "field": "duration",
        "type": "integer"
      }
    },
    {
      "convert": {
        "field": "count",
        "type": "integer"
      }
    }
  ]
}

After posting that pipeline to Elasticsearch, we configure the appender to use it as follows

   <appender name="ELASTIC" class="com.internetitem.logback.elasticsearch.ElasticsearchAppender">
      <url>${elastic.url}/_bulk/?pipeline=to_number</url>
      <index>nhcs-logback</index>
      <type>logback</type>
      <loggerName>es-logger</loggerName> <!-- optional -->
... etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants