LogstashLayout plugin provides a Log4j 2.x layout with customizable and Logstash-friendly JSON formatting.

By default, LogstashLayout ships the official JSONEventLayoutV1 stated by log4j-jsonevent-layout Log4j 1.x plugin. Compared to JSONLayout included in Log4j 2.x and log4j-jsonevent-layout, LogstashLayout provides the following additional features:

• Additional fields can be added. (See template and templateUri parameters.)
• JSON schema structure can be customized. (See template and templateUri parameters.)
• Timestamp formatting can be customized. (See dateTimeFormatPattern and timeZoneId parameters.)

Usage

Add the log4j2-logstash-layout dependency to your POM file

<dependency>
    <groupId>com.vlkan.log4j2</groupId>
    <artifactId>log4j2-logstash-layout</artifactId>
    <version>${log4j2-logstash-layout.version}</version>
</dependency>

together with a valid log4j-core dependency:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j2.version}</version>
</dependency>

Below you can find a sample log4j2.xml snippet employing LogstashLayout.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
    <Appenders>
        <Console name="CONSOLE" target="SYSTEM_OUT">
            <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
                            templateUri="classpath:LogstashJsonEventLayoutV1.json"
                            prettyPrintEnabled="true"
                            stackTraceEnabled="true"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="CONSOLE"/>
        </Root>
    </Loggers>
</Configuration>

This generates an output as follows:

{
  "exception": {
    "exception_class": "java.lang.RuntimeException",
    "exception_message": "test",
    "stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
  },
  "line_number": 12,
  "class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
  "@version": 1,
  "source_host": "varlik",
  "message": "Hello, error!",
  "thread_name": "main",
  "@timestamp": "2017-05-25T19:56:23.370+02:00",
  "level": "ERROR",
  "file": "LogstashLayoutDemo.java",
  "method": "main",
  "logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}

LogstashLayout is configured with the following parameters:

Parameter Name	Type	Description
prettyPrintEnabled	boolean	enables pretty-printer (defaults to false)
locationInfoEnabled	boolean	includes the filename and line number in the output (defaults to false)
stackTraceEnabled	boolean	includes stack traces (defaults to false)
emptyPropertyExclusionEnabled	boolean	exclude empty and null properties (defaults to true)
dateTimeFormatPattern	String	timestamp formatter pattern (defaults to yyyy-MM-dd'T'HH:mm:ss.SSSZZZ)
timeZoneId	String	time zone id (defaults to TimeZone.getDefault().getID())
mdcKeyPattern	String	regex to filter MDC keys (does not apply to direct mdc:key access)
ndcPattern	String	regex to filter NDC items
template	String	inline JSON template for generating the output (has priority over templateUri)
templateUri	String	JSON template for generating the output (defaults to classpath:LogstashJsonEventLayoutV1.json)

templateUri denotes the URI pointing to the JSON template that will be used while formatting the log events. By default, LogstashLayout ships the JSONEventLayoutV1 in LogstashJsonEventLayoutV1.json within the classpath:

{
  "mdc": "${json:mdc}",
  "ndc": "${json:ndc}",
  "exception": {
    "exception_class": "${json:exceptionClassName}",
    "exception_message": "${json:exceptionMessage}",
    "stacktrace": "${json:exceptionStackTrace}"
  },
  "line_number": "${json:sourceLineNumber}",
  "class": "${json:sourceClassName}",
  "@version": 1,
  "source_host": "${hostName}",
  "message": "${json:message}",
  "thread_name": "${json:threadName}",
  "@timestamp": "${json:timestamp}",
  "level": "${json:level}",
  "file": "${json:sourceFileName}",
  "method": "${json:sourceMethodName}",
  "logger_name": "${json:loggerName}"
}

In case of need, you can create your own templates with a structure tailored to your needs. That is, you can add new fields, remove or rename existing ones, change the structure, etc. Please note that templateUri parameter only supports file and classpath URI schemes.

Below is the list of known template variables that will be replaced while rendering the JSON output.

Variable Name	Description
exceptionClassName	logEvent.getThrown().getClass().getCanonicalName()
exceptionMessage	logEvent.getThrown().getMessage()
exceptionStackTrace	logEvent.getThrown().printStackTrace() (inactive when stackTraceEnabled=false)
exceptionRootCauseClassName	the innermost exceptionClassName in causal chain
exceptionRootCauseMessage	the innermost exceptionMessage in causal chain
exceptionRootCauseStackTrace	the innermost exceptionStackTrace in causal chain
level	logEvent.getLevel()
loggerName	logEvent.getLoggerName()
mdc	Mapped Diagnostic Context Map<String, String> returned by logEvent.getContextData()
mdc:key	Mapped Diagnostic Context String associated with key (mdcKeyPattern is discarded)
message	logEvent.getMessage()
ndc	Nested Diagnostic Context String[] returned by logEvent.getContextStack()
sourceClassName	logEvent.getSource().getClassName()
sourceFileName	logEvent.getSource().getFileName() (inactive when locationInfoEnabled=false)
sourceLineNumber	logEvent.getSource().getLineNumber() (inactive when locationInfoEnabled=false)
sourceMethodName	logEvent.getSource().getMethodName()
threadName	logEvent.getThreadName()
timestamp	logEvent.getTimeMillis() formatted using dateTimeFormatPattern and timeZoneId

JSON field lookups are performed using the ${json:<variable-name>} scheme where <variable-name> is defined as <resolver-name>[:<resolver-key>]. Characters following colon (:) are treated as the resolver-key of which as of now only supported by mdc resolver.

Log4j 2.x Lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in templates too. Though note that while ${json:...} template variables is expected to occupy an entire field, that is, "level": "${json:level}", a lookup can be mixed within a regular text: "myCustomField": "Hello, ${env:USER}!".

See layout-demo directory for a sample application using the LogstashLayout.

Fat JAR

Project also contains a log4j2-logstash-layout-fatjar artifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception of log4j-core, that you need to include separately.

This might come handy if you want to use this plugin along with already compiled applications, e.g., Elasticsearch 5.x, which requires Log4j 2.x.

Appender Support

log4j2-logstash-layout is all about providing a highly customizable JSON schema for your logs. Though this does not necessarily mean that all of its features are expected to be supported by every appender in the market. For instance, while prettyPrintEnabled=true works fine with log4j2-redis-appender, it should be turned off for Logstash's log4j-json file input type. (See Pretty printing in Logstash issue.) Make sure you configure log4j2-logstash-layout properly in a way that is aligned with your appender of preference.

Contributors

• Yaroslav Skopets

License

Copyright © 2017-2018 Volkan Yazıcı

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.