On top of the Core component that sends logs to Loki this project gives you Logback appender.
-
Grab the no-dependency version of the appender:
<dependency> <groupId>pl.tkowalcz.tjahzi</groupId> <artifactId>logback-appender-nodep</artifactId> <version>0.1.0</version> </dependency>
If you already use a compatible version of Netty in your project then to reduce the size of your dependencies include regular appender distribution:
<dependency> <groupId>pl.tkowalcz.tjahzi</groupId> <artifactId>logback-appender</artifactId> <version>0.1.0</version> </dependency>
-
Include minimal appender configuration in your existing logback configuration:
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<efficientLayout>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</efficientLayout>
<label>
<name>server</name>
<value>${HOSTNAME}</value>
</label>
</appender>- Reference the appender from inside one of your logger definitions:
<root level="debug">
<appender-ref ref="Loki"/>
</root>Tjahzi sends POST requests to /loki/api/v1/push HTTP endpoint. Specifying
e.g. <host>loki.mydomain.com</host><port>3100</port>
will configure the appender to call to URL: http://loki.mydomain.com:3100/loki/api/v1/push.
Tjahzi provides its own pattern layout class ch.qos.logback.core.pattern.EfficientPatternLayout. It is based on
existing PatternLayout from Logback and uses exact same code for formatting events. The key difference is that it
swaps out some parts of the implementation and introduces reusable thread local buffers for constructing log line and
allocation free string to bytes encoder.
Tjahzi appender supports all kinds of layouts. To enable EfficientPatternLayout us following configuration:
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
...
<efficientLayout>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</efficientLayout>
</appender>This example sets up a root logger with Loki appender. The appender definition must point to
class pl.tkowalcz.tjahzi.logback.LokiAppender.
<configuration>
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<efficientLayout>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</efficientLayout>
<header>
<name>X-Org-Id</name>
<value>Codewise</value>
</header>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
<logLevelLabel>
log_level
</logLevelLabel>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>Connection is configured by providing parameters like host or port explicitly in dedicated tags or by using a URL that has them all "inline". First we will show how the individual parameters work. At a minimum Tjahzi needs host and port configuration to connect to Loki, e.g.:
<host>example.com</host>
<port>3100</port>If port is equal to 443 then SSL will be used. You can also configure SSL manually:
<host>example.com</host>
<port>3100</port>
<useSSL>true</useSSL>You can also override the default endpoint to which Tjahzi sends data. This can be useful if Loki is behind reverse proxy and additional path mapping is used:
<host>example.com</host>
<port>3100</port>
<logEndpoint>/monitoring/loki/api/v1/push</logEndpoint>All these parameters can be configured in one place using a URL:
<url>https://example.com:56654/monitoring/loki/api/v1/push</url>Note that all previously mentioned tags (host, port, useSSL, logEndpoint) cannot be used when using URL.
URL consists of four parts: protocol, host, port, path. Some of them may be omitted and there are defaults that depend on contents of other parts of the URL. This table has a rundown of all viable configurations:
| Section | Default | Comment |
|---|---|---|
| Protocol | None (must be provided) | Supported protocols are http and https. Https is equivalent to setting useUSSL |
| Host | None (must be provided) | |
| Port | 80 for http, 443 for https | You can use any port and SSL will still be used if protocol is set to https |
| Path | '/loki/api/v1/push' |
Some examples of correct URLs:
<url>http://example.com</url>
<url>https://example.com:56654</url>
<url>http://example.com/monitoring/loki/api/v1/push</url>
<url>https://example.com:3100/monitoring/foo/bar</url>Contents of the properties are automatically interpolated by Logback ( see here). All environment, system etc. variable references will be replaced by their values during initialization of the appender.
MDC is supported via mdcLogLabel tag. It will dynamically extract MDC value associated with its content and will turn
it into a label.
Click to expand example
<configuration debug="true">
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
<host>${loki.host}</host>
<port>${loki.port}</port>
<efficientLayout>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</efficientLayout>
<label>
<name>server</name>
<value>127.0.0.1</value>
</label>
<!-- MDC -->
<mdcLogLabel>
trace_id
</mdcLogLabel>
<!-- MDC -->
<mdcLogLabel>
span_id
</mdcLogLabel>
</appender>
<root level="debug">
<appender-ref ref="Loki"/>
</root>
</configuration>You can include logger name and logging thread name as labels by using dedicated configuration tags. It will dynamically extract these values and will turn them into a label.
<appender name="Loki" class="pl.tkowalcz.tjahzi.logback.LokiAppender">
...
<loggerNameLabel>
logger_name
</loggerNameLabel>
<threadNameLabel>
thread_name
</threadNameLabel>
</appender>Let's go through the example config above and analyze configuration options (Note: Tags are case-insensitive).
| Tag | Description |
|---|---|
| Host | Network host address of Loki instance. Either IP address or host name. It will be passed to Netty and end up being resolved by call to InetSocketAddress.createUnresolved. |
| Port | Self-explanatory :) |
| Encoder | You need to specify pattern for the layout. See section below |
Note: Instead of 'host' and 'port' you can specify URL. See this section.
You can define layout and pattern in a traditional Logback way:
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>or use Tjahzi optimized efficient, low allocation encoder:
<efficientLayout>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</efficientLayout>| Tag | Default value | Description |
|---|---|---|
| Url | - | Configure connection in one place instead of using host, port etc. See this section. |
| UseSSL | true if port == '443' | Enable secure (HTTPS) communication regardless of configured port number. |
| LogEndpoint | /loki/api/v1/push |
Overrides the default endpoint to which Tjahzi sends data. This can be useful if Loki is behind reverse proxy and additional path mapping is used. |
| Header | - | This tag can be used multiple times to specify additional headers that are passed to Loki instance. One example is to pass a X-Scope-OrgID header when running Loki in multi-tenant mode. |
| Label | - | Specify additional labels attached to each log line sent via this appender instance. See also note about label naming. |
| LogLevelLabel | - | If defined then log level label of configured name will be added to each line sent to Loki. It will contain Logback log level e.g. INFO, WARN etc. See also note about label naming. |
| BufferSizeMegabytes | 32 MB | Size of the log buffer. Must be power of two between 1MB and 1GB. See log buffer sizing for more explanations. |
| MaxRetries | 3 | Maximum number of retries to perform when delivering log message to Loki. Log buffer data is delivered in order, one batch after the other, so too much retries will block delivery of subsequent log batches (on the other hand if we need to retry many times then next batches will probably fail too). |
| ConnectTimeoutMillis | 5s | This configures socket connect timeout when connecting to Loki. After unsuccessful connection attempt it will continue to retry indefinitely employing exponential backoff (initial backoff = 250ms, maximum backoff = 30s, multiplier = 3). |
| ReadTimeoutMillis | 60s | Sets socket read timeout on Loki connection. |
| UseOffHeapBuffer | true | Whether Tjahzi should allocate native buffer for Log buffer component. We can go into a rabbit hole of divagations what are the implications of this. Most important in our view is that having 10s or 100s of MB of space taken out of heap is not very friendly to garbage collector which might have to occasionally copy it around. |
| BatchSize | 100 KB | Like in promtail configuration maximum batch size (in bytes) of logs to accumulate before sending the batch to Loki. |
| BatchWait | 5s | Like in promtail configuration maximum amount of time to wait before sending a batch, even if that batch isn't full. |
| logShipperWakeupIntervalMillis | 10 | The agent that reads data from log buffer, compresses it and sends to Loki via http is called LogShipper. This property controls how often it wakes up to perform its duties. Other properties control how often the data should be sent to Loki (batchSize, batchWait) this one just control how often to wake up and check for these conditions. In versions before 0.9.17 it was left at default 1ms which caused high CPU usage on some setups. |
| shutdownTimeoutSeconds | 10s | On logging system shutdown (or config reload) Tjahzi will flush its internal buffers so that no logs are lost. This property sets limit on how long to wait for this to complete before proceeding with shutdown. |
| useDaemonThreads | false | If set to true Tjahzi will run all it's threads as daemon threads. Use this option if you do not want to explicitly close the logging system and still want to make sure Tjahzi internal threads will not prevent JVM from closing down. Note that this can result in unflushed logs not being delivered when the JVM is closed. |