From 5e2440b215acc4475ac6e83bb3b1a85971ec6a0b Mon Sep 17 00:00:00 2001 From: Devin Bost Date: Tue, 18 May 2021 01:44:51 -0600 Subject: [PATCH] [docs] Issue #10389 - Documenting function log level customization (#10617) This PR documents the feature added by #10389 and provides some additional information about how to use Log4j in Pulsar to configure Pulsar Function log levels. --- site2/docs/functions-develop.md | 184 +++++++++++++++++- .../version-2.7.2/functions-develop.md | 182 +++++++++++++++++ 2 files changed, 365 insertions(+), 1 deletion(-) diff --git a/site2/docs/functions-develop.md b/site2/docs/functions-develop.md index a14e537568924..3b0780d38a30a 100644 --- a/site2/docs/functions-develop.md +++ b/site2/docs/functions-develop.md @@ -656,6 +656,185 @@ $ bin/pulsar-admin functions create \ All logs produced by `LoggingFunction` above can be accessed via the `persistent://public/default/logging-function-logs` topic. +#### Customize Function log level +Additionally, you can use the XML file, `functions_log4j2.xml`, to customize the function log level. +To customize the function log level, create or update `functions_log4j2.xml` in your Pulsar conf directory (for example, `/etc/pulsar/` on bare-metal, or `/pulsar/conf` on Kubernetes) to contain contents such as: + +```xml + + pulsar-functions-instance + 30 + + + pulsar.log.appender + RollingFile + + + pulsar.log.level + debug + + + bk.log.level + debug + + + + + Console + SYSTEM_OUT + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + RollingFile + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.log + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}-%d{MM-dd-yyyy}-%i.log.gz + true + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + 1 + true + + + 1 GB + + + 0 0 0 * * ? + + + + + ${sys:pulsar.function.log.dir} + 2 + + */${sys:pulsar.function.log.file}*log.gz + + + 30d + + + + + + BkRollingFile + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.bk + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.bk-%d{MM-dd-yyyy}-%i.log.gz + true + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + 1 + true + + + 1 GB + + + 0 0 0 * * ? + + + + + ${sys:pulsar.function.log.dir} + 2 + + */${sys:pulsar.function.log.file}.bk*log.gz + + + 30d + + + + + + + + org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper + ${sys:bk.log.level} + false + + BkRollingFile + + + + ${sys:pulsar.log.level} + + ${sys:pulsar.log.appender} + ${sys:pulsar.log.level} + + + + +``` + +The properties set like: +```xml + + pulsar.log.level + debug + +``` +propagate to places where they are referenced, such as: +```xml + + ${sys:pulsar.log.level} + + ${sys:pulsar.log.appender} + ${sys:pulsar.log.level} + + +``` +In the above example, debug level logging would be applied to ALL function logs. +This may be more verbose than you desire. To be more selective, you can apply different log levels to different classes or modules. For example: + +```xml + + com.example.module + info + false + + ${sys:pulsar.log.appender} + + +``` +You can be more specific as well, such as applying a more verbose log level to a class in the module, such as: +```xml + + com.example.module.className + debug + false + + Console + + +``` + +Each `` entry allows you to output the log to a target specified in the definition of the Appender. + +Additivity pertains to whether log messages will be duplicated if multiple Logger entries overlap. +To disable additivity, specify +```xml +false +``` +as shown in examples above. Disabling additivity prevents duplication of log messages when one or more `` entries contain classes or modules that overlap. + +The `` is defined in the `` section, such as: +```xml + + Console + SYSTEM_OUT + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + +``` + Pulsar Functions that use the Python SDK have access to a logging object that can be used to produce logs at the chosen log level. The following example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`. @@ -683,6 +862,7 @@ $ bin/pulsar-admin functions create \ ``` All logs produced by `LoggingFunction` above can be accessed via the `logging-function-logs` topic. +Additionally, you can specify the function log level through the broker XML file as described in [Customize Function log level](#customize-function-log-level). The following Go Function example shows different log levels based on the function input. @@ -709,7 +889,9 @@ func main() { } ``` -When you use `logTopic` related functionalities in Go Function, import `github.com/apache/pulsar/pulsar-function-go/logutil`, and you do not have to use the `getLogger()` context object. +When you use `logTopic` related functionalities in Go Function, import `github.com/apache/pulsar/pulsar-function-go/logutil`, and you do not have to use the `getLogger()` context object. + +Additionally, you can specify the function log level through the broker XML file, as described here: [Customize Function log level](#customize-function-log-level) diff --git a/site2/website/versioned_docs/version-2.7.2/functions-develop.md b/site2/website/versioned_docs/version-2.7.2/functions-develop.md index 1af512ec9f719..16779f8ef75bd 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-develop.md @@ -657,6 +657,185 @@ $ bin/pulsar-admin functions create \ All logs produced by `LoggingFunction` above can be accessed via the `persistent://public/default/logging-function-logs` topic. +#### Customize Function log level +Additionally, you can use the XML file, `functions_log4j2.xml`, to customize the function log level. +To customize the function log level, create or update `functions_log4j2.xml` in your Pulsar conf directory (for example, `/etc/pulsar/` on bare-metal, or `/pulsar/conf` on Kubernetes) to contain contents such as: + +```xml + + pulsar-functions-instance + 30 + + + pulsar.log.appender + RollingFile + + + pulsar.log.level + debug + + + bk.log.level + debug + + + + + Console + SYSTEM_OUT + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + RollingFile + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.log + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}-%d{MM-dd-yyyy}-%i.log.gz + true + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + 1 + true + + + 1 GB + + + 0 0 0 * * ? + + + + + ${sys:pulsar.function.log.dir} + 2 + + */${sys:pulsar.function.log.file}*log.gz + + + 30d + + + + + + BkRollingFile + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.bk + ${sys:pulsar.function.log.dir}/${sys:pulsar.function.log.file}.bk-%d{MM-dd-yyyy}-%i.log.gz + true + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + + + 1 + true + + + 1 GB + + + 0 0 0 * * ? + + + + + ${sys:pulsar.function.log.dir} + 2 + + */${sys:pulsar.function.log.file}.bk*log.gz + + + 30d + + + + + + + + org.apache.pulsar.functions.runtime.shaded.org.apache.bookkeeper + ${sys:bk.log.level} + false + + BkRollingFile + + + + ${sys:pulsar.log.level} + + ${sys:pulsar.log.appender} + ${sys:pulsar.log.level} + + + + +``` + +The properties set like: +```xml + + pulsar.log.level + debug + +``` +propagate to places where they are referenced, such as: +```xml + + ${sys:pulsar.log.level} + + ${sys:pulsar.log.appender} + ${sys:pulsar.log.level} + + +``` +In the above example, debug level logging would be applied to ALL function logs. +This may be more verbose than you desire. To be more selective, you can apply different log levels to different classes or modules. For example: + +```xml + + com.example.module + info + false + + ${sys:pulsar.log.appender} + + +``` +You can be more specific as well, such as applying a more verbose log level to a class in the module, such as: +```xml + + com.example.module.className + debug + false + + Console + + +``` + +Each `` entry allows you to output the log to a target specified in the definition of the Appender. + +Additivity pertains to whether log messages will be duplicated if multiple Logger entries overlap. +To disable additivity, specify +```xml +false +``` +as shown in examples above. Disabling additivity prevents duplication of log messages when one or more `` entries contain classes or modules that overlap. + +The `` is defined in the `` section, such as: +```xml + + Console + SYSTEM_OUT + + %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n + + +``` + Pulsar Functions that use the Python SDK have access to a logging object that can be used to produce logs at the chosen log level. The following example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`. @@ -684,6 +863,7 @@ $ bin/pulsar-admin functions create \ ``` All logs produced by `LoggingFunction` above can be accessed via the `logging-function-logs` topic. +Additionally, you can specify the function log level through the broker XML file as described in [Customize Function log level](#customize-function-log-level). The following Go Function example shows different log levels based on the function input. @@ -712,6 +892,8 @@ func main() { When you use `logTopic` related functionalities in Go Function, import `github.com/apache/pulsar/pulsar-function-go/logutil`, and you do not have to use the `getLogger()` context object. +Additionally, you can specify the function log level through the broker XML file, as described here: [Customize Function log level](#customize-function-log-level) + ### Pulsar admin