Skip to content

Commit

Permalink
Update Pulsar Functions in 2.4.1 release (apache#5133)
Browse files Browse the repository at this point in the history
Fixes apache#4554 

### Motivation
Adopt new structure for Pulsar Functions in 2.4.1 release.

### Modifications
Adopt changes in apache#5118 to the `version-2.4.1` folder.
  • Loading branch information
Jennifer88huang-zz authored and sijie committed Sep 6, 2019
1 parent 1302c40 commit 3a696d2
Show file tree
Hide file tree
Showing 9 changed files with 447 additions and 1,421 deletions.
721 changes: 0 additions & 721 deletions site2/website/versioned_docs/version-2.4.1/functions-api.md

This file was deleted.

238 changes: 119 additions & 119 deletions site2/website/versioned_docs/version-2.4.1/functions-cli.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions site2/website/versioned_docs/version-2.4.1/functions-debug.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: version-2.4.1-functions-debug
title: How to debug Pulsar Functions
sidebar_label: Debug
title: Debug Pulsar Functions
sidebar_label: How-to: Debug
original_id: functions-debug
---

Expand Down
14 changes: 7 additions & 7 deletions site2/website/versioned_docs/version-2.4.1/functions-deploy.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: version-2.4.1-functions-deploy
title: Pulsar Functions Deploy
sidebar_label: Deploy
title: Deploy Pulsar Functions
sidebar_label: How-to: Deploy
original_id: functions-deploy
---

Expand Down Expand Up @@ -32,8 +32,8 @@ Function name | You can specify any value for the class name (except org, librar
Tenant | Derived from names of the input topics. If the input topics are under the `marketing` tenant, which means the topic names have the form `persistent://marketing/{namespace}/{topicName}`, the tenant is `marketing`.
Namespace | Derived from names of the input topics. If the input topics are under the `asia` namespace under the `marketing` tenant, which means the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace is `asia`.
Output topic | `{input topic}-{function name}-output`. For example, if an input topic name of a function is `incoming`, and the function name is `exclamation`, then the name of the output topic is `incoming-exclamation-output`.
Subscription type | For `at-least-once` and `at-most-once` [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) mode is applied by default; for `effectively-once` guarantees, the [`FAILOVER`](concepts-messaging.md#failover) mode is applied.
Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md)
Subscription type | For `at-least-once` and `at-most-once` [processing guarantees](functions-overview.md#processing-guarantees), the [`SHARED`](concepts-messaging.md#shared) mode is applied by default; for `effectively-once` guarantees, the [`FAILOVER`](concepts-messaging.md#failover) mode is applied.
Processing guarantees | [`ATLEAST_ONCE`](functions-overview.md#processing-guarantees)
Pulsar service URL | `pulsar://localhost:6650`

### Example of default arguments
Expand Down Expand Up @@ -153,23 +153,23 @@ $ bin/pulsar-admin functions create \
```

> #### Resources are *per instance*
> The resources that you apply to a given Pulsar Function are applied to each [instance](#parallelism) of the function. For example, if you apply 8 GB of RAM to a function with a parallelism of 5, you are applying 40 GB of RAM for the function in total. Make sure that you take the parallelism (the number of instances) factor into your resource calculations.
> The resources that you apply to a given Pulsar Function are applied to each instance of the function. For example, if you apply 8 GB of RAM to a function with a parallelism of 5, you are applying 40 GB of RAM for the function in total. Make sure that you take the parallelism (the number of instances) factor into your resource calculations.
## Trigger Pulsar Functions

If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function output (if any) via the command line.

> Triggering a function is to invoke a function by producing a message on one of the input topics. With the [`pulsar-admin functions trigger`](reference-pulsar-admin.md#trigger) command, you can send messages to functions without using the [`pulsar-client`](reference-cli-tools.md#pulsar-client) tool or a language-specific client library.
To learn how to trigger a function, you can start with [Python function](functions-api.md#functions-for-python) that returns a simple string based on the input.
To learn how to trigger a function, you can start with Python function that returns a simple string based on the input.

```python
# myfunc.py
def process(input):
return "This function has been triggered with a value of {0}".format(input)
```

You can run the function in [local run mode](functions-deploying.md#local-run-mode).
You can run the function in [local run mode](functions-deploy.md#local-run-mode).

```bash
$ bin/pulsar-admin functions create \
Expand Down
46 changes: 33 additions & 13 deletions site2/website/versioned_docs/version-2.4.1/functions-develop.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
id: version-2.4.1-functions-develop
title: Develop Pulsar Functions
sidebar_label: Develop functions
sidebar_label: How-to: Develop
original_id: functions-develop
---

Expand Down Expand Up @@ -89,7 +89,7 @@ For complete code, see [here](https://github.com/apache/pulsar/blob/master/pulsa
<!--END_DOCUSAURUS_CODE_TABS-->

## Schema registry
Pulsar has a built in [Schema Registry](concepts-schema-registry) and comes bundled with a variety of popular schema types(avro, json and protobuf). Pulsar Functions can leverage existing schema information from input topics and derive the input type. The schema registry applies for output topic as well.
Pulsar has a built in schema registry and comes bundled with a variety of popular schema types(avro, json and protobuf). Pulsar Functions can leverage existing schema information from input topics and derive the input type. The schema registry applies for output topic as well.

## SerDe
SerDe stands for **Ser**ialization and **De**serialization. Pulsar Functions uses SerDe when publishing data to and consuming data from Pulsar topics. How SerDe works by default depends on the language you use for a particular function.
Expand Down Expand Up @@ -118,7 +118,7 @@ public interface SerDe<T> {
<!--Python-->
In Python, the default SerDe is identity, meaning that the type is serialized as whatever type the producer function returns.

You can specify the SerDe when [creating](functions-deploying.md#cluster-mode) or [running](functions-deploying.md#local-run-mode) functions.
You can specify the SerDe when [creating](functions-deploy.md#cluster-mode) or [running](functions-deploy.md#local-run-mode) functions.

```bash
$ bin/pulsar-admin functions create \
Expand Down Expand Up @@ -244,24 +244,24 @@ In order to use this class in Pulsar Functions, you have two options:
In both languages, however, you can write custom SerDe logic for more complex, application-specific types.

## Context
Both the [Java](#java-sdk-functions), [Python](#python-sdk-functions) and [Go](#go-sdk-functions) SDKs provide access to a **context object** that can be used by a function. This context object provides a wide variety of information and functionality to the function.
Java, Python and Go SDKs provide access to a **context object** that can be used by a function. This context object provides a wide variety of information and functionality to the function.

* The name and ID of a Pulsar Function.
* The message ID of each message. Each Pulsar message is automatically assigned with an ID.
* The name of the topic to which the message is sent.
* The names of all input topics as well as the output topic associated with the function.
* The name of the class used for [SerDe](#serialization-and-deserialization-serde).
* The name of the class used for [SerDe](#serde).
* The [tenant](reference-terminology.md#tenant) and namespace associated with the function.
* The ID of the Pulsar Functions instance running the function.
* The version of the function.
* The [logger object](functions-overview.md#logging) used by the function, which can be used to create function log messages.
* Access to arbitrary [user configuration](#user-configuration) values supplied via the CLI.
* An interface for recording [metrics](functions-metrics.md).
* An interface for storing and retrieving state in [state storage](functions-overview.md#state-storage).
* The [logger object](functions-develop.md#logger) used by the function, which can be used to create function log messages.
* Access to arbitrary [user configuration](#user-config) values supplied via the CLI.
* An interface for recording [metrics](#metrics).
* An interface for storing and retrieving state in [state storage](#state-storage).

<!--DOCUSAURUS_CODE_TABS-->
<!--Java-->
The {@inject: javadoc:Context:/client/org/apache/pulsar/functions/api/Context} interface provides a number of methods that you can use to access the function [context](#context). The various method signatures for the `Context` interface are listed as follows.
The [Context](https://github.com/apache/pulsar/blob/master/pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java) interface provides a number of methods that you can use to access the function [context](#context). The various method signatures for the `Context` interface are listed as follows.

```java
public interface Context {
Expand Down Expand Up @@ -361,6 +361,26 @@ func (c *FunctionContext) GetUserConfMap() map[string]interface{} {
}
```

The following example uses several methods available via the `Context` object.

```
import (
"context"
"fmt"
"github.com/apache/pulsar/pulsar-function-go/pf"
)
func contextFunc(ctx context.Context) {
if fc, ok := pf.FromContext(ctx); ok {
fmt.Printf("function ID is:%s, ", fc.GetFuncID())
fmt.Printf("function version is:%s\n", fc.GetFuncVersion())
}
}
```

For complete code, see [here](https://github.com/apache/pulsar/blob/master/pulsar-function-go/examples/contextFunc.go#L29-L34).

<!--END_DOCUSAURUS_CODE_TABS-->

### User config
Expand Down Expand Up @@ -469,7 +489,7 @@ class UserConfigFunction(Function):

<!--DOCUSAURUS_CODE_TABS-->
<!--Java-->
Pulsar Functions that use the [Java SDK](#java-sdk-functions) have access to an [SLF4j](https://www.slf4j.org/) [`Logger`](https://www.slf4j.org/api/org/apache/log4j/Logger.html) object that can be used to produce logs at the chosen log level. The following example logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`.
Pulsar Functions that use the Java SDK have access to an [SLF4j](https://www.slf4j.org/) [`Logger`](https://www.slf4j.org/api/org/apache/log4j/Logger.html) object that can be used to produce logs at the chosen log level. The following example logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`.

```java
import org.apache.pulsar.functions.api.Context;
Expand Down Expand Up @@ -506,7 +526,7 @@ $ bin/pulsar-admin functions create \
All logs produced by `LoggingFunction` above can be accessed via the `persistent://public/default/logging-function-logs` topic.

<!--Python-->
Pulsar Functions that use the [Python SDK](#python-sdk-functions) have access to a logging object that can be used to produce logs at the chosen log level. The following example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`.
Pulsar Functions that use the Python SDK have access to a logging object that can be used to produce logs at the chosen log level. The following example function that logs either a `WARNING`- or `INFO`-level log based on whether the incoming string contains the word `danger`.

```python
from pulsar import Function
Expand Down Expand Up @@ -564,7 +584,7 @@ When you use `logTopic` related functionalities in Go Function, import `github.c
## Metrics
Pulsar Functions can publish arbitrary metrics to the metrics interface which can be queried.

> If a Pulsar Function uses the language-native interface for [Java](functions-api.md#java-native-functions) or [Python](#python-native-functions), that function is not able to publish metrics and stats to Pulsar.
> If a Pulsar Function uses the language-native interface for Java or Python, that function is not able to publish metrics and stats to Pulsar.
<!--DOCUSAURUS_CODE_TABS-->
<!--Java-->
Expand Down
Loading

0 comments on commit 3a696d2

Please sign in to comment.