Skip to content

Commit

Permalink
Pulsar Functions resource docs (apache#1760)
Browse files Browse the repository at this point in the history
* add section on function resources to overview

* add resource flags to YAML

* add section to deployment doc

* add runtime info
  • Loading branch information
lucperkins authored and sijie committed Jun 9, 2018
1 parent 3e74aff commit cb4f6ad
Show file tree
Hide file tree
Showing 4 changed files with 64 additions and 6 deletions.
12 changes: 12 additions & 0 deletions site/_data/cli/pulsar-admin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ commands:
- name: localrun
description: Run a Pulsar Function locally
options:
- flags: --cpu
description: The CPU to allocate to each function instance (in number of cores)
- flags: --ram
description: The RAM to allocate to each function instance (in bytes)
- flags: --disk
description: The disk space to allocate to each function instance (in bytes)
- flags: --brokerServiceUrl
description: The URL of the Pulsar broker
- flags: --className
Expand Down Expand Up @@ -163,6 +169,12 @@ commands:
- name: create
description: Creates a new Pulsar Function on the target infrastructure
options:
- flags: --cpu
description: The CPU to allocate to each function instance (in number of cores)
- flags: --ram
description: The RAM to allocate to each function instance (in bytes)
- flags: --disk
description: The disk space to allocate to each function instance (in bytes)
- flags: --className
description: The name of the function's class
- flags: --customSerdeInputs
Expand Down
2 changes: 0 additions & 2 deletions site/docs/latest/admin-api/persistent-topics.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,6 @@ Permission can be fetched using [`permissions`](../../reference/CliTools#permiss

#### pulsar-admin

TODO: admin

```shell
$ pulsar-admin persistent permissions \
persistent://test-tenant/ns1/tp1 \
Expand Down
26 changes: 25 additions & 1 deletion site/docs/latest/functions/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ When managing Pulsar Functions, you'll need to specify a variety of information

Parameter | Default
:---------|:-------
Function name | Whichever value is specified for the class name. For example, `--className org.example.MyFunction` would give the function a name of `MyFunction`
Function name | Whichever value is specified for the class name (minus org, library, etc.). The flag `--className org.example.MyFunction`, for example, would give the function a name of `MyFunction`.
Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`.
Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`.
Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`.
Expand Down Expand Up @@ -143,6 +143,30 @@ $ bin/pulsar-admin functions update \
--functionConfigFile function-config.yaml
```

### Function instance resources {#resources}

When you run Pulsar Functions in [cluster run](#cluster-run) mode, you can specify the resources that are assigned to each function [instance](#parallelism):

Resource | Specified as... | Runtimes
:--------|:----------------|:--------
CPU | The number of cores | Docker (coming soon)
RAM | The number of bytes | Process, Docker
Disk space | The number of bytes | Docker

Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function:

```bash
$ bin/pulsar-admin functions create \
--jar target/my-functions.jar \
--className org.example.functions.MyFunction \
--cpu 8 \
--ram 8589934592 \
--disk 10737418240
```

{% include admonition.html type="warning" title="Resources are *per instance*"
content="The resources that you apply to a given Pulsar Function are applied to each [instance](#parallelism) of the function. If you apply 8 GB of RAM to a function with a paralellism of 5, for example, then you are applying 40 GB of RAM total for the function. You should always make sure to factor paralellism---i.e. the number of instances---into your resource calculations." %}

## Triggering Pulsar Functions {#triggering}

If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function's output (if any) via the command line.
Expand Down
30 changes: 27 additions & 3 deletions site/docs/latest/functions/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,9 +194,10 @@ Pulsar Functions can currently be written in [Java](../../functions/api#java) an
## The Pulsar Functions API {#api}
* Type safe (bytes versus specific types)
* SerDe (built-in vs. custom)
* Pulsar messages are always just bytes, but Pulsar Functions handles data types for you *unless* you need custom types
The Pulsar Functions API enables you to create processing logic that is:
* Type safe. Pulsar Functions can process raw bytes or more complex, application-specific types.
* Based on SerDe (**Ser**ialization/**De**serialization). A variety of types are supported "out of the box" but you can also create your own custom SerDe logic.
### Function context {#context}
Expand Down Expand Up @@ -317,6 +318,29 @@ $ bin/pulsar-admin functions create \
--parallelism 5
```

### Function instance resources {#resources}

When you run Pulsar Functions in [cluster run](#cluster-run) mode, you can specify the resources that are assigned to each function [instance](#parallelism):

Resource | Specified as... | Runtimes
:--------|:----------------|:--------
CPU | The number of cores | Docker (coming soon)
RAM | The number of bytes | Process, Docker
Disk space | The number of bytes | Docker

Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function:

```bash
$ bin/pulsar-admin functions create \
--jar target/my-functions.jar \
--className org.example.functions.MyFunction \
--cpu 8 \
--ram 8589934592 \
--disk 10737418240
```

For more information on resources, see the [Deploying and Managing Pulsar Functions](../deployment#resources) documentation.

### Logging

Pulsar Functions created using the [Pulsar Functions SDK(#sdk) can send logs to a log topic that you specify as part of the function's configuration. The function created using the command below, for example, would produce all logs on the `persistent://public/default/my-func-1-log` topic:
Expand Down

0 comments on commit cb4f6ad

Please sign in to comment.