Skip to content

Commit

Permalink
[doc] update doc for debezium connector and log-topic message (apache…
Browse files Browse the repository at this point in the history
…#11451)

* update doc

* update
  • Loading branch information
nlu90 authored Jul 27, 2021
1 parent ee202d0 commit 8e7bfe0
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 12 deletions.
5 changes: 5 additions & 0 deletions site2/docs/functions-debug.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,11 @@ $ bin/pulsar-admin functions create \
# Other function configs
```

The message published to log topic contains several properties for better reasoning:
- `loglevel` -- the level of the log message.
- `fqn` -- fully qualified function name pushes this log message.
- `instance` -- the ID of the function instance pushes this log message.

## Use Functions CLI

With [Pulsar Functions CLI](reference-pulsar-admin.md#functions), you can debug Pulsar Functions with the following subcommands:
Expand Down
17 changes: 5 additions & 12 deletions site2/docs/io-debezium-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ The configuration of Debezium source connector has the following properties.
| `database.history` | true | null | The name of the database history class. |
| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements. <br/><br/>**Note: this topic is for internal use only and should not be used by consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `pulsar.service.url` | true | null | Pulsar cluster service URL. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
| `json-with-envelope` | false | false | Present the message only consist of payload.

Expand Down Expand Up @@ -81,7 +80,6 @@ You can use one of the following methods to create a configuration file.
"database.history.pulsar.service.url": "pulsar://127.0.0.1:6650",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"pulsar.service.url": "pulsar://127.0.0.1:6650",
"offset.storage.topic": "offset-topic"
}
```
Expand Down Expand Up @@ -116,9 +114,6 @@ You can use one of the following methods to create a configuration file.
key.converter: "org.apache.kafka.connect.json.JsonConverter"
value.converter: "org.apache.kafka.connect.json.JsonConverter"

## PULSAR_SERVICE_URL_CONFIG
pulsar.service.url: "pulsar://127.0.0.1:6650"

## OFFSET_STORAGE_TOPIC_CONFIG
offset.storage.topic: "offset-topic"
```
Expand Down Expand Up @@ -217,7 +212,7 @@ You can use one of the following methods to create a configuration file.
"plugin.name": "pgoutput",
"schema.whitelist": "public",
"table.whitelist": "public.users",
"pulsar.service.url": "pulsar://127.0.0.1:6650"
"database.history.pulsar.service.url": "pulsar://127.0.0.1:6650"
}
```

Expand Down Expand Up @@ -247,7 +242,7 @@ You can use one of the following methods to create a configuration file.
table.whitelist: "public.users"

## PULSAR_SERVICE_URL_CONFIG
pulsar.service.url: "pulsar://127.0.0.1:6650"
database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"
```

Notice that `pgoutput` is a standard plugin of Postgres introduced in version 10 - [see Postgres architecture docu](https://www.postgresql.org/docs/10/logical-replication-architecture.html). You don't need to install anything, just make sure the WAL level is set to `logical` (see docker command below and [Postgres docu](https://www.postgresql.org/docs/current/runtime-config-wal.html)).
Expand Down Expand Up @@ -361,7 +356,7 @@ You need to create a configuration file before using the Pulsar Debezium connect
"mongodb.password": "dbz",
"mongodb.task.id": "1",
"database.whitelist": "inventory",
"pulsar.service.url": "pulsar://127.0.0.1:6650"
"database.history.pulsar.service.url": "pulsar://127.0.0.1:6650"
}
```

Expand All @@ -386,9 +381,7 @@ You need to create a configuration file before using the Pulsar Debezium connect
mongodb.password: "dbz",
mongodb.task.id: "1",
database.whitelist: "inventory",

## PULSAR_SERVICE_URL_CONFIG
pulsar.service.url: "pulsar://127.0.0.1:6650"
database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"
```

### Usage
Expand Down Expand Up @@ -429,7 +422,7 @@ This example shows how to change the data of a MongoDB table using the Pulsar De
--destination-topic-name debezium-mongodb-topic \
--tenant public \
--namespace default \
--source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user": "debezium","mongodb.password": "dbz","mongodb.task.id": "1","database.whitelist": "inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}'
--source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user": "debezium","mongodb.password": "dbz","mongodb.task.id": "1","database.whitelist": "inventory","database.history.pulsar.service.url": "pulsar://127.0.0.1:6650"}'
```

* Use the **YAML** configuration file as shown previously.
Expand Down

0 comments on commit 8e7bfe0

Please sign in to comment.