Skip to content

Commit 3aa4ae5

Browse files
zhangchengming601zhangchengming601
authored
[Doc] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (apache#5102)
1 parent 4acd370 commit 3aa4ae5

File tree

4 files changed

+656
-0
lines changed

4 files changed

+656
-0
lines changed

‎docs/en/connector-v2/sink/DB2.md

+171
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# DB2
2+
3+
> JDBC DB2 Sink Connector
4+
5+
## Support Those Engines
6+
7+
> Spark<br/>
8+
> Flink<br/>
9+
> SeaTunnel Zeta<br/>
10+
11+
## Key Features
12+
13+
- [x] [exactly-once](../../concept/connector-v2-features.md)
14+
- [ ] [cdc](../../concept/connector-v2-features.md)
15+
16+
> Use `Xa transactions` to ensure `exactly-once`. So only support `exactly-once` for the database which is
17+
> support `Xa transactions`. You can set `is_exactly_once=true` to enable it.
18+
19+
## Description
20+
21+
Write data through jdbc. Support Batch mode and Streaming mode, support concurrent writing, support exactly-once
22+
semantics (using XA transaction guarantee).
23+
24+
## Supported DataSource Info
25+
26+
| Datasource | Supported Versions | Driver | Url | Maven |
27+
|------------|----------------------------------------------------------|--------------------------------|-----------------------------------|-----------------------------------------------------------------------|
28+
| DB2 | Different dependency version has different driver class. | com.ibm.db2.jdbc.app.DB2Driver | jdbc:db2://127.0.0.1:50000/dbname | [Download](https://mvnrepository.com/artifact/com.ibm.db2.jcc/db2jcc) |
29+
30+
## Database Dependency
31+
32+
> Please download the support list corresponding to 'Maven' and copy it to the '$SEATNUNNEL_HOME/plugins/jdbc/lib/' working directory<br/>
33+
> For example DB2 datasource: cp db2-connector-java-xxx.jar $SEATNUNNEL_HOME/plugins/jdbc/lib/
34+
35+
## Data Type Mapping
36+
37+
| DB2 Data type | SeaTunnel Data type |
38+
|------------------------------------------------------------------------------------------------------|---------------------|---|
39+
| BOOLEAN | BOOLEAN |
40+
| SMALLINT | SHORT |
41+
| INT<br/>INTEGER<br/> | INTEGER |
42+
| BIGINT | LONG |
43+
| DECIMAL<br/>DEC<br/>NUMERIC<br/>NUM | DECIMAL(38,18) |
44+
| REAL | FLOAT |
45+
| FLOAT<br/>DOUBLE<br/>DOUBLE PRECISION<br/>DECFLOAT | DOUBLE |
46+
| CHAR<br/>VARCHAR<br/>LONG VARCHAR<br/>CLOB<br/>GRAPHIC<br/>VARGRAPHIC<br/>LONG VARGRAPHIC<br/>DBCLOB | STRING |
47+
| BLOB | BYTES |
48+
| DATE | DATE |
49+
| TIME | TIME |
50+
| TIMESTAMP | TIMESTAMP |
51+
| ROWID<br/>XML | Not supported yet |
52+
53+
## Sink Options
54+
55+
| Name | Type | Required | Default | Description |
56+
|-------------------------------------------|---------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
57+
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:db2://127.0.0.1:50000/dbname |
58+
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use DB2 the value is `com.ibm.db2.jdbc.app.DB2Driver`. |
59+
| user | String | No | - | Connection instance user name |
60+
| password | String | No | - | Connection instance password |
61+
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority |
62+
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
63+
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
64+
| primary_keys | Array | No | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql. |
65+
| support_upsert_by_query_primary_key_exist | Boolean | No | false | Choose to use INSERT sql, UPDATE sql to process update events(INSERT, UPDATE_AFTER) based on query primary key exists. This configuration is only used when database unsupport upsert syntax. **Note**: that this method has low performance |
66+
| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete. |
67+
| max_retries | Int | No | 0 | The number of retries to submit failed (executeBatch) |
68+
| batch_size | Int | No | 1000 | For batch writing, when the number of buffered records reaches the number of `batch_size` or the time reaches `batch_interval_ms`<br/>, the data will be flushed into the database |
69+
| batch_interval_ms | Int | No | 1000 | For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the database |
70+
| is_exactly_once | Boolean | No | false | Whether to enable exactly-once semantics, which will use Xa transactions. If on, you need to<br/>set `xa_data_source_class_name`. |
71+
| generate_sink_sql | Boolean | No | false | Generate sql statements based on the database table you want to write to |
72+
| xa_data_source_class_name | String | No | - | The xa data source class name of the database Driver, for example, DB2 is `com.db2.cj.jdbc.Db2XADataSource`, and<br/>please refer to appendix for other data sources |
73+
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
74+
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
75+
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
76+
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |
77+
78+
### Tips
79+
80+
> If partition_column is not set, it will run in single concurrency, and if partition_column is set, it will be executed in parallel according to the concurrency of tasks.
81+
82+
## Task Example
83+
84+
### Simple:
85+
86+
> This example defines a SeaTunnel synchronization task that automatically generates data through FakeSource and sends it to JDBC Sink. FakeSource generates a total of 16 rows of data (row.num=16), with each row having two fields, name (string type) and age (int type). The final target table is test_table will also be 16 rows of data in the table. Before run this job, you need create database test and table test_table in your DB2. And if you have not yet installed and deployed SeaTunnel, you need to follow the instructions in [Install SeaTunnel](../../start-v2/locally/deployment.md) to install and deploy SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel Engine](../../start-v2/locally/quick-start-seatunnel-engine.md) to run this job.
87+
88+
```
89+
# Defining the runtime environment
90+
env {
91+
# You can set flink configuration here
92+
execution.parallelism = 1
93+
job.mode = "BATCH"
94+
}
95+
96+
source {
97+
# This is a example source plugin **only for test and demonstrate the feature source plugin**
98+
FakeSource {
99+
parallelism = 1
100+
result_table_name = "fake"
101+
row.num = 16
102+
schema = {
103+
fields {
104+
name = "string"
105+
age = "int"
106+
}
107+
}
108+
}
109+
# If you would like to get more information about how to configure seatunnel and see full list of source plugins,
110+
# please go to https://seatunnel.apache.org/docs/category/source-v2
111+
}
112+
113+
transform {
114+
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
115+
# please go to https://seatunnel.apache.org/docs/category/transform-v2
116+
}
117+
118+
sink {
119+
jdbc {
120+
url = "jdbc:db2://127.0.0.1:50000/dbname"
121+
driver = "com.ibm.db2.jdbc.app.DB2Driver"
122+
user = "root"
123+
password = "123456"
124+
query = "insert into test_table(name,age) values(?,?)"
125+
}
126+
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
127+
# please go to https://seatunnel.apache.org/docs/category/sink-v2
128+
}
129+
```
130+
131+
### Generate Sink SQL
132+
133+
> This example not need to write complex sql statements, you can configure the database name table name to automatically generate add statements for you
134+
135+
```
136+
sink {
137+
jdbc {
138+
url = "jdbc:db2://127.0.0.1:50000/dbname"
139+
driver = "com.ibm.db2.jdbc.app.DB2Driver"
140+
user = "root"
141+
password = "123456"
142+
# Automatically generate sql statements based on database table names
143+
generate_sink_sql = true
144+
database = test
145+
table = test_table
146+
}
147+
}
148+
```
149+
150+
### Exactly-once :
151+
152+
> For accurate write scene we guarantee accurate once
153+
154+
```
155+
sink {
156+
jdbc {
157+
url = "jdbc:db2://127.0.0.1:50000/dbname"
158+
driver = "com.ibm.db2.jdbc.app.DB2Driver"
159+
160+
max_retries = 0
161+
user = "root"
162+
password = "123456"
163+
query = "insert into test_table(name,age) values(?,?)"
164+
165+
is_exactly_once = "true"
166+
167+
xa_data_source_class_name = "com.db2.cj.jdbc.Db2XADataSource"
168+
}
169+
}
170+
```
171+

0 commit comments

Comments
 (0)