Skip to content

Commit 41e6288

Browse files
authored
[Improve][Connector-V2] Support INFINI Easysearch (apache#5933)
1 parent 78abe2f commit 41e6288

File tree

56 files changed

+4516
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+4516
-2
lines changed

config/plugin_config

+1
Original file line numberDiff line numberDiff line change
@@ -76,4 +76,5 @@ connector-tablestore
7676
connector-selectdb-cloud
7777
connector-hbase
7878
connector-amazonsqs
79+
connector-easysearch
7980
--end--
+202
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# INFINI Easysearch
2+
3+
## Support Those Engines
4+
5+
> Spark<br/>
6+
> Flink<br/>
7+
> SeaTunnel Zeta<br/>
8+
9+
## Description
10+
11+
A sink plugin which use send data to `INFINI Easysearch`.
12+
13+
## Using Dependency
14+
15+
> Depenndency [easysearch-client](https://central.sonatype.com/artifact/com.infinilabs/easysearch-client)
16+
>
17+
## Key features
18+
19+
- [ ] [exactly-once](../../concept/connector-v2-features.md)
20+
- [x] [cdc](../../concept/connector-v2-features.md)
21+
22+
:::tip
23+
24+
Engine Supported
25+
26+
* Supported all versions released by [INFINI Easysearch](https://www.infini.com/download/?product=easysearch).
27+
28+
:::
29+
30+
## Data Type Mapping
31+
32+
| Easysearch Data Type | SeaTunnel Data Type |
33+
|-----------------------------|----------------------|
34+
| STRING<br/>KEYWORD<br/>TEXT | STRING |
35+
| BOOLEAN | BOOLEAN |
36+
| BYTE | BYTE |
37+
| SHORT | SHORT |
38+
| INTEGER | INT |
39+
| LONG | LONG |
40+
| FLOAT<br/>HALF_FLOAT | FLOAT |
41+
| DOUBLE | DOUBLE |
42+
| Date | LOCAL_DATE_TIME_TYPE |
43+
44+
## Sink Options
45+
46+
| name | type | required | default value |
47+
|-------------------------|---------|----------|---------------|
48+
| hosts | array | yes | - |
49+
| index | string | yes | - |
50+
| primary_keys | list | no | |
51+
| key_delimiter | string | no | `_` |
52+
| username | string | no | |
53+
| password | string | no | |
54+
| max_retry_count | int | no | 3 |
55+
| max_batch_size | int | no | 10 |
56+
| tls_verify_certificate | boolean | no | true |
57+
| tls_verify_hostnames | boolean | no | true |
58+
| tls_keystore_path | string | no | - |
59+
| tls_keystore_password | string | no | - |
60+
| tls_truststore_path | string | no | - |
61+
| tls_truststore_password | string | no | - |
62+
| common-options | | no | - |
63+
64+
### hosts [array]
65+
66+
`INFINI Easysearch` cluster http address, the format is `host:port` , allowing multiple hosts to be specified. Such as `["host1:9200", "host2:9200"]`.
67+
68+
### index [string]
69+
70+
`INFINI Easysearch` `index` name.Index support contains variables of field name,such as `seatunnel_${age}`,and the field must appear at seatunnel row.
71+
If not, we will treat it as a normal index.
72+
73+
### primary_keys [list]
74+
75+
Primary key fields used to generate the document `_id`, this is cdc required options.
76+
77+
### key_delimiter [string]
78+
79+
Delimiter for composite keys ("_" by default), e.g., "$" would result in document `_id` "KEY1$KEY2$KEY3".
80+
81+
### username [string]
82+
83+
security username
84+
85+
### password [string]
86+
87+
security password
88+
89+
### max_retry_count [int]
90+
91+
one bulk request max try size
92+
93+
### max_batch_size [int]
94+
95+
batch bulk doc max size
96+
97+
### tls_verify_certificate [boolean]
98+
99+
Enable certificates validation for HTTPS endpoints
100+
101+
### tls_verify_hostname [boolean]
102+
103+
Enable hostname validation for HTTPS endpoints
104+
105+
### tls_keystore_path [string]
106+
107+
The path to the PEM or JKS key store. This file must be readable by the operating system user running SeaTunnel.
108+
109+
### tls_keystore_password [string]
110+
111+
The key password for the key store specified
112+
113+
### tls_truststore_path [string]
114+
115+
The path to PEM or JKS trust store. This file must be readable by the operating system user running SeaTunnel.
116+
117+
### tls_truststore_password [string]
118+
119+
The key password for the trust store specified
120+
121+
### common options
122+
123+
Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
124+
125+
## Examples
126+
127+
Simple
128+
129+
```bash
130+
sink {
131+
Easysearch {
132+
hosts = ["localhost:9200"]
133+
index = "seatunnel-${age}"
134+
}
135+
}
136+
```
137+
138+
CDC(Change data capture) event
139+
140+
```bash
141+
sink {
142+
Easysearch {
143+
hosts = ["localhost:9200"]
144+
index = "seatunnel-${age}"
145+
146+
# cdc required options
147+
primary_keys = ["key1", "key2", ...]
148+
}
149+
}
150+
```
151+
152+
SSL (Disable certificates validation)
153+
154+
```hocon
155+
sink {
156+
Easysearch {
157+
hosts = ["https://localhost:9200"]
158+
username = "admin"
159+
password = "admin"
160+
161+
tls_verify_certificate = false
162+
}
163+
}
164+
```
165+
166+
SSL (Disable hostname validation)
167+
168+
```hocon
169+
sink {
170+
Easysearch {
171+
hosts = ["https://localhost:9200"]
172+
username = "admin"
173+
password = "admin"
174+
175+
tls_verify_hostname = false
176+
}
177+
}
178+
```
179+
180+
SSL (Enable certificates validation)
181+
182+
```hocon
183+
sink {
184+
Easysearch {
185+
hosts = ["https://localhost:9200"]
186+
username = "admin"
187+
password = "admin"
188+
189+
tls_keystore_path = "${your Easysearch home}/config/certs/http.p12"
190+
tls_keystore_password = "${your password}"
191+
}
192+
}
193+
```
194+
195+
## Changelog
196+
197+
### 2.3.4 2023-11-16
198+
199+
- Add Easysearch Sink Connector
200+
- Support http/https protocol
201+
- Support CDC write DELETE/UPDATE/INSERT events
202+

0 commit comments

Comments
 (0)