English | 中文
-
We are recruiting Big data platform development engineers.If you want more information about the position, please add WeChat ID [ysqwhiletrue] or email your resume to [email protected].
-
We use DingTalk to communicate,You can search the group number [30537511] or scan the QR code below to join the communication group
FlinkX is a data synchronization tool based on Flink. FlinkX can collect static data, such as MySQL, HDFS, etc, as well as real-time changing data, such as MySQL binlog, Kafka, etc. FlinkX currently includes the following features:
-
Most plugins support concurrent reading and writing of data, which can greatly improve the speed of reading and writing;
-
Some plug-ins support the function of failure recovery, which can restore tasks from the failed location and save running time; Failure Recovery
-
The Reader plugin for relational databases supports interval polling. It can continuously collect changing data; Interval Polling
-
Some databases support opening Kerberos security authentication; Kerberos
-
Limit the reading speed of Reader plugins and reduce the impact on business databases;
-
Save the dirty data when writing data;
-
Limit the maximum number of dirty data;
-
Multiple running modes: Local,Standalone,Yarn Session,Yarn Per;
The following databases are currently supported:
| Database Type | Reader | Writer | |
|---|---|---|---|
| Batch Synchronization | MySQL | doc | doc |
| Oracle | doc | doc | |
| SqlServer | doc | doc | |
| PostgreSQL | doc | doc | |
| DB2 | doc | doc | |
| GBase | doc | doc | |
| ClickHouse | doc | doc | |
| PolarDB | doc | doc | |
| SAP Hana | doc | doc | |
| Teradata | doc | doc | |
| Phoenix | doc | doc | |
| 达梦 | doc | doc | |
| Cassandra | doc | doc | |
| ODPS | doc | doc | |
| HBase | doc | doc | |
| MongoDB | doc | doc | |
| Kudu | doc | doc | |
| ElasticSearch | doc | doc | |
| FTP | doc | doc | |
| HDFS | doc | doc | |
| Carbondata | doc | doc | |
| Redis | doc | ||
| Hive | doc | ||
| Stream Synchronization | Kafka | doc | doc |
| EMQX | doc | doc | |
| MySQL Binlog | doc | ||
| MongoDB Oplog | doc | ||
| PostgreSQL WAL | doc |
Documentation | Old Documentation
FlinkX is under the Apache 2.0 license. See the LICENSE file for details.