Skip to content

Commit

Permalink
Add Support for Sybase Anywhere (wgzhao#825)
Browse files Browse the repository at this point in the history
  • Loading branch information
@wgzhao
wgzhao authored Jun 3, 2023
1 parent 5e99c0d commit 3d92a45
Show file tree
Hide file tree
Showing 23 changed files with 973 additions and 1 deletion.
40 changes: 40 additions & 0 deletions docs/assets/jobs/sybasereader.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"job": {
"content": [
{
"reader": {
"name": "sybasereader",
"parameter": {
"column": [
"*"
],
"connection": [
{
"jdbcUrl": [
"jdbc:sybase:Tds:127.0.0.1:5000/master"
],
"table": [
"dbo.ijdbc_function_escapes"
]
}
],
"username": "sa",
"password": "password"
}
},
"writer": {
"name": "streamwriter",
"parameter": {
"print": "true"
}
}
}
],
"setting": {
"speed": {
"bytes": -1,
"channel": 1
}
}
}
}
61 changes: 61 additions & 0 deletions docs/assets/jobs/sybasewriter.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"job": {
"content": [
{
"writer": {
"name": "sybasewriter",
"parameter": {
"column": [
"id","name","salary","created_at", "updated_at"
],
"connection": [
{
"jdbcUrl": "jdbc:sybase:Tds:127.0.0.1:5000/master"
,
"table": [
"dbo.addax_writer"
]
}
],
"username": "sa",
"password": "password"
}
},
"reader": {
"name": "streamreader",
"parameter": {
"column": [
{
"random": "100,1000",
"type": "long"
},
{
"random": "10,100",
"type": "string"
},
{
"random": "10,1000",
"type": "double"
},
{
"incr":"2022-01-01 13:00:00,2,d",
"type":"date"
},
{
"incr":"2023-01-01 13:00:00,2,d",
"type":"date"
}
],
"sliceRecordCount": 100
}
}
}
],
"setting": {
"speed": {
"bytes": -1,
"channel": 1
}
}
}
}
163 changes: 163 additions & 0 deletions docs/assets/output/sybasereader.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
___ _ _
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\

:: Addax version :: (v4.1.1-SNAPSHOT)

2023-06-02 23:29:28.800 [ main] INFO VMInfo - VMInfo# operatingSystem class => com.sun.management.internal.OperatingSystemImpl
2023-06-02 23:29:28.819 [ main] INFO Engine -
{
"content":{
"reader":{
"name":"sybasereader",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":[
"jdbc:sybase:Tds:127.0.0.1:5000/master"
],
"table":[
"dbo.ijdbc_function_escapes"
]
}
],
"username":"sa",
"password":"*****"
}
},
"writer":{
"name":"streamwriter",
"parameter":{
"print":"true"
}
}
},
"setting":{
"speed":{
"bytes":-1,
"channel":1
}
}
}

2023-06-02 23:29:28.878 [ main] INFO JobContainer - The jobContainer begins to process the job.
2023-06-02 23:29:28.887 [ job-0] WARN SybaseReader$Job - The plugin(SybaseReader) need to setup fetchSize to improve performance.
2023-06-02 23:29:29.532 [ job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl [jdbc:sybase:Tds:127.0.0.1:5000/master].
2023-06-02 23:29:29.533 [ job-0] WARN OriginalConfPretreatmentUtil - There are some risks in the column configuration. Because you did not configure the columns to read the database table, changes in the number and types of fields in your table may affect the correctness of the task or even cause errors.
2023-06-02 23:29:29.537 [ job-0] INFO JobContainer - The Reader.Job [sybasereader] perform prepare work .
2023-06-02 23:29:29.537 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] perform prepare work .
2023-06-02 23:29:29.537 [ job-0] INFO JobContainer - Job set Channel-Number to 1 channel(s).
2023-06-02 23:29:29.541 [ job-0] INFO JobContainer - The Reader.Job [sybasereader] is divided into [1] task(s).
2023-06-02 23:29:29.541 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] is divided into [1] task(s).
2023-06-02 23:29:29.581 [ job-0] INFO JobContainer - The Scheduler launches [1] taskGroup(s).
2023-06-02 23:29:29.595 [ taskGroup-0] INFO TaskGroupContainer - The taskGroupId=[0] started [1] channels for [1] tasks.
2023-06-02 23:29:29.598 [ taskGroup-0] INFO Channel - The Channel set byte_speed_limit to -1, No bps activated.
2023-06-02 23:29:29.598 [ taskGroup-0] INFO Channel - The Channel set record_speed_limit to -1, No tps activated.
2023-06-02 23:29:29.609 [ reader-0-0] INFO CommonRdbmsReader$Task - Begin reading records by executing SQL query: [select * from dbo.ijdbc_function_escapes ].
2023-06-02 23:29:29.683 [ reader-0-0] INFO CommonRdbmsReader$Task - Finished reading records by executing SQL query: [select * from dbo.ijdbc_function_escapes ].
abs abs(%1)
acos acos(%1)
asin asin(%1)
atan atan(%1)
atan2 atn2(%1, %2)
ceiling ceiling(%1)
cos cos(%1)
cot cot(%1)
degrees degrees(%1)
exp exp(%1)
floor floor(%1)
log log(%1)
log10 log10(%1)
pi pi()
power power(%1, %2)
radians radians(%1)
rand rand(%1)
round round(%1, %2)
sign sign(%1)
sin sin(%1)
sqrt sqrt(%1)
tan tan(%1)
ascii ascii(%1)
char char(%1)
concat %1 + %2
difference difference(%1, %2)
insert stuff(%1, %2, %3, %4)
length char_length(%1)
lcase lower(%1)
ltrim ltrim(%1)
repeat replicate(%1, %2)
right right(%1, %2)
rtrim rtrim(%1)
soundex soundex(%1)
space space(%1)
substring substring(%1, %2, %3)
ucase upper(%1)
curdate getdate()
curtime getdate()
dayname datename(dw, %1)
dayofmonth datepart(dd, %1)
dayofweek datepart(dw, %1)
dayofyear datepart(dy, %1)
hour datepart(hh, %1)
minute datepart(mi, %1)
month datepart(mm, %1)
monthname datename(mm, %1)
now getdate()
quarter datepart(qq, %1)
second datepart(ss, %1)
timestampaddsql_tsi_second dateadd(ss, %2, %3)
timestampaddsql_tsi_minute dateadd(mi, %2, %3)
timestampaddsql_tsi_hour dateadd(hh, %2, %3)
timestampaddsql_tsi_day dateadd(dd, %2, %3)
timestampaddsql_tsi_week dateadd(wk, %2, %3)
timestampaddsql_tsi_month dateadd(mm, %2, %3)
timestampaddsql_tsi_quarter dateadd(qq, %2, %3)
timestampaddsql_tsi_year dateadd(yy, %2, %3)
timestampdiffsql_tsi_second datediff(ss, %2, %3)
timestampdiffsql_tsi_minute datediff(mi, %2, %3)
timestampdiffsql_tsi_hour datediff(hh, %2, %3)
timestampdiffsql_tsi_day datediff(dd, %2, %3)
timestampdiffsql_tsi_week datediff(wk, %2, %3)
timestampdiffsql_tsi_month datediff(mm, %2, %3)
timestampdiffsql_tsi_quarter datediff(qq, %2, %3)
timestampdiffsql_tsi_year datediff(yy, %2, %3)
week datepart(wk, %1)
year datepart(yy, %1)
database db_name()
ifnull isnull(%1, %2)
user user_name()
convertsql_binary convert(varbinary(255), %1)
convertsql_bit convert(bit, %1)
convertsql_char convert(varchar(255), %1)
convertsql_date convert(datetime, %1)
convertsql_decimal convert(decimal(36, 18), %1)
convertsql_double convert(float, %1)
convertsql_float convert(float, %1)
convertsql_integer convert(int, %1)
convertsql_longvarbinary convert(varbinary(255), %1)
convertsql_longvarchar convert(varchar(255), %1)
convertsql_real convert(real, %1)
convertsql_smallint convert(smallint, %1)
convertsql_time convert(datetime, %1)
convertsql_timestamp convert(datetime, %1)
convertsql_tinyint convert(tinyint, %1)
convertsql_varbinary convert(varbinary(255), %1)
convertsql_varchar convert(varchar(255), %1)
2023-06-02 23:29:32.625 [ job-0] INFO AbstractScheduler - The scheduler has completed all tasks.
2023-06-02 23:29:32.627 [ job-0] INFO JobContainer - The Writer.Job [streamwriter] perform post work.
2023-06-02 23:29:32.628 [ job-0] INFO JobContainer - The Reader.Job [sybasereader] perform post work.
2023-06-02 23:29:32.632 [ job-0] INFO StandAloneJobContainerCommunicator - Total 88 records, 2317 bytes | Speed 772B/s, 29 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.073s | Percentage 100.00%
2023-06-02 23:29:32.635 [ job-0] INFO JobContainer -
Job start at : 2023-06-02 23:29:28
Job end at : 2023-06-02 23:29:32
Job took secs : 3s
Average bps : 772B/s
Average rps : 29rec/s
Number of rec : 88
Failed record : 0
Binary file modified docs/images/supported_databases.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
65 changes: 65 additions & 0 deletions docs/reader/sybasereader.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Sybase Reader

SybaseReader 插件实现了从 [Sybase][1] 读取数据

## 示例

我们可以用 Docker 容器来启动一个 Sybase 数据库

```shell
docker run -tid --rm -h dksybase --name sybase -p 5000:5000 ifnazar/sybase_15_7 bash /sybase/start
```

下面的配置是读取该表到终端的作业:

=== "job/sybasereader.json"

```json
--8<-- "jobs/sybasereader.json"
```

将上述配置文件保存为 `job/sybase2stream.json`

### 执行采集命令

执行以下命令进行数据采集

```shell
bin/addax.sh job/sybase2stream.json
```

## 参数说明

| 配置项 | 是否必须 | 类型 | 默认值 | 描述 |
|:----------|:----:|--------|-------|------------------------------------------------------------------|
| jdbcUrl | 是 | list | 无 | 对端数据库的JDBC连接信息，jdbcUrl按照RDBMS官方规范. |
| driver | 否 | string | 无 | 自定义驱动类名，解决兼容性问题，详见下面描述 |
| username | 是 | string | 无 | 数据源的用户名 |
| password | 否 | string | 无 | 数据源指定用户名的密码 |
| table | 是 | list | 无 | 所选取的需要同步的表名,使用JSON数据格式，当配置为多张表时，用户自己需保证多张表是同一表结构 |
| column | 是 | list | 无 | 所配置的表中需要同步的列名集合，详细描述 [rdbmreader](../rdbmsreader) |
| splitPk | 否 | string | 无 | 使用splitPk代表的字段进行数据分片，详细描述见 [rdbmreader](../rdbmsreader) |
| autoPk | 否 | bool | false | 是否自动猜测分片主键，`3.2.6` 版本引入 |
| where | 否 | string | 无 | 针对表的筛选条件 |
| querySql | 否 | list | 无 | 使用自定义的SQL而不是指定表来获取数据，当配置了这一项之后，Addax系统就会忽略 `table`，`column`这些配置项 |
| fetchSize | 否 | int | 2048 | Sybase 要求配置 fetchSize 来提升性能 |


## 类型转换

目前 SybaseReader 支持大部分 Sybase 类型，但也存在部分个别类型没有支持的情况，请注意检查你的类型。

下面列出类型转换列表:


| Addax 内部类型 | SqlServer 数据类型 |
|------------|-------------------------------------------------------------------|
| Long | bigint, int, smallint, tinyint |
| Double | float, decimal, real, numeric |
| String | char,nchar,ntext,nvarchar,text,varchar,nvarchar(MAX),varchar(MAX) |
| Date | date, datetime, time |
| Boolean | bit |
| Bytes | binary,varbinary,varbinary(MAX),timestamp, image |


[1]: https://www.sap.com/products/technology-platform/sql-anywhere.html
55 changes: 55 additions & 0 deletions docs/writer/sybasewriter.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Sybase Writer

SybaseWriter 插件实现了写入数据到 [Sybase][1] 库表的功能。

## 配置样例

我们可以用 Docker 容器来启动一个 Sybase 数据库

```shell
docker run -tid --rm -h dksybase --name sybase -p 5000:5000 ifnazar/sybase_15_7 bash /sybase/start
```

然后创建一张如下表

```sql
create table addax_writer
(
id int,
name varchar(255),
salary float(2),
created_at datetime,
updated_at datetime
);
```

再使用下面的任务配置文件

```json
--8<-- "jobs/sybasewriter.json"
```

## 参数说明

| 配置项 | 是否必须 | 默认值 | 描述 |
| :-------- | :------: | ------ |--------------------------------------------------|
| jdbcUrl | 是 | 无 | 对端数据库的JDBC连接信息，jdbcUrl按照RDBMS官方规范，并可以填写连接附件控制信息 |
| username | 是 | 无 | 数据源的用户名 |
| password | 否 | 无 | 数据源指定用户名的密码 |
| writeMode | 否 | insert | 写入方式，支持 insert， update，详见下文 |
| table | 是 | 无 | 所选取的需要同步的表名,使用JSON数据格式，当配置为多张表时，用户自己需保证多张表是同一表结构 |
| column | 是 | 无 | 所配置的表中需要同步的列名集合，详细描述见 [rdbmswriter][2] |
| splitPk | 否 | 无 | 使用splitPk代表的字段进行数据分片，详细描述见 [rdbmsreader][3] |
| preSql | 否 | 无 | 数据写入前先执行的sql语句 |
| postSql | 否 | 无 | 数据写入完成后,再执行的SQL语句 |
| batchSize | 否 | 1024 | 定义了插件和数据库服务器端每次批量数据获取条数，调高该值可能导致 Addax 出现OOM |
| session | 否 | 无 | 针对本地连接,修改会话配置 |


## 类型转换

类似 SqlServerReader ，目前 SqlServerWriter 支持大部分 SqlServer 类型，但也存在部分个别类型没有支持的情况，请注意检查你的类型。

[1]: https://www.sap.com/products/technology-platform/sql-anywhere.html
[2]: ../rdbmswriter
[3]: ../../reader/rdbmsreader
Loading

0 comments on commit 3d92a45

Please sign in to comment.