Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

index名似乎不能带*号 #2

Open
john5480 opened this issue Jun 25, 2021 · 0 comments
Open

index名似乎不能带*号 #2

john5480 opened this issue Jun 25, 2021 · 0 comments

Comments

@john5480
Copy link

试执行时,似乎index带了*号就报错了
报错内容:



2021-06-25 10:20:09.493 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2021-06-25 10:20:09.494 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-06-25 10:20:09.494 [main] INFO  JobContainer - DataX jobContainer starts job.
2021-06-25 10:20:09.496 [main] INFO  JobContainer - Set jobId = 0
2021-06-25 10:20:09.507 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2021-06-25 10:20:09.508 [job-0] INFO  JobContainer - DataX Reader.Job [elasticsearchreader] do prepare work .
2021-06-25 10:20:09.836 [job-0] INFO  ESClient - ======= RestHighLevelClient 初始化成功 =======
2021-06-25 10:20:09.837 [job-0] INFO  EsReader$Job - index:[netdevsyslog_syslogcs], type:[log]
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
2021-06-25 10:20:10.018 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[ESReader-02], Description:[index不存在.].  - ElasticsearchStatusException[method [HEAD], host [http://18.100.16.125:9200], URI [/netdevsyslog_syslogcs?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 400 Bad Request]]; nested: ResponseException[method [HEAD], host [http://18.100.16.125:9200], URI [/netdevsyslog_syslogcs?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 400 Bad Request]];
        at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.plugin.reader.elasticsearchreader.EsReader$Job.prepare(EsReader.java:68) ~[elasticsearchreader-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.JobContainer.prepareJobReader(JobContainer.java:715) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:308) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
        at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
2021-06-25 10:20:10.021 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2021-06-25 10:20:10.021 [job-0] INFO  EsReader$Job - ============elasticsearch reader job destroy=================
2021-06-25 10:20:10.022 [job-0] ERROR Engine - 

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[ESReader-02], Description:[index不存在.].  - ElasticsearchStatusException[method [HEAD], host [http://18.100.16.125:9200], URI [/netdevsyslog_syslogcs?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 400 Bad Request]]; nested: ResponseException[method [HEAD], host [http://18.100.16.125:9200], URI [/netdevsyslog_syslogcs?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 400 Bad Request]];
        at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
        at com.alibaba.datax.plugin.reader.elasticsearchreader.EsReader$Job.prepare(EsReader.java:68)
        at com.alibaba.datax.core.job.JobContainer.prepareJobReader(JobContainer.java:715)
        at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:308)
        at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)
        at com.alibaba.datax.core.Engine.start(Engine.java:92)
        at com.alibaba.datax.core.Engine.entry(Engine.java:171)
        at com.alibaba.datax.core.Engine.main(Engine.java:204)

json文件

{
  "job": {
    "setting": {
      "speed": {
        "channel": 5,
        "byte": 10485760
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "elasticsearchreader",
          "parameter": {
            "endpoints": "10.0.0.1:9200",
            "accessId": "xxxx",
            "accessKey": "xxxx",
            "index": "syslogcs*",
            "type": "log",
            "searchType": "dfs_query_then_fetch",
            "headers": {},
            "scroll": "3m",
            "search": [
             {
	"from": 0,
	"query": {
		"bool": {
			"must": [
				{
					"bool": {
						"must_not": [
							{
								"bool": {
									"must_not": [
										{
											"exists": {
												"field": "identifier",
												"boost": 1
											}
										}
									],
									"disable_coord": false,
									"adjust_pure_negative": true,
									"boost": 1
								}
							}
						],
						"disable_coord": false,
						"adjust_pure_negative": true,
						"boost": 1
					}
				},
				{
					"match_phrase": {
						"maprules": {
							"query": "rule_21",
							"slop": 0,
							"boost": 1
						}
					}
				}
			],
			"disable_coord": false,
			"adjust_pure_negative": true,
			"boost": 1
		}
	},
	"aggregations": {
		"@ip": {
			"terms": {
				"size": 10000,
				"field": "@ip",
				"min_doc_count": 1,
				"order": [
					{
						"_count": "desc"
					},
					{
						"_term": "asc"
					}
				],
				"show_term_doc_count_error": false,
				"shard_min_doc_count": 0
			},
			"aggregations": {
				"date_histogram_timestamp": {
					"date_histogram": {
						"format": "yyyy-MM-dd HH:mm:ss",
						"keyed": false,
						"field": "timestamp",
						"min_doc_count": 0,
						"interval": "1d",
						"offset": 0,
						"order": {
							"_key": "asc"
						},
						"time_zone": "Asia/Shanghai"
					},
					"aggregations": {
"count": {
							"value_count": {
								"field": "_index"
							}
						}
					}
				}
			}
		}
	}
}
            ],
            "table": {
              "name": "netdevsyslog_syslogcs*",
              "column": [
                {
                  "name": "@timestamp"
                },
                {
                  "name": "@sourceip"
                },
                {
                  "name": "@message"
                }
              ]
            }
          }
        },
        "writer": {
          "name": "streamwriter",
          "parameter": {
"path": "/tmp/out",
            "fileName":"new-inflxudb.csv",
            "print": true,
            "encoding": "UTF-8"
          }
        }
      }
    ]
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant