logstash获取kafka信息报错,请各位老师指导!
1.filebeat配置文件:
filebeat.inputs:
- type: log
paths:
- "/app/cspapps/logs/performance.log"
encoding: UTF-8
force_close_files: true
tail_files: true
ignore_older: 50m
publish_async: true
close_older: 15m
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
output.kafka:
hosts: ["10.2.110.56:9092", "10.2.110.79:9092", "10.2.110.80:9092"]
topic: "activecard-performance"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: none
max_message_bytes: 1000000
2.logstash配置文件:
input {
kafka{
bootstrap_servers => ["10.2.110.56:9092,10.2.110.79:9092,10.2.110.80:9092"]
auto_offset_reset => "latest"
consumer_threads => 10
topics => "activecard-performance"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.2.110.67:9200",
"10.2.110.74:9200",
"10.2.110.75:9200",
"10.2.110.76:9200"]
index => "activecard-performance-%{+YYYY.MM.dd}"
}
}
3.启动logstash报错信息:
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-2, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-5, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-7, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-3, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,504][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-8, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,505][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-1, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-6, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-4, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
1.filebeat配置文件:
filebeat.inputs:
- type: log
paths:
- "/app/cspapps/logs/performance.log"
encoding: UTF-8
force_close_files: true
tail_files: true
ignore_older: 50m
publish_async: true
close_older: 15m
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
output.kafka:
hosts: ["10.2.110.56:9092", "10.2.110.79:9092", "10.2.110.80:9092"]
topic: "activecard-performance"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: none
max_message_bytes: 1000000
2.logstash配置文件:
input {
kafka{
bootstrap_servers => ["10.2.110.56:9092,10.2.110.79:9092,10.2.110.80:9092"]
auto_offset_reset => "latest"
consumer_threads => 10
topics => "activecard-performance"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.2.110.67:9200",
"10.2.110.74:9200",
"10.2.110.75:9200",
"10.2.110.76:9200"]
index => "activecard-performance-%{+YYYY.MM.dd}"
}
}
3.启动logstash报错信息:
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-2, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-5, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-7, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-3, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,504][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-8, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,505][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-1, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-6, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-4, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
2 个回复
rochy - rochy_he
赞同来自:
你可以检查一下
jhondom
赞同来自: