使用 shuf 来打乱一个文件中的行或是选择文件中一个随机的行。

logstash启动报错问题

Logstash | 作者 jhondom | 发布于2019年04月08日 | 阅读数:3094

logstash获取kafka信息报错,请各位老师指导!
1.filebeat配置文件:
filebeat.inputs:
- type: log
paths:
- "/app/cspapps/logs/performance.log"
encoding: UTF-8
force_close_files: true
tail_files: true
ignore_older: 50m
publish_async: true
close_older: 15m
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
output.kafka:
hosts: ["10.2.110.56:9092", "10.2.110.79:9092", "10.2.110.80:9092"]
topic: "activecard-performance"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: none
max_message_bytes: 1000000

2.logstash配置文件:
input {
kafka{
bootstrap_servers => ["10.2.110.56:9092,10.2.110.79:9092,10.2.110.80:9092"]
auto_offset_reset => "latest"
consumer_threads => 10
topics => "activecard-performance"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["10.2.110.67:9200",
"10.2.110.74:9200",
"10.2.110.75:9200",
"10.2.110.76:9200"]
index => "activecard-performance-%{+YYYY.MM.dd}"
}
}

3.启动logstash报错信息:
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-2, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-5, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,495][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-7, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,499][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-3, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,504][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-8, groupId=logstash] Error while fetching metadata with correlation id 10 : {activecard-pe
rformance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,505][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-1, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-6, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
[2019-04-08T09:09:27,506][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-4, groupId=logstash] Error while fetching metadata with correlation id 9 : {activecard-per
formance=UNKNOWN_TOPIC_OR_PARTITION}
已邀请:

rochy - rochy_he

赞同来自:

看报错信息 ​UNKNOWN_TOPIC_OR_PARTITION 应该是说没有这个 topic 或者这个 topic 里面没有数据
你可以检查一下

jhondom

赞同来自:

谢谢,已经解决。重新创建topic后正常了。

要回复问题请先登录注册