架构filebeat+kafka+elk
filbeat配置
filebeat警告信息
filbeat配置
#------------------------------- Kafka output ----------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["kfk1:9092", "kfk2:9092"]
# message topic selection + partitioning
#topic: '%{[fields.log_topic]}'
topic: 'citymain77'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
filebeat警告信息
2018-03-02T09:53:31+08:00 INFO Harvester started for file: /app/log/citymain.log.2017-11-10.log
2018-03-02T09:53:31+08:00 INFO Harvester started for file: /app/log/citymain.log.2017-12-04.log
2018-03-02T09:53:32+08:00 WARN kafka message: Initializing new client
2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for all topics from broker 10.25.80.146:9092
2018-03-02T09:53:32+08:00 WARN Connected to broker at 10.25.80.146:9092 (unregistered)
2018-03-02T09:53:32+08:00 WARN client/brokers registered new broker #232 at szd-l0086693.siapp.h:9092
2018-03-02T09:53:32+08:00 WARN client/brokers registered new broker #233 at szd-l0086692.siapp.h:9092
2018-03-02T09:53:32+08:00 WARN kafka message: Successfully initialized new client
2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for [citymain231] from broker 10.25.80.146:9092
2018-03-02T09:53:32+08:00 WARN kafka message: client/metadata found some partitions to be leaderless
2018-03-02T09:53:32+08:00 WARN client/metadata retrying after 250ms... (3 attempts remaining)
2018-03-02T09:53:32+08:00 WARN client/metadata fetching metadata for [citymain231] from broker 10.25.80.146:9092
2018-03-02T09:53:32+08:00 WARN producer/broker/233 starting up
2018-03-02T09:53:32+08:00 WARN producer/broker/233 state change to [open] on citymain231/0
2018-03-02T09:53:32+08:00 WARN Connected to broker at szd-l0086692.siapp.h:9092 (registered as #233)
3 个回复
locatelli
赞同来自:
-- 'describe'你的topic,看是不是有partition没有leader
-- 如果所有partition都正常,下一步排除通信问题,从客户端到broker的通信时间如果太长也有可能会造成类似的错误。比如你的bootstrap server写的是kfk1,log里显示的是szd-xxxx,应该尽量使用broker的FQDN
Lincoln - 80s
赞同来自:
--logsatsh日志
zhangmeng
赞同来自: