愚者求师之过,智者从师之长。

filebeat 重复发送日志到kafka

Beats | 作者 chachabusi | 发布于2019年05月03日 | 阅读数:3389

filebeat 重复发送日志到kafka   发生这个情况的时候 filebeat 都会报如下的错误
百度了一些会发生重复发送日志的情况,但是都和我的情况不太符合  比如日志切割的时候名字重复之类的,似乎看起filebeat 在不停的和kafka断开建立连接
 
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-monitor/0
2019-05-02T23:26:46+08:00 WARN Connected to broker at 172.25.81.126:9092 (registered as #7)
2019-05-02T23:26:46+08:00 WARN producer/broker/2 starting up
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-monitor/4
2019-05-02T23:26:46+08:00 WARN Connected to broker at 172.25.81.127:9092 (registered as #8)
2019-05-02T23:26:46+08:00 WARN producer/broker/9 starting up
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-monitor/2
2019-05-02T23:26:46+08:00 WARN producer/broker/1 starting up
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-monitor/3
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-server/2
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-server/0
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-server/1
2019-05-02T23:26:46+08:00 WARN producer/broker/8 starting up
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-monitor/1
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-server/7
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-server/8
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-server/9
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-server/10
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-server/11
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-server/16
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-server/17
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-server/18
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-server/20
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-server/19
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-server/25
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-monitor/8
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-monitor/9
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-server/27
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-server/26
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-server/28
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-server/29
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-monitor/10
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-monitor/11
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-monitor/12
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-monitor/17
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-monitor/18
2019-05-02T23:26:46+08:00 WARN producer/broker/9 state change to [open] on ec-kfc-delivery-monitor/19
2019-05-02T23:26:46+08:00 WARN producer/broker/1 state change to [open] on ec-kfc-delivery-monitor/20
2019-05-02T23:26:46+08:00 WARN producer/broker/2 state change to [open] on ec-kfc-delivery-monitor/21
2019-05-02T23:26:46+08:00 WARN producer/broker/3 starting up
2019-05-02T23:26:46+08:00 WARN producer/broker/7 state change to [open] on ec-kfc-delivery-monitor/26
2019-05-02T23:26:46+08:00 WARN producer/broker/3 state change to [open] on ec-kfc-delivery-server/3
2019-05-02T23:26:46+08:00 WARN producer/broker/3 state change to [open] on ec-kfc-delivery-server/12
2019-05-02T23:26:46+08:00 WARN producer/broker/3 state change to [open] on ec-kfc-delivery-server/21
2019-05-02T23:26:46+08:00 WARN producer/broker/8 state change to [open] on ec-kfc-delivery-monitor/27
2019-05-02T23:26:46+08:00 WARN producer/broker/3 state change to [open] on ec-kfc-delivery-monitor/13
 
似乎和我下面这个链接里的情况有点相似
 
已邀请:

sa_linux - https://ops-coffee.cn

赞同来自:

如果filebeat在处理发送事件时还没有等到output的响应就意外关闭或僵死了,新发送的日志状态没有来得及记录进registry文件中,那么在filebeat重新启动后会去读取registry文件记录的信息重新发送日志,这确保了所有日志都被发送过,但可能会有重复的日志被发送

更详细的解释参考这个:
https://mp.weixin.qq.com/s/H5QiZ10KSdUXFW5LdH79wA

要回复问题请先登录注册