要不要也来分享分享一下啊

filebeat如何优化?

匿名 | 发布于2018年05月16日 | 阅读数:8894

服务器配置
4vCore  8G内存 上面运行了java服务和filebeat
filebeat版本6.1.2

流程
filebeat -> kafka -> logstash output file (kafka , logstash 和 filebeat 在不同的机器上.)

业务繁忙时 ,日志量也比较大,用top看filebeat进程占用cpu非常的高.想问大家filebeat如何优化.
filebeat上 日志获取的频率不需要太快.


下面是之前使用的filebeat配置:
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log
paths:
- /logs/wx/*.log
fields:
log_topic: wx
document_type: wx
#================================ Outputs =====================================
#----------------------------- kafka ---------------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["e01:9092", "e02:9092", "e03:9092"]

# message topic selection + partitioning
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false

version: 0.10.0.0
required_acks: 1
compression: none
max_message_bytes: 1000000



下面是看过文档自己刚修改后的配置:
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- input_type: log
paths:
- /logs/wx/*.log
fields:
log_topic: wx
document_type: wx
scan_frequency: 60s
backoff: 10s
#================================ General ======================================
queue:
mem:
events: 8192
flush.min_events: 12288
fulsh.timeout: 60s


#================================ Outputs =====================================
#----------------------------- kafka ---------------------------------------
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["e01:9092", "e02:9092", "e03:9092"]

# message topic selection + partitioning
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false

version: 0.10.0.0
required_acks: 1
compression: none
max_message_bytes: 1000000

还有那些参数可以优化吗? 另外 我这种写法不知道有没有问题, 重启filebeat的时候是没有报错的,不知道有没有生效,有什么方法能看到filebeat启动后各项参数的值吗?
 
已邀请:

apeletMarshal - 小猿帅

赞同来自:

queue.mem.events的值要大于queue.mem.flush.min_events

要回复问题请先登录注册