filebeat采集数量大的日志出现延迟问题

发现当filebeat采集Windows服务器上的IIS日志时,出现延迟投递的问题。甚至有些日志要等待几天才会到ES中。
IIS日志量平均每台服务器一天有20GB左右。
filebeat配置如下:
#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- type: log

enabled: true

paths:
- D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\MessageTracking\*.LOG

tags: ["mtklog"]

tail_files: true
ignore_older: 7d
close_renamed: disabled

- type: log

enabled: true

paths:
- D:\inetpub\logs\LogFiles\W3SVC*\u_ex*

tags: ["iislog"]

tail_files: true
ignore_older: 7d
close_renamed: disabled

processors:
- drop_fields:
fields: ["@timestamp", "beat.name","beat.version"]


#============================= Filebeat modules ===============================

filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading
reload.enabled: false

# Period on which files under path should be checked for changes
#reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging


#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["xxx.xxx.xxx.xxx:9700", "
xxx.xxx.xxx.xxx:9700"]
loadbalance: true

通过filebeat监控可以看到filebeat传输数据很不稳定:

屏幕快照_2018-07-11_17.15_.38_.png

 
已邀请:

JElbert - es菜鸟选手

赞同来自:

日志量大一般都选传输给Kafka 然后再给Logstash 

zqc0512

赞同来自:

直接上logtash试试,解析日志内容。
延时几天肯定有问题的。

要回复问题请先登录注册