filebeat版本6.0,kafka版本0.11.0.1.
filebeat配置如下:
filebeat日志如下:
如果使用filebeat5.5的话,就一切正常.
---------------------------------------
补充
1,手动插入数据后,filebeat的那个日志是一直重复刷新的.上面只是节选一段.
2,filebeat下载的是6.0的tar包.解压后修改了yml,直接使用./filebeat启动的.
3,filebeat5.5可以正常的使用,主要是想试试6.0版本Filebeat prospectors中的document_type有没有被禁用.如果禁用了的话,topic: '%{[type]}'要如何获取文档类型.
filebeat配置如下:
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
- type: log
paths:
- /var/log/lhf
document_type: lhftcc
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ kafka =====================================
output.kafka:
# initial brokers for reading cluster metadata
hosts: ["test02:9092"]
# message topic selection + partitioning
topic: '%{[type]}'
partition.round_robin:
reachable_only: false
version: 0.10.0.0
required_acks: 1
compression: none
max_message_bytes: 1000000
启动之后,手动向/var/log/lhf插入数据.filebeat日志如下:
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send unwait-signal to consumer
2017-11-29T14:36:18+08:00 INFO done
2017-11-29T14:36:18+08:00 INFO retryer: send wait signal to consumer
在kafka上没有看到创建topic.如果使用filebeat5.5的话,就一切正常.
---------------------------------------
补充
1,手动插入数据后,filebeat的那个日志是一直重复刷新的.上面只是节选一段.
2,filebeat下载的是6.0的tar包.解压后修改了yml,直接使用./filebeat启动的.
3,filebeat5.5可以正常的使用,主要是想试试6.0版本Filebeat prospectors中的document_type有没有被禁用.如果禁用了的话,topic: '%{[type]}'要如何获取文档类型.
2 个回复
keyso
赞同来自: luohuanfeng
fields: log_topic: test
topic: '%{[fields.log_topic]}'
zhangmeng
赞同来自: