ELK,萌萌哒

filebeat 与logstash 和 kafka 的连接问题

Beats | 作者 Jokers | 发布于2020年06月25日 | 阅读数:180

社区有没有大佬,我不管怎么配置 filebeat 将日志输出到 logstash 还是 kakfa 他都不会去进行连接。我也不知道哪里出问题了。求社区帮忙看看。
filebeat 设置
```
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/containers/qbjc-background*.log
fields:
type: qbjc-background
output.kafka:
enabled: true
hosts: ["kakfaip:kakfaport"]
topic: beats
```
kafka 的topic 已经创建。
filebeat的启动日志:
``
2020-06-23T14:42:51.582+0800 INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.4.2
2020-06-23T14:42:51.583+0800 INFO [publisher] pipeline/module.go:97 Beat name: node50
2020-06-23T14:42:51.584+0800 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-23T14:42:51.584+0800 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-06-23T14:42:51.584+0800 INFO instance/beat.go:422 filebeat start running.
2020-06-23T14:42:51.585+0800 INFO registrar/registrar.go:145 Loading registrar data from /usr/local/filebeat-7.4.2-linux-x86_64/data/registry/filebeat/data.json
2020-06-23T14:42:51.585+0800 INFO registrar/registrar.go:152 States Loaded from registrar: 1
2020-06-23T14:42:51.585+0800 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-06-23T14:42:51.585+0800 INFO crawler/crawler.go:72 Loading Inputs: 1
2020-06-23T14:42:51.586+0800 INFO log/input.go:152 Configured paths: [/var/log/containers/eim-background-797bd9ff7c-qtblz_default_eim-background-1a0d2764d8d596cd80de88b7b118befe1fd6daf5c372ad15981c5f31851ad879.log]
2020-06-23T14:42:51.586+0800 INFO input/input.go:114 Starting input of type: log; ID: 11516756834528010919
2020-06-23T14:42:51.586+0800 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-06-23T14:42:51.586+0800 INFO cfgfile/reload.go:171 Config reloader started
2020-06-23T14:42:51.586+0800 INFO cfgfile/reload.go:226 Loading of config files completed.
2020-06-23T14:42:54.573+0800 INFO add_cloud_metadata/add_cloud_metadata.go:87 add_cloud_metadata: hosting provider type not detected.
2020-06-23T14:43:21.588+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":37}},"total":{"ticks":90,"time":{"ms":102},"value":0},"user":{"ticks":60,"time":{"ms":65}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":30045}},"memstats":{"gc_next":8101936,"memory_alloc":4408160,"memory_total":13384248,"rss":55906304},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"kafka"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":40},"load":{"1":0.34,"15":1.52,"5":1.06,"norm":{"1":0.0085,"15":0.038,"5":0.0265}}}}}}
2020-06-23T14:43:51.587+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":4}},"total":{"ticks":100,"time":{"ms":8},"value":100},"user":{"ticks":60,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":60044}},"memstats":{"gc_next":8101936,"memory_alloc":4623008,"memory_total":13599096},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.21,"15":1.47,"5":0.96,"norm":{"1":0.0052,"15":0.0368,"5":0.024}}}}}}
2020-06-23T14:44:21.587+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":5}},"total":{"ticks":110,"time":{"ms":6},"value":110},"user":{"ticks":70,"time":{"ms":1}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":90044}},"memstats":{"gc_next":8101936,"memory_alloc":4922904,"memory_total":13898992},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.09,"15":1.49,"5":1.06,"norm":{"1":0.0273,"15":0.0373,"5":0.0265}}}}}}
2020-06-23T14:44:51.588+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":2}},"total":{"ticks":120,"time":{"ms":18},"value":120},"user":{"ticks":80,"time":{"ms":16}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":120046}},"memstats":{"gc_next":8120656,"memory_alloc":4204176,"memory_total":14304296},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":2.35,"15":1.57,"5":1.35,"norm":{"1":0.0588,"15":0.0393,"5":0.0338}}}}}}
2020-06-23T14:45:21.587+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":2}},"total":{"ticks":120,"time":{"ms":6},"value":120},"user":{"ticks":80,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":150045}},"memstats":{"gc_next":8120656,"memory_alloc":4540936,"memory_total":14641056},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.49,"15":1.52,"5":1.24,"norm":{"1":0.0373,"15":0.038,"5":0.031}}}}}}
2020-06-23T14:45:51.587+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":6}},"total":{"ticks":140,"time":{"ms":6},"value":140},"user":{"ticks":90}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":180046}},"memstats":{"gc_next":8120656,"memory_alloc":4673920,"memory_total":14774040},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.32,"15":1.51,"5":1.21,"norm":{"1":0.033,"15":0.0378,"5":0.0303}}}}}}
2020-06-23T14:46:21.587+0800 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":2}},"total":{"ticks":140,"time":{"ms":6},"value":140},"user":{"ticks":90,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"a5765668-8322-4dca-a40d-7f4436b8e430","uptime":{"ms":210043}},"memstats":{"gc_next":8120656,"memory_alloc":4982232,"memory_total":15082352},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.1,"15":1.49,"5":1.18,"norm":{"1":0.0275,"15":0.0373,"5":0.0295}}}}}}
```
已邀请:

tacsklet - 公司有用到es

赞同来自:

启动也没有报错,应该是连接上了,可以检查一下设置的路径下能不能匹配到日志文件,再看看kafka里面有没有相关消息和错误日志

要回复问题请先登录注册