行动是治愈恐惧的良药,而犹豫、拖延将不断滋养恐惧。

logstash过滤分析nginx日志,启动报错

Logstash | 作者 lucky_girl | 发布于2017年12月15日 | 阅读数:9530

logstash从filebeat收集日志,版本都是6.0.1
报错信息如下:
[2017-12-15T11:32:45,692][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion of nil into String", "backtrace"=>["org/jruby/RubyString.java:3370:in `include?'", "(eval):143:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):141:in `block in initialize'", "(eval):127:in `block in filter_func'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:501:in `filter_batch'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:477:in `worker_loop'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:439:in `block in start_workers'"], :thread=>"#<Thread:0x48607bcb@/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:290 sleep>"}
[2017-12-15T11:32:45,692][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<TypeError: no implicit conversion of nil into String>, :backtrace=>["org/jruby/RubyString.java:3370:in `include?'", "(eval):143:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):141:in `block in initialize'", "(eval):127:in `block in filter_func'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:501:in `filter_batch'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:477:in `worker_loop'", "/usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:439:in `block in start_workers'"]}
TypeError: no implicit conversion of nil into String
                include? at org/jruby/RubyString.java:3370
     block in initialize at (eval):143
                    each at org/jruby/RubyArray.java:1734
     block in initialize at (eval):141
    block in filter_func at (eval):127
            filter_batch at /usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:501
             worker_loop at /usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:477
  block in start_workers at /usr/local/logstash/logstash-core/lib/logstash/pipeline.rb:439
已邀请:

cqlray

赞同来自: lucky_girl

不知道你之前有没有启动正常过,所以我建议你先把这个文件拷贝到其他目录去,不要放到这个logstash目录下,比如~目录,然后重启logstash看有问题问题。 
因为我安装时出现根本启动不了的问题,我贴一个之前碰到的问题和解决办法给你看看:
 
如果显示logstash.service holdoff time over, scheduling restart.那么看一下运行时间,如果显示Active: active (running) since 二 2017-12-12 01:47:04 UTC; 8s ago,显示时间小于36秒,那么半分钟后查询,如果时间还是小于36秒,那么通过如下命令查看日志信息:
$ tail -n 50 /var/log/logstash/logstash-plain.log
查看日志最后50行信息,如果提示如下信息:
[2017-12-12T01:32:27,299][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>,
那么就需要使用命令将文件夹权限付给logstash,命令如下:
$ sudo chown -R logstash: /var/lib/logstash/
执行完毕后重新启动,重新查看运行状态是否正常。

cqlray

赞同来自:

你先执行命令检查一下配置是否正确。我安装6.0.1时也发现以前的filter不能用了,从源码里边搜索nginx,能搜到一个可用filter。

lucky_girl - 95 IT女

赞同来自:

检查配置文件是正确的
 

cqlray

赞同来自:

我查了一下记录的文档,执行下面命令:
$ sudo /usr/share/logstash/bin/logstash --config.test_and_exit --path.settings /etc/logstash/
如果显示Configuration OK则表示没有任何语法错误。
 如果还是出错,就只能查看日志报什么错啦:
tail -n 50 /var/log/logstash/logstash-plain.log

tail -n 50 /var/log/logstash/logstash-plain.log | grep error

 

lucky_girl - 95 IT女

赞同来自:

嗯,这个是最后50行的错误

cqlray

赞同来自:

执行上面的命令检查一下看看。

cqlray

赞同来自:

参考一下源代码里边的配置:https://github.com/elastic/log ... .conf

lucky_girl - 95 IT女

赞同来自:

错误看不懂耶,/usr/local/logstash/config/pipelines.yml,这个文件是有内容的,没有改动过

lucky_girl - 95 IT女

赞同来自:

好的,谢谢你耐心的解答,我看看

cqlray

赞同来自:

没事,我也正好刚踩过坑

lucky_girl - 95 IT女

赞同来自:

其实我感觉是版本的原因,6.0.1里面type听说是被废除了的,不能这么用,但是我不知道怎么引用
下面的图,第一张是我filebeat的配置
第二张是logstash的配置,从filebeat收集,根据document_type的名字来命名elasticsearch的名字
 

cqlray

赞同来自:

document _type不会让系统起不来,我看了一下日志,我的测试环境没改,只是警告,如下:
[WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature

要回复问题请先登录注册