行动是治愈恐惧的良药,而犹豫、拖延将不断滋养恐惧。

logstash启动报错

Logstash | 作者 jhondom | 发布于2019年04月04日 | 阅读数:8941

新增系统日志接入elk时启动logstash报错!请老师指点!!!!
1.配置信息:
1)filebeat配置信息:
filebeat.inputs:
- type: log
  paths:
    - "/app/cspapps/logs/com-sinosafe-activecard/performance.log"
  encoding: UTF-8
  force_close_files: true
  tail_files: true
  ignore_older: 50m
  publish_async: true
  close_older: 15m
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
output.kafka:
  hosts: ["10.2.114.56:9092", "10.2.114.79:9092", "10.2.114.80:9092"]
  topic: 'activecard-performance'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: none
  max_message_bytes: 1000000
 
2)filebeat启动正常

2019-04-04T15:12:03.887+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [activecard-performance] from broker 10.2.114.56:9092
2019-04-04T15:12:03.888+0800 INFO kafka/log.go:53 kafka message: client/metadata found some partitions to be leaderless
2019-04-04T15:12:03.888+0800 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-04-04T15:12:03.888+0800 INFO [publish] pipeline/retry.go:168   done
2019-04-04T15:12:09.642+0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker 10.2.114.56:9092
 
3)logstash配置信息:
input {
      kafka{
        bootstrap_servers => ["10.2.114.56:9092,10.2.114.79:9092,10.2.114.80:9092"]
        auto_offset_reset => "latest"
        consumer_threads => 10
        topics => "activecard-performance"
        codec => "json"
      }
}
filter {
  mutate {
    gsub => [ "message", "\n", " "]
  }
  mutate {
    gsub => ["message", "\t", " "]
    remove_field => ["offset"]
  }
  grok { 
    match => {
         "message" =>"\[%{USERNAME:traceid},%{USERNAME:buinessid},[0-9a-zA-Z]+\] --- \[%{GREEDYDATA}\] :\[%{GREEDYDATA:uriPath}\]:\[%{USERNAME:methodName}\]:\[%{NUMBER:proceedTime}\]"
        }               
  }
  mutate {  
    convert => [ "proceedTime", "float"]         
  }
  mutate {
    remove_field => ["beat"]
    remove_field => ["ip"]
    remove_field => ["ipprospector.type"]
    remove_field => ["input.type"]
    remove_field => ["@version"]
  }  
  if "_grokparsefailure" in [tags]
  {
    mutate {remove_field => 'tags'} 
  }
  if [uriPath] == "*"
  {
    mutate {remove_field => 'uriPath'} 
  }
}
output {
    elasticsearch {
        hosts => ["10.2.114.67:9200","10.2.114.68:9200","10.2.114.69:9200","10.2.114.70:9200","10.2.114.71:9200","10.2.114.72:9200","10.2.114.73:9200","10.2.114.74:9200","10.2.114.75:9200","10.2.114.76:9200","10.2.114.77:9200","10.2.114.78:9200"]
        index => "activecard-performance-%{+YYYY.MM.dd}"
        }
}
 
4)启动logstash报错信息:
Sending Logstash logs to /app/logs/activecard.log which is now configured via log4j2.properties
[2019-04-04T14:59:55,979][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-04-04T14:59:56,488][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.3"}
[2019-04-04T14:59:58,601][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaNet::URISyntaxException", :message=>"Illegal character in scheme name at index 0: 1\n0.2.114.76:9200", :backtrace=>["java.net.URI$Parser.fail(java/net/URI.java:2848)", "java.net.URI$Parser.checkChars(java/net/URI.java:3021)", "java.net.URI$Parser.checkChar(java/net/URI.java:3031)", "java.net.URI$Parser.parse(java/net/URI.java:3047)", "java.net.URI.<init>(java/net/URI.java:588)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:278)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1001)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.util.safe_uri.initialize(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/util/safe_uri.rb:21)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.validate_value(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:513)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.block in process_parameter_value(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:328)", "org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2472)", "org.jruby.RubyArray.map(org/jruby/RubyArray.java:2486)", "org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.process_parameter_value(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:328)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.block in validate_check_parameter_values(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:351)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.block in validate_check_parameter_values(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:345)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1362)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1359)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:662)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:647)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1319)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1354)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1343)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.validate_check_parameter_values(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:344)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate_check_parameter_values$0$__VARARGS__(app/logstash/logstash_minus_6_dot_4_dot_3/logstash_minus_core/lib/logstash/config//app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.validate(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:234)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$validate$0$__VARARGS__(app/logstash/logstash_minus_6_dot_4_dot_3/logstash_minus_core/lib/logstash/config//app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.config_init(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb:85)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$config_init$0$__VARARGS__(app/logstash/logstash_minus_6_dot_4_dot_3/logstash_minus_core/lib/logstash/config//app/logstash/logstash-6.4.3/logstash-core/lib/logstash/config/mixin.rb)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.outputs.base.initialize(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/outputs/base.rb:60)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1001)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:908)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.initialize(org/logstash/config/ir/compiler/OutputStrategyExt.java:224)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.call(org/logstash/config/ir/compiler/OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:30)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:217)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:166)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.pipeline.plugin(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline.rb:71)", "RUBY.<eval>((eval):87)", "org.jruby.RubyKernel.evalCommon(org/jruby/RubyKernel.java:1027)", "org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(org/jruby/RubyKernel$INVOKER$s$0$3$eval19.gen)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.pipeline.initialize(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline.rb:49)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.pipeline.initialize(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.pipeline_action.create.execute(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline_action/create.rb:38)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(app/logstash/logstash_minus_6_dot_4_dot_3/logstash_minus_core/lib/logstash/pipeline_action//app/logstash/logstash-6.4.3/logstash-core/lib/logstash/pipeline_action/create.rb)", "app.logstash.logstash_minus_6_dot_4_dot_3.logstash_minus_core.lib.logstash.agent.block in converge_state(/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/agent.rb:309)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2019-04-04T14:59:58,658][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaNet::URISyntaxException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in `create'", "org/logstash/execution/ConvergeResultExt.java:34:in `add'", "/app/logstash/logstash-6.4.3/logstash-core/lib/logstash/agent.rb:321:in `block in converge_state'"]}
[2019-04-04T14:59:58,712][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
filebeat.jpg filebeat启动信息.jpg logstash报错1.jpg logstash报错.jpg
已邀请:

bellengao - 博客: https://www.jianshu.com/u/e0088e3e2127

赞同来自:

grok filter里,[]用两个斜杠\\进行转义试试

rochy - rochy_he

赞同来自:

看起来可能是   hosts => ["10.2.114.67:9200","10.2.114.68:9200","10.2.114.69:9200","10.2.114.70:9200","10.2.114.71:9200","10.2.114.72:9200","10.2.114.73:9200","10.2.114.74:9200","10.2.114.75:9200","10.2.114.76:9200","10.2.114.77:9200","10.2.114.78:9200"]
 
这部分配置是出现了换行符,导致 URL 解析错误,请该为:
  hosts => ["10.2.114.67:9200",
                    "10.2.114.68:9200",
                    "10.2.114.69:9200",
                    "10.2.114.70:9200",
                    "10.2.114.71:9200"]
这样的配置试试

jhondom

赞同来自:

谢谢,按照提供的方法已解决。

要回复问题请先登录注册