我们产线上使用 logstash-input-file 从文件抓数据,然后通过logstash-output-kafka 将数据发送到kafka,现在就是output线程使用CPU很高。
1. 环境: CT7.
2. 版本: Logstash 5.6.9 AND logstash-output-kafka 4.0.5
3. TOP的结果:
两个worker消耗CPU
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3714 root 20 0 4253288 769760 16332 R 86.7 4.7 3324:30 [main]>worker0
3715 root 20 0 4253288 769760 16332 R 86.7 4.7 3320:29 [main]>worker1
4. thread dump 结果(看起来是卡在发送的encode阶段)
"[main]>worker0" #33 daemon prio=5 os_prio=0 tid=0x00007f344021e000 nid=0xe82 runnable [0x00007f344e92f000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:304)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:918)
at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:43)
at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:453)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:430)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:353)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:451)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:312)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at rubyjit.LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969.chained_1_rescue_2$RUBY$SYNTHETIC__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.11/lib/logstash/outputs/kafka.rb:258)
at rubyjit.LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969.block_0$RUBY$__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.11/lib/logstash/outputs/kafka.rb:256)
at rubyjit$LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969$block_0$RUBY$__file__.call(rubyjit$LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969$block_0$RUBY$__file__)
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)
5. 文件编码
[root@vssj1cal001 logs]# file vssj1cal001.webex.com_access_log.2019-08-29.txt
vssj1cal001.webex.com_access_log.2019-08-29.txt: ASCII text, with very long lines
现在我知道问题可能是数据的编码问题,但是我不知道该如何排查,我检查了读取文件的除了有一些大的event之外,还没看出来有啥问题,请教一下我改如何排查这个问题,谢谢!
1. 环境: CT7.
2. 版本: Logstash 5.6.9 AND logstash-output-kafka 4.0.5
3. TOP的结果:
两个worker消耗CPU
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3714 root 20 0 4253288 769760 16332 R 86.7 4.7 3324:30 [main]>worker0
3715 root 20 0 4253288 769760 16332 R 86.7 4.7 3320:29 [main]>worker1
4. thread dump 结果(看起来是卡在发送的encode阶段)
"[main]>worker0" #33 daemon prio=5 os_prio=0 tid=0x00007f344021e000 nid=0xe82 runnable [0x00007f344e92f000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:304)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:918)
at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:43)
at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:453)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:430)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:353)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:451)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:312)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at rubyjit.LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969.chained_1_rescue_2$RUBY$SYNTHETIC__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.11/lib/logstash/outputs/kafka.rb:258)
at rubyjit.LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969.block_0$RUBY$__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.11/lib/logstash/outputs/kafka.rb:256)
at rubyjit$LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969$block_0$RUBY$__file__.call(rubyjit$LogStash::Outputs::Kafka$$retrying_send_01d07881566e5bd041a791dccd6743425c81cea8589431969$block_0$RUBY$__file__)
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)
5. 文件编码
[root@vssj1cal001 logs]# file vssj1cal001.webex.com_access_log.2019-08-29.txt
vssj1cal001.webex.com_access_log.2019-08-29.txt: ASCII text, with very long lines
现在我知道问题可能是数据的编码问题,但是我不知道该如何排查,我检查了读取文件的除了有一些大的event之外,还没看出来有啥问题,请教一下我改如何排查这个问题,谢谢!
1 个回复
laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net
赞同来自:
是不是又复杂写入,当前机器的资源(CPU、线程池、队列)处理能力有限导致的。
核实一下。