好的想法是十分钱一打,真正无价的是能够实现这些想法的人。

es配置的pipeline无法解析多行

Beats | 作者 nothing | 发布于2019年10月23日 | 阅读数:3747

流程: filebeat读取java日志->es中配置了pipeline预处理->根据模板写入es
问题: filebeat做了multiline配置,可以多行合并,但是经过pipeline的grok预处理以后就只能读取第一行了.如果清空pipeline的processors配置则一切正常.
 
这是我filebeat的配置
filebeat.inputs:
- type: log
enabled: true
paths:
- /Users/liule/Desktop/demo.log
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true

setup.ilm.enabled: false

output.elasticsearch:
hosts: ["localhost:9200"]
pipeline: "java_pipeline"
index: "iotrta-%{+yyyy.MM.dd}"

setup.template.name: "log-java"
setup.template.pattern: "log-java-*"

queue.mem:
events: 256
flush.min_events: 128

 这是我pipeline
QQ20191023-153944@2x.png

 
这是我template
QQ20191023-154122@2x.png

 
 
这是我的demo文件
2019-10-17 11:29:45.011  INFO 12365 --- [main] c.d.AbnormalInterruptTaskFixRunner       : dddddddddddddd
2019-10-17 11:30:45.011 INFO 12365 --- [main] c.d.AbnormalInterruptTaskFixRunner : test
2019-10-17 11:31:34.243 ERROR 12365 --- [XNIO-2 I/O-10] io.undertow.request.io : UT005090: Unexpected failure

java.dddddd.dddddddd: dddddddddddddd
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
at aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

2019-10-17 11:31:34.243 ERROR 12365 --- [XNIO-2 I/O-11] io.undertow.request.io : UT005090: Unexpected failure

java.lang.NoClassDefFoundError: Could not initialize class io.undertow.UndertowMessages
at io.undertow.server.DefaultByteBufferPool$DefaultPooledBuffer.getBuffer(DefaultByteBufferPool.java:260) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:833) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:648) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1127) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]

 
 
 
 
 
 
已邀请:

CononYc

赞同来自:

可以使用自定义的匹配表达式
参阅elasticsearch 官方文档 Ingest node ->grok-processor
表达式参阅:github.com/kkos/oniguruma/blob/master/doc/RE
匹配多行文本的表达式为(.m:.*)
在pipeline下配置
"pattern_definitions" : { "MULTI_LINE" : "(.m:.*)"}

使用
{
    "description": "...",
    "processors": [
        {
            "grok": {
                "field": "message",
                "patterns": [
                    "my dog is colored %{MULTI_LINE:msg}"
                ],
                "pattern_definitions": {
                    "MULTI_LINE": "(.m:.*)"
                }
            }
        }
    ]
}

要回复问题请先登录注册