愚者求师之过,智者从师之长。

logstash filter split with \n

Logstash | 作者 ajaxhe | 发布于2016年04月10日 | 阅读数:5310

背景:
kafka中一个message中存在多行的情况,因此我需要将从kafka中一条消息在logstash中裂变成多条记录,logstash配置如下:
input {
#stdin { }
kafka {
zk_connect => '127.0.0.1:2181'
group_id => "logstash"
#topic_id => "bestv_access_log"
topic_id => "test"
reset_beginning => false
#reset_beginning => true
consumer_threads => 5
decorate_events => true
codec => plain
}
}

filter {
split {
field => "message"
terminator => "\n"
}
}

output {
stdout {
codec => rubydebug
}
}
测试producer python代码如下:
with topic.get_producer() as producer:  # Create Kafka producer on the given topic
while True:
#msg = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
msg = ''
msg += 'ajaxhe\nanother ajaxhe'
print msg
producer.produce(msg) # Send the message to Kafka
time.sleep(5)
但结果却不符合预期:
{
"message" => "ajaxhe\nanother ajaxhe",
"@version" => "1",
"@timestamp" => "2016-04-10T12:24:38.963Z",
"kafka" => {
"msg_size" => 21,
"topic" => "test",
"consumer_group" => "logstash",
"partition" => 1,
"offset" => 34,
"key" => nil
}
}
{
"message" => "ajaxhe\nanother ajaxhe",
"@version" => "1",
"@timestamp" => "2016-04-10T12:24:43.962Z",
"kafka" => {
"msg_size" => 21,
"topic" => "test",
"consumer_group" => "logstash",
"partition" => 0,
"offset" => 27,
"key" => nil
}
}
我将分割符换成:
terminator => "#"

调整logstash配置以及producer,能够正常切分,难道是对split filter不支持转义符号的缘故?
with topic.get_producer() as producer:  # Create Kafka producer on the given topic
while True:
#msg = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
msg = ''
msg += 'ajaxhe#another ajaxhe'
print msg
producer.produce(msg) # Send the message to Kafka
time.sleep(5)
预期输出:
{
"message" => "ajaxhe",
"@version" => "1",
"@timestamp" => "2016-04-10T12:27:36.277Z",
"kafka" => {
"msg_size" => 21,
"topic" => "test",
"consumer_group" => "logstash",
"partition" => 0,
"offset" => 39,
"key" => nil
}
}
{
"message" => "another ajaxhe",
"@version" => "1",
"@timestamp" => "2016-04-10T12:27:36.277Z",
"kafka" => {
"msg_size" => 21,
"topic" => "test",
"consumer_group" => "logstash",
"partition" => 0,
"offset" => 39,
"key" => nil
}
}

 
已邀请:

medcl - 今晚打老虎。

赞同来自:

换其他分割符试试呢?如『\t』,也许是个bug

要回复问题请先登录注册