使用 nohup 或 disown 如果你要让某个进程运行在后台。

json filter 解析日志 input 源 redis 和 stdin 行为表现不一样的问题

Logstash | 作者 shermanzhou | 发布于2017年07月07日 | 阅读数:3789

案例一:

日志源:标准输入 stdin
通道: stdin => logstash => elasticsearch
logstash配置:
input {
stdin {}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "zcq_index"
}
stdout { codec => rubydebug }
}


通过标准输入:
{"applicaiton":"merchant","level":"INFO","log_type":"visit","user_id":"1000","ip":"112.252.150.88","domain":"www.xxx.com","url":"/xxx/xxx/xxx","method":"GET","request":"{...}","response":"{...}","execute_time":"231","uuid":"676f0e631a734c449832cfece5cc0127","server_hostname":"test012","server_ip":"112.42.0.102"}

最后存入到 elasticsearch 格式为:
{
"request": "{...}",
"method": "GET",
"level": "INFO",
"ip": "112.252.150.88",
"message": "{\"applicaiton\":\"merchant\",\"level\":\"INFO\",\"type\":\"visit\",\"user_id\":\"1000\",\"ip\":\"112.252.150.88\",\"domain\":\"www.xxx.com\",\"url\":\"/xxx/xxx/xxx\",\"method\":\"GET\",\"request\":\"{\\\"a\\\":\\\"b\\\"}\",\"response\":\"{...}\",\"execute_time\":\"231\",\"uuid\":\"676f0e631a734c449832cfece5cc0127\",\"server_hostname\":\"pre004.qc.com\",\"server_ip\":\"112.42.0.102\"}",
"type": "visit",
"uuid": "676f0e631a734c449832cfece5cc0127",
"url": "/xxx/xxx/xxx",
"server_hostname": "test012",
"@timestamp": "2017-07-07T02:48:45.509Z",
"user_id": "1000",
"response": "{...}",
"execute_time": "231",
"domain": "www.xxx.com",
"@version": "1",
"host": "zcq-MacBook-Pro.local",
"server_ip": "112.42.0.102",
"applicaiton": "merchant"
}


logstash 成功提取了 message 里的JSON字段,并且保留了message字符串。

案例二:

日志源:redis
通道: redis => logstash => elasticsearch
logstash配置:
input {
redis {
data_type => "list"
key => "logstash-list"
host => "10.100.0.185"
port => 6379
threads => 1
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "zcq_index"
}
stdout { codec => rubydebug }
}

通过redis-cli输入:
{"applicaiton":"merchant","level":"INFO","log_type":"visit","user_id":"1000","ip":"112.252.150.88","domain":"www.xxx.com","url":"/xxx/xxx/xxx","method":"GET","request":"{...}","response":"{...}","execute_time":"231","uuid":"676f0e631a734c449832cfece5cc0127","server_hostname":"test012","server_ip":"112.42.0.102"}

最后存入到 elasticsearch 格式为:
{
"request": "{...}",
"method": "GET",
"level": "INFO",
"ip": "112.252.150.88",
"type": "visit",
"uuid": "676f0e631a734c449832cfece5cc0127",
"url": "/xxx/xxx/xxx",
"server_hostname": "test012",
"@timestamp": "2017-07-07T02:55:42.102Z",
"user_id": "1000",
"response": "{...}",
"execute_time": "231",
"domain": "www.xxx.com",
"@version": "1",
"host": "zcq-MacBook-Pro.local",
"server_ip": "112.42.0.102",
"applicaiton": "merchant"
}

logstash成功提取了message里的JSON字段,但是并没有保留 message字符串。

问题: 
message呢?为什么案例一有,案例二没有存到ES?
 
已邀请:

wyntergreg

赞同来自:

案例一的JSON里加个message字段即可,值是什么无所谓,覆盖一下。

要回复问题请先登录注册