filebeat和ELK全用了6.2.4了,kafka是1.1.0,filebeat写入kafka后,所有信息都保存在message字段中,怎么才能把message里面的字段都单独分离出来呢?
Beats | 作者 a505100745 | 发布于2018年05月07日 | 阅读数:9634
我用filebeat收集的是json格式的nginx日志,filebeat收集的信息用output:kafka到达kafka后是这样:
发现所有需要的信息都在message段里面,现在如何能把message里面的字段都单独分离出来呢?
类似于下面这个(从网上找的),让message中的每个字段都单独列出来:
{"@timestamp":"2018-05-07T14:47:43.586Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4","topic":"elk-nginx"},"source":"/usr/local/nginx/logs/access_json.log","offset":741815,"json":{},"message":"{ \"@timestamp\": \"2018-05-07T19:04:25+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"73\", \"request\": \"POST /jsrpc.php?output=json-rpc HTTP/1.1\", \"request_method\": \"POST\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.076\", \"request_body\": \"\"{\\x22jsonrpc\\x22: \\x222.0\\x22, \\x22method\\x22: \\x22zabbix.status\\x22, \\x22params\\x22: {}, \\x22auth\\x22: \\x228e125896e94285e47e1313be49d5cb55\\x22, \\x22id\\x22: 3}\" }}","prospector":{"type":"log"},"beat":{"name":"master","hostname":"master","version":"6.2.4"}}
logstash用input-kafka后是这样:{
"source" => "/usr/local/nginx/logs/access_json.log",
"offset" => 738665,
"message" => "{ \"@timestamp\": \"2018-05-07T19:04:08+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"25686\", \"request\": \"GET /overview.php?ddreset=1 HTTP/1.1\", \"request_method\": \"GET\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.153\", \"request_body\": \"\"-\" }}",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"type" => "accesslog",
"beat" => {
"name" => "master",
"version" => "6.2.4",
"hostname" => "master"
},
"json" => {},
"prospector" => {
"type" => "log"
},
"@timestamp" => 2018-05-07T14:47:43.586Z
}
发现所有需要的信息都在message段里面,现在如何能把message里面的字段都单独分离出来呢?
类似于下面这个(从网上找的),让message中的每个字段都单独列出来:
{
"message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",
"@version" => "1",
"@timestamp" => "2017-03-30T08:27:09.539Z",
"path" => "/var/log/nginx/access.log",
"host" => "spark4",
"type" => "nginxlog",
"remote_ip" => "192.168.154.2",
"timestamp" => "30/Mar/2017:01:27:09 -0700",
"method" => "GET",
"request" => "/index.html",
"httpversion" => "1.1",
"status" => "304",
"bytes" => "0",
"referer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",
"xforward" => "\"-\""
}
10 个回复
yokv
赞同来自: a505100745 、xm110224 、lianjie
yokv
赞同来自:
a505100745
赞同来自:
a505100745
赞同来自:
tiandou
赞同来自:
chachabusi - 新手妹子运维,希望多多关照
赞同来自:
aslan
赞同来自:
aslan
赞同来自:
json {
source => "message"
}
我将外边大的message字段用json转下就可以把message字段中各个字段拿出来了
sailershen
赞同来自:
jlhde123
赞同来自:
- dissect: tokenizer: "%{key1} - %{key2} "