filebeat和ELK全用了6.2.4了,kafka是1.1.0,filebeat写入kafka后,所有信息都保存在message字段中,怎么才能把message里面的字段都单独分离出来呢?

作者 a505100745 | 发布于2018年05月07日 | 阅读数:1373

我用filebeat收集的是json格式的nginx日志,filebeat收集的信息用output:kafka到达kafka后是这样:
{"@timestamp":"2018-05-07T14:47:43.586Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4","topic":"elk-nginx"},"source":"/usr/local/nginx/logs/access_json.log","offset":741815,"json":{},"message":"{ \"@timestamp\": \"2018-05-07T19:04:25+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"73\", \"request\": \"POST /jsrpc.php?output=json-rpc HTTP/1.1\", \"request_method\": \"POST\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.076\", \"request_body\": \"\"{\\x22jsonrpc\\x22: \\x222.0\\x22, \\x22method\\x22: \\x22zabbix.status\\x22, \\x22params\\x22: {}, \\x22auth\\x22: \\x228e125896e94285e47e1313be49d5cb55\\x22, \\x22id\\x22: 3}\" }}","prospector":{"type":"log"},"beat":{"name":"master","hostname":"master","version":"6.2.4"}}
logstash用input-kafka后是这样:
{
"source" => "/usr/local/nginx/logs/access_json.log",
"offset" => 738665,
"message" => "{ \"@timestamp\": \"2018-05-07T19:04:08+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"25686\", \"request\": \"GET /overview.php?ddreset=1 HTTP/1.1\", \"request_method\": \"GET\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.153\", \"request_body\": \"\"-\" }}",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"type" => "accesslog",
"beat" => {
"name" => "master",
"version" => "6.2.4",
"hostname" => "master"
},
"json" => {},
"prospector" => {
"type" => "log"
},
"@timestamp" => 2018-05-07T14:47:43.586Z
}

发现所有需要的信息都在message段里面,现在如何能把message里面的字段都单独分离出来呢?
类似于下面这个(从网上找的),让message中的每个字段都单独列出来:
{
"message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",
"@version" => "1",
"@timestamp" => "2017-03-30T08:27:09.539Z",
"path" => "/var/log/nginx/access.log",
"host" => "spark4",
"type" => "nginxlog",
"remote_ip" => "192.168.154.2",
"timestamp" => "30/Mar/2017:01:27:09 -0700",
"method" => "GET",
"request" => "/index.html",
"httpversion" => "1.1",
"status" => "304",
"bytes" => "0",
"referer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",
"xforward" => "\"-\""
}

 
已邀请:

yokv

赞同来自: a505100745 xm110224 lianjie

 
{
"@timestamp": "2018-05-07T19:04:08+08:00",
"remote_addr": "192.168.6.1",
"remote_user": "-",
"status": "200",
"body_bytes_sent": "25686",
"request": "GET /overview.php?ddreset=1 HTTP/1.1",
"request_method": "GET",
"http_referrer": "http://192.168.6.71/overview.php?ddreset=1",
"http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331",
"http_x_forwarded_for": "-",
"request_time": "0.153",
"request_body": "" - " }}
你的json格式有错吧

request_body那多个"和},你的nginx日志格式配错了吧

yokv

赞同来自:

filebeat配置加上试试
json.keys_under_root: true
json.overwrite_keys: true

 

a505100745

赞同来自:

[b]这个是配置文件:[/b]
filebeat.prospectors:
- type: log
enabled: true
paths:
- /usr/local/nginx/logs/access_json.log
json.message_key: log
json.keys_under_root: true
json.overwrite_keys: true
exclude_lines: ['^DBG',"^$"]
document_type: access-log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.kafka:
hosts: ["192.168.6.71:9092","192.168.6.72:9092","192.168.6.73:9092"]
topic: elk-nginx
required_acks: 1

a505100745

赞同来自:

此问题已解决,谢谢大神的帮助

tiandou

赞同来自:

能看下你是怎么把message里面的字段都单独分离出来的?

chachabusi - 新手妹子运维,希望多多关照

赞同来自:

我也遇到这个问题了 filebeat推到kafka的消息包含时间戳的各种信息 ,但是我只要message 楼主时怎么解决的

aslan

赞同来自:

你是怎么将beat字段从message字段中取出来的呢

aslan

赞同来自:

不好意思 我没权限直接回复你刚的问题,
json {

      source => "message"

    }
我将外边大的message字段用json转下就可以把message字段中各个字段拿出来了

要回复问题请先登录注册