filebeat和ELK全用了6.2.4了，kafka是1.1.0，filebeat写入kafka后，所有信息都保存在message字段中，怎么才能把message里面的字段都单独分离出来呢？

Beats | 作者 a505100745 | 发布于2018年05月07日 | 阅读数：9553

我用filebeat收集的是json格式的nginx日志，filebeat收集的信息用output:kafka到达kafka后是这样：

{"@timestamp":"2018-05-07T14:47:43.586Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4","topic":"elk-nginx"},"source":"/usr/local/nginx/logs/access_json.log","offset":741815,"json":{},"message":"{ \"@timestamp\": \"2018-05-07T19:04:25+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"73\", \"request\": \"POST /jsrpc.php?output=json-rpc HTTP/1.1\", \"request_method\": \"POST\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.076\", \"request_body\": \"\"{\\x22jsonrpc\\x22: \\x222.0\\x22, \\x22method\\x22: \\x22zabbix.status\\x22, \\x22params\\x22: {}, \\x22auth\\x22: \\x228e125896e94285e47e1313be49d5cb55\\x22, \\x22id\\x22: 3}\" }}","prospector":{"type":"log"},"beat":{"name":"master","hostname":"master","version":"6.2.4"}}

logstash用input-kafka后是这样：

{

        "source" => "/usr/local/nginx/logs/access_json.log",

        "offset" => 738665,

       "message" => "{ \"@timestamp\": \"2018-05-07T19:04:08+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"25686\", \"request\": \"GET /overview.php?ddreset=1 HTTP/1.1\", \"request_method\": \"GET\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.153\", \"request_body\": \"\"-\" }}",

          "tags" => [

        [0] "_jsonparsefailure"

    ],

      "@version" => "1",

          "type" => "accesslog",

          "beat" => {

            "name" => "master",

         "version" => "6.2.4",

        "hostname" => "master"

    },

          "json" => {},

    "prospector" => {

        "type" => "log"

    },

    "@timestamp" => 2018-05-07T14:47:43.586Z

}

发现所有需要的信息都在message段里面，现在如何能把message里面的字段都单独分离出来呢？
类似于下面这个（从网上找的），让message中的每个字段都单独列出来：

{

        "message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",

       "@version" => "1",

     "@timestamp" => "2017-03-30T08:27:09.539Z",

           "path" => "/var/log/nginx/access.log",

           "host" => "spark4",

           "type" => "nginxlog",

      "remote_ip" => "192.168.154.2",

      "timestamp" => "30/Mar/2017:01:27:09 -0700",

         "method" => "GET",

        "request" => "/index.html",

    "httpversion" => "1.1",

         "status" => "304",

          "bytes" => "0",

        "referer" => "\"-\"",

          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",

       "xforward" => "\"-\""

}

10 个回复

yokv

赞同来自: a505100745 、xm110224 、lianjie

{

	"@timestamp": "2018-05-07T19:04:08+08:00",

	"remote_addr": "192.168.6.1",

	"remote_user": "-",

	"status": "200",

	"body_bytes_sent": "25686",

	"request": "GET /overview.php?ddreset=1 HTTP/1.1",

	"request_method": "GET",

	"http_referrer": "http://192.168.6.71/overview.php?ddreset=1&quot;,

	"http_user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331",

	"http_x_forwarded_for": "-",

	"request_time": "0.153",

	"request_body": "" - " }}

你的json格式有错吧



request_body那多个"和},你的nginx日志格式配错了吧

yokv

filebeat配置加上试试

json.keys_under_root: true

json.overwrite_keys: true

a505100745

[b]这个是配置文件：[/b]

filebeat.prospectors:

- type: log

  enabled: true

  paths:

    - /usr/local/nginx/logs/access_json.log

  json.message_key: log

  json.keys_under_root: true

  json.overwrite_keys: true

  exclude_lines: ['^DBG',"^$"]

  document_type: access-log

filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:

  index.number_of_shards: 3

setup.kibana:

output.kafka:

  hosts: ["192.168.6.71:9092","192.168.6.72:9092","192.168.6.73:9092"]

  topic: elk-nginx

  required_acks: 1