原始日志:
101.90.254.100 - - [16/Apr/2018:18:01:11 +0800] "GET /new/api/site_login_ijf.htm?referer=&user_fingerprint=&app_key=adhu5487540&callback=getSiteLoginInfo15238728&v=9663 HTTP/1.1" 200 490 "https://display.test.com/site_ ... ot%3B "Mozilla/5.0 (Linux; U; Android 7.0; zh-CN; KNT-AL20 Build/HUAWEIKNT-AL20) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.6.4.950 UWS/2.11.0.46 Mobile Safari/537.36 UCBS/2.11.0.46_180226200455 NebulaSDK/1.8.100112 Nebula AlipayDefined(nt:4G,ws:360|0|4.0) AliApp(AP/10.1.18.708) AlipayClient/10.1.18.708 Language/zh-Hans useStatusBar/true" "-" "display.test.com" 127.0.0.1:17290 200 - "text/javascript; charset=utf-8" 0.028 > 0.028
保存到kafka中的:
{"@timestamp":"2018-04-16T10:01:15.120Z","beat":{"hostname":"sh-103250","name":"sh-103250","version":"5.4.0"},"fields":{"type":"nginx_log"},"input_type":"log","message":"101.90.254.100 - - [16/Apr/2018:18:01:11 +0800] \"GET /new/api/site_login_ijf.htm?referer=\u0026user_fingerprint=\u0026app_key=adhu5487540dc\u0026callback=getSiteLoginInfo1523872870\u0026v=9663 HTTP/1.1\" 200 490 \"https://display.test.com/site_ ... 540dc\" \"Mozilla/5.0 (Linux; U; Android 7.0; zh-CN; KNT-AL20 Build/HUAWEIKNT-AL20) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.6.4.950 UWS/2.11.0.46 Mobile Safari/537.36 UCBS/2.11.0.46_180226200455 NebulaSDK/1.8.100112 Nebula AlipayDefined(nt:4G,ws:360|0|4.0) AliApp(AP/10.1.18.708) AlipayClient/10.1.18.708 Language/zh-Hans useStatusBar/true\" \"-\" \"display.test.com\" 127.0.0.1:17290 200 - \"text/javascript; charset=utf-8\" 0.028 \u003e 0.028","offset":19006739,"source":"/usr/local/nginx/logs/display.test.com.log","tags":["nginx_log"],"type":"log"}
多了不少转义符,另外logstash从kafka中读取会更多转义符,请解决
101.90.254.100 - - [16/Apr/2018:18:01:11 +0800] "GET /new/api/site_login_ijf.htm?referer=&user_fingerprint=&app_key=adhu5487540&callback=getSiteLoginInfo15238728&v=9663 HTTP/1.1" 200 490 "https://display.test.com/site_ ... ot%3B "Mozilla/5.0 (Linux; U; Android 7.0; zh-CN; KNT-AL20 Build/HUAWEIKNT-AL20) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.6.4.950 UWS/2.11.0.46 Mobile Safari/537.36 UCBS/2.11.0.46_180226200455 NebulaSDK/1.8.100112 Nebula AlipayDefined(nt:4G,ws:360|0|4.0) AliApp(AP/10.1.18.708) AlipayClient/10.1.18.708 Language/zh-Hans useStatusBar/true" "-" "display.test.com" 127.0.0.1:17290 200 - "text/javascript; charset=utf-8" 0.028 > 0.028
保存到kafka中的:
{"@timestamp":"2018-04-16T10:01:15.120Z","beat":{"hostname":"sh-103250","name":"sh-103250","version":"5.4.0"},"fields":{"type":"nginx_log"},"input_type":"log","message":"101.90.254.100 - - [16/Apr/2018:18:01:11 +0800] \"GET /new/api/site_login_ijf.htm?referer=\u0026user_fingerprint=\u0026app_key=adhu5487540dc\u0026callback=getSiteLoginInfo1523872870\u0026v=9663 HTTP/1.1\" 200 490 \"https://display.test.com/site_ ... 540dc\" \"Mozilla/5.0 (Linux; U; Android 7.0; zh-CN; KNT-AL20 Build/HUAWEIKNT-AL20) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.6.4.950 UWS/2.11.0.46 Mobile Safari/537.36 UCBS/2.11.0.46_180226200455 NebulaSDK/1.8.100112 Nebula AlipayDefined(nt:4G,ws:360|0|4.0) AliApp(AP/10.1.18.708) AlipayClient/10.1.18.708 Language/zh-Hans useStatusBar/true\" \"-\" \"display.test.com\" 127.0.0.1:17290 200 - \"text/javascript; charset=utf-8\" 0.028 \u003e 0.028","offset":19006739,"source":"/usr/local/nginx/logs/display.test.com.log","tags":["nginx_log"],"type":"log"}
多了不少转义符,另外logstash从kafka中读取会更多转义符,请解决
5 个回复
ribbel - linux运维
赞同来自:
football025
赞同来自:
football025
赞同来自:
a505100745
赞同来自:
football025
赞同来自:
input的时候要codec => json
output 要 codec => line { format => "%{message}"}