找到问题的解决办法了么?

【求助】elk的索引名,在filebeat定义的名字和解析过后不一致

Logstash | 作者 sweetpotato | 发布于2018年09月11日 | 阅读数:3927

input {
beats {
port => 5044
}
}
filter {
ruby {
code => "event.set('localTime', Time.now.getlocal('+08:00').strftime('%Y.%m.%d'))"
}
if [type] =~ /nginx_err/ {
grok {
match => [
"message" , "%{DATESTAMP:log_timestamp} \[%{WORD:state}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, upstream: \"%{URI:upstream}\")?(?:, host: %{QS:domain})?(?:, referrer: \"%{URI:referrer}\")"
]
}
geoip {
source => "client"
target => "geoip"
database => "/opt/test/GeoLite2-City/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
date {
match => [ "timestamp" , "YYYY/MM/dd HH:mm:ss" ]
remove_field => [ "timestamp" ]
}
}else{
if [type] =~ /nginx_access/ {
mutate {
gsub => [
"message", "\n", " "
]
}
json {
source => "message"
remove_field => "message"
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/opt/test/GeoLite2-City/GeoLite2-City.mmdb"
}
}
}
}
output {
elasticsearch {
hosts => ["172.17.12.180:9200"]
sniffing => false
manage_template => false
index => "%{type}-%{localTime}"
document_type => "%{type}"
}
}

 
上面是logstash的配置,就是说 
我的日志是json格式的,直接filter使用json,然后加上geoip不会报错,但是也不会有这个字段
 
 
if [type] =~ /nginx_access/ {
mutate {
gsub => [
"message", "\n", " "
]
}
json {
source => "message"
remove_field => "message"
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/opt/test/GeoLite2-City/GeoLite2-City.mmdb"
}
}
以下是日志
{"session_id": "-", "type":"nginx" ,"remote_addr": "123.141.64.130","refer": "http://lu-pro.t/center/main/index_test&quot;,"time": "06/Sep/2018:11:33:27 +0800",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36","method":"GET", "request": "GET /
center/statistics/order HTTP/1.1","status": 200,"body_bytes_sent":95, "x_forwarded_for": "-","request_time": 0.014,"bytes_sent" :498,"request_length": 842,"request_body": "-" }
ip字段是remote_addr
 
 
请问json格式处理后,也不能给geoip拿到该字段吗
已邀请:

sweetpotato - 90IT男

赞同来自:

噢 可以了,不过有别的问题,就是这个index名
我明明是用nginx_access,但是不知道为什么,转换过去就只变成nginx-{时间}
filebeat:
prospectors:
-
paths:
- /opt/test/error.log
document_type: test_nginx_err

-
paths:
- /opt/test/access.log
document_type: nginx_access

registry_file: /var/lib/filebeat/registry

output:
logstash: hosts: ["127.0.0.1:5044"] bulk_max_size: 1024

shipper:

logging:
files:
rotateeverybytes: 1048500 # = 10MB
green  open   nginx-2018.09.07              NQqyGubtTQOyaToWgPLnWQ   5   1          2            0     44.5kb         22.2kb
green open nginx-2018.09.11 YxChGwK-Qey-taXOuJgPlA 5 1 5 0 123.3kb 61.6kb

luohuanfeng

赞同来自:

你logstash的output{}里面写了
index => "%{type}-%{localTime}"
 

要回复问题请先登录注册