想把message里的内容，拆成time, content
原来的message不再分词，转换成keyword
content转换成text
time转换成date
但是发现es6.1.4 beats6.1.4的pipeline不能转换成date text
如下：
PUT _ingest/pipeline/test_parttern_01{
"description" : "test pipeline",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:time}\\s*\\|\\s*%{DATA:level}\\s*\\|\\s*%{str:content}"
],
"pattern_definitions": {
"str": "[\\s\\S]*"
}
}
},
{
"convert": {
"field": "content",
"type": "text"
}
},
{
"convert": {
"field": "time",
"type": "date"
}
}
]
}
分别报错：
"reason": "[type] type [text] not supported, cannot convert field.",
"reason": "[type] type [date] not supported, cannot convert field.",
请大神指导应该怎么处理？
如何把原来的：message不作分词，拆分的fields再去转换成date , text
原来的message不再分词，转换成keyword
content转换成text
time转换成date
但是发现es6.1.4 beats6.1.4的pipeline不能转换成date text
如下：
PUT _ingest/pipeline/test_parttern_01{
"description" : "test pipeline",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:time}\\s*\\|\\s*%{DATA:level}\\s*\\|\\s*%{str:content}"
],
"pattern_definitions": {
"str": "[\\s\\S]*"
}
}
},
{
"convert": {
"field": "content",
"type": "text"
}
},
{
"convert": {
"field": "time",
"type": "date"
}
}
]
}
分别报错：
"reason": "[type] type [text] not supported, cannot convert field.",
"reason": "[type] type [date] not supported, cannot convert field.",
请大神指导应该怎么处理？
如何把原来的：message不作分词，拆分的fields再去转换成date , text
4 个回复
bellengao - 博客: https://www.jianshu.com/u/e0088e3e2127
赞同来自: bzg119
{
"date": {
"field": "time",
"formats": ["ISO8601"],
"timezone": "Asia/Shanghai",
"ignore_failure": true
},
"remove": {
"field": "time"
}
}
bzg119
赞同来自:
pipeline提取的字段，在fields.yml怎么定义？
提取的字段，想转换成text可以分词
bzg119
赞同来自:
谢谢答复。
我用的是filebeat6.1.4
fields.yml里增加了配置：定义了一个content
- name: message
type: text
ignore_above: 0
required: true
description: >
The content of the line read from the log file.
- name: content
type: text
ignore_above: 0
required: true
description: >
The content of the line read from the log file.
然后，filebeat创建索引后，去查询索引，发现content还是keyword
"content": {
"type": "keyword",
"ignore_above": 1024
}
那个dynamic_templates我没有配置，不知道在filebeat.yml还是fields.yml里配置。我看内容，应该是默认将string当成keyword的默认配置，我不用配置吧。。
bzg119
赞同来自:
filebeat可以直接修改dynamic_templates的定义吗？