用了Elasticsearch,一口气上5T

logstash从ES获取数据储存到mongoDB遇到的问题

Logstash | 作者 LiPain | 发布于2018年01月23日 | 阅读数:6030

各位大牛好,初学logstash,最近在做logstash从ES获取数据储存到mongoDB用于备份日志恢复到ES,但是在配置文件写好之后执行的阶段出现了一个可能关于时间戳类型的错误,一直误解,请大牛解惑。
问题如下
配置文件
input {
mongodb {
uri => 'mongodb://内网IP:27017/user_space'
placeholder_db_dir => '//home/qianyi/Downloads'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'user_space'
}
}
filter {
}
output {
file {
path => "//home/qianyi/Downloads/mongons.log"
}
elasticsearch {
hosts => "内网IP:9200"
index => "mongo_log"
document_id=>"%{uId}"
}
stdout{
codec=>rubydebug{}
}
}
按照此配置已经可以正常的取出数据,但是会报错
[2018-01-22T17:58:47.351966 #14686] DEBUG -- : MONGODB | QUERY | namespace=system_log.gs_log selector={:_id=>{:$gt=>BSON::ObjectId('5a61975db209e35843000007')}} flags= limit=30 skip=0 project=nil | runtime: 10.8860ms
[2018-01-22T17:58:47,353][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}
D, [2018-01-22T17:58:47.366423 #14686] DEBUG -- : MONGODB | QUERY | namespace=system_log.gs_log selector={:_id=>{:$gt=>BSON::ObjectId('5a61975db209e35843000007')}} flags= limit=30 skip=0 project=nil | runtime: 11.7218ms
[2018-01-22T17:58:47,368][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}
D, [2018-01-22T17:58:47.379617 #14686] DEBUG -- : MONGODB | QUERY | namespace=system_log.gs_log selector={:_id=>{:$gt=>BSON::ObjectId('5a61975db209e35843000007')}} flags= limit=30 skip=0 project=nil | runtime: 9.9361ms
[2018-01-22T17:58:47,381][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}
D, [2018-01-22T17:58:47.392635 #14686] DEBUG -- : MONGODB | QUERY | namespace=system_log.gs_log selector={:_id=>{:$gt=>BSON::ObjectId('5a61975db209e35843000007')}} flags= limit=30 skip=0 project=nil | runtime: 9.8879ms
[2018-01-22T17:58:47,394][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}
D, [2018-01-22T17:58:47.405800 #14686] DEBUG -- : MONGODB | QUERY | namespace=system_log.gs_log selector={:_id=>{:$gt=>BSON::ObjectId('5a61975db209e35843000007')}} flags= limit=30 skip=0 project=nil | runtime: 10.1318ms
[2018-01-22T17:58:47,410][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}
如上
 
错误的主体信息为【MongoDB Input threw an exception, restarting {:exception=>#<TypeError: wrong argument type String (expected LogStash::Timestamp)>}】
应该是和数据类型有关,但是对于mongodb了解表叫浅薄,是什么问题,请大牛指点,谢谢。
ES中的日志数据如下
{
"_index": "gs_log",
"_type": "gs_log",
"_id": "CFDD0186D689ED8C1D00571A16F26ED5",
"_version": 2,
"_score": 1,
"_source": {
"ip": "101.226.208.9",
"type": "gs_log",
"noteType": "N001940200",
"msg": {
"model": "Api",
"msg": {
"Method": "GET",
"GET": {
"area_id": "147"
},
"AcceptType": false,
"URI": "/index.php?m=api&a=find_area&area_id=147",
"ContentType": "text/plain;charset=UTF-8",
"POST": [ ]
}
},
"cTime": "2018-01-11 00:12:53",
"@version": "1",
"space": null,
"source": "N001950500",
"host": "qianyi-VB",
"action": "find_area",
"user": null,
"path": "//home/qianyi/Downloads/test_7.log",
"cTimeStamp": 1515600773,
"group": "Home",
"model": "Api",
"uId": "CFDD0186D689ED8C1D00571A16F26ED5",
"nodeId": null,
"@timestamp": "2018-01-12T06:08:07.078Z"
}
}
已邀请:

要回复问题请先登录注册