使用logstash-input-jdbc导入数据后,文档变少了
匿名 | 发布于2017年08月02日 | 阅读数:4282
使用logstash-input-jdbc从sql server导入某张表里的数据,使用数据库表的自增id作为es文档的_id。数据库的自增id的值是从1到60556,总共有60556条记录。但导入后发现es的索引里只有43196条记录,另外有1486条记录被删除。这里有2个问题:
1. 没有手工删除文档,为什么在导入的过程中有文档被删除了。
2. 算上被删除的文档数量,ES的文档数量也比原始数据库表里的记录数量要少。
请问要怎样排查原因?
ES里有如下日志:
[2017-08-02T09:14:14,400][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][229] overhead, spent [301ms] collecting in the last [1s]
[2017-08-02T09:14:16,408][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][231] overhead, spent [266ms] collecting in the last [1s]
[2017-08-02T09:14:25,451][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][240] overhead, spent [313ms] collecting in the last [1s]
[2017-08-02T09:14:27,613][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][242] overhead, spent [347ms] collecting in the last [1.1s]
[2017-08-02T09:14:31,647][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][246] overhead, spent [270ms] collecting in the last [1s]
[2017-08-02T09:14:40,993][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][255] overhead, spent [464ms] collecting in the last [1.1s]
[2017-08-02T09:14:43,006][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][257] overhead, spent [306ms] collecting in the last [1s]
logstash的日志未发现异常,并且是读完了60556条记录。
1. 没有手工删除文档,为什么在导入的过程中有文档被删除了。
2. 算上被删除的文档数量,ES的文档数量也比原始数据库表里的记录数量要少。
请问要怎样排查原因?
ES里有如下日志:
[2017-08-02T09:14:14,400][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][229] overhead, spent [301ms] collecting in the last [1s]
[2017-08-02T09:14:16,408][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][231] overhead, spent [266ms] collecting in the last [1s]
[2017-08-02T09:14:25,451][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][240] overhead, spent [313ms] collecting in the last [1s]
[2017-08-02T09:14:27,613][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][242] overhead, spent [347ms] collecting in the last [1.1s]
[2017-08-02T09:14:31,647][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][246] overhead, spent [270ms] collecting in the last [1s]
[2017-08-02T09:14:40,993][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][255] overhead, spent [464ms] collecting in the last [1.1s]
[2017-08-02T09:14:43,006][INFO ][o.e.m.j.JvmGcMonitorService] [vs11] [gc][257] overhead, spent [306ms] collecting in the last [1s]
logstash的日志未发现异常,并且是读完了60556条记录。
3 个回复
laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net
赞同来自: zhangyufu
zhangyufu - 学习使我快乐
赞同来自:
gzliudan
赞同来自: