身安不如心安,屋宽不如心宽 。

es 其中一台莫名踢出集群 请各帮忙分析一下原因

Elasticsearch | 作者 mingaijie1 | 发布于2018年01月19日 | 阅读数:3453

20台ecs,收到报警其中一台脱离集群,然后我去log查看如下日志:
[2018-01-19T21:08:48,481][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][young][4312117][1523595] duration [860ms], collections [1]/[1.6s], total [860ms]/[20.6h], memory [25.1g
b]->[25.2gb]/[31.8gb], all_pools {[young] [390.3mb]->[13.4mb]/[1.4gb]}{[survivor] [46mb]->[191.3mb]/[191.3mb]}{[old] [24.7gb]->[25gb]/[30.1gb]}[2018-01-19T21:08:48,481][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312117] overhead, spent [860ms] collecting in the last [1.6s]
[2018-01-19T21:08:54,654][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312123] overhead, spent [412ms] collecting in the last [1.1s]
[2018-01-19T21:08:59,655][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312128] overhead, spent [536ms] collecting in the last [1s]
[2018-01-19T21:09:18,658][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312147] overhead, spent [584ms] collecting in the last [1s]
[2018-01-19T21:09:24,659][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312153] overhead, spent [314ms] collecting in the last [1s]
[2018-01-19T21:09:26,659][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312155] overhead, spent [488ms] collecting in the last [1s]
[2018-01-19T21:09:28,660][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312157] overhead, spent [525ms] collecting in the last [1s]
[2018-01-19T21:09:34,093][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312162] overhead, spent [683ms] collecting in the last [1.4s]
[2018-01-19T21:09:35,892][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312163] overhead, spent [933ms] collecting in the last [1.7s]
[2018-01-19T21:09:37,548][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][young][4312164][1523626] duration [1.2s], collections [1]/[1.6s], total [1.2s]/[20.6h], memory [26.4gb]
->[27.1gb]/[31.8gb], all_pools {[young] [14.4mb]->[187.1mb]/[1.4gb]}{[survivor] [35mb]->[191.3mb]/[191.3mb]}{[old] [26.4gb]->[26.7gb]/[30.1gb]}[2018-01-19T21:09:37,548][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312164] overhead, spent [1.2s] collecting in the last [1.6s]
[2018-01-19T21:09:38,616][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312165] overhead, spent [637ms] collecting in the last [1s]
[2018-01-19T21:09:39,735][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312166] overhead, spent [627ms] collecting in the last [1.1s]
[2018-01-19T21:09:45,293][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][young][4312170][1523632] duration [1.5s], collections [2]/[2.5s], total [1.5s]/[20.6h], memory [27.2gb]
->[27.7gb]/[31.8gb], all_pools {[young] [62.9mb]->[104.2mb]/[1.4gb]}{[survivor] [75.4mb]->[30.9mb]/[191.3mb]}{[old] [27.1gb]->[27.5gb]/[30.1gb]}[2018-01-19T21:09:45,293][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312170] overhead, spent [1.5s] collecting in the last [2.5s]
[2018-01-19T21:09:47,190][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][young][4312171][1523633] duration [1.3s], collections [1]/[1.8s], total [1.3s]/[20.6h], memory [27.7gb]
->[28.1gb]/[31.8gb], all_pools {[young] [104.2mb]->[984.7kb]/[1.4gb]}{[survivor] [30.9mb]->[191.3mb]/[191.3mb]}{[old] [27.5gb]->[27.9gb]/[30.1gb]}[2018-01-19T21:09:47,190][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312171] overhead, spent [1.3s] collecting in the last [1.8s]
[2018-01-19T21:09:49,001][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch_25] [gc][4312172] overhead, spent [1.2s] collecting in the last [1.8s]
已邀请:

xinfanwang

赞同来自: medcl

不响应了集群的请求包了吧。其它master节点认为这个节点挂了。这个节点一直报overhead。JVM GC回收不了多少空间。[old] [27.5gb]->[27.9gb]/[30.1gb]。

要回复问题请先登录注册