es集群gc问题

作者 wwwlll | 发布于2017年09月25日 | 阅读数:7289

[2017-09-22T09:19:00,655][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779980][148521] duration [23.9s], collections [1]/[24.6s], total [23.9s]/[4.1h], memory [29.7gb]->[29.3gb]/[30.8gb], all_pools {[young] [803.4mb]->[347.8mb]/[1.4gb]}{[survivor] [40.8mb]->[0b]/[191.3mb]}{[old] [28.8gb]->[29gb]/[29.1gb]}
[2017-09-22T09:19:00,747][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779980] overhead, spent [24.1s] collecting in the last [24.6s]
[2017-09-22T09:19:23,515][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779981][148522] duration [21.9s], collections [1]/[22.8s], total [21.9s]/[4.1h], memory [29.3gb]->[29.1gb]/[30.8gb], all_pools {[young] [347.8mb]->[76.2mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:19:23,515][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779981] overhead, spent [22.3s] collecting in the last [22.8s]
[2017-09-22T09:19:48,815][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779983][148523] duration [23.7s], collections [1]/[24.2s], total [23.7s]/[4.1h], memory [30.7gb]->[30.4gb]/[30.8gb], all_pools {[young] [1.4gb]->[1.3gb]/[1.4gb]}{[survivor] [141.6mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:19:48,815][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779983] overhead, spent [23.7s] collecting in the last [24.2s]
[2017-09-22T09:20:12,046][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779985][148524] duration [21.8s], collections [1]/[22.2s], total [21.8s]/[4.1h], memory [30.7gb]->[29gb]/[30.8gb], all_pools {[young] [1.4gb]->[28.7mb]/[1.4gb]}{[survivor] [122.2mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29gb]/[29.1gb]}
[2017-09-22T09:20:12,047][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779985] overhead, spent [21.8s] collecting in the last [22.2s]
[2017-09-22T09:20:36,439][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779986][148525] duration [23.8s], collections [1]/[24.3s], total [23.8s]/[4.1h], memory [29gb]->[29.1gb]/[30.8gb], all_pools {[young] [28.7mb]->[4.3mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:20:36,439][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779986] overhead, spent [23.8s] collecting in the last [24.3s]
[2017-09-22T09:20:58,895][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779987][148526] duration [21.9s], collections [1]/[22.4s], total [21.9s]/[4.1h], memory [29.1gb]->[29.1gb]/[30.8gb], all_pools {[young] [4.3mb]->[1.4mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:20:58,895][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779987] overhead, spent [21.9s] collecting in the last [22.4s]
[2017-09-22T09:21:23,694][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779988][148527] duration [23.7s], collections [1]/[24.7s], total [23.7s]/[4.1h], memory [29.1gb]->[30.2gb]/[30.8gb], all_pools {[young] [1.4mb]->[1gb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:21:23,694][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779988] overhead, spent [23.7s] collecting in the last [24.7s]
[2017-09-22T09:21:47,572][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779990][148528] duration [22.6s], collections [1]/[22.8s], total [22.6s]/[4.1h], memory [30.7gb]->[29.1gb]/[30.8gb], all_pools {[young] [1.4gb]->[66.7mb]/[1.4gb]}{[survivor] [151.5mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:21:47,572][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779990] overhead, spent [22.6s] collecting in the last [22.8s]
[2017-09-22T09:22:13,027][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779992][148529] duration [23.4s], collections [1]/[24.4s], total [23.4s]/[4.1h], memory [30.7gb]->[30.4gb]/[30.8gb], all_pools {[young] [1.4gb]->[1.3gb]/[1.4gb]}{[survivor] [89.9mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:22:13,032][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779992] overhead, spent [23.4s] collecting in the last [24.4s]
[2017-09-22T09:22:35,663][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779993][148530] duration [21.6s], collections [1]/[22.6s], total [21.6s]/[4.1h], memory [30.4gb]->[29.1gb]/[30.8gb], all_pools {[young] [1.3gb]->[70.6mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:22:35,663][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779993] overhead, spent [21.6s] collecting in the last [22.6s]
[2017-09-22T09:22:59,552][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779994][148531] duration [23.3s], collections [1]/[23.8s], total [23.3s]/[4.1h], memory [29.1gb]->[29.1gb]/[30.8gb], all_pools {[young] [70.6mb]->[36.3mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:22:59,552][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779994] overhead, spent [23.3s] collecting in the last [23.8s]
[2017-09-22T09:23:47,539][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779997][148533] duration [45.2s], collections [2]/[45.8s], total [45.2s]/[4.1h], memory [30.7gb]->[29.1gb]/[30.8gb], all_pools {[young] [1.4gb]->[8.2mb]/[1.4gb]}{[survivor] [173.3mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:23:47,539][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779997] overhead, spent [45.2s] collecting in the last [45.8s]
[2017-09-22T09:24:11,978][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1779998][148534] duration [23.8s], collections [1]/[24.4s], total [23.8s]/[4.1h], memory [29.1gb]->[29gb]/[30.8gb], all_pools {[young] [8.2mb]->[4mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29.1gb]->[29gb]/[29.1gb]}
[2017-09-22T09:24:11,978][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1779998] overhead, spent [23.8s] collecting in the last [24.4s]
[2017-09-22T09:24:37,304][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1780001][148535] duration [23.2s], collections [1]/[23.3s], total [23.2s]/[4.2h], memory [30.6gb]->[29gb]/[30.8gb], all_pools {[young] [1.4gb]->[28.7mb]/[1.4gb]}{[survivor] [162.5mb]->[0b]/[191.3mb]}{[old] [29gb]->[29gb]/[29.1gb]}
[2017-09-22T09:24:37,304][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1780001] overhead, spent [23.2s] collecting in the last [23.3s]
[2017-09-22T09:25:02,598][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1780003][148536] duration [23.7s], collections [1]/[24.2s], total [23.7s]/[4.2h], memory [30.3gb]->[29gb]/[30.8gb], all_pools {[young] [1.1gb]->[6mb]/[1.4gb]}{[survivor] [191.3mb]->[0b]/[191.3mb]}{[old] [29gb]->[29gb]/[29.1gb]}
[2017-09-22T09:25:02,598][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1780003] overhead, spent [24.1s] collecting in the last [24.2s]
[2017-09-22T09:25:25,435][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1780004][148537] duration [22.3s], collections [1]/[22.8s], total [22.3s]/[4.2h], memory [29gb]->[29.1gb]/[30.8gb], all_pools {[young] [6mb]->[32.3mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29gb]->[29.1gb]/[29.1gb]}
[2017-09-22T09:25:25,435][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1780004] overhead, spent [22.3s] collecting in the last [22.8s]
[2017-09-22T09:25:50,017][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1780006][148538] duration [23.2s], collections [1]/[23.5s], total [23.2s]/[4.2h], memory [30.7gb]->[29.8gb]/[30.8gb], all_pools {[young] [1.4gb]->[801.2mb]/[1.4gb]}{[survivor] [177.8mb]->[0b]/[191.3mb]}{[old] [29.1gb]->[29gb]/[29.1gb]}
[2017-09-22T09:25:50,018][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][1780006] overhead, spent [23.2s] collecting in the last [23.5s]
[2017-09-22T09:26:13,420][WARN ][o.e.m.j.JvmGcMonitorService] . [gc][old][1780007][148539] duration [22.7s], collections [1]/[23.1s], total [22.7s]/[4.2h], memory [29.8gb]->[29.4gb]/[30.8gb], all_pools {[young] [801.2mb]->[412.2mb]/[1.4gb]}{[survivor] [0b]->[0b]/[191.3mb]}{[old] [29gb]->[29.1gb]/[29.1gb]}


12台机器的集群 5.3.2 java1.8.131 已经配置内存锁定31g。现在不定期的其中某台会出现大量gc的记录然后导致集群无法工作,当我手工停掉这台es进程后,其他所有节点也产生了这样gc记录,最后没办法只能所有节点一起重启。

请问有啥办法吗还是配置我需要注意什么
已邀请:

kennywu76 - wood@Ctrip

赞同来自: wwwlll luohuanfeng

cyberdak - 58.com

赞同来自: wwwlll

 看这个记录大概觉得是CMS 然后FULL GC太多,年轻带只有1.4G,意味着堆分配不合理。。
 
那么最简单的办法就是直接切换到G1GC。
 
 

Loading Zhang

赞同来自: wwwlll

使用G1

ybtsdst - focus on lucene & es

赞同来自: wwwlll

集群在干啥? old区一直是满的

hufuman

赞同来自: wwwlll

完整的ES启动命令行发一下
 
机器的配置发一下

simooge - 80后IT男

赞同来自:

兄弟,解决了么?

要回复问题请先登录注册