elasticsearch 频繁fullgc

机器情况：       两台es集群，每台机器一个实例，整个集群一个index,数据量大概在1千五百万级别，ssd硬盘，内存我给分配的是16g ,es版本为5.2.2
 
问题描述：
      full gc有点频繁，并且每次gc的时候，其他模块【从es查数据并进行逻辑处理的模块】响应时间会变长，大概比没有gc的时候长100ms的样子，持续时间长达两个小时
      按理说时间变长，可以理解，但是持续两个小时就不能理解了，不太明白什么原因
想问：
     1：怎样能解决fullgc频繁的问题
     2：每次fullgc 对其他模块影响为什么那么大，持续时间那么长，你们有遇到相似的情况吗？
 
 
cpu和gc的一些情况：
    



 
es启动参数：

CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 

-XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=17179869184 

-XX:MaxHeapSize=17179869184 -XX:MaxNewSize=2442764288 -XX:MaxTenuringThreshold=6 -XX:NewSize=2442764288 

-XX:OldPLABSize=16 -XX:OldSize=4885528576 -XX:+PrintClassHistogram -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime 

-XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 

-XX:+UseCMSInitiAtingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

 
GC的log日志如下：      

2017-07-31T16:13:21.784+0800: 14.332: [CMS-concurrent-mark-start]

2017-07-31T16:13:21.786+0800: 14.335: [CMS-concurrent-mark: 0.002/0.002 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 

2017-07-31T16:13:21.786+0800: 14.335: [CMS-concurrent-preclean-start]

2017-07-31T16:13:21.811+0800: 14.360: [CMS-concurrent-preclean: 0.025/0.025 secs] [Times: user=0.06 sys=0.00, real=0.03 secs] 

2017-07-31T16:13:21.811+0800: 14.360: [CMS-concurrent-abortable-preclean-start]

 CMS: abort preclean due to time 2017-07-31T16:13:26.968+0800: 19.517: [CMS-concurrent-abortable-preclean: 3.123/5.157 secs] [Times: user=8.40 sys=0.37, real=5.15 secs] 

2017-07-31T16:13:26.969+0800: 19.517: [GC (CMS Final Remark) [YG occupancy: 977994 K (2146944 K)]19.517: [Rescan (parallel) , 0.0431862 secs]19.561: [weak refs processing, 0.0001164 secs]19.561: [class unloading, 0.0070301 secs]19.568: [scrub symbol table, 0.0047379 secs]19.572: [scrub string table, 0.0006645 secs][1 CMS-remark: 0K(14391744K)] 977994K(16538688K), 0.0575850 secs] [Times: user=1.14 sys=0.01, real=0.06 secs] 

2017-07-31T16:13:27.026+0800: 19.575: [CMS-concurrent-sweep-start]

2017-07-31T16:13:27.030+0800: 19.578: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 

2017-07-31T16:13:27.030+0800: 19.578: [CMS-concurrent-reset-start]

2017-07-31T16:13:27.059+0800: 19.608: [CMS-concurrent-reset: 0.029/0.029 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 

2017-08-01T00:08:29.999+0800: 28522.547: [CMS-concurrent-mark-start]

2017-08-01T00:08:31.091+0800: 28523.640: [CMS-concurrent-mark: 0.972/1.092 secs] [Times: user=20.49 sys=0.23, real=1.10 secs] 

2017-08-01T00:08:31.091+0800: 28523.640: [CMS-concurrent-preclean-start]

2017-08-01T00:08:32.381+0800: 28524.930: [CMS-concurrent-preclean: 1.157/1.290 secs] [Times: user=19.79 sys=0.29, real=1.29 secs] 

2017-08-01T00:08:32.381+0800: 28524.930: [CMS-concurrent-abortable-preclean-start]

2017-08-01T00:08:32.656+0800: 28525.204: [CMS-concurrent-abortable-preclean: 0.220/0.275 secs] [Times: user=3.93 sys=0.04, real=0.27 secs] 

2017-08-01T00:08:32.657+0800: 28525.206: [GC (CMS Final Remark) [YG occupancy: 1066372 K (2146944 K)]28525.206: [Rescan (parallel) , 0.0681478 secs]28525.274: [weak refs processing, 1.9562686 secs]28527.230: [class unloading, 0.0420890 secs]28527.272: [scrub symbol table, 0.0068975 secs]28527.279: [scrub string table, 0.0015076 secs][1 CMS-remark: 10807502K(14391744K)] 11873875K(16538688K), 2.7892374 secs] [Times: user=3.99 sys=0.01, real=2.79 secs] 

2017-08-02T00:16:22.163+0800: 115394.711: [CMS-concurrent-sweep-start]

2017-08-02T00:16:43.017+0800: 115415.566: [CMS-concurrent-sweep: 17.712/20.855 secs] [Times: user=324.85 sys=4.71, real=20.85 secs] 

2017-08-02T00:16:43.023+0800: 115415.572: [CMS-concurrent-reset-start]

2017-08-02T00:16:43.093+0800: 115415.642: [CMS-concurrent-reset: 0.059/0.070 secs] [Times: user=1.30 sys=0.03, real=0.07 secs] 

2017-08-02T01:17:51.465+0800: 119084.014: [CMS-concurrent-mark-start]

2017-08-02T01:17:52.124+0800: 119084.673: [CMS-concurrent-mark: 0.558/0.659 secs] [Times: user=12.99 sys=0.17, real=0.66 secs] 

2017-08-02T01:17:52.124+0800: 119084.673: [CMS-concurrent-preclean-start]

2017-08-02T01:17:52.808+0800: 119085.356: [CMS-concurrent-preclean: 0.589/0.683 secs] [Times: user=10.97 sys=0.14, real=0.69 secs] 

2017-08-02T01:17:52.808+0800: 119085.356: [CMS-concurrent-abortable-preclean-start]

2017-08-02T01:17:52.899+0800: 119085.447: [CMS-concurrent-abortable-preclean: 0.090/0.091 secs] [Times: user=1.10 sys=0.01, real=0.09 secs] 

2017-08-02T01:17:52.902+0800: 119085.450: [GC (CMS Final Remark) [YG occupancy: 1528335 K (2146944 K)]119085.450: [Rescan (parallel) , 0.0963117 secs]119085.547: [weak refs processing, 1.0689985 secs]119086.616: [class unloading, 0.0385312 secs]119086.654: [scrub symbol table, 0.0095100 secs]119086.664: [scrub string table, 0.0016191 secs][1 CMS-remark: 10797729K(14391744K)] 12326065K(16538688K), 1.6050374 secs] [Times: user=3.14 sys=0.01, real=1.60 secs] 

2017-08-02T01:17:54.507+0800: 119087.055: [CMS-concurrent-sweep-start]

2017-08-02T01:18:12.358+0800: 119104.906: [CMS-concurrent-sweep: 15.287/17.851 secs] [Times: user=269.87 sys=3.89, real=17.84 secs] 

2017-08-02T01:18:12.358+0800: 119104.906: [CMS-concurrent-reset-start]

2017-08-02T01:18:12.394+0800: 119104.942: [CMS-concurrent-reset: 0.036/0.036 secs] [Times: user=0.42 sys=0.01, real=0.04 secs] 

2017-08-02T02:20:51.571+0800: 122864.119: [CMS-concurrent-mark-start]

2017-08-02T02:20:52.094+0800: 122864.642: [CMS-concurrent-mark: 0.465/0.523 secs] [Times: user=9.99 sys=0.17, real=0.53 secs] 

2017-08-02T02:20:52.094+0800: 122864.642: [CMS-concurrent-preclean-start]

2017-08-02T02:20:52.742+0800: 122865.291: [CMS-concurrent-preclean: 0.537/0.649 secs] [Times: user=9.85 sys=0.15, real=0.65 secs] 

2017-08-02T02:20:52.743+0800: 122865.291: [CMS-concurrent-abortable-preclean-start]

2017-08-02T02:20:53.181+0800: 122865.729: [CMS-concurrent-abortable-preclean: 0.372/0.438 secs] [Times: user=6.67 sys=0.12, real=0.44 secs] 

2017-08-02T02:20:53.183+0800: 122865.732: [GC (CMS Final Remark) [YG occupancy: 1390290 K (2146944 K)]122865.732: [Rescan (parallel) , 0.0969662 secs]122865.829: [weak refs processing, 1.1002808 secs]122866.929: [class unloading, 0.0394111 secs]122866.969: [scrub symbol table, 0.0076265 secs]122866.976: [scrub string table, 0.0016416 secs][1 CMS-remark: 10796833K(14391744K)] 12187123K(16538688K), 1.6322704 secs] [Times: user=3.13 sys=0.01, real=1.63 secs] 

2017-08-02T02:20:54.816+0800: 122867.365: [CMS-concurrent-sweep-start]

2017-08-02T02:21:12.781+0800: 122885.329: [CMS-concurrent-sweep: 15.299/17.965 secs] [Times: user=272.00 sys=4.71, real=17.96 secs] 

2017-08-02T02:21:12.781+0800: 122885.330: [CMS-concurrent-reset-start]

2017-08-02T02:21:12.826+0800: 122885.375: [CMS-concurrent-reset: 0.045/0.045 secs] [Times: user=0.64 sys=0.01, real=0.05 secs] 

2017-08-02T03:13:08.688+0800: 126001.236: [CMS-concurrent-mark-start]

2017-08-02T03:13:09.351+0800: 126001.900: [CMS-concurrent-mark: 0.603/0.663 secs] [Times: user=13.30 sys=0.14, real=0.66 secs] 

2017-08-02T03:13:09.351+0800: 126001.900: [CMS-concurrent-preclean-start]

2017-08-02T03:13:10.131+0800: 126002.679: [CMS-concurrent-preclean: 0.703/0.780 secs] [Times: user=11.50 sys=0.13, real=0.78 secs] 

2017-08-02T03:13:10.131+0800: 126002.679: [CMS-concurrent-abortable-preclean-start]

2017-08-02T03:13:10.227+0800: 126002.776: [CMS-concurrent-abortable-preclean: 0.092/0.096 secs] [Times: user=1.47 sys=0.03, real=0.09 secs] 

2017-08-02T03:13:10.231+0800: 126002.779: [GC (CMS Final Remark) [YG occupancy: 1850862 K (2146944 K)]126002.779: [Rescan (parallel) , 0.1232905 secs]126002.903: [weak refs processing, 1.1937147 secs]126004.096: [class unloading, 0.0344156 secs]126004.131: [scrub symbol table, 0.0060543 secs]126004.137: [scrub string table, 0.0013539 secs][1 CMS-remark: 10795897K(14391744K)] 12646759K(16538688K), 1.7954516 secs] [Times: user=3.81 sys=0.00, real=1.79 secs] 

2017-08-02T03:13:12.026+0800: 126004.575: [CMS-concurrent-sweep-start]

2017-08-02T03:13:32.581+0800: 126025.129: [CMS-concurrent-sweep: 17.822/20.554 secs] [Times: user=302.97 sys=4.62, real=20.56 secs] 

2017-08-02T03:13:32.581+0800: 126025.129: [CMS-concurrent-reset-start]

2017-08-02T03:13:32.618+0800: 126025.166: [CMS-concurrent-reset: 0.037/0.037 secs] [Times: user=0.59 sys=0.01, real=0.03 secs] 

2017-08-02T04:00:12.924+0800: 128825.472: [CMS-concurrent-mark-start]

2017-08-02T04:00:13.647+0800: 128826.195: [CMS-concurrent-mark: 0.672/0.723 secs] [Times: user=13.46 sys=0.16, real=0.73 secs] 

2017-08-02T04:00:13.647+0800: 128826.196: [CMS-concurrent-preclean-start]

2017-08-02T04:00:14.648+0800: 128827.197: [CMS-concurrent-preclean: 0.902/1.001 secs] [Times: user=13.47 sys=0.22, real=1.00 secs] 

2017-08-02T04:00:14.648+0800: 128827.197: [CMS-concurrent-abortable-preclean-start]

2017-08-02T04:00:14.897+0800: 128827.445: [CMS-concurrent-abortable-preclean: 0.247/0.248 secs] [Times: user=2.77 sys=0.06, real=0.24 secs] 

2017-08-02T04:00:14.905+0800: 128827.453: [GC (CMS Final Remark) [YG occupancy: 1869407 K (2146944 K)]128827.453: [Rescan (parallel) , 0.0987141 secs]128827.552: [weak refs processing, 1.6173477 secs]128829.169: [class unloading, 0.0355365 secs]128829.205: [scrub symbol table, 0.0057739 secs]128829.211: [scrub string table, 0.0013439 secs][1 CMS-remark: 10798371K(14391744K)] 12667779K(16538688K), 2.3460907 secs] [Times: user=4.20 sys=0.01, real=2.35 secs] 

2017-08-02T04:00:17.251+0800: 128829.799: [CMS-concurrent-sweep-start]

2017-08-02T04:00:37.970+0800: 128850.519: [CMS-concurrent-sweep: 17.840/20.719 secs] [Times: user=301.05 sys=4.38, real=20.71 secs] 

2017-08-02T04:00:37.970+0800: 128850.519: [CMS-concurrent-reset-start]

2017-08-02T04:00:38.035+0800: 128850.584: [CMS-concurrent-reset: 0.036/0.065 secs] [Times: user=0.84 sys=0.01, real=0.07 secs] 

2017-08-02T04:52:25.570+0800: 131958.119: [CMS-concurrent-mark-start]

2017-08-02T04:52:26.480+0800: 131959.029: [CMS-concurrent-mark: 0.809/0.910 secs] [Times: user=16.61 sys=0.19, real=0.91 secs] 

2017-08-02T04:52:26.480+0800: 131959.029: [CMS-concurrent-preclean-start]

2017-08-02T04:52:27.521+0800: 131960.070: [CMS-concurrent-preclean: 0.902/1.041 secs] [Times: user=14.51 sys=0.18, real=1.04 secs] 

2017-08-02T04:52:27.521+0800: 131960.070: [CMS-concurrent-abortable-preclean-start]

2017-08-02T04:52:27.774+0800: 131960.322: [CMS-concurrent-abortable-preclean: 0.212/0.253 secs] [Times: user=3.51 sys=0.05, real=0.25 secs] 

2017-08-02T04:52:27.776+0800: 131960.325: [GC (CMS Final Remark) [YG occupancy: 954177 K (2146944 K)]131960.325: [Rescan (parallel) , 0.0780706 secs]131960.403: [weak refs processing, 1.6247166 secs]131962.027: [class unloading, 0.0344207 secs]131962.062: [scrub symbol table, 0.0058886 secs]131962.068: [scrub string table, 0.0013479 secs][1 CMS-remark: 10798781K(14391744K)] 11752958K(16538688K), 2.3393911 secs] [Times: user=3.47 sys=0.01, real=2.34 secs]