es集群节点失败

昨天晚上es集群节点down机 查看了一下日志 检出了WARN 大神能帮忙看下吗
 
 
[2019-01-09T23:16:29,013][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494215] overhead, spent [37s] collecting in the last [37.6s]
[2019-01-09T23:17:08,642][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494216][144] duration [38.8s], collections [1]/[39.6s], total [38.8s]/[15m], memory [15.1gb]->[15.2gb]/[15.8gb], all_pools {[young] [205.4mb]->[287.4mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:17:08,642][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494216] overhead, spent [38.8s] collecting in the last [39.6s]
[2019-01-09T23:17:46,084][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494217][145] duration [36.8s], collections [1]/[37.4s], total [36.8s]/[15.7m], memory [15.2gb]->[15.2gb]/[15.8gb], all_pools {[young] [287.4mb]->[295.1mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:17:46,084][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494217] overhead, spent [36.8s] collecting in the last [37.4s]
[2019-01-09T23:18:29,314][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494218][146] duration [42.4s], collections [1]/[43.2s], total [42.4s]/[16.4m], memory [15.2gb]->[15.2gb]/[15.8gb], all_pools {[young] [295.1mb]->[307.2mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:18:29,314][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494218] overhead, spent [42.4s] collecting in the last [43.2s]
[2019-01-09T23:19:08,252][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494219][147] duration [38.2s], collections [1]/[38.9s], total [38.2s]/[17m], memory [15.2gb]->[15.2gb]/[15.8gb], all_pools {[young] [307.2mb]->[326.6mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:19:08,252][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494219] overhead, spent [38.2s] collecting in the last [38.9s]
[2019-01-09T23:19:52,306][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494220][148] duration [43.3s], collections [1]/[44s], total [43.3s]/[17.7m], memory [15.2gb]->[15.3gb]/[15.8gb], all_pools {[young] [326.6mb]->[395.4mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:19:52,306][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494220] overhead, spent [43.3s] collecting in the last [44s]
[2019-01-09T23:20:30,964][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494221][149] duration [38s], collections [1]/[38.6s], total [38s]/[18.4m], memory [15.3gb]->[15.3gb]/[15.8gb], all_pools {[young] [395.4mb]->[378mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:20:30,964][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494221] overhead, spent [38s] collecting in the last [38.6s]
[2019-01-09T23:21:11,134][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494222][150] duration [39.5s], collections [1]/[40.1s], total [39.5s]/[19m], memory [15.3gb]->[15.3gb]/[15.8gb], all_pools {[young] [378mb]->[376.6mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:21:11,134][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494222] overhead, spent [39.5s] collecting in the last [40.1s]
[2019-01-09T23:21:50,225][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494223][151] duration [38.4s], collections [1]/[39s], total [38.4s]/[19.7m], memory [15.3gb]->[15.3gb]/[15.8gb], all_pools {[young] [376.6mb]->[384mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:21:50,255][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494223] overhead, spent [38.4s] collecting in the last [39s]
[2019-01-09T23:22:33,915][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][old][10494224][152] duration [42.8s], collections [1]/[43.6s], total [42.8s]/[20.4m], memory [15.3gb]->[15.3gb]/[15.8gb], all_pools {[young] [384mb]->[407.7mb]/[865.3mb]}{[survivor] [0b]->[0b]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2019-01-09T23:22:33,915][WARN ][o.e.m.j.JvmGcMonitorService] [vm172-18-66-204.ksc.com] [gc][10494224] overhead, spent [42.8s] collecting in the last [43.6s]
[2019-01-09T23:22:33,938][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,942][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,943][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,943][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,943][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,956][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,957][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,957][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,957][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
[2019-01-09T23:22:33,958][WARN ][o.e.t.n.Netty4Transport  ] [vm172-18-66-204.ksc.com] write and flush on the network layer failed (channel: [id: 0x67085a71, L:0.0.0.0/0.0.0.0:33998 ! R:172.18.66.205/172.18.66.205:9300])
 
 
 
前面反复出现了这些错误信息 
[2019-01-09T22:56:07,740][DEBUG][o.e.a.s.TransportSearchAction] [vm172-18-66-204.ksc.com] [cposorder_20180908][1], node[vw122wIJRiGUhFry56ew5Q], [R], s[STARTED], a[id=VRYE2HhhQ2qBEdfgaTWC
cQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[cposorder], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildca
rds_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[PS1723], routing='null', preference='null', requestCache=null, s
croll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, source={
  "from" : 0,
  "size" : 1,
  "query" : {
    "bool" : {
      "must" : [
        {
          "constant_score" : {
            "filter" : {
              "terms" : {
                "orderNumber" : [
                  "252801070203201901091546997550301210480941219",
                  "252801070203201901091547006146031423921070047",
                  "252801070203201901091546999246039957329644313",
                  "252801070203201901091547004746438650160560850",
                  "252801070203201901091547025346183134822062430",
                  "252801070203201901091547031946145740796005842",
                  "252801070203201901091547030901021443721197923"
                ],
                "boost" : 1.0
              }
            },
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "match" : {
            "orderNumber" : {
              "query" : "252801070203201901091547006146031423921070047",
              "operator" : "OR",
              "prefix_length" : 0,
              "max_expansions" : 50,
              "fuzzy_transpositions" : true,
              "lenient" : false,
              "zero_terms_query" : "NONE",
              "boost" : 1.0
            }
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "explain" : false,
  "sort" : [
    {
      "orderType" : {
        "order" : "desc"
      }
    }
  ]
}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [vm172-18-66-205.ksc.com][172.18.66.205:9300][indices:data/read/search[can_match]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@2919cbfe on EsThreadPoolExe
cutor[search, queue capacity = 2000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@44add53b[Running, pool size = 80, active threads = 80, queued tasks = 2000, completed ta
sks = 2697856722]]