ELK，萌萌哒

elasticsearch集群单节点cpu跑满问题

Elasticsearch | 作者 shwtz | 发布于2019年12月13日 | 阅读数：4977
分享到：QQ空间新浪微博微信QQ好友印象笔记有道云笔记
8个数据节点3个主节点集群。某天突然间node-7的cpu和load彪满，导致整个集群写入和查询收到很大的影响，数据都开始延迟了。。。

经过单独的服务器测试，确认服务器的读写io，cpu和内存分配没有硬件上的问题。
第一次跑满的时候发现gc日志很多，但是和其他节点也差不多。
同时发现：这个节点的写入队列即使排到2000，也很少发生rejected，而是一直排着。
其他节点写入队列多的时候一般都是200多就开始rejected了，不知道和这个是不是有关系。

nodes.png

我把数据导入停了，然后load还是上图那样，去看node-7的write队列，都是满的：
node_name name        active queue rejected
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    search           0     0        0
node-7    write            8   568        0
node-1    flush            0     0        0
 
queue的峰值能有2000，但是一个拒绝都没有，奇了怪了。。。
------补充-------------
把写数据的程序打开后，bulk线程很快又满了，然后有rejected发生：
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    generic          3     0        0
node-7    refresh          1     0        0
node-7    search           0     0        0
node-7    write            8   377       11
----------------
 
通过查看该节点的jvm和gc细节，发现从有写入开始，cpu就开始飙升了：

node7_jvm_gc.png

 
 
通过hot_threads API，发现该节点上很多lucene的操作：
::: {4HWpYCG}{4HWpYCGoSJ6pTNa7OcO81g}{Zn-M2TBTSZa2WRD0euXbYA}{:9300}{ml.machine_memory=33605414912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
   Hot threads at 2019-12-12T15:49:38.791Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {node-4}{24tuyouFRWarHtFRu-sG2g}{qsblW_7RSZCYA9eojjLgbw}{:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:38.942Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {node-5}{-DPFdrNaT9i4ZxrAhi75XA}{lL58OwhrSfSyFlwZwDJR4g}{1:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:38.942Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   91.0% (455ms out of 500ms) cpu usage by thread 'elasticsearch[node-5][generic][T#544]'
     2/10 snapshots sharing following 28 elements
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3125/704687181.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     8/10 snapshots sharing following 9 elements
       sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
       java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
       java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
       java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

::: {node-7}{2VK4lPggS1OE3nfCVFtpVQ}{OHNjRmuJSIK0ZJGwkPAkjw}{:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:42.141Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   89.3% (446.7ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][generic][T#4]'
     3/10 snapshots sharing following 25 elements
       org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)
       org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupSeqNo(PerThreadIDVersionAndSeqNoLookup.java:167)
       org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.loadDocIdAndSeqNo(VersionsAndSeqNoResolver.java:164)
       org.elasticsearch.index.engine.InternalEngine.compareOpToLuceneDocBasedOnSeqNo(InternalEngine.java:728)
       org.elasticsearch.index.engine.InternalEngine.planIndexingAsNonPrimary(InternalEngine.java:1022)
       org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1041)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:928)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 33 elements
       org.wltea.analyzer.dic.DictSegment.match(DictSegment.java:93)
       org.wltea.analyzer.dic.Dictionary.matchInMainDict(Dictionary.java:349)
       org.wltea.analyzer.core.CJKSegmenter.analyze(CJKSegmenter.java:80)
       org.wltea.analyzer.core.IKSegmenter.next(IKSegmenter.java:113)
       org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:88)
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 28 elements
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     3/10 snapshots sharing following 16 elements
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
   
   85.6% (428ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][generic][T#2]'
     2/10 snapshots sharing following 51 elements
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:221)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 56 elements
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 50 elements
       org.apache.lucene.util.Sorter.upper(Sorter.java:121)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:86)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     3/10 snapshots sharing following 46 elements
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     unique snapshot
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:218)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
   
   82.8% (414ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][write][T#7]'
     2/10 snapshots sharing following 55 elements
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:229)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 52 elements
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     4/10 snapshots sharing following 49 elements
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 46 elements
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
 
----------------------12-13补充-----------------------------------
今天把集群升级了下，到7.4.2了，然后还是有这个问题，节点线程和队列参数没有调整过。现在有写入的时候还是这个状态：

node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    write            8   413        0
 
到400多的排队了，还不rejected。然后节点的cpu和load都很高，异常地高：

node7_top.png

 
又看了一下，到队列到800多有rejected了：
node-8    flush            0     0        0
node-8    force_merge      0     0        0
node-8    refresh          0     0        0
node-8    write            0     0        0
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    write            8   857        2
node-5    flush            1     0        0
node-5    force_merge      0     0        0
node-5    refresh          1     0        0
node-5    write            4     0     7456
master-1  flush            0     0        0
master-1  force_merge      0     0        0
master-1  refresh          0     0        0
master-1  write            0     0        0
node-3    flush            0     0        0
node-3    force_merge      0     0        0
node-3    refresh          1     0        0
node-3    write            0     0        0
 
可以看到，其他节点的写入都很正常，基本不用排队。
2019-12-13 添加评论

没有找到相关结果

    已邀请:

    locatelli

    赞同来自:

    听起来像是这个节点的thread_pool的size跟别的节点不一样？可以检查一下设置。
    另外写入任何index都是一样的情况吗？如果只是写入特定的index才发生那么可能跟index的设置也有关系。
    2019-12-13 0 1
    分享

    laoyang360 - 《一本书讲透Elasticsearch》作者，Elastic认证工程师 [死磕Elasitcsearch]知识星球地址：http://t.cn/RmwM3N9；微信公众号：铭毅天下; 博客：https://elastic.blog.csdn.net

    赞同来自:

    节点的角色是什么？ 建议增加协调节点，区分节点角色。怀疑这个节点连接的是客户端，充当协调节点的角色。
    这个节点的线程池和队列的关系，有没有参考官方文档进行配置。
     
    2019-12-13 0 1
    分享

    easyice - 张超

    赞同来自:

    这个问题和业务索引有关，写入该节点的索引应该有很多使用外部 id 方式入库的，找一下确认，然后看看该索引的分片分布情况
    2020-07-05 0 0
    分享

    pony_maggie - 公众号:犀牛饲养员的技术笔记

    赞同来自:

    感觉应该是业务数据在这个节点上有热点，看下业务场景，路由那些
    2020-07-07 0 0
    分享

    shwtz - 学物理想做演员的IT男

    赞同来自:

    之后没有遇到过这问题了。
    之前@laoyang360 提到过协调节点的问题。我回想了一下。目前集群是有一个专门的协调节点。我提问的时候，可能还真就没有。所以有可能还真是协调节点的问题。。

    不过当真如此的话，我就有问题了。
    我用python 的elasticsearch客户端，配置es地址的时候，如果配置的ip就是node-7，那么node-7默认成为了协调节点。但是我看源码，实际发送请求的时候，由于我打开了sniff选项，所以客户端实际是连接的es的全部数据节点的连接池，然后使用round-robin的方式进行轮询调用，应该不会导致node-7单节点一直成为协调节点而影响该节点的性能吧。。。
    难道说，配置ip指定的es的节点的地址，就会一直把ta当成协调节点吗。。。
    2020-07-09 0 0
    分享
    为什么被折叠? 0 个回复被折叠

    要回复问题请先登录或注册

    Copyright © 2024 · CC BY-NC-SA 3.0

    本站服务器及带宽由 提供赞助