用了Elasticsearch,一口气上5T

elasticsearch集群单节点cpu跑满问题

Elasticsearch | 作者 shwtz | 发布于2019年12月13日 | 阅读数:933

8个数据节点3个主节点集群。某天突然间node-7的cpu和load彪满,导致整个集群写入和查询收到很大的影响,数据都开始延迟了。。。

经过单独的服务器测试,确认服务器的读写io,cpu和内存分配没有硬件上的问题。
第一次跑满的时候发现gc日志很多,但是和其他节点也差不多。
同时发现:这个节点的写入队列即使排到2000,也很少发生rejected,而是一直排着。
其他节点写入队列多的时候一般都是200多就开始rejected了,不知道和这个是不是有关系。

nodes.png

我把数据导入停了,然后load还是上图那样,去看node-7的write队列,都是满的:
node_name name        active queue rejected
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    search           0     0        0
node-7    write            8   568        0
node-1    flush            0     0        0
 
queue的峰值能有2000,但是一个拒绝都没有,奇了怪了。。。
------补充-------------
把写数据的程序打开后,bulk线程很快又满了,然后有rejected发生:
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    generic          3     0        0
node-7    refresh          1     0        0
node-7    search           0     0        0
node-7    write            8   377       11
----------------
 
通过查看该节点的jvm和gc细节,发现从有写入开始,cpu就开始飙升了:

node7_jvm_gc.png

 
 
通过hot_threads API,发现该节点上很多lucene的操作:
::: {4HWpYCG}{4HWpYCGoSJ6pTNa7OcO81g}{Zn-M2TBTSZa2WRD0euXbYA}{:9300}{ml.machine_memory=33605414912, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
   Hot threads at 2019-12-12T15:49:38.791Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {node-4}{24tuyouFRWarHtFRu-sG2g}{qsblW_7RSZCYA9eojjLgbw}{:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:38.942Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {node-5}{-DPFdrNaT9i4ZxrAhi75XA}{lL58OwhrSfSyFlwZwDJR4g}{1:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:38.942Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   91.0% (455ms out of 500ms) cpu usage by thread 'elasticsearch[node-5][generic][T#544]'
     2/10 snapshots sharing following 28 elements
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3125/704687181.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     8/10 snapshots sharing following 9 elements
       sun.misc.Unsafe.park(Native Method)
       java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
       java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
       java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
       java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)

::: {node-7}{2VK4lPggS1OE3nfCVFtpVQ}{OHNjRmuJSIK0ZJGwkPAkjw}{:9300}{xpack.installed=true}
   Hot threads at 2019-12-12T15:49:42.141Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   89.3% (446.7ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][generic][T#4]'
     3/10 snapshots sharing following 25 elements
       org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:507)
       org.elasticsearch.common.lucene.uid.PerThreadIDVersionAndSeqNoLookup.lookupSeqNo(PerThreadIDVersionAndSeqNoLookup.java:167)

       org.elasticsearch.common.lucene.uid.VersionsAndSeqNoResolver.loadDocIdAndSeqNo(VersionsAndSeqNoResolver.java:164)
       org.elasticsearch.index.engine.InternalEngine.compareOpToLuceneDocBasedOnSeqNo(InternalEngine.java:728)
       org.elasticsearch.index.engine.InternalEngine.planIndexingAsNonPrimary(InternalEngine.java:1022)
       org.elasticsearch.index.engine.InternalEngine.indexingStrategyForOperation(InternalEngine.java:1041)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:928)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 33 elements
       org.wltea.analyzer.dic.DictSegment.match(DictSegment.java:93)
       org.wltea.analyzer.dic.Dictionary.matchInMainDict(Dictionary.java:349)
       org.wltea.analyzer.core.CJKSegmenter.analyze(CJKSegmenter.java:80)
       org.wltea.analyzer.core.IKSegmenter.next(IKSegmenter.java:113)
       org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:88)
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 28 elements
       org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:793)
       org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430)
       org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
       org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     3/10 snapshots sharing following 16 elements
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
   
   85.6% (428ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][generic][T#2]'
     2/10 snapshots sharing following 51 elements
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:221)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)

       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 56 elements
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)

       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 50 elements
       org.apache.lucene.util.Sorter.upper(Sorter.java:121)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:86)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     3/10 snapshots sharing following 46 elements
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)

       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     unique snapshot
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:218)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.
java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:417)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:514)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1377)
       org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1364)
       org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$2(RecoveryTarget.java:363)
       org.elasticsearch.indices.recovery.RecoveryTarget$$Lambda$3113/2135080965.get(Unknown Source)
       org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:197)
       org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:338)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:518)
       org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.messageReceived(PeerRecoveryTargetService.java:478)
       org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
   
   82.8% (414ms out of 500ms) cpu usage by thread 'elasticsearch[node-7][write][T#7]'
     2/10 snapshots sharing following 55 elements
       org.apache.lucene.util.TimSorter.doRotate(TimSorter.java:229)
       org.apache.lucene.util.Sorter.rotate(Sorter.java:164)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:89)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 52 elements
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:92)
       org.apache.lucene.util.Sorter.mergeInPlace(Sorter.java:91)
       org.apache.lucene.util.TimSorter.merge(TimSorter.java:188)
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     4/10 snapshots sharing following 49 elements
       org.apache.lucene.util.TimSorter.mergeAt(TimSorter.java:169)
       org.apache.lucene.util.TimSorter.ensureInvariants(TimSorter.java:144)
       org.apache.lucene.util.TimSorter.sort(TimSorter.java:200)
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
     2/10 snapshots sharing following 46 elements
       org.apache.lucene.index.SortingLeafReader$SortingPostingsEnum.<init>(SortingLeafReader.java:908)
       org.apache.lucene.index.SortingLeafReader$SortingTermsEnum.postings(SortingLeafReader.java:135)
       org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:122)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
       org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
       org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:95)
       org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:166)
       org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
       org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
       org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:401)
       org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:479)
       org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
       org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
       org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1175)
       org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120)
       org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793)
       org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:753)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:420)
       org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:388)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:375)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:82)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:645)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:605)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2783)
       org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
       org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
       org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2721)
       org.elasticsearch.index.shard.IndexShard$$Lambda$3287/1241864703.accept(Unknown Source)
       org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2828)
       org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2720)
       org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1009)
       org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:715)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:588)
       org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:567)
       org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
       org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       java.lang.Thread.run(Thread.java:748)
 
----------------------12-13补充-----------------------------------
今天把集群升级了下,到7.4.2了,然后还是有这个问题,节点线程和队列参数没有调整过。现在有写入的时候还是这个状态:

node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    write            8   413        0
 
到400多的排队了,还不rejected。然后节点的cpu和load都很高,异常地高:

node7_top.png

 
又看了一下,到队列到800多有rejected了:
node-8    flush            0     0        0
node-8    force_merge      0     0        0
node-8    refresh          0     0        0
node-8    write            0     0        0
node-7    flush            0     0        0
node-7    force_merge      0     0        0
node-7    refresh          0     0        0
node-7    write            8   857        2
node-5    flush            1     0        0
node-5    force_merge      0     0        0
node-5    refresh          1     0        0
node-5    write            4     0     7456
master-1  flush            0     0        0
master-1  force_merge      0     0        0
master-1  refresh          0     0        0
master-1  write            0     0        0
node-3    flush            0     0        0
node-3    force_merge      0     0        0
node-3    refresh          1     0        0
node-3    write            0     0        0
 
可以看到,其他节点的写入都很正常,基本不用排队。
已邀请:

locatelli

赞同来自:

听起来像是这个节点的thread_pool的size跟别的节点不一样?可以检查一下设置。
另外写入任何index都是一样的情况吗?如果只是写入特定的index才发生那么可能跟index的设置也有关系。

laoyang360 - Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

节点的角色是什么? 建议增加协调节点,区分节点角色。怀疑这个节点连接的是客户端,充当协调节点的角色。
这个节点的线程池和队列的关系,有没有参考官方文档进行配置。
 

easyice - 张超

赞同来自:

这个问题和业务索引有关,写入该节点的索引应该有很多使用外部 id 方式入库的,找一下确认,然后看看该索引的分片分布情况

pony_maggie - 公众号:犀牛饲养员的技术笔记

赞同来自:

感觉应该是业务数据在这个节点上有热点,看下业务场景,路由那些

shwtz - 学物理想做演员的IT男

赞同来自:

之后没有遇到过这问题了。
之前@laoyang360 提到过协调节点的问题。我回想了一下。目前集群是有一个专门的协调节点。我提问的时候,可能还真就没有。所以有可能还真是协调节点的问题。。

不过当真如此的话,我就有问题了。
我用python 的elasticsearch客户端,配置es地址的时候,如果配置的ip就是node-7,那么node-7默认成为了协调节点。但是我看源码,实际发送请求的时候,由于我打开了sniff选项,所以客户端实际是连接的es的全部数据节点的连接池,然后使用round-robin的方式进行轮询调用,应该不会导致node-7单节点一直成为协调节点而影响该节点的性能吧。。。
难道说,配置ip指定的es的节点的地址,就会一直把ta当成协调节点吗。。。

要回复问题请先登录注册