搜索结果正在快递途中

ES集群多分片下单个node的cpu飙高持续不下

Elasticsearch | 作者 econben | 发布于2019年09月16日 | 阅读数:2245

目前在压测环境对2分片6副本的ES进行模拟生产环境的压测,总共7个node,发现其中一个node的cpu飙高到100%,持续不下,其余node的cpu在75%左右并且持续在这个上下波动。
1)docs数据量总共2G左右,没有明显的数据偏移的情况
2)压测进行的时候,没有index的写入操作,也就是说只有查询的请求到ES集群
3)ES集群 version 5.2
4)线程池的配置参数如下
thread_pool.bulk.queue_size: 4096
thread_pool.index.queue_size: 10240
thread_pool.search.queue_size: 20480
thread_pool.get.queue_size: 10240
5)附件是挑选了两个不同的业务api接口进行压测时抓取的stack信息
 
6)如下是出现次数较多的线程stack。抓取的全部stack详见附件。
Thread 6694: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=209 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)
 - org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking() @bci=8, line=251 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.ReleasableLock.close() @bci=4, line=49 (Compiled frame)
 - org.elasticsearch.index.engine.InternalEngine.refresh(java.lang.String) @bci=48, line=647 (Compiled frame)
 - org.elasticsearch.index.engine.InternalEngine.get(org.elasticsearch.index.engine.Engine$Get, java.util.function.Function) @bci=160, line=343 (Compiled frame)
 - org.elasticsearch.index.shard.IndexShard.get(org.elasticsearch.index.engine.Engine$Get) @bci=15, line=612 (Compiled frame)
 - org.elasticsearch.index.get.ShardGetService.innerGet(java.lang.String, java.lang.String, java.lang.String[], boolean, long, org.elasticsearch.index.VersionType, org.elasticsearch.search.fetch.subphase.FetchSourceContext) @bci=222, line=165 (Compiled frame)
 - org.elasticsearch.index.get.ShardGetService.get(java.lang.String, java.lang.String, java.lang.String[], boolean, long, org.elasticsearch.index.VersionType, org.elasticsearch.search.fetch.subphase.FetchSourceContext) @bci=24, line=83 (Compiled frame)
 - org.elasticsearch.action.update.UpdateHelper.prepare(org.elasticsearch.action.update.UpdateRequest, org.elasticsearch.index.shard.IndexShard, java.util.function.LongSupplier) @bci=48, line=78 (Compiled frame)
 - org.elasticsearch.action.update.TransportUpdateAction.shardOperation(org.elasticsearch.action.update.UpdateRequest, org.elasticsearch.action.ActionListener, int) @bci=53, line=179 (Compiled frame)
 - org.elasticsearch.action.update.TransportUpdateAction.shardOperation(org.elasticsearch.action.update.UpdateRequest, org.elasticsearch.action.ActionListener) @bci=4, line=172 (Compiled frame)
 - org.elasticsearch.action.update.TransportUpdateAction.shardOperation(org.elasticsearch.action.support.single.instance.InstanceShardOperationRequest, org.elasticsearch.action.ActionListener) @bci=6, line=69 (Compiled frame)
 - org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(org.elasticsearch.action.support.single.instance.InstanceShardOperationRequest, org.elasticsearch.transport.TransportChannel) @bci=14, line=246 (Compiled frame)
 - org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(org.elasticsearch.transport.TransportRequest, org.elasticsearch.transport.TransportChannel) @bci=6, line=242 (Compiled frame)
 - org.elasticsearch.transport.TransportRequestHandler.messageReceived(org.elasticsearch.transport.TransportRequest, org.elasticsearch.transport.TransportChannel, org.elasticsearch.tasks.Task) @bci=3, line=33 (Compiled frame)
 - org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(org.elasticsearch.transport.TransportRequest, org.elasticsearch.transport.TransportChannel) @bci=59, line=69 (Compiled frame)
 - org.elasticsearch.transport.TcpTransport$RequestHandler.doRun() @bci=12, line=1488 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun() @bci=28, line=596 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.AbstractRunnable.run() @bci=1, line=37 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1149 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
 
Thread 18507: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1269 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=161 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
已邀请:

要回复问题请先登录注册