用了Elasticsearch,一口气上5T

20w条数据频繁出现内存溢出??

Elasticsearch | 作者 ting44 | 发布于2018年01月28日 | 阅读数:4045


2018-01-28T16:07:57,952][WARN ][o.e.t.TransportService   ] [BxDKy7F] Received response for a request that has timed out, sent [36174ms] ago, timed out [8ms] ago, action [cluster:monitor/nodes/stats[n]], node [{BxDKy7F}{BxDKy7F2QGugzNMdsSYKTA}{s-ehXTOLQCu-vvDIWZc5cQ}{192.168.10.106}{192.168.10.106:9300}], id [26936]
[2018-01-28T16:08:29,109][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][old][12981][40] duration [1m], collections [2]/[1m], total [1m]/[19.7m], memory [5.8gb]->[5.8gb]/[5.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [108.1mb]->[107.1mb]/[108.1mb]}{[old] [4.9gb]->[4.9gb]/[4.9gb]}
[2018-01-28T16:08:29,109][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][12981] overhead, spent [1m] collecting in the last [1m]
[2018-01-28T16:10:25,397][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][old][12982][41] duration [32.6s], collections [1]/[32.6s], total [32.6s]/[20.3m], memory [5.8gb]->[5.8gb]/[5.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [107.1mb]->[107.9mb]/[108.1mb]}{[old] [4.9gb]->[4.9gb]/[4.9gb]}
[2018-01-28T16:12:29,039][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][12982] overhead, spent [32.6s] collecting in the last [32.6s]
[2018-01-28T16:28:30,837][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][old][12983][78] duration [16.2m], collections [37]/[18m], total [16.2m]/[36.5m], memory [5.8gb]->[5.8gb]/[5.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [107.9mb]->[108mb]/[108.1mb]}{[old] [4.9gb]->[4.9gb]/[4.9gb]}
[2018-01-28T16:28:30,851][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][12983] overhead, spent [16.2m] collecting in the last [18m]
[2018-01-28T16:29:32,056][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][old][12984][83] duration [2.4m], collections [5]/[2.4m], total [2.4m]/[38.9m], memory [5.8gb]->[5.8gb]/[5.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [108mb]->[107.1mb]/[108.1mb]}{[old] [4.9gb]->[4.9gb]/[4.9gb]}
[2018-01-28T16:29:32,058][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][12984] overhead, spent [2.4m] collecting in the last [2.4m]
[2018-01-28T16:29:58,816][WARN ][o.e.t.TransportService   ] [BxDKy7F] Received response for a request that has timed out, sent [87918ms] ago, timed out [26757ms] ago, action [cluster:monitor/nodes/stats[n]], node [{BxDKy7F}{BxDKy7F2QGugzNMdsSYKTA}{s-ehXTOLQCu-vvDIWZc5cQ}{192.168.10.106}{192.168.10.106:9300}], id [26988]
[2018-01-28T16:30:34,849][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][old][12985][84] duration [26.7s], collections [1]/[26.7s], total [26.7s]/[39.4m], memory [5.8gb]->[5.8gb]/[5.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [107.1mb]->[106.6mb]/[108.1mb]}{[old] [4.9gb]->[4.9gb]/[4.9gb]}
[2018-01-28T16:30:34,851][WARN ][o.e.m.j.JvmGcMonitorService] [BxDKy7F] [gc][12985] overhead, spent [26.7s] collecting in the last [26.7s]
[2018-01-28T16:29:58,794][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [BxDKy7F] failed to execute on node [BxDKy7F2QGugzNMdsSYKTA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [BxDKy7F][192.168.10.106:9300][cluster:monitor/nodes/stats[n]] request_id [26988] timed out after [61161ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:925) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-01-28T16:23:18,804][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [BxDKy7F] failed to execute on node [BxDKy7F2QGugzNMdsSYKTA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [BxDKy7F][192.168.10.106:9300][cluster:monitor/nodes/stats[n]] request_id [26936] timed out after [36166ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:925) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2018-01-28T16:29:00,741][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[BxDKy7F][flush][T#14]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.DirectMonotonicWriter.<init>(DirectMonotonicWriter.java:56) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.util.packed.DirectMonotonicWriter.getInstance(DirectMonotonicWriter.java:136) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addBinaryField(Lucene54DocValuesConsumer.java:398) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addTermsDict(Lucene54DocValuesConsumer.java:443) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addSortedField(Lucene54DocValuesConsumer.java:578) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.addSortedSetField(Lucene54DocValuesConsumer.java:618) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:131) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:182) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:258) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:142) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:444) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3007) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3242) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3205) ~[lucene-core-6.5.0.jar:6.5.0 4b16c9a10c3c00cafaf1fc92ec3276a7bc7b8c95 - jimczi - 2017-03-21 20:40:22]
at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1540) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.engine.InternalEngine.flush(InternalEngine.java:1036) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.shard.IndexShard.flush(IndexShard.java:780) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.indices.flush.SyncedFlushService.performPreSyncedFlush(SyncedFlushService.java:414) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.indices.flush.SyncedFlushService.access$1000(SyncedFlushService.java:70) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:696) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.indices.flush.SyncedFlushService$PreSyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:692) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:627) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
这个es里面只有20多w条文档,总共才230m,我给es分配了6g堆内存,想不明白还会出现内存溢出!!!
已邀请:

laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

批量入库的吗?批量值大了,可能

要回复问题请先登录注册