不为失败找理由,要为成功找方法。

内存溢出问题

Elasticsearch | 作者 jsh0401 | 发布于2017年12月18日 | 阅读数:8591

你们python api用http连接es,连得都是协调节点吗,连多台他会自动负载均衡吧?
插入数据时协调节点为什么压力会那么大老内存溢出?数据难道会先加载到协调节点?
我连4台挂了2台, 4台机器都是配置64g,绝对超不过64g的,不知道是为什么挂的,可能是并发量太大了?
我插入4亿多条数据,用mapreduce, 
if len(actions) == 9000:
    for i in helpers.parallel_bulk(es, actions,
                             thread_count=3, chunk_size=3000):
 
[2017-12-16T22:05:43,000][WARN ][o.e.m.j.JvmGcMonitorService] [es_0 ] [gc][110776] overhead, spent [9.7s] collecting in the last [9.8s]
[2017-12-16T22:05:43,000][DEBUG][o.e.x.m.MonitoringService] [es_0 ] monitoring execution is skipped until previous execution terminated
[2017-12-16T22:05:43,000][DEBUG][i.n.b.PoolThreadCache    ] Freed 10 thread-local buffer(s) from thread: threadDeathWatcher-2-1
[2017-12-16T22:05:43,000][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer
        at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:140)
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:81)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:78)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
        at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:99)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:140)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:745)
[2017-12-16T22:05:43,001][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es_0 ] fatal error in thread [Thread-7], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-12-16T22:05:43,001][WARN ][o.e.h.n.Netty4HttpServerTransport] [es_0 ] caught exception while handling client http traffic, closing connection [id: 0x383a92ab, L:/10.87.126.27:8600 - R:/10.87.195.21:39600]
java.lang.OutOfMemoryError: Java heap space
[2017-12-16T22:05:43,001][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es_0 ] fatal error in thread [Thread-8], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-12-16T22:05:43,001][WARN ][o.e.h.n.Netty4HttpServerTransport] [es_0 ] caught exception while handling client http traffic, closing connection [id: 0x9eee5d49, L:/10.87.126.27:8600 - R:/10.87.247.14:39292]
java.lang.OutOfMemoryError: Java heap space
 
已邀请:

ting44

赞同来自:

你这个4亿条数据奔溃了,还说的过去,我这个30w就莫名挂了,怎么想都想不明白!!

novia - 1&0

赞同来自:

设置的几个ruduce?之前我也是mapreduce插入,设置的reduce太多,还有单条数据较大,导致ES内存溢出

刚学es

赞同来自:

你好,也遇到相同问题,请问解决了吗

jianjianhe

赞同来自:

生成dump日志,可以分析下哪里出现内存泄漏或溢出问题,找出问题所在

要回复问题请先登录注册