RemoteTransportException[[node_xxx.xxx.xxx][xxx.xxx.xxx.xxx:xxxx][indices:data/read/search[phase/query]]]; nested: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] wo
uld be larger than limit of [5569511424/5.1gb]];
Caused by: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] would be larger than limit of [5569511424/5.1gb]]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.common.util.BigArrays.adjustBreaker(BigArrays.java:396)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:433)
at org.elasticsearch.common.util.BigArrays.newDoubleArray(BigArrays.java:640)
at org.elasticsearch.search.aggregations.metrics.sum.SumAggregator.<init>(SumAggregator.java:58)
at org.elasticsearch.search.aggregations.metrics.sum.SumAggregator$Factory.doCreateInternal(SumAggregator.java:124)
at org.elasticsearch.search.aggregations.metrics.sum.SumAggregator$Factory.doCreateInternal(SumAggregator.java:108)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:76)
at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:69)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregator.<init>(TermsAggregator.java:141)
at org.elasticsearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.<init>(AbstractStringTermsAggregator.java:40)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.<init>(GlobalOrdinalsStringTermsAggregator.java:79)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$WithHash.<init>(GlobalOrdinalsStringTermsAggregator.java:275)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$ExecutionMode$3.create(TermsAggregatorFactory.java:92)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:243)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactory.asMultiBucketAggregator(AggregatorFactory.java:119)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:196)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:76)
at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:69)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregator.<init>(TermsAggregator.java:141)
at org.elasticsearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.<init>(AbstractStringTermsAggregator.java:40)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.<init>(GlobalOrdinalsStringTermsAggregator.java:79)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$WithHash.<init>(GlobalOrdinalsStringTermsAggregator.java:275)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$ExecutionMode$3.create(TermsAggregatorFactory.java:92)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:243)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactory.asMultiBucketAggregator(AggregatorFactory.java:119)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:196)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:76)
at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:69)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregator.<init>(TermsAggregator.java:141)
at org.elasticsearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.<init>(AbstractStringTermsAggregator.java:40)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.<init>(GlobalOrdinalsStringTermsAggregator.java:79)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator$WithHash.<init>(GlobalOrdinalsStringTermsAggregator.java:275)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$ExecutionMode$3.create(TermsAggregatorFactory.java:92)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:243)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:76)
at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:69)
at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregator.<init>(TermsAggregator.java:141)
at org.elasticsearch.search.aggregations.bucket.terms.AbstractStringTermsAggregator.<init>(AbstractStringTermsAggregator.java:40)
at org.elasticsearch.search.aggregations.bucket.terms.GlobalOrdinalsStringTermsAggregator.<init>(GlobalOrdinalsStringTermsAggregator.java:79)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory$ExecutionMode$2.create(TermsAggregatorFactory.java:72)
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:243)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:64)
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:102)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:87)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:78)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:104)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
6 个回复
kennywu76 - Wood
赞同来自: novia 、famoss
我看了下代码,ES里很多计算,包括terms aggregation,都是通过BigArrays这个工具类申请大数组来存放计算数据的。 不管申请什么类型数据(long,int,byte, etc..)的数组,都会在实际分配内存之前,调用adjustBreaker()方法,将新申请数组的大小交给breaker service来判断是否应该熔断。
这个方法内部会调用
breaker.addEstimateBytesAndMaybeBreak, 传入新申请的数组字节大小,作为一个增量值delta计入总的request的总字节数里面,然后判断是否超过request breaker设置的上限。 如果超过则触发一个"<reused_arrays>"类型的异常,也就是你贴的日志里看到的错误信息:
breaker service对于一个node来说是全局的,也就是说这个结点上所有申请的bigarray的内存消耗会进行累计。
另外这些申请的大数组,都是继承自AbstractArray这个抽象类,这个类是releasable的,其close()方法里会调用
bigArrays.adjustBreaker将释放的内存从breaker的记录里减去。
因此breaker是以所有进行中的查询聚合生成持有的bigarrays占用的内存为基础,再将新的内存申请累加上去,判断是否应该触发熔断掉,从而尽量避免内存消耗过大导致结点挂掉。
根据你描述的有一段时间一直熔断的特征,要么是当时一直有大内存消耗的聚合请求在进来,要么是有大聚合耗时非常长,导致申请的bigarrays长时间被占用没有释放,并且接近request breaker临界值,致使新进来的聚合不断的被熔断。重启就好了,可能只是因为释放掉了这些进行中的高消耗查询。
建议下次再遇到问题时,用hot threads api看一下,高消耗的线程在做什么。
PS. 我看的代码是5.6的, 不排除早期版本熔断存在一些不完善的地方。
kennywu76 - Wood
赞同来自:
novia - 1&0
赞同来自:
novia - 1&0
赞同来自:
Caused by: CircuitBreakingException[[request] Data too large, data for [<reused_arrays>] would be larger than limit of [5569511424/5.1gb]]
栈信息应该是下面的(抱歉,当时日志没有及时抓到):
novia - 1&0
赞同来自:
从这监控看,确实每次请求之后会释放,但是从上面的warn日志看,es为什么会有个监控一直检测这个值呢?难道有一次查询超出后,就会出问题么
weizijun - elasticsearch fan
赞同来自: