用了Elasticsearch,一口气上5T

增加协调节点能缓解 CircuitBreakingException: [parent] Data too large 的熔断问题吗?

Elasticsearch | 作者 hl1221hl | 发布于2020年04月20日 | 阅读数:3201

大并发(查询+插入)时, 节点机器的cpu及load都很低的情况下, 出发了熔断, 导致节点下线shared迁移. 像请教一下, 添加协调节点是否能缓解这中问题呢?
 服务器运行环境信息: 8c/32g Centos
软件版本: 7.4.0

问题发生的上下文信息: 大并发查询及插入时, 节点cpu/load很低, 触发熔断

可以完整复现问题的代码:
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<transport_request>] would be [15802623402/14.7gb], which is larger than the limit of [15679356928/14.6gb], real usage: [15802618848/14.7gb], new bytes reserved: [4554/4.4kb], usages [request=0/0b, fielddata=873/873b, in_flight_requests=4554/4.4kb, accounting=28454346/27.1mb]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1727)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1704)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1467)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:836)
 

补充完整的异常和错误信息:

failed to perform indices:data/write/bulk[s] on replica [index-0][23], node[UID], [R], s[STARTED], a[id=UID] org.elasticsearch.transport.RemoteTransportException: [UID][172.19.0.11:9300][indices:data/write/bulk[s][r]] Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [15748097796/14.6gb], which is larger than the limit of [15679356928/14.6gb], real usage: [15748092896/14.6gb], new bytes reserved: [4900/4.7kb], usages [request=0/0b, fielddata=873/873b, in_flight_requests=5684/5.5kb, accounting=51361686/48.9mb] at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit
已邀请:

locatelli

赞同来自:

这基本上是7.x中一个已知的问题。可以试着关掉“indices.breaker.total.use_real_memory”

hl1221hl

赞同来自:

GC 从 G1 改为 CMS后, 没问题了. 不过GC时cpu峰值很高, 没有G1时平稳

要回复问题请先登录注册