有个人长的像洋葱,走着走着就哭了…….

使用RestHighLevelClient获取Cluster setting信息BLOCK死锁【已解决】

Elasticsearch | 作者 yangjianxuan | 发布于2023年05月18日 | 阅读数:2956

大家好:
有个问题请教一下我使用elasticsearch-rest-high-level-client-6.8.17.jar中提供的RestHighLevelClient获取Cluster setting信息时,线程死锁。
跟踪了一下源码,毫无头绪,请问一下大家有没有什么排查思路可以借鉴。
 
调用代码如下:
return restHighLevelClient.cluster().getSettings(clusterGetSettingsRequest, RequestOptions.DEFAULT);
堆栈信息如下:
Thread 204677: (state = BLOCKED)
 - org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse.lambda$static$1(org.elasticsearch.common.xcontent.XContentParser, java.lang.Void) @bci=1, line=62 (Interpreted frame)
 - org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse$$Lambda$785.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=5 (Interpreted frame)
 - org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObject$1(org.elasticsearch.common.xcontent.ContextParser, org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=3, line=146 (Interpreted frame)
 - org.elasticsearch.common.xcontent.AbstractObjectParser$$Lambda$786.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=6 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$1(java.util.function.BiConsumer, org.elasticsearch.common.xcontent.ContextParser, org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=6, line=213 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ObjectParser$$Lambda$788.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=11 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ObjectParser.parseValue(org.elasticsearch.common.xcontent.XContentParser, org.elasticsearch.common.xcontent.ObjectParser$FieldParser, java.lang.String, java.lang.Object, java.lang.Object) @bci=9, line=314 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ObjectParser.parseSub(org.elasticsearch.common.xcontent.XContentParser, org.elasticsearch.common.xcontent.ObjectParser$FieldParser, java.lang.String, java.lang.Object, java.lang.Object) @bci=80, line=326 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ObjectParser.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=237, line=168 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=16, line=169 (Interpreted frame)
 - org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=3, line=161 (Interpreted frame)
 - org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse.fromXContent(org.elasticsearch.common.xcontent.XContentParser) @bci=5, line=143 (Interpreted frame)
 - org.elasticsearch.client.ClusterClient$$Lambda$775.apply(java.lang.Object) @bci=4 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.parseEntity(org.apache.http.HttpEntity, org.elasticsearch.common.CheckedFunction) @bci=164, line=2061 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.Response) @bci=6, line=1698 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient$$Lambda$751.apply(java.lang.Object) @bci=12 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(java.lang.Object, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=93, line=1782 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.performRequest(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=35, line=1735 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=14, line=1697 (Interpreted frame)
 - org.elasticsearch.client.ClusterClient.getSettings(org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsRequest, org.elasticsearch.client.RequestOptions) @bci=19, line=116 (Interpreted frame)
 
已邀请:

yangjianxuan

赞同来自:

补充一下堆栈图片
 

yangjianxuan

赞同来自:

debug看了下报错信息
IMG_20230518_215138.jpg

yangjianxuan

赞同来自:

我来结案了,搞了两天终于知道问题原因了。
1、还是从堆栈信息入手:
org.elasticsearch.common.settings.Settings
org.elasticsearch.common.settings.Settings$Builder
这两个类在初始化时好像死锁了
 
Thread 18683: (state = BLOCKED)
 - org.elasticsearch.common.settings.Settings.<clinit>() @bci=0, line=84 (Interpreted frame)
 - org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.parseSettingsField(org.elasticsearch.common.xcontent.XContentParser, java.lang.String, java.util.Map, java.util.Map) @bci=115, line=147 (Interpreted frame)
 -
org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.parseIndexEntry(org.elasticsearch.common.xcontent.XContentParser, java.util.Map, java.util.Map) @bci=39, line=166 (Interpreted frame)
 - org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.fromXContent(org.elasticsearch.common.xcontent.XContentParser) @bci=86, line=182 (Interpreted frame)
 - org.elasticsearch.client.IndicesClient$$Lambda$750.apply(java.lang.Object) @bci=4 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.parseEntity(org.apache.http.HttpEntity, org.elasticsearch.common.CheckedFunction) @bci=164, line=2061 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.Response) @bci=6, line=1698 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient$$Lambda$751.apply(java.lang.Object) @bci=12 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(java.lang.Object, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=93, line=1782 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.performRequest(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=35, line=1735 (Interpreted frame)
 - org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=14, line=1697 (Interpreted frame)
 - org.elasticsearch.client.IndicesClient.getSettings(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest, org.elasticsearch.client.RequestOptions) @bci=19, line=858 (Interpreted frame)
 
 
Thread 18681: (state = BLOCKED)
 - org.elasticsearch.common.settings.Settings$Builder.build() @bci=8, line=1242 (Interpreted frame)
 - org.elasticsearch.common.settings.Settings$Builder.<clinit>() @bci=7, line=782 (Interpreted frame)
 - org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest.<init>(java.lang.String) @bci=16, line=56 (Interpreted frame)

yangjianxuan

赞同来自:

2、翻了一下Settings的源码
Setings类初始化时会去new Settings.Builder()
public final class Settings implements ToXContentFragment {
public static final Settings EMPTY = (new Settings.Builder()).build();

build()方法会去new Settings
public static class Builder { public static final Settings EMPTY_SETTINGS = (new Settings.Builder()).build();

public Settings build() {
this.processLegacyLists(this.map);
return new Settings(this.map, (SecureSettings)this.secureSettings.get());
}

yangjianxuan

赞同来自:

3、GetSettingsResponse调用parseSettingsField函数,初始化Settings获取Settings锁,然后去new Builder但builder已经被锁住
private static void parseSettingsField(XContentParser parser, String currentIndexName, Map<String, Settings> indexToSettings, Map<String, Settings> indexToDefaultSettings) throws IOException {
if (parser.currentToken() == Token.START_OBJECT) {
String var4 = parser.currentName();
byte var5 = -1;
switch(var4.hashCode()) {
case 644280914:
if (var4.equals("defaults")) {
var5 = 1;
}
break;
case 1434631203:
if (var4.equals("settings")) {
var5 = 0;
}
}

switch(var5) {
case 0:
indexToSettings.put(currentIndexName, Settings.fromXContent(parser));
break;
case 1:
[b]indexToDefaultSettings.put(currentIndexName, Settings.fromXContent(parser));
[/b] break;
default:
parser.skipChildren();
}
} else if (parser.currentToken() == Token.START_ARRAY) {
parser.skipChildren();
}

parser.nextToken();
}
 
4、UpdateSettingsRequest初始化Builder时将Builder类锁住,调用builder方法时去new Settings需要获取Settings类的锁,但是获取不到,因为已经被GetSettingsResponse的调用parseSettingsField时锁住
public class UpdateSettingsRequest extends AcknowledgedRequest<UpdateSettingsRequest> implements Replaceable, ToXContentObject {
private String[] indices;
private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, true);
private Settings settings;
private boolean preserveExisting;

public UpdateSettingsRequest() {
this.settings = Builder.EMPTY_SETTINGS;
this.preserveExisting = false;
}

public UpdateSettingsRequest(String... indices) {
[b]this.settings = Builder.EMPTY_SETTINGS;
[/b] this.preserveExisting = false;
this.indices = indices;
}
因此,形成死锁
 

yangjianxuan

赞同来自:

编写单元测试验证想法
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.xcontent.XContentParser;
import org.junit.Test;

import java.io.IOException;

/**
* Description:
*
* @author yangjianxuan 2023/5/19
*/
public class YjxTest {

@test
public void test() throws InterruptedException {
#Settings aa= Settings.Builder.EMPTY_SETTINGS;
#System.out.println("hahaha:" + aa);

Thread thread = new Thread() {
@Override
public void run() {
while(true) {
System.out.println("1:");
Settings aa = Settings.Builder.EMPTY_SETTINGS;
System.out.println("11:" + aa);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
};
thread.start();

Thread thread1 = new Thread() {
@Override
public void run() {
while(true) {
System.out.println("2:");
XContentParser parser = null;
try {
Settings.fromXContent(parser);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("22:");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
};
thread1.start();

Thread.sleep(1000000);
}
}
1、此时直接运行就会出现死锁
2、解决方法,将注释的两行代码打开,提前加载Builder类,破坏死锁条件
最后,虽然问题解决了,但是理论还是比较匮乏,希望有懂的大神指点一下,这个锁机制。
以及,有没有直观地定位问题的方法。因为看到堆栈信息,基本就是靠猜,感觉有点误打误撞了。

要回复问题请先登录注册