使用RestHighLevelClient获取Cluster setting信息BLOCK死锁【已解决】

Elasticsearch | 作者 yangjianxuan | 发布于2023年05月18日 | 阅读数：2558

大家好：
有个问题请教一下我使用elasticsearch-rest-high-level-client-6.8.17.jar中提供的RestHighLevelClient获取Cluster setting信息时，线程死锁。
跟踪了一下源码，毫无头绪，请问一下大家有没有什么排查思路可以借鉴。

调用代码如下：

return restHighLevelClient.cluster().getSettings(clusterGetSettingsRequest, RequestOptions.DEFAULT);

堆栈信息如下：
Thread 204677: (state = BLOCKED)
- org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse.lambda$static$1(org.elasticsearch.common.xcontent.XContentParser, java.lang.Void) @bci=1, line=62 (Interpreted frame)
- org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse$$Lambda$785.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=5 (Interpreted frame)
- org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObject$1(org.elasticsearch.common.xcontent.ContextParser, org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=3, line=146 (Interpreted frame)
- org.elasticsearch.common.xcontent.AbstractObjectParser$$Lambda$786.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=6 (Interpreted frame)
- org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$1(java.util.function.BiConsumer, org.elasticsearch.common.xcontent.ContextParser, org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=6, line=213 (Interpreted frame)
- org.elasticsearch.common.xcontent.ObjectParser$$Lambda$788.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=11 (Interpreted frame)
- org.elasticsearch.common.xcontent.ObjectParser.parseValue(org.elasticsearch.common.xcontent.XContentParser, org.elasticsearch.common.xcontent.ObjectParser$FieldParser, java.lang.String, java.lang.Object, java.lang.Object) @bci=9, line=314 (Interpreted frame)
- org.elasticsearch.common.xcontent.ObjectParser.parseSub(org.elasticsearch.common.xcontent.XContentParser, org.elasticsearch.common.xcontent.ObjectParser$FieldParser, java.lang.String, java.lang.Object, java.lang.Object) @bci=80, line=326 (Interpreted frame)
- org.elasticsearch.common.xcontent.ObjectParser.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object, java.lang.Object) @bci=237, line=168 (Interpreted frame)
- org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=16, line=169 (Interpreted frame)
- org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(org.elasticsearch.common.xcontent.XContentParser, java.lang.Object) @bci=3, line=161 (Interpreted frame)
- org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsResponse.fromXContent(org.elasticsearch.common.xcontent.XContentParser) @bci=5, line=143 (Interpreted frame)
- org.elasticsearch.client.ClusterClient$$Lambda$775.apply(java.lang.Object) @bci=4 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.parseEntity(org.apache.http.HttpEntity, org.elasticsearch.common.CheckedFunction) @bci=164, line=2061 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.Response) @bci=6, line=1698 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient$$Lambda$751.apply(java.lang.Object) @bci=12 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(java.lang.Object, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=93, line=1782 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.performRequest(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=35, line=1735 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=14, line=1697 (Interpreted frame)
- org.elasticsearch.client.ClusterClient.getSettings(org.elasticsearch.action.admin.cluster.settings.ClusterGetSettingsRequest, org.elasticsearch.client.RequestOptions) @bci=19, line=116 (Interpreted frame)

6 个回复

yangjianxuan

补充一下堆栈图片

yangjianxuan

debug看了下报错信息

yangjianxuan

我来结案了，搞了两天终于知道问题原因了。
1、还是从堆栈信息入手：
org.elasticsearch.common.settings.Settings
org.elasticsearch.common.settings.Settings$Builder
这两个类在初始化时好像死锁了

Thread 18683: (state = BLOCKED)
- org.elasticsearch.common.settings.Settings.<clinit>() @bci=0, line=84 (Interpreted frame)
- org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.parseSettingsField(org.elasticsearch.common.xcontent.XContentParser, java.lang.String, java.util.Map, java.util.Map) @bci=115, line=147 (Interpreted frame)
- org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.parseIndexEntry(org.elasticsearch.common.xcontent.XContentParser, java.util.Map, java.util.Map) @bci=39, line=166 (Interpreted frame)
- org.elasticsearch.action.admin.indices.settings.get.GetSettingsResponse.fromXContent(org.elasticsearch.common.xcontent.XContentParser) @bci=86, line=182 (Interpreted frame)
- org.elasticsearch.client.IndicesClient$$Lambda$750.apply(java.lang.Object) @bci=4 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.parseEntity(org.apache.http.HttpEntity, org.elasticsearch.common.CheckedFunction) @bci=164, line=2061 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$13(org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.Response) @bci=6, line=1698 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient$$Lambda$751.apply(java.lang.Object) @bci=12 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(java.lang.Object, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=93, line=1782 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.performRequest(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=35, line=1735 (Interpreted frame)
- org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(org.elasticsearch.action.ActionRequest, org.elasticsearch.common.CheckedFunction, org.elasticsearch.client.RequestOptions, org.elasticsearch.common.CheckedFunction, java.util.Set) @bci=14, line=1697 (Interpreted frame)
- org.elasticsearch.client.IndicesClient.getSettings(org.elasticsearch.action.admin.indices.settings.get.GetSettingsRequest, org.elasticsearch.client.RequestOptions) @bci=19, line=858 (Interpreted frame)

Thread 18681: (state = BLOCKED)
- org.elasticsearch.common.settings.Settings$Builder.build() @bci=8, line=1242 (Interpreted frame)
- org.elasticsearch.common.settings.Settings$Builder.<clinit>() @bci=7, line=782 (Interpreted frame)
- org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest.<init>(java.lang.String) @bci=16, line=56 (Interpreted frame)

yangjianxuan

2、翻了一下Settings的源码
Setings类初始化时会去new Settings.Builder()

public final class Settings implements ToXContentFragment {

    public static final Settings EMPTY = (new Settings.Builder()).build();

build()方法会去new Settings

public static class Builder { public static final Settings EMPTY_SETTINGS = (new Settings.Builder()).build();

public Settings build() {

            this.processLegacyLists(this.map);

            return new Settings(this.map, (SecureSettings)this.secureSettings.get());

        }

yangjianxuan

3、GetSettingsResponse调用parseSettingsField函数，初始化Settings获取Settings锁，然后去new Builder但builder已经被锁住

private static void parseSettingsField(XContentParser parser, String currentIndexName, Map<String, Settings> indexToSettings, Map<String, Settings> indexToDefaultSettings) throws IOException {

        if (parser.currentToken() == Token.START_OBJECT) {

            String var4 = parser.currentName();

            byte var5 = -1;

            switch(var4.hashCode()) {

            case 644280914:

                if (var4.equals("defaults")) {

                    var5 = 1;

                }

                break;

            case 1434631203:

                if (var4.equals("settings")) {

                    var5 = 0;

                }

            }



            switch(var5) {

            case 0:

                indexToSettings.put(currentIndexName, Settings.fromXContent(parser));

                break;

            case 1:

                [b]indexToDefaultSettings.put(currentIndexName, Settings.fromXContent(parser));

[/b]                break;

            default:

                parser.skipChildren();

            }

        } else if (parser.currentToken() == Token.START_ARRAY) {

            parser.skipChildren();

        }



        parser.nextToken();

    }

4、UpdateSettingsRequest初始化Builder时将Builder类锁住，调用builder方法时去new Settings需要获取Settings类的锁，但是获取不到，因为已经被GetSettingsResponse的调用parseSettingsField时锁住

public class UpdateSettingsRequest extends AcknowledgedRequest<UpdateSettingsRequest> implements Replaceable, ToXContentObject {

    private String[] indices;

    private IndicesOptions indicesOptions = IndicesOptions.fromOptions(false, false, true, true);

    private Settings settings;

    private boolean preserveExisting;



    public UpdateSettingsRequest() {

        this.settings = Builder.EMPTY_SETTINGS;

        this.preserveExisting = false;

    }



    public UpdateSettingsRequest(String... indices) {

        [b]this.settings = Builder.EMPTY_SETTINGS;

[/b]        this.preserveExisting = false;

        this.indices = indices;

    }

因此，形成死锁

yangjianxuan

编写单元测试验证想法

import org.elasticsearch.common.settings.Settings;

import org.elasticsearch.common.xcontent.XContentParser;

import org.junit.Test;



import java.io.IOException;



/**

 * Description:

 *

 * @author yangjianxuan 2023/5/19

 */

public class YjxTest {



    @test

    public void test() throws InterruptedException {

        #Settings aa= Settings.Builder.EMPTY_SETTINGS;

        #System.out.println("hahaha:" + aa);



        Thread thread = new Thread() {

            @Override

            public void run() {

                while(true) {

                    System.out.println("1:");

                    Settings aa = Settings.Builder.EMPTY_SETTINGS;

                    System.out.println("11:" + aa);

                    try {

                        Thread.sleep(1000);

                    } catch (InterruptedException e) {

                        e.printStackTrace();

                    }

                }

            }

        };

        thread.start();



        Thread thread1 = new Thread() {

            @Override

            public void run() {

                while(true) {

                    System.out.println("2:");

                    XContentParser parser = null;

                    try {

                        Settings.fromXContent(parser);

                    } catch (Exception e) {

                        e.printStackTrace();

                    }

                    System.out.println("22:");

                    try {

                        Thread.sleep(1000);

                    } catch (InterruptedException e) {

                        e.printStackTrace();

                    }

                }

            }

        };

        thread1.start();



        Thread.sleep(1000000);

    }

}

1、此时直接运行就会出现死锁
2、解决方法，将注释的两行代码打开，提前加载Builder类，破坏死锁条件
最后，虽然问题解决了，但是理论还是比较匮乏，希望有懂的大神指点一下，这个锁机制。
以及，有没有直观地定位问题的方法。因为看到堆栈信息，基本就是靠猜，感觉有点误打误撞了。

要回复问题请先登录或注册

使用RestHighLevelClient获取Cluster setting信息BLOCK死锁【已解决】

6 个回复

发起人

相关问题

问题状态

使用RestHighLevelClient获取Cluster setting信息BLOCK死锁【已解决】

与内容相关的链接

6 个回复

发起人

相关问题

问题状态