好的想法是十分钱一打,真正无价的是能够实现这些想法的人。

elasticsearch能记录每次的查询吗

Elasticsearch | 作者 redhat | 发布于2018年05月15日 | 阅读数:11181

我们的elasticsearch集群有多个系统在调用,之前发现有个系统的查询语句有问题,频繁调用,请求大量数据,导致集群cpu飙升。我们现在想记录每次对es的访问,及返回的数据量,请问目前有这种监控工具吗?
已邀请:

JackGe

赞同来自: redhat CarrieJin

记录用户查询日志有两种方案1. 编写插件(实现比较简单),将查询日志记录到es日志文件中,示例代码如
public class SearchLoggerPlugin extends Plugin {

@Override
public String name() {
return "requestLogger";
}

@Override
public String description() {
return "attempt to log search request through TCP.";
}

public void onModule(final ActionModule module) {
module.registerFilter(SearchRequestActionFilter.class);
}
}

public class SearchRequestActionFilter extends ActionFilter.Simple {

private final ESLogger tcpSearchLogger;

private static final String INDEX_TCP_SEARCH_LOG_PREFIX = "index.tcp.search.log";

@Inject
public SearchRequestActionFilter(Settings settings) {
super(settings);
this.tcpSearchLogger = Loggers.getLogger(INDEX_TCP_SEARCH_LOG_PREFIX);
}

@Override
public int order() {
return 0;
}

@SuppressWarnings("rawtypes")
@Override
public boolean apply(String action, ActionRequest actionRequest,
final ActionListener listener) {
tcpSearchLogger.info("request info....");
return true;
}

@SuppressWarnings("rawtypes")
@Override
public boolean apply(String s, ActionResponse actionResponse,
ActionListener actionListener) {
tcpSearchLogger.info("response info....");
return true;
}
}
2. 实现查询网关,可以扩展审计限流,记录日志,故障迁移等功能。需要仿照es网络通信源码实现网关(实现较复杂)。

code4j - coder github: https://github.com/rpgmakervx

赞同来自:

我们是服务层记录查询语句。说白了就是肯定你有个服务或者 应用去调用查询语句,在那个服务或者应用中把query dsl 通过日志打下来就行了

bill

赞同来自:

/{index}/_stats/search命令可以返回一些search的统计数据。如果定时执行,可以看出一些查询性能的问题。

# curl -XGET '192.168.1.111:9200/log_test1/_stats/search?pretty'
{
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "failed" : 0
  },
  "_all" : {
    "primaries" : {
      "search" : {
        "open_contexts" : 0,
        "query_total" : 1,
        "query_time_in_millis" : 0,
        "query_current" : 0,
        "fetch_total" : 0,
        "fetch_time_in_millis" : 0,
        "fetch_current" : 0,
        "scroll_total" : 0,
        "scroll_time_in_millis" : 0,
        "scroll_current" : 0
      }
    },
    "total" : {
      "search" : {
        "open_contexts" : 0,
        "query_total" : 4,
        "query_time_in_millis" : 0,
        "query_current" : 0,
        "fetch_total" : 0,
        "fetch_time_in_millis" : 0,
        "fetch_current" : 0,
        "scroll_total" : 0,
        "scroll_time_in_millis" : 0,
        "scroll_current" : 0
      }
    }
  },
  "indices" : {
    "log_test1" : {
      "primaries" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 1,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0
        }
      },
      "total" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 4,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0
        }
      }
    }
  }
}

laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

es有类似慢查询的日志,可以配置,打印到日志。

要回复问题请先登录注册