最近看到 INFINI Gateway 新增了一个 bulk_reshuffle filter, 于是便简单地测试一下这个功能。(Gateway 下载地址 以及 参考文档)

测试机器配置

测试所需软件及版本

1. Elasticsearch 7.10
2. Kibana 7.10
3. INFINI Gateway 最新版本
4. Logstash 7.10
5. Metricbeat 7.10

本文就省略以上软件的下载和安装步骤了。 另外本文中测试 Elasticsearch 集群含两个节点，每个节点配置内存都为 1GB ，其他参数均为默认。

测试步骤

准备测试数据文件

本文测试数据文件 nginx_mock_log ,文件中每行结构如下：

{"timestamp":1611540661651,"method":"POST","msg":"mock log"}

大概一千多万条

Logstatsh 使用 Input file 模式直接输出数据到 Elasticsearch

编辑 Logstash 配置 test.conf 如下：

input{
     file {
        path => ["/test/nginx_mock_log"]
        type => "file_monitor"
        start_position => "beginning"
    }
}

output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "nginx_mock_log"
        http_compression => false
    }
}

在 kibana 中创建索引 nginx_mock_log ，将主分片设置为2（为了体现出 Gateway的性能优势， 主分片数应设置大于1）， 配置如下：

PUT nginx_mock_log
{
"mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "path" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
        "number_of_shards" : "2"
    }
}

运行 Logstash

/usr/local/logstash/bin/logstash -f test.conf

打开 Kibana Stack Monitorning 查看 Indexing Rate 监控指标如下图：

从图中可以看到索引速率基本保持在4300/s 上下

Logstatsh 使用 Input file 模式输出数据到 Gateway

进入 Kibana 删除索引 nginx_mock_log 并重建

DELETE nginx_mock_log

PUT nginx_mock_log
{
"mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "path" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
        "number_of_shards" : "2"
    }
}

修改 Logstash 配置 test.conf 如下：

input{
     file {
        path => ["/test/nginx_mock_log"]
        type => "file_monitor"
        start_position => "beginning"
    }
}

output{
    elasticsearch {
        hosts => ["localhost:8000"]
        index => "nginx_mock_log"
        http_compression => false
    }
}

修改 Gateway 配置文件 gateway.yaml 如下：

path.data: data
path.logs: log

entry:
  - name: es_gateway #your gateway endpoint
    enabled: true
    router: default
    network:
      binding: localhost:8000
      reuse_port: true #you can start multi gateway instance, they share same port, to full utilize system's resources

flow:
  - name: bulk_es_test
    filter: #comment out any filter sections, like you don't need cache or rate-limiter
      - name: bulk_reshuffle
        parameters:
          elasticsearch: dev
          level: node
          mode: async
      - name: elasticsearch
        parameters:
          elasticsearch: dev
          refresh:
            enabled: true
            interval: 30s
  - name: request_logging
    filter:
      - name: request_logging
        parameters:
          queue_name: request_logging
router:
  - name: default
    default_flow: bulk_es_test
    tracing_flow: request_logging

elasticsearch:
- name: dev
  enabled: true
  endpoint: http://localhost:9200 # if your elasticsearch is using https, your gateway should be listen on as https as well
  basic_auth: #used to discovery full cluster nodes, or check elasticsearch's health and versions
    username: elastic
    password: yV6syH3KLt4DxqMlCyag
  discovery: # auto discovery elasticsearch cluster nodes
    enabled: true
    refresh:
      enabled: true

modules:
- name: elastic
  enabled: true
  elasticsearch: dev
  store:
    enabled: true
  orm:
    enabled: true
    init_template: true
    template_name: ".infini-default1"
    index_prefix: "gateway_"

- name: pipeline
  enabled: true
  runners:
    - name: nodes_index
      enabled: true
      max_go_routine: 2
      threshold_in_ms: 0
      timeout_in_ms: 5000
      pipeline_id: bulk_request_ingest
    - name: request_logging_test_name
      enabled: true
      max_go_routine: 2
      threshold_in_ms: 0
      timeout_in_ms: 5000
      pipeline_id: request_logging_index

pipelines:
- name: bulk_request_ingest
  start:
    joint: bulk_indexing
    enabled: true
    parameters:
      elasticsearch: "dev"
      timeout: "5s"
      worker_size: 10
      bulk_size_in_mb: 1 #in MB
- name: request_logging_index
  start:
    joint: json_indexing
    enabled: true
    parameters:
      index_name: "gateway_requests"
      elasticsearch: "dev"
      input_queue: "request_logging"
      timeout: "5s"
      worker_size: 10
      bulk_size_in_mb: 1 #in MB

queue:
  min_msg_size: 1
  max_msg_size: 5000000000
  max_bytes_per_file: 53687091200
  sync_every_records: 100000 # sync by records count
  sync_timeout_in_ms: 10000 # sync by time in million seconds
  write_chan_buffer: 1000
  read_chan_buffer: 1000

以上各配置节点含义，请参考 Gateway 文档

启动 Gateway ./gateway

删除 Logstash data 目录

rm -rf /usr/local/logstash/data

启动 Logstash

/usr/local/logstash/bin/logstash -f test.conf

打开 Kibana Stack Monitorning 查看 Indexing Rate 监控指标如下图：

从上图后半部分可以看到索引速率可以保持在 25000/s 上下（一会儿的功夫，一千多万条数据导入ES完事了）

前面看到 Gateway 配置开启了 request_logging，因此可以在 Kibana Dashboard 里面的 INFINI Gateway Dashboard 查看请求信息，如下图：

注意，上面图中的请求速率是 _bulk 请求的速率，不是索引速率

总结

从测试结果来看，相同环境下，用 Logstash elasticsearch output 输出数据到 Gateway 的方式比 Logstash elasticsearch output 直接到 ES 的方式速率快了4倍，不得不说这速率是真的杠杠的。至于能不能通过参数调优再提升速率呢？大家有兴趣的自己下载测试吧！最后感谢 medcl 大神出品。

1.使用word_delimiter构建自定义分析器

https://stackoverflow.com/questions/28499140/regular-expression-in-elasticsearch

2.使用glove测试es向量相似搜索

https://blog.mimacom.com/elastic-cosine-similarity-word-embeddings/

3.Lucene 中的 Stored Fields 存储优化

https://www.infoq.cn/article/rgsyhhhckwdv4b65csyx

• 编辑：bsll

• 归档：https://ela.st/cn-daily-all

• 订阅：https://ela.st/cn-daily-sub

• 沙龙：https://ela.st/cn-meetup

我们的es从5.3.2升级到7.4.1，遇到一个脚本的性能问题。同样的一段Sort Script脚本, 在ES 7.4.1的执行要比ES 5.3.2要慢至少一倍。一个主要的特点就是departure_city_ids可能会很多，通过doc获取一个deparature_city_ids的数据, 数据的大小最大的时候达到两千多个左右。Type为keyword类型。

• 具体的sort script脚本实例已经贴在下面了。

脚本实例:

"sort": [
    {
      "_script": {
        "script": {
          "inline": "return doc['departure_city_ids'].size()",
          "lang": "painless"
        },
        "type": "number",
        "order": "desc"
      }
    }
  ]

https://discuss.elastic.co/t/the-performance-of-script-based-sorting-in-es-7-4-2-verison-is-very-poor/261241

1.es常见分析器介绍

https://medium.com/betacom/elasticsearch-text-fields-analyzers-9cd4ee3aadcd

2.es近似最近邻向量搜索插件

https://github.com/rixwew/elasticsearch-approximate-nearest-neighbor

3.利用bert使es支持语义搜索

https://github.com/md-experiments/elastic_transformers

• 编辑：bsll

• 归档：https://ela.st/cn-daily-all

• 订阅：https://ela.st/cn-daily-sub

• 沙龙：https://ela.st/cn-meetup