1.如何使用terraform自动化部署Elastic Cloud

https://xeraa.net/blog/2020_how-to-automate-elastic-cloud-with-terraform/

2.使用ILM构建hot-warm-cold架构（自备梯子）

https://medium.com/bestcloudforme/data-aging-with-elasticsearch-hot-warm-cold-infrastructure-6e5eef4c9e65

3.ES底层字符串数组和字符串的索引区别

https://stackoverflow.com/questions/65271653/internals-of-array-of-strings-vs-concatenated-string-in-elasticsearch

• 编辑：bsll

• 归档：https://ela.st/cn-daily-all

• 订阅：https://ela.st/cn-daily-sub

• 沙龙：https://ela.st/cn-meetup

[program:gateway]

command = /app/logger/gateway/gateway-linux64

username = appuser

autostart=false

autorestart=true

startsecs=3

priority=1007

stdout_logfile=/app/logger/gateway/log/infini_gateway.log

path.data: data

path.logs: log

entry:

- name: es_gateway #your gateway endpoint

enable: true

router: default #configure your gateway's routing flow

network:

binding: 0.0.0.0:8000

reuse_port: true #you can start multi gateway instance, they share same port, to full utilize system's resources

tls:

enabled: false #if your es is using https, the gateway entrypoint should enable https too

- name: cache_first

filter: #comment out any filter sections, like you don't need cache or rate-limiter

- name: get_cache_1

type: get_cache

- name: rate_limit_1

type: rate_limit

parameters:

message: "Hey, You just reached our request limit!"

rules: #configure match rules against request's PATH, eg: /_cluster/health, match the first rule and return

- pattern: "/(?P<index_name>test.*?)/_search" #use regex pattern to match index, will match any /$index/_search, and limit each index with max_qps ~=100

max_qps: 1000

group: index_name

- name: elasticsearch_1

type: elasticsearch

parameters:

elasticsearch: default #elasticsearch configure reference name

max_connection: 1000 #max tcp connection to upstream, default for all nodes

max_response_size: -1 #default for all nodes

balancer: weight

- host: 192.168.3.201:9200 #the format is host:port

weight: 100

- host: 192.168.3.202:9200

weight: 100

discovery:

enabled: false

- name: set_cache_1

type: set_cache

parameters:

cache_ttl: 1000s

# max_cache_items: 100000

- name: request_logging # this flow is used for request logging, refer to `router`'s `tracing_flow`

filter:

- name: request_path_filter_1

type: request_path_filter

parameters:

must: #must match all rules to continue

prefix:

- /test

- name: request_logging_1

type: request_logging

parameters:

queue_name: request_logging

elasticsearch.customHeaders: { "app": "kibana" }

- name: request_header_filter1 # filter out the requests that we are not interested, reduce tracing pressure

type: request_header_filter

parameters:

exclude: # any rule match will marked request as filtered

- app: kibana # in order to filter kibana's access log, config `elasticsearch.customHeaders: { "app": "kibana" }` to your kibana's config `/config/kibana.yml`

router:

- name: default

tracing_flow: request_logging #a flow will execute after request finish

default_flow: cache_first

rules: #rules can't be conflicted with each other, will be improved in the future

- id: 1 # this rule means match every requests, and sent to `cache_first` flow

method:

- "*"

pattern:

- /

# priority: 1

flow:

- cache_first # after match, which processing flow will go through

elasticsearch:

- name: default

enabled: true

endpoint: http://localhost:9200 # if your elasticsearch is using https, your gateway should be listen on as https as well

version: 7.9.1 #optional, used to select es adaptor, can be done automatically after connect to es

index_prefix: gateway_

basic_auth: #used to discovery full cluster nodes, or check elasticsearch's health and versions

username: elastic

password: pass

modules:

- name: elastic

enabled: true

elasticsearch: default

init_template: true

- name: pipeline

enabled: true

runners:

- name: primary

enabled: true

max_go_routine: 1

threshold_in_ms: 0

timeout_in_ms: 5000

pipeline_id: request_logging_index

pipelines:

- name: request_logging_index

start:

joint: json_indexing

enabled: true

parameters:

index_name: "gateway_requests"

elasticsearch: "default"

input_queue: "request_logging"

timeout: "60s"

worker_size: 1

bulk_size_in_mb: 10 #in MB

process:

queue:

min_msg_size: 1

max_msg_size: 50000000

max_bytes_per_file: 53687091200

sync_every_records: 100000 # sync by records count

sync_timeout_in_ms: 10000 # sync by time in million seconds

read_chan_buffer: 0

enabled: false

host: 127.0.0.1

port: 8125

namespace: gateway.

• Content tier （内容层）节点处理诸如产品目录之类的内容的索引和查询负载。
• Hot tier （热层） 节点处理诸如日志或指标之类的时间序列数据的索引负载，并保存你最近，最常访问的数据。
• Warm tier （温层）节点保存的时间序列数据访问频率较低，并且很少需要更新。
• Cold tier （冷层）节点保存时间序列数据，这些数据偶尔会被访问，并且通常不会更新。

• 在 Discover 界面中制作
• 使用 table 可视化进行制作
• 使用 TSVB  来制作
• 使用 Lens 进行制作

1.复杂嵌套语句的搜索及优化

https://stackoverflow.com/questions/65237747/elasticsearch-query-by-generic-properties-keywords-and-numeric-values/65243079#65243079

2.Lucene vs Solr vs ElasticSearch

https://anytxt.net/how-to-choose-a-full-text-search-engine/

3.基于ES的问答系统

https://github.com/jetnew/DrFAQ

• 编辑：bsll

• 归档：https://ela.st/cn-daily-all

• 订阅：https://ela.st/cn-daily-sub

• 沙龙：https://ela.st/cn-meetup