行动是治愈恐惧的良药,而犹豫、拖延将不断滋养恐惧。

Elasticsearch 迁移工具 ESM 更新 0.4.4

资料分享 | 作者 medcl | 发布于2020年05月14日 | | 阅读数:1329

timg.jpeg

 ESM 0.4.4 修复一堆 bug,支持几个新的特性:
  • 可用于生成测试数据,一般用于压力测试,基于源 ES 或者导入到本地的 JSON 数据,随机修改 ID,可以指定重复次数
  • 修复 Routing 参数在不同 ES 版本下的参数差异,支持 1.x\2.x\3.x\5.x 到 6.x\7.x 的相互导入
  • 修复终端下不能切换到后台执行的 bug,可以以 crontab 定时执行
  • 支持指定 _source 字段导出
  • 支持 _source 字段重命名
  • 支持文档 _type 重命名

 
下载地址:
 
https://github.com/medcl/esm/releases/tag/v0.4.4
 
 
生成测试数据示例:
 
1.直接以来源 es 的 my_index1 的索引数据生成到目标 es 集群的索引 my_index2,产生 10 份一样的数据
./bin/esm -s http://localhost:9201 -d http://localhost:9200 -x my_index1 -y 
my_index2
-n elastic:pass --regenerate_id --repeat_times=10

2.先导出索引文档到本地的文件 dump.json
./bin/esm -s http://localhost:9201 -x my_index1 -o dump.json
再基于这份样本,生成 10 份一样的数据到目标集群
./bin/esm -i dunp.json -d  http://localhost:9201 -y target-index1  --regenerate_id  --repeat_times=10 


更多使用示例参照项目 README

[尊重社区原创,转载请保留或注明出处]
本文地址:http://elasticsearch.cn/article/13773


8 个评论

./esm -s elastic:xxx@10.19.1.14:9200 -d elastic:xxx@10.19.1.95:9200 -x b--news-perf --log=debug
[06-23 01:40:45] [DBG] [main.go:523,ClusterVersion] {
"name" : "hz-es-client-19-1-14",
"cluster_name" : "my-application",
"cluster_uuid" : "vVRGFMZfRUiuJo-_WMRNbA",
"version" : {
"number" : "6.8.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fad4e1",
"build_date" : "2019-06-18T13:16:52.517138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

[06-23 01:40:45] [DBG] [main.go:116,main] source es is V6,6.8.1
[06-23 01:41:09] [ERR] [v6.go:99,NewScroll] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
[06-23 01:41:09] [ERR] [main.go:149,main] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
稍等,我来测试一下 6.8.1
这个索引数据是从旧版本升级上来的么?
这个索引是新建的.
我用集群里任何一个索引测试貌似都有问题.

POST mydba_logstash_index/logs
{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "2020-06-16T14:36:39.000Z",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}


POST mydba_logstash_index/logs
{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "1592289396859",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}

./esm -s elastic:xxx@10.199.151.14:9200 -d elastic:xxx@10.199.151.95:9200 -x mydba_logstash_index -y mydba_logstash_index --log=trace

[06-23 12:38:37] [DBG] [main.go:523,ClusterVersion] {
"name" : "hz-es-client-199-151-14",
"cluster_name" : "my-application",
"cluster_uuid" : "vVRGFMZfRUiuJo-_WMRNbA",
"version" : {
"number" : "6.8.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fad4e1",
"build_date" : "2019-06-18T13:16:52.517138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

[06-23 12:38:37] [DBG] [main.go:116,main] source es is V6,6.8.1
[06-23 12:38:37] [TRC] [v6.go:89,NewScroll] new scroll,{"_scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAs-4WUjZBSHhiUHNUYjZJalRQNkFHSmJ4dw==","took":18,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"mydba_logstash_index","_type":"logs","_id":"k_l813IBQM_nnOshyGjN","_score":1.0,"_source":{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "2020-06-16T14:36:39.000Z",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}
},{"_index":"mydba_logstash_index","_type":"logs","_id":"sfl813IBQM_nnOsh0miT","_score":1.0,"_source":{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "1592289396859",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}
}]}}
[06-23 12:38:37] [ERR] [v6.go:99,NewScroll] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
[06-23 12:38:37] [ERR] [main.go:149,main] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
MAPPING: mydba_logstash_index

{
"mapping": {
"logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"sourceIp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"traceId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
你下载的 ESM 是什么版本呢?是最新的么?
下载这个https://github.com/medcl/esm/releases/tag/v0.4.5-SNAPSHOT,4.4 有一个 bug
v0.4.5 已经可以了, 谢谢!!

要回复文章请先登录注册