找到问题的解决办法了么？

Elasticsearch 迁移工具 ESM 更新 0.4.4

资料分享 | 作者 medcl | 发布于2020年05月14日 | | 阅读数：8566
分享到：QQ空间新浪微博微信QQ好友印象笔记有道云笔记
timg.jpeg

 ESM 0.4.4 修复一堆 bug，支持几个新的特性：
  • 可用于生成测试数据，一般用于压力测试，基于源 ES 或者导入到本地的 JSON 数据，随机修改 ID，可以指定重复次数
  • 修复 Routing 参数在不同 ES 版本下的参数差异，支持 1.x\2.x\3.x\5.x 到 6.x\7.x 的相互导入
  • 修复终端下不能切换到后台执行的 bug，可以以 crontab 定时执行
  • 支持指定 _source 字段导出
  • 支持 _source 字段重命名
  • 支持文档 _type 重命名

 
下载地址：
 
https://github.com/medcl/esm/releases/tag/v0.4.4
 
 
生成测试数据示例：
 
1.直接以来源 es 的 my_index1 的索引数据生成到目标 es 集群的索引 my_index2，产生 10 份一样的数据
./bin/esm -s http://localhost:9201 -d http://localhost:9200 -x my_index1 -y 

my_index2

 -n elastic:pass  --regenerate_id  --repeat_times=10

2.先导出索引文档到本地的文件 dump.json
./bin/esm -s http://localhost:9201 -x my_index1 -o dump.json
再基于这份样本，生成 10 份一样的数据到目标集群
./bin/esm -i dunp.json -d  http://localhost:9201 -y target-index1  --regenerate_id  --repeat_times=10

更多使用示例参照项目 README
[尊重社区原创，转载请保留或注明出处]
本文地址：http://elasticsearch.cn/article/13773
3
laoyang360 gomatu wayne

10 个评论

tithonus

tithonus

./esm -s elastic:xxx@10.19.1.14:9200 -d elastic:xxx@10.19.1.95:9200 -x b--news-perf --log=debug
[06-23 01:40:45] [DBG] [main.go:523,ClusterVersion] {
"name" : "hz-es-client-19-1-14",
"cluster_name" : "my-application",
"cluster_uuid" : "vVRGFMZfRUiuJo-_WMRNbA",
"version" : {
"number" : "6.8.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fad4e1",
"build_date" : "2019-06-18T13:16:52.517138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

[06-23 01:40:45] [DBG] [main.go:116,main] source es is V6,6.8.1
[06-23 01:41:09] [ERR] [v6.go:99,NewScroll] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
[06-23 01:41:09] [ERR] [main.go:149,main] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
2020-06-23 01:45
medcl

medcl

稍等，我来测试一下 6.8.1
2020-06-23 11:21
medcl

medcl 回复 tithonus

这个索引数据是从旧版本升级上来的么？
2020-06-23 11:58
tithonus

tithonus 回复 medcl

这个索引是新建的.
我用集群里任何一个索引测试貌似都有问题.

POST mydba_logstash_index/logs
{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "2020-06-16T14:36:39.000Z",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}


POST mydba_logstash_index/logs
{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "1592289396859",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}

./esm -s elastic:xxx@10.199.151.14:9200 -d elastic:xxx@10.199.151.95:9200 -x mydba_logstash_index -y mydba_logstash_index --log=trace

[06-23 12:38:37] [DBG] [main.go:523,ClusterVersion] {
"name" : "hz-es-client-199-151-14",
"cluster_name" : "my-application",
"cluster_uuid" : "vVRGFMZfRUiuJo-_WMRNbA",
"version" : {
"number" : "6.8.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fad4e1",
"build_date" : "2019-06-18T13:16:52.517138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

[06-23 12:38:37] [DBG] [main.go:116,main] source es is V6,6.8.1
[06-23 12:38:37] [TRC] [v6.go:89,NewScroll] new scroll,{"_scroll_id":"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAs-4WUjZBSHhiUHNUYjZJalRQNkFHSmJ4dw==","took":18,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"mydba_logstash_index","_type":"logs","_id":"k_l813IBQM_nnOshyGjN","_score":1.0,"_source":{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "2020-06-16T14:36:39.000Z",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}
},{"_index":"mydba_logstash_index","_type":"logs","_id":"sfl813IBQM_nnOsh0miT","_score":1.0,"_source":{
"traceId" : "9055b02343504acb96e5ef3a37cb53e2",
"@timestamp" : "1592289396859",
"sourceIp" : "192.168.78.5",
"timestamp" : "1592289396809"
}
}]}}
[06-23 12:38:37] [ERR] [v6.go:99,NewScroll] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
[06-23 12:38:37] [ERR] [main.go:149,main] json: cannot unmarshal number into Go struct field .hits.total of type struct { Value int "json:\"value,omitempty\""; Relation string "json:\"relation,omitempty\"" }
2020-06-23 12:41
tithonus

tithonus

MAPPING: mydba_logstash_index

{
"mapping": {
"logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"sourceIp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"traceId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
2020-06-23 12:43
medcl

medcl

你下载的 ESM 是什么版本呢？是最新的么？
2020-06-23 17:08
medcl

medcl

下载这个https://github.com/medcl/esm/releases/tag/v0.4.5-SNAPSHOT，4.4 有一个 bug
2020-06-23 17:09
tithonus

tithonus 回复 medcl

v0.4.5 已经可以了, 谢谢!!
2020-06-23 17:23
yolanda_chen

yolanda_chen

大佬，我用的vo.4.5，索引迁移过程一直刷这个错误：[v0.go:66,Bulk] data is empyt, skip
界面进度显示： bulk 38/45162 一直卡着不动
2020-11-11 11:44
wanjunbao

wanjunbao

请问新版的v0.6.0怎么编译安装
2023-03-16 18:46

要回复文章请先登录或注册

Copyright © 2024 · CC BY-NC-SA 3.0

本站服务器及带宽由 提供赞助