ES5.0 批量导入数据,设置导入结束后刷新不生效
匿名 | 发布于2016年12月05日 | 阅读数:8797
我想在批量导入结束后再让ES执行刷新操作,提高导入性能,
但是实际上我看到数据一直是刷新的,因为我在head插件的概览界面看到索引docs数量是一直增加的
这不就表示ES实际上已经再刷新了,并没有执行我的设置
elasticsearch.yml配置:
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /esdata/elasticsearch/data
path.logs: /esdata/elasticsearch/logs
network.host: 192.168.24.84
http.port: 8201
transport.tcp.port: 8301
bootstrap.memory_lock: true
索引的Mapping设置:
{"state": "open",
"settings": {
"index": {
"refresh_interval": "-1",
"number_of_shards": "4",
"translog": {
"flush_threshold_size": "10240mb",
"sync_interval": "60m",
"durability": "async"
}... ...
JAVA客户端代码:
client.admin()
.indices()
.prepareUpdateSettings(indexName)
.setSettings(Settings.builder()
.put("index.refresh_interval", "-1")
.put("index.translog.flush_threshold_size", "10240mb")
.put("index.translog.durability", "async")
)
.get();
BulkProcessor bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener()
{
public void beforeBulk(long executionId, BulkRequest request){
}
@Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response){
}
@Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure){
}
}).setBulkActions(5000).setBulkSize(new ByteSizeValue(1L, ByteSizeUnit.PB)).setFlushInterval(TimeValue.timeValueHours(1L)).setConcurrentRequests(10).build();
IndexRequest iq = new IndexRequest(indexName, indexType).source(doc).setRefreshPolicy(RefreshPolicy.NONE);
for (int i = 0; i < 1000000; i++)
{
bulkProcessor.add(iq);
}
bulkProcessor.awaitClose(3, TimeUnit.MINUTES);
client.admin().indices().prepareUpdateSettings(indexName).setSettings(Settings.builder().put("index.refresh_interval", "1s")).get();
但是实际上我看到数据一直是刷新的,因为我在head插件的概览界面看到索引docs数量是一直增加的
这不就表示ES实际上已经再刷新了,并没有执行我的设置
elasticsearch.yml配置:
http.cors.enabled: true
http.cors.allow-origin: "*"
path.data: /esdata/elasticsearch/data
path.logs: /esdata/elasticsearch/logs
network.host: 192.168.24.84
http.port: 8201
transport.tcp.port: 8301
bootstrap.memory_lock: true
索引的Mapping设置:
{"state": "open",
"settings": {
"index": {
"refresh_interval": "-1",
"number_of_shards": "4",
"translog": {
"flush_threshold_size": "10240mb",
"sync_interval": "60m",
"durability": "async"
}... ...
JAVA客户端代码:
client.admin()
.indices()
.prepareUpdateSettings(indexName)
.setSettings(Settings.builder()
.put("index.refresh_interval", "-1")
.put("index.translog.flush_threshold_size", "10240mb")
.put("index.translog.durability", "async")
)
.get();
BulkProcessor bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener()
{
public void beforeBulk(long executionId, BulkRequest request){
}
@Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response){
}
@Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure){
}
}).setBulkActions(5000).setBulkSize(new ByteSizeValue(1L, ByteSizeUnit.PB)).setFlushInterval(TimeValue.timeValueHours(1L)).setConcurrentRequests(10).build();
IndexRequest iq = new IndexRequest(indexName, indexType).source(doc).setRefreshPolicy(RefreshPolicy.NONE);
for (int i = 0; i < 1000000; i++)
{
bulkProcessor.add(iq);
}
bulkProcessor.awaitClose(3, TimeUnit.MINUTES);
client.admin().indices().prepareUpdateSettings(indexName).setSettings(Settings.builder().put("index.refresh_interval", "1s")).get();
3 个回复
240475588
赞同来自:
240475588
赞同来自:
index.refresh_interval
不生效,感觉没更新一个文档都会刷新
Jaret
赞同来自: