客户端采用java api BulkProcessor bulk action 为5000,size 为20M,解析小文件速度很快,但是add到BulkProcessor 里面,提交到es就很慢,速度才2千/S,集群规模是8台es集群,32G,24cpu的,其中客户端采用多线程,每个线程里面都有BulkProcessor 对象,每个线程里面处理的数据都丢进BulkProcessor 里,之前处理速度很快,这两天就很慢了,其中创建索引是:
IndexRequest indexRequest = new IndexRequest(index, type, _id)
.source(xContentBuilder);
UpdateRequest updateRequest = new UpdateRequest(index, type, _id)
.doc(xContentBuilder)
.upsert(indexRequest);
bulkProcessor.add(updateRequest);
IndexRequest indexRequest = new IndexRequest(index, type, _id)
.source(xContentBuilder);
UpdateRequest updateRequest = new UpdateRequest(index, type, _id)
.doc(xContentBuilder)
.upsert(indexRequest);
bulkProcessor.add(updateRequest);
0 个回复