执行 bulk 索引文档的时候，用 index 或者 create 类型并且自定义 doc id 的情况下，是否会像 update 一样每次都要去 get 一遍原始文档？ 比如下面的这条命令：

POST _bulk



{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }

{ "field1" : "value1" }

{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }

{ "field1" : "value3" }

git clone https://github.com/elastic/elasticsearch.git --depth=1

gradle idea
 

457 public IndexResult index(Index index) throws IOException

558 private IndexingStrategy planIndexingAsPrimary(Index index) throws IOException {
 

• optimizedAppendOnly
• skipDueToVersionConflict
• processNormally
• overrideExistingAsIfNotThere
• skipAsStale

389 private VersionValue resolveDocVersion(final Operation op) throws IOException {

1. 尝试从 versionMap 中读取待写入文档的 version，也即从内存中读取。versionMap 会暂存还没有 commit 到磁盘的文档版本信息。
2. 如果第 1 步中没有读到，则从 index 中读取，也即从文件中读取。

    public Result prepare(UpdateRequest request, IndexShard indexShard, LongSupplier nowInMillis) {

        final GetResult getResult = indexShard.getService().get(request.type(), request.id(),

                new String[]{RoutingFieldMapper.NAME, ParentFieldMapper.NAME, TTLFieldMapper.NAME, TimestampFieldMapper.NAME},

                true, request.version(), request.versionType(), FetchSourceContext.FETCH_SOURCE);

        return prepare(indexShard.shardId(), request, getResult, nowInMillis);

    }

1. 获取待更新文档的数据
2. 执行更新文档的操作

350 public GetResult get(Get get, Function<String, Searcher> searcherFactory, LongConsumer onRefresh) throws EngineException {

Avoid sparsityedit

The data-structures behind Lucene, which Elasticsearch relies on in order to index and store data, work best with dense data, ie. when all documents have the same fields. This is especially true for fields that have norms enabled (which is the case for text fields by default) or doc values enabled (which is the case for numerics, date, ip and keyword by default).

The reason is that Lucene internally identifies documents with so-called doc ids, which are integers between 0 and the total number of documents in the index. These doc ids are used for communication between the internal APIs of Lucene: for instance searching on a term with a matchquery produces an iterator of doc ids, and these doc ids are then used to retrieve the value of the norm in order to compute a score for these documents. The way this norm lookup is implemented currently is by reserving one byte for each document. The norm value for a given doc id can then be retrieved by reading the byte at index doc_id. While this is very efficient and helps Lucene quickly have access to the norm values of every document, this has the drawback that documents that do not have a value will also require one byte of storage.

In practice, this means that if an index has M documents, norms will require M bytes of storage per field, even for fields that only appear in a small fraction of the documents of the index. Although slightly more complex with doc values due to the fact that doc values have multiple ways that they can be encoded depending on the type of field and on the actual data that the field stores, the problem is very similar. In case you wonder: fielddata, which was used in Elasticsearch pre-2.0 before being replaced with doc values, also suffered from this issue, except that the impact was only on the memory footprint since fielddata was not explicitly materialized on disk.

Note that even though the most notable impact of sparsity is on storage requirements, it also has an impact on indexing speed and search speed since these bytes for documents that do not have a field still need to be written at index time and skipped over at search time.

It is totally fine to have a minority of sparse fields in an index. But beware that if sparsity becomes the rule rather than the exception, then the index will not be as efficient as it could be.

This section mostly focused on norms and doc values because those are the two features that are most affected by sparsity. Sparsity also affect the efficiency of the inverted index (used to index text/keyword fields) and dimensional points (used to index geo_point and numerics) but to a lesser extent.

Here are some recommendations that can help avoid sparsity:

Fields that exist in one type will also consume resources for documents of types where this field does not exist. This is a general issue with Lucene indices: they don’t like sparsity. Sparse postings lists can’t be compressed efficiently because of high deltas between consecutive matches. And the issue is even worse with doc values: for speed reasons, doc values often reserve a fixed amount of disk space for every document, so that values can be addressed efficiently. This means that if Lucene establishes that it needs one byte to store all value of a given numeric field, it will also consume one byte for documents that don’t have a value for this field. Future versions of Elasticsearch will have improvements in this area but I would still advise you to model your data in a way that will limit sparsity as much as possible.