elasticsearch能否支持中文拼音纠错?

回复

wokeyi 发起了问题 • 2 人关注 • 0 个回复 • 83 次浏览 • 2017-09-15 11:50 • 来自相关话题

为何要避免往ES里写入稀疏数据

kennywu76 发表了文章 • 1 个评论 • 112 次浏览 • 2017-09-15 11:24 • 来自相关话题

转几篇文章,让大家知晓,当前版本(<=5.x) 为何要避免将稀疏的数据写入ES。 随着ES/Lucene编码的改进,这个问题未来版本可能会得到改善,特别是ES6.0/Lucene7.0优化了doc_values对稀疏数据的编码方式。
 
https://www.elastic.co/guide/e ... rsity  

Avoid sparsityedit

The data-structures behind Lucene, which Elasticsearch relies on in order to index and store data, work best with dense data, ie. when all documents have the same fields. This is especially true for fields that have norms enabled (which is the case for text fields by default) or doc values enabled (which is the case for numerics, date, ip and keyword by default).

The reason is that Lucene internally identifies documents with so-called doc ids, which are integers between 0 and the total number of documents in the index. These doc ids are used for communication between the internal APIs of Lucene: for instance searching on a term with a matchquery produces an iterator of doc ids, and these doc ids are then used to retrieve the value of the norm in order to compute a score for these documents. The way this norm lookup is implemented currently is by reserving one byte for each document. The norm value for a given doc id can then be retrieved by reading the byte at index doc_id. While this is very efficient and helps Lucene quickly have access to the norm values of every document, this has the drawback that documents that do not have a value will also require one byte of storage.

In practice, this means that if an index has M documents, norms will require M bytes of storage per field, even for fields that only appear in a small fraction of the documents of the index. Although slightly more complex with doc values due to the fact that doc values have multiple ways that they can be encoded depending on the type of field and on the actual data that the field stores, the problem is very similar. In case you wonder: fielddata, which was used in Elasticsearch pre-2.0 before being replaced with doc values, also suffered from this issue, except that the impact was only on the memory footprint since fielddata was not explicitly materialized on disk.

Note that even though the most notable impact of sparsity is on storage requirements, it also has an impact on indexing speed and search speed since these bytes for documents that do not have a field still need to be written at index time and skipped over at search time.

It is totally fine to have a minority of sparse fields in an index. But beware that if sparsity becomes the rule rather than the exception, then the index will not be as efficient as it could be.

This section mostly focused on norms and doc values because those are the two features that are most affected by sparsity. Sparsity also affect the efficiency of the inverted index (used to index text/keyword fields) and dimensional points (used to index geo_point and numerics) but to a lesser extent.

Here are some recommendations that can help avoid sparsity:

https://www.elastic.co/blog/index-vs-type

Fields that exist in one type will also consume resources for documents of types where this field does not exist. This is a general issue with Lucene indices: they don’t like sparsity. Sparse postings lists can’t be compressed efficiently because of high deltas between consecutive matches. And the issue is even worse with doc values: for speed reasons, doc values often reserve a fixed amount of disk space for every document, so that values can be addressed efficiently. This means that if Lucene establishes that it needs one byte to store all value of a given numeric field, it will also consume one byte for documents that don’t have a value for this field. Future versions of Elasticsearch will have improvements in this area but I would still advise you to model your data in a way that will limit sparsity as much as possible.

https://www.elastic.co/blog/sp ... ucene
[url=https://issues.apache.org/jira/browse/LUCENE-6863]https://issues.apache.org/jira/browse/LUCENE-6863​[/url] 
https://www.elastic.co/blog/el ... eased 查看全部
转几篇文章,让大家知晓,当前版本(<=5.x) 为何要避免将稀疏的数据写入ES。 随着ES/Lucene编码的改进,这个问题未来版本可能会得到改善,特别是ES6.0/Lucene7.0优化了doc_values对稀疏数据的编码方式。
 
https://www.elastic.co/guide/e ... rsity  


Avoid sparsityedit

The data-structures behind Lucene, which Elasticsearch relies on in order to index and store data, work best with dense data, ie. when all documents have the same fields. This is especially true for fields that have norms enabled (which is the case for text fields by default) or doc values enabled (which is the case for numerics, date, ip and keyword by default).

The reason is that Lucene internally identifies documents with so-called doc ids, which are integers between 0 and the total number of documents in the index. These doc ids are used for communication between the internal APIs of Lucene: for instance searching on a term with a matchquery produces an iterator of doc ids, and these doc ids are then used to retrieve the value of the norm in order to compute a score for these documents. The way this norm lookup is implemented currently is by reserving one byte for each document. The norm value for a given doc id can then be retrieved by reading the byte at index doc_id. While this is very efficient and helps Lucene quickly have access to the norm values of every document, this has the drawback that documents that do not have a value will also require one byte of storage.

In practice, this means that if an index has M documents, norms will require M bytes of storage per field, even for fields that only appear in a small fraction of the documents of the index. Although slightly more complex with doc values due to the fact that doc values have multiple ways that they can be encoded depending on the type of field and on the actual data that the field stores, the problem is very similar. In case you wonder: fielddata, which was used in Elasticsearch pre-2.0 before being replaced with doc values, also suffered from this issue, except that the impact was only on the memory footprint since fielddata was not explicitly materialized on disk.

Note that even though the most notable impact of sparsity is on storage requirements, it also has an impact on indexing speed and search speed since these bytes for documents that do not have a field still need to be written at index time and skipped over at search time.

It is totally fine to have a minority of sparse fields in an index. But beware that if sparsity becomes the rule rather than the exception, then the index will not be as efficient as it could be.

This section mostly focused on norms and doc values because those are the two features that are most affected by sparsity. Sparsity also affect the efficiency of the inverted index (used to index text/keyword fields) and dimensional points (used to index geo_point and numerics) but to a lesser extent.

Here are some recommendations that can help avoid sparsity:


https://www.elastic.co/blog/index-vs-type


Fields that exist in one type will also consume resources for documents of types where this field does not exist. This is a general issue with Lucene indices: they don’t like sparsity. Sparse postings lists can’t be compressed efficiently because of high deltas between consecutive matches. And the issue is even worse with doc values: for speed reasons, doc values often reserve a fixed amount of disk space for every document, so that values can be addressed efficiently. This means that if Lucene establishes that it needs one byte to store all value of a given numeric field, it will also consume one byte for documents that don’t have a value for this field. Future versions of Elasticsearch will have improvements in this area but I would still advise you to model your data in a way that will limit sparsity as much as possible.


https://www.elastic.co/blog/sp ... ucene
[url=https://issues.apache.org/jira/browse/LUCENE-6863]https://issues.apache.org/jira/browse/LUCENE-6863​[/url] 
https://www.elastic.co/blog/el ... eased

Screen_Shot_2017-09-15_at_11.29_.44_.png

除了设置mapping为not analyzied,还有其他方法让一个string字段不分词吗?

shellmeng 回复了问题 • 8 人关注 • 6 个回复 • 4537 次浏览 • 2017-09-15 11:13 • 来自相关话题

Elasticsearch 聚合时,如何获取非聚合字段的值

shellmeng 回复了问题 • 5 人关注 • 4 个回复 • 156 次浏览 • 2017-09-15 11:11 • 来自相关话题

启动es错误

回复

3014089145a 发起了问题 • 1 人关注 • 0 个回复 • 70 次浏览 • 2017-09-15 10:40 • 来自相关话题

elasticsearch搜索大文档时highlight导致CPU占满

kennywu76 回复了问题 • 4 人关注 • 2 个回复 • 142 次浏览 • 2017-09-15 10:29 • 来自相关话题

稀疏索引字段是否对性能有影响?

kennywu76 回复了问题 • 3 人关注 • 1 个回复 • 93 次浏览 • 2017-09-14 21:18 • 来自相关话题

各位大神,你们在ELK迁移的时候,要先把原来的elk服务停掉吗?那这段时间数据没有写进elk怎么办?

kennywu76 回复了问题 • 3 人关注 • 1 个回复 • 84 次浏览 • 2017-09-14 21:06 • 来自相关话题

es 时间查询问题

rockybean 回复了问题 • 2 人关注 • 1 个回复 • 88 次浏览 • 2017-09-14 18:14 • 来自相关话题

query_string 两个特定关键词,诡异的返回10s以上问题?

rockybean 回复了问题 • 2 人关注 • 1 个回复 • 83 次浏览 • 2017-09-14 18:04 • 来自相关话题