设置参数 `node.name` 可以自定义 Elasticsearch 节点的名字。 此条 Tips 由 medcl 贡献。

使用Elasticsearch实现同段和同句搜索

Elasticsearch | 作者 trycatchfinal | 发布于2020年03月07日 | | 阅读数：5620
分享到：QQ空间新浪微博微信QQ好友印象笔记有道云笔记

同句搜索要求搜索多个关键词时，返回的文章不只要包含关键词，而且这些关键词必须在同一句中。
同段搜素类似，只是范围为同一段落。

SpanQuery

同段、同句搜索，使用常用的term、match查询，没有找到办法可以实现。
Elasticsearch提供了SpanQuery，官方文档中如下的介绍：

Span queries are low-level positional queries which provide expert control over the order and proximity of the specified terms. These are typically used to implement very specific queries on legal documents or patents.

上面提到，SpanQuery常常应用在法律或专利的特定搜索。这些领域，常常提供同段 /同句搜索 。
下面我们看一下三种类型的SpanQuery，能否实现我们的需求:

准备数据

PUT article

POST article/_mapping
{
  "properties": {
    "maincontent": {
      "type": "text"
    }
  }
}

POST article/_doc/1
{
   "maincontent":"the quick red fox jumps over the sleepy cat"
}

POST article/_doc/2
{
   "maincontent":"the quick brown fox jumps over the lazy dog"
}

SpanTermQuery

SpanTermQuery 和 Term Query类似, 下面的查询会返回_id为1的doc。 the quick red fox jumps over the sleepy cat

POST article/_search
{
  "profile": "true",
  "query": {
    "span_term": {
      "maincontent": {
        "value": "red"
      }
    }
  }
}

SpanNearQuery

SpanNearQuery 表示邻近搜索，查找多个term是否邻近，slop可以设置邻近距离，如果设置为0，那么代表两个term是挨着的，相当于matchphase in_order参数，代表文档中的term和查询设置的term保持相同的顺序。

POST article/_search
{
  "query": {
    "span_near": {
      "clauses": [
        {
          "span_term": {
            "maincontent": {
              "value": "quick"
            }
          }
        },
        {
          "span_term": {
            "maincontent": {
              "value": "brown"
            }
          }
        }
      ],
      "slop": 0,
      "in_order": true
    }
  }
}

上面的查询会返回_id为2的doc。

the quick brown fox jumps over the lazy dog

SpanNotQuery

SpanNotQuery非常重要，它要求两个SpanQuery的跨度，不能够重合。
看下面的例子:

  • include: 匹配的SpanQuery，例子为需要一个包含quick和fox两个词的邻近搜索。
  • exclude：设置一个SpanQuery，要求include中的SpanQuery不能包含这个SpanQuery 
    POST article/_search
{
"query": {
"span_not": {
  "include": {
    "span_near": {
      "clauses": [
        {
          "span_term": {
            "maincontent": {
              "value": "quick"
            }
          }
        },
        {
          "span_term": {
            "maincontent": {
              "value": "fox"
            }
          }
        }
      ],
      "slop": 1,
      "in_order": true
    }
  },
  "exclude": {
    "span_term": {
      "maincontent": {
        "value": "red"
      }
    }
  }
}
}
}

    上面的查询会返回_id为2的doc。
    因为_id为1的文档，虽然quick red fox符合include中的SpanQuery,但是red也符合exclude中的SpanQuery。因此，这篇文章需要排除掉。 the quick red fox jumps over the sleepy cat

同句/同段搜索原理

同句搜索，反向来说，就是搜索词不能够跨句。再进一步，就是搜索词之间不能够有。 、？、！等其他标点符号。
其对应的查询类似如下：

POST article/_search
{
  "query": {
    "span_not": {
      "include": {
        "span_near": {
          "clauses": [
            {
              "span_term": {
                "maincontent": {
                  "value": "word1"
                }
              }
            },
            {
              "span_term": {
                "maincontent": {
                  "value": "word2"
                }
              }
            }
          ],
          "slop": 1,
          "in_order": true
        }
      },
      "exclude": {
        "span_term": {
          "maincontent": {
            "value": "。/？/！"
          }
        }
      }
    }
  }
}

同段搜素类似，对应分隔符变为\n,或者<p>,</p>

同段/同句搜索实现

文本为HTML格式

创建索引

PUT sample1
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "analysis": {
      "analyzer": {
        "maincontent_analyzer": {
          "type": "custom",
          "char_filter": [
            "sentence_paragrah_mapping",
            "html_strip"
          ],
          "tokenizer": "ik_max_word"
        }
      },
      "char_filter": {
        "sentence_paragrah_mapping": {
          "type": "mapping",
          "mappings": [
            """<h1> => \u0020paragraph\u0020""",
            """</h1> => \u0020sentence\u0020paragraph\u0020 """,
            """<h2> => \u0020paragraph\u0020""",
            """</h2> => \u0020sentence\u0020paragraph\u0020 """,
            """<p> => \u0020paragraph\u0020""",
            """</p> => \u0020sentence\u0020paragraph\u0020 """,
            """! => \u0020sentence\u0020 """,
            """? => \u0020sentence\u0020 """,
            """。 => \u0020sentence\u0020 """,
            """？ => \u0020sentence\u0020 """,
            """！ => \u0020sentence\u0020"""
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "mainContent": {
        "type": "text",
        "analyzer": "maincontent_analyzer",
        "search_analyzer": "ik_smart"
      }
    }
  }
}

我们创建了一个名称为sentence_paragrah_mapping的char filter，它的目的有两个：

  • 替换p,h1,h2标签为统一的分段符：paragraph；
  • 替换中英文 ！,？,。 标点符号为统一的分页符：sentence。

有几个细节，需要说明：

  • paragraph和sentence前后都需要添加空格，并且需要使用Unicode \u0020表示空格。

    # 期望
hello world! => hello world sentence
# 不合理的配置，可能会出现下面的情况
hello world! => hello worldsentence
  • </p>,</h1>,</h2>的结尾标签需要添加paragraph和sentence两个分隔符，避免结尾没有标点符号的情况
# 期望
<h1>hello world</h1> <p>hello china</p> => paragraph hello world sentence paragraph hello china sentence

# </p>,</h1>,</h2>只使用paragraph替换的结果
# 此时 hello world hello china 为同句
<h1>hello world</h1> <p>hello china</p> => paragraph hello world  paragraph hello china sentence

# 上面配置结果有些冗余:有两个连续的paragraph 
# 如果能保证HTML文本都符合标准，可以只替换</p>,</h1>,</h2>，不替换<p>,<h1>,<h2>
<h1>hello world</h1> <p>hello china</p> => paragraph hello world sentence paragraph paragraph hello china sentence
  • 注意sentence_paragrah_mapping和html_strip的配置顺序

插入测试数据

POST sample1/_doc/1
{
  "mainContent":"<p>java python javascript</p><p>oracle mysql sqlserver</p>"
} 

# 测试分词
POST sample1/_analyze
{
  "text": ["<p>java python javascript</p><p>oracle mysql sqlserver</p>"],
  "analyzer": "maincontent_analyzer"
}

# 返回结果
{
  "tokens" : [
    {
      "token" : "paragraph",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "ENGLISH",
      "position" : 0
    },
    {
      "token" : "java",
      "start_offset" : 3,
      "end_offset" : 7,
      "type" : "ENGLISH",
      "position" : 1
    },
    {
      "token" : "python",
      "start_offset" : 8,
      "end_offset" : 14,
      "type" : "ENGLISH",
      "position" : 2
    },
    {
      "token" : "javascript",
      "start_offset" : 15,
      "end_offset" : 25,
      "type" : "ENGLISH",
      "position" : 3
    },
    {
      "token" : "sentence",
      "start_offset" : 26,
      "end_offset" : 28,
      "type" : "ENGLISH",
      "position" : 4
    },
    {
      "token" : "paragraph",
      "start_offset" : 28,
      "end_offset" : 28,
      "type" : "ENGLISH",
      "position" : 5
    },
    {
      "token" : "paragraph",
      "start_offset" : 30,
      "end_offset" : 31,
      "type" : "ENGLISH",
      "position" : 6
    },
    {
      "token" : "oracle",
      "start_offset" : 32,
      "end_offset" : 38,
      "type" : "ENGLISH",
      "position" : 7
    },
    {
      "token" : "mysql",
      "start_offset" : 39,
      "end_offset" : 44,
      "type" : "ENGLISH",
      "position" : 8
    },
    {
      "token" : "sqlserver",
      "start_offset" : 45,
      "end_offset" : 54,
      "type" : "ENGLISH",
      "position" : 9
    },
    {
      "token" : "sentence",
      "start_offset" : 55,
      "end_offset" : 57,
      "type" : "ENGLISH",
      "position" : 10
    },
    {
      "token" : "paragraph",
      "start_offset" : 57,
      "end_offset" : 57,
      "type" : "ENGLISH",
      "position" : 11
    }
  ]
}

测试查询

  • 同段查询：java python
GET sample1/_search
{
  "query": {
    "span_not": {
      "include": {
        "span_near": {
          "clauses": [
            {
              "span_term": {
                "mainContent": {
                  "value": "java"
                }
              }
            },
            {
              "span_term": {
                "mainContent": {
                  "value": "python"
                }
              }
            }
          ],
          "slop": 12,
          "in_order": false
        }
      },
      "exclude": {
        "span_term": {
          "mainContent": {
            "value": "paragraph"
          }
        }
      }
    }
  }
}

//结果
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.1655603,
    "hits" : [
      {
        "_index" : "sample1",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.1655603,
        "_source" : {
          "mainContent" : "<p>java python javascript</p><p>oracle mysql sqlserver</p>"
        }
      }
    ]
  }
}
  • 同段查询：java oracle
GET sample1/_search
{
  "query": {
    "span_not": {
      "include": {
        "span_near": {
          "clauses": [
            {
              "span_term": {
                "mainContent": {
                  "value": "java"
                }
              }
            },
            {
              "span_term": {
                "mainContent": {
                  "value": "oracle"
                }
              }
            }
          ],
          "slop": 12,
          "in_order": false
        }
      },
      "exclude": {
        "span_term": {
          "mainContent": {
            "value": "paragraph"
          }
        }
      }
    }
  }
}

#结果：没有文档返回
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

纯文本格式

纯文本和HTML的区别是段落分割符不同，使用\n.

创建索引

PUT sample2
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "analysis": {
      "analyzer": {
        "maincontent_analyzer": {
          "type": "custom",
          "char_filter": [
            "sentence_paragrah_mapping"
          ],
          "tokenizer": "ik_max_word"
        }
      },
      "char_filter": {
        "sentence_paragrah_mapping": {
          "type": "mapping",
          "mappings": [
            """\n => \u0020sentence\u0020paragraph\u0020 """,
            """! => \u0020sentence\u0020 """,
            """? => \u0020sentence\u0020 """,
            """。 => \u0020sentence\u0020 """,
            """？ => \u0020sentence\u0020 """,
            """！ => \u0020sentence\u0020"""
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "mainContent": {
        "type": "text",
        "analyzer": "maincontent_analyzer",
        "search_analyzer": "ik_smart"
      }
    }
  }
}

测试分词

POST sample2/_analyze
{
  "text": ["java python javascript\noracle mysql sqlserver"],
  "analyzer": "maincontent_analyzer"
}

# 结果
{
  "tokens" : [
    {
      "token" : "java",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "ENGLISH",
      "position" : 0
    },
    {
      "token" : "python",
      "start_offset" : 5,
      "end_offset" : 11,
      "type" : "ENGLISH",
      "position" : 1
    },
    {
      "token" : "javascript",
      "start_offset" : 12,
      "end_offset" : 22,
      "type" : "ENGLISH",
      "position" : 2
    },
    {
      "token" : "sentence",
      "start_offset" : 22,
      "end_offset" : 22,
      "type" : "ENGLISH",
      "position" : 3
    },
    {
      "token" : "paragraph",
      "start_offset" : 22,
      "end_offset" : 22,
      "type" : "ENGLISH",
      "position" : 4
    },
    {
      "token" : "oracle",
      "start_offset" : 23,
      "end_offset" : 29,
      "type" : "ENGLISH",
      "position" : 5
    },
    {
      "token" : "mysql",
      "start_offset" : 30,
      "end_offset" : 35,
      "type" : "ENGLISH",
      "position" : 6
    },
    {
      "token" : "sqlserver",
      "start_offset" : 36,
      "end_offset" : 45,
      "type" : "ENGLISH",
      "position" : 7
    }
  ]
}
[尊重社区原创，转载请保留或注明出处]
本文地址：http://elasticsearch.cn/article/13677
5
medcl HelloClyde Marine tacsklet muhao

4 个评论

HelloClyde

HelloClyde

这篇文章真的可以，好顶赞！（ps.我们以前做同段同句都是用position做的）
2020-03-09 14:43
EchoYuan

EchoYuan

SpanNotQuery那里，我觉得可以添加更详细的例子来说明，SpanNotQuery确实不好理解。
以下是我自己做的实验结果：
1. `exclude.span_term.maincontent.value` in [`quick`, `fox`, `jumps`, `over`]，无文档返回；
2. `exclude.span_term.maincontent.value` == `red`，只返回了`the quick brown fox jumps over the lazy dog`这一个文档；
3. `exclude.span_term.maincontent.value` == `brown`，只返回了`the quick red fox jumps over the sleepy cat`这一个文档；
4. `exclude.span_term.maincontent.value` in [`the`, `over`, `lazy`, `dog`, `sleepy`, `cat`]，即如果是content中`quick`之前的所有terms或`over`之后的所有terms，都会返回这两个文档。

结论：
`exclude`和`must_not`的工作方式不一样，它并不会把符合自身条件的docs查询出来然后再从`include`的结果中remove掉它们，而只是在条件这一层面上判断是否包含在`include`的条件范围内。
2021-05-25 16:47
dxyz

dxyz

同句查询能用OR AND这些么大佬？
2021-09-22 09:01
gaojun1212

gaojun1212

打扰问一下，使用char_filter替换的时候，如何保障替换字符之后的内容分词的start_offset，还是原来的内容，例如：
原始文本“java\npython”，不替换前，Python的start_offset为：5，通过char_filter替换paragraph之后，python这个词的start_offset如何继续保证为5，而不是15
2022-04-27 13:53

要回复文章请先登录或注册

Copyright © 2024 · CC BY-NC-SA 3.0

本站服务器及带宽由 提供赞助