即使是不成熟的尝试,也胜于胎死腹中的策略。

logstash可以更新或删除elasticsearch中数据吗

Logstash | 作者 hwzhizun1 | 发布于2018年03月09日 | 阅读数:7223

比如我有一张mysql表,需要每天导入到elasticsearch中,并按mysql主键进行更新,各位大神,这个需求可以做吗?
已邀请:

BrickXu - BlackOps@Qunar

赞同来自: rochy

当然可以啊,elsticsearch output的action专门干这个的,请直接看https://www.elastic.co/guide/e ... ction


  • index: indexes a document (an event from Logstash).
  • delete: deletes a document by id (An id is required for this action)
  • create: indexes a document, fails if a document by that id already exists in the index.
  • update: updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the upsert option. NOTE: This does not work and is not supported in Elasticsearch 1.x. Please upgrade to ES 2.x or greater to use this feature with Logstash!

dongne

赞同来自:

可以直接调用 restful 接口处理 es 啊,为什么要用 logstash 呢

laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

除了楼上,这边已验证过logstash_input_jdbc支持增、更新操作,删除不支持。更新注意id的唯一性

super9du

赞同来自:

我的 conf 配置如下,但是多个jdbc一起导入es更新不成功什么原因?配置有误吗?
input{
jdbc {
type => "111"

jdbc_connection_string => "jdbc:mysql://localhost:3306/111?characterEncoding=UTF-8&useSSL=false&serverTimezone=CTT"
jdbc_user => "root"
jdbc_password => "111"
jdbc_driver_library => "/home/logstash-7.3.1/mysql/mysql-connector-java-8.0.18.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_paging_enabled => true
jdbc_page_size => 1000

record_last_run => true
use_column_value => false
tracking_column => "modify_date"
tracking_column_type => "timestamp"
last_run_metadata_path => "/home/logstash-7.3.1/data/111.txt"

statement => "select * from 111 where modify_date > :sql_last_value"
schedule => "* * * * *"
}
}
output{
if [type] == "111" {
elasticsearch {
action => "index"
hosts => ["localhost:9201","localhost:9202"]
index => "111"
document_id => "%{id}"
template_overwrite => true
manage_template => false
}
stdout {
codec => json_lines
}
}
}

要回复问题请先登录注册