好的想法是十分钱一打,真正无价的是能够实现这些想法的人。

logstash同步mysql数据到elasticsearch时数据错乱

Logstash | 作者 hezhiqiang | 发布于2018年04月25日 | 阅读数:2732

我有两个表:kid_question_category和kid_question,分别同步到kid_question_category和kid_question两个单独的索引,但发现有些kid_question_category表的数据会跑到kid_question索引中,大家有遇到过类似问题吗?怎么解决?
 
我写了两个配置文件:
一、kid_question_category.conf
input {
        jdbc {
                jdbc_driver_library => "/home/mysql-connector-java-5.1.45-bin.jar"
                jdbc_driver_class => "com.mysql.jdbc.Driver"
                jdbc_connection_string => "jdbc:mysql://192.168.6.67:3306/devkiddo?useUnicode=true&characterEncoding=utf8"
                jdbc_user => "root"
                jdbc_password => "xxxx"
                schedule => "* * * * *"
                statement => "SELECT * FROM kid_question_category WHERE modified_at >= :sql_last_value"
                last_run_metadata_path => "/home/logstash_last_run_metadata_path_kid_question_category"
                tracking_column => "modified_at"
                tracking_column_type => "timestamp"
                jdbc_default_timezone => "Asia/Shanghai"
        }
}

output {
        elasticsearch {
                index => "kid_question_category"
                document_id => "%{id}"
                document_type => "kid_question_category"
                user => "logstash"
                password => "xxxx"
        }
}
二、kid_question.conf
input {
        jdbc {
                jdbc_driver_library => "/home/mysql-connector-java-5.1.45-bin.jar"
                jdbc_driver_class => "com.mysql.jdbc.Driver"
                jdbc_connection_string => "jdbc:mysql://192.168.6.67:3306/devkiddo?useUnicode=true&characterEncoding=utf8"
                jdbc_user => "root"
                jdbc_password => "xxxx"
                schedule => "* * * * *"
                statement => "SELECT * FROM kid_question e WHERE e.modified_at >= :sql_last_value"
                last_run_metadata_path => "/home/logstash_last_run_metadata_path_kid_question"
                tracking_column => "modified_at"
                tracking_column_type => "timestamp"
                jdbc_default_timezone => "Asia/Shanghai"
        }
}

output {
        elasticsearch {
                index => "kid_question"
                document_id => "%{id}"
                document_type => "kid_question"
                user => "logstash"
                password => "xxxx"
        }
}
已邀请:

strglee

赞同来自: hezhiqiang

input {
jdbc {
jdbc_driver_library => "/home/mysql-connector-java-5.1.45-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://192.168.6.67:3306/devkiddo?useUnicode=true&characterEncoding=utf8"
jdbc_user => "root"
jdbc_password => "xxxx"
schedule => "* * * * *"
statement => "SELECT * FROM kid_question_category WHERE modified_at >= :sql_last_value"
last_run_metadata_path => "/home/logstash_last_run_metadata_path_kid_question_category"
tracking_column => "modified_at"
tracking_column_type => "timestamp"
jdbc_default_timezone => "Asia/Shanghai"
type => "kid_question_category" # 定义type
}
}

output {
if [type] == "kid_question_category" { # 判断 type
elasticsearch {
index => "kid_question_category"
document_id => "%{id}"
user => "logstash"
password => "xxxx"
}
}
}

laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

要回复问题请先登录注册