用了Elasticsearch,一口气上5T

用户logstash同步mysql数据到es中7.4.1版本以后输出的sql日志中没有分页信息

Logstash | 作者 future | 发布于2019年12月09日 | 阅读数:1870

问题描述:
在使用logstash同步mysql数据库中的数据,在使用logstash7.4.1以前的版本,输出的sql日志如下:
[2019-12-09T10:50:01,585][INFO ][logstash.inputs.jdbc     ] (0.009082s) SELECT version()
[2019-12-09T10:50:01,634][INFO ][logstash.inputs.jdbc     ] (0.001152s) SELECT version()
[2019-12-09T10:50:01,841][INFO ][logstash.inputs.jdbc     ] (0.002531s) SELECT count(*) AS `count` FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1
[2019-12-09T10:50:01,892][INFO ][logstash.inputs.jdbc     ] (0.009893s) SELECT * FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1 OFFSET 0
[2019-12-09T10:50:02,032][INFO ][logstash.inputs.jdbc     ] (0.001794s) SELECT * FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1 OFFSET 1
[2019-12-09T10:50:02,043][INFO ][logstash.inputs.jdbc     ] (0.001604s) SELECT * FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1 OFFSET 2
[2019-12-09T10:50:02,060][INFO ][logstash.inputs.jdbc     ] (0.001694s) SELECT * FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1 OFFSET 3
[2019-12-09T10:50:02,076][INFO ][logstash.inputs.jdbc     ] (0.001766s) SELECT * FROM (select * from t_test where updatetime > '1970-01-01 08:00:00' and updatetime < date_add(NOW(), interval -1 second) order by updatetime asc) AS `t1` LIMIT 1 OFFSET 4
 
使用logstash7.4.1以后的版本,输出的sql日志如下:
[2019-12-09T15:42:01,445][INFO ][logstash.inputs.jdbc     ][main] (0.009087s) select * from t_test where  updatetime < date_add(NOW(), interval -1 second) order by updatetime asc
 
 
logstash input 脚步如下:
jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test"
    jdbc_user => "root"
    jdbc_password => "test"
    jdbc_default_timezone =>"Asia/Shanghai"
    jdbc_paging_enabled => true
    jdbc_page_size => 1
    schedule => "* * * * *"
    record_last_run => true
    use_column_value => true
    tracking_column => "updatetime"
    tracking_column_type => "timestamp"
    last_run_metadata_path => "/Users/future/work/logstash-7.4.2/lastrunmetadata/test"
    statement => "select * from t_test where  updatetime < date_add(NOW(), interval -1 second) order by updatetime asc"
  }
 
请问为什么在7.4.1及以后的版本在输出的sql日志中没有LIMIT 1 OFFSET 0分页的相关信息?
 
已邀请:

GLC

赞同来自:

用了logstash的分页,就不能用order by 
 
  jdbc_page_size => 1  分页数量也太少了。

要回复问题请先登录注册