用了Elasticsearch,一口气上5T

请教各位大神,我的logstash提取数据后,是无限自循环的数据,怎么办呢?

Logstash | 作者 ljx95315 | 发布于2018年08月02日 | 阅读数:3870

这是一个公司要求logstash测试的配置,从oracle中读取数据,导入到mysql中去
input {
    stdin {
    }
    jdbc {
      jdbc_driver_library => "/usr/share/logstash/config/ojdbc6.jar"
      jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
      jdbc_connection_string => "jdbc:oracle:thin:@//192.168.10.5:1521/orcl"
      jdbc_user => "sdystr"
      jdbc_password => "shandong"

      jdbc_paging_enabled => "true"
      jdbc_page_size => "50000"
      jdbc_default_timezone => "Asia/Shanghai"

      statement => "SELECT ID FROM PERSON_INFO where rownum <= 10 and create_datetime >= :sql_last_value"

      # schedule => "* * * * * Asia/Shanghai"
      # type => ""#####LAB_SERVER_NO#####""

      record_last_run => "true"
      use_column_value => "true"
      tracking_column_type => "timestamp"
      tracking_column => "ID"
      last_run_metadata_path => "/usr/share/logstash/config/last_datetime/PERSON_INFO"
      clean_run => "false"
      lowercase_column_names => "false"
    }
}

filter {
   json {
        source => "message"
        remove_field => ["message","@version","host","@timestamp"]
    }
}

output {
jdbc {
     driver_jar_path => "/opt/logstash-6.3.1/config/mysql-connector-java-8.0.11.jar"
     driver_class => "com.mysql.jdbc.Driver"
     connection_string => "jdbc:mysql://192.168.10.61:3306/ystr2?user=ystr2&password=ystr2"
     statement => ["insert into PERSON_INFO(ID) values(?)","%{ID}"]
    }
}
 
现在问题是,获取数据后发现,mysql上的ID字段是无限循环的,如附件所示,我只检索了10个数据,但是这些数据总是无限循环。
循环_logstash.PNG

 
请教一下各位
 
已邀请:

laoyang360 - 《一本书讲透Elasticsearch》作者,Elastic认证工程师 [死磕Elasitcsearch]知识星球地址:http://t.cn/RmwM3N9;微信公众号:铭毅天下; 博客:https://elastic.blog.csdn.net

赞同来自:

确认下你的自增id还是时间字段?
确认方法:配置的文件会打印输出的

要回复问题请先登录注册