绊脚石乃是进身之阶。

logstash6.2运行一段时间就挂了

Logstash | 作者 hexiaohong | 发布于2018年05月16日 | 阅读数:10043

运行1-2天就挂了,先报如下错误,几分钟后就挂了
[2018-05-15T20:52:59,712][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 5,867 milliseconds ago.  The last packet sent successfully to the server was 5,867 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 22,365 milliseconds ago.  The last packet sent successfully to the server was 9,451 milliseconds ago."}
[2018-05-15T20:53:09,060][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 3,647 milliseconds ago.  The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:52:59,713][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 19,578 milliseconds ago.  The last packet sent successfully to the server was 0 milliseconds ago."}
[2018-05-15T20:53:47,835][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Trying again {:error_message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet successfully received from the server was 27,170 milliseconds ago.  The last packet sent successfully to the server was 4,982 milliseconds ago."}
 

数据库连接不上十分钟后,接着就是堆内存溢出:
[2018-05-16T16:34:42,975][WARN ][logstash.inputs.jdbc     ] Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::JavaLang::OutOfMemoryError: Java heap space>}
[2018-05-16T16:34:43,073][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space
 
logstash配置如下;
input {
    stdin {
    }
    jdbc {
            jdbc_connection_string => "jdbc:mysql://*.*.*.*:3306/TEST"
            jdbc_user => "*******************"
            jdbc_password => "*******************"
            jdbc_driver_library => "/data/logstash/lib/mysql-connector-java-5.1.22.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_paging_enabled => "true"
            # 使用前验证连接是否有效
            jdbc_validate_connection => true
            # 多久进行连接有效验证(4小时)
            jdbc_validation_timeout => 14400
            # 连接失败后最大重试次数
            connection_retry_attempts => 50
            # 连接失败后重试时间间隔
            connection_retry_attempts_wait_time => 1

            #使用其他字段追踪,而不是用时间
            use_column_value => true
            #追踪的字段
            tracking_column => FUPDATE_TIME
            record_last_run => true
            jdbc_default_timezone =>"Asia/Shanghai"
            #上一个sql_last_value值的存放文件路径, 必须要在文件中指定字段的初始值
            last_run_metadata_path => "/data/logstash/config/work/time/station_reply.txt"
            #是否将 column 名称转小写
            lowercase_column_names => false
            jdbc_page_size => "10000"
            statement_filepath => "/data/logstash/config/work/sql/reply.sql"
            schedule => "* * * * *"
            type => "reply"
    }
    ...(一共16个这样的jdbc,因为有16张分表)
}    
 
求指导,感谢
已邀请:

rockybean - Elastic Certified Engineer, ElasticStack Fans,公众号:ElasticTalk

赞同来自:

从日志看是你的 dtabase 无法连接了,你查看下是不是 database 的连接池满了

hexiaohong

赞同来自:

问题已经解决。
原因是:logstash6.2.2对应的logstash-input-jdbc插件版本是4.3.3,从插件官方更新日志来看4.3.4还修复了内存泄露问题:

1526984000(1).png

 
[root@localhost bin]# ./logstash-plugin update logstash-input-jdbc
Updating logstash-input-jdbc
Updated logstash-input-jdbc 4.3.3 to 4.3.9
[root@localhost bin]# 
 
总结一下:其实我的这个最大的问题在于内存泄露,因为随着运行,占用内存越来越大,不释放,最后堆内存溢出,导致logstash挂了或者管道关闭。
 

要回复问题请先登录注册