即使是不成熟的尝试,也胜于胎死腹中的策略。

mysql数据迁移到es,使用logstash报错

Logstash | 作者 Kyrie | 发布于2019年10月22日 | 阅读数:546

报错信息如下:
[2019-10-22T10:00:00,228][ERROR][logstash.inputs.jdbc     ] Unable to connect to database. Tried 1 times {:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}
{ 2014 rufus-scheduler intercepted an error:
  2014   job:
  2014     Rufus::Scheduler::CronJob "* * * * *" {}
  2014   error:
  2014     2014
  2014     Sequel::DatabaseConnectionError
  2014     Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  2014       com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(com/mysql/cj/jdbc/exceptions/SQLError.java:174)
  2014       com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(com/mysql/cj/jdbc/exceptions/SQLExceptionsMapping.java:64)
  2014       com.mysql.cj.jdbc.ConnectionImpl.createNewIO(com/mysql/cj/jdbc/ConnectionImpl.java:827)
  2014       com.mysql.cj.jdbc.ConnectionImpl.<init>(com/mysql/cj/jdbc/ConnectionImpl.java:447)
  2014       com.mysql.cj.jdbc.ConnectionImpl.getInstance(com/mysql/cj/jdbc/ConnectionImpl.java:237)
  2014       com.mysql.cj.jdbc.NonRegisteringDriver.connect(com/mysql/cj/jdbc/NonRegisteringDriver.java:199)
  2014       java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
  2014       org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:468)
  2014       org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:326)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.adapters.jdbc.connect(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/adapters/jdbc.rb:215)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.adapters.jdbc.RUBY$method$connect$0$__VARARGS__(usr/local/elasticsearch/logstash_minus_6_dot_5_dot_4/vendor/bundle/jruby/$2_dot_3_dot_0/gems/sequel_minus_5_dot_14_dot_0/lib/sequel/adapters//usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/adapters/jdbc.rb)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.connection_pool.make_new(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/connection_pool.rb:127)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/connection_pool/threaded.rb:206)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/connection_pool/threaded.rb:138)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/connection_pool/threaded.rb:90)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.database.connecting.synchronize(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/database/connecting.rb:270)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.database.connecting.test_connection(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/database/connecting.rb:279)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.database.connecting.connect(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/database/connecting.rb:58)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.sequel_minus_5_dot_14_dot_0.lib.sequel.core.connect(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/sequel-5.14.0/lib/sequel/core.rb:121)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.plugin_mixins.jdbc.jdbc.block in jdbc_connect(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)
  2014       org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1292)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:112)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:165)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.plugin_mixins.jdbc.jdbc.execute_statement(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.inputs.jdbc.execute_query(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_jdbc_minus_4_dot_3_dot_13.lib.logstash.inputs.jdbc.block in run(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:258)
  2014       org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
  2014       org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.do_call(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.do_trigger(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.block in start_work_thread(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300)
  2014       org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
  2014       org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.block in start_work_thread(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299)
  2014       org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1292)
  2014       org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)
  2014       usr.local.elasticsearch.logstash_minus_6_dot_5_dot_4.vendor.bundle.jruby.$2_dot_3_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.block in start_work_thread(/usr/local/elasticsearch/logstash-6.5.4/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289)
  2014       org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
  2014       org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
  2014       java.lang.Thread.run(java/lang/Thread.java:748)
  2014   tz:
  2014     ENV['TZ']: 
  2014     Time.now: 2019-10-22 10:00:00 +0800
  2014   scheduler:
  2014     object_id: 2008
  2014     opts:
  2014       {:max_work_threads=>1}
  2014       frequency: 0.3
  2014       scheduler_lock: #<Rufus::Scheduler::NullLock:0x5ec5deb0>
  2014       trigger_lock: #<Rufus::Scheduler::NullLock:0x4a7e66f3>
  2014     uptime: 133.752668 (2m13s752)
  2014     down?: false
  2014     threads: 2
  2014       thread: #<Thread:0x5cbfeb65>
  2014       thread_key: rufus_scheduler_2008
  2014       work_threads: 1
  2014         active: 1
  2014         vacant: 0
  2014         max_work_threads: 1
  2014       mutexes: {}
  2014     jobs: 1
  2014       at_jobs: 0
  2014       in_jobs: 0
  2014       every_jobs: 0
  2014       interval_jobs: 0
  2014       cron_jobs: 1
  2014     running_jobs: 1
  2014     work_queue: 0
} 2014 .
mysql.conf文件内容如下:
input {
stdin {
}
jdbc {
# mysql 数据库链接,shop为数据库名
jdbc_connection_string => "jdbc:mysql://localhost:3306/community?serverTimezone=GMT%2B8&useUnicode=true&characterEncoding=utf-8"
# 用户名和密码
jdbc_user => "root"
jdbc_password => ""
# 驱动
jdbc_driver_library => "/usr/local/elasticsearch/logstash-6.5.4/mysqletc/mysql-connector-java-8.0.17.jar"
# 驱动类名
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
# 执行的sql 文件路径+名称
statement_filepath => "select * from tb_question"
# 设置监听间隔 各字段含义(由左至右)分、时、天、月、年,全部为*默认含义为每分钟都更新
schedule => "* * * * *"
# 索引类型
type => "question"
}
}

filter {
json {
source => "message"
remove_field => ["message"]
}
}

output {
elasticsearch {
hosts => ["192.168.1.98:9200"]
index => "community"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
已邀请:

rochy - rochy_he

赞同来自: Kyrie

Unable to connect to database. Tried 1 times 
检查一下数据库连接

要回复问题请先登录注册