logstash thread_pool rejected:1800

最近发现logstash一直在报一个错,还请各位大神帮忙解答一下:
/10.10.250.37:81/, :error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-04-11T11:35:52,398][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-04-11T11:35:52,999][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out {:url=>http://10.10.250.37:81/, :error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-04-11T11:35:52,999][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>8}
[2018-04-11T11:35:55,657][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.10.250.37:81/, :path=>"/"}
[2018-04-11T11:35:55,659][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x771329db URL:http://10.10.250.37:81/&gt;}
[2018-04-11T11:36:01,067][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out {:url=>http://10.10.250.37:81/, :error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2018-04-11T11:36:01,067][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.10.250.37:81/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2018-04-11T11:36:01,217][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2018-04-11T11:36:01,218][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>16}

[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
[2018-04-11T11:36:59,864][ERROR][logstash.outputs.elasticsearch] Action
查看es threads_pool情况
"thread_pool" : {
"bulk" : {
"threads" : 8,
"queue" : 28,
"active" : 8,
"rejected" : 414,
"largest" : 8,
"completed" : 695034
请问这个压力是来自es还是logstash?
已邀请:

yayg2008

赞同来自:

简单理解就是你的ES bulk线程忙不过来了。现在你的bulk8线程火力全开,队列还是有堆积。从而导致logstash连接数被用完。

要回复问题请先登录注册