身安不如心安,屋宽不如心宽 。

flume读取kafka数据写入ES2.1+shield

Elasticsearch | 作者 suwensen | 发布于2016年01月17日 | 阅读数:8098

备注:新建立一个ES的cluster.name,flume启动,能正常写入ES一段时间,但是一会就开始报一下错误了。
改了flume的sink源码,像官网那样加入了shield
请大家帮忙看下有没有遇见此问题的。

17 一月 2016 11:22:10,117 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.elasticsearch.ElasticSearchSink.process:221) - Failed to commit transaction. Transaction rolled back.
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{192.168.49.3}{home1/192.168.49.3:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:206)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
17 一月 2016 11:22:10,119 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
已邀请:

suwensen - 如影随形

赞同来自: sky7644

已经改好flume插件,已经调试好了。

medcl - 今晚打老虎。

赞同来自:

es能正常写入一段时间,说明连接配置是对的,你看看是不是es挂了?

sky7644

赞同来自:

先确定是真的已经写入到ES中了吗,你说的一段时间之后报错,会不会看到的只是前面在打包batch的过程。

要回复问题请先登录注册