点击visualize的时候告警:Visualize:java.lang.lllegalStateException:Field daVisualize
Kibana | 作者 velion | 发布于2015年11月30日 | 阅读数:16393
环境版本:elasticsearch-2.1.0/kibana-4.3.0/java 1.8.0_65
kibana相关页面告警:
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2015.11.29","node":"ODyds4KgQsqrxczx1ana8A","reason":{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"illegal_state_exception","reason":"Field data loading is forbidden on path"}}}}]}} KbnError@http://www.xxx.cn:80/bundles/c ... 64:30 RequestFailure@http://www.xxx.cn:80/bundles/c ... 97:19 http://www.xxx.cn:80/bundles/k ... 05:57 http://www.xxx.cn:80/bundles/c ... 91:28 http://www.xxx.cn:80/bundles/c ... 60:31 map@[native code] map@http://www.xxx.cn:80/bundles/c ... 59:34 callResponseHandlers@http://www.xxx.cn:80/bundles/k ... 77:26 http://www.xxx.cn:80/bundles/k ... 84:37 processQueue@http://www.xxx.cn:80/bundles/c ... 09:31 http://www.xxx.cn:80/bundles/c ... 25:40 $eval@http://www.xxx.cn:80/bundles/c ... 53:29 $digest@http://www.xxx.cn:80/bundles/c ... 64:37 $apply@http://www.xxx.cn:80/bundles/c ... 61:32 done@http://www.xxx.cn:80/bundles/c ... 10:54 completeRequest@http://www.xxx.cn:80/bundles/c ... 08:16 requestLoaded@http://www.xxx.cn:80/bundles/c ... 49:25
elasticsearch相关日志告警:
[2015-11-30 00:04:53,395][DEBUG][action.search.type ] [Thumbelina] [logstash-2015.11.29][2], node[acrTX4O0RciN8ppbSdfoww], [P], v[4], s[STARTED], a[id=ogwkZP0yQCCgatW0_tnvnw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@aef2ffe] lastShard [true]
RemoteTransportException[[Thumbelina][192.168.1.76:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
Caused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:343)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:165)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:82)
at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$FieldData.bytesValues(ValuesSource.java:195)
at org.elasticsearch.search.aggregations.bucket.terms.StringTermsAggregator.getLeafCollector(StringTermsAggregator.java:73)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:132)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:38)
at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:117)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:763)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:324)
... 11 more
Vian 00:18:36
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Field data loading is forbidden on response
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:156)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:76)
… 19 more
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on response
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.fail(DisabledIndexFieldData.java:68)
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.loadDirect(DisabledIndexFieldData.java:54)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:163)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:156)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
… 23 more
补充:
根据页面提示,无法访问"index":"logstash-2015.11.29","node":"ODyds4KgQsqrxczx1ana8A"
首先通过直接访问那个node,发现无法访问,通过手工post提交到对应node后,能单独get到,但问题依然存在,告警内容相同。
kibana相关页面告警:
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2015.11.29","node":"ODyds4KgQsqrxczx1ana8A","reason":{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on path","caused_by":{"type":"illegal_state_exception","reason":"Field data loading is forbidden on path"}}}}]}} KbnError@http://www.xxx.cn:80/bundles/c ... 64:30 RequestFailure@http://www.xxx.cn:80/bundles/c ... 97:19 http://www.xxx.cn:80/bundles/k ... 05:57 http://www.xxx.cn:80/bundles/c ... 91:28 http://www.xxx.cn:80/bundles/c ... 60:31 map@[native code] map@http://www.xxx.cn:80/bundles/c ... 59:34 callResponseHandlers@http://www.xxx.cn:80/bundles/k ... 77:26 http://www.xxx.cn:80/bundles/k ... 84:37 processQueue@http://www.xxx.cn:80/bundles/c ... 09:31 http://www.xxx.cn:80/bundles/c ... 25:40 $eval@http://www.xxx.cn:80/bundles/c ... 53:29 $digest@http://www.xxx.cn:80/bundles/c ... 64:37 $apply@http://www.xxx.cn:80/bundles/c ... 61:32 done@http://www.xxx.cn:80/bundles/c ... 10:54 completeRequest@http://www.xxx.cn:80/bundles/c ... 08:16 requestLoaded@http://www.xxx.cn:80/bundles/c ... 49:25
elasticsearch相关日志告警:
[2015-11-30 00:04:53,395][DEBUG][action.search.type ] [Thumbelina] [logstash-2015.11.29][2], node[acrTX4O0RciN8ppbSdfoww], [P], v[4], s[STARTED], a[id=ogwkZP0yQCCgatW0_tnvnw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@aef2ffe] lastShard [true]
RemoteTransportException[[Thumbelina][192.168.1.76:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
Caused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:343)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:165)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:82)
at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$FieldData.bytesValues(ValuesSource.java:195)
at org.elasticsearch.search.aggregations.bucket.terms.StringTermsAggregator.getLeafCollector(StringTermsAggregator.java:73)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:132)
at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:38)
at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:117)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:763)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:324)
... 11 more
Vian 00:18:36
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Field data loading is forbidden on response
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:156)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:76)
… 19 more
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on response
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.fail(DisabledIndexFieldData.java:68)
at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.loadDirect(DisabledIndexFieldData.java:54)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:163)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:156)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
… 23 more
补充:
根据页面提示,无法访问"index":"logstash-2015.11.29","node":"ODyds4KgQsqrxczx1ana8A"
首先通过直接访问那个node,发现无法访问,通过手工post提交到对应node后,能单独get到,但问题依然存在,告警内容相同。
2 个回复
medcl - 今晚打老虎。
赞同来自:
参照:[url=https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#fielddata-format]https://www.elastic.co/guide/e ... ormat[/url]
velion - 90后it男
赞同来自:
具体细节:
https://github.com/elastic/ela ... v%3D1