查询cdn日志报错,单term日志大于32K,看报错是成功写入了es,但是,不能查询出来

大家好:
最近发现,cdn日志,处理进es,但是,不能正确查询到结果,求解答。
cdn日志是每100行正常日志上报到服务器,所以,单行日志可能会很大,request_body中可能好多日志的集合,kv形式的。
es版本为:1.6.0,写入es的工具用的childe的hangout
怀疑是,es写入了单个term大于32k了,然后无法查询。
通过kibana页面查询,页面报错是:Courier Fetch: 2 of 4 shards failed.
查看es master的报错是:
[2015-12-18 16:21:08,696][DEBUG][action.search.type       ] [bes01] [1305683] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes04][inet[/10.10.10.166:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][1]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@5065570e>!]: Fetch Failed [Failed to highlight field [request_body]]
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
    at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
    at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
    at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
    at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
    ... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43666
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
    ... 20 more
[2015-12-18 16:21:08,766][DEBUG][action.search.type       ] [bes01] [1307503] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes03][inet[/10.10.10.165:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.17][3]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[298],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@40297afd>!]: Fetch Failed [Failed to highlight field [request_body]]
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
    at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42779
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
    at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
    at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
    at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
    ... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42779
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
    ... 20 more
[2015-12-18 16:21:09,507][DEBUG][action.search.type       ] [bes01] [1307427] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes06][inet[/10.10.10.168:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][1]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1f14b0e>!]: Fetch Failed [Failed to highlight field [request_body]]
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
    at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42325
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
    at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
    at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
    at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
    ... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 42325
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
    ... 20 more
[2015-12-18 16:21:09,547][DEBUG][action.search.type       ] [bes01] [1305672] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes05][inet[/10.10.10.167:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][0]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@633c498c>!]: Fetch Failed [Failed to highlight field [request_body]]
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
    at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44259
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
    at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
    at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
    at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
    ... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 44259
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
    ... 20 more
[2015-12-18 16:21:09,587][DEBUG][action.search.type       ] [bes01] [1305685] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [bes04][inet[/10.10.10.166:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.fetch.FetchPhaseExecutionException: [logstash-es-img-2015.12.16][2]: query[filtered(did:02f715e2072fc7265b14f4470f565b2b)->BooleanFilter(+cache(@timestamp:[1450167667735 TO 1450426867736]))],from[0],size[288],sort[<custom:"@timestamp": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@5acd403d>!]: Fetch Failed [Failed to highlight field [request_body]]
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:133)
    at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:133)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:504)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32889
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:493)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:396)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:371)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:351)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getLeafContext(WeightedSpanTermExtractor.java:360)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:216)
    at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
    at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
    at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
    at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
    at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:119)
    ... 10 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32889
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)
    at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:467)
    ... 20 more
已邀请:

samir

赞同来自:

curl  http://127.0.0.1:9200/logstash-*/_cache/clear   ##清理缓存即可。试试吧 

要回复问题请先登录注册