在 Mapping 里面，将 dynamic 参数设置成 strict 可以拒绝索引包含未知字段的文档。 此条 Tips 由 medcl 贡献。

求助。ES集群运行一段时间就会不断的full gc。。。请看日志

Elasticsearch | 作者 sdkjfssls | 发布于2017年05月14日 | 阅读数：4605
分享到：QQ空间新浪微博微信QQ好友印象笔记有道云笔记
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4b80b10d5b, pid=23725, tid=139960519792384
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x47ad5b]  Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x20b
#
# Failed to write core dump. Core dumps have been disabled. To enable c
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f4b7c1e6000):  GCTaskThread [stack: 0x00007f4b19009000,0x00007f4b1910a000] [id=23761]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000209ccae90

Registers:
RAX=0x00007f4b814bc360, RBX=0x00007f4b814d7bf8, RCX=0x0000000000000003, RDX=0x0000000209ccae80
RSP=0x00007f4b19108770, RBP=0x00007f4b19108870, RSI=0x00000006413995e8, RDI=0x00000007feba90d8
R8 =0x00007f4b814e1800, R9 =0x00007f4b7c1e2fa8, R10=0x00007f4b7c1e3128, R11=0x00007f4b810075e0
R12=0x00007f4b7c1ed280, R13=0x00007f4b19108940, R14=0x00007f4b7c1e3030, R15=0x00007f4b19108940
RIP=0x00007f4b80b10d5b, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f4b19108770)
0x00007f4b19108770:   0000000000000000 00007f4b19108810
0x00007f4b19108780:   00007f4b19108820 00007f4b19108790
0x00007f4b19108790:   00007f4b814867d0 00007f4b7c1e6f00
0x00007f4b191087a0:   00007f4b7c22f0c0 00007f4b7c1e2cf0
0x00007f4b191087b0:   00007f4b7c1e3238 00007fff65fffa00
0x00007f4b191087c0:   0000000458790000 0000000074f0e000
0x00007f4b191087d0:   00000007feb00028 000000000001fffb
0x00007f4b191087e0:   00007f4b7c1e2fa8 00007f4b7c1ed280
0x00007f4b191087f0:   00007f4b7c1e3128 00000007feba9248
0x00007f4b19108800:   00007f4b03af9a50 00007f4b19108940
0x00007f4b19108810:   05c93cb400003108 00007f4b7c1e7470
0x00007f4b19108820:   05e16c8e0000655f 00007fff65fffa47
0x00007f4b19108830:   05e16c8f0000655f 00007f4b80f25c2e
0x00007f4b19108840:   00007f4b7c22f0c0 00007f4b19108940
0x00007f4b19108850:   00000007feba91c8 000000000e990640
0x00007f4b19108860:   00007f4b7c1e3030 00007f4b19108940
0x00007f4b19108870:   00007f4b19108890 00007f4b80b111c9
0x00007f4b19108880:   0000000074c83239 0000000074c8e000
0x00007f4b19108890:   00007f4b191088e0 00007f4b809a53ad
0x00007f4b191088a0:   00007f4b19108920 0000000001d320c8
0x00007f4b191088b0:   0000000001d32381 00000007feb00028
0x00007f4b191088c0:   00007f4b7c111840 00007f4b03af9a50
0x00007f4b191088d0:   00007f4b03af99d0 00000007fae00000
0x00007f4b191088e0:   00007f4b19108a40 00007f4b80b026d6
0x00007f4b191088f0:   0000000419108980 00007f4b19108940
0x00007f4b19108900:   00007f4b191089e0 00007f4b191089c0
0x00007f4b19108910:   00007f4b191089d0 00007f4b19108a0c
0x00007f4b19108920:   0000000219108a40 0000000000100000
0x00007f4b19108930:   00007f4b7c1118a8 0000000000000010
0x00007f4b19108940:   00007f4b81486910 00007f4b7c1e2cf0
0x00007f4b19108950:   0000000458790000 0000000074f0e000
0x00007f4b19108960:   00000007feb00028 000000000001fffb 

Instructions: (pc=0x00007f4b80b10d5b)
0x00007f4b80b10d3b:   59 16 99 00 80 3b 00 0f 84 80 02 00 00 48 8b 05
0x00007f4b80b10d4b:   29 e8 98 00 8b 56 08 8b 48 08 48 d3 e2 48 03 10
0x00007f4b80b10d5b:   48 8b 42 10 48 8d 7a 10 48 8b 95 18 ff ff ff ff
0x00007f4b80b10d6b:   90 60 02 00 00 48 8b 05 61 ef 98 00 8b 00 85 c0 

Register to memory mapping:

RAX=0x00007f4b814bc360: <offset 0xe26360> in /usr/java/jdk1.7.0_80/jre/lib/amd64/server/libjvm.so at 0x00007f4b80696000
RBX=0x00007f4b814d7bf8: <offset 0xe41bf8> in /usr/java/jdk1.7.0_80/jre/lib/amd64/server/libjvm.so at 0x00007f4b80696000
RCX=0x0000000000000003 is an unknown value
RDX=0x0000000209ccae80 is an unknown value
RSP=0x00007f4b19108770 is an unknown value
RBP=0x00007f4b19108870 is an unknown value
RSI=0x00000006413995e8 is an unknown value
RDI=0x00000007feba90d8 is an oop
[B 
 - klass: {type array byte}
 - length: 35
R8 =0x00007f4b814e1800: <offset 0xe4b800> in /usr/java/jdk1.7.0_80/jre/lib/amd64/server/libjvm.so at 0x00007f4b80696000
R9 =0x00007f4b7c1e2fa8 is an unknown value
R10=0x00007f4b7c1e3128 is an unknown value
R11=0x00007f4b810075e0: <offset 0x9715e0> in /usr/java/jdk1.7.0_80/jre/lib/amd64/server/libjvm.so at 0x00007f4b80696000
R12=0x00007f4b7c1ed280 is an unknown value
R13=0x00007f4b19108940 is an unknown value
R14=0x00007f4b7c1e3030 is an unknown value
R15=0x00007f4b19108940 is an unknown value


Stack: [0x00007f4b19009000,0x00007f4b1910a000],  sp=0x00007f4b19108770,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x47ad5b]  Par_MarkFromRootsClosure::scan_oops_in_oop(HeapWord*)+0x20b
V  [libjvm.so+0x47b1c9]  Par_MarkFromRootsClosure::do_bit(unsigned long)+0xc9
V  [libjvm.so+0x30f3ad]  BitMap::iterate(BitMapClosure*, unsigned long, unsigned long)+0xbd
V  [libjvm.so+0x46c6d6]  CMSConcMarkingTask::do_scan_and_mark(int, CompactibleFreeListSpace*)+0x326
V  [libjvm.so+0x47aa19]  CMSConcMarkingTask::work(unsigned int)+0xe9
V  [libjvm.so+0x9bc0de]  YieldingFlexibleGangWorker::loop()+0xae
V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f4858093000 JavaThread "elasticsearch[node-100][[materialfac_2][1]: Lucene Merge Thread #393]" daemon [_thread_blocked, id=2825, stack(0x00007f479bb68000,0x00007f479bc69000)]
  0x00007f4ac400d000 JavaThread "elasticsearch[node-100][flush][T#278]" daemon [_thread_blocked, id=2577, stack(0x00007f46f67b9000,0x00007f46f68ba000)]
  0x00007f4ac4009800 JavaThread "elasticsearch[node-100][flush][T#277]" daemon [_thread_blocked, id=2576, stack(0x00007f4780889000,0x00007f478098a000)]
  0x00007f4ac4008800 JavaThread "elasticsearch[node-100][refresh][T#461]" daemon [_thread_blocked, id=2573, stack(0x00007f449d79e000,0x00007f449d89f000)]
  0x00007f485810d800 JavaThread "elasticsearch[node-100][warmer][T#114]" daemon [_thread_blocked, id=2443, stack(0x00007f47ae25d000,0x00007f47ae35e000)]
  0x00007f4ac4008000 JavaThread "elasticsearch[node-100][refresh][T#460]" daemon [_thread_blocked, id=1759, stack(0x00007f48a97d8000,0x00007f48a98d9000)]
  0x00007f4ac400b800 JavaThread "elasticsearch[node-100][flush][T#274]" daemon [_thread_blocked, id=448, stack(0x00007f4565295000,0x00007f4565396000)]
  0x00007f49c009c800 JavaThread "elasticsearch[node-100][generic][T#33]" daemon [_thread_blocked, id=32698, stack(0x00007f478098a000,0x00007f4780a8b000)]
  0x00007f4ac4007000 JavaThread "elasticsearch[node-100][refresh][T#455]" daemon [_thread_blocked, id=30699, stack(0x00007f449dd5a000,0x00007f449de5b000)]
  0x00007f4858084000 JavaThread "elasticsearch[node-100][warmer][T#112]" daemon [_thread_blocked, id=28709, stack(0x00007f449f8e8000,0x00007f449f9e9000)]
  0x00007f48c00c8000 JavaThread "elasticsearch[node-100][management][T#4]" daemon [_thread_blocked, id=5264, stack(0x00007f449d89f000,0x00007f449d9a0000)]
  0x00007f4a8c084800 JavaThread "elasticsearch[node-100][listener][T#10]" daemon [_thread_blocked, id=5211, stack(0x00007f48aaaeb000,0x00007f48aabec000)]
  0x00007f4a88085000 JavaThread "elasticsearch[node-100][listener][T#9]" daemon [_thread_blocked, id=5210, stack(0x00007f48aa5e6000,0x00007f48aa6e7000)]
  0x00007f4a94081000 JavaThread "elasticsearch[node-100][listener][T#8]" daemon [_thread_blocked, id=5175, stack(0x00007f478247c000,0x00007f478257d000)]
  0x00007f4a80081000 JavaThread "elasticsearch[node-100][listener][T#7]" daemon [_thread_blocked, id=5174, stack(0x00007f479ba67000,0x00007f479bb68000)]
  0x00007f4a8c084000 JavaThread "elasticsearch[node-100][listener][T#6]" daemon [_thread_blocked, id=5173, stack(0x00007f4852d3f000,0x00007f4852e40000)]
  0x00007f4a7808d000 JavaThread "elasticsearch[node-100][listener][T#5]" daemon [_thread_blocked, id=5172, stack(0x00007f4498065000,0x00007f4498166000)]
  0x00007f4a8c082000 JavaThread "elasticsearch[node-100][listener][T#4]" daemon [_thread_blocked, id=5163, stack(0x00007f47afd47000,0x00007f47afe48000)]
  0x00007f4a84081000 JavaThread "elasticsearch[node-100][listener][T#3]" daemon [_thread_blocked, id=5162, stack(0x00007f4498166000,0x00007f4498267000)]
  0x00007f4a8c081000 JavaThread "elasticsearch[node-100][listener][T#2]" daemon [_thread_blocked, id=5161, stack(0x00007f449d9a0000,0x00007f449daa1000)]
  0x00007f4a88084000 JavaThread "elasticsearch[node-100][listener][T#1]" daemon [_thread_blocked, id=5160, stack(0x00007f48ab1f2000,0x00007f48ab2f3000)]
  0x00007f49f408b800 JavaThread "elasticsearch[node-100][index][T#24]" daemon [_thread_blocked, id=26603, stack(0x00007f4498267000,0x00007f4498368000)]
  0x00007f493c091800 JavaThread "elasticsearch[node-100][index][T#23]" daemon [_thread_blocked, id=26598, stack(0x00007f4498368000,0x00007f4498469000)]
  0x00007f4958094000 JavaThread "elasticsearch[node-100][index][T#22]" daemon [_thread_blocked, id=26597, stack(0x00007f449bd19000,0x00007f449be1a000)]
  0x00007f49c81b5000 JavaThread "elasticsearch[node-100][index][T#21]" daemon [_thread_blocked, id=26596, stack(0x00007f449be1a000,0x00007f449bf1b000)]
  0x00007f49d0092800 JavaThread "elasticsearch[node-100][index][T#20]" daemon [_thread_blocked, id=26595, stack(0x00007f48a91d2000,0x00007f48a92d3000)]
  0x00007f49d8086800 JavaThread "elasticsearch[node-100][index][T#19]" daemon [_thread_blocked, id=26594, stack(0x00007f48a98d9000,0x00007f48a99da000)]
  0x00007f49f408b000 JavaThread "elasticsearch[node-100][index][T#18]" daemon [_thread_blocked, id=26592, stack(0x00007f48a9fe0000,0x00007f48aa0e1000)]
  0x00007f493c090800 JavaThread "elasticsearch[node-100][index][T#17]" daemon [_thread_blocked, id=26590, stack(0x00007f48aa1e2000,0x00007f48aa2e3000)]
  0x00007f4958093000 JavaThread "elasticsearch[node-100][index][T#16]" daemon [_thread_blocked, id=26589, stack(0x00007f48aa3e4000,0x00007f48aa4e5000)]
  0x00007f49c81b4800 JavaThread "elasticsearch[node-100][index][T#15]" daemon [_thread_blocked, id=26588, stack(0x00007f49ac0c1000,0x00007f49ac1c2000)]
  0x00007f4958092000 JavaThread "elasticsearch[node-100][index][T#14]" daemon [_thread_blocked, id=26473, stack(0x00007f49ac6c7000,0x00007f49ac7c8000)]
  0x00007f49c81b3800 JavaThread "elasticsearch[node-100][index][T#13]" daemon [_thread_blocked, id=26472, stack(0x00007f48ab6f7000,0x00007f48ab7f8000)]
  0x00007f49d0083000 JavaThread "elasticsearch[node-100][management][T#3]" daemon [_thread_blocked, id=26027, stack(0x00007f48a92d3000,0x00007f48a93d4000)]
  0x00007f49d008a800 JavaThread "elasticsearch[node-100][index][T#12]" daemon [_thread_blocked, id=25163, stack(0x00007f49ac7c8000,0x00007f49ac8c9000)]
  0x00007f49d8105800 JavaThread "elasticsearch[node-100][index][T#11]" daemon [_thread_blocked, id=25162, stack(0x00007f48a99da000,0x00007f48a9adb000)]
  0x00007f49f4089000 JavaThread "elasticsearch[node-100][index][T#10]" daemon [_thread_blocked, id=25161, stack(0x00007f49ac3c4000,0x00007f49ac4c5000)]
  0x00007f493c089800 JavaThread "elasticsearch[node-100][index][T#9]" daemon [_thread_blocked, id=25160, stack(0x00007f48aaeef000,0x00007f48aaff0000)]
  0x00007f4958085800 JavaThread "elasticsearch[node-100][index][T#8]" daemon [_thread_blocked, id=25159, stack(0x00007f48aa7e8000,0x00007f48aa8e9000)]
  0x00007f49c8089800 JavaThread "elasticsearch[node-100][index][T#7]" daemon [_thread_blocked, id=25150, stack(0x00007f48ab4f5000,0x00007f48ab5f6000)]
  0x00007f49d0094000 JavaThread "elasticsearch[node-100][index][T#6]" daemon [_thread_blocked, id=25148, stack(0x00007f48ab0f1000,0x00007f48ab1f2000)]
  0x00007f49d808a800 JavaThread "elasticsearch[node-100][index][T#5]" daemon [_thread_blocked, id=25143, stack(0x00007f48aa0e1000,0x00007f48aa1e2000)]
  0x00007f49c8087800 JavaThread "elasticsearch[node-100][management][T#2]" daemon [_thread_blocked, id=25072, stack(0x00007f48aaff0000,0x00007f48ab0f1000)]
  0x00007f4958085000 JavaThread "elasticsearch[node-100][index][T#4]" daemon [_thread_blocked, id=24985, stack(0x00007f48aabec000,0x00007f48aaced000)]
  0x00007f49c8081000 JavaThread "elasticsearch[node-100][index][T#3]" daemon [_thread_blocked, id=24976, stack(0x00007f48a9cdd000,0x00007f48a9dde000)]
  0x00007f4958001000 JavaThread "elasticsearch[node-100][index][T#2]" daemon [_thread_blocked, id=24629, stack(0x00007f48abdfe000,0x00007f48abeff000)]
  0x00007f495c097000 JavaThread "elasticsearch[node-100][bulk][T#24]" daemon [_thread_blocked, id=24620, stack(0x00007f48ab8f9000,0x00007f48ab9fa000)]
  0x00007f4970096800 JavaThread "elasticsearch[node-100][bulk][T#23]" daemon [_thread_blocked, id=24619, stack(0x00007f48a93d4000,0x00007f48a94d5000)]
  0x00007f49e0090000 JavaThread "elasticsearch[node-100][bulk][T#22]" daemon [_thread_blocked, id=24618, stack(0x00007f48a96d7000,0x00007f48a97d8000)]
  0x00007f49ec08f000 JavaThread "elasticsearch[node-100][bulk][T#21]" daemon [_thread_blocked, id=24617, stack(0x00007f48aa8e9000,0x00007f48aa9ea000)]
  0x00007f4974091000 JavaThread "elasticsearch[node-100][bulk][T#20]" daemon [_thread_blocked, id=24614, stack(0x00007f48aadee000,0x00007f48aaeef000)]
  0x00007f49e809a000 JavaThread "elasticsearch[node-100][bulk][T#19]" daemon [_thread_blocked, id=24612, stack(0x00007f48a95d6000,0x00007f48a96d7000)]
  0x00007f495c096800 JavaThread "elasticsearch[node-100][bulk][T#18]" daemon [_thread_blocked, id=24611, stack(0x00007f48abcfd000,0x00007f48abdfe000)]
  0x00007f49e009b000 JavaThread "elasticsearch[node-100][bulk][T#17]" daemon [_thread_blocked, id=24610, stack(0x00007f48abeff000,0x00007f48ac000000)]
  0x00007f4970095000 JavaThread "elasticsearch[node-100][bulk][T#16]" daemon [_thread_blocked, id=24609, stack(0x00007f49ac8c9000,0x00007f49ac9ca000)]
  0x00007f497408f800 JavaThread "elasticsearch[node-100][bulk][T#15]" daemon [_thread_blocked, id=24608, stack(0x00007f48aa2e3000,0x00007f48aa3e4000)]
  0x00007f495c095800 JavaThread "elasticsearch[node-100][bulk][T#14]" daemon [_thread_blocked, id=24607, stack(0x00007f48abafb000,0x00007f48abbfc000)]
  0x00007f49ec08d800 JavaThread "elasticsearch[node-100][bulk][T#13]" daemon [_thread_blocked, id=24606, stack(0x00007f48a9edf000,0x00007f48a9fe0000)]
  0x00007f497008e000 JavaThread "elasticsearch[node-100][bulk][T#12]" daemon [_thread_blocked, id=24605, stack(0x00007f49ac5c6000,0x00007f49ac6c7000)]
  0x00007f49e8089000 JavaThread "elasticsearch[node-100][bulk][T#11]" daemon [_thread_blocked, id=24604, stack(0x00007f49ac1c2000,0x00007f49ac2c3000)]
  0x00007f49e009a000 JavaThread "elasticsearch[node-100][bulk][T#10]" daemon [_thread_blocked, id=24603, stack(0x00007f48ab9fa000,0x00007f48abafb000)]
  0x00007f49ec08c000 JavaThread "elasticsearch[node-100][bulk][T#9]" daemon [_thread_blocked, id=24602, stack(0x00007f49ac2c3000,0x00007f49ac3c4000)]
  0x00007f4974003800 JavaThread "elasticsearch[node-100][bulk][T#8]" daemon [_thread_blocked, id=24600, stack(0x00007f48aa6e7000,0x00007f48aa7e8000)]
  0x00007f495c007000 JavaThread "elasticsearch[node-100][bulk][T#7]" daemon [_thread_blocked, id=24599, stack(0x00007f48ab3f4000,0x00007f48ab4f5000)]
  0x00007f497008d000 JavaThread "elasticsearch[node-100][bulk][T#6]" daemon [_thread_blocked, id=24598, stack(0x00007f48aa9ea000,0x00007f48aaaeb000)]
  0x00007f49e8088000 JavaThread "elasticsearch[node-100][bulk][T#5]" daemon [_thread_blocked, id=24597, stack(0x00007f48ab2f3000,0x00007f48ab3f4000)]
  0x00007f49e0097000 JavaThread "elasticsearch[node-100][bulk][T#4]" daemon [_thread_blocked, id=24596, stack(0x00007f48a9dde000,0x00007f48a9edf000)]
  0x00007f49ec08b000 JavaThread "elasticsearch[node-100][bulk][T#3]" daemon [_thread_blocked, id=24595, stack(0x00007f48a9adb000,0x00007f48a9bdc000)]
  0x00007f49c8083800 JavaThread "elasticsearch[node-100][index][T#1]" daemon [_thread_blocked, id=24588, stack(0x00007f4852e40000,0x00007f4852f41000)]
  0x00007f4964002000 JavaThread "elasticsearch[node-100][bulk][T#2]" daemon [_thread_blocked, id=24426, stack(0x00007f449de5b000,0x00007f449df5c000)]
  0x00007f496c003000 JavaThread "elasticsearch[node-100][bulk][T#1]" daemon [_thread_blocked, id=24425, stack(0x00007f449df5c000,0x00007f449e05d000)]
  0x00007f493408b000 JavaThread "elasticsearch[node-100][get][T#24]" daemon [_thread_blocked, id=24412, stack(0x00007f449e05d000,0x00007f449e15e000)]
  0x00007f49a4099800 JavaThread "elasticsearch[node-100][get][T#23]" daemon [_thread_blocked, id=24411, stack(0x00007f49ad5d6000,0x00007f49ad6d7000)]
  0x00007f49e008b000 JavaThread "elasticsearch[node-100][get][T#22]" daemon [_thread_blocked, id=24410, stack(0x00007f49ad7d8000,0x00007f49ad8d9000)]
  0x00007f49a008f800 JavaThread "elasticsearch[node-100][get][T#21]" daemon [_thread_blocked, id=24409, stack(0x00007f49ad6d7000,0x00007f49ad7d8000)]
  0x00007f49a4099000 JavaThread "elasticsearch[node-100][get][T#20]" daemon [_thread_blocked, id=24400, stack(0x00007f49adfe0000,0x00007f49ae0e1000)]
  0x00007f493408a000 JavaThread "elasticsearch[node-100][get][T#19]" daemon [_thread_blocked, id=24399, stack(0x00007f44ab765000,0x00007f44ab866000)]
  0x00007f498c093800 JavaThread "elasticsearch[node-100][get][T#18]" daemon [_thread_blocked, id=24394, stack(0x00007f449e15e000,0x00007f449e25f000)]
  0x00007f497c1e2800 JavaThread "elasticsearch[node-100][get][T#17]" daemon [_thread_blocked, id=24393, stack(0x00007f449e25f000,0x00007f449e360000)]
  0x00007f498c08b000 JavaThread "elasticsearch[node-100][get][T#16]" daemon [_thread_blocked, id=24392, stack(0x00007f44ab664000,0x00007f44ab765000)]
  0x00007f4934088000 JavaThread "elasticsearch[node-100][get][T#15]" daemon [_thread_blocked, id=24391, stack(0x00007f44a823f000,0x00007f44a8340000)]
  0x00007f49a008d800 JavaThread "elasticsearch[node-100][get][T#14]" daemon [_thread_blocked, id=24390, stack(0x00007f44ab15f000,0x00007f44ab260000)]
  0x00007f49e8084800 JavaThread "elasticsearch[node-100][get][T#13]" daemon [_thread_blocked, id=24389, stack(0x00007f44ab260000,0x00007f44ab361000)]
  0x00007f493808e000 JavaThread "elasticsearch[node-100][get][T#12]" daemon [_thread_blocked, id=24388, stack(0x00007f44ab361000,0x00007f44ab462000)]
  0x00007f498c08c000 JavaThread "elasticsearch[node-100][get][T#11]" daemon [_thread_blocked, id=24387, stack(0x00007f44ab462000,0x00007f44ab563000)]
  0x00007f49a808a000 JavaThread "elasticsearch[node-100][get][T#10]" daemon [_thread_blocked, id=24378, stack(0x00007f44ab563000,0x00007f44ab664000)]
  0x00007f49e8083000 JavaThread "elasticsearch[node-100][get][T#9]" daemon [_thread_blocked, id=24372, stack(0x00007f4750f9b000,0x00007f475109c000)]
  0x00007f498c089800 JavaThread "elasticsearch[node-100][get][T#8]" daemon [_thread_blocked, id=24370, stack(0x00007f44b5227000,0x00007f44b5328000)]
  0x00007f49a008b800 JavaThread "elasticsearch[node-100][get][T#7]" daemon [_thread_blocked, id=24369, stack(0x00007f44f14d1000,0x00007f44f15d2000)]
  0x00007f49a8087800 JavaThread "elasticsearch[node-100][get][T#6]" daemon [_thread_blocked, id=24368, stack(0x00007f44f99a7000,0x00007f44f9aa8000)]
  0x00007f49dc09e800 JavaThread "elasticsearch[node-100][get][T#5]" daemon [_thread_blocked, id=24367, stack(0x00007f49adcdd000,0x00007f49addde000)]
  0x00007f499808a800 JavaThread "elasticsearch[node-100][get][T#4]" daemon [_thread_blocked, id=24366, stack(0x00007f49adadb000,0x00007f49adbdc000)]
  0x00007f49e01de000 JavaThread "elasticsearch[node-100][get][T#3]" daemon [_thread_blocked, id=24365, stack(0x00007f49adedf000,0x00007f49adfe0000)]
  0x00007f49e4096800 JavaThread "elasticsearch[node-100][get][T#2]" daemon [_thread_blocked, id=24356, stack(0x00007f49addde000,0x00007f49adedf000)]
  0x00007f49d41f8000 JavaThread "elasticsearch[node-100][get][T#1]" daemon [_thread_blocked, id=24355, stack(0x00007f49adbdc000,0x00007f49adcdd000)]
  0x00007f49400ab800 JavaThread "elasticsearch[node-100][suggest][T#24]" daemon [_thread_blocked, id=24349, stack(0x00007f47329b5000,0x00007f4732ab6000)]
  0x00007f4948086800 JavaThread "elasticsearch[node-100][suggest][T#23]" daemon [_thread_blocked, id=24348, stack(0x00007f4732ab6000,0x00007f4732bb7000)]
  0x00007f4998086800 JavaThread "elasticsearch[node-100][suggest][T#22]" daemon [_thread_blocked, id=24347, stack(0x00007f4732bb7000,0x00007f4732cb8000)]
  0x00007f49a0089800 JavaThread "elasticsearch[node-100][suggest][T#21]" daemon [_thread_blocked, id=24346, stack(0x00007f4732cb8000,0x00007f4732db9000)]
  0x00007f49d41f4800 JavaThread "elasticsearch[node-100][suggest][T#20]" daemon [_thread_blocked, id=24345, stack(0x00007f474c3d6000,0x00007f474c4d7000)]
  0x00007f49e8188800 JavaThread "elasticsearch[node-100][suggest][T#19]" daemon [_thread_blocked, id=24344, stack(0x00007f474c4d7000,0x00007f474c5d8000)]
  0x00007f49380ee000 JavaThread "elasticsearch[node-100][suggest][T#18]" daemon [_thread_blocked, id=24343, stack(0x00007f474c5d8000,0x00007f474c6d9000)]
  0x00007f49340a4800 JavaThread "elasticsearch[node-100][suggest][T#17]" daemon [_thread_blocked, id=24342, stack(0x00007f474c6d9000,0x00007f474c7da000)]
  0x00007f498c18e800 JavaThread "elasticsearch[node-100][suggest][T#16]" daemon [_thread_blocked, id=24341, stack(0x00007f474c7da000,0x00007f474c8db000)]
  0x00007f4998172000 JavaThread "elasticsearch[node-100][suggest][T#15]" daemon [_thread_blocked, id=24340, stack(0x00007f474c8db000,0x00007f474c9dc000)]
  0x00007f49400a1000 JavaThread "elasticsearch[node-100][suggest][T#14]" daemon [_thread_blocked, id=24339, stack(0x00007f474c9dc000,0x00007f474cadd000)]
  0x00007f49480f3000 JavaThread "elasticsearch[node-100][suggest][T#13]" daemon [_thread_blocked, id=24338, stack(0x00007f474cadd000,0x00007f474cbde000)]
  0x00007f49c80a8000 JavaThread "elasticsearch[node-100][suggest][T#12]" daemon [_thread_blocked, id=24337, stack(0x00007f474cbde000,0x00007f474ccdf000)]
  0x00007f49e4094800 JavaThread "elasticsearch[node-100][suggest][T#11]" daemon [_thread_blocked, id=24336, stack(0x00007f49ad8d9000,0x00007f49ad9da000)]
  0x00007f49dc143800 JavaThread "elasticsearch[node-100][suggest][T#10]" daemon [_thread_blocked, id=24335, stack(0x00007f474ccdf000,0x00007f474cde0000)]
  0x00007f49a4091000 JavaThread "elasticsearch[node-100][suggest][T#9]" daemon [_thread_blocked, id=24332, stack(0x00007f474d20c000,0x00007f474d30d000)]
  0x00007f49a0197800 JavaThread "elasticsearch[node-100][suggest][T#8]" daemon [_thread_blocked, id=24331, stack(0x00007f474d30d000,0x00007f474d40e000)]
  0x00007f49480f1000 JavaThread "elasticsearch[node-100][suggest][T#7]" daemon [_thread_blocked, id=24315, stack(0x00007f474d53f000,0x00007f474d640000)]
  0x00007f49c8090000 JavaThread "elasticsearch[node-100][suggest][T#6]" daemon [_thread_blocked, id=24314, stack(0x00007f474d7d5000,0x00007f474d8d6000)]
  0x00007f49d41f2800 JavaThread "elasticsearch[node-100][suggest][T#5]" daemon [_thread_blocked, id=24313, stack(0x00007f474d8d6000,0x00007f474d9d7000)]
  0x00007f497c087000 JavaThread "elasticsearch[node-100][suggest][T#4]" daemon [_thread_blocked, id=24311, stack(0x00007f4750d2c000,0x00007f4750e2d000)]
  0x00007f49a4090000 JavaThread "elasticsearch[node-100][search][T#37]" daemon [_thread_blocked, id=24310, stack(0x00007f4750e2d000,0x00007f4750f2e000)]
  0x00007f493808c000 JavaThread "elasticsearch[node-100][search][T#36]" daemon [_thread_blocked, id=24309, stack(0x00007f475109c000,0x00007f475119d000)]
  0x00007f4934086000 JavaThread "elasticsearch[node-100][search][T#35]" daemon [_thread_blocked, id=24308, stack(0x00007f475119d000,0x00007f475129e000)]
  0x00007f494009f800 JavaThread "elasticsearch[node-100][search][T#34]" daemon [_thread_blocked, id=24307, stack(0x00007f475129e000,0x00007f475139f000)]
  0x00007f49a8085000 JavaThread "elasticsearch[node-100][search][T#33]" daemon [_thread_blocked, id=24306, stack(0x00007f475139f000,0x00007f47514a0000)]
  0x00007f49e4093000 JavaThread "elasticsearch[node-100][search][T#32]" daemon [_thread_blocked, id=24305, stack(0x00007f47514a0000,0x00007f47515a1000)]
  0x00007f49c808d800 JavaThread "elasticsearch[node-100][search][T#31]" daemon [_thread_blocked, id=24304, stack(0x00007f47515a1000,0x00007f47516a2000)]
  0x00007f4948084800 JavaThread "elasticsearch[node-100][search][T#30]" daemon [_thread_blocked, id=24303, stack(0x00007f47516a2000,0x00007f47517a3000)]
  0x00007f49d4093800 JavaThread "elasticsearch[node-100][search][T#29]" daemon [_thread_blocked, id=24302, stack(0x00007f47517a3000,0x00007f47518a4000)]
  0x00007f497c085000 JavaThread "elasticsearch[node-100][search][T#28]" daemon [_thread_blocked, id=24301, stack(0x00007f47518a4000,0x00007f47519a5000)]
  0x00007f49d0090000 JavaThread "elasticsearch[node-100][search][T#27]" daemon [_thread_blocked, id=24300, stack(0x00007f47519a5000,0x00007f4751aa6000)]
  0x00007f49e0083000 JavaThread "elasticsearch[node-100][search][T#26]" daemon [_thread_blocked, id=24299, stack(0x00007f4751aa6000,0x00007f4751ba7000)]
  0x00007f498c086000 JavaThread "elasticsearch[node-100][search][T#25]" daemon [_thread_blocked, id=24298, stack(0x00007f4751ba7000,0x00007f4751ca8000)]
  0x00007f49dc098800 JavaThread "elasticsearch[node-100][search][T#24]" daemon [_thread_blocked, id=24297, stack(0x00007f4751ca8000,0x00007f4751da9000)]
  0x00007f4998084800 JavaThread "elasticsearch[node-100][search][T#23]" daemon [_thread_blocked, id=24296, stack(0x00007f4751da9000,0x00007f4751eaa000)]
  0x00007f49d8096800 JavaThread "elasticsearch[node-100][search][T#22]" daemon [_thread_blocked, id=24295, stack(0x00007f4751eaa000,0x00007f4751fab000)]
  0x00007f49e808b000 JavaThread "elasticsearch[node-100][search][T#21]" daemon [_thread_blocked, id=24294, stack(0x00007f4751fab000,0x00007f47520ac000)]
  0x00007f49a0084000 JavaThread "elasticsearch[node-100][search][T#20]" daemon [_thread_blocked, id=24293, stack(0x00007f47520ac000,0x00007f47521ad000)]
  0x00007f49a4083000 JavaThread "elasticsearch[node-100][search][T#19]" daemon [_thread_blocked, id=24292, stack(0x00007f47521ad000,0x00007f47522ae000)]
  0x00007f493808a000 JavaThread "elasticsearch[node-100][suggest][T#3]" daemon [_thread_blocked, id=24291, stack(0x00007f4755476000,0x00007f4755577000)]
  0x00007f49a8083000 JavaThread "elasticsearch[node-100][search][T#18]" daemon [_thread_blocked, id=24290, stack(0x00007f4755629000,0x00007f475572a000)]
  0x00007f497c083000 JavaThread "elasticsearch[node-100][search][T#17]" daemon [_thread_blocked, id=24289, stack(0x00007f4758911000,0x00007f4758a12000)]
  0x00007f4934085000 JavaThread "elasticsearch[node-100][suggest][T#2]" daemon [_thread_blocked, id=24288, stack(0x00007f475d3f9000,0x00007f475d4fa000)]
  0x00007f498c083000 JavaThread "elasticsearch[node-100][search][T#16]" daemon [_thread_blocked, id=24287, stack(0x00007f475d7e3000,0x00007f475d8e4000)]
  0x00007f49e4091800 JavaThread "elasticsearch[node-100][search][T#15]" daemon [_thread_blocked, id=24286, stack(0x00007f475daac000,0x00007f475dbad000)]
  0x00007f4998082800 JavaThread "elasticsearch[node-100][search][T#14]" daemon [_thread_blocked, id=24285, stack(0x00007f476289f000,0x00007f47629a0000)]
  0x00007f49d4091800 JavaThread "elasticsearch[node-100][search][T#13]" daemon [_thread_blocked, id=24284, stack(0x00007f47668d3000,0x00007f47669d4000)]
  0x00007f49400e7000 JavaThread "elasticsearch[node-100][search][T#12]" daemon [_thread_blocked, id=24283, stack(0x00007f4772c0d000,0x00007f4772d0e000)]
  0x00007f4948082000 JavaThread "elasticsearch[node-100][search][T#11]" daemon [_thread_blocked, id=24274, stack(0x00007f4772d0e000,0x00007f4772e0f000)]
  0x00007f49a0082800 JavaThread "elasticsearch[node-100][search][T#10]" daemon [_thread_blocked, id=24254, stack(0x00007f4772e0f000,0x00007f4772f10000)]
  0x00007f49a4081000 JavaThread "elasticsearch[node-100][search][T#9]" daemon [_thread_blocked, id=24238, stack(0x00007f4772f10000,0x00007f4773011000)]
  0x00007f49e0081000 JavaThread "elasticsearch[node-100][search][T#8]" daemon [_thread_blocked, id=24237, stack(0x00007f4776fa3000,0x00007f47770a4000)]
  0x00007f49a8081000 JavaThread "elasticsearch[node-100][search][T#7]" daemon [_thread_blocked, id=24236, stack(0x00007f47772f8000,0x00007f47773f9000)]
  0x00007f49e8081000 JavaThread "elasticsearch[node-100][search][T#6]" daemon [_thread_blocked, id=24235, stack(0x00007f47773f9000,0x00007f47774fa000)]
  0x00007f497c081000 JavaThread "elasticsearch[node-100][search][T#5]" daemon [_thread_blocked, id=24234, stack(0x00007f477770b000,0x00007f477780c000)]
  0x00007f4938088800 JavaThread "elasticsearch[node-100][search][T#4]" daemon [_thread_blocked, id=24233, stack(0x00007f478060f000,0x00007f4780710000)]
  0x00007f498c081000 JavaThread "elasticsearch[node-100][suggest][T#1]" daemon [_thread_blocked, id=24232, stack(0x00007f4780a8b000,0x00007f4780b8c000)]
  0x00007f4998081000 JavaThread "elasticsearch[node-100][search][T#3]" daemon [_thread_blocked, id=24231, stack(0x00007f4790ccc000,0x00007f4790dcd000)]
  0x00007f49a0081000 JavaThread "elasticsearch[node-100][search][T#2]" daemon [_thread_blocked, id=24230, stack(0x00007f4790dcd000,0x00007f4790ece000)]
  0x00007f49d008e800 JavaThread "elasticsearch[node-100][search][T#1]" daemon [_thread_blocked, id=24229, stack(0x00007f47acea5000,0x00007f47acfa6000)]
  0x00007f4b7c00a000 JavaThread "DestroyJavaVM" [_thread_blocked, id=23727, stack(0x00007f4b80595000,0x00007f4b80696000)]
  0x00007f4b7e78d800 JavaThread "elasticsearch[keepAlive/2.3.4]" [_thread_blocked, id=24218, stack(0x00007f4852ff0000,0x00007f48530f1000)]
  0x00007f4b7e78a000 JavaThread "elasticsearch[node-100][http_server_boss][T#1]{New I/O server boss #147}" daemon [_thread_in_native, id=24217, stack(0x00007f48530f1000,0x00007f48531f2000)]
  0x00007f4b7e75f000 JavaThread "elasticsearch[node-100][http_server_worker][T#48]{New I/O worker #146}" daemon [_thread_in_native, id=24216, stack(0x00007f48531f2000,0x00007f48532f3000)]
  0x00007f4b7e734000 JavaThread "elasticsearch[node-100][http_server_worker][T#47]{New I/O worker #145}" daemon [_thread_in_native, id=24215, stack(0x00007f48532f3000,0x00007f48533f4000)]
  0x00007f4b7e709000 JavaThread "elasticsearch[node-100][http_server_worker][T#46]{New I/O worker #144}" daemon [_thread_in_native, id=24214, stack(0x00007f48533f4000,0x00007f48534f5000)]
  0x00007f4b7c637800 JavaThread "elasticsearch[node-100][http_server_worker][T#45]{New I/O worker #143}" daemon [_thread_in_native, id=24213, stack(0x00007f48534f5000,0x00007f48535f6000)]
  0x00007f4b7c635000 JavaThread "elasticsearch[node-100][http_server_worker][T#44]{New I/O worker #142}" daemon [_thread_in_native, id=24212, stack(0x00007f48535f6000,0x00007f48536f7000)]
  0x00007f4b7c633000 JavaThread "elasticsearch[node-100][http_server_worker][T#43]{New I/O worker #141}" daemon [_thread_in_native, id=24211, stack(0x00007f48536f7000,0x00007f48537f8000)]
  0x00007f4b7c631000 JavaThread "elasticsearch[node-100][http_server_worker][T#42]{New I/O worker #140}" daemon [_thread_in_native, id=24210, stack(0x00007f48537f8000,0x00007f48538f9000)]
  0x00007f4b7c62f000 JavaThread "elasticsearch[node-100][http_server_worker][T#41]{New I/O worker #139}" daemon [_thread_in_native, id=24209, stack(0x00007f48538f9000,0x00007f48539fa000)]
  0x00007f4b7c62d000 JavaThread "elasticsearch[node-100][http_server_worker][T#40]{New I/O worker #138}" daemon [_thread_in_native, id=24208, stack(0x00007f48539fa000,0x00007f4853afb000)]
  0x00007f4b7c62b000 JavaThread "elasticsearch[node-100][http_server_worker][T#39]{New I/O worker #137}" daemon [_thread_in_native, id=24207, stack(0x00007f4853afb000,0x00007f4853bfc000)]
  0x00007f4b7c629000 JavaThread "elasticsearch[node-100][http_server_worker][T#38]{New I/O worker #136}" daemon [_thread_in_native, id=24206, stack(0x00007f4853bfc000,0x00007f4853cfd000)]
  0x00007f4b7c61d800 JavaThread "elasticsearch[node-100][http_server_worker][T#37]{New I/O worker #135}" daemon [_thread_in_native, id=24205, stack(0x00007f4853cfd000,0x00007f4853dfe000)]
  0x00007f4b7c61b800 JavaThread "elasticsearch[node-100][http_server_worker][T#36]{New I/O worker #134}" daemon [_thread_in_native, id=24204, stack(0x00007f4853dfe000,0x00007f4853eff000)]
  0x00007f4b7c619800 JavaThread "elasticsearch[node-100][http_server_worker][T#35]{New I/O worker #133}" daemon [_thread_in_native, id=24203, stack(0x00007f4853eff000,0x00007f4854000000)]
  0x00007f4b7c617800 JavaThread "elasticsearch[node-100][http_server_worker][T#34]{New I/O worker #132}" daemon [_thread_in_native, id=24202, stack(0x00007f48a80c1000,0x00007f48a81c2000)]
  0x00007f4b7c615800 JavaThread "elasticsearch[node-100][http_server_worker][T#33]{New I/O worker #131}" daemon [_thread_in_native, id=24201, stack(0x00007f48a81c2000,0x00007f48a82c3000)]
  0x00007f4b7c613800 JavaThread "elasticsearch[node-100][http_server_worker][T#32]{New I/O worker #130}" daemon [_thread_in_native, id=24200, stack(0x00007f48a82c3000,0x00007f48a83c4000)]
  0x00007f4b7c611800 JavaThread "elasticsearch[node-100][http_server_worker][T#31]{New I/O worker #129}" daemon [_thread_in_native, id=24198, stack(0x00007f48a83c4000,0x00007f48a84c5000)]
  0x00007f4b7c59b000 JavaThread "elasticsearch[node-100][http_server_worker][T#30]{New I/O worker #128}" daemon [_thread_in_native, id=24195, stack(0x00007f48a84c5000,0x00007f48a85c6000)]
  0x00007f4b7c598800 JavaThread "elasticsearch[node-100][http_server_worker][T#29]{New I/O worker #127}" daemon [_thread_in_native, id=24194, stack(0x00007f48a85c6000,0x00007f48a86c7000)]
  0x00007f4b7c597000 JavaThread "elasticsearch[node-100][http_server_worker][T#28]{New I/O worker #126}" daemon [_thread_in_native, id=24192, stack(0x00007f48a86c7000,0x00007f48a87c8000)]
  0x00007f4b7c595000 JavaThread "elasticsearch[node-100][http_server_worker][T#27]{New I/O worker #125}" daemon [_thread_in_native, id=24190, stack(0x00007f48a87c8000,0x00007f48a88c9000)]
  0x00007f4b7c593000 JavaThread "elasticsearch[node-100][http_server_worker][T#26]{New I/O worker #124}" daemon [_thread_in_native, id=24189, stack(0x00007f48a88c9000,0x00007f48a89ca000)]
  0x00007f4b7c591000 JavaThread "elasticsearch[node-100][http_server_worker][T#25]{New I/O worker #123}" daemon [_thread_in_native, id=24187, stack(0x00007f48a89ca000,0x00007f48a8acb000)]
  0x00007f4b7c58f000 JavaThread "elasticsearch[node-100][http_server_worker][T#24]{New I/O worker #122}" daemon [_thread_in_native, id=24185, stack(0x00007f48a8acb000,0x00007f48a8bcc000)]
  0x00007f4b7c58d000 JavaThread "elasticsearch[node-100][http_server_worker][T#23]{New I/O worker #121}" daemon [_thread_in_native, id=24181, stack(0x00007f48a8bcc000,0x00007f48a8ccd000)]
  0x00007f4b7c58b800 JavaThread "elasticsearch[node-100][http_server_worker][T#22]{New I/O worker #120}" daemon [_thread_in_native, id=24180, stack(0x00007f48a8ccd000,0x00007f48a8dce000)]
  0x00007f4b7c5a5000 JavaThread "elasticsearch[node-100][http_server_worker][T#21]{New I/O worker #119}" daemon [_thread_in_native, id=24179, stack(0x00007f48a8dce000,0x00007f48a8ecf000)]
  0x00007f4b7c5a2800 JavaThread "elasticsearch[node-100][http_server_worker][T#20]{New I/O worker #118}" daemon [_thread_in_native, id=24176, stack(0x00007f48a8ecf000,0x00007f48a8fd0000)]
  0x00007f4b7c5a0800 JavaThread "elasticsearch[node-100][http_server_worker][T#19]{New I/O worker #117}" daemon [_thread_in_native, id=24175, stack(0x00007f48a8fd0000,0x00007f48a90d1000)]
  0x00007f4b7c4d5800 JavaThread "elasticsearch[node-100][http_server_worker][T#18]{New I/O worker #116}" daemon [_thread_in_native, id=24172, stack(0x00007f48a90d1000,0x00007f48a91d2000)]
  0x00007f4b7c4d3800 JavaThread "elasticsearch[node-100][http_server_worker][T#17]{New I/O worker #115}" daemon [_thread_in_native, id=24168, stack(0x00007f48a94d5000,0x00007f48a95d6000)]
  0x00007f4b7c4d1800 JavaThread "elasticsearch[node-100][http_server_worker][T#16]{New I/O worker #114}" daemon [_thread_in_native, id=24161, stack(0x00007f48a9bdc000,0x00007f48a9cdd000)]
  0x00007f4b7c4cf000 JavaThread "elasticsearch[node-100][http_server_worker][T#15]{New I/O worker #113}" daemon [_thread_in_native, id=24152, stack(0x00007f48aa4e5000,0x00007f48aa5e6000)]
  0x00007f4b7c5d7000 JavaThread "elasticsearch[node-100][http_server_worker][T#14]{New I/O worker #112}" daemon [_thread_in_native, id=24144, stack(0x00007f48aaced000,0x00007f48aadee000)]
  0x00007f4b7c5d5000 JavaThread "elasticsearch[node-100][http_server_worker][T#13]{New I/O worker #111}" daemon [_thread_in_native, id=24135, stack(0x00007f48ab5f6000,0x00007f48ab6f7000)]
  0x00007f49dc08b000 JavaThread "elasticsearch[node-100][fetch_shard_store][T#16]" daemon [_thread_blocked, id=24133, stack(0x00007f48ab7f8000,0x00007f48ab8f9000)]
  0x00007f4b7c5d2800 JavaThread "elasticsearch[node-100][http_server_worker][T#12]{New I/O worker #110}" daemon [_thread_in_native, id=24129, stack(0x00007f48abbfc000,0x00007f48abcfd000)]
  0x00007f4b7c5d0800 JavaThread "elasticsearch[node-100][http_server_worker][T#11]{New I/O worker #109}" daemon [_thread_in_native, id=24121, stack(0x00007f49ac4c5000,0x00007f49ac5c6000)]
  0x00007f4b7c4a6800 JavaThread "elasticsearch[node-100][http_server_worker][T#10]{New I/O worker #108}" daemon [_thread_in_native, id=24116, stack(0x00007f49ac9ca000,0x00007f49acacb000)]
  0x00007f4b7c4a5000 JavaThread "elasticsearch[node-100][http_server_worker][T#9]{New I/O worker #107}" daemon [_thread_in_native, id=24115, stack(0x00007f49acacb000,0x00007f49acbcc000)]
  0x00007f4b7c4a3000 JavaThread "elasticsearch[node-100][http_server_worker][T#8]{New I/O worker #106}" daemon [_thread_in_native, id=24114, stack(0x00007f49acbcc000,0x00007f49acccd000)]
  0x00007f4b7c4a1000 JavaThread "elasticsearch[node-100][http_server_worker][T#7]{New I/O worker #105}" daemon [_thread_in_native, id=24113, stack(0x00007f49acccd000,0x00007f49acdce000)]
  0x00007f49dc081000 JavaThread "elasticsearch[node-100][management][T#1]" daemon [_thread_blocked, id=24112, stack(0x00007f49acdce000,0x00007f49acecf000)]
  0x00007f4b7c5c2800 JavaThread "elasticsearch[node-100][http_server_worker][T#6]{New I/O worker #104}" daemon [_thread_in_native, id=24111, stack(0x00007f49acecf000,0x00007f49acfd0000)]
  0x00007f4b7c5c0800 JavaThread "elasticsearch[node-100][http_server_worker][T#5]{New I/O worker #103}" daemon [_thread_in_native, id=24110, stack(0x00007f49acfd0000,0x00007f49ad0d1000)]
  0x00007f4b7c5be800 JavaThread "elasticsearch[node-100][http_server_worker][T#4]{New I/O worker #102}" daemon [_thread_in_native, id=24107, stack(0x00007f49ad0d1000,0x00007f49ad1d2000)]
  0x00007f4b7c5bc800 JavaThread "elasticsearch[node-100][http_server_worker][T#3]{New I/O worker #101}" daemon [_thread_in_native, id=24106, stack(0x00007f49ad1d2000,0x00007f49ad2d3000)]
  0x00007f4b7c5bb800 JavaThread "elasticsearch[node-100][http_server_worker][T#2]{New I/O worker #100}" daemon [_thread_in_native, id=24104, stack(0x00007f49ad2d3000,0x00007f49ad3d4000)]
  0x00007f4b7dfc4800 JavaThread "elasticsearch[node-100][http_server_worker][T#1]{New I/O worker #99}" daemon [_thread_in_native, id=24102, stack(0x00007f49ad3d4000,0x00007f49ad4d5000)]
  0x00007f49fc001000 JavaThread "elasticsearch[node-100][transport_client_timer][T#1]{Hashed wheel timer #1}" daemon [_thread_blocked, id=23904, stack(0x00007f49ad9da000,0x00007f49adadb000)]
  0x00007f4b7dfa6800 JavaThread "elasticsearch[node-100][clusterService#updateTask][T#1]" daemon [_thread_blocked, id=23897, stack(0x00007f49ae0e1000,0x00007f49ae1e2000)]
  0x00007f4b7dfa1800 JavaThread "elasticsearch[node-100][[http_server_boss.default]][T#1]{New I/O server boss #98}" daemon [_thread_in_native, id=23896, stack(0x00007f49ae1e2000,0x00007f49ae2e3000)]
  0x00007f4b7df8f800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#48]{New I/O worker #97}" daemon [_thread_blocked, id=23895, stack(0x00007f49ae2e3000,0x00007f49ae3e4000)]
  0x00007f4b7df65000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#47]{New I/O worker #96}" daemon [_thread_blocked, id=23894, stack(0x00007f49ae3e4000,0x00007f49ae4e5000)]
  0x00007f4b7df3a000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#46]{New I/O worker #95}" daemon [_thread_blocked, id=23893, stack(0x00007f49ae4e5000,0x00007f49ae5e6000)]
  0x00007f4b7df0f000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#45]{New I/O worker #94}" daemon [_thread_in_native, id=23892, stack(0x00007f49ae5e6000,0x00007f49ae6e7000)]
  0x00007f4b7dee4000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#44]{New I/O worker #93}" daemon [_thread_blocked, id=23891, stack(0x00007f49ae6e7000,0x00007f49ae7e8000)]
  0x00007f4b7deb9000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#43]{New I/O worker #92}" daemon [_thread_in_native, id=23890, stack(0x00007f49ae7e8000,0x00007f49ae8e9000)]
  0x00007f4b7de8e000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#42]{New I/O worker #91}" daemon [_thread_in_native, id=23889, stack(0x00007f49ae8e9000,0x00007f49ae9ea000)]
  0x00007f4b7de63000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#41]{New I/O worker #90}" daemon [_thread_in_native, id=23888, stack(0x00007f49ae9ea000,0x00007f49aeaeb000)]
  0x00007f4b7de38000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#40]{New I/O worker #89}" daemon [_thread_in_native, id=23887, stack(0x00007f49aeaeb000,0x00007f49aebec000)]
  0x00007f4b7de0d000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#39]{New I/O worker #88}" daemon [_thread_in_native, id=23886, stack(0x00007f49aebec000,0x00007f49aeced000)]
  0x00007f4b7dde2800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#38]{New I/O worker #87}" daemon [_thread_in_native, id=23885, stack(0x00007f49aeced000,0x00007f49aedee000)]
  0x00007f4b7ddb8000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#37]{New I/O worker #86}" daemon [_thread_in_native, id=23884, stack(0x00007f49aedee000,0x00007f49aeeef000)]
  0x00007f4b7dd8d000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#36]{New I/O worker #85}" daemon [_thread_in_native, id=23883, stack(0x00007f49aeeef000,0x00007f49aeff0000)]
  0x00007f4b7dd62000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#35]{New I/O worker #84}" daemon [_thread_in_native, id=23882, stack(0x00007f49aeff0000,0x00007f49af0f1000)]
  0x00007f4b7dd37000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#34]{New I/O worker #83}" daemon [_thread_in_native, id=23881, stack(0x00007f49af0f1000,0x00007f49af1f2000)]
  0x00007f4b7dd0b800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#33]{New I/O worker #82}" daemon [_thread_in_native, id=23880, stack(0x00007f49af1f2000,0x00007f49af2f3000)]
  0x00007f4b7dce0800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#32]{New I/O worker #81}" daemon [_thread_in_native, id=23879, stack(0x00007f49af2f3000,0x00007f49af3f4000)]
  0x00007f4b7dcb5800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#31]{New I/O worker #80}" daemon [_thread_in_native, id=23878, stack(0x00007f49af3f4000,0x00007f49af4f5000)]
  0x00007f4b7dc8a800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#30]{New I/O worker #79}" daemon [_thread_in_native, id=23877, stack(0x00007f49af4f5000,0x00007f49af5f6000)]
  0x00007f4b7dc5f800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#29]{New I/O worker #78}" daemon [_thread_blocked, id=23876, stack(0x00007f49af5f6000,0x00007f49af6f7000)]
  0x00007f4b7dc34800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#28]{New I/O worker #77}" daemon [_thread_in_native, id=23875, stack(0x00007f49af6f7000,0x00007f49af7f8000)]
  0x00007f4b7dc0a000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#27]{New I/O worker #76}" daemon [_thread_in_native, id=23874, stack(0x00007f49af7f8000,0x00007f49af8f9000)]
  0x00007f4b7dbdf000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#26]{New I/O worker #75}" daemon [_thread_in_native, id=23873, stack(0x00007f49af8f9000,0x00007f49af9fa000)]
  0x00007f4b7dbb4000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#25]{New I/O worker #74}" daemon [_thread_blocked, id=23872, stack(0x00007f49af9fa000,0x00007f49afafb000)]
  0x00007f4b7db89000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#24]{New I/O worker #73}" daemon [_thread_blocked, id=23871, stack(0x00007f49afafb000,0x00007f49afbfc000)]
  0x00007f4b7db5e000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#23]{New I/O worker #72}" daemon [_thread_in_native, id=23870, stack(0x00007f49afbfc000,0x00007f49afcfd000)]
  0x00007f4b7db33000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#22]{New I/O worker #71}" daemon [_thread_blocked, id=23869, stack(0x00007f49afcfd000,0x00007f49afdfe000)]
  0x00007f4b7db07800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#21]{New I/O worker #70}" daemon [_thread_in_native, id=23868, stack(0x00007f49afdfe000,0x00007f49afeff000)]
  0x00007f4b7dadc800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#20]{New I/O worker #69}" daemon [_thread_blocked, id=23867, stack(0x00007f49afeff000,0x00007f49b0000000)]
  0x00007f4b7dab1800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#19]{New I/O worker #68}" daemon [_thread_blocked, id=23866, stack(0x00007f4ae404e000,0x00007f4ae414f000)]
  0x00007f4b7da86800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#18]{New I/O worker #67}" daemon [_thread_in_native, id=23865, stack(0x00007f4ae414f000,0x00007f4ae4250000)]
  0x00007f4b7da5b800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#17]{New I/O worker #66}" daemon [_thread_in_native, id=23864, stack(0x00007f4ae4250000,0x00007f4ae4351000)]
  0x00007f4b7da30800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#16]{New I/O worker #65}" daemon [_thread_in_native, id=23863, stack(0x00007f4ae4351000,0x00007f4ae4452000)]
  0x00007f4b7da05000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#15]{New I/O worker #64}" daemon [_thread_in_native, id=23862, stack(0x00007f4ae4452000,0x00007f4ae4553000)]
  0x00007f4b7d9da000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#14]{New I/O worker #63}" daemon [_thread_in_native, id=23861, stack(0x00007f4ae4553000,0x00007f4ae4654000)]
  0x00007f4b7d9af000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#13]{New I/O worker #62}" daemon [_thread_blocked, id=23860, stack(0x00007f4ae4654000,0x00007f4ae4755000)]
  0x00007f4b7d984000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#12]{New I/O worker #61}" daemon [_thread_in_native, id=23859, stack(0x00007f4ae4755000,0x00007f4ae4856000)]
  0x00007f4b7d959800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#11]{New I/O worker #60}" daemon [_thread_blocked, id=23858, stack(0x00007f4ae4856000,0x00007f4ae4957000)]
  0x00007f4b7d92e800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#10]{New I/O worker #59}" daemon [_thread_blocked, id=23857, stack(0x00007f4ae4957000,0x00007f4ae4a58000)]
  0x00007f4b7d903800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#9]{New I/O worker #58}" daemon [_thread_blocked, id=23856, stack(0x00007f4ae4a58000,0x00007f4ae4b59000)]
  0x00007f4b7d8d9000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#8]{New I/O worker #57}" daemon [_thread_blocked, id=23855, stack(0x00007f4ae4b59000,0x00007f4ae4c5a000)]
  0x00007f4b7d8ae800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#7]{New I/O worker #56}" daemon [_thread_blocked, id=23854, stack(0x00007f4ae4c5a000,0x00007f4ae4d5b000)]
  0x00007f4b7d883000 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#6]{New I/O worker #55}" daemon [_thread_blocked, id=23853, stack(0x00007f4ae4d5b000,0x00007f4ae4e5c000)]
  0x00007f4b7d858800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#5]{New I/O worker #54}" daemon [_thread_blocked, id=23852, stack(0x00007f4ae4e5c000,0x00007f4ae4f5d000)]
  0x00007f4b7d82d800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#4]{New I/O worker #53}" daemon [_thread_in_native, id=23851, stack(0x00007f4ae4f5d000,0x00007f4ae505e000)]
  0x00007f4b7d802800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#3]{New I/O worker #52}" daemon [_thread_blocked, id=23850, stack(0x00007f4ae505e000,0x00007f4ae515f000)]
  0x00007f4b7d7d8800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#2]{New I/O worker #51}" daemon [_thread_in_native, id=23849, stack(0x00007f4ae515f000,0x00007f4ae5260000)]
  0x00007f4b7d793800 JavaThread "elasticsearch[node-100][[http_server_worker.default]][T#1]{New I/O worker #50}" daemon [_thread_in_native, id=23848, stack(0x00007f4ae5260000,0x00007f4ae5361000)]
  0x00007f4b7d774800 JavaThread "elasticsearch[node-100][transport_client_boss][T#1]{New I/O boss #49}" daemon [_thread_in_native, id=23847, stack(0x00007f4ae5361000,0x00007f4ae5462000)]
  0x00007f4b7d75e000 JavaThread "elasticsearch[node-100][transport_client_worker][T#48]{New I/O worker #48}" daemon [_thread_in_native, id=23846, stack(0x00007f4ae5462000,0x00007f4ae5563000)]
  0x00007f4b7d733000 JavaThread "elasticsearch[node-100][transport_client_worker][T#47]{New I/O worker #47}" daemon [_thread_in_native, id=23845, stack(0x00007f4ae5563000,0x00007f4ae5664000)]
  0x00007f4b7d708000 JavaThread "elasticsearch[node-100][transport_client_worker][T#46]{New I/O worker #46}" daemon [_thread_in_native, id=23844, stack(0x00007f4ae5664000,0x00007f4ae5765000)]
  0x00007f4b7d6dd000 JavaThread "elasticsearch[node-100][transport_client_worker][T#45]{New I/O worker #45}" daemon [_thread_in_native, id=23843, stack(0x00007f4ae5765000,0x00007f4ae5866000)]
  0x00007f4b7d6b2000 JavaThread "elasticsearch[node-100][transport_client_worker][T#44]{New I/O worker #44}" daemon [_thread_in_native, id=23842, stack(0x00007f4ae5866000,0x00007f4ae5967000)]
  0x00007f4b7d686800 JavaThread "elasticsearch[node-100][transport_client_worker][T#43]{New I/O worker #43}" daemon [_thread_in_native, id=23841, stack(0x00007f4ae5967000,0x00007f4ae5a68000)]
  0x00007f4b7d65b800 JavaThread "elasticsearch[node-100][transport_client_worker][T#42]{New I/O worker #42}" daemon [_thread_in_native, id=23840, stack(0x00007f4ae5a68000,0x00007f4ae5b69000)]
  0x00007f4b7d630800 JavaThread "elasticsearch[node-100][transport_client_worker][T#41]{New I/O worker #41}" daemon [_thread_in_native, id=23839, stack(0x00007f4ae5b69000,0x00007f4ae5c6a000)]
  0x00007f4b7d605800 JavaThread "elasticsearch[node-100][transport_client_worker][T#40]{New I/O worker #40}" daemon [_thread_in_native, id=23838, stack(0x00007f4ae5c6a000,0x00007f4ae5d6b000)]
  0x00007f4b7d5da800 JavaThread "elasticsearch[node-100][transport_client_worker][T#39]{New I/O worker #39}" daemon [_thread_in_native, id=23837, stack(0x00007f4ae5d6b000,0x00007f4ae5e6c000)]
  0x00007f4b7d5af800 JavaThread "elasticsearch[node-100][transport_client_worker][T#38]{New I/O worker #38}" daemon [_thread_in_native, id=23836, stack(0x00007f4ae5e6c000,0x00007f4ae5f6d000)]
  0x00007f4b7d584000 JavaThread "elasticsearch[node-100][transport_client_worker][T#37]{New I/O worker #37}" daemon [_thread_in_native, id=23835, stack(0x00007f4ae5f6d000,0x00007f4ae606e000)]
  0x00007f4b7d559000 JavaThread "elasticsearch[node-100][transport_client_worker][T#36]{New I/O worker #36}" daemon [_thread_in_native, id=23834, stack(0x00007f4ae606e000,0x00007f4ae616f000)]
  0x00007f4b7d52e000 JavaThread "elasticsearch[node-100][transport_client_worker][T#35]{New I/O worker #35}" daemon [_thread_in_native, id=23833, stack(0x00007f4af80ff000,0x00007f4af8200000)]
  0x00007f4b7d503000 JavaThread "elasticsearch[node-100][transport_client_worker][T#34]{New I/O worker #34}" daemon [_thread_in_native, id=23832, stack(0x00007f4af8200000,0x00007f4af8301000)]
  0x00007f4b7d4d8000 JavaThread "elasticsearch[node-100][transport_client_worker][T#33]{New I/O worker #33}" daemon [_thread_in_native, id=23831, stack(0x00007f4af8301000,0x00007f4af8402000)]
  0x00007f4b7d4ad000 JavaThread "elasticsearch[node-100][transport_client_worker][T#32]{New I/O worker #32}" daemon [_thread_in_native, id=23830, stack(0x00007f4af8402000,0x00007f4af8503000)]
  0x00007f4b7d481800 JavaThread "elasticsearch[node-100][transport_client_worker][T#31]{New I/O worker #31}" daemon [_thread_in_native, id=23829, stack(0x00007f4af8503000,0x00007f4af8604000)]
  0x00007f4b7d456800 JavaThread "elasticsearch[node-100][transport_client_worker][T#30]{New I/O worker #30}" daemon [_thread_in_native, id=23828, stack(0x00007f4af8604000,0x00007f4af8705000)]
  0x00007f4b7d42b800 JavaThread "elasticsearch[node-100][transport_client_worker][T#29]{New I/O worker #29}" daemon [_thread_in_native, id=23827, stack(0x00007f4af8705000,0x00007f4af8806000)]
  0x00007f4b7d400800 JavaThread "elasticsearch[node-100][transport_client_worker][T#28]{New I/O worker #28}" daemon [_thread_in_native, id=23826, stack(0x00007f4af8806000,0x00007f4af8907000)]
  0x00007f4b7d3d5800 JavaThread "elasticsearch[node-100][transport_client_worker][T#27]{New I/O worker #27}" daemon [_thread_in_native, id=23825, stack(0x00007f4af8907000,0x00007f4af8a08000)]
  0x00007f4b7d3aa800 JavaThread "elasticsearch[node-100][transport_client_worker][T#26]{New I/O worker #26}" daemon [_thread_in_native, id=23824, stack(0x00007f4af8a08000,0x00007f4af8b09000)]
  0x00007f4b7d37f000 JavaThread "elasticsearch[node-100][transport_client_worker][T#25]{New I/O worker #25}" daemon [_thread_in_native, id=23823, stack(0x00007f4af8b09000,0x00007f4af8c0a000)]
  0x00007f4b7d354000 JavaThread "elasticsearch[node-100][transport_client_worker][T#24]{New I/O worker #24}" daemon [_thread_in_native, id=23822, stack(0x00007f4af8c0a000,0x00007f4af8d0b000)]
  0x00007f4b7d329000 JavaThread "elasticsearch[node-100][transport_client_worker][T#23]{New I/O worker #23}" daemon [_thread_in_native, id=23821, stack(0x00007f4af8d0b000,0x00007f4af8e0c000)]
  0x00007f4b7d2fe000 JavaThread "elasticsearch[node-100][transport_client_worker][T#22]{New I/O worker #22}" daemon [_thread_in_native, id=23820, stack(0x00007f4af8e0c000,0x00007f4af8f0d000)]
  0x00007f4b7d2d3000 JavaThread "elasticsearch[node-100][transport_client_worker][T#21]{New I/O worker #21}" daemon [_thread_in_native, id=23819, stack(0x00007f4af8f0d000,0x00007f4af900e000)]
  0x00007f4b7d2a8000 JavaThread "elasticsearch[node-100][transport_client_worker][T#20]{New I/O worker #20}" daemon [_thread_in_native, id=23818, stack(0x00007f4af900e000,0x00007f4af910f000)]
  0x00007f4b7d27c800 JavaThread "elasticsearch[node-100][transport_client_worker][T#19]{New I/O worker #19}" daemon [_thread_in_native, id=23817, stack(0x00007f4b000e3000,0x00007f4b001e4000)]
  0x00007f4b7d251800 JavaThread "elasticsearch[node-100][transport_client_worker][T#18]{New I/O worker #18}" daemon [_thread_in_native, id=23816, stack(0x00007f4b001e4000,0x00007f4b002e5000)]
  0x00007f4b7d226800 JavaThread "elasticsearch[node-100][transport_client_worker][T#17]{New I/O worker #17}" daemon [_thread_in_native, id=23815, stack(0x00007f4b002e5000,0x00007f4b003e6000)]
  0x00007f4b7d1fb800 JavaThread "elasticsearch[node-100][transport_client_worker][T#16]{New I/O worker #16}" daemon [_thread_in_native, id=23814, stack(0x00007f4b003e6000,0x00007f4b004e7000)]
  0x00007f4b7d1d0800 JavaThread "elasticsearch[node-100][transport_client_worker][T#15]{New I/O worker #15}" daemon [_thread_in_native, id=23813, stack(0x00007f4b004e7000,0x00007f4b005e8000)]
  0x00007f4b7d1a5800 JavaThread "elasticsearch[node-100][transport_client_worker][T#14]{New I/O worker #14}" daemon [_thread_in_native, id=23812, stack(0x00007f4b005e8000,0x00007f4b006e9000)]
  0x00007f4b7d17a800 JavaThread "elasticsearch[node-100][transport_client_worker][T#13]{New I/O worker #13}" daemon [_thread_in_native, id=23811, stack(0x00007f4b006e9000,0x00007f4b007ea000)]
  0x00007f4b7d14f000 JavaThread "elasticsearch[node-100][transport_client_worker][T#12]{New I/O worker #12}" daemon [_thread_in_native, id=23810, stack(0x00007f4b007ea000,0x00007f4b008eb000)]
  0x00007f4b7d124000 JavaThread "elasticsearch[node-100][transport_client_worker][T#11]{New I/O worker #11}" daemon [_thread_in_native, id=23809, stack(0x00007f4b008eb000,0x00007f4b009ec000)]
  0x00007f4b7d0f9000 JavaThread "elasticsearch[node-100][transport_client_worker][T#10]{New I/O worker #10}" daemon [_thread_in_native, id=23808, stack(0x00007f4b009ec000,0x00007f4b00aed000)]
  0x00007f4b7d0ce000 JavaThread "elasticsearch[node-100][transport_client_worker][T#9]{New I/O worker #9}" daemon [_thread_in_native, id=23807, stack(0x00007f4b00aed000,0x00007f4b00bee000)]
  0x00007f4b7d0a3000 JavaThread "elasticsearch[node-100][transport_client_worker][T#8]{New I/O worker #8}" daemon [_thread_in_native, id=23806, stack(0x00007f4b00bee000,0x00007f4b00cef000)]
  0x00007f4b7d078000 JavaThread "elasticsearch[node-100][transport_client_worker][T#7]{New I/O worker #7}" daemon [_thread_in_native, id=23805, stack(0x00007f4b00cef000,0x00007f4b00df0000)]
  0x00007f4b7d04c800 JavaThread "elasticsearch[node-100][transport_client_worker][T#6]{New I/O worker #6}" daemon [_thread_in_native, id=23804, stack(0x00007f4b00df0000,0x00007f4b00ef1000)]
  0x00007f4b7cf64800 JavaThread "elasticsearch[node-100][transport_client_worker][T#5]{New I/O worker #5}" daemon [_thread_in_native, id=23803, stack(0x00007f4b00ef1000,0x00007f4b00ff2000)]
  0x00007f4b7cf63000 JavaThread "elasticsearch[node-100][transport_client_worker][T#4]{New I/O worker #4}" daemon [_thread_in_native, id=23802, stack(0x00007f4b00ff2000,0x00007f4b010f3000)]
  0x00007f4b7cf5d800 JavaThread "elasticsearch[node-100][transport_client_worker][T#3]{New I/O worker #3}" daemon [_thread_in_native, id=23801, stack(0x00007f4b010f3000,0x00007f4b011f4000)]
  0x00007f4b7cf5c000 JavaThread "elasticsearch[node-100][transport_client_worker][T#2]{New I/O worker #2}" daemon [_thread_in_native, id=23800, stack(0x00007f4b011f4000,0x00007f4b012f5000)]
  0x00007f4b7cf61000 JavaThread "elasticsearch[node-100][transport_client_worker][T#1]{New I/O worker #1}" daemon [_thread_in_native, id=23799, stack(0x00007f4b012f5000,0x00007f4b013f6000)]
  0x00007f4b7cf4a000 JavaThread "elasticsearch[node-100][[ttl_expire]]" daemon [_thread_blocked, id=23798, stack(0x00007f4b013f6000,0x00007f4b014f7000)]
  0x00007f4b7cefc800 JavaThread "elasticsearch[node-100][scheduler][T#1]" daemon [_thread_blocked, id=23797, stack(0x00007f4b014f7000,0x00007f4b015f8000)]
  0x00007f4b7cd9f000 JavaThread "pool-1-thread-1" [_thread_blocked, id=23796, stack(0x00007f4b015f8000,0x00007f4b016f9000)]
  0x00007f4b7c894000 JavaThread "elasticsearch[node-100][[timer]]" daemon [_thread_blocked, id=23779, stack(0x00007f4b01b6b000,0x00007f4b01c6c000)]
  0x00007f4b7c263000 JavaThread "Service Thread" daemon [_thread_blocked, id=23773, stack(0x00007f4b0256d000,0x00007f4b0266e000)]
  0x00007f4b7c260800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=23772, stack(0x00007f4b0266e000,0x00007f4b0276f000)]
  0x00007f4b7c25e800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=23771, stack(0x00007f4b0276f000,0x00007f4b02870000)]
  0x00007f4b7c25c000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=23770, stack(0x00007f4b02870000,0x00007f4b02971000)]
  0x00007f4b7c25a000 JavaThread "Surrogate Locker Thread (Concurrent GC)" daemon [_thread_blocked, id=23769, stack(0x00007f4b02971000,0x00007f4b02a72000)]
  0x00007f4b7c239000 JavaThread "Finalizer" daemon [_thread_blocked, id=23768, stack(0x00007f4b02a72000,0x00007f4b02b73000)]
  0x00007f4b7c237000 JavaThread "Reference Handler" daemon [_thread_blocked, id=23767, stack(0x00007f4b02b73000,0x00007f4b02c74000)]

Other Threads:
  0x00007f4b7c233000 VMThread [stack: 0x00007f4b02c74000,0x00007f4b02d75000] [id=23766]
  0x00007f4b7c26e800 WatcherThread [stack: 0x00007f4b0246c000,0x00007f4b0256d000] [id=23774]

=>0x00007f4b7c1e6000 (exited) GCTaskThread [stack: 0x00007f4b19009000,0x00007f4b1910a000] [id=23761]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007f4b7c007f80] Heap_lock - owner thread: 0x00007f498c086000

Heap
 par new generation   total 1380160K, used 1327622K [0x00000003fae00000, 0x0000000458790000, 0x0000000458790000)
  eden space 1226816K, 100% used [0x00000003fae00000, 0x0000000445c10000, 0x0000000445c10000)
  from space 153344K,  65% used [0x0000000445c10000, 0x000000044be81800, 0x000000044f1d0000)
  to   space 153344K,   0% used [0x000000044f1d0000, 0x000000044f1d0000, 0x0000000458790000)
 concurrent mark-sweep generation total 15243712K, used 11443113K [0x0000000458790000, 0x00000007fae00000, 0x00000007fae00000)
 concurrent-mark-sweep perm gen total 83968K, used 63351K [0x00000007fae00000, 0x0000000800000000, 0x0000000800000000)

Card table byte_map: [0x00007f4b719d0000,0x00007f4b739fa000] byte_map_base: 0x00007f4b6f9f9000

Polling page: 0x00007f4b8173b000

Code Cache  [0x00007f4b78bda000, 0x00007f4b7a4da000, 0x00007f4b7bbda000)
 total_blobs=8571 nmethods=7968 adapters=553 free_code_cache=25001Kb largest_free_block=24160192

Compilation events (10 events):
Event: 743388.791 Thread 0x00007f4b7c260800 nmethod 11001 0x00007f4b7a394250 code [0x00007f4b7a394380, 0x00007f4b7a3943b8]
Event: 743388.793 Thread 0x00007f4b7c25e800 nmethod 11000 0x00007f4b7a393b90 code [0x00007f4b7a393ce0, 0x00007f4b7a393e88]
Event: 743388.793 Thread 0x00007f4b7c25e800 11002             java.lang.invoke.LambdaForm$MH/252998801::guard (66 bytes)
Event: 743388.796 Thread 0x00007f4b7c260800 11003             java.lang.invoke.LambdaForm$BMH/1474060855::reinvoke (28 bytes)
Event: 743388.797 Thread 0x00007f4b7c260800 nmethod 11003 0x00007f4b7a3937d0 code [0x00007f4b7a393920, 0x00007f4b7a393a18]
Event:
2017-05-14 添加评论

没有找到相关结果

    已邀请:

    Ricky_Lau - 码农，，

    赞同来自:

    解决 Full GC 的问题一般是有套路的，我是不了解任何关于你的情况 ，我给一个通用的笨办法。
     
    1. 确定的的GC 垃圾回收器， 是G1 还是CMS
     
         若是G1 ，先把其中一台有Full GC的节点内存翻一倍（最大不）超过30GB。
               G1配置如下即可（最大停顿时间根据自己的业务进行调整）
                     JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
                     JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=200"
         若是CMS ，也把其中一台有Full GC的节点内存翻一倍（最大不）超过30GB。
                CMS 回收器若没有配置对应的策略会产生大量的碎片，我建议的GC参数如下（以ES 分配30G内存为例）
                      JAVA_OPTS="$JAVA_OPTS -Xmn6G"
                      JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
                      JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
                      JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=60"
                      JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
                      JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSCompactAtFullCollection"
                      JAVA_OPTS="$JAVA_OPTS -XX:CMSFullGCsBeforeCompaction=3"
                      JAVA_OPTS="$JAVA_OPTS -XX:SurvivorRatio=6"
                      JAVA_OPTS="$JAVA_OPTS -XX:MaxTenuringThreshold=6"
                      JAVA_OPTS="$JAVA_OPTS -XX:+CMSScavengeBeforeRemark"
     
    2. 在 $ES_HOME/bin/elasticsearch.in.sh 处修改gc 日志的存放位置
             看附件
     
     
    3. 允许ES节点 生成日志（GC日志分割大小，以及是否以进程名后缀的策略自己百度去）
             在$ES_HOME/bin/elasticsearch 配置文件中添加
              ES_USE_GC_LOGGING=true
     
    4. 重启节点，检查GC
     
    5. 也可以把GC日志下载下来，通过GCVIEW来分析。
             https://github.com/chewiebug/GCViewer
     
    2017-05-14 0 0
    分享

    Ricky_Lau - 码农，，

    赞同来自:

    如果还是会FGC，那么检查业务查询方式是否有问题，是否有大量异步并发聚合，TTL删除等异常请求导致。
    2017-05-14 0 0
    分享

    ybtsdst - focus on lucene & es

    赞同来自:

    这个也得贴出来gc的时候集群在做什么吧, 写入还是查询?
    2017-05-15 0 0
    分享
    为什么被折叠? 0 个回复被折叠

    要回复问题请先登录或注册

    Copyright © 2024 · CC BY-NC-SA 3.0

    本站服务器及带宽由 提供赞助