mysql hang and srv_error_monitor_thread using 100% cpu（已解决）

昨天晚上，运维过来说有台生产服务器的mysql cpu一直100%，新的客户端登录不了，但是已经在运行的应用都正常可用。

登录服务器后，top -H看了下，其中一个线程的cpu 一直100%，其他的几乎都空闲。

MySQL thread id 14560536, OS thread handle 0x7f1255ef1700, query id 31889137761 10.26.124.8 osm cleaning up

top - 18:56:26 up 62 days, 3:55, 3 users, load average: 1.08, 1.13, 1.18
Tasks: 1503 total, 2 running, 1501 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.9%us, 0.2%sy, 0.0%ni, 86.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16333644k total, 15440672k used, 892972k free, 45652k buffers
Swap: 4153340k total, 0k used, 4153340k free, 1050524k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15669 mysql 20 0 8338m 5.9g 5808 R 100.0 38.1 162:29.18 mysqld
29679 rabbitmq 20 0 4315m 154m 1780 S 1.3 1.0 1162:54 beam.smp
3887 root 20 0 16092 2360 940 R 0.7 0.0 0:00.27 top
426 root 20 0 68020 6136 4244 S 0.3 0.0 0:03.34 AliYunDun
427 root 20 0 68020 6136 4244 S 0.3 0.0 0:02.62 AliYunDun
430 root 20 0 68020 6136 4244 S 0.3 0.0 0:08.37 AliYunDun
432 root 20 0 68020 6136 4244 S 0.3 0.0 0:05.34 AliYunDun
1247 root 20 0 245m 49m 1604 S 0.3 0.3 28:40.31 AliHids
1249 root 20 0 245m 49m 1604 S 0.3 0.3 10:17.96 AliHids
3653 root 20 0 113m 1572 932 S 0.3 0.0 0:00.11 wrapper
12293 root 20 0 9722m 1.3g 5940 S 0.3 8.6 20:10.09 java
16457 root 20 0 8710m 166m 5772 S 0.3 1.0 1:52.67 java
16914 root 20 0 9864m 168m 5808 S 0.3 1.1 19:30.97 java
29680 rabbitmq 20 0 4315m 154m 1780 S 0.3 1.0 578:21.03 beam.smp
1 root 20 0 19356 800 500 S 0.0 0.0 0:02.80 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:15.65 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:32.07 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:05.98 watchdog/0

这个问题已经发生过接近10来次了，只不过之前除了一两次是其他的生产系统外，大部分发生在开发环境，一直也没找到原因，一旦kill这个线程，就导致mysql实例被kill并自动重启了。

因为之前发生过，没解决，所以刚好那台服务器的Innodb_monitor开着，但是根据时间往后退算到差不多出问题的那一刻，只看到有一个事务从出问题开始一直持续运行了差不多20多分钟，如下：

FILE I/O
--
MySQL thread id 14560536, OS thread handle 0x7f122db3c700, query id 31889113153 10.26.124.8 osm cleaning up
---TRANSACTION 604639618, ACTIVE 1248 sec
MySQL thread id 14561064, OS thread handle 0x7f122db3c700, query id 31887588399 125.118.111.42 osm cleaning up
--------
FILE I/O
--
MySQL thread id 14560536, OS thread handle 0x7f122db3c700, query id 31889113154 10.26.124.8 osm cleaning up
---TRANSACTION 604639618, ACTIVE 1268 sec
MySQL thread id 14561064, OS thread handle 0x7f122db3c700, query id 31887588399 125.118.111.42 osm cleaning up
--------
FILE I/O
--
MySQL thread id 14560536, OS thread handle 0x7f122db3c700, query id 31889113373 10.26.124.8 osm cleaning up
---TRANSACTION 604639618, ACTIVE 1288 sec
MySQL thread id 14561064, OS thread handle 0x7f122db3c700, query id 31887588399 125.118.111.42 osm cleaning up

因为所有的系统配置参数、mysql都是统一的，OS版本、内核版本都是一致的，所以应该不是版本和某个参数导致的问题。

跟相关运维确认的时候，只说用gui连接过，但是没有做运行慢的操作比如直接通过navicat进行导出等操作。所以这个问题比较像之前遇到的内存问题，光靠模拟它好像永远不出来，但某个时间就出来了。

strace和ltrace也没看出什么问题，那会儿实在比较郁闷。。。折腾到后来，mysqld不知怎么被不小心的killed了。

早上回来，继续研究，搜了bing半天，最后在mysql buglist发现就是5.5.23之前有个performance_schema的bug，跟我们的现象极为相似，如下：

http://bugs.mysql.com/bug.php?id=64491

http://lists.mysql.com/commits/143350?f=plain

one thread call sched_yield(), but other threads are't ready run. so cause cpu 100%.

但根据开发者所说这个问题在5.5.23以及5.6.6就已经修复，我们使用的是percona server 5.6.31，理论上应该不会出现这个bug的。

因为现在没有当时的现场了，只能等下次出现gdb上去看下，那个出问题的线程在执行什么操作了，还是非c++出身gdb不熟导致的，哎。。。这个负责性能优化的gdb/wireshark真得熟悉的滚瓜烂熟，不然有些诡异的现象光通过服务器本身提供的API和CLI和一些高级工具真不能定位到原因啊。。

这个问题昨天在另外一个测试环境又出现了，当时就用gdb看了下出问题的那个线程，根据当时的线程状态来看，是srv_error_monitor_thread线程执行thd_kill时得不到mutex所致，正常其状态应该如下：

(gdb) thread 21
[Switching to thread 21 (Thread 0x7f2c203ff700 (LWP 12665))]#0 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) bt
#0 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a06317 in os_cond_wait_timed (event=0x7f2c427541d0, time_in_usec=Unhandled dwarf expression opcode 0xf3
)
at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/storage/innobase/os/os0sync.cc:159
#2 os_event_wait_time_low (event=0x7f2c427541d0, time_in_usec=Unhandled dwarf expression opcode 0xf3
) at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/storage/innobase/os/os0sync.cc:591
#3 0x0000000000a6b991 in srv_error_monitor_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/storage/innobase/srv/srv0srv.cc:2193
#4 0x0000003d30207aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003d2fee893d in clone () from /lib64/libc.so.6

因为在5.6.31和5.7.15版本均出现过这个问题，所以可以进一步确定不是版本的问题。

http://bugs.mysql.com/bug.php?id=60682

1226，今天这个问题又出现了，准确的说，发生时间应该是在前天晚上12点左右，详细的现状如下：

[root@iZbp1ebrtsmjsxqcnds5gkZ ~]# top
top - 09:01:55 up 18 days, 21:40, 3 users, load average: 1.17, 1.20, 1.17
Tasks: 986 total, 2 running, 984 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.8%us, 0.4%sy, 0.0%ni, 73.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.1%st
Mem: 8057964k total, 7536780k used, 521184k free, 173284k buffers
Swap: 4194300k total, 108644k used, 4085656k free, 1567672k cached
Unknown command - try 'h' for help
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12633 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:27.58 mysqld
12634 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:00.00 mysqld
12635 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.60 mysqld
12636 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:12.46 mysqld
12637 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.41 mysqld
12638 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.79 mysqld
12639 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.65 mysqld
12640 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.55 mysqld
12641 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.47 mysqld
12642 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.77 mysqld
12643 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.30 mysqld
12644 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.56 mysqld
12645 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:13.84 mysqld
12646 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.51 mysqld
12647 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.77 mysqld
12648 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.85 mysqld
12649 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:12.15 mysqld
12650 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.35 mysqld
12651 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.83 mysqld
12652 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:11.55 mysqld
12664 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:06.94 mysqld
12665 mysql 20 0 3310m 1.9g 9448 R 99.8 25.4 2037:27 mysqld
12666 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:01.44 mysqld
12667 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:15.27 mysqld
12668 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:05.69 mysqld
12669 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 2:37.74 mysqld
12670 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:08.36 mysqld
12671 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:00.00 mysqld
12672 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:00.87 mysqld
12673 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:01.47 mysqld
12674 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:08.41 mysqld  
12677 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:00.00 mysqld
12678 mysql 20 0 3310m 1.9g 9448 S 0.0 25.4 0:00.00 mysqld

[root@iZbp1ebrtsmjsxqcnds5gkZ ~]# cd /usr/local/mysql/bin/
[root@iZbp1ebrtsmjsxqcnds5gkZ bin]# gdb mysqld 12633
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/Percona-Server-5.6.31-rel77.0-Linux.x86_64.ssl101/bin/mysqld...done.
Attaching to program: /usr/local/Percona-Server-5.6.31-rel77.0-Linux.x86_64.ssl101/bin/mysqld, process 12633
Reading symbols from /usr/local/Percona-Server-5.6.31-rel77.0-Linux.x86_64.ssl101/lib/mysql/libjemalloc.so.1...done.
Loaded symbols for /usr/local/Percona-Server-5.6.31-rel77.0-Linux.x86_64.ssl101/lib/mysql/libjemalloc.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 17182]
[New LWP 17178]
[New LWP 17177]
[New LWP 17176]
[New LWP 17175]
[New LWP 27521]
[New LWP 21368]
[New LWP 14998]
[New LWP 13101]
[New LWP 11038]
[New LWP 11010]
[New LWP 10995]
[New LWP 4516]
[New LWP 31723]
[New LWP 30090]
[New LWP 22653]
[New LWP 19885]
[New LWP 16066]
[New LWP 2171]
[New LWP 12678]
[New LWP 12677]
[New LWP 12674]
[New LWP 12673]
[New LWP 12672]
[New LWP 12671]
[New LWP 12670]
[New LWP 12669]
[New LWP 12668]
[New LWP 12667]
[New LWP 12666]
[New LWP 12665]
[New LWP 12664]
[New LWP 12652]
[New LWP 12651]
[New LWP 12650]
[New LWP 12649]
[New LWP 12648]
[New LWP 12647]
[New LWP 12646]
[New LWP 12645]
[New LWP 12644]
[New LWP 12643]
[New LWP 12642]
[New LWP 12641]
[New LWP 12640]
[New LWP 12639]
[New LWP 12638]
[New LWP 12637]
[New LWP 12636]
[New LWP 12635]
[New LWP 12634]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libaio.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libaio.so.1
Reading symbols from /usr/lib64/libnuma.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnuma.so.1
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-15.el6_5.1.x86_64 libaio-0.3.107-10.el6.x86_64 libcom_err-1.41.12-18.el6_5.1.x86_64 libgcc-4.4.7-4.el6.x86_64 libselinux-2.0.94-5.3.el6_4.1.x86_64 libstdc++-4.4.7-4.el6.x86_64 nss-softokn-freebl-3.14.3-10.el6_5.x86_64 numactl-2.0.7-8.el6.x86_64 openssl-1.0.1e-30.el6_6.5.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) info threads
52 Thread 0x7f2c40df1700 (LWP 12634) 0x0000003d2fe33857 in sigwaitinfo () from /lib64/libc.so.6
51 Thread 0x7f2c2c7ff700 (LWP 12635) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
50 Thread 0x7f2c2bdfe700 (LWP 12636) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
49 Thread 0x7f2c2b3fd700 (LWP 12637) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
48 Thread 0x7f2c2a9fc700 (LWP 12638) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
47 Thread 0x7f2c29ffb700 (LWP 12639) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
46 Thread 0x7f2c295fa700 (LWP 12640) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
45 Thread 0x7f2c28bf9700 (LWP 12641) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
44 Thread 0x7f2c281f8700 (LWP 12642) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
43 Thread 0x7f2c277f7700 (LWP 12643) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
42 Thread 0x7f2c26df6700 (LWP 12644) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
41 Thread 0x7f2c263f5700 (LWP 12645) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
40 Thread 0x7f2c259f4700 (LWP 12646) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
39 Thread 0x7f2c24ff3700 (LWP 12647) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
38 Thread 0x7f2c245f2700 (LWP 12648) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
37 Thread 0x7f2c23bf1700 (LWP 12649) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
36 Thread 0x7f2c231f0700 (LWP 12650) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
35 Thread 0x7f2c227ef700 (LWP 12651) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
34 Thread 0x7f2c21dee700 (LWP 12652) 0x0000003d2fa00614 in ?? () from /lib64/libaio.so.1
33 Thread 0x7f2c1efff700 (LWP 12664) 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
32 Thread 0x7f2c203ff700 (LWP 12665) 0x00000000006c91cd in thd_kill (id=229858) at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/sql/sql_class.cc:902
31 Thread 0x7f2c1e5fe700 (LWP 12666) 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
30 Thread 0x7f2c1dbfd700 (LWP 12667) 0x0000003d2fee13b3 in select () from /lib64/libc.so.6
29 Thread 0x7f2c1d1fc700 (LWP 12668) 0x0000003d3020b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
28 Thread 0x7f2c1c7fb700 (LWP 12669) 0x0000003d2fee13b3 in select () from /lib64/libc.so.6
27 Thread 0x7f2c1bdfa700 (LWP 12670) 0x0000003d2fee13b3 in select () from /lib64/libc.so.6
26 Thread 0x7f2c1b3f9700 (LWP 12671) 0x0000003d3020b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
25 Thread 0x7f2c1a9f8700 (LWP 12672) 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
24 Thread 0x7f2c19ff7700 (LWP 12673) 0x0000003d3020ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
23 Thread 0x7f2c17fff700 (LWP 12674) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
22 Thread 0x7f2c175fe700 (LWP 12677) 0x0000003d3020f585 in sigwait () from /lib64/libpthread.so.0
21 Thread 0x7f2c175bd700 (LWP 12678) 0x0000003d3020b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
20 Thread 0x7f2c0ffff700 (LWP 2171) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
19 Thread 0x7f2ba6bff700 (LWP 16066) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
18 Thread 0x7f2c0ff3c700 (LWP 19885) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
17 Thread 0x7f2c1757c700 (LWP 22653) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
16 Thread 0x7f2c730ff700 (LWP 30090) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
15 Thread 0x7f2c7307d700 (LWP 31723) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
14 Thread 0x7f2c1753b700 (LWP 4516) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
13 Thread 0x7f2c0ffbe700 (LWP 10995) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
12 Thread 0x7f2c174b9700 (LWP 11010) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
11 Thread 0x7f2c17478700 (LWP 11038) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
10 Thread 0x7f2c174fa700 (LWP 13101) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
9 Thread 0x7f2c117ff700 (LWP 14998) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
8 Thread 0x7f2c147ff700 (LWP 21368) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
7 Thread 0x7f2c0ff7d700 (LWP 27521) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
6 Thread 0x7f2c730be700 (LWP 17175) 0x0000003d2fee8f33 in epoll_wait () from /lib64/libc.so.6
5 Thread 0x7f2c12bff700 (LWP 17176) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
4 Thread 0x7f2ba6bbe700 (LWP 17177) 0x0000003d2fee8f33 in epoll_wait () from /lib64/libc.so.6
3 Thread 0x7f2ba6b7d700 (LWP 17178) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0
2 Thread 0x7f2ba6aba700 (LWP 17182) 0x0000003d2fee8f33 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 0x7f2c73137840 (LWP 12633) 0x0000003d3020e334 in __lll_lock_wait () from /lib64/libpthread.so.0

(gdb) thread 32

#0 0x00000000006c91cd in thd_kill (id=229858) at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/sql/sql_class.cc:902
#1 0x0000000000a6ba46 in srv_error_monitor_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /mnt/workspace/percona-server-5.6-binaries-release/label_exp/centos6-64/percona-server-5.6.31-77.0/storage/innobase/srv/srv0srv.cc:2177
#2 0x0000003d30207aa1 in start_thread () from /lib64/libpthread.so.0
#3 0x0000003d2fee893d in clone () from /lib64/libc.so.6

http://forums.mysql.com/read.php?132,332546,356777#msg-356777

20170105，今天有个新的环境又连续出现这个问题，现在感觉有可能跟thread_concurrency的设置可能有关系，我们基本上设置为了cpu核心数。可参考：

https://dev.mysql.com/doc/refman/5.6/en/innodb-performance-thread_concurrency.html

http://assets.en.oreilly.com/1/event/36/High%20Concurrency%20MySQL%20Presentation.pdf

http://serverfault.com/questions/695975/mysql-innodb-processes-waiting-on-futex

https://bugs.launchpad.net/percona-server/+bug/1489454

https://bugs.mysql.com/bug.php?id=48929

http://bugs.mysql.com/bug.php?id=54982

https://www.percona.com/blog/2011/12/02/kernel_mutex-problem-or-double-throughput-with-single-variable/

https://www.percona.com/blog/2011/12/02/kernel_mutex-problem-cont-or-triple-your-throughput/

https://www.percona.com/blog/2011/07/28/how-innodb-contention-may-manifest-itself/

http://blog.csdn.net/cug_jiang126com/article/details/53114583

好久没出现问题0209，这两天一个环境又出现了这个问题，现象还是一样，pstack看了下，似乎问题出现线程池空闲后的kill时、得不到995锁（据查应该是pthread的lock_thread_count），如下所示：

Thread 33 (Thread 0x7fefcbbff700 (LWP 25641)):
#0 0x00000000006c91cd in thd_kill ()
#1 0x0000000000a6ba46 in srv_error_monitor_thread ()
#2 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#3 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 32 (Thread 0x7fefc89fe700 (LWP 25642)):
#0 0x000000310620ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a06317 in os_event_wait_time_low(os_event*, unsigned long, long) ()
#2 0x0000000000a6b3ac in srv_monitor_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 31 (Thread 0x7fefc7ffd700 (LWP 25643)):
#0 0x0000003105ee13b3 in select () from /lib64/libc.so.6
#1 0x0000000000a06adf in os_thread_sleep(unsigned long) ()
#2 0x0000000000a6ce30 in srv_master_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7fefc75fc700 (LWP 25644)):
#0 0x000000310620b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a061eb in os_event_wait_low(os_event*, long) ()
#2 0x0000000000a6e5c0 in srv_purge_coordinator_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fefc6bfb700 (LWP 25645)):
#0 0x0000003105ee13b3 in select () from /lib64/libc.so.6
#1 0x0000000000a06adf in os_thread_sleep(unsigned long) ()
#2 0x0000000000ae63f2 in buf_flush_page_cleaner_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fefc61fa700 (LWP 25646)):
#0 0x0000003105ee13b3 in select () from /lib64/libc.so.6
#1 0x0000000000a06adf in os_thread_sleep(unsigned long) ()
#2 0x0000000000ae73b7 in buf_flush_lru_manager_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fefc57f9700 (LWP 25647)):
#0 0x000000310620b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a061eb in os_event_wait_low(os_event*, long) ()
#2 0x0000000000adfa6c in buf_dump_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fefc4df8700 (LWP 25648)):
#0 0x000000310620ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a06317 in os_event_wait_time_low(os_event*, unsigned long, long) ()
#2 0x0000000000b1db03 in dict_stats_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fefc43f7700 (LWP 25649)):
#0 0x000000310620ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000a06317 in os_event_wait_time_low(os_event*, unsigned long, long) ()
#2 0x0000000000a9cc41 in ib_wqueue_timedwait(ib_wqueue_t*, long) ()
#3 0x0000000000b527a3 in fts_optimize_thread(void*) ()
#4 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fefc39f6700 (LWP 25653)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x0000000000795cd9 in timer_thread(void*) ()
#4 0x0000000000985e26 in pfs_spawn_thread ()
#5 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fefc2ff5700 (LWP 25656)):
#0 0x000000310620f585 in sigwait () from /lib64/libpthread.so.0
#1 0x00000000005ae25b in signal_hand ()
#2 0x0000000000985e26 in pfs_spawn_thread ()
#3 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fefc2fb4700 (LWP 25657)):
#0 0x000000310620b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000087a067 in Event_queue::cond_wait(THD*, timespec*, PSI_stage_info_v1 const*, char const*, char const*, unsigned int) ()
#2 0x000000000087a117 in Event_queue::get_top_for_execution_if_time(THD*, Event_queue_element_for_exec**) ()
#3 0x000000000087b461 in Event_scheduler::run(THD*) ()
#4 0x000000000087b5cc in event_scheduler_thread ()
#5 0x0000000000985e26 in pfs_spawn_thread ()
#6 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fefc2d6b700 (LWP 4586)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fef247be700 (LWP 12528)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b2910 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fef20bff700 (LWP 18609)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fefc2e6f700 (LWP 19620)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fef867ff700 (LWP 20515)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fef8677d700 (LWP 20531)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fef5dbff700 (LWP 20557)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fefc2dac700 (LWP 20944)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fefc07be700 (LWP 21142)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fef8673c700 (LWP 21183)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fef2477d700 (LWP 21307)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fefc2f32700 (LWP 21310)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fefc2f73700 (LWP 21409)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fefc2eb0700 (LWP 21473)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fefc2e2e700 (LWP 21474)):
#0 0x0000003105ee8f33 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000793de3 in listener ()
#2 0x00000000007943ca in worker_main(void*) ()
#3 0x0000000000985e26 in pfs_spawn_thread ()
#4 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fefbc7ff700 (LWP 21476)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fef247ff700 (LWP 21805)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fefc2ef1700 (LWP 22123)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fefc2ded700 (LWP 23602)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b28f0 in remove_global_thread(THD*) ()
#4 0x000000000079379f in threadpool_remove_connection(THD*) ()
#5 0x00000000007942df in worker_main(void*) ()
#6 0x0000000000985e26 in pfs_spawn_thread ()
#7 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#8 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fef1ffff700 (LWP 26329)):
#0 0x0000003105ee8f33 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000793de3 in listener ()
#2 0x00000000007943ca in worker_main(void*) ()
#3 0x0000000000985e26 in pfs_spawn_thread ()
#4 0x0000003106207aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003105ee893d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff12177f840 (LWP 25607)):
#0 0x000000310620e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000310620960e in _L_lock_995 () from /lib64/libpthread.so.0
#2 0x0000003106209576 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000005b110f in handle_connections_sockets() ()
#4 0x00000000005ba58f in mysqld_main(int, char**) ()
#5 0x0000003105e1ed5d in __libc_start_main () from /lib64/libc.so.6
#6 0x00000000005ab5ed in _start ()
[root@iZbp112kwadw1qt8emked5Z mysql]#

把thread_cache_size设置为max-connections/thread-pool-size，应该是可以解决。参考如下：

http://docs.linuxtone.org/ebooks/MySQL/linuxperformancetuningandstabilizationtips-mysqlconf2010.pdf

https://bugs.mysql.com/bug.php?id=53045

http://www.th7.cn/db/mysql/201608/201249.shtml