mds0: Many clients (191) failing to respond to cache pressure

cephfs时我们产品依赖的主要分布式操作系统，但似乎很不给面子，压力测试的时候经常出问题。

目录

• 背景
• 后续的努力
• 临时的解决办法

背景

集群环境出现的问题: mds0: Many clients (191) failing to respond to cache pressure

背景：三个节点，100多个客户端mount，服务器可用内存仅剩100MB，ceph报错如下：

[root@node1 ceph]# ceph -s
    cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
     health HEALTH_WARN
            mds0: Many clients (191) failing to respond to cache pressure
     monmap e1: 3 mons at {node1=192.168.0.1:6789/0,node2=192.168.0.2:6789/0,node3=192.168.0.3:6789/0}
            election epoch 22, quorum 0,1,2 node1,node2,node3
      fsmap e924: 1/1/1 up {0=node1=up:active}, 2 up:standby
     osdmap e71: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds
      pgmap v48336: 576 pgs, 3 pools, 82382 MB data, 176 kobjects
            162 GB used, 5963 GB / 6126 GB avail
                 576 active+clean
  client io 0 B/s rd, 977 kB/s wr, 19 op/s rd, 116 op/s wr

至今问题也没有解决。(我的意思是说没有弄清楚Capacity的机制，如果抱着解决不了问题，就解决提出问题的人的思路，可以参考第三部分。)

mds日志如下：

2019-11-12 16:00:17.679876 7fa6a5040700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.236623 secs
2019-11-12 16:00:17.679914 7fa6a5040700  0 log_channel(cluster) log [WRN] : slow request 34.236623 seconds old, received at 2019-11-12 15:59:43.326917: client_request(client.154893:13683 open #1000005cb77 2019-11-12 15:59:43.293037) currently failed to xlock, waiting
2019-11-12 16:03:27.614474 7fa6a5040700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.350555 secs
2019-11-12 16:03:27.614523 7fa6a5040700  0 log_channel(cluster) log [WRN] : slow request 34.350555 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:57.615297 7fa6a5040700  0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 64.351379 secs
2019-11-12 16:03:57.615322 7fa6a5040700  0 log_channel(cluster) log [WRN] : slow request 64.351379 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:58.181330 7fa6a5040700  0 log_channel(cluster) log [WRN] : client.155079 isn't responding to mclientcaps(revoke), ino 1000003e360 pending pAsxLsXsxFcb issued pAsxLsXsxFsxcrwb, sent 64.458260 seconds ago

后续的努力

自己找环境重现，用的一个测试服务器，安装了一个Ubuntu系统，然后进行测试。惊喜的发现，同一个客户端不管我mount多少个目录，与后端的连接始终都只有那两个。

但重现过程中还是出现类似的问题了。

mds0: Client ubuntu:guest failing to respond to capability release

静置一段时间之后出现了如下错误:

[root@ceph741 ~]# ceph -s
    cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
     health HEALTH_WARN
            mds0: Client ubuntu:guest failing to respond to capability release
            mds0: Client ubuntu:guest failing to advance its oldest client/flush tid
     monmap e2: 3 mons at {ceph741=192.168.15.112:6789/0,ceph742=192.168.15.113:6789/0,ceph743=192.168.15.114:6789/0}
            election epoch 38, quorum 0,1,2 ceph741,ceph742,ceph743
      fsmap e8989: 1/1/1 up {0=ceph743=up:active}, 2 up:standby
     osdmap e67: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds
      pgmap v847657: 576 pgs, 3 pools, 20803 MB data, 100907 objects
            44454 MB used, 241 GB / 284 GB avail
                 576 active+clean
  client io 59739 B/s rd, 3926 kB/s wr, 58 op/s rd, 770 op/s wr

临时的解决办法

临时的解决办法就是把出问题的客户端干掉。

步骤主要命令：

ceph tell  mds.0 session ls
ceph tell mds.0 session evict id=249632

其中id是问题client的id。那么问题客户端比其他客户端哪里不同呢，实话说，我也不知道，大家可以看下：

参考：

https://www.jianshu.com/p/d1e0e32346ac

http://www.talkwithtrend.com/Article/242905

https://www.jianshu.com/p/fa49e40f6133