场景描述:
收到zabbix监控报警,晋中生产机器出现磁盘空间不足报警。
远程到该服务器,排查占员工磁盘空间的原因,发现tomcat日志过多,于是清除3天前的日志。
日志清理后,发现磁盘空间没有释放,如下是处理过程和原因分析:
日志清理后,查看磁盘空间:
[root@jzreport2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 50G 29G 64% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
原因应该是已经删除的文件仍然被某个进程占用,所以导致磁盘空间得不到释放,如下命令可以查看已经删除的空间爱你被哪些进程占用。
[root@jzreport2 ~]# lsof |grep -i delete
java 4825 root 13r REG 253,0 416582455 5377302 /geelyapp/tomcat-ime/logs/catalina-2016-07-26.out (deleted)
java 4825 root 15r REG 253,0 3815677 5377477 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-22.txt (deleted)
java 4825 root 16r REG 253,0 7384392 5377242 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-30.txt (deleted)
java 4825 root 18r REG 253,0 364016974 5377293 /geelyapp/tomcat-ime/logs/catalina-2016-07-28.out (deleted)
java 4825 root 24r REG 253,0 19699678 5377464 /geelyapp/tomcat-ime/logs/catalina-2016-07-22.out (deleted)
java 4825 root 27r REG 253,0 368883739 5377240 /geelyapp/tomcat-ime/logs/catalina-2016-07-30.out (deleted)
java 4825 root 28r REG 253,0 358485718 5377247 /geelyapp/tomcat-ime/logs/catalina-2016-07-31.out (deleted)
java 4825 root 31r REG 253,0 7239318 5377338 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-28.txt (deleted)
java 4825 root 32r REG 253,0 5689600 5382395 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-24.txt (deleted)
java 4825 root 33r REG 253,0 355019854 5377250 /geelyapp/tomcat-ime/logs/catalina-2016-08-01.out (deleted)
java 4825 root 35r REG 253,0 2988975349 5377294 /geelyapp/tomcat-ime/logs/catalina-2016-07-25.out (deleted)
java 4825 root 37r REG 253,0 5417099 5377320 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-27.txt (deleted)
java 4825 root 39r REG 253,0 7357142 5377249 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-31.txt (deleted)
java 4825 root 40r REG 253,0 371975503 5377318 /geelyapp/tomcat-ime/logs/catalina-2016-07-27.out (deleted)
java 4825 root 41r REG 253,0 5413317330 5382387 /geelyapp/tomcat-ime/logs/catalina-2016-07-23.out (deleted)
java 4825 root 42r REG 253,0 353928941 5377258 /geelyapp/tomcat-ime/logs/catalina-2016-08-02.out (deleted)
java 4825 root 44r REG 253,0 19902702 5382393 /geelyapp/tomcat-ime/logs/catalina-2016-07-24.out (deleted)
java 4825 root 45r REG 253,0 7333147 5377252 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-01.txt (deleted)
java 4825 root 46r REG 253,0 358733756 5377265 /geelyapp/tomcat-ime/logs/catalina-2016-08-03.out (deleted)
java 4825 root 48r REG 253,0 371727923 5377271 /geelyapp/tomcat-ime/logs/catalina-2016-08-04.out (deleted)
java 4825 root 50r REG 253,0 7328014 5377260 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-02.txt (deleted)
java 4825 root 51r REG 253,0 385891702 5377279 /geelyapp/tomcat-ime/logs/catalina-2016-08-05.out (deleted)
java 4825 root 54r REG 253,0 7336642 5377267 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-03.txt (deleted)
java 4825 root 55r REG 253,0 7446032 5377307 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-06.txt (deleted)
java 4825 root 61r REG 253,0 7375397 5377274 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-04.txt (deleted)
java 4825 root 62r REG 253,0 419590833 5377332 /geelyapp/tomcat-ime/logs/catalina-2016-08-08.out (deleted)
java 4825 root 63r REG 253,0 7534494 5377304 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-26.txt (deleted)
java 4825 root 64r REG 253,0 6619016 5382398 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-25.txt (deleted)
java 4825 root 65r REG 253,0 5664294 5382389 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-23.txt (deleted)
java 4825 root 66r REG 253,0 7408877 5377282 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-05.txt (deleted)
java 4825 root 67r REG 253,0 7379473 5377350 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-29.txt (deleted)
java 4825 root 69r REG 253,0 379479937 5377347 /geelyapp/tomcat-ime/logs/catalina-2016-07-29.out (deleted)
java 4825 root 70r REG 253,0 382905142 5377342 /geelyapp/tomcat-ime/logs/catalina-2016-08-09.out (deleted)
java 4825 root 72r REG 253,0 441397404 5377364 /geelyapp/tomcat-ime/logs/catalina-2016-08-11.out (deleted)
java 4825 root 73r REG 253,0 393485017 5377295 /geelyapp/tomcat-ime/logs/catalina-2016-08-06.out (deleted)
java 4825 root 76r REG 253,0 7386336 5377330 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-07.txt (deleted)
java 4825 root 77r REG 253,0 371199837 5377323 /geelyapp/tomcat-ime/logs/catalina-2016-08-07.out (deleted)
java 4825 root 79r REG 253,0 7475422 5377336 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-08.txt (deleted)
java 4825 root 81r REG 253,0 360021471 5377379 /geelyapp/tomcat-ime/logs/catalina-2016-08-13.out (deleted)
java 4825 root 83r REG 253,0 7401499 5377344 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-09.txt (deleted)
java 4825 root 84r REG 253,0 7421931 5377360 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-10.txt (deleted)
java 4825 root 86r REG 253,0 367824636 5377389 /geelyapp/tomcat-ime/logs/catalina-2016-08-15.out (deleted)
java 4825 root 88r REG 253,0 7594223 5377367 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-11.txt (deleted)
java 4825 root 89r REG 253,0 373697996 5377358 /geelyapp/tomcat-ime/logs/catalina-2016-08-10.out (deleted)
java 4825 root 90r REG 253,0 7218216 5377377 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-12.txt (deleted)
java 4825 root 94r REG 253,0 7207574 5377381 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-13.txt (deleted)
java 4825 root 96r REG 253,0 363153109 5377375 /geelyapp/tomcat-ime/logs/catalina-2016-08-12.out (deleted)
java 4825 root 97r REG 253,0 6953046 5377386 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-14.txt (deleted)
java 4825 root 100r REG 253,0 7443747 5377390 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-15.txt (deleted)
java 4825 root 102r REG 253,0 301548658 5377387 /geelyapp/tomcat-ime/logs/catalina-2016-08-14.out (deleted)
java 4825 root 104r REG 253,0 8306681 5377453 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-16.txt (deleted)
java 4825 root 105r REG 253,0 530882613 5377597 /geelyapp/tomcat-ime/logs/catalina-2016-08-19.out (deleted)
java 4825 root 110r REG 253,0 8246802 5377581 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-17.txt (deleted)
java 4825 root 111r REG 253,0 608724746 5377450 /geelyapp/tomcat-ime/logs/catalina-2016-08-16.out (deleted)
java 4825 root 113r REG 253,0 8433313 5377592 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-18.txt (deleted)
java 4825 root 114r REG 253,0 8173165 5377599 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-19.txt (deleted)
java 4825 root 117r REG 253,0 8061150 5377605 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-20.txt (deleted)
java 4825 root 119r REG 253,0 7948469 5377611 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-21.txt (deleted)
java 4825 root 121r REG 253,0 612650394 5377589 /geelyapp/tomcat-ime/logs/catalina-2016-08-18.out (deleted)
java 4825 root 127r REG 253,0 627820458 5377580 /geelyapp/tomcat-ime/logs/catalina-2016-08-17.out (deleted)
java 4825 root 142r REG 253,0 475492585 5377603 /geelyapp/tomcat-ime/logs/catalina-2016-08-20.out (deleted)
zabbix_ag 15644 zabbix 3wW REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15646 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15647 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15648 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15649 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15650 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
catalina. 25220 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
cronolog 25221 root 2u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
java 25232 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
以上可以看出已经删除的tomcat日志被进程4825所占用:(本次的4825进程是logstash进程)
重启4825进程后,空间释放:
[root@jzreport2 logstash-2.1.1]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 32G 47G 41% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
系统的日志采集在logstach中配置的是*号匹配日期,tomcat日志删除后,可能会出现空间不释放问题,可以参考上述处理过程进行处理。
- linux删除文件空间不释放问题解决
目录 场景描述 原因 解决方法 在线清空文件 场景描述 某天,收到Prometheus报警,生产中某台机器出现磁盘空间不足报警,该台服务器是mysql其中一台从库,远程登录到该服务后,排查磁盘空间的原 ...
- linux 删除文件空间未释放问题
现象:我们测试环境上,导出数据文件时,由于作业报错,重复导出,空间使用到达100%,按理说,导出的文件时在相同的路径下,文件名也是一致的,会自动替换. 那么之前导出的文件会被删除,问题就出现在删除这一 ...
- linux删除文件未释放空间问题处理
linux删除文件未释放空间问题处理 或者 /根分区满了 (我的根分区是/dev/sda1,/dev/sda1满了) http://blog.csdn.net/donghustone/article/ ...
- 【转】Linux删除文件未释放空间问题处理
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- Linux 删除文件未释放空间问题处理,下清空或删除大文件
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- linux删除文件未释放
https://access.redhat.com/solutions/2316 $ /usr/sbin/lsof | grep deleted ora 25575 data 33u REG 65,6 ...
- Linux文件删除空间未释放
当系统空间使用量过大需要清理空间或者清理某个文件时,有时会出现执行了删除命令之后磁盘空间并没有释放,很多人首次遇到该情况时会比较困惑,在考虑是不是像windows系统的回收站一样,删除只是逻辑删除到回 ...
- 解决linux删除文件后空间没有释放问题
linux删除文件后沒有释放空间 今天发现一台服务器的home空间满了,于是要清空没用的文件,当我删除文件后,发现可用空间沒有变化 os:centos4.7 现象: 发现当前磁盘空间使用情况: [ro ...
- Linux 删除文件后空间不释放【原创】
删除MySQL备份文件后,查找文件所在目录发现文件已经备删除了,但是空间没有释放,还是83% 解决方法: lsof|grep -i delete 发现进程还在,杀掉进程 kill -9 5377 再次 ...
随机推荐
- vue 强制刷新组件重新渲染
实现功能:使用富文本编辑器编写文章,然后把编写成功的文章用子组件显示. 问题描述:父组件给子组件传递数据,子组件第一次调用数据的时候页面渲染是正常的,当数据变化的时候,子组件的页面渲染就失效了. 问题 ...
- U1. 广度优先搜索(BFS)和 广度优先搜索(DFS)
广度优先搜索用栈(stack)来实现,整个过程可以想象成一个倒立的树形: 1.把根节点压入栈中. 2.每次从栈中弹出一个元素,搜索所有在它下一级的元素,把这些元素压入栈中.并把这个元素记为它下一级元素 ...
- 使用idea上传项目初始化版本到coding
1.在coding创建项目 2.使用idea命令控制台初始化本地仓库 3.将代码提交到本地仓库,git add . 或者 git add <filename> 4.将本地仓库文件推送到co ...
- GC(Garbage Collection)
GC(Garbage Collection) GC背景 创建对象会消耗内存,如果不回收对象占用的内存,内存使用率会越来越高,最终出现OutOfMemoryError(OOM) 在C++中专 ...
- springcloud断路器Dashboard监控仪表盘的使用
断路器Dashboard监控仪表盘的使用 在原有的orderserverfeignhystrix服务中使用 1.增加依赖仓库 <dependency> <g ...
- 第一个vue程序:hello,vlue
<!DOCTYPE html> <html> <head> <meta charset="utf-8"> <meta http ...
- asp.net core-14.JWT认证授权 生成 JWT Token
源码下载 语言组织能力不好 ,看这个 视频 用visual studio code打开文件,运行dotnet watch run 之后在postman里面去访问 拿到Token后
- (十三)JSON的使用(下)
(接上一章节) 4.2 使用ajax交互方式 使用ajax的转换思想 A:异步请求在页面中发起,也要在页面中接收数据并展现. B:ajax中的数据不能从作用域中获取,只能从respons ...
- 通俗化理解Spring3 IoC的原理和主要组件
♣什么是IoC? ♣通俗化理解IoC原理 ♣IoC好处 ♣工厂模式 ♣IoC的主要组件 ♣IoC的应用实例 ♣附:实例代码 1.什么是IoC(控制反转)? Spring3框架的核心是实现控制反转( ...
- JFinal(1)JFinal helloworld
** java的极速开放框架:Final 是基于 Java 语言的极速 WEB + ORM 框架,其核心设计目标是开发迅速.代码量少.学习简单.功能强大.轻量级.易扩展.Restful MVC架构,设 ...