执行SQL查询导致磁盘耗尽故障演示

a fellow in IMG wechat group 2 met an error about running out of disk space when using MySQL query today,now let's mimic the situation:

first,i'll create a test table and a procedure,then using the procedure to insert 1000W records into test table "test":

(root@localhost mysql3306.sock)[zlm]08:46:36>create table test(

    -> id int unsigned not null,  --Notice,there is no primary key and any other key here.

    -> name char(50) not null default ''

    -> ) engine=innodb charset=utf8;

Query OK, 0 rows affected (0.01 sec)

(root@localhost mysql3306.sock)[zlm]08:46:45>delimiter $$

(root@localhost mysql3306.sock)[zlm]08:46:54>create procedure pro_test()

    -> begin

    -> declare i int;

    -> set i=10000000;

    -> while i>0 do

    -> insert into test(id,name) values (i,'aaron8219');

    -> set i=i-1;

    -> end while;

    -> end $$

Query OK, 0 rows affected (0.00 sec)

(root@localhost mysql3306.sock)[zlm]08:46:54>delimiter ;

(root@localhost mysql3306.sock)[zlm]08:47:13>call pro_test;

^C^C -- query aborted  --I'm afraid the disk space will be insufficient,so i cancel the procedure by Ctrl+C.

Terminated

it's the incremental change result of the command "df -h" output below at the begining to the point that i cancel insertation:

 [root@zlm3 08:47:20 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.1G  3.4G  60% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:47:22 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.8G  2.6G  70% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:51:27 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  6.2G  2.3G  74% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:53:06 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.0G  1.4G  84% /  -- The free disk space became 16% (maybe less) when i cancel the operation.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   77G  9.6G  89% /vagrant

 [root@zlm3 08:57:30 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:05 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6182  6157  0 08:46 pts/0    00:00:00 mysql

 root      6389  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:13 /data/mysql/mysql3306]

 #kill 6182  -- After cancel the operation,i kill the mysql process in this session.

 [root@zlm3 08:58:22 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:17 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6403  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #

in the first session,relogin with mysql client,check the status of the table test:

 #mysql

 Welcome to the MySQL monitor.  Commands end with ; or \g.

 Your MySQL connection id is 7

 Server version: 5.7.21-log MySQL Community Server (GPL)

 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

 Oracle is a registered trademark of Oracle Corporation and/or its

 affiliates. Other names may be trademarks of their respective

 owners.

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 (root@localhost mysql3306.sock)[(none)]08:58:35>use zlm;

 Reading table information for completion of table and column names

 You can turn off this feature to get a quicker startup with -A

 Database changed

 (root@localhost mysql3306.sock)[zlm]08:59:08>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 | test          |

 +---------------+

 4 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]09:00:49>select table_schema,concat(truncate(sum(data_length)/1024/1024,2),'MB') as data_size,

     -> concat(truncate(sum(index_length)/1024/1024,2),'MB') as index_size

     -> from information_schema.tables

     -> where table_schema='zlm' and table_name='test';

 +--------------+-----------+------------+

 | table_schema | data_size | index_size |

 +--------------+-----------+------------+

 | zlm          | 656.00MB  | 0.00MB     |  --Now the table "test" has alread been a big table,the size turned into 656Mb.

 +--------------+-----------+------------+

 1 row in set (0.10 sec)

 (root@localhost mysql3306.sock)[zlm]09:01:34>select count(*) from test;

 +----------+

 | count(*) |

 +----------+

 |  9070823 | --Almost 1000W records around.

 +----------+

 1 row in set (19.16 sec)

 (root@localhost mysql3306.sock)[zlm]09:02:29>show master status;

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | File             | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                               |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | mysql-bin.000057 | 244082592 |              |                  | 5c77c31b-4add-11e8-81e2-080027de0e0e:1-13527727 |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 1 row in set (0.06 sec)

 (root@localhost mysql3306.sock)[zlm]09:12:58>

let's check the binlog file,these files really ocuppied a huge amount disk space:

[root@zlm3 08:58:42 /data/mysql/mysql3306/logs]

#ls -l

total 2384300

-rw-r----- 1 mysql mysql      3831 May 28 03:20 mysql-bin.000019

-rw-r----- 1 mysql mysql      9564 May 28 11:37 mysql-bin.000020

-rw-r----- 1 mysql mysql      4761 May 29 11:27 mysql-bin.000021

-rw-r----- 1 mysql mysql       217 May 30 11:29 mysql-bin.000022

-rw-r----- 1 mysql mysql       217 May 31 03:20 mysql-bin.000023

-rw-r----- 1 mysql mysql       613 May 31 03:29 mysql-bin.000024

-rw-r----- 1 mysql mysql      1009 May 31 03:35 mysql-bin.000025

-rw-r----- 1 mysql mysql       217 May 31 03:36 mysql-bin.000026

-rw-r----- 1 mysql mysql       217 May 31 03:37 mysql-bin.000027

-rw-r----- 1 mysql mysql       217 May 31 03:38 mysql-bin.000028

-rw-r----- 1 mysql mysql       217 May 31 03:40 mysql-bin.000029

-rw-r----- 1 mysql mysql      1563 May 31 03:45 mysql-bin.000030

-rw-r----- 1 mysql mysql       217 May 31 06:50 mysql-bin.000031

-rw-r----- 1 mysql mysql       217 May 31 06:59 mysql-bin.000032

-rw-r----- 1 mysql mysql       217 May 31 07:04 mysql-bin.000033

-rw-r----- 1 mysql mysql       217 May 31 07:13 mysql-bin.000034

-rw-r----- 1 mysql mysql       217 May 31 07:15 mysql-bin.000035

-rw-r----- 1 mysql mysql       217 May 31 07:42 mysql-bin.000036

-rw-r----- 1 mysql mysql       461 May 31 08:22 mysql-bin.000037

-rw-r----- 1 mysql mysql       217 May 31 08:25 mysql-bin.000038

-rw-r----- 1 mysql mysql       613 May 31 10:37 mysql-bin.000039

-rw-r----- 1 mysql mysql       369 May 31 10:41 mysql-bin.000040

-rw-r----- 1 mysql mysql       613 May 31 11:28 mysql-bin.000041

-rw-r----- 1 mysql mysql      3141 Jun  1 10:10 mysql-bin.000042

-rw-r----- 1 mysql mysql      5677 Jun  1 11:38 mysql-bin.000043

-rw-r----- 1 mysql mysql       217 Jun  4 07:54 mysql-bin.000044

-rw-r----- 1 mysql mysql       194 Jun  4 07:57 mysql-bin.000045

-rw-r----- 1 mysql mysql       217 Jun  4 07:57 mysql-bin.000046

-rw-r----- 1 mysql mysql       217 Jun  4 11:23 mysql-bin.000047

-rw-r----- 1 mysql mysql 268435609 Jun  5 08:48 mysql-bin.000048

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:50 mysql-bin.000049

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:52 mysql-bin.000050

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:54 mysql-bin.000051

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:55 mysql-bin.000052

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:57 mysql-bin.000053

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:58 mysql-bin.000054

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:00 mysql-bin.000055

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:02 mysql-bin.000056  --From binlog 48 to 56,each one exhausted the max size at 256M.

-rw-r----- 1 mysql mysql  25366220 Jun  5 09:02 mysql-bin.000057  --This is the last binlog file contain the last several records.

-rw-r----- 1 mysql mysql      1716 Jun  5 09:02 mysql-bin.index

[root@zlm3 09:09:43 /data/mysql/mysql3306/logs]

#mysqlbinlog -vv --base64-output=decode-rows mysql-bin.000057| tail -20

#180605  9:03:45 server id 1023306  end_log_pos 244082462 CRC32 0x9db85944     Query    thread_id=5    exec_time=0    error_code=0

SET TIMESTAMP=1528182225/*!*/;

BEGIN

/*!*/;

# at 244082462

#180605  9:03:45 server id 1023306  end_log_pos 244082511 CRC32 0x8bb85bae     Table_map: `zlm`.`test` mapped to number 104

# at 244082511

#180605  9:03:45 server id 1023306  end_log_pos 244082561 CRC32 0x6ede8301     Write_rows: table id 104 flags: STMT_END_F

### INSERT INTO `zlm`.`test`

### SET

###   @1=1 /* INT meta=0 nullable=0 is_null=0 */

###   @2='aaron8219' /* STRING(150) meta=65174 nullable=0 is_null=0 */

# at 244082561

#180605  9:03:45 server id 1023306  end_log_pos 244082592 CRC32 0xf983b21b     Xid = 30000069  --The "end_log_pos 244082592" is equal with the output result of the postion in "show master status;" command.

COMMIT/*!*/;

SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; --GTID_NEXT='AUTOMATIC' means there're no more sequential binlogs behind.

DELIMITER ;

# End of log file

/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

[root@zlm3 09:11:21 /data/mysql/mysql3306/logs]

#

now,let's do some testing query below(using distinct/group by):

 (root@localhost mysql3306.sock)[zlm]09:16:31>select count(distinct(id)) from test group by name;

 ERROR 3 (HY000): Error writing file '/data/mysql/mysql3306/tmp/MYCUMGkU' (Errcode: 28 - No space left on device)

 (root@localhost mysql3306.sock)[zlm]09:18:34>system ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6667  0.0  0.1 113116  1360 pts/0    S+   09:17   0:00 sh -c  ps aux|grep mysql

 root      6669  0.0  0.0 112640   944 pts/0    R+   09:17   0:00 grep mysql

 (root@localhost mysql3306.sock)[zlm]10:22:13>

eventually,the query ended with error "Errcode:28 - no space left on device",obviously the mysqld process has dead now.check the disk space,it's 99% in Use% column,why does the disk space continuously increase?let's see the official document about this bellow:

8.4.4 Internal Temporary Table Use in MySQL

In some cases, the server creates internal temporary tables while processing statements. Users have no direct control over when this occurs.

The server creates temporary tables under conditions such as these:

Evaluation of UNION statements, with some exceptions described later.
Evaluation of some views, such those that use the TEMPTABLE algorithm, UNION, or aggregation.
Evaluation of derived tables (see Section 13.2.11.8, “Derived Tables”).
Evaluation of common table expressions (see Section 13.2.13, “WITH Syntax (Common Table Expressions)”).
Tables created for subquery or semi-join materialization (see Section 8.2.2, “Optimizing Subqueries, Derived Tables, View References, and Common Table Expressions”).
Evaluation of statements that contain an ORDER BY clause and a different GROUP BY clause, or for which the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue.
Evaluation of DISTINCT combined with ORDER BY may require a temporary table.
For queries that use the SQL_SMALL_RESULT modifier, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.
To evaluate INSERT ... SELECT statements that select from and insert into the same table, MySQL creates an internal temporary table to hold the rows from the SELECT, then inserts those rows into the target table. See Section 13.2.6.1, “INSERT ... SELECT Syntax”.
Evaluation of multiple-table UPDATE statements.
Evaluation of GROUP_CONCAT() or COUNT(DISTINCT) expressions.

来源： https://dev.mysql.com/doc/refman/8.0/en/internal-temporary-tables.html

here's the conclusion,whenever above conditions occured,it will induce MySQL server to do sort operation in memory or in disk(this is too bad thing),hence,in our test above,the disk space was exhausted soon.

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  122M  99% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 09:19:20 /data/mysql/mysql3306]

 #du -sh /data

 3.8G    /data --Even though the "df -h" shows that it's out of space already,but the "/data/" dir still don't increase at all.

 [root@zlm3 09:20:21 /data/mysql/mysql3306]

 #mysqladmin var|grep -i tmp --Check the parameter about key word "tmp".

 | default_tmp_storage_engine                               | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | innodb_temp_data_file_path                               | ibtmp1:12M:autoextend                                                                                                                                                                                                                                                                                                                                                                                            |

 | innodb_tmpdir                                            |                                                                                                                                                                                                                                                                                                                                                                                                                  |

 | internal_tmp_disk_storage_engine                         | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | max_tmp_tables                                           | 32                                                                                                                                                                                                                                                                                                                                                                                                               |

 | slave_load_tmpdir                                        | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 | socket                                                   | /tmp/mysql3306.sock                                                                                                                                                                                                                                                                                                                                                                                              |

 | tmp_table_size                                           | 100663296                                                                                                                                                                                                                                                                                                                                                                                                        |

 | tmpdir                                                   | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 [root@zlm3 09:22:54 /data/mysql/mysql3306/tmp]

 #ls -l --No files here at all,why?'cause it has been deleted,see below:

 total 0

 [root@zlm3 09:22:56 /data/mysql/mysql3306/tmp]

 #lsof -n|grep deleted --List open files using "lsof" command.there're lots of deleted tmp file used by mysqld process

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 mysqld    6031        mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031        mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031        mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031        mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6036   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6036   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6036   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6036   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6037   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6037   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6037   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6037   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6038   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6038   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6038   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6038   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6039   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6039   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6039   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6039   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6040   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6040   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6040   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6040   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6041   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6041   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6041   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6041   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6042   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6042   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6042   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6042   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6043   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6043   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6043   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6043   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6044   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6044   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6044   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6044   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6045   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6045   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6045   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6045   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6046   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6046   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6046   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6046   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6047   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6047   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6047   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6047   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6049   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6049   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6049   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6049   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6050   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6050   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6050   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6050   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6051   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6051   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6051   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6051   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6052   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6052   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6052   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6052   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6053   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6053   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6053   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6053   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6054   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6054   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6054   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6054   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6055   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6055   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6055   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6055   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6056   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6056   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6056   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6056   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6057   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6057   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6057   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6057   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6058   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6058   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6058   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6058   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6059   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6059   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6059   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6059   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6060   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6060   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6060   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6060   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6061   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6061   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6061   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6061   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6062   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6062   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6062   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6062   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6064   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6064   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6064   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6064   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6387   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6387   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6387   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6387   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 [root@zlm3 09:24:32 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted | awk -F ' ' '{printf "kill -9 %s\n" $2}' > /root/free_handle_space.sh

 awk: cmd. line:1: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy format string

     `kill -9 %s

 566'

              ^ ran out for this one --Is this a gramma mistake here?i've been tested before and it's ok,should it be the lack of disk space?

 [root@zlm3 09:53:07 /data/mysql/mysql3306/tmp]

 #

the process id here is 6031,i supposed that i can kill the process to release the handler of files and get back the disk space:

 [root@zlm3 09:56:06 /data/mysql/mysql3306/tmp]

 #kill 6031

 [root@zlm3 10:13:42 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted  --The lots of deleted tmp file has gone now.

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #

what shocked me is that the disk space was not released after my killing operation:

 [root@zlm3 10:14:17 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Still 99% in Use% cloumn

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:15:25 ~]

 #

what's wrong with it?what if i kill the mysql process?

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6656  0.0  0.0 112640   960 pts/1    R+   10:21   0:00 grep --color=auto mysql

 [root@zlm3 10:21:31 /data/mysql/mysql3306/tmp]

 #kill 6420

 [root@zlm3 10:29:39 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6679  0.0  0.0 112640   960 pts/1    R+   10:29   0:00 grep --color=auto mysql

 [root@zlm3 10:29:49 /data/mysql/mysql3306/tmp]

 #

nothing changed,the disk space is still full:

 [root@zlm3 10:29:47 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Change nothing,99% again.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:29:55 ~]

 #

after i drop the table "test",it just release 10% disk sapce,where has the free space gone?

 [root@zlm3 10:29:55 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.5G  962M  89% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:34:16 ~]

 #

then,i use "reset master;",the dike space finally came back:

 (root@localhost mysql3306.sock)[zlm]10:34:10>reset master;  --It's utterly forbidden to use this command on product MySQL server,take care.

 Query OK, 0 rows affected (0.08 sec)

 (root@localhost mysql3306.sock)[zlm]10:41:44>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 +---------------+

 3 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]10:42:00>

 [root@zlm3 10:34:16 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.0G  3.5G  60% /  --Now,all the space came back(before the insertation).

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:41:50 ~]

 #cd /data/mysql/mysql3306/logs

 [root@zlm3 10:47:49 /data/mysql/mysql3306/logs]

 #ls -l

 total 8

 -rw-r----- 1 mysql mysql 154 Jun  5 10:41 mysql-bin.000001

 -rw-r----- 1 mysql mysql  44 Jun  5 10:41 mysql-bin.index

 [root@zlm3 10:47:50 /data/mysql/mysql3306/logs]

 #

Summary:

the test what i made just now was not so good to explain what i wanna say,perhaps i didn't use the accurate sample data.
generally speaking,the file handler is the common reason why the deletion operation doesn't release disk space.
operations such as "distinct","group by" in MySQL query will use temperory table to sort on disk,it depend on the several parameter("max_tmp_tables","max_heap_table_size" and etc.)you've set in config file "my.cnf".
not using index will lead to huge amount of sort operation which will generate poor performance of MySQL server.
before you kill the process to release the disk space,do think twice that the deleted file is really unnecessary any more.

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

EF5中执行 sql语句使用Database.ExecuteSqlCommand 返回影响的行数 ; EF5执行sql查询语句 Database.SqlQuery 带返回值
一: 执行sql语句,返回受影响的行数在mysql里面,如果没有影响,那么返回行数为 -1 ,sqlserver 里面还没有测试过 using (var ctx = new MyDbConte ...
Oracle执行SQL查询语句的步骤
Oracle执行SQL查询语句的步骤如果用户在SQL*Plus下输入了如下查询语句:SELECT * FROM dept: 查询语句的处理主要包括三个过程:编译(parse).执行(execute) ...
oracle执行sql查询语句出现错误ORA-00942：表或视图不存在
情况是这样,A库的用户名和表空间分别为SH , SH 把业务表SH所有数据从A库,导入到B库, 表空间为SH,用户名为SP 在B库里面执行sql查询语句出现错误ORA-00942:表或视图不存在语句 ...
ado执行sql查询出现“发送数据流时出现算术溢出”错误
开发一个数据采集监控系统,比较变态的是有将近2000项数据.根据数据类型分多个表存储.数据库访问层采用ado.最近发现当一条sql一次性查询1700多个字段数据后就出现“发送数据流时出现算术溢出”错误 ...
MySQL数据库详解（一）执行SQL查询语句时,其底层到底经历了什么?
一条SQL查询语句是如何执行的? 前言大家好,我是WZY,今天我们学习下MySQL的基础框架,看一件事千万不要直接陷入细节里,你应该先鸟瞰其全貌,这样能够帮助你从高维度理解问题.同样,对于MyS ...
python MySQL执行SQL查询结果返回字典
写自动化测试的时候我希望执行数据库前置任务,把数据库查询的结果作为请求的参数,但是正常返回结果为列表嵌套里面,这样就会影响到关键字准确的获取,特别的受限于SQL的查询字段的的顺序,所以希望返回的单条数 ...
MySQL执行sql查询并上传至远程服务器
最近项目中有需要做一个shell脚本,可以对一个数据库执行sql操作,并将结果转为txt,筛选结果用tab隔开,保存至一个远程服务器上,以供其他人用Excel读取用txt中的内容. MySQL中将结果 ...
Oracle 在函数或存储过程中执行sql查询字符串并将结果值赋值给变量
请看黄色部分 --区县指标 THEN TVALUE_SQL := 'SELECT TO_CHAR(' || CUR_ROW.MAIN_FIELD || ') FROM ' || CUR_ROW.END ...
HBase 查询导致RegionServer OOM故障复盘
背景:我司作为某运营商公司的技术咨询公司,发现有第三方开发公司在使用HBase 1.1.2 (HDP 2.4.2.258版本)一段时间使用正常后,从某一天开始报OOM,从而导致RegionServer ...

随机推荐

Windows核心编程(第5版)----关闭内核对象
无论怎样创建内核对象,都要向系统指明将通过调用 CloseHandle 来结束对该对象的操作: BOOL CloseHandle(HANDLE hobj); 该函数首先检查调用进程的句柄表,以确保传递 ...
VC添加文件到工程出错问题--FileTool.dll
原文:http://blog.csdn.net/bingdianlanxin/article/details/45112737 在我们的软件开发中,经常需要导入其他文件到我们的工程. 一般,我们会选择 ...
python SyntaxError: EOL while scanning string literal
错误原因是,字符串以 \ 结尾或者字符串缺少引号. 写代码拼接windows 路径出现这个错误, 查资料才知道 python中字符串不能以 \ 结尾我的代码如下 import os dirname ...
Stack vs Heap
http://gribblelab.org/CBootcamp/7_Memory_Stack_vs_Heap.html Table of Contents Stack vs Heap The Stac ...
【转】关于色彩空间sRGB和Adobe RGB...
以前对这个问题一直很困惑.... 下面采用问答形式.A:相机中的SRGB和ADOBE RGB有什么区别 ADOBE貌似颜色淡.什么时候有用处呢? B:不是这样的,ADOBE RGB的色域要比SRGB要 ...
【Leetcode】【Medium】Unique Binary Search Trees
Given n, how many structurally unique BST's (binary search trees) that store values 1...n? For examp ...
Yii正则验证
required : 必须值验证属性 [['字段名'],required,'requiredValue'=>'必填值','message'=>'提示信息']; #说明:CRequiredV ...
ignore_user_abort函数制定计划任务
PHP中的ignore_user_abort函数是当用户关掉终端后脚本不停止仍然在执行,可以用它来实现计划任务与持续进程,下面会通过实例讨论ignore_user_abort()函数的作用与用法. ...
January 16 2017 Week 3 Monday
In love, folly is always sweet. 恋爱中,干傻事总是让人感到十分美妙. Love can easily get us in over our heads, so it i ...
看了xici有写给孩子的信，maybe我也要写给孩子一些东西了
看了xici有写给孩子的信,maybe我也要写给孩子一些东西了

执行SQL查询导致磁盘耗尽故障演示

8.4.4 Internal Temporary Table Use in MySQL

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

随机推荐

热门专题