执行SQL查询导致磁盘耗尽故障演示

a fellow in IMG wechat group 2 met an error about running out of disk space when using MySQL query today,now let's mimic the situation:

first,i'll create a test table and a procedure,then using the procedure to insert 1000W records into test table "test":

(root@localhost mysql3306.sock)[zlm]08:46:36>create table test(

    -> id int unsigned not null,  --Notice,there is no primary key and any other key here.

    -> name char(50) not null default ''

    -> ) engine=innodb charset=utf8;

Query OK, 0 rows affected (0.01 sec)

(root@localhost mysql3306.sock)[zlm]08:46:45>delimiter $$

(root@localhost mysql3306.sock)[zlm]08:46:54>create procedure pro_test()

    -> begin

    -> declare i int;

    -> set i=10000000;

    -> while i>0 do

    -> insert into test(id,name) values (i,'aaron8219');

    -> set i=i-1;

    -> end while;

    -> end $$

Query OK, 0 rows affected (0.00 sec)

(root@localhost mysql3306.sock)[zlm]08:46:54>delimiter ;

(root@localhost mysql3306.sock)[zlm]08:47:13>call pro_test;

^C^C -- query aborted  --I'm afraid the disk space will be insufficient,so i cancel the procedure by Ctrl+C.

Terminated

it's the incremental change result of the command "df -h" output below at the begining to the point that i cancel insertation:

 [root@zlm3 08:47:20 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.1G  3.4G  60% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:47:22 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.8G  2.6G  70% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:51:27 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  6.2G  2.3G  74% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   76G   11G  88% /vagrant

 [root@zlm3 08:53:06 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.0G  1.4G  84% /  -- The free disk space became 16% (maybe less) when i cancel the operation.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   77G  9.6G  89% /vagrant

 [root@zlm3 08:57:30 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:05 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6182  6157  0 08:46 pts/0    00:00:00 mysql

 root      6389  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:13 /data/mysql/mysql3306]

 #kill 6182  -- After cancel the operation,i kill the mysql process in this session.

 [root@zlm3 08:58:22 /data/mysql/mysql3306]

 #ps -ef|grep mysql

 mysql     6031     1 12 08:00 ?        00:07:17 mysqld --defaults-file=/data/mysql/mysql3306/my.cnf

 root      6403  6085  0 08:58 pts/1    00:00:00 grep --color=auto mysql

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #

in the first session,relogin with mysql client,check the status of the table test:

 #mysql

 Welcome to the MySQL monitor.  Commands end with ; or \g.

 Your MySQL connection id is 7

 Server version: 5.7.21-log MySQL Community Server (GPL)

 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

 Oracle is a registered trademark of Oracle Corporation and/or its

 affiliates. Other names may be trademarks of their respective

 owners.

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 (root@localhost mysql3306.sock)[(none)]08:58:35>use zlm;

 Reading table information for completion of table and column names

 You can turn off this feature to get a quicker startup with -A

 Database changed

 (root@localhost mysql3306.sock)[zlm]08:59:08>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 | test          |

 +---------------+

 4 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]09:00:49>select table_schema,concat(truncate(sum(data_length)/1024/1024,2),'MB') as data_size,

     -> concat(truncate(sum(index_length)/1024/1024,2),'MB') as index_size

     -> from information_schema.tables

     -> where table_schema='zlm' and table_name='test';

 +--------------+-----------+------------+

 | table_schema | data_size | index_size |

 +--------------+-----------+------------+

 | zlm          | 656.00MB  | 0.00MB     |  --Now the table "test" has alread been a big table,the size turned into 656Mb.

 +--------------+-----------+------------+

 1 row in set (0.10 sec)

 (root@localhost mysql3306.sock)[zlm]09:01:34>select count(*) from test;

 +----------+

 | count(*) |

 +----------+

 |  9070823 | --Almost 1000W records around.

 +----------+

 1 row in set (19.16 sec)

 (root@localhost mysql3306.sock)[zlm]09:02:29>show master status;

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | File             | Position  | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                               |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 | mysql-bin.000057 | 244082592 |              |                  | 5c77c31b-4add-11e8-81e2-080027de0e0e:1-13527727 |

 +------------------+-----------+--------------+------------------+-------------------------------------------------+

 1 row in set (0.06 sec)

 (root@localhost mysql3306.sock)[zlm]09:12:58>

let's check the binlog file,these files really ocuppied a huge amount disk space:

[root@zlm3 08:58:42 /data/mysql/mysql3306/logs]

#ls -l

total 2384300

-rw-r----- 1 mysql mysql      3831 May 28 03:20 mysql-bin.000019

-rw-r----- 1 mysql mysql      9564 May 28 11:37 mysql-bin.000020

-rw-r----- 1 mysql mysql      4761 May 29 11:27 mysql-bin.000021

-rw-r----- 1 mysql mysql       217 May 30 11:29 mysql-bin.000022

-rw-r----- 1 mysql mysql       217 May 31 03:20 mysql-bin.000023

-rw-r----- 1 mysql mysql       613 May 31 03:29 mysql-bin.000024

-rw-r----- 1 mysql mysql      1009 May 31 03:35 mysql-bin.000025

-rw-r----- 1 mysql mysql       217 May 31 03:36 mysql-bin.000026

-rw-r----- 1 mysql mysql       217 May 31 03:37 mysql-bin.000027

-rw-r----- 1 mysql mysql       217 May 31 03:38 mysql-bin.000028

-rw-r----- 1 mysql mysql       217 May 31 03:40 mysql-bin.000029

-rw-r----- 1 mysql mysql      1563 May 31 03:45 mysql-bin.000030

-rw-r----- 1 mysql mysql       217 May 31 06:50 mysql-bin.000031

-rw-r----- 1 mysql mysql       217 May 31 06:59 mysql-bin.000032

-rw-r----- 1 mysql mysql       217 May 31 07:04 mysql-bin.000033

-rw-r----- 1 mysql mysql       217 May 31 07:13 mysql-bin.000034

-rw-r----- 1 mysql mysql       217 May 31 07:15 mysql-bin.000035

-rw-r----- 1 mysql mysql       217 May 31 07:42 mysql-bin.000036

-rw-r----- 1 mysql mysql       461 May 31 08:22 mysql-bin.000037

-rw-r----- 1 mysql mysql       217 May 31 08:25 mysql-bin.000038

-rw-r----- 1 mysql mysql       613 May 31 10:37 mysql-bin.000039

-rw-r----- 1 mysql mysql       369 May 31 10:41 mysql-bin.000040

-rw-r----- 1 mysql mysql       613 May 31 11:28 mysql-bin.000041

-rw-r----- 1 mysql mysql      3141 Jun  1 10:10 mysql-bin.000042

-rw-r----- 1 mysql mysql      5677 Jun  1 11:38 mysql-bin.000043

-rw-r----- 1 mysql mysql       217 Jun  4 07:54 mysql-bin.000044

-rw-r----- 1 mysql mysql       194 Jun  4 07:57 mysql-bin.000045

-rw-r----- 1 mysql mysql       217 Jun  4 07:57 mysql-bin.000046

-rw-r----- 1 mysql mysql       217 Jun  4 11:23 mysql-bin.000047

-rw-r----- 1 mysql mysql 268435609 Jun  5 08:48 mysql-bin.000048

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:50 mysql-bin.000049

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:52 mysql-bin.000050

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:54 mysql-bin.000051

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:55 mysql-bin.000052

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:57 mysql-bin.000053

-rw-r----- 1 mysql mysql 268435737 Jun  5 08:58 mysql-bin.000054

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:00 mysql-bin.000055

-rw-r----- 1 mysql mysql 268435737 Jun  5 09:02 mysql-bin.000056  --From binlog 48 to 56,each one exhausted the max size at 256M.

-rw-r----- 1 mysql mysql  25366220 Jun  5 09:02 mysql-bin.000057  --This is the last binlog file contain the last several records.

-rw-r----- 1 mysql mysql      1716 Jun  5 09:02 mysql-bin.index

[root@zlm3 09:09:43 /data/mysql/mysql3306/logs]

#mysqlbinlog -vv --base64-output=decode-rows mysql-bin.000057| tail -20

#180605  9:03:45 server id 1023306  end_log_pos 244082462 CRC32 0x9db85944     Query    thread_id=5    exec_time=0    error_code=0

SET TIMESTAMP=1528182225/*!*/;

BEGIN

/*!*/;

# at 244082462

#180605  9:03:45 server id 1023306  end_log_pos 244082511 CRC32 0x8bb85bae     Table_map: `zlm`.`test` mapped to number 104

# at 244082511

#180605  9:03:45 server id 1023306  end_log_pos 244082561 CRC32 0x6ede8301     Write_rows: table id 104 flags: STMT_END_F

### INSERT INTO `zlm`.`test`

### SET

###   @1=1 /* INT meta=0 nullable=0 is_null=0 */

###   @2='aaron8219' /* STRING(150) meta=65174 nullable=0 is_null=0 */

# at 244082561

#180605  9:03:45 server id 1023306  end_log_pos 244082592 CRC32 0xf983b21b     Xid = 30000069  --The "end_log_pos 244082592" is equal with the output result of the postion in "show master status;" command.

COMMIT/*!*/;

SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; --GTID_NEXT='AUTOMATIC' means there're no more sequential binlogs behind.

DELIMITER ;

# End of log file

/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;

/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

[root@zlm3 09:11:21 /data/mysql/mysql3306/logs]

#

now,let's do some testing query below(using distinct/group by):

 (root@localhost mysql3306.sock)[zlm]09:16:31>select count(distinct(id)) from test group by name;

 ERROR 3 (HY000): Error writing file '/data/mysql/mysql3306/tmp/MYCUMGkU' (Errcode: 28 - No space left on device)

 (root@localhost mysql3306.sock)[zlm]09:18:34>system ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6667  0.0  0.1 113116  1360 pts/0    S+   09:17   0:00 sh -c  ps aux|grep mysql

 root      6669  0.0  0.0 112640   944 pts/0    R+   09:17   0:00 grep mysql

 (root@localhost mysql3306.sock)[zlm]10:22:13>

eventually,the query ended with error "Errcode:28 - no space left on device",obviously the mysqld process has dead now.check the disk space,it's 99% in Use% column,why does the disk space continuously increase?let's see the official document about this bellow:

8.4.4 Internal Temporary Table Use in MySQL

In some cases, the server creates internal temporary tables while processing statements. Users have no direct control over when this occurs.

The server creates temporary tables under conditions such as these:

Evaluation of UNION statements, with some exceptions described later.
Evaluation of some views, such those that use the TEMPTABLE algorithm, UNION, or aggregation.
Evaluation of derived tables (see Section 13.2.11.8, “Derived Tables”).
Evaluation of common table expressions (see Section 13.2.13, “WITH Syntax (Common Table Expressions)”).
Tables created for subquery or semi-join materialization (see Section 8.2.2, “Optimizing Subqueries, Derived Tables, View References, and Common Table Expressions”).
Evaluation of statements that contain an ORDER BY clause and a different GROUP BY clause, or for which the ORDER BY or GROUP BY contains columns from tables other than the first table in the join queue.
Evaluation of DISTINCT combined with ORDER BY may require a temporary table.
For queries that use the SQL_SMALL_RESULT modifier, MySQL uses an in-memory temporary table, unless the query also contains elements (described later) that require on-disk storage.
To evaluate INSERT ... SELECT statements that select from and insert into the same table, MySQL creates an internal temporary table to hold the rows from the SELECT, then inserts those rows into the target table. See Section 13.2.6.1, “INSERT ... SELECT Syntax”.
Evaluation of multiple-table UPDATE statements.
Evaluation of GROUP_CONCAT() or COUNT(DISTINCT) expressions.

来源： https://dev.mysql.com/doc/refman/8.0/en/internal-temporary-tables.html

here's the conclusion,whenever above conditions occured,it will induce MySQL server to do sort operation in memory or in disk(this is too bad thing),hence,in our test above,the disk space was exhausted soon.

 [root@zlm3 08:58:29 /data/mysql/mysql3306]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  122M  99% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 09:19:20 /data/mysql/mysql3306]

 #du -sh /data

 3.8G    /data --Even though the "df -h" shows that it's out of space already,but the "/data/" dir still don't increase at all.

 [root@zlm3 09:20:21 /data/mysql/mysql3306]

 #mysqladmin var|grep -i tmp --Check the parameter about key word "tmp".

 | default_tmp_storage_engine                               | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | innodb_temp_data_file_path                               | ibtmp1:12M:autoextend                                                                                                                                                                                                                                                                                                                                                                                            |

 | innodb_tmpdir                                            |                                                                                                                                                                                                                                                                                                                                                                                                                  |

 | internal_tmp_disk_storage_engine                         | InnoDB                                                                                                                                                                                                                                                                                                                                                                                                           |

 | max_tmp_tables                                           | 32                                                                                                                                                                                                                                                                                                                                                                                                               |

 | slave_load_tmpdir                                        | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 | socket                                                   | /tmp/mysql3306.sock                                                                                                                                                                                                                                                                                                                                                                                              |

 | tmp_table_size                                           | 100663296                                                                                                                                                                                                                                                                                                                                                                                                        |

 | tmpdir                                                   | /data/mysql/mysql3306/tmp                                                                                                                                                                                                                                                                                                                                                                                        |

 [root@zlm3 09:22:54 /data/mysql/mysql3306/tmp]

 #ls -l --No files here at all,why?'cause it has been deleted,see below:

 total 0

 [root@zlm3 09:22:56 /data/mysql/mysql3306/tmp]

 #lsof -n|grep deleted --List open files using "lsof" command.there're lots of deleted tmp file used by mysqld process

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 mysqld    6031        mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031        mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031        mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031        mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6036   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6036   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6036   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6036   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6037   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6037   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6037   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6037   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6038   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6038   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6038   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6038   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6039   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6039   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6039   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6039   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6040   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6040   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6040   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6040   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6041   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6041   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6041   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6041   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6042   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6042   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6042   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6042   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6043   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6043   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6043   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6043   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6044   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6044   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6044   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6044   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6045   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6045   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6045   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6045   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6046   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6046   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6046   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6046   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6047   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6047   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6047   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6047   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6049   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6049   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6049   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6049   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6050   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6050   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6050   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6050   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6051   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6051   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6051   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6051   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6052   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6052   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6052   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6052   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6053   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6053   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6053   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6053   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6054   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6054   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6054   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6054   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6055   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6055   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6055   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6055   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6056   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6056   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6056   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6056   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6057   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6057   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6057   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6057   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6058   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6058   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6058   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6058   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6059   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6059   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6059   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6059   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6060   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6060   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6060   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6060   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6061   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6061   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6061   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6061   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6062   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6062   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6062   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6062   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6064   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6064   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6064   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6064   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 mysqld    6031 6387   mysql    6u      REG              253,0         0   33919399 /data/mysql/mysql3306/tmp/ibuwlBKV (deleted)

 mysqld    6031 6387   mysql    7u      REG              253,0         0   33919400 /data/mysql/mysql3306/tmp/ibhJgkTN (deleted)

 mysqld    6031 6387   mysql    8u      REG              253,0         0   33919402 /data/mysql/mysql3306/tmp/ibrjB3fy (deleted)

 mysqld    6031 6387   mysql   13u      REG              253,0         0   33919403 /data/mysql/mysql3306/tmp/ibEqcSBq (deleted)

 [root@zlm3 09:24:32 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted | awk -F ' ' '{printf "kill -9 %s\n" $2}' > /root/free_handle_space.sh

 awk: cmd. line:1: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy format string

     `kill -9 %s

 566'

              ^ ran out for this one --Is this a gramma mistake here?i've been tested before and it's ok,should it be the lack of disk space?

 [root@zlm3 09:53:07 /data/mysql/mysql3306/tmp]

 #

the process id here is 6031,i supposed that i can kill the process to release the handler of files and get back the disk space:

 [root@zlm3 09:56:06 /data/mysql/mysql3306/tmp]

 #kill 6031

 [root@zlm3 10:13:42 /data/mysql/mysql3306/tmp]

 #lsof -n | grep deleted  --The lots of deleted tmp file has gone now.

 tuned      566         root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 gmain      566  719    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  721    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  723    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 tuned      566  728    root  txt       REG              253,0      7136   17004368 /usr/bin/python2.7;5b15ee3d (deleted)

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #

what shocked me is that the disk space was not released after my killing operation:

 [root@zlm3 10:14:17 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Still 99% in Use% cloumn

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:15:25 ~]

 #

what's wrong with it?what if i kill the mysql process?

 [root@zlm3 10:14:06 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6420  0.0  0.2 134112  2292 pts/0    S+   08:58   0:00 mysql

 root      6656  0.0  0.0 112640   960 pts/1    R+   10:21   0:00 grep --color=auto mysql

 [root@zlm3 10:21:31 /data/mysql/mysql3306/tmp]

 #kill 6420

 [root@zlm3 10:29:39 /data/mysql/mysql3306/tmp]

 #ps aux|grep mysql

 root      6679  0.0  0.0 112640   960 pts/1    R+   10:29   0:00 grep --color=auto mysql

 [root@zlm3 10:29:49 /data/mysql/mysql3306/tmp]

 #

nothing changed,the disk space is still full:

 [root@zlm3 10:29:47 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  8.3G  134M  99% /  --Change nothing,99% again.

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:29:55 ~]

 #

after i drop the table "test",it just release 10% disk sapce,where has the free space gone?

 [root@zlm3 10:29:55 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  7.5G  962M  89% /

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:34:16 ~]

 #

then,i use "reset master;",the dike space finally came back:

 (root@localhost mysql3306.sock)[zlm]10:34:10>reset master;  --It's utterly forbidden to use this command on product MySQL server,take care.

 Query OK, 0 rows affected (0.08 sec)

 (root@localhost mysql3306.sock)[zlm]10:41:44>show tables;

 +---------------+

 | Tables_in_zlm |

 +---------------+

 | t1            |

 | t2            |

 | t3            |

 +---------------+

 3 rows in set (0.00 sec)

 (root@localhost mysql3306.sock)[zlm]10:42:00>

 [root@zlm3 10:34:16 ~]

 #df -h

 Filesystem               Size  Used Avail Use% Mounted on

 /dev/mapper/centos-root  8.4G  5.0G  3.5G  60% /  --Now,all the space came back(before the insertation).

 devtmpfs                 488M     0  488M   0% /dev

 tmpfs                    497M     0  497M   0% /dev/shm

 tmpfs                    497M  6.6M  491M   2% /run

 tmpfs                    497M     0  497M   0% /sys/fs/cgroup

 /dev/sda1                497M  118M  379M  24% /boot

 none                      87G   78G  8.3G  91% /vagrant

 [root@zlm3 10:41:50 ~]

 #cd /data/mysql/mysql3306/logs

 [root@zlm3 10:47:49 /data/mysql/mysql3306/logs]

 #ls -l

 total 8

 -rw-r----- 1 mysql mysql 154 Jun  5 10:41 mysql-bin.000001

 -rw-r----- 1 mysql mysql  44 Jun  5 10:41 mysql-bin.index

 [root@zlm3 10:47:50 /data/mysql/mysql3306/logs]

 #

Summary:

the test what i made just now was not so good to explain what i wanna say,perhaps i didn't use the accurate sample data.
generally speaking,the file handler is the common reason why the deletion operation doesn't release disk space.
operations such as "distinct","group by" in MySQL query will use temperory table to sort on disk,it depend on the several parameter("max_tmp_tables","max_heap_table_size" and etc.)you've set in config file "my.cnf".
not using index will lead to huge amount of sort operation which will generate poor performance of MySQL server.
before you kill the process to release the disk space,do think twice that the deleted file is really unnecessary any more.

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

EF5中执行 sql语句使用Database.ExecuteSqlCommand 返回影响的行数 ; EF5执行sql查询语句 Database.SqlQuery 带返回值
一: 执行sql语句,返回受影响的行数在mysql里面,如果没有影响,那么返回行数为 -1 ,sqlserver 里面还没有测试过 using (var ctx = new MyDbConte ...
Oracle执行SQL查询语句的步骤
Oracle执行SQL查询语句的步骤如果用户在SQL*Plus下输入了如下查询语句:SELECT * FROM dept: 查询语句的处理主要包括三个过程:编译(parse).执行(execute) ...
oracle执行sql查询语句出现错误ORA-00942：表或视图不存在
情况是这样,A库的用户名和表空间分别为SH , SH 把业务表SH所有数据从A库,导入到B库, 表空间为SH,用户名为SP 在B库里面执行sql查询语句出现错误ORA-00942:表或视图不存在语句 ...
ado执行sql查询出现“发送数据流时出现算术溢出”错误
开发一个数据采集监控系统,比较变态的是有将近2000项数据.根据数据类型分多个表存储.数据库访问层采用ado.最近发现当一条sql一次性查询1700多个字段数据后就出现“发送数据流时出现算术溢出”错误 ...
MySQL数据库详解（一）执行SQL查询语句时,其底层到底经历了什么?
一条SQL查询语句是如何执行的? 前言大家好,我是WZY,今天我们学习下MySQL的基础框架,看一件事千万不要直接陷入细节里,你应该先鸟瞰其全貌,这样能够帮助你从高维度理解问题.同样,对于MyS ...
python MySQL执行SQL查询结果返回字典
写自动化测试的时候我希望执行数据库前置任务,把数据库查询的结果作为请求的参数,但是正常返回结果为列表嵌套里面,这样就会影响到关键字准确的获取,特别的受限于SQL的查询字段的的顺序,所以希望返回的单条数 ...
MySQL执行sql查询并上传至远程服务器
最近项目中有需要做一个shell脚本,可以对一个数据库执行sql操作,并将结果转为txt,筛选结果用tab隔开,保存至一个远程服务器上,以供其他人用Excel读取用txt中的内容. MySQL中将结果 ...
Oracle 在函数或存储过程中执行sql查询字符串并将结果值赋值给变量
请看黄色部分 --区县指标 THEN TVALUE_SQL := 'SELECT TO_CHAR(' || CUR_ROW.MAIN_FIELD || ') FROM ' || CUR_ROW.END ...
HBase 查询导致RegionServer OOM故障复盘
背景:我司作为某运营商公司的技术咨询公司,发现有第三方开发公司在使用HBase 1.1.2 (HDP 2.4.2.258版本)一段时间使用正常后,从某一天开始报OOM,从而导致RegionServer ...

随机推荐

关于 ie9 不执行 js 的问题
当写js用 console.log()调试,但没有注释,在ie 9 浏览器上运行时,console.log() 后面的脚本不能再执行了. 为了ie,记得把console.log() 注释
Django—XSS及CSRF
一.XSS 跨站脚本攻击(Cross Site Scripting),为不和层叠样式表(Cascading Style Sheets, CSS)的缩写混淆,故将跨站脚本攻击缩写为XSS.恶意攻击者往W ...
Android蓝牙自动配对Demo，亲测好使！！！
蓝牙自动配对,即搜索到其它蓝牙设备之后直接进行配对,不需要弹出配对确认框或者密钥输入框. 转载请注明出处http://blog.csdn.net/qq_25827845/article/details ...
MySQL数据库(1)----入门级操作
1.在服务器主机上以 root 用户登陆,创建位于其他客户端的新用户: mysql> CREATE USER 'newuser'@'192.168.1.109' IDENTIFIED BY 'p ...
Android Timer和TimerTask
以下内容根据 The JavaTM Tutorial 和相关API doc翻译整理,以供日后参考: 1.概览 Timer是一种定时器工具,用来在一个后台线程计划执行指定任务.它可以计划执行一个任务一次 ...
Page Cache的落地问题
除非特别说明,否则本文提到的写操作都是 buffer write/write back. 起因前几天讨论到一个问题:Linux 下文件 close成功,会不会触发 “刷盘”? 其实这个问题根本不用讨 ...
Redhat Linux 7.3 虚拟机通过USB挂载NTFS格式的移动硬盘
分为如下几个步骤: 一.设置本地yum,安装gcc(如果本机已经安装gcc,则跳过此步) 在虚拟机连接linux iso安装盘查看光盘挂载情况 mkdir /iso mount /dev/cdrom ...
BASE64编码的字符进行URL传输丢失特殊字符的问题
因为BASE64的编码里含有“+”号等特殊字符,在url传输的时候会把+号编程空格,解决这个问题的方法: 请求时把BASE64编码进行url的编码再进行传输接收时把BASE64编码进行url的解码 ...
SQL点点滴滴_删除临时表
select into 创建的表属于临时表,判断是否存在的方法 select c_adno,c_con_no into #temp from tb_contract IF OBJECT_ID( 'te ...
Java接口与多态
接口可以理解为一种特殊的类,里面全部是由全局常量(static final)和公共的抽象方法所组成接口的定义格式接口的数据成员,只允许被public, static, final修饰. 接口的方 ...

执行SQL查询导致磁盘耗尽故障演示

8.4.4 Internal Temporary Table Use in MySQL

执行SQL查询导致磁盘耗尽故障演示的更多相关文章

随机推荐

热门专题