Recovering an InnoDB table from only an .ibd file.

Sometime you may need to recover a table when all you have is the .ibd file. In this case, if you try to load it into a new instance, your likely to encounter some errors about the table id not matching. And there is not really a way around this.

However, I’ve found two work-arounds for this:

Note: You will need the .ibd file and the CREATE TABLE statement for each table you want to recover using these methods.

  1. Simulate the internal InnoDB table counter. That is, create work tables (with innodb_file_per_table enabled) until you have the internal pointer of table id equal to (1 – id_of_ibd_table_you_need_to_restore). (See Method #1)
  2. Manually hex edit the .ibd file, changing the table id. (See Method #2)

*Note: There are some internal structures with this meta information, so you’ll need to dump/import that data after you get it loaded, so you avoid unpleasantries that will inevitably show their face.

Method #1 – Create work tables

1. Start up clean/fresh instance of MySQL with innodb_file_per_table enabled.

2. Now, we need to find the table id that MySQL is currently set at, as well as the table id for the table we need to recover.

Note:
Step 2 (2a – 2f) is simply to find the table id that is stored inside of the .ibd file. I’ve written a PHP script to determine this, so using the script can save a bunch of time. See the bottom of this page (under “Associated Files”) for the exact script.

2a. Create a test database:

mysql> CREATE DATABASE test1;
mysql> USE test1;

2b. Issue the create table command for the table:

mysql> CREATE TABLE `product` (
`PRODUCT_ID` bigint(20) unsigned NOT NULL auto_increment,
`BRAND_ID` int(10) unsigned default NULL,
`PRODUCT_TYPE_ID` int(10) unsigned default NULL,
`GROUP_ID` int(10) unsigned default NULL,
`PRODUCT_NAME` varchar(500) NOT NULL,
`DEFAULT_EMAIL_ID` varchar(48) default NULL,
`PRODUCT_STATUS` tinyint(1) NOT NULL,
`CLIENT_ID` bigint(20) unsigned default NULL,
`LAST_MODIFIED_BY` varchar(45) NOT NULL,
`LAST_MODIFIED_DATE` datetime NOT NULL,
PRIMARY KEY (`PRODUCT_ID`)
) ENGINE=InnoDB;

2c. Discard the tablespace, which will delete the newly created .ibd file:

mysql> ALTER TABLE product DISCARD TABLESPACE;

2d. Copy the pre-existing .ibd file to the datadir/test1 folder

2e. Import this tablespace:

mysql> ALTER TABLE product IMPORT TABLESPACE;

This should produce the following error (at least this is most likely). The only way it would not is if MySQL’s current table id matched that of the preexisting ibd table id. In which case, you can now dump your table.

ERROR 1030 (HY000): Got error -1 from storage engine

2f. So, now to check the error log (manually). Look for the following entry:

081010 11:47:40  InnoDB: Error: tablespace id in file
'.test1product.ibd' is 1193, but in the InnoDB
InnoDB: data dictionary it is 1.

So, now we know the internal table id is at 1, and that of the ibd table is 1193.

3. Clean up working database:

3a. Manually move the ibd file from the $datadir to a safe location (as you will need this file again).

3b. Drop this table.

mysql> DROP TABLE product;

Note this does not re-set the internal table counter.

4. You’ll need to create the number of tables you need to increase the internal table id value.

In this case, you’d create 1191 test InnoDB tables (already at 1, and need to leave 1 for the actual table, so 1193-2=1191). Run below in a loop.

for ($1=1; $i<=1191; $1++) {
CREATE TABLE t# (id int) ENGINE=InnoDB;
}

I accomplished this via a simple php script. See the bottom of this page (under "Associated Files") for the exact script.

5. After these are created, go ahead and drop this database and all tables (as they are not needed).

DROP DB test1;

6. Now, re-perform steps 2a through 2e.

mysql> CREATE DATABASE test1;
mysql> USE test1;
mysql> CREATE TABLE `product` ( ... ) ENGINE=InnoDB;
mysql> ALTER TABLE product DISCARD TABLESPACE; <-- Here is where you copy back the original ibd file to /$datadir/test1/ --> mysql> ALTER TABLE product IMPORT TABLESPACE;

Success!

mysql> show tables;
+-----------------+
| Tables_in_test1 |
+-----------------+
| product |
+-----------------+
1 row in set (0.00 sec)

7. Now, dump the table using mysqldump, and then you can import this to any MySQL instance. Note, you must dump this and re-import it, or you'll run into problems.

However, it's possible to encounter crashes and/or reports of corruption in the logs.

If this happens, try to force innodb recovery (which is most likely), and then dump the table.

Start by setting innodb_force_recovery=1 (and try 2,3,4,5,6) until the dump works.

For this example table, I had to set innodb_force_recovery=5 before the dump would succeed.

The # in the output file name is the value I had innodb_force_recovery set to when trying to perform the dump:

C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump1.txt
mysqldump: Couldn't execute 'show table status like 'product'':
Lost connection to MySQL server during query (2013) C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump2.txt
mysqldump: Couldn't execute 'show table status like 'product'':
Lost connection to MySQL server during query (2013) C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump3.txt
mysqldump: Couldn't execute 'show table status like 'product'':
Lost connection to MySQL server during query (2013) C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump4.txt
mysqldump: Couldn't execute 'SELECT /*!40001 SQL_NO_CACHE */
* FROM `product`': Lost connection to MySQL server during
query (2013) C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump5.txt C:Program FilesMySQLmysql-5.0.68bin>
mysqladmin -u root -P 3385 shutdown C:Program FilesMySQLmysql-5.0.68bin>
mysqldump -uroot -P3385 test1 product > product_dump6.txt

In fact, in this case, I could have simply started with 5. This is because the error log stated this:

InnoDB: Error: trying to access update undo rec field 19
in index PRIMARY of table test1/product
InnoDB: but index has only 12 fields

So, I knew there was a problem trying to look at the undo logs, and from the manual, a setting of 5 says this:

"Do not look at undo logs when starting the database: InnoDB treats even incomplete transactions as committed"

However, it's best to start at 1 and work your way forward so as to prevent as much data loss as possible.

Method #2 - Hex Edit .ibd file

First of all, you'll need to backup everything (ibdata files, ib_logfile(s), data). I'd also perform a mysqldump of everything you currently have, just so you have a mysqldump of it in the event that you need it.

Also, very important, be sure to make a copy of the .ibd file for the specific table in question.

Lastly, get a copy of the CREATE TABLE statement that will recreate this table.

Then, you'll follow the steps #1-5 (but do not perform step #6 yet) outlined on the following page:

http://dev.mysql.com/doc/refman/5.0/en/adding-and-removing.html

Let me post them here for completeness, however:

  1. Use mysqldump to dump all your InnoDB tables.
  2. Stop the server.
  3. Remove all the existing tablespace files, including the ibdata and ib_log files. If you want to keep a backup copy of the information, then copy all the ib* files to another location before the removing the files in your MySQL installation.
  4. Remove any .frm files for InnoDB tables.
  5. Configure a new tablespace.
  6. Restart the server.
  7. Import the dump files.

At this point, MySQL should be running fine with an empty slate (and should have just re-created your new ibdata and log files).

Now, you'll want to recreate the table (just using the CREATE TABLE output from above), and its database to hold it.

Then, you'll basically be following the steps #1-3 outlined on the following page:

http://dev.mysql.com/doc/refman/5.0/en/multiple-tablespaces.html

1. Issue this ALTER TABLE statement:

ALTER TABLE tbl_name DISCARD TABLESPACE;

Caution: This statement deletes the current .ibd file.

2. Put the backup .ibd file back in the proper database directory (the one that you copied above).

3. Issue this ALTER TABLE statement:

ALTER TABLE tbl_name IMPORT TABLESPACE;

Everything should go smoothly until step #3 (above). More than likely, this will produce an error like the following on your console:

"Got error -1 from storage engine"

Now, if you look in the error log, you'll see something like:

"InnoDB: Error: tablespace id in file '.testt2.ibd' is 2,
but in the InnoDB data dictionary it is 1."

It would not produce the above error and would work fine if the existing table already had a tablespace id of 1. However, this is unlikely.

So, assuming you see the above errors, then you can modify the tablespace id actual ibd file using a hex editor. I would do this on a different copy of the ibd file (other than the original, just in case).

Note that I used "Freeware Hex Editor XVI32" for Windows for this step. Start the program, and then open the .ibd file. You'll see each byte in it's own cell. You can then click on a cell, click edit, and then edit that byte. (http://www.chmaas.handshake.de/delphi/freeware/xvi32/xvi32.htm)

Now, in this file, there are 2 places where this tablespace id is located.

For me, and I assume it should be the same for you, but just look at the values to be sure, I see the tablespace id values listed at position 37 and 41 (positions 25 and 29 in hex). In the actual hex column, if you're previous tablespace id was 2, then in positions 37 and 41, you'd see 02 and 02.

(Note these positions can change. For instance, I tested on a table with an internal id of 1193. This in hex is 04A9. However, when searching the file, for the first instance of the table id, I found the '04' in position 39 and 'A9' in position 40. Then, for the second instance of the table id, the '04' was at position 43 and the 'A9' was at position 44. So, you'll have to convert the table id to hex, and then search for that value, near the beginning of the file.)

Note that this value (02) may vary depending on what your actual tablespace id is.

Then, simply modify both of those fields to 01, and save the file.

Then, re-do the following 3 steps:

1. ALTER TABLE tbl_name DISCARD TABLESPACE;
2. Put the newly saved .ibd file back in the proper database directory
3. ALTER TABLE tbl_name IMPORT TABLESPACE;

This time, step #3 works fine.

It is at this point you should dump/import the data. At least, get a good mysqldump of this table now. You may find that this causes corruption in InnoDB, and you may need to start MySQL using --force-innodb-recovery.

http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html Forcing InnoDB Recovery

Associated Files :: PHP Scripts

Simple PHP script - Used to create a number of InnoDB tables (to increase internal table id counter):

$dbhost = "localhost:3385";
$dbname = "test1";
$dbuser = "root";
$dbpwd = ""; mysql_connect($dbhost,$dbuser,$dbpwd) or die(mysql_error()); for ($i = 1033; $i <= 1190; $i++) {
$dbquery = "CREATE TABLE test1.t" . $i . " (id int) ENGINE=InnoDB"; echo "" . $dbquery . ""; $result = mysql_db_query($dbname,$dbquery) or die(mysql_error()); $j = 0; while($row = mysql_fetch_array($result)) {
$j++;
echo $row[0];
}
} mysql_close();

PHP Internal Table ID Finder - Used to determine the internal Table ID from the binary .ibd file:

/*
Tested with tables from 4.1.23, 5.0.68, 5.1.28, and 6.0.7.
*/ // Set the filename
$filename = "C:\Users\Chris\Desktop\mysql\working\ibds\z1.ibd"; // Read 2 bytes in at a time
$offset = 2; // Echo filename and path
echo "filename = $filename

"; // Open the filename - need 'rb' for binary file on Windows
$handle = fopen($filename, "rb"); // Define redundant, local variables for possible later functionality and/or checks
$ibd_id_bin = 0;
$ibd_id_hex = 0;
$ibd_id_dec = 0;
$ibd_id_bin2 = 0;
$ibd_id_hex2 = 0;
$ibd_id_dec2 = 0; // Find the filesize (note: below command messes up script)
//$filesize = filesize($filename)); // Only loop through first 21 bytes - as table is is in $array[18] and $array[20]
for ($z = 0; $z <= 20; $z++) { // Set variable $contents equal to 2 ($offset) bytes of binary data
$contents = fread($handle, $offset); // Convert $contents from binary data to hex data
$contents2 = bin2hex($contents); // Convert $contents2 from hex data to decimal data
$contents3 = hexdec($contents2); // Debug Output
//echo "contents[$z] = " . $contents . "
";
//echo "contents2[$z] = " . $contents2 . "

";
//echo "contents3[$z] = " . $contents3 . "

"; // If position 19, array position [18], then store the values
if ($z == 18) {
$ibd_id_bin = $contents;
$ibd_id_hex = $contents2;
$ibd_id_dec = $contents3;
} // If position 21, array position [20], then store the values
if ($z == 20) {
$ibd_id_bin2 = $contents;
$ibd_id_hex2 = $contents2;
$ibd_id_dec2 = $contents3;
}
}
fclose($handle); // More Debug output
//echo "

The table id is $ibd_id_dec

";
//echo "

The table id is $ibd_id_dec2

"; // Check to see if both values are equal. If so, then it's
// most certain this is the correct value.
// If not, then there's a chance the positions are off for
// this table (due to versions, etc.).
if ($ibd_id_dec == $ibd_id_dec2) {
echo "

The table id is $ibd_id_dec

";
} else {
echo "The values from positions [18] and [20] did not match,";
echo "so please enable debug output, and check for proper positions.";
}

Recovering InnoDB table from an .ibd file.的更多相关文章

  1. MySQL 5.6 Reference Manual-14.7 InnoDB Table Compression

    14.7 InnoDB Table Compression 14.7.1 Overview of Table Compression 14.7.2 Enabling Compression for a ...

  2. MySQL 5.6 Reference Manual-14.6 InnoDB Table Management

    14.6 InnoDB Table Management 14.6.1 Creating InnoDB Tables 14.6.2 Moving or Copying InnoDB Tables to ...

  3. mysql InnoDB通过.frm和.ibd恢复表和数据

    ibdata1是一个用来构建innodb系统表空间的文件,这个文件包含了innodb表的元数据.撤销记录.修改buffer和双写buffer.如果file-per-table选项打开的话,该文件则不一 ...

  4. InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pa

    2016-09-14T09:17:37.713955Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleane ...

  5. windows mysql5.7 InnoDB 通过frm与ibd对数据进行恢复

    参考:https://www.jianshu.com/p/50a2e13cd5cf 安装MySQL Utilities 下载地址:https://dev.mysql.com/downloads/uti ...

  6. MySQL 8 InnoDB Table 和 Page 压缩

    压缩用一点CPU换取磁盘IO.内存空间.磁盘空间. 在有Secondary Indexes 的表中,使用压缩更加明显,相关索引数据也会压缩. InnoDB 表压缩 对表压缩只需要在Create Tab ...

  7. win10安装mysql时报错[MY-012576] [InnoDB] Unable to create temporary file; errno: 2

    报错信息 解决: 在my.ini文件里面的 [mysqld]区段内加入: #自己指定的临时文件目录 tmpdir="临时目录" 添加好后初始化成功 接下来启动mysql服务的时候报 ...

  8. windows 下使用 zip安装包安装MySQL 5.7

    以下内容参考官方文档:http://dev.mysql.com/doc/refman/5.7/en/windows-start-command-line.html 解压缩zip到D:\mysql-5. ...

  9. MySQL/MariaDB数据库的服务器配置

    MySQL/MariaDB数据库的服务器配置 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.MySQL中的系统数据库 1>.mysql数据库 是mysql的核心数据库,类 ...

随机推荐

  1. Markdown基本语法总结

    一.标题 在想要设置为标题的文字前面加#来表示 一个#是一级标题,二个#是二级标题,以此类推.支持六级标题. 注:标准语法一般在#后跟个空格再写文字. 示例: # 这是一级标题 ## 这是二级标题 # ...

  2. java翻译到mono C#实现系列(2) mono实现GridView 横向滚动

    群里的朋友问GridView 横向滚动怎么实现,我就百度了,参考http://blog.csdn.net/lonely_fireworks/article/details/7841134写了个mono ...

  3. [Xamarin.Android] 儲存資料於Windows Azure (转帖)

    在準備討論Xamarin.Android 如何整合GCM與Windows Azure來實作Push Notification之前, 先來了解如何將Xamarin.Android 與Windows Az ...

  4. MySQL error2003错误原因以及解决方案

    转自:http://hi.baidu.com/tianxia339/item/8e8849111461ea7e7a5f2540 出现ERROR 2003 (HY000): Can't connect ...

  5. 利用setTimeout来实现setInterval

    在Js中,当我们要在一定间隔时间内不断执行同一函数,我们可以使用setInterval函数,但setInterval在某些情况下使用时也存在一定问题. 1.不去关心回调函数是否还在运行 在某些情况下, ...

  6. Android动画原理

    1 Frame Animation:大体意思就是将UI设计的多张图片组成的动画,然后在将他们组合起来连贯进行播放,类似于早期电影的工作原理. 2 Tween Animation:是对某个View进行一 ...

  7. Capture Conversion解读

    Let G name a generic type declaration with n type parameters A1,...,An with corresponding bounds U1, ...

  8. js解决千分符问题

    js脚本function: //js数字千分符处理 function commafy(num) { num = num + ""; var re = /(-?\d+)(\d{3}) ...

  9. Mac 安装tensorflow

    一. 安装 TensorFlow谷歌的官网和开源项目都有介绍各个系统的安装和使用(官网:https://www.tensorflow.org/install git: https://github.c ...

  10. Debian 9 启动后进入命令行

    打开 default grub 配置 $ sudo vi /etc/default/grub 修改以下3处内容 1. 找到行 GRUB_CMDLINE_LINUX_DEFAULT="quie ...