【MySQL】TokuDB引擎初探（MySQL升级为Percona，MySQL升级为MariaDB）

参考：http://blog.sina.com.cn/s/blog_4673e6030102v46l.html

参考：http://hcymysql.blog.51cto.com/5223301/1431698

参考：http://blog.csdn.net/yueliangdao0608/article/details/21230627

参考：http://code.openark.org/blog/mysql/tokudb-configuration-variables-of-interest

参考：http://blog.itpub.net/22418990/viewspace-1251497/

官网说明：http://docs.tokutek.com/tokudb/tokudb-index-variables.html

官网说明：https://mariadb.com/kb/en/mariadb/tokudb-system-variables/#tokudb_pk_insert_mode

备份：http://mysql.taobao.org/monthly/2015/12/06/

因为MariaDB 10版本需要单独编译TokuDB引擎，编译需要GCC >= 4.7和CMake >=2.8.8版本，暂没有进行单独编译，后续再测试。

官网TokuDB源码下面的说明：

Before you start, make sure you have a C++11-compatible compiler (GCC >= 4.7 is recommended), as well as CMake >=2.8.8, and the libraries and header files for valgrind,zlib, and Berkeley DB. We are using the gcc 4.7 in devtoolset-1.1.

测试版本：Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101.tar.gz

测试版本：Percona-Server-5.6.24-rel72.2-TokuDB.Linux.x86_64.ssl101.tar.gz

OS:Centos 6.2

# - path of TokuDB file - #
> tokudb_data_dir
> tokudb_log_dir
> tokudb_tmp_dir

# - variables of TokuDB engine - #
> tokudb_cache_size (Dynamic: NO,Default Value: Half of the total system memory)
This variable configures the size in bytes of the TokuDB cache table. The default cache table size is 1/2 of physical memory. Tokutek highly recommends using the default setting if using buffered IO, if using direct IO then consider setting this parameter to 80% of available memory.

> tokudb_directio (Dynamic: NO,Default Value: OFF)
When set to ON, TokuDB employs Direct IO rather than Buffered IO for writes. When using Direct IO consider increasing tokudb cache size from its default of 1/2 physical memory. The default values is OFF.

> tokudb_commit_sync (Dynamic: Yes,Default Value: ON)
Session variable tokudb_commit_sync controls whether or not the transaction log is flushed when a transaction commits. The default behavior is that the transaction log is flushed by the commit.
Setting this parameter to off may make the system run faster. However, transactions committed since the last checkpoint are not guaranteed to survive a crash.

> tokudb_fsync_log_period (Dynamic: Yes,Default Value: 0)
Controls the frequency, in milliseconds, for fsync() operations. If set to 0 then the fsync() behavior is only controlled by the tokudb commit sync, which is on or off. The default values is 0.

> tokudb_row_format (Dynamic: Yes,Default Value: tokudb_zlib)
tokudb_default, tokudb_zlib: Use the zlib library,
tokudb_fast, tokudb_quicklz: Use the quicklz library, the lightest compression with low CPU usage,
tokudb_small, tokudb_lzma: Use the lzma library. the highest compression and highest CPU usage
tokudb_uncompressed: No compression is used.
http://docs.tokutek.com/tokudb/tokudb-index-using-tokudb.html#tokudb-compression-details

> tokudb_read_buf_size
> tokudb_loader_memory_size
上面两个官网的说明没有读懂...求高人指点

二进制包解压后把toku包中的文件放到server包中的路径下

注意配置要添加

[mysqld_safe]
malloc_lib = /usr/local/Percona-Mysql/lib/mysql/libjemalloc.so

[mysqld]
plugin_dir = /usr/local/Percona-Mysql/lib/mysql/plugin/
plugin_load=ha_tokudb.so

修改系统Huge pages配置

[root@test percona]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
[root@test percona]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defarg

否则会报错

#### 系统Huge pages配置需要修改，Centos我的是redhat_transparent_hugepage/enabled和defrag
[ERROR] TokuDB: Huge pages are enabled, disable them before continuing
[ERROR] ************************************************************
[ERROR]
[ERROR]                         @@@@@@@@@@@
[ERROR]                       @@'         '@@
[ERROR]                      @@    _     _  @@
[ERROR]                      |    (.)   (.)  |
[ERROR]                      |             ` |
[ERROR]                      |        >    ' |
[ERROR]                      |     .----.    |
[ERROR]                      ..   |.----.|  ..
[ERROR]                       ..  '      ' ..
[ERROR]                         .._______,.
[ERROR]
[ERROR] TokuDB will not run with transparent huge pages enabled.
[ERROR] Please disable them to continue.
[ERROR] (echo never > /sys/kernel/mm/transparent_hugepage/enabled)
[ERROR]
[ERROR] ************************************************************
[ERROR] Plugin 'TokuDB' init function returned error.
[ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.

####  配置malloc_lib放错位置，应该放在[mysqld_safe]下
[ERROR] /usr/local/Percona-Server-5.6.-rel72.-Linux.x86_64.ssl101/bin/mysqld: unknown variable 'malloc_lib=/usr/local/Percona-Server-5.6.24-rel72.2-Linux.x86_64.ssl101/lib/mysql/libjemalloc.so'

#### 配置malloc_lib缺项
[ERROR] TokuDB is not initialized because jemalloc is not loaded
[ERROR] Plugin 'TokuDB' init function returned error.
[ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.

#### 启动失败后删除TokuDB遗留文件
[ERROR] TokuDB: Persistent environment information is missing (but log exists)
[ERROR] TokuDB unknown error
[ERROR] Plugin 'TokuDB' init function returned error.
[ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.

然后进入MySQL确认TokuDB可用

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                                    | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| InnoDB             | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES          | YES  | YES        |
| CSV                | YES     | CSV storage engine                                                         | NO           | NO   | NO         |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                                      | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears)             | NO           | NO   | NO         |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                  | NO           | NO   | NO         |
| TokuDB             | YES     | Tokutek TokuDB Storage Engine with Fractal Tree(tm) Technology             | YES          | YES  | YES        |
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                         | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                                     | NO           | NO   | NO         |
| MyISAM             | YES     | MyISAM storage engine                                                      | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                             | NULL         | NULL | NULL       |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
 rows in set (0.00 sec)

【性能测试】

由于使用的是Percona 5.6版本，默认performance_schema是开启的，所以关掉后再试。

建两张测试表，表字段类型长度一样引擎不一样

CREATE TABLE `test1_innodb` \ `test1_tokudb` (
  `a` bigint(20) NOT NULL AUTO_INCREMENT,
  `b` varchar(255) DEFAULT NULL,
  `c` varchar(255) DEFAULT NULL,
  `d` text,
  `e` varchar(255) DEFAULT NULL,
  `f` varchar(255) DEFAULT NULL,
  `g` varchar(255) DEFAULT NULL,
  `h` varchar(255) DEFAULT NULL,
  `i` text,
  `j` varchar(255) DEFAULT NULL,
  `k` varchar(255) DEFAULT NULL,
  `l` varchar(255) DEFAULT NULL,
  `m` varchar(255) DEFAULT NULL,
  `n` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`a`)
) ENGINE=InnoDB \ TokuDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8

##### 测试1

binlog_format = MIX
sync_binlog = 0
innodb_flush_log_at_trx_commit = 1
innodb_buffer_pool_size = 1G
tokudb_cache_size = 1G
tokudb_row_format = tokudb_zlib
tokudb_commit_sync = ON
tokudb_fsync_log_period = 0

mysql> load data local infile 'test.csv' into table test_innodb  ...;
Query OK, 65536 rows affected (51.46 sec)
Records: 65536  Deleted: 0  Skipped: 0  Warnings: 0

mysql> load data local infile 'test.csv' into table test_tokudb  ...;
Query OK, 65536 rows affected (31.86 sec)
Records: 65536  Deleted: 0  Skipped: 0  Warnings: 0

##### 统计信息为information_schema.tables的table_length,index_length,total(table_length+index_length)，换算完单位为G
mysql> test_innodb;
+---------+---------+---------+
| DATA    | idx     | total   |
+---------+---------+---------+
| 2.66791 | 0.00000 | 2.66791 |
+---------+---------+---------+

mysql> test_tokudb;
+---------+---------+---------+
| DATA    | idx     | total   |
+---------+---------+---------+
| 1.44171 | 0.00000 | 1.44171 |
+---------+---------+---------+

[root@test percona]# ll -h */*test_*
-rw-rw---- 1 mysql mysql  17K Jun 30 11:11 test/test_innodb.frm
-rw-rw---- 1 mysql mysql 2.8G Jun 30 11:24 test/test_innodb.ibd
-rw-rw---- 1 mysql mysql  17K Jun 30 11:10 test/test_tokudb.frm
-rw-rw---- 1 mysql mysql  16M Jun 30 11:26 toku_data/_test_test_tokudb_main_90_2_1b.tokudb
-rw-rw---- 1 mysql mysql  64K Jun 30 11:27 toku_data/_test_test_tokudb_status_90_1_1b.tokudb

##### 插入数据后，发现系统中tokudb数据文件大小与实例中统计的表大小相差很多，暂不分析原因，后面测试实例中统计信息的依据性。

##### 测试2

tokudb_row_format = tokudb_small
其余配置不变

mysql> load data local infile 'test.csv' into table test_tokudb_small  ...;
Query OK, 65536 rows affected (33.91 sec)
Records: 65536  Deleted: 0  Skipped: 0  Warnings: 0

mysql> test_tokudb_small;
+---------+---------+---------+
| DATA    | idx     | total   |
+---------+---------+---------+
| 1.44171 | 0.00000 | 1.44171 |
+---------+---------+---------+
1 row in set (0.00 sec)

[root@test percona]# ll -h */*test_toku*
-rw-rw---- 1 mysql mysql 17K Jun 30 11:10 test/test_tokudb.frm
-rw-rw---- 1 mysql mysql 17K Jun 30 11:36 test/test_tokudb_small.frm
-rw-rw---- 1 mysql mysql 16M Jun 30 11:26 toku_data/_test_test_tokudb_main_90_2_1b.tokudb
-rw-rw---- 1 mysql mysql 16M Jun 30 11:38 toku_data/_test_test_tokudb_small_main_a1_2_1b.tokudb
-rw-rw---- 1 mysql mysql 64K Jun 30 11:38 toku_data/_test_test_tokudb_small_status_a1_1_1b.tokudb
-rw-rw---- 1 mysql mysql 64K Jun 30 11:27 toku_data/_test_test_tokudb_status_90_1_1b.tokudb

##### 使用tokudb_small的表发现数据文件大小与tokudb_zlib相差并不多
##### 使用tokudb_small\tokudb_lzma CPU负载会比较高
##### tokudb_small, tokudb_lzma: Use the lzma library. the highest compression and highest CPU usage
##### 试了一下更换tokudb_row_format不影响表数据文件

##### 删除完数据使用optimize table测试磁盘回收，发现innodb表数据文件空间释放掉，而且速度很快，但是tokudb表数据文件空间没有释放，重启也未释放

#####rename table更换表名但是tokudb表数据文件名字不会更改，表结构.frm会变更

【MySQL 5.5.41官方带TokuDB版本】

版本：mysql-5.5.41-tokudb-7.5.5-linux-x86_64

加载TokuDB与MariaDB版本语法不一样

root@mysql_5541.sock::[tmp]
->INSTALL PLUGIN TOKUDB SONAME 'ha_tokudb.so';

测试数据

CREATE TABLE `test2` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `inta` bigint(20) DEFAULT NULL,
  `char1` varchar(255) DEFAULT NULL,
  `char2` varchar(255) DEFAULT NULL,
  `time1` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `char3` varchar(255) DEFAULT NULL,
  `char4` varchar(255) DEFAULT NULL,
  `intb` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=TokuDB;

begin;insert into test2(inta,char1,char2,time1,char3,char4,intb) values (12345678901,'','',now(),'','',12345678901);commit;

单线程写测试结果

tokudb_directio=OFF
real    13m1.932s

tokudb_directio=ON
real    11m46.517s

tokudb_directio=ON
tokudb_commit_sync            = OFF
tokudb_fsync_log_period       =
real    9m18.774s

tokudb_directio=OFF
tokudb_commit_sync            = OFF
tokudb_fsync_log_period       =
real    9m3.667s

并发5线程和10线程写速度测试，单线程数据50W

分别测试tokudb_directio开和关对性能的影响

tokudb_commit_sync=OFF
tokudb_fsync_log_period=1000

5线程 tokudb_directio=OFF
real    16m1.031s

10线程 tokudb_directio=OFF
real    22m0.711s

10线程 tokudb_directio=ON
real    22m37.935s

10线程更新数据速度测试，单线程数据10W

CREATE TABLE `test2` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `inta` bigint(20) DEFAULT NULL,
  `char1` varchar(255) DEFAULT NULL,
  `char2` varchar(255) DEFAULT NULL,
  `time1` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `char3` varchar(255) DEFAULT NULL,
  `char4` varchar(255) DEFAULT NULL,
  `intb` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=TokuDB;

...
begin;update test2 set inta=99998 ,char1='char1' where id=023567;commit;
begin;update test2 set inta=99999 ,char1='char1' where id=760214;commit;
begin;update test2 set inta=100000 ,char1='char1' where id=560488;commit;

用时:
real    3m51.970s

【MariaDB 10编译TokuDB】

参考：http://blog.sina.com.cn/s/blog_4673e6030102v46l.html

参考：http://blog.sina.com.cn/s/blog_4673e6030102v46k.html

参考：http://www.ithov.com/linux/130504.shtml

按照上面的参考按照高版本的gcc和cmake。(C++11-compatible compiler (GCC >= 4.7 is recommended), as well as CMake >=2.8.8)

如果出现报错/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found参考上面链接的方法解决

如果报下面的按照jemalloc 

CMake Warning at storage/tokudb/CMakeLists.txt:17 (MESSAGE):
TokuDB is enabled, but jemalloc is not. This configuration is not
supported

编译完的ha_tokudb.so放在官方同版本分发包中就可以使用了

MariaDB [(none)]> INSTALL SONAME 'ha_tokudb';

使用注意：

1.MariaDB分发包解压复制到/usr/local/下
2.MariaDB配置文件初始化时不要带TokuDB引擎参数
3.初始化实例系统表
4.启动实例执行“install soname 'ha_tokudb;'”
5.停止实例并将配置文件添加TokuDB引擎参数
　　如果使用默认参数可以不停实例
　　如果要修改tokudb_data_dir/tokudb_log_dir/tokudb_tmp_dir需要删除实例数据目录下面TokuDB加载文件，-BASH "rm -rf *tokudb*"

不建议tokudb使用自定义数据文件、临时文件、日志文件路径，自定义路径会导致tokudb引擎文件字典中使用绝对路径，如果实例数据库目录要在服务器系统移动位置时出现找不到数据文件的错误

6.第一次加载成功TokuDB，之后启动自动加载，但是tokudb_*配置错误会导致引擎注册启动失败，TokuDB引擎表无法使用

【MySLQ 5.5.21实例升级Percona 5.6.24】

tokudb最新5.5下载地址：http://www.tokutek.com/download.php

官方安装都需要停止实例安装tokudb引擎

参考：http://docs.tokutek.com/tokudb/tokudb-index-installation.html

测试从MySQL-5.5.21升级到Percona-Server-5.6.24

1、停止实例

2、使用Percona-Server-5.6.24启动实例

3、使用Percona-Server-5.6.24的mysql_upgrade -u -p -S 升级系统表结构

4、启动实例

[root@test Percona]# /usr/local/Percona-Server-5.6.-rel72.-Linux.x86_64.ssl101/bin/mysql_upgrade -uroot -p -S /tmp/mysql_3306.sock
Enter password:
Looking for 'mysql' as: /usr/local/Percona-Server-5.6.-rel72.-Linux.x86_64.ssl101/bin/mysql
Looking for 'mysqlcheck' as: /usr/local/Percona-Server-5.6.-rel72.-Linux.x86_64.ssl101/bin/mysqlcheck
Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock'
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock'
Warning: Using a password on the command line interface can be insecure.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Running 'mysql_fix_privilege_tables'...
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock'
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck' with connection arguments: '--socket=/tmp/mysql_3306.sock'
Warning: Using a password on the command line interface can be insecure.
test.t1                                            OK
tmp.tb                                             OK
tmp.tb_merge_goldeye_main_role                     OK
OK

【MySQL 5.5.37实例升级MariaDB 10.0.20】

官网对于mysql_upgrade的说明：https://mariadb.com/kb/en/mariadb/mysql_upgrade/

1、MariaDB 10.0.20官网分发包，下载解压

2、MySQL 5.5.37挺实例

3、MariaDB 10.0.20启动实例，如果实例启动正常，errlog报错忽略

4、MariaDB 10.0.20分发包中的mysql_upgrade升级实例系统表，查看升级状态

5、实例停止再启动，errlog无报错

[root@test mariadb]# ./bin/mysql_upgrade -uroot -p -S /tmp/mysql_3306.sock
Enter password:
MySQL upgrade detected
Phase /: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Phase /: Fixing views from mysql
Phase /: Running 'mysql_fix_privilege_tables'
Phase /: Fixing table and database names
Phase /: Checking and upgrading tables
Processing databases
information_schema
jiangxu
jiangxu.jiangxu1                                   OK
jiangxu.jiangxu2                                   OK
jiangxu.jiangxu3                                   OK
performance_schema
test
tmp
Phase /: Running 'FLUSH PRIVILEGES'
OK

【功能特性说明】：支持辅助聚集索引、在线添加索引、在线修改列属性等

辅助聚集索引 --> 使得通过该索引的查询都为覆盖索引，速度更快

在线添加索引 --> tokudb对于alter table add index 和create index处理方式不同。

Using the ALTER TABLE command for creating an index will create the index offline (with the table unavailable for inserts or queries), regardless of the value of tokudb_create_index_online. The only way to hot create an index is to use the CREATE INDEX command.

Hot creating an index will be slower than creating the index offline, and progress depends how busy the mysqld server is with other tasks. 

在线修改列属性 --> 有一些修改并不支持。

Hot column expansion operations are only supported to char, varchar, varbinary, and integer data types. Hot column expansion is not supported if the given column is part of the primary key or any secondary keys.

Hot column rename does not support the following data types: TIME, ENUM, BLOB, TINYBLOB, MEDIUMBLOB, LONGBLOB. Renaming columns of these types will revert to the standard MySQL blocking behavior.

上面只是官网的一小部分，详细看下面链接原文

参考：http://docs.tokutek.com/tokudb/tokudb-index-using-tokudb.html

【锁的说明】

实例主从复制建议使用binlog_format=ROW

参考：http://docs.tokutek.com/tokudb/tokudb-index-troubleshooting.html