Using innodb_large_prefix to avoid ERROR 1071

单列索引限制
上面有提到单列索引限制767，起因是256×3-1。这个3是字符最大占用空间（utf8）。但是在5.5以后，开始支持4个字节的uutf8。255×4>767, 于是增加了一个参数叫做 innodb_large_prefix。
这个参数默认值是OFF。当改为ON时，允许列索引最大达到3072。
联合索引3072
我们知道InnoDB一个page的默认大小是16k。由于是Btree组织，要求叶子节点上一个page至少要包含两条记录（否则就退化链表了）。
所以一个记录最多不能超过8k。
又由于InnoDB的聚簇索引结构，一个二级索引要包含主键索引，因此每个单个索引不能超过4k （极端情况，pk和某个二级索引都达到这个限制）。
由于需要预留和辅助空间，扣掉后不能超过3500，取个“整数”就是(1024*3)。
具体参见丁奇的分享： http://www.mysqlops.com/2012/09/ ... 6%E7%9A%84tips.html

原文 http://mechanics.flite.com/blog/2014/07/29/using-innodb-large-prefix-to-avoid-error-1071/

If you’ve ever tried to add an index that includes a long varchar column to an InnoDB table in MySQL, you may have seen this error:

ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

The character limit depends on the character set you use. For example if you uselatin1 then the largest column you can index is varchar(767) , but if you useutf8 then the limit is varchar(255) . There is also a separate 3072 byte limit per index. The 767 byte limit is per column, so you can include multiple columns (each 767 bytes or smaller) up to 3072 total bytes per index, but no column longer than 767 bytes. (MyISAM is a little different. It has a 1000 byte index length limit, but no separate column length limit within that).

One workaround for these limits is to only index a prefix of the longer columns, but what if you want to index more than 767 bytes of a column in InnoDB?

In that case you should consider using innodb_large_prefix, which was introduced in MySQL 5.5.14 and allows you to include columns up to 3072 bytes long in InnoDB indexes. It does not affect the index limit, which is still 3072 bytes as quoted inthe manual :

The InnoDB internal maximum key length is 3500 bytes, but MySQL itself restricts this to 3072 bytes. This limit applies to the length of the combined index key in a multi-column index.

Read on for details and examples about innodb_large_prefix.

Here are a few pre-requisites for using innodb_large_prefix:

At the database level you have to use innodb_file_format=BARRACUDA
At the table level you have to use ROW_FORMAT=DYNAMIC orROW_FORMAT=COMPRESSED

The default file format is still Antelope for backwards compatibility, and the default row format is COMPACT.

You can set both innodb_file_format and innodb_large_prefixdynamically, but you should also set them in my.cnf so they survive a restart. innodb_file_format 默认值为Antelope, innodb_large_prefixdynamically默认值为OFF.

Here’s an example. If I try to create this table with innodb_large_prefixdisabled I get an error:

mysql> create table if not exists utf8_test (
    ->   day date not null,
    ->   product_id int not null,
    ->   dimension1 varchar(500) character set utf8 collate utf8_bin not null,
    ->   dimension2 varchar(500) character set utf8 collate utf8_bin not null,
    ->   unique index unique_index (day, product_id, dimension1, dimension2)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

If I enable innodb_large_prefix I can create the table successfully:

mysql> set global innodb_file_format = BARRACUDA;
Query OK, 0 rows affected (0.00 sec)

mysql> set global innodb_large_prefix = ON;
Query OK, 0 rows affected (0.00 sec)

mysql> create table if not exists utf8_test (
    ->   day date not null,
    ->   product_id int not null,
    ->   dimension1 varchar(500) character set utf8 collate utf8_bin not null,
    ->   dimension2 varchar(500) character set utf8 collate utf8_bin not null,
    ->   unique index unique_index (day, product_id, dimension1, dimension2)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.02 sec)

在/etc/my.cnf中添加这两行,效果一样: innodb_file_format=Barracuda, innodb_large_prefix=ON

The examples are similar for latin1 , but I can use columns three times as long since it’s a single-byte character set.

mysql> create table if not exists latin1_test (
    ->   day date not null,
    ->   product_id int not null,
    ->   dimension1 varchar(1500) not null,
    ->   dimension2 varchar(1500) not null,
    ->   unique index unique_index (day, product_id, dimension1, dimension2)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

mysql> set global innodb_file_format = BARRACUDA;
Query OK, 0 rows affected (0.00 sec)

mysql> set global innodb_large_prefix = ON;
Query OK, 0 rows affected (0.00 sec)

mysql> create table if not exists latin1_test (
    ->   day date not null,
    ->   product_id int not null,
    ->   dimension1 varchar(1500) not null,
    ->   dimension2 varchar(1500) not null,
    ->   unique index unique_index (day, product_id, dimension1, dimension2)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.02 sec)

And here’s what happens if I try to create an index longer than 3072 bytes, 1500*3=4500>3072:

mysql> create table if not exists long_index_test (
    ->   day date not null,
    ->   product_id int not null,
    ->   dimension1 varchar(1500) not null,
    ->   dimension2 varchar(1500) not null,
    ->   dimension3 varchar(1500) not null,
    ->   unique index unique_index (day, product_id, dimension1, dimension2, dimension3)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
ERROR 1071 (42000): Specified key was too long; max key length is 3072 bytes

只要各个column的各列不大于3072就不会报错:

create table if not exists tmp1 (
     x0 varchar() not null,
     x1 varchar() not null,
     x2 varchar() not null,
     x3 varchar() not null,
     unique index unique_index (x3)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;

Query OK, 0 rows affected (0.23 sec)

有一column超过3072即报错:

mysql> create table if not exists tmp1 (
    ->      x3 varchar() not null,
    ->      unique index unique_index (x3)
    ->     ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
ERROR  (): Specified key was too  bytes

stackoverflow上的回答:

Using DYNAMIC or COMPRESSED means that InnoDB stores varchar/text/blob fields that don't fit in the page completely off-page. But other than those columns, which then only count 20 bytes per column, the InnoDB row size limit has not changed; it's still limited to about 8000 bytes per row.

InnoDB only supports indexes of 767 bytes per column. You can raise this 3072 bytes by setting innodb_large_prefix=1 and using either DYNAMIC or COMPRESSED row format.

Using COMPRESSED row format does not make InnoDB support longer indexes.

Regarding performance, this is one of those cases where "it depends." Compression is generally a tradeoff between storage size and CPU load to compress and uncompress. It's true that this takes a bit more CPU to work with compressed data, but you have to keep in mind that database servers are typically waiting for I/O and have CPU resources to spare.

But not always -- if you do complex queries against data that is in the buffer pool, you may be constrained by CPU more than I/O. So it depends on many factors, like how well your data fits in RAM, the type of queries you run and how many queries per second, as well as hardware specs. Too many factors for anyone else to be able to answer for your application on your server. You'll just have to test it.

Re your comment:

One possibility is that the index is not fitting in the buffer pool. Performance degrades significantly if an index search needs to load pages and evict pages during every SELECT query. An EXPLAIN analysis can't tell you whether the index fits in the buffer pool.

I don't know how many columns or what data types of the columns in your index, but if you are indexing long varchar columns you should consider using prefix indexes (or decreasing the length of the columns).

You could also get more RAM and increase the size of the buffer pool.

Official doc 5.6

Limits on InnoDB Tables

The InnoDB internal maximum key length is 3500 bytes, but MySQL itself restricts this to 3072 bytes. This limit applies to the length of the combined index key in a multi-column index.

http://dev.mysql.com/doc/refman/5.7/en/innodb-restrictions.html

The MyISAM Storage Engine

The maximum key length is 1000 bytes. This can also be changed by changing the source and recompiling. For the case of a key longer than 250 bytes, a larger key block size than the default of 1024 bytes is used.

http://dev.mysql.com/doc/refman/5.7/en/myisam-storage-engine.html

However your table is declared as InnoDB. So I don't know what to think

There's also a clue at the end of this old bug

If you need this you should really look at MySQL 5.5 and the innodb_large_prefix option that is available from 5.5.14 (July 2011) onwards because it probably does what you are looking for:

"Enable this option to allow index key prefixes longer than 767 bytes (up to 3072 bytes), for InnoDB tables that use the DYNAMIC and COMPRESSED row formats. (Creating such tables also requires the option values innodb_file_format=barracuda and innodb_file_per_table=true.) See Section 13.3.15, “Limits on InnoDB Tables” for the relevant maximums associated with index key prefixes under various settings.

Enabling Compression for a Table

Before creating a compressed table, make sure the innodb_file_per_table configuration option is enabled, and innodb_file_format is set to Barracuda. You can set these parameters in the MySQL configuration file my.cnf or my.ini, or with theSET statement without shutting down the MySQL server.

To enable compression for a table, you use the clauses ROW_FORMAT=COMPRESSED, KEY_BLOCK_SIZE, or both in a CREATE TABLE or ALTER TABLE statement.

To create a compressed table, you might use statements like these:

SET GLOBAL innodb_file_per_table=1;
SET GLOBAL innodb_file_format=Barracuda;
CREATE TABLE t1
 (c1 INT PRIMARY KEY)
 ROW_FORMAT=COMPRESSED
 KEY_BLOCK_SIZE=8;

If you specify ROW_FORMAT=COMPRESSED, you can omit KEY_BLOCK_SIZE; the default compressed page size of 8KB is used.
If you specify KEY_BLOCK_SIZE, you can omit ROW_FORMAT=COMPRESSED; compression is enabled automatically.
To determine the best value for KEY_BLOCK_SIZE, typically you create several copies of the same table with different values for this clause, then measure the size of the resulting .ibd files and see how well each performs with a realistic workload.
For additional performance-related configuration options, see Section 14.12.3, “Tuning Compression for InnoDB Tables”.

The default uncompressed size of InnoDB data pages is 16KB. Depending on the combination of option values, MySQL uses a page size of 1KB, 2KB, 4KB, 8KB, or 16KB for the .ibd file of the table. The actual compression algorithm is not affected by theKEY_BLOCK_SIZE value; the value determines how large each compressed chunk is, which in turn affects how many rows can be packed into each compressed page.

Setting KEY_BLOCK_SIZE=16 typically does not result in much compression, since the normal InnoDB page size is 16KB. This setting may still be useful for tables with many long BLOB, VARCHAR or TEXT columns, because such values often do compress well, and might therefore require fewer overflow pages as described in Section 14.12.5, “How Compression Works for InnoDB Tables”.

All indexes of a table (including the clustered index) are compressed using the same page size, as specified in the CREATE TABLE or ALTER TABLE statement. Table attributes such as ROW_FORMAT and KEY_BLOCK_SIZE are not part of the CREATE INDEXsyntax, and are ignored if they are specified (although you see them in the output of the SHOW CREATE TABLE statement).

Restrictions on Compressed Tables

Because MySQL versions prior to 5.1 cannot process compressed tables, using compression requires specifying the configuration parameter innodb_file_format=Barracuda, to avoid accidentally introducing compatibility issues.

Table compression is also not available for the InnoDB system tablespace. The system tablespace (space 0, the ibdata* files) can contain user data, but it also contains internal system information, and therefore is never compressed. Thus, compression applies only to tables (and indexes) stored in their own tablespaces, that is, created with the innodb_file_per_table option enabled.

Compression applies to an entire table and all its associated indexes, not to individual rows, despite the clause name ROW_FORMAT.

MyISAM和InnoDB的行格式ROW_FORMAT

MyISAM行存储

MyISAM有3种行存储格式：fixed/dynamic/compressed；

其中fixed为默认格式，只有当表不包含变长字段(varchar/varbinary/blob/text)时使用，该每行都是固定的，所以很容易获取行在页上的具体位置，存取效率比较高，但是占用磁盘空间较多；

dynamic

每行都有一个行头部，包含bitmap，用以记录那些列为空(NULL列不算为空)；

相比于fixed，其有如下特性：

所有字符串列都是动态存储的，除非长度小于4；

字符类型若长度为0/数字类型为0都会不占用存储空间，由bitmap标注，NULL值不包含在内；

如果要update行，其扩展后很容易导致行链接既而产生碎片，一旦crash若link丢失则比较难恢复，fixed模式update不会产生碎片；

compressed只能通过myisampack创建且为只读；

MyISAM的索引文件包含一个flag记录基表是否正常关闭，如果mysqld启动时指定了--myisam-recover-options，则在打开表时检测并自动修复表

InnoDB行存储

Innodb plugin新引入Barracuda梭子鱼，其包含compressed/dynamic两种行格式，而之前的compact/redundant统属于antelope羚羊；

Barracuda VS antelope

由innodb_file_format(动态)参数决定，目前可选值由Antelope和Barracuda，默认为前者；要想要此参数生效，

因为共享表空间默认为Antelope，因此要想使用Barracuda为默认值，还必须先声明innodb_file_per_table；

Innodb_file_format用于控制行格式，全局变量可动态调整，5.5默认依旧是Antelope；

下面只看antelope格式

Redundant行结构

字段长度偏移列表

记录头信息

列1数据

列2数据

….

行头部为字段长度偏移信息，包括变长和非变长的, 还包含了3个隐藏列：RowID(没有主键时使用)/Transaction ID/Roll Point; 而compact只包含变长的，节约了空间；

冗余行格式没有NULL标志位；对于redundant格式，varchar为Null时不占用空间，但是char为NULL需要占用空间，因为其没有Null标志位；

记录头信息占用6个字节，比compact多1字节；

对于定长char，若为NULL依旧填充整个字段，而varchar为Null时不占用空间；

记录头信息，与compact相比，多了黑体字部分，缺失record_type

名称	长度bit	功能
Deleted_flag	1	是否被删除
Min_rec_flag	1	1则表示该记录为预先被定义的最小记录
N_owned	4	该记录拥有的总记录数
Heap_no	13	索引中该行的排序记录
N_fields	10	记录中列数量
1byte_offs_flag	1	偏移量列表是1字节还是2字节
Next_recorder	16	下一条记录相对位置
()	1	未知
()	1	未知

Create table test(t1 varchar(10), t2 varchar(10), t3 char(10),t4 varchar(10)) charset=latin1 row_format=redundant;

--该表有3个变长列

Insert into test values(‘a’,’bb’,’bb’,’ccc’);

使用hexdump –C –v test.idb查看其二进制代码

--长度偏移列表，

compact行格式

字段长度偏移列表

NULL标志位

记录头信息

列1数据

列2数据

….

5.0引入

行头存放该行内变长字段的length，当列小于255字节时占用1个字节，大于255而小于65535时占用2个字节；故varchar最大长度为2的16次方-1；

第2个指示该行是否有NULL值，占用1字节；NULL列不占用数据存储空间；

记录头信息：5个字节共计40bit，用于链接相邻的记录案的行级锁

名称	长度bit	功能
Deleted_flag	1	是否被删除
Min_rec_flag	1	1则表示该记录为预先被定义的最小记录
N_owned	4	该记录拥有的总记录数
Heap_no	13	索引中该行的排序记录
Record_type		行类型 0=普通 1=B+节点指针
Next_recorder	16	下一条记录相对位置
()	1	未知
()	1	未知

除此之外，每页还有两个隐含字段：

DB_TRX_ID：6字节，记录最近的一个事务标示符

DB_ROLL_ID：7字节，指向回滚日志记录

--若没有主键，则还会有DB_ROW_ID：6字节，包含在clustered索引中

创建一个compact行格式的表

Create table test(t1 varchar(10), t2 varchar(10), t3 char(10),t4 varchar(10)) row_format=compact;

--该表有3个变长列

Insert into test values(‘a’,’bb’,’bb’,’ccc’);

使用hexdump –C –v test.idb查看其二进制代码

第一行

03 02 01—变长字段长度列表(逆序)，实际顺序为01 02 03，这也是t1,t2,t4的实际长度

00—Null标志位，第一行没有NULL

00 00 10 00 2c—记录头信息，5字节，后4个字节指向下一个记录next_recorder

00 00 00 2b 68 00—6字节rowid，因为没有主键

00 00 00 00 06 05 –事务ID，6字节

80 00 00 00 32 01 10—回滚指针，7字节

61 –列1

62 62 –列2

62 62 20 20 20 20 20 20 20 20 –列3，char会填充余下部分

63 63 63 –列4

余下的为列数据，其中t3由于采用固定长度，故会填充满10个字节;

第二行

Insert into test values(‘d’,null,null,’fff’);

03 01--变长字段长度列表，逆序

06-- Null标志位，有NULL值，转换为二进制00000110，表示第2/3列为null

……

64—列1数据

66 66 66—列4数据，而第2/3列为NULL不占用存储空间

注：对于redundant格式，varchar为Null时同样不占用空间，但是char为NULL需要占用空间，因为其没有Null标志位

行溢出

Innodb表为IOT，采用了B+数类型，故每个页面至少要存储2行数据，如果行过大则会产生行溢出；

理论上mysql的varchar可存储65525字节，强于oracle的4000，但对于InnoDB其实际上限为65532，且该值为表所有varchar列长度总和；对于utf8字符集，一个字符占3个字节，则其上限又缩小为1/3；

如果强行创建varchar(65535)的字段，在sql_mode不为restricted的情况下，其会被隐式转换为mediumtext；

不论是varchar还是blob/text，只要保证一个16k的页面能容下2行数据，应该不会行溢出；

而一旦行溢出，字段前768字节依旧存放于当前页面，数据一般使用B-tree Node页，而溢出的行存放于Uncompress Blob页；

而barracuda采用了完全行溢出，即只保留字段的前20字节；