原文地址:http://www.tocker.ca/2013/10/24/improving-the-performance-of-large-tables-in-mysql.html

Today I wanted to take a look at improving the performance of tables that cause performance problems based largely on their size. Some of this advice also applies to databases that are large in-aggregate over many tables, but I always find the individually large table a special-case that is problematic.

What you will normally find is that the speed that the table can be modified will trend down as the size increases. Here is what I am going to call the typical B+Tree index performance over time:

So we should expect degradation of performance due to the structure of the index, but there are actually some ways that we can try and stretch out the curve, and not degrade as quickly.

Ten potential ways to reduce large table impact:

    1. Make sure to use InnoDB instead of MyISAM. MyISAM can be faster at inserts to the end of a table, but it has both table locking (limiting updates and deletes) and uses a single lock to protect the key buffer when loading data to/from disk, resulting in contention. It also does not have the change buffering feature described just below.

    2. InnoDB has change buffering (previously called the insert buffer), which is a feature to delay building secondary indexes that are not unique, and merge writes. It's further described by Facebook here. It's not shown in the graph above, but it can boost insert performance by quite a lot, and it's enabled by default. It was greatly improved in MySQL 5.5, so it is time to upgrade if you haven't.

    3. Partitioning may reduce the size of indexes, effectively reducing the table
      into many smaller tables. It also reduces internal index->lockcontention, something that has been greatly improved in the MySQL 5.7.2 DMR.

    4. Use innodb page compression. For some workloads (particularly those with lots of char/varchar/text data types) compression will allow the data to be more compact, stretching out that performance curve for longer. It may also allow you to more easily justify SSDs which are typically smaller in capacity. InnoDB page compression was improved a lot in MySQL 5.6, courtesy of Facebook providing a series of patches.

    5. Sort and bulk load data into tables. Inserting in order will result in fewer page splits (which will perform worse on tables not in memory), and the bulk loading is not specifically related to the table size, but it will help reduce redo log pressure.

    6. Remove any unnecessary indexes on the table, paying particular attention to UNIQUE indexes as these disable change buffering. Don't use a UNIQUE index if you have no reason for that constraint; prefer a regular INDEX.

    7. Related to the points 5 & 6, the type of primary key also matters. It is much better to use either an INT or BIGINT datatype than say a GUID, which will have a curve that degrades much faster. Having no PRIMARY KEY will also affect performance negatively.

    8. If bulk loading a fresh table, delay creating any indexes besides the PRIMARY KEY. If you create them once all data is loaded, then InnoDB is able to apply a pre-sort and bulk load process which is both faster and results in typically more compact indexes. This optimization became true in MySQL 5.5.

    9. More memory can actually help here too. I frequently see people under spec memory on new database servers compared to what it actually costs these days. Simple advice: If SHOW ENGINE INNODB STATUSshows any reads/s under BUFFER POOL AND MEMORY and the number of Free buffers (also under BUFFER POOL AND MEMORY) is zero, you could benefit from more (assuming you have sized innodb_buffer_pool_sizecorrectly on your server. See here.)

    10. As well as memory, SSDs can help too. Much of the performance drop shown on the curve can be attributed to additional IO which is created as the table gets bigger. While a hard drive can do 200 operations per second (IOPS), a typical SSD will do 20K+

Ten ways to improve the performance of large tables in MySQL--转载的更多相关文章

  1. Five Invaluable Techniques to Improve Regex Performance

    Regular expressions are powerful, but with great power comes great responsibility. Because of the wa ...

  2. 8 ways to improve ASP.NET Web API performance

    ASP.NET Web API is a great piece of technology. Writing Web API is so easy that many developers don’ ...

  3. Effective Modern C++ 42 Specific Ways to Improve Your Use of C++11 and C++14

    Item 1: Understand template type deduction. Item 2: Understand auto type deduction. Item 3: Understa ...

  4. to improve sqlite performance

    INSERT is really slow - I can only do few dozen INSERTs per second http://www.sqlite.org/faq.html#q1 ...

  5. build a real-time analytics dashboard to visualize the number of orders getting shipped every minute to improve the performance of their logistics for an e-commerce portal

    https://cloudxlab.com/blog/real-time-analytics-dashboard-with-apache-spark-kafka/

  6. LMAX Disruptor – High Performance, Low Latency and Simple Too 转载

    原文地址:http://www.symphonious.net/2011/07/11/lmax-disruptor-high-performance-low-latency-and-simple-to ...

  7. Packet for query is too large(1767212 > 1048576)mysql在存储图片时提示图片过大

    原网址:http://blog.csdn.net/bigbird2012/article/details/6304417 错误现象:Packet for query is too large(1767 ...

  8. Why MySQL could be slow with large tables ?

    https://www.percona.com/blog/2006/06/09/why-mysql-could-be-slow-with-large-tables/

  9. 关于数据库报Packet for query is too large (1986748 > 1048576)(mysql写入数据过大)的解决办法

    方法2 (很妥协,很纠结的办法) 进入mysql server 在mysql 命令行中运行 set global max_allowed_packet = 2*1024*1024*10 然后关闭掉这此 ...

随机推荐

  1. 脚本实现自动化安装lamp&lnmp

    #备注:前提是将lnmp和lnmp自动化脚本写好放在相应的路径, 脚本已写好,请查看我博客中的 shell脚本 专栏! #!/bin/bash #安装lamp或者lnmp path=/server/s ...

  2. 记VS2008初始化Microsoft Visual Sourcesafe源码管理提供程序时失败

    之前vs2008开发一直都好好,不知道为什么如今的vs2008打开和关闭都老是弹出这这个初始化Microsoft Visual Sourcesafe源码管理提供程序时失败的错误 解决方法非常easy: ...

  3. Tiling POJ 2506 【大数】

    id=2506">http://poj.org/problem?id=2506 Description In how many ways can you tile a 2xn rect ...

  4. HDU 5375 Gray code (简单dp)

    题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=5375 题面: Gray code Time Limit: 2000/1000 MS (Java/Oth ...

  5. 一入python深似海--对象的属性

    Python中一切皆是对象,每一个对象都能够有多个属性.Python是怎样管理这些属性呢?我们来探讨一下. 属性的__dict__系统 对象的属性包括两部分:类属性和对象属性.对象的属性可能来自于其类 ...

  6. 正确地在QML应用中使用fontsize

    我们知道我们有时须要显示text文本.可是,在QML应用中.我们应该怎样选择font的大小呢?在今天的这篇文章中,我们将展示在Ubuntu平台中的不同文字的大小.我们能够通过FontUtils来帮我们 ...

  7. 63.note.js之 Mongodb在Nodejs上的配置及session会话机制的实现

    转自:https://www.cnblogs.com/alvin_xp/p/4751784.html 1.第一步安装mongodb数据库,这直接官网下载,这里不介绍. 2.也可以使用npm实现直接下载 ...

  8. Elasticsearch之REST

    REST 简介-定义 REST (REpresentation State Transfer)描述了一个架构样式的网络系统,比如 web 应用程序.它首次出现在 2000 年 Roy Fielding ...

  9. JCameraView 仿微信拍照Android控件(点击拍照,长按录小视频)

    JCameraView 控件介绍 这是一个模仿微信拍照的Android开源控件,主要的功能有如下: 点击拍照. 前后摄像头的切换. 长按录视频(视频长度为10秒内). 长按录视频的时候,手指上滑可以放 ...

  10. 一台服务器安装运行多个Tomcat及注册服务

    项目需要,自己配置了一下,顺便分享出来. 1.下载对应版本Tomcat,这里下载Tomcat7.0.65.zip; 下载地址:http://archive.apache.org/dist/tomcat ...