Ten ways to improve the performance of large tables in MySQL--转载
原文地址:http://www.tocker.ca/2013/10/24/improving-the-performance-of-large-tables-in-mysql.html
Today I wanted to take a look at improving the performance of tables that cause performance problems based largely on their size. Some of this advice also applies to databases that are large in-aggregate over many tables, but I always find the individually large table a special-case that is problematic.
What you will normally find is that the speed that the table can be modified will trend down as the size increases. Here is what I am going to call the typical B+Tree index performance over time:
So we should expect degradation of performance due to the structure of the index, but there are actually some ways that we can try and stretch out the curve, and not degrade as quickly.
Ten potential ways to reduce large table impact:
Make sure to use InnoDB instead of MyISAM. MyISAM can be faster at inserts to the end of a table, but it has both table locking (limiting updates and deletes) and uses a single lock to protect the key buffer when loading data to/from disk, resulting in contention. It also does not have the change buffering feature described just below.
InnoDB has change buffering (previously called the insert buffer), which is a feature to delay building secondary indexes that are not unique, and merge writes. It's further described by Facebook here. It's not shown in the graph above, but it can boost insert performance by quite a lot, and it's enabled by default. It was greatly improved in MySQL 5.5, so it is time to upgrade if you haven't.
Partitioning may reduce the size of indexes, effectively reducing the table
into many smaller tables. It also reduces internal index->lockcontention, something that has been greatly improved in the MySQL 5.7.2 DMR.Use innodb page compression. For some workloads (particularly those with lots of char/varchar/text data types) compression will allow the data to be more compact, stretching out that performance curve for longer. It may also allow you to more easily justify SSDs which are typically smaller in capacity. InnoDB page compression was improved a lot in MySQL 5.6, courtesy of Facebook providing a series of patches.
Sort and bulk load data into tables. Inserting in order will result in fewer page splits (which will perform worse on tables not in memory), and the bulk loading is not specifically related to the table size, but it will help reduce redo log pressure.
Remove any unnecessary indexes on the table, paying particular attention to UNIQUE indexes as these disable change buffering. Don't use a UNIQUE index if you have no reason for that constraint; prefer a regular INDEX.
Related to the points 5 & 6, the type of primary key also matters. It is much better to use either an
INT
orBIGINT
datatype than say aGUID
, which will have a curve that degrades much faster. Having no PRIMARY KEY will also affect performance negatively.If bulk loading a fresh table, delay creating any indexes besides the PRIMARY KEY. If you create them once all data is loaded, then InnoDB is able to apply a pre-sort and bulk load process which is both faster and results in typically more compact indexes. This optimization became true in MySQL 5.5.
More memory can actually help here too. I frequently see people under spec memory on new database servers compared to what it actually costs these days. Simple advice: If
SHOW ENGINE INNODB STATUS
shows any reads/s underBUFFER POOL AND MEMORY
and the number of Free buffers (also underBUFFER POOL AND MEMORY
) is zero, you could benefit from more (assuming you have sizedinnodb_buffer_pool_size
correctly on your server. See here.)As well as memory, SSDs can help too. Much of the performance drop shown on the curve can be attributed to additional IO which is created as the table gets bigger. While a hard drive can do 200 operations per second (IOPS), a typical SSD will do 20K+
Ten ways to improve the performance of large tables in MySQL--转载的更多相关文章
- Five Invaluable Techniques to Improve Regex Performance
Regular expressions are powerful, but with great power comes great responsibility. Because of the wa ...
- 8 ways to improve ASP.NET Web API performance
ASP.NET Web API is a great piece of technology. Writing Web API is so easy that many developers don’ ...
- Effective Modern C++ 42 Specific Ways to Improve Your Use of C++11 and C++14
Item 1: Understand template type deduction. Item 2: Understand auto type deduction. Item 3: Understa ...
- to improve sqlite performance
INSERT is really slow - I can only do few dozen INSERTs per second http://www.sqlite.org/faq.html#q1 ...
- build a real-time analytics dashboard to visualize the number of orders getting shipped every minute to improve the performance of their logistics for an e-commerce portal
https://cloudxlab.com/blog/real-time-analytics-dashboard-with-apache-spark-kafka/
- LMAX Disruptor – High Performance, Low Latency and Simple Too 转载
原文地址:http://www.symphonious.net/2011/07/11/lmax-disruptor-high-performance-low-latency-and-simple-to ...
- Packet for query is too large(1767212 > 1048576)mysql在存储图片时提示图片过大
原网址:http://blog.csdn.net/bigbird2012/article/details/6304417 错误现象:Packet for query is too large(1767 ...
- Why MySQL could be slow with large tables ?
https://www.percona.com/blog/2006/06/09/why-mysql-could-be-slow-with-large-tables/
- 关于数据库报Packet for query is too large (1986748 > 1048576)(mysql写入数据过大)的解决办法
方法2 (很妥协,很纠结的办法) 进入mysql server 在mysql 命令行中运行 set global max_allowed_packet = 2*1024*1024*10 然后关闭掉这此 ...
随机推荐
- Swift学习笔记(13)--属性 (Properties)
普通属性用var和let即可,本文不做详述 1.延迟存储属性 延迟存储属性是指当第一次被调用的时候才会计算其初始值的属性.在属性声明前使用@lazy来标示一个延迟存储属性. class DataImp ...
- [Javascript] Specify this using .call() or .apply()
A function's this argument is usually set implicitly, depending on how the function is called. Ordin ...
- HBase概念学习(八)开发一个类twitter系统之表设计
这边文章先将可能的需求分析一下,设计出HBase表,下一步再開始编写client代码. TwiBase系统 1.背景 为了加深HBase基本概念的学习,參考HBase实战这本书实际动手做了这个样例. ...
- Scala学习之爬豆瓣电影
简单使用Scala和Jsoup对豆瓣电影进行爬虫,技术比較简单易学. 写文章不易,欢迎大家採我的文章,以及给出实用的评论,当然大家也能够关注一下我的github:多谢. 1.爬虫前期准备 找好须要抓取 ...
- 使用PHP中的curl发送请求
使用CURL发送请求的基本流程 使用CURL的PHP扩展完成一个HTTP请求的发送一般有以下几个步骤: 初始化连接句柄: 设置CURL选项: 执行并获取结果: 释放VURL连接句柄. 下面的程序片段是 ...
- command---调用指定的指令并执行
command命令调用指定的指令并执行,命令执行时不查询shell函数.command命令只能够执行shell内部的命令. 语法 command(参数) 参数 指令:需要调用的指令及参数. 实例 使用 ...
- 如何在win10上同时安装python2和python3
哎,其实本人已经用惯了python2,听说python3的语法有很多不一样的地方,那我之前写的算法改起来岂不是日了狗了吗?所以一直没改用python3.但是谷歌的那个TensorFlow,在windo ...
- 学习WWDC的好资源!
学习WWDC的好资源. 大家都知道.要看Apple每年一度的WWDC,仅仅要到它的Developer站点去就能够了.那里有每年的研讨会视频,并且还能够下载每一个视频的SD或HD视频文件,以及相关的演示 ...
- Service-监听手机来电
public class MonitorPhone extends Activity { TelephonyManager tManager; @Override protected void onC ...
- BZOJ5020: [THUWC 2017]在美妙的数学王国中畅游(LCT,泰勒展开,二项式定理)
Description 数字和数学规律主宰着这个世界. 机器的运转, 生命的消长, 宇宙的进程, 这些神秘而又美妙的过程无不可以用数学的语言展现出来. 这印证了一句古老的名言: ...