MySQL主从同步延迟

早上接到open-falcon报警，一台mysql从库同步延迟2w多秒，mysql版本比较老，用的5.1.37。

连接从库查找原因：

　　

show processlist一下，查看哪些线程在跑。

看到Time=25565，也就是说这个线程保持当前状态25565秒，一直在执行Updating操作。怀疑是由php分析数据脚本引起的，把这条语句发给开发，问问执行的是什么操作，开发说是每个用户登录，游戏角色等级提高就执行一次update操作，又问开发有没有脚本执行这个update语句，开发也说不清。。没办法，只能去脚本机搜索定时任务中php脚本了，脚本非常多crontab -l 列出所有脚本，配合awk，grep搜索所有php脚本，然后过滤脚本中内容update，有几个是有update操作的，逐个分析，并查看脚本记录的日志。执行tail -f /opt/wwwroot/daemon/run/datasource/newdata/fast_channel_run.log时，发现了大量update的记录，又向开发确认下，原因基本确定，让开发优化php脚本了。

解决思路：

　　给开发提建议，优化程序逻辑，多个update合成一条语句执行。

降低update频率

　　升级mysql版本，mysql-5.1性能比新版差的太多，但短期不现实。

　　实在延迟跟不上了，重做了salve。

下面是show processlist的一些说明，摘自官方文档。

13.7.5.30. SHOW PROCESSLIST Syntax

SHOW [FULL] PROCESSLIST

SHOW PROCESSLIST shows you which threads are running. You can also get this information from the INFORMATION_SCHEMA PROCESSLIST table or the mysqladmin processlist command. If you have the PROCESS privilege, you can see all threads. Otherwise, you can see only your own threads (that is, threads associated with the MySQL account that you are using). If you do not use the FULL keyword, only the first 100 characters of each statement are shown in the Info field.

Process information is also available from the performance_schema.threads table. However, access to threads does not require a mutex and has minimal impact on server performance. INFORMATION_SCHEMA.PROCESSLIST and SHOW PROCESSLIST have negative performance consequences because they require a mutex. threads also shows information about background threads, which INFORMATION_SCHEMA.PROCESSLIST and SHOW PROCESSLIST do not. This means that threads can be used to monitor activity the other thread information sources cannot.

The SHOW PROCESSLIST statement is very useful if you get the “too many connections” error message and want to find out what is going on. MySQL reserves one extra connection to be used by accounts that have the SUPER privilege, to ensure that administrators should always be able to connect and check the system (assuming that you are not giving this privilege to all your users).

Threads can be killed with the KILL statement. See Section 13.7.6.4, “KILL Syntax”.

Here is an example of SHOW PROCESSLIST output:

mysql> SHOW FULL PROCESSLIST\G
*************************** 1. row ***************************
Id: 1
User: system user
Host:
db: NULL
Command: Connect
Time: 1030455
State: Waiting for master to send event
Info: NULL

The columns produced by SHOW PROCESSLIST have the following meanings:

• Id

  The connection identifier.

• User

  The MySQL user who issued the statement. If this is system user, it refers to a nonclient thread spawned by the server to handle tasks internally. This could be the I/O or SQL thread used on replication slaves or a delayed-row handler. unauthenticated user refers to a thread that has become associated with a client connection but for which authentication of the client user has not yet been done. event_scheduler refers to the thread that monitors scheduled events. For system user, there is no host specified in the Host column.

• Host

  The host name of the client issuing the statement (except for system user where there is no host). SHOW PROCESSLIST reports the host name for TCP/IP connections in host_name:client_port format to make it easier to determine which client is doing what.

• db

  The default database, if one is selected, otherwise NULL.

• Command

  The type of command the thread is executing. For descriptions for thread commands, see Section 8.12.5, “Examining Thread Information”. The value of this column corresponds to the COM_xxx commands of the client/server protocol and Com_xxx status variables. See Section 5.1.6, “Server Status Variables”

• Time

  The time in seconds that the thread has been in its current state.

• State

  An action, event, or state that indicates what the thread is doing. Descriptions for State values can be found at Section 8.12.5, “Examining Thread Information”.

  Most states correspond to very quick operations. If a thread stays in a given state for many seconds, there might be a problem that needs to be investigated.

  For the SHOW PROCESSLIST statement, the value of State is NULL.

• Info

  The statement the thread is executing, or NULL if it is not executing any statement. The statement might be the one sent to the server, or an innermost statement if the statement executes other statements. For example, if a CALL statement executes a stored procedure that is executing a SELECT statement, the Info value shows the SELECT statement.