Archive MySQL Data In Chunks Using Stored Procedure
sqladminon September 26, 2018
In a DBA’s day to day activities, we are doing Archive operation on our transnational database servers to improve your queries and control the Disk space. The archive is a most expensive operation since its involved a huge number of Read and Write will be performed. So its mandatory to run the archive queries in chunks. The archive is depended on business use. Many of us need a copy of the data on an archive database to refer later. To perform the archive we can just simply run the delete query with the limit. But we need to run the query again and again until the matched rows count is 0. We can create a procedure to do this in a while loop. I have created one such procedure to archive many tables.
Why Archive is an expensive operation?
Generally how we are arching the data is delete from table_name where column_name <= some_value; If you are running on a table which needs to be deleted around 15million records, then you need the undo log file to hold all of these records. There will be a heavy IO happening in the Disk. And lt’ll lock the rows and some other locks will be held until the Archive complete. Replication may delay because of this.
When Archive is going to mess up the production?
- Running archive commands on a heavy traffic time.
- Archive without a proper where clause.
- Delete data without limit.
- Performing archive contrition on a not indexed column.
- Continuously run the delete query in chunks on a replication environment. {without sleep(1 or few seconds}.
How to perform the archive properly?
- To do this, the first condition is use limit in the delete.
- Create an index on the where clause.
- At least do sleep 1sec for each chuck which will be good for a replication infra.
- Set autocommit=1
- Optional: Set transaction isolation to Read Committed.
- Do not mention the number of loops without knowing the actual loop counts to process the complete delete.
My approach to this:
Inspired by Rick James’s Blog, I have prepared a single stored procedure to perform archive on multiple tables. We just need to pass the table name, date column and then date to archive. I have tested with datetime and Primary key column.
Archive a single table:
The below procedure will perform delete on table test and remove older than 10 days records.
use sqladmin; DROP PROCEDURE
IF EXISTS archive;
delimiter //
CREATE PROCEDURE
archive()
begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
DELETE
FROM test
WHERE dop < DATE(Date_sub(Now(), INTERVAL 10 day))
LIMIT 10000;
SET rows = row_count();
select sleep(1);
commit;
END WHILE;
END //
delimiter ;
Archive multiple tables:
This procedure will help you to archive multiple tables, you just need to pass the table name, column name and the date for the archive. I love to use this
use sqladmin; DROP PROCEDURE
IF EXISTS sqladmin_archive;
delimiter //
CREATE PROCEDURE
sqladmin_archive(IN archive_dbname varchar(100), IN archive_table varchar(100), IN archive_column varchar(100), IN archive_date varchar(100)) begin
DECLARE rows INT;
DECLARE rows_deleted INT;
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET rows = 1;
SET rows_deleted = 10000;
WHILE rows > 0
do
SET autocommit=1;
SET @query =CONCAT('DELETE FROM ',archive_dbname,'.',archive_table,' WHERE ',archive_column,' <= "',archive_date ,'" LIMIT 10000;');
PREPARE arcive_stmt FROM @query;
EXECUTE arcive_stmt;
SET rows = row_count();
SET rows = row_count();
select sleep(1);
commit;
DEALLOCATE PREPARE arcive_stmt;
END WHILE;
END //
delimiter ; -- Execute this procedure
CALL sqladmin_archive ('mydb','test_table','created_at','2018-09-12');
Take dump before archive with where clause:
This script is my favorite one, but this depends on the above stored procedure. This shell script will take the dump of the table with where clause of the date that we want to archive. You can customize this as per your requirement.
#!/bin/bash # pass variables
archive_dbname=$1
archive_table=$2
archive_column=$3
days_to_archive=$4
archive_date="'"`date +'%Y-%m-%d' --date="-$days_to_archive day"`"'"
where_clause=$archive_column'<='$archive_date
dump_file=$archive_table_`date +'%Y-%m-%d' --date="-$days_to_archive day"`".sql" # Dump the table
echo "DUMP Starting for the table $archive_table ....."
mysqldump -u root $archive_dbname $archive_table --where=$where_clause > $dump_file
echo "DUMP Done......" # Archive the data
echo "Deleting the data on the table $archive_table ....."
mysql -u root sqladmin -e"CALL sqladmin_archive('$archive_dbname','$archive_table','$archive_column',$archive_date);"
echo "Deleting is Done ....."
Example Archive:
This example, Im going to archive a table called test. The column started_at contains the timestamp value. I want to remove older than 15 days data in the table. This table is located in the database name called sqladmin.
./archive_script.sh sqladmin test started_at 15
Archive MySQL Data In Chunks Using Stored Procedure的更多相关文章
- JDBC连接执行 MySQL 存储过程报权限错误:User does not have access to metadata required to determine stored procedure parameter types. If rights can not be granted,
国内私募机构九鼎控股打造APP,来就送 20元现金领取地址:http://jdb.jiudingcapital.com/phone.html 内部邀请码:C8E245J (不写邀请码,没有现金送) 国 ...
- Net连接mysql的公共Helper类MySqlHelper.cs带MySql.Data.dll下载
MySqlHelper.cs代码如下: using System; using System.Collections.Generic; using System.Linq; using System. ...
- How To Call Stored Procedure In Hibernate
How To Call Stored Procedure In Hibernate In this tutorial, you will learn how to call a store proce ...
- [转]Mapping Stored Procedure Parameters in SSIS OLE DB Source Editor
本文转自:http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-o ...
- 关于Linux和Windows下部署mysql.data.dll的注册问题
mysql ado.net connector下载地址: http://dev.mysql.com/downloads/connector/net/ 选择版本: Generally Available ...
- Stored Procedure 里的 WITH RECOMPILE 到底是干麻的?
在 SQL Server 创建或修改「存储过程(stored procedure)」时,可加上 WITH RECOMPILE 选项,但多数文档或书籍都写得语焉不详,或只解释为「每次执行此存储过程时,都 ...
- [转]Dynamic SQL & Stored Procedure Usage in T-SQL
转自:http://www.sqlusa.com/bestpractices/training/scripts/dynamicsql/ Dynamic SQL & Stored Procedu ...
- Retrieving Out Params From a Stored Procedure With Python
http://www.rodneyoliver.com/blog/2013/08/08/retrieving-out-params-from-a-stored-procedure-with-pytho ...
- [转]Easy Stored Procedure Output Oracle Select
本文转自:http://www.oraclealchemist.com/oracle/easy-stored-procedure-output/ I answered a question on a ...
随机推荐
- TensorFlow object detection API应用--配置
目标检测在图形识别的基础上有了更进一步的应用,但是代码也更加繁琐,TensorFlow专门为此开设了一个object detection API,接下来看看怎么使用它. object detectio ...
- tensorflow 根据节点名称获取节点
ckpt_file = os.path.join(self.args.bert_dir, 'model.ckpt-6123') # 获取图的reader reader = self.tf_instan ...
- 第一章 Java Web工作原理
一:在本章我们将学到如下的内容 >HTTP协议原理 >服务器端Web编程原理 >Servlet与Web容器 >Java Web应用程序的组成 >Tomcat介绍 一:1. ...
- 使用vertical-align实现垂直对齐
关于垂直对齐,之前研究过好几次了,但感觉每次都没研究透彻,做了几个效果,就觉得自己掌握了,实在是自欺欺人.真乃搞技术的大忌. 这两天又下定决心重新开始研究vertical-allign这个高深莫测的属 ...
- UVA 1605 Building for UN(思维)
题目链接: https://cn.vjudge.net/problem/UVA-1605#author=0 /* 问题 设计一个包含若干层的联合国大厦,其中每一层都是等大的网格,每个格子分配给一个国家 ...
- 状态压缩·一(状态压缩DP)
描述 小Hi和小Ho在兑换到了喜欢的奖品之后,便继续起了他们的美国之行,思来想去,他们决定乘坐火车前往下一座城市——那座城市即将举行美食节! 但是不幸的是,小Hi和小Ho并没有能够买到很好的火车票—— ...
- Linux分区方式及关闭iptables和selinux的方式
分区方式一般有三种 第一种:数据不是很重要 /boot(系统的引导分区): 系统引导的信息/软件 系统的内核 200M swap( 交换分区): 为了避免系统内存用光了导致系统 宕机 如果系统内存 ...
- SQL Server T—SQL 视图 事务
一 视图 视图是存储在数据库中的查询的SQL 语句, 视图是从一个或多个表或视图中导出的表,是一张虚表,只能对视图进行查询,不能增.删.改. 对视图进行修改要在相应的基本表中进行修改,修改会自动的反应 ...
- Java基础——详尽说明try-catch-finally的用法
问:Java异常处理机制,理解了吗?Java异常处理,真的掌握了吗?什么是自定义异常?catch体里遇到return 是怎么处理?finally 体里有return怎么处理?catch 和 final ...
- [转载] Spring框架——AOP前置、后置、环绕、异常通知
通知类型: 步骤: 1. 定义接口 2. 编写对象(被代理对象=目标对象) 3. 编写通知(前置通知目标方法调用前调用) 4. 在beans.xml文件配置 4.1 配置 被代理对象=目标对象 4.2 ...