生产环境zabbix3.2上亿的表数据通过表分区的方式进行历史数据清理
生产环境zabbix3.2上亿的表数据通过表分区的方式进行历史数据清理 zabbix服务器经常报警io过载，在报警的时候发现是数据库在删除历史数据时耗时较长
数据库积攒了大量的历史数据信息，主要集中在zabbix的history、history_uint、history_str、history_text、trends、trends_uint这几个表中
需要进行清理，两种清理方式：
.直接清空表，最省事，速度最快，缺陷是会丢失所有的监控历史数据 具体清理语句如下：
use zabbix;
truncate table history;
truncate table history_uint;
truncate table history_log;
truncate table history_str;
truncate table history_text;
truncate table trends;
truncate table trends_uint; truncate table event_recovery;
truncate table events;(有外键关联，无法直接清空表)
解决办法：
mysql> truncate table events;
ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (`zabbix`.`acknowledges`, CONSTRAINT `c_acknowledges_2` FOREIGN KEY (`eventid`) REFERENCES `zabbix`.`events` (`eventid`)) mysql> SET foreign_key_checks=0;
mysql> truncate table events;
Query OK, 0 rows affected (3.37 sec) mysql> SET foreign_key_checks=1;
Query OK, 0 rows affected (0.13 sec) .删除一段时间以前的数据，速度慢，可以保留需要的数据
相对折中的解决方案是先将表分区，然后清理表分区，这样处理相对比较稳妥，清理速度也较快 清理前先备份数据库：
.备份脚本 # cat /opt/mysql_bak.sh
#!/usr/bin/env bash # Program: MySQL 增量备份脚本 使用 percona xtrabackup
# Author : jack
# Date : --
# update :2017.11. by tom
##进入程序目录
cd /usr/local/worksh/xtrabackup_cron/bin
# 读取配置文件中的所有变量值, 设置为全局变量
# 配置文件
conf_file="../conf/mysql_increment_hot_backup.conf"
# mysql 用户
user=`sed '/^user=/!d;s/.*=//' $conf_file`
# mysql 密码
#password=`sed '/^password=/!d;s/.*=//' $conf_file`
password="pass" # mysql 备份目录
backup_dir=`sed '/^backup_dir=/!d;s/.*=//' $conf_file`
# percona-xtrabackup 备份软件路径
xtrabackup_dir=`sed '/^xtrabackup_dir=/!d;s/.*=//' $conf_file`
# 全备是在一周的第几天
full_backup_week_day=`sed '/^full_backup_week_day=/!d;s/.*=//' $conf_file`
# mysql 全备前缀标识
full_backup_prefix=`sed '/^full_backup_prefix=/!d;s/.*=//' $conf_file`
# mysql 增量备前缀标识
increment_prefix=`sed '/^increment_prefix=/!d;s/.*=//' $conf_file`
# mysql 配置文件
mysql_conf_file=`sed '/^mysql_conf_file=/!d;s/.*=//' $conf_file`
# 备份错误日志文件
error_log=`sed '/^error_log=/!d;s/.*=//' $conf_file`
# 备份索引文件
index_file=`sed '/^index_file=/!d;s/.*=//' $conf_file` # 备份日期
backup_date=`date +%F`
# 备份日期
backup_time=`date +%H-%M-%S`
# 备份日期
backup_week_day=`date +%u` # 设置备份线程数
backup_thread= # 创建相关目录
log_dir=../log
var_dir=../var
mkdir -p $backup_dir
mkdir -p $log_dir
mkdir -p $var_dir # 全量备份
function full_backup() {
backup_folder=${full_backup_prefix}_${backup_date}_${backup_time}_${backup_week_day} mkdir -p $backup_dir/$backup_folder
$xtrabackup_dir/bin/innobackupex --defaults-file=$mysql_conf_file --user=$user --password=$password --parallel=$backup_thread --no-timestamp $backup_dir/$backup_folder > $log_dir/${backup_folder}.log >&
return $?
} # 增量备份
function increment_backup() {
backup_folder=${increment_prefix}_${backup_date}_${backup_time}_${backup_week_day}
incr_base_folder=`sed -n '$p' $index_file | \
awk -F '[, {}]*' '{print $3}' | \
awk -F ':' '{print $2}'` mkdir -p $backup_dir/$backup_folder
$xtrabackup_dir/bin/innobackupex \
--defaults-file=$mysql_conf_file \
--user=$user \
--password=$password \
--no-timestamp \
--incremental \
$backup_dir/$backup_folder \
--incremental-basedir=$backup_dir/$incr_base_folder > $log_dir/${backup_folder}.log >&
return $?
} # 删除之前的备份(一般在全备完成后使用)
function delete_before_backup() {
cat $index_file | awk -F '[, {}]*' '{print $3}' | \
awk -v backup_dir=$backup_dir -F ':' '{if($2!=""){printf("rm -rf %s/%s\n", backup_dir, $2)}}' | \
/bin/bash cat $index_file | awk -F '[, {}]*' '{print $3}' | \
awk -v log_dir=$log_dir -F ':' '{if($2!=""){printf("rm -rf %s/%s.log\n", log_dir, $2)}}' | \
/bin/bash
} # 备份索引文件
function backup_index_file() {
cp $index_file ${index_file}_$(date -d "1 day ago" +%F)
} # 备份索引文件
function send_index_file_to_remote() {
echo 'send index file ok'
} # 添加索引, 索引记录了当前最新的备份
function append_index_to_file() {
echo "{week_day:$backup_week_day, \
dir:${}_${backup_date}_${backup_time}_${backup_week_day}, \
type:${}, \
date:${backup_date}}" >> $index_file
} # 记录 错误消息到文件
function logging_backup_err() {
echo "{week_day:$backup_week_day, \
dir:${}_${backup_date}_${backup_time}_${backup_week_day}, \
type:${}, \
date:${backup_date}}" >> $error_log
} # 清空索引
function purge_index_from_file() {
> $index_file
} # 清空错误日志信息
function purge_err_log() {
> $error_log
} # 打包备份
function tar_backup_file() {
echo "tar $1 ok"
} # 发送备份到远程
function send_backup_to_remote() {
echo "send $1 remote ok"
} # 判断是应该全备还是增量备份
# :full, :incr
function get_backup_type() {
full_backup_week_day=`sed '/^full_backup_week_day=/!d;s/.*=//' $conf_file`
backup_type=
if [ "$full_backup_week_day" -eq `date +%u` ]; then
backup_type=
else
backup_type=
fi
if [ ! -n "`cat $index_file`" ]; then
backup_type=
fi
return $backup_type
} # 测试配置文件正确性
function test_conf_file() {
# 判断每个变量是否在配置文件中有配置，没有则退出程序
if [ ! -n "$user" ]; then echo 'fail: configure file user not set'; exit ; fi
if [ ! -n "$password" ]; then echo 'fail: configure file password not set'; exit ; fi
if [ ! -n "$backup_dir" ]; then echo 'fail: configure file backup_dir not set'; exit ; fi
if [ ! -n "$full_backup_week_day" ]; then echo 'fail: configure file full_backup_week_day not set'; exit ; fi
if [ ! -n "$full_backup_prefix" ]; then echo 'fail: configure file full_backup_prefix not set'; exit ; fi
if [ ! -n "$increment_prefix" ]; then echo 'fail: configure file increment_prefix not set'; exit ; fi
if [ ! -n "$mysql_conf_file" ]; then echo 'fail: configure file mysql_conf_file not set'; exit ; fi
if [ ! -n "$error_log" ]; then echo 'fail: configure file error_log not set'; exit ; fi
if [ ! -n "$index_file" ]; then echo 'fail: configure file index_file not set'; exit ; fi
} # 执行
function run() {
# 检测配置文件值
test_conf_file # 判断是执行全备还是曾量备份
get_backup_type
backup_type=$?
case $backup_type in
)
# 全量备份
full_backup
backup_ok=$?
if [ -eq "$backup_ok" ]; then
# 全备成功
# # 打包最新备份
# tar_backup_file $full_backup_prefix
# # 将tar备份发送到远程
# send_backup_to_remote $full_backup_prefix
# 备份索引文件
backup_index_file
# # 发送索引文件到远程
# send_index_file_to_remote
# 清除之前的备份
delete_before_backup
# 清除索引文件
purge_index_from_file
# 添加索引, 索引记录了当前最新的备份
append_index_to_file $full_backup_prefix
else
# 全备失败
# 删除备份目录
rm -rf ${backup_dir}/${full_backup_prefix}_${backup_date}_${backup_time}_${backup_week_day}
# 记录错误日志
logging_backup_err $full_backup_prefix
fi
;;
)
# 增量备份
increment_backup
backup_ok=$?
if [ -eq "$backup_ok" ]; then
# 增量备份成功
# # 打包最新备份
# tar_backup_file $increment_prefix
# # 将tar备份发送到远程
# send_backup_to_remote $increment_prefix
# 添加索引, 索引记录了当前最新的备份
append_index_to_file $increment_prefix
else
# 增量备份失败
# 删除备份目录
rm -rf ${backup_dir}/${full_backup_prefix}_${backup_date}_${backup_time}_${backup_week_day}
# 记录错误日志
logging_backup_err $increment_prefix
fi
;;
esac
} run
chown -R apache.users /data/MySQL_Data_Backup .备份数据库脚本需要用到的配置 # 授权语句
show grants for xtrabackup@'localhost';
GRANT RELOAD, PROCESS, SUPER, LOCK TABLES, REPLICATION CLIENT, CREATE TABLESPACE ON *.* TO 'xtrabackup'@'localhost' identified by 'pass'; [root:/data/MySQL_Data_Backup]# cat /usr/local/worksh/xtrabackup_cron/conf/mysql_increment_hot_backup.conf
# mysql 用户名
user=xtrabackup # mysql 密码
password="pass" # 备份存放路径
backup_dir=/data/mysql_db_backup # percona-xtrabackup 备份软件路径
xtrabackup_dir=/usr # 全备是在一周的第几天，可以根据当前时间选择备份的时机，比如今天是星期一，需要全量备份就可以把数字改为1
full_backup_week_day= # 全量备信息名称 前缀
full_backup_prefix=full # 增量备信息名称 前缀
increment_prefix=incr # mysql配置文件
mysql_conf_file=/etc/my.cnf # 错误日志文件(更具此文件知道备份是否成功)
# format:
# {week_day:,dir:full/incr_2015--29_00--00_7,type:full/incr,date:--}
error_log=../var/mysql_increment_hot_backup.err # 索引文件
# format:
# {week_day:,dir:full/incr_2015--29_00--00_7,type:full/incr}
index_file=../var/mysql_increment_hot_backup.index .查表的整体情况 # 线上eus_zabbix数据库的记录 mysql> select table_name, (data_length+index_length)// as total_mb, table_rows from information_schema.tables where table_schema='zabbix';
+----------------------------+----------------+------------+
| table_name | total_mb | table_rows |
+----------------------------+----------------+------------+
| acknowledges | 0.06250000 | |
| actions | 0.04687500 | |
| alerts | 5.68750000 | |
| application_discovery | 0.04687500 | |
| application_prototype | 0.04687500 | |
| application_template | 0.39062500 | |
| applications | 0.35937500 | |
| auditlog | 2.01562500 | |
| auditlog_details | 0.26562500 | |
| autoreg_host | 0.03125000 | |
| conditions | 0.03125000 | |
| config | 0.04687500 | |
| corr_condition | 0.03125000 | |
| corr_condition_group | 0.03125000 | |
| corr_condition_tag | 0.01562500 | |
| corr_condition_tagpair | 0.01562500 | |
| corr_condition_tagvalue | 0.01562500 | |
| corr_operation | 0.03125000 | |
| correlation | 0.01562500 | |
| dbversion | 0.01562500 | |
| dchecks | 0.03125000 | |
| dhosts | 0.03125000 | |
| drules | 0.04687500 | |
| dservices | 0.04687500 | |
| escalations | 0.03125000 | |
| event_recovery | 231.89062500 | |
| event_tag | 0.03125000 | |
| events | 1561.00000000 | |
| expressions | 0.03125000 | |
| functions | 2.01562500 | |
| globalmacro | 0.01562500 | |
| globalvars | 0.01562500 | |
| graph_discovery | 0.12500000 | |
| graph_theme | 0.03125000 | |
| graphs | 1.96875000 | |
| graphs_items | 1.79687500 | |
| group_discovery | 0.03125000 | |
| group_prototype | 0.06250000 | |
| groups | 0.03125000 | |
| history | 19667.17187500 | |
| history_log | 0.01562500 | |
| history_str | 777.90625000 | |
| history_text | 654.35937500 | |
| history_uint | 45977.17187500 | |
| host_discovery | 0.04687500 | |
| host_inventory | 0.01562500 | |
| hostmacro | 0.03125000 | |
| hosts | 0.20312500 | |
| hosts_groups | 0.04687500 | |
| hosts_templates | 0.10937500 | |
| housekeeper | 0.06250000 | |
| httpstep | 0.03125000 | |
| httpstepitem | 0.04687500 | |
| httptest | 0.07812500 | |
| httptestitem | 0.04687500 | |
| icon_map | 0.04687500 | |
| icon_mapping | 0.04687500 | |
| ids | 0.01562500 | |
| images | 1.53125000 | |
| interface | 0.04687500 | |
| interface_discovery | 0.03125000 | |
| item_application_prototype | 0.04687500 | |
| item_condition | 0.09375000 | |
| item_discovery | 1.76562500 | |
| items | 13.25000000 | |
| items_applications | 4.89062500 | |
| maintenances | 0.01562500 | |
| maintenances_groups | 0.04687500 | |
| maintenances_hosts | 0.04687500 | |
| maintenances_windows | 0.04687500 | |
| mappings | 0.03125000 | |
| media | 0.04687500 | |
| media_type | 0.03125000 | |
| opcommand | 0.03125000 | |
| opcommand_grp | 0.04687500 | |
| opcommand_hst | 0.04687500 | |
| opconditions | 0.03125000 | |
| operations | 0.03125000 | |
| opgroup | 0.04687500 | |
| opinventory | 0.01562500 | |
| opmessage | 0.03125000 | |
| opmessage_grp | 0.04687500 | |
| opmessage_usr | 0.04687500 | |
| optemplate | 0.04687500 | |
| problem | 6.50000000 | |
| problem_tag | 0.04687500 | |
| profiles | 0.50000000 | |
| proxy_autoreg_host | 0.01562500 | |
| proxy_dhistory | 0.01562500 | |
| proxy_history | 0.01562500 | |
| regexps | 0.03125000 | |
| rights | 0.04687500 | |
| screen_user | 0.04687500 | |
| screen_usrgrp | 0.04687500 | |
| screens | 0.04687500 | |
| screens_items | 0.03125000 | |
| scripts | 0.06250000 | |
| service_alarms | 0.04687500 | |
| services | 0.03125000 | |
| services_links | 0.04687500 | |
| services_times | 0.03125000 | |
| sessions | 0.20312500 | |
| slides | 0.04687500 | |
| slideshow_user | 0.04687500 | |
| slideshow_usrgrp | 0.04687500 | |
| slideshows | 0.04687500 | |
| sysmap_element_url | 0.03125000 | |
| sysmap_url | 0.03125000 | |
| sysmap_user | 0.04687500 | |
| sysmap_usrgrp | 0.04687500 | |
| sysmaps | 0.07812500 | |
| sysmaps_elements | 0.09375000 | |
| sysmaps_link_triggers | 0.04687500 | |
| sysmaps_links | 0.06250000 | |
| task | 0.01562500 | |
| task_close_problem | 0.01562500 | |
| timeperiods | 0.01562500 | |
| trends | 1489.54687500 | |
| trends_uint | 2278.60937500 | |
| trigger_depends | 0.04687500 | |
| trigger_discovery | 0.20312500 | |
| trigger_tag | 0.03125000 | |
| triggers | 3.07812500 | |
| users | 0.03125000 | |
| users_groups | 0.04687500 | |
| usrgrp | 0.03125000 | |
| valuemaps | 0.03125000 | |
+----------------------------+----------------+------------+
rows in set (0.01 sec) # 数据库的占用空间情况 mysql> select TABLE_SCHEMA, concat(truncate(sum(data_length)//,),' MB') as data_size,
-> concat(truncate(sum(index_length)//,),'MB') as index_size
-> from information_schema.tables
-> group by TABLE_SCHEMA
-> order by data_length desc; +--------------------+-------------+------------+
| TABLE_SCHEMA | data_size | index_size |
+--------------------+-------------+------------+
| zabbix | 54649.39 MB | .93MB |
| nosql_eye | 0.12 MB | .06MB |
| performance_schema | 0.00 MB | .00MB |
| mysql | 2.43 MB | .21MB |
| information_schema | 0.15 MB | .00MB |
| sys | 0.01 MB | .00MB |
+--------------------+-------------+------------+
rows in set (0.19 sec) # 要关闭zabbix_server 否则可能会出现大量连接不到客户端的报警信息 mysql> show full processlist\G Connection id:
Current database: mysql *************************** . row ***************************
Id:
User: zabbix
Host: 127.0.0.1:
db: zabbix
Command: Query
Time:
State: copy to tmp table
Info: ALTER TABLE history PARTITION BY RANGE( clock ) (
PARTITION p20190524 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-25 00:00:00")),
PARTITION p20190525 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-26 00:00:00")),
PARTITION p20190526 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-27 00:00:00")),
PARTITION p20190527 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-28 00:00:00")),
PARTITION p20190528 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-29 00:00:00")),
PARTITION p20190529 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-30 00:00:00")),
PARTITION p20190530 VALUES LESS THAN (UNIX_TIMESTAMP("2019-05-31 00:00:00")),
PARTITION p20190531 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-01 00:00:00")),
PARTITION p20190601 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-02 00:00:00")),
PARTITION p20190602 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-03 00:00:00")),
PARTITION p20190603 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-04 00:00:00")),
PARTITION p20190604 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-05 00:00:00")),
PARTITION p20190605 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-06 00:00:00")),
PARTITION p20190606 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-07 00:00:00")),
PARTITION p20190607 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-08 00:00:00")),
PARTITION p20190608 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-09 00:00:00")),
PARTITION p20190609 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-10 00:00:00")),
PARTITION p20190610 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-11 00:00:00")),
PARTITION p20190611 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-12 00:00:00")),
PARTITION p20190612 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-13 00:00:00")),
PARTITION p20190613 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-14 00:00:00")),
PARTITION p20190614 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-15 00:00:00")),
PARTITION p20190615 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-16 00:00:00")),
PARTITION p20190616 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-17 00:00:00")),
PARTITION p20190617 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-18 00:00:00")),
PARTITION p20190618 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-19 00:00:00")),
PARTITION p20190619 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-20 00:00:00")),
PARTITION p20190620 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-21 00:00:00")),
PARTITION p20190621 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-22 00:00:00")),
PARTITION p20190622 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-23 00:00:00")),
PARTITION p20190623 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-24 00:00:00")),
PARTITION p20190624 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-25 00:00:00")),
PARTITION p20190625 VALUES LESS THAN (UNIX_TIMESTAMP("2019-06-26 00:00:00"))
)
Rows_sent:
Rows_examined:
*************************** . row ***************************
Id:
User: root
Host: 127.0.0.1:
db: mysql
Command: Query
Time:
State: starting
Info: show full processlist
Rows_sent:
Rows_examined:
rows in set (0.08 sec) mysql> # 在表分区的时候 因为history和history_uint 超过1亿条数据，需要的时间较长
history 接近2亿条数据，下午两点 10分开始 ,到3点11分左右结束，用时1个小时多一点
history_uint 从3点10分开始 5点40左右结束，用时 2个半小时
history_uint | 45977.17187500 | [root@aliyun-american-guigu-zabbix:/opt]# ./zabbix_db_partition.sh
Ready to partition tables. Ready to update permissions of Zabbix user to create routines Enter root DB user: root
Enter root password: pass
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR () at line : Can't find any matching row in the user table Do you want to backup the database (recommended) (Y/n): n
Are you certain you have a backup (y/N):
y Ready to proceed: Starting yearly partioning at:
and ending at:
With days of daily history mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR () at line : Can't find any matching row in the user table Do you want to backup the database (recommended) (Y/n): n
Are you certain you have a backup (y/N):
y Ready to proceed: Starting yearly partioning at:
and ending at:
With days of daily history Ready to proceed (Y/n):
Y
Altering table: history
Altering table: history_log
Altering table: history_str
Altering table: history_text
Altering table: history_uint
Altering table: trends
Altering table: trends_uint
Creating monthly partitions for table: trends
Creating monthly partitions for table: trends_uint
Creating daily partitions for table: history
Creating daily partitions for table: history_log
Creating daily partitions for table: history_str
Creating daily partitions for table: history_text
Creating daily partitions for table: history_uint Ready to apply script to database, this may take a while.(Y/n):
Y
mysql: [Warning] Using a password on the command line interface can be insecure.
Altering tables
history
history_log
history_str
history_text
history_uint
trends
trends_uint
trends
trends_uint
history
history_log
history_str
history_text
history_uint
Installing procedures If Zabbix Version = 2.0
Do you want to update the /etc/zabbix/zabbix_server.conf
to disable housekeeping (Y/n): n Do you want to update the crontab (Y/n): Y
The crontab entry can be either in /etc/cron.daily, or added
to the crontab for root Do you want to add this to the /etc/cron.daily directory (Y/n): Y Enter email of who should get the daily housekeeping reports: zhengjj@wondershare.cn # 分区是将表从物理上分割开 [root@aliyun-american-guigu-zabbix:/data/mysql_data/zabbix]# ll trends*
-rw-r----- mysql mysql Jun : trends.frm
-rw-r----- mysql mysql Jun : trends#P#p201901.ibd
-rw-r----- mysql mysql Jun : trends#P#p201902.ibd
-rw-r----- mysql mysql Jun : trends#P#p201903.ibd
-rw-r----- mysql mysql Jun : trends#P#p201904.ibd
-rw-r----- mysql mysql Jun : trends#P#p201905.ibd
-rw-r----- mysql mysql Jun : trends#P#p201906.ibd
-rw-r----- mysql mysql Jun : trends#P#p201907.ibd
-rw-r----- mysql mysql Jun : trends_uint.frm
-rw-r----- mysql mysql Jun : trends_uint#P#p201901.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201902.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201903.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201904.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201905.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201906.ibd
-rw-r----- mysql mysql Jun : trends_uint#P#p201907.ibd # 依次清理history、history_uint等表分区
# 查看表分区情况
mysql> select partition_name part,partition_expression expr,partition_description descr,table_rows from information_schema.partitions where table_schema=schema() and table_name='history';
+-----------+--------+------------+------------+
| part | expr | descr | table_rows |
+-----------+--------+------------+------------+
| p20190524 | clock | | |
| p20190525 | clock | | |
| p20190526 | clock | | |
| p20190527 | clock | | |
| p20190528 | clock | | |
| p20190529 | clock | | |
| p20190530 | clock | | |
| p20190531 | clock | | |
| p20190601 | clock | | |
| p20190602 | clock | | |
| p20190603 | clock | | |
| p20190604 | clock | | |
| p20190605 | clock | | |
| p20190606 | clock | | |
| p20190607 | clock | | |
| p20190608 | clock | | |
| p20190609 | clock | | |
| p20190610 | clock | | |
| p20190611 | clock | | |
| p20190612 | clock | | |
| p20190613 | clock | | |
| p20190614 | clock | | |
| p20190615 | clock | | |
| p20190616 | clock | | |
| p20190617 | clock | | |
| p20190618 | clock | | |
| p20190619 | clock | | |
| p20190620 | clock | | |
| p20190621 | clock | | |
| p20190622 | clock | | |
| p20190623 | clock | | |
| p20190624 | clock | | |
| p20190625 | clock | | |
+-----------+--------+------------+------------+
rows in set (0.00 sec) mysql> alter table history drop partition p20190524;
Query OK, rows affected (0.20 sec)
Records: Duplicates: Warnings: mysql> select partition_name part,partition_expression expr,partition_description descr,table_rows from information_schema.partitions where table_schema=schema() and table_name='history';
+-----------+--------+------------+------------+
| part | expr | descr | table_rows |
+-----------+--------+------------+------------+
| p20190525 | clock | | |
| p20190526 | clock | | |
| p20190527 | clock | | |
| p20190528 | clock | | |
| p20190529 | clock | | |
| p20190530 | clock | | |
| p20190531 | clock | | |
| p20190601 | clock | | |
| p20190602 | clock | | |
| p20190603 | clock | | |
| p20190604 | clock | | |
| p20190605 | clock | | |
| p20190606 | clock | | |
| p20190607 | clock | | |
| p20190608 | clock | | |
| p20190609 | clock | | |
| p20190610 | clock | | |
| p20190611 | clock | | |
| p20190612 | clock | | |
| p20190613 | clock | | |
| p20190614 | clock | | |
| p20190615 | clock | | |
| p20190616 | clock | | |
| p20190617 | clock | | |
| p20190618 | clock | | |
| p20190619 | clock | | |
| p20190620 | clock | | |
| p20190621 | clock | | |
| p20190622 | clock | | |
| p20190623 | clock | | |
| p20190624 | clock | | |
| p20190625 | clock | | |
+-----------+--------+------------+------------+
rows in set (0.00 sec) mysql> alter table history_uint drop partition p20190524; # 清理后的结果
mysql> select table_name, (data_length+index_length)// as total_mb, table_rows from information_schema.tables where table_schema='zabbix';
+----------------------------+----------------+------------+
| table_name | total_mb | table_rows |
+----------------------------+----------------+------------+
| acknowledges | 0.06250000 | |
| actions | 0.04687500 | |
| alerts | 5.75000000 | |
| application_discovery | 0.04687500 | |
| application_prototype | 0.04687500 | |
| application_template | 0.39062500 | |
| applications | 0.35937500 | |
| auditlog | 2.01562500 | |
| auditlog_details | 0.26562500 | |
| autoreg_host | 0.03125000 | |
| conditions | 0.03125000 | |
| config | 0.04687500 | |
| corr_condition | 0.03125000 | |
| corr_condition_group | 0.03125000 | |
| corr_condition_tag | 0.01562500 | |
| corr_condition_tagpair | 0.01562500 | |
| corr_condition_tagvalue | 0.01562500 | |
| corr_operation | 0.03125000 | |
| correlation | 0.01562500 | |
| dbversion | 0.01562500 | |
| dchecks | 0.03125000 | |
| dhosts | 0.03125000 | |
| drules | 0.04687500 | |
| dservices | 0.04687500 | |
| escalations | 0.03125000 | |
| event_recovery | 231.89062500 | |
| event_tag | 0.03125000 | |
| events | 1561.00000000 | |
| expressions | 0.03125000 | |
| functions | 2.01562500 | |
| globalmacro | 0.01562500 | |
| globalvars | 0.01562500 | |
| graph_discovery | 0.12500000 | |
| graph_theme | 0.03125000 | |
| graphs | 1.96875000 | |
| graphs_items | 1.79687500 | |
| group_discovery | 0.03125000 | |
| group_prototype | 0.06250000 | |
| groups | 0.03125000 | |
| history | 7314.17187500 | |
| history_log | 1.03125000 | |
| history_str | 269.62500000 | |
| history_text | 196.07812500 | |
| history_uint | 15734.87500000 | |
| host_discovery | 0.04687500 | |
| host_inventory | 0.01562500 | |
| hostmacro | 0.03125000 | |
| hosts | 0.20312500 | |
| hosts_groups | 0.04687500 | |
| hosts_templates | 0.10937500 | |
| housekeeper | 0.01562500 | |
| httpstep | 0.03125000 | |
| httpstepitem | 0.04687500 | |
| httptest | 0.07812500 | |
| httptestitem | 0.04687500 | |
| icon_map | 0.04687500 | |
| icon_mapping | 0.04687500 | |
| ids | 0.01562500 | |
| images | 1.53125000 | |
| interface | 0.04687500 | |
| interface_discovery | 0.03125000 | |
| item_application_prototype | 0.04687500 | |
| item_condition | 0.09375000 | |
| item_discovery | 1.76562500 | |
| items | 13.25000000 | |
| items_applications | 4.89062500 | |
| maintenances | 0.01562500 | |
| maintenances_groups | 0.04687500 | |
| maintenances_hosts | 0.04687500 | |
| maintenances_windows | 0.04687500 | |
| mappings | 0.03125000 | |
| media | 0.04687500 | |
| media_type | 0.03125000 | |
| opcommand | 0.03125000 | |
| opcommand_grp | 0.04687500 | |
| opcommand_hst | 0.04687500 | |
| opconditions | 0.03125000 | |
| operations | 0.03125000 | |
| opgroup | 0.04687500 | |
| opinventory | 0.01562500 | |
| opmessage | 0.03125000 | |
| opmessage_grp | 0.04687500 | |
| opmessage_usr | 0.04687500 | |
| optemplate | 0.04687500 | |
| problem | 6.48437500 | |
| problem_tag | 0.04687500 | |
| profiles | 0.50000000 | |
| proxy_autoreg_host | 0.01562500 | |
| proxy_dhistory | 0.01562500 | |
| proxy_history | 0.01562500 | |
| regexps | 0.03125000 | |
| rights | 0.04687500 | |
| screen_user | 0.04687500 | |
| screen_usrgrp | 0.04687500 | |
| screens | 0.04687500 | |
| screens_items | 0.03125000 | |
| scripts | 0.06250000 | |
| service_alarms | 0.04687500 | |
| services | 0.03125000 | |
| services_links | 0.04687500 | |
| services_times | 0.03125000 | |
| sessions | 0.20312500 | |
| slides | 0.04687500 | |
| slideshow_user | 0.04687500 | |
| slideshow_usrgrp | 0.04687500 | |
| slideshows | 0.04687500 | |
| sysmap_element_url | 0.03125000 | |
| sysmap_url | 0.03125000 | |
| sysmap_user | 0.04687500 | |
| sysmap_usrgrp | 0.04687500 | |
| sysmaps | 0.07812500 | |
| sysmaps_elements | 0.09375000 | |
| sysmaps_link_triggers | 0.04687500 | |
| sysmaps_links | 0.06250000 | |
| task | 0.01562500 | |
| task_close_problem | 0.01562500 | |
| timeperiods | 0.01562500 | |
| trends | 832.81250000 | |
| trends_uint | 1179.21875000 | |
| trigger_depends | 0.04687500 | |
| trigger_discovery | 0.20312500 | |
| trigger_tag | 0.03125000 | |
| triggers | 3.07812500 | |
| users | 0.03125000 | |
| users_groups | 0.04687500 | |
| usrgrp | 0.03125000 | |
| valuemaps | 0.03125000 | |
+----------------------------+----------------+------------+
rows in set (0.01 sec) # 分区脚本获取地址
https://dl.cactifans.com/zabbix/partitiontables_gt_zbx34.sh 脚本默认详情数据保留30天，趋势数据保留12个月，如需修改，请修改以下内容：
daily_history_min=
monthly_history_min= 脚本默认连接数据库信息，更改成你的：
DBHOST=localhost
DBUSER=zabbix
DBPASS=zabbix [root@zabbix:~]# cat /opt/zabbix_db_partition.sh
#!/bin/bash
#
# This script will partition your zabbix database to improve the efficiency.
# It will also create stored procedures to do the necessary housekeeping,
# and create a cronjob to do this on a daily basis
#
# This script inspired by the following:
# http://zabbixzone.com/zabbix/partitioning-tables/
#
# While the basic SQL is from the above page, this script both creates the necessary
# SQL for the desired tables, and can create new partitions as the time goes on
# assuming that the cronjob has been properly entered.
# #
# Who to email with cron output
#
EMAIL="root@localhost" #
# How long to keep the daily history
#
daily_history_min= #
# How long to keep the monthly history (months)
#
monthly_history_min= #
# Years to create the monthly partitions for
#
first_year=`date +"%Y"`
last_year=$first_year
cur_month=`date +"%m"|sed 's/^0*//'`
if [ $cur_month -eq ]; then
last_year=$((first_year+))
cur_month=
fi y=`date +"%Y"` SQL="/tmp/partition.sql"
PATHTOCRON="/usr/local/zabbix/cron.d"
PATHTOMAILBIN="/usr/bin/mail"
DUMP_FILE=/tmp/zabbix.sql function usage {
cat <<_EOF_ $ [-h host][-u user][-p password][-d min_days][-y startyear][-n][-s][-e email_address][-b] -h host database host
-u user db user
-p password user password
-d min_days Minimum number of days of history to keep (default: $daily_history_min)
-m min_months Minimum number of months to keep trends (default: $monthly_history_min)
-y startyear First year to set up with partitions
-n noninteractive Run without questions - careful, make sure you know what is going to happen. Needs my.cnf with correct permissions.
-b backup Create backup of DB in $DUMP_FILE before alterations (only works with non-interactive mode, -n)
-s simulate Create SQL file that would be executed for examination ($SQL)
-e email Email address to receive partition update report (default: $EMAIL) After running this script, don't forget to disable housekeeping if
you didn't have the script disable it, and add the following cronjob ### Option: DisableHousekeeping
# If set to , disables housekeeping.
#
# Mandatory: no
# Range: -
################### Uncomment and change the following line to in
################### Then restart the zabbix server
DisableHousekeeping= Cron job * * * $PATHTOCRON/housekeeping.sh _EOF_
exit
} #DBHOST=localhost
DBHOST="127.0.0.1"
DBUSER=zabbix
DBPASS="zabbix"
SIMULATE=
NONINTERACTIVE=
BACKUP=
while getopts "m:nsbe:h:u:p:d:y:?h" flag; do
case $flag in
h) DBHOST=$OPTARG ;;
u) DBUSER=$OPTARG ;;
p) DBPASS=$OPTARG ;;
e) EMAIL=$OPTARG ;;
s) SIMULATE= ;;
n) NONINTERACTIVE= ;;
b) BACKUP= ;;
d) h=$OPTARG
if [ $h -gt ] >/dev/null; then
daily_history_min=$h
else
echo "Invalid daily history min, exiting"
exit
fi
;;
m) h=$OPTARG
if [ $h -gt ] >/dev/null; then
monthly_history_min=$h
else
echo "Invalid monthly history min, exiting"
exit
fi
;; y) yy=$OPTARG
if [ $yy -lt $y -a $yy -gt ] >/dev/null; then
first_year=$yy
else
echo "Invalid year, exiting"
exit
fi
;;
?|h) usage ;;
esac
done
shift $((OPTIND-)) if [ $NONINTERACTIVE != ]; then
echo "Ready to partition tables."
fi if [ $SIMULATE = ]; then
if [ $NONINTERACTIVE = ]; then
mysql -B -h $DBHOST -e "GRANT CREATE ROUTINE ON zabbix.* TO '$DBUSER'@'localhost';"
# echo "GRANT LOCK TABLES ON zabbix.* TO '${DBUSER}'@'${DBHOST}' IDENTIFIED BY '${DBPASS}';" | mysql -h${DBHOST} -u${DBADMINUSER} --password=${DBADMINPASS}
mysql -h $DBHOST -e "GRANT LOCK TABLES ON zabbix.* TO '$DBUSER'@'$DBHOST' IDENTIFIED BY '$DBPASS';"
if [ $BACKUP = ]; then
mysqldump --opt -h $DBHOST -u $DBUSER -p$DBPASS zabbix --result-file=$DUMP_FILE
rc=$?
if [ $rc -ne ]; then
echo "Error during mysqldump, exit code: $rc"
fi
fi
else
echo -e "\nReady to update permissions of Zabbix user to create routines\n"
echo -n "Enter root DB user: "
read DBADMINUSER
echo -n "Enter $DBADMINUSER password: "
read DBADMINPASS
mysql -B -h $DBHOST -u $DBADMINUSER -p$DBADMINPASS -e "GRANT CREATE ROUTINE ON zabbix.* TO '$DBUSER'@'localhost';"
echo -e "\n" echo -ne "\nDo you want to backup the database (recommended) (Y/n): "
read yn
if [ "$yn" != "n" -a "$yn" != "N" ]; then
echo -e "\nEnter output file, press return for default of $DUMP_FILE"
read df
[ "$df" != "" ] && DUMP_FILE=$df #
# Lock tables is needed for a good mysqldump
#
echo "GRANT LOCK TABLES ON zabbix.* TO '${DBUSER}'@'${DBHOST}' IDENTIFIED BY '${DBPASS}';" | mysql -h${DBHOST} -u${DBADMINUSER} --password=${DBADMINPASS} mysqldump --opt -h ${DBHOST} -u ${DBUSER} -p${DBPASS} zabbix --result-file=${DUMP_FILE}
rc=$?
if [ $rc -ne ]; then
echo "Error during mysqldump, rc: $rc"
echo "Do you wish to continue (y/N): "
read yn
[ "yn" != "y" -a "$yn" != "Y" ] && exit
else
echo "Mysqldump succeeded!, proceeding with upgrade..."
fi
else
echo "Are you certain you have a backup (y/N): "
read yn
[ "$yn" != 'y' -a "$yn" != "Y" ] && exit
fi
fi
fi if [ $NONINTERACTIVE = ]; then
yn='y'
else
echo -e "\n\nReady to proceed:" echo -e "\nStarting yearly partioning at: $first_year"
echo "and ending at: $last_year"
echo "With $daily_history_min days of daily history"
echo -e "\n\nReady to proceed (Y/n): "
read yn
[ "$yn" = 'n' -o "$yn" = "N" ] && exit
fi DAILY="history history_log history_str history_text history_uint"
DAILY_IDS="itemid id itemid id itemid" MONTHLY="trends trends_uint"
#"acknowledges alerts auditlog events service_alarms"
MONTHLY_IDS="" TABLES="$DAILY $MONTHLY"
IDS="$DAILY_IDS $MONTHLY_IDS" if [ $NONINTERACTIVE != ]; then
echo "Use zabbix; SELECT 'Altering tables';" >$SQL
else
echo "Use zabbix;" >$SQL
fi
cnt=
for i in $TABLES; do
if [ $NONINTERACTIVE != ]; then
echo "Altering table: $i"
echo "SELECT '$i';" >>$SQL
fi
cnt=$((cnt+))
case $i in
history_log)
#echo "ALTER TABLE $i DROP KEY history_log_2;" >>$SQL
#echo "ALTER TABLE $i ADD KEY history_log_2(itemid, id);" >>$SQL
#echo "ALTER TABLE $i DROP PRIMARY KEY ;" >>$SQL
#id=`echo $IDS | cut -f$cnt -d" "`
#echo "ALTER TABLE $i ADD KEY ${i}id ($id);" >>$SQL
;;
history_text)
#echo "ALTER TABLE $i DROP KEY history_text_2;" >>$SQL
#echo "ALTER TABLE $i ADD KEY history_text_2 (itemid, clock);" >>$SQL
#echo "ALTER TABLE $i DROP PRIMARY KEY ;" >>$SQL
#id=`echo $IDS | cut -f$cnt -d" "`
#echo "ALTER TABLE $i ADD KEY ${i}id ($id);" >>$SQL
;;
esac
done echo -en "\n" >>$SQL
for i in $MONTHLY; do
if [ $NONINTERACTIVE != ]; then
echo "Creating monthly partitions for table: $i"
echo "SELECT '$i';" >>$SQL
fi
echo "ALTER TABLE $i PARTITION BY RANGE( clock ) (" >>$SQL
for y in `seq $first_year $last_year`; do
last_month=
[ $y -eq $last_year ] && last_month=$((cur_month+))
for m in `seq $last_month`; do
[ $m -lt ] && m="0$m"
ms=`date +"%Y-%m-01" -d "$m/01/$y +1 month"`
pname="p${y}${m}"
echo -n "PARTITION $pname VALUES LESS THAN (UNIX_TIMESTAMP(\"$ms 00:00:00\"))" >>$SQL
[ $m -ne $last_month -o $y -ne $last_year ] && echo -n "," >>$SQL
echo -ne "\n" >>$SQL
done
done
echo ");" >>$SQL
done for i in $DAILY; do
if [ $NONINTERACTIVE != ]; then
echo "Creating daily partitions for table: $i"
echo "SELECT '$i';" >>$SQL
fi
echo "ALTER TABLE $i PARTITION BY RANGE( clock ) (" >>$SQL
for d in `seq -$daily_history_min `; do
ds=`date +"%Y-%m-%d" -d "$d day +1 day"`
pname=`date +"%Y%m%d" -d "$d day"`
echo -n "PARTITION p$pname VALUES LESS THAN (UNIX_TIMESTAMP(\"$ds 00:00:00\"))" >>$SQL
[ $d -ne ] && echo -n "," >>$SQL
echo -ne "\n" >>$SQL
done
echo ");" >>$SQL
done ###############################################################
if [ $NONINTERACTIVE != ]; then
cat >>$SQL <<_EOF_
SELECT "Installing procedures";
_EOF_
fi cat >>$SQL <<_EOF_
/**************************************************************
MySQL Auto Partitioning Procedure for Zabbix 1.8
http://zabbixzone.com/zabbix/partitioning-tables/ Author: Ricardo Santos (rsantos at gmail.com)
Version: 20110518
**************************************************************/
DELIMITER //
DROP PROCEDURE IF EXISTS zabbix.create_zabbix_partitions; //
CREATE PROCEDURE zabbix.create_zabbix_partitions ()
BEGIN
_EOF_ ############################################################### for i in $DAILY; do
echo " CALL zabbix.create_next_partitions(\"zabbix\",\"$i\");" >>$SQL
echo " CALL zabbix.drop_old_partitions(\"zabbix\",\"$i\");" >>$SQL
done
echo -en "\n" >>$SQL
for i in $MONTHLY; do
echo " CALL zabbix.create_next_monthly_partitions(\"zabbix\",\"$i\");" >>$SQL
echo " CALL zabbix.drop_old_monthly_partitions(\"zabbix\",\"$i\");" >>$SQL
done ###############################################################
cat >>$SQL <<_EOF_
END // DROP PROCEDURE IF EXISTS zabbix.create_next_partitions; //
CREATE PROCEDURE zabbix.create_next_partitions (SCHEMANAME varchar(), TABLENAME varchar())
BEGIN
DECLARE NEXTCLOCK timestamp;
DECLARE PARTITIONNAME varchar();
DECLARE CLOCK int;
SET @totaldays = ;
SET @i = ;
createloop: LOOP
SET NEXTCLOCK = DATE_ADD(NOW(),INTERVAL @i DAY);
SET PARTITIONNAME = DATE_FORMAT( NEXTCLOCK, 'p%Y%m%d' );
SET CLOCK = UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD( NEXTCLOCK ,INTERVAL DAY),'%Y-%m-%d 00:00:00'));
CALL zabbix.create_partition( SCHEMANAME, TABLENAME, PARTITIONNAME, CLOCK );
SET @i=@i+;
IF @i > @totaldays THEN
LEAVE createloop;
END IF;
END LOOP;
END // DROP PROCEDURE IF EXISTS zabbix.drop_old_partitions; //
CREATE PROCEDURE zabbix.drop_old_partitions (SCHEMANAME varchar(), TABLENAME varchar())
BEGIN
DECLARE OLDCLOCK timestamp;
DECLARE PARTITIONNAME varchar();
DECLARE CLOCK int;
SET @mindays = $daily_history_min;
SET @maxdays = @mindays+;
SET @i = @maxdays;
droploop: LOOP
SET OLDCLOCK = DATE_SUB(NOW(),INTERVAL @i DAY);
SET PARTITIONNAME = DATE_FORMAT( OLDCLOCK, 'p%Y%m%d' );
CALL zabbix.drop_partition( SCHEMANAME, TABLENAME, PARTITIONNAME );
SET @i=@i-;
IF @i <= @mindays THEN
LEAVE droploop;
END IF;
END LOOP;
END // DROP PROCEDURE IF EXISTS zabbix.create_next_monthly_partitions; //
CREATE PROCEDURE zabbix.create_next_monthly_partitions (SCHEMANAME varchar(), TABLENAME varchar())
BEGIN
DECLARE NEXTCLOCK timestamp;
DECLARE PARTITIONNAME varchar();
DECLARE CLOCK int;
SET @totalmonths = ;
SET @i = ;
createloop: LOOP
SET NEXTCLOCK = DATE_ADD(NOW(),INTERVAL @i MONTH);
SET PARTITIONNAME = DATE_FORMAT( NEXTCLOCK, 'p%Y%m' );
SET CLOCK = UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD( NEXTCLOCK ,INTERVAL MONTH),'%Y-%m-01 00:00:00'));
CALL zabbix.create_partition( SCHEMANAME, TABLENAME, PARTITIONNAME, CLOCK );
SET @i=@i+;
IF @i > @totalmonths THEN
LEAVE createloop;
END IF;
END LOOP;
END // DROP PROCEDURE IF EXISTS zabbix.drop_old_monthly_partitions; //
CREATE PROCEDURE zabbix.drop_old_monthly_partitions (SCHEMANAME varchar(), TABLENAME varchar())
BEGIN
DECLARE OLDCLOCK timestamp;
DECLARE PARTITIONNAME varchar();
DECLARE CLOCK int;
SET @minmonths = $monthly_history_min;
SET @maxmonths = @minmonths+;
SET @i = @maxmonths;
droploop: LOOP
SET OLDCLOCK = DATE_SUB(NOW(),INTERVAL @i MONTH);
SET PARTITIONNAME = DATE_FORMAT( OLDCLOCK, 'p%Y%m' );
CALL zabbix.drop_partition( SCHEMANAME, TABLENAME, PARTITIONNAME );
SET @i=@i-;
IF @i <= @minmonths THEN
LEAVE droploop;
END IF;
END LOOP;
END // DROP PROCEDURE IF EXISTS zabbix.create_partition; //
CREATE PROCEDURE zabbix.create_partition (SCHEMANAME varchar(), TABLENAME varchar(), PARTITIONNAME varchar(), CLOCK int)
BEGIN
DECLARE RETROWS int;
SELECT COUNT() INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_name = PARTITIONNAME; IF RETROWS = THEN
SELECT CONCAT( "create_partition(", SCHEMANAME, ",", TABLENAME, ",", PARTITIONNAME, ",", CLOCK, ")" ) AS msg;
SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME,
' ADD PARTITION (PARTITION ', PARTITIONNAME, ' VALUES LESS THAN (', CLOCK, '));' );
PREPARE STMT FROM @sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END // DROP PROCEDURE IF EXISTS zabbix.drop_partition; //
CREATE PROCEDURE zabbix.drop_partition (SCHEMANAME varchar(), TABLENAME varchar(), PARTITIONNAME varchar())
BEGIN
DECLARE RETROWS int;
SELECT COUNT() INTO RETROWS
FROM information_schema.partitions
WHERE table_schema = SCHEMANAME AND table_name = TABLENAME AND partition_name = PARTITIONNAME; IF RETROWS = THEN
SELECT CONCAT( "drop_partition(", SCHEMANAME, ",", TABLENAME, ",", PARTITIONNAME, ")" ) AS msg;
SET @sql = CONCAT( 'ALTER TABLE ', SCHEMANAME, '.', TABLENAME,
' DROP PARTITION ', PARTITIONNAME, ';' );
PREPARE STMT FROM @sql;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
END IF;
END //
DELIMITER ;
_EOF_ if [ $SIMULATE = ]; then
exit
fi if [ $NONINTERACTIVE = ]; then
yn='y'
else
echo -e "\n\nReady to apply script to database, this may take a while.(Y/n): "
read yn
fi
if [ "$yn" != "n" -a "$yn" != "N" ]; then
mysql --skip-column-names -h ${DBHOST} -u ${DBUSER} -p${DBPASS} <$SQL
fi conf=/etc/zabbix/zabbix_server.conf
if [ $NONINTERACTIVE = ]; then
yn='y'
else
echo -e "\nIf Zabbix Version = 2.0 \nDo you want to update the /etc/zabbix/zabbix_server.conf"
echo -n "to disable housekeeping (Y/n): "
read yn
fi
if [ "$yn" != "n" -a "$yn" != "N" ]; then
cp $conf ${conf}.bak
sed -i "s/^# DisableHousekeeping=0/DisableHousekeeping=1/" $conf
sed -i "s/^DisableHousekeeping=0/DisableHousekeeping=1/" $conf
/etc/init.d/zabbix-server stop
sleep
/etc/init.d/zabbix-server start >& > /dev/null
fi tmpfile=/tmp/cron$$
if [ $NONINTERACTIVE = ]; then
yn='y'
else
echo -ne "\nDo you want to update the crontab (Y/n): "
read yn
fi
if [ "$yn" != "n" -a "$yn" != "N" ]; then
where=
while [ "$where" = "" ]; do
if [ $NONINTERACTIVE = ]; then
where='Y'
else
echo "The crontab entry can be either in /etc/cron.daily, or added"
echo -e "to the crontab for root\n"
echo -n "Do you want to add this to the /etc/cron.daily directory (Y/n): "
read where
fi
[ "$where" = "" -o "$where" = "y" ] && where="Y"
if [ "$where" != "y" -a "$where" != "Y" -a "$where" != "n" -a "$where" != "N" ]; then
where=""
echo "Response not recognized, please try again"
fi
done if [ $NONINTERACTIVE != ]; then
echo -en "\nEnter email of who should get the daily housekeeping reports: "
read mailto
fi
[ "$mailto" = "" ] && mailto=$EMAIL
mkdir -p $PATHTOCRON
cat >$PATHTOCRON/housekeeping.sh <<_EOF_
#!/bin/bash MAILTO=$mailto
tmpfile=/tmp/housekeeping\$\$ date >\$tmpfile
/usr/bin/mysql --skip-column-names -B -h localhost -u${DBUSER} -p${DBPASS} zabbix -e "CALL create_zabbix_partitions();" >>\$tmpfile >&
$PATHTOMAILBIN -s "Zabbix MySql Partition Housekeeping" \$MAILTO <\$tmpfile
rm -f \$tmpfile
_EOF_
chmod +x $PATHTOCRON/housekeeping.sh
chown -R zabbix.zabbix /usr/local/zabbix
if [ "$where" = "Y" ]; then
cat >/etc/cron.daily/zabbixhousekeeping <<_EOF_
#!/bin/bash
$PATHTOCRON/housekeeping.sh
_EOF_
chmod +x /etc/cron.daily/zabbixhousekeeping
else
crontab -l >$tmpfile
cat >>$tmpfile <<_EOF_
* * * $PATHTOCRON/housekeeping.sh
_EOF_
crontab $tmpfile
rm $tmpfile
fi
fi
#############################
# 上面的分表脚本执行以后会生成以下自动每天执行分表的计划任务
# 为了安全起见最好自定义每天的计划任务，如果没有自动分表，zabbix获取的监控数据就无法入库，进而不能触发报警，如果刚好有关键业务出现问题没有触发报警，就很悲剧了
[root@aliyun-zabbix:~]# cat /etc/cron.daily/zabbixhousekeeping
#!/bin/bash
/usr/local/zabbix/cron.d/housekeeping.sh
[root@aliyun-zabbix:~]# cat /usr/local/zabbix/cron.d/housekeeping.sh
#!/bin/bash
MAILTO=jack@chinasoft.com
tmpfile=/tmp/housekeeping$$
date >$tmpfile
/usr/local/bin/mysql --skip-column-names -B -h localhost -uzabbix -p'zabbix' zabbix -e "CALL create_zabbix_partitions();" >>$tmpfile 2>&1
/usr/bin/mail -s "Zabbix MySql Partition Housekeeping" $MAILTO <$tmpfile
rm -f $tmpfile
[root@aliyun-zabbix:~]# crontab -l
## zabbix auto partion: history,history_uint,history_str,history_text,history_log,trends,trends_uint
05 00 * * * /bin/bash /usr/local/zabbix/cron.d/housekeeping.sh > /dev/null 2>&1
# 关闭hoursekeeping
生产环境zabbix3.2上亿的表数据通过表分区的方式进行历史数据清理的更多相关文章
- SQL语句-批量插入表(表数据插表)
批量插入表(表数据插表) ****1.INSERT INTO SELECT语句语句形式为:Insert into Table2(field1,field2,...) select value1,val ...
- Mysql 复制表数据（表结构相同）
[1]Mysql 复制表数据(表结构相同) -- 方式一: create table table_name_dest as select * from table_name_src; -- 方式二: ...
- Sentinel上生产环境只差一步，监控数据持久化
之前介绍了Sentinel相关的文章,小伙伴在生产实践中不知道有没有这个疑问?我们的Sentinel控制台监控的数据只能看最近5分钟的,如图 那么就导致历史数据是查看不了的,那肯定是不行的,在生产环境 ...
- 深度学习Tensorflow生产环境部署（上·环境准备篇）
最近在研究Tensorflow Serving生产环境部署,尤其是在做服务器GPU环境部署时,遇到了不少坑.特意总结一下,当做前车之鉴. 1 系统背景 系统是ubuntu16.04 ubuntu@ub ...
- 四步法分析定位生产环境下MySQL上千条SQL中的问题所在
第一步:通过以下两种方式之一来打开慢查询功能 (1)方式一:通过修改mysql的my.cnf文件 如果是5.0或5.1等版本需要增加以下选项: log-slow-queries="mysql ...
- NanoProfiler - 适合生产环境的性能监控类库 之 大数据篇
上期回顾 上一期:NanoProfiler - 适合生产环境的性能监控类库 之 基本功能篇 上次介绍了NanoProfiler的基本功能,提到,NanoProfiler实现了MiniProfiler欠 ...
- plsql 数据迁移——导出表结构，表数据，表序号
场景:项目开发完之后要部署在不同的环境进行测试,这时候就需要将数据库中的表结构,序号,数据进行迁移,这时候就需要能够熟练的使用plsql. 问题: 导出的表结构,在另一个数据库中无法导入 部分表的数据 ...
- mysql 复制表数据，表结构的3种方法
什么时候我们会用到复制表?例如:我现在对一张表进行操作,但是怕误删数据,所以在同一个数据库中建一个表结构一样,表数据也一样的表,以作备份.如果用mysqldump比较麻烦,备份.MYD,.MYI这样的 ...
- mysql复制表数据或表结构到新表中
MySQL复制表数据到新表的几个步骤. 1.MySQL复制表结构及数据到新表 CREATE TABLE new_table SELECT * FROM old_table; 2.只复制表结构到新表 C ...
随机推荐
- LeetCode - 86、分隔链表
给定一个链表和一个特定值 x,对链表进行分隔,使得所有小于 x 的节点都在大于或等于 x 的节点之前. 你应当保留两个分区中每个节点的初始相对位置. 示例: 输入: head = 1->4-&g ...
- test20190829 神大校赛模拟
100+100+0=200,聪明搬题人题面又出锅了. 最短路径(path) 给定有向图,包含 n 个节点和 m 条有向边. 一条A 到 B 的路径是最短路径当且仅当不存在另一条从A 到 B 的路径比它 ...
- Yum 安装memcached 与缓存清空
1.安装 root@pts/0 # yum -y install memcached 2.启动服务 root@pts/0 # /etc/init.d/memcached start 3 ...
- drf框架 - 序列化组件 | ModelSerializer (查,增,删,改)
ModelSerializer 序列化准备: 配置 settings.py # 注册rest_framework框架 INSTALLED_APPS = [ ... 'rest_framework' ] ...
- set/priority_queue的运算符重载
#include<bits/stdc++.h> using namespace std; struct cmp { bool operator ()(int a, int b) //重载小 ...
- nginx优化、负载均衡、rewrite
nginx优化 # 普通用户启动 (useradd nginx -s /sbin/nologin -M) user nginx; # 配置nginx worker进程个数 #worker_proces ...
- javaweb学习笔记（三）
一.javaweb高级(Filter和Listener)的简单介绍 1.过滤器Filter (https://www.cnblogs.com/vanl/p/5742501.html) ①定义 Filt ...
- learning java AWT MenuBar Menu MenuItem菜单
import java.awt.*; import java.awt.event.ActionListener; import java.awt.event.KeyEvent; import java ...
- 洛谷 P1725 琪露诺 题解
P1725 琪露诺 题目描述 在幻想乡,琪露诺是以笨蛋闻名的冰之妖精. 某一天,琪露诺又在玩速冻青蛙,就是用冰把青蛙瞬间冻起来.但是这只青蛙比以往的要聪明许多,在琪露诺来之前就已经跑到了河的对岸.于是 ...
- 建立自己的键盘栈(shortcutkeyStack)
建立自己的键盘栈(shortcutkeyStack) 作为一名开发者, 快捷键是必不可少的, 并且各种开发工具都有提供快捷键. 但是各种工具(IDE,编辑器)因为历史或者其他不可抗原因(比如键盘的布局 ...