ELASTIC的备份与恢复
前言
elasticsearch官方并没有提供合适的备份工具,然而生产场景中备份却是的确需要的。
本文介绍了使用自己写的php脚本以及第三方工具来进行索引的备份,恢复以及删除等操作。
全量备份
恢复
删除索引
<?php
date_default_timezone_set('Asia/Shanghai'); $longopts = array(
'index:',
'query:',
'start:',
'end:',
);
$options = getopt('a', $longopts); if(!isset($options['index'])) {
fwrite(STDERR, "index必须设置索引地址\n");
exit(1);
} $components = parse_url($options['index']);
if(!isset($components['path'])) {
fwrite(STDERR, 'index不可为空');
exit(1);
} $host = "{$components['scheme']}://{$components['host']}:{$components['port']}";
$index = basename($components['path']); $query = isset($options['query']) ? $options['query'] : '{"query":{"match_all":{}}}';
if(isset($options['start'])) {
$start_time = strtotime($options['start']);
$start = date('Y-m-d', $start_time).'T00:00:00+0800';
if(isset($options['end'])) {
$end_time = strtotime($options['end']);
$end = date('Y-m-d', $end_time).'T00:00:00+0800';
} else {
$end = date('Y-m-d', $start_time+86400).'T00:00:00+0800';
} $field = strpos($index, 'analysis')!==false ? 'create_time' : 'date';
$query = '{"size":1000,"_source":false,"query":{"filtered":{"filter":{"range":{"'.$field.'":{"gte":"'.$start.'","lt":"'.$end.'"}}}}}}';
} $scroll_id = null;
$retry = 0;
$num = 0;
while(true) {
if(is_null($scroll_id)) {
$result = post("{$host}/{$index}/_search?scroll=2m", $query);
} else {
$result = get("{$host}/_search/scroll?scroll=2m&scroll_id={$scroll_id}");
} $json = json_decode($result, true);
if(!isset($json['_scroll_id'])) {
fwrite(STDERR, "查询失败:索引-{$index} 起始-{$start} 截止-{$end}\n");
sleep(5);
$retry++;
if($retry>10) {
exit(4);
}
} $scroll_id = $json['_scroll_id'];
$bulk = [];
foreach($json['hits']['hits'] as $row) {
unset($row['_score']);
$bulk[] = json_encode(['delete'=>$row]);
$num++;
} if(count($json['hits']['hits'])==0)
{
break;
} $check = post("{$host}/_bulk", implode("\n", $bulk)."\n");
fwrite(STDOUT, "{$num}\n");
usleep(100000);
} echo "deleted:{$num}\n"; function get($url)
{
$handle = curl_init();
//curl_setopt($handle, CURLOPT_POST, 1);
curl_setopt($handle, CURLOPT_HEADER, 0);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($handle, CURLOPT_URL, $url);
//curl_setopt($handle, CURLOPT_POSTFIELDS, $data);
return curl_exec($handle);
} function post($url, $data)
{
$handle = curl_init();
curl_setopt($handle, CURLOPT_POST, 1);
curl_setopt($handle, CURLOPT_HEADER, 0);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($handle, CURLOPT_URL, $url);
curl_setopt($handle, CURLOPT_POSTFIELDS, $data);
return curl_exec($handle);
}
索引增量备份
#!/bin/bash #如果有参数,则第一个参数为备份日期,否则默认备份昨天数据
if [ -z "$1" ];then
start=$(date +%Y-%m-%d --date '-1 day 00:00:00')
end=$(date +%Y-%m-%d --date 'today 00:00:00')
else
start=$(date +%Y-%m-%d --date $)
end=$(date +%Y-%m-%d --date "$1 +1 day 00:00:00")
fi #如果是每月1号则加一天,解决时区导致的数据跨索引问题
if [ $(date +%d --date $end) -eq "" ]
then
end=$(date +%Y-%m-%d --date "$end +1 day 00:00:00")
fi php esdump.php --input=http://127.0.0.1:9200/logstash-event-$(date +%Y.%m --date $start) --start $start --end $end 2>> backup_error_$start.log | gzip > event/logstash-event-$(date +%Y.%m_%d --date $start).json.gz 2>> backup_error_$start.log &
<?php
date_default_timezone_set('Asia/Shanghai'); $longopts = array(
'input:',
'output:',
'query:',
'start:',
'end:',
);
$options = getopt('a', $longopts); if(!isset($options['input'])) {
fwrite(STDERR, "input必须设置索引地址\n");
exit(1);
} $check = get($options['input'].'/_count');
if($check===false) {
fwrite(STDERR, "input索引地址无效:{$options['input']}\n");
exit(2);
} $check = json_decode($check, true);
if(!isset($check['count'])) {
fwrite(STDERR, "input索引地址无效:{$options['input']}\n");
exit(3);
} $components = parse_url($options['input']);
$host = "{$components['scheme']}://{$components['host']}:{$components['port']}";
$index = basename($components['path']); $query = isset($options['query']) ? $options['query'] : '{"query":{"match_all":{}}}';
if(isset($options['start'])) {
$start_time = strtotime($options['start']);
$start = date('Y-m-d', $start_time).'T00:00:00+0800';
if(isset($options['end'])) {
$end_time = strtotime($options['end']);
$end = date('Y-m-d', $end_time).'T00:00:00+0800';
} else {
$end = date('Y-m-d', $start_time+86400).'T00:00:00+0800';
} $field = strpos($index, 'analysis')!==false ? 'create_time' : 'date';
$query = '{"size":1000,"sort":{"'.$field.'":{"order":"asc"}},"query":{"filtered":{"filter":{"range":{"'.$field.'":{"gte":"'.$start.'","lt":"'.$end.'"}}}}}}'; if(strpos($index, 'eventlogs')!==false) {
$query = '{"size":1000,"sort":{"'.$field.'":{"order":"asc"}},"query":{"filtered":{"filter":{"bool":{'.
'"must":[{"range":{"date":{"gte":"'.$start.'","lte":"'.$end.'"}}}],'.
'"must_not":[{"exists": {"field":"nsp3hq"}},{"exists": {"field":"q0i8u1"}},{"exists": {"field":"eyn916"}},{"exists": {"field":"20mqd8"}},'.
'{"exists": {"field":"wwbkux"}},{"exists": {"field":"r5ua96"}},{"exists": {"field":"easiz"}},{"exists": {"field":"dexusu"}},{"exists": {"field":"earts"}},'.
'{"exists": {"field":"ealu"}},{"exists": {"field":"ealf"}},{"exists": {"field":"eal"}},{"exists": {"field":"ears"}},{"exists": {"field":"ealuf"}},'.
'{"exists": {"field":"ealus"}},{"exists": {"field":"eaatf"}},{"exists": {"field":"enail"}},{"exists": {"field":"enuail"}},{"exists": {"field":"test"}}]'.
'}}}}}';
}
} $scroll_id = null;
$retry = 0;
$num = 0;
while(true) {
if(is_null($scroll_id)) {
$result = post("{$host}/{$index}/_search?scroll=2m", $query);
} else {
$result = get("{$host}/_search/scroll?scroll=2m&scroll_id={$scroll_id}");
} $json = json_decode($result, true);
if(!isset($json['_scroll_id'])) {
fwrite(STDERR, "查询失败:索引-{$index} 起始-{$start} 截止-{$end}\n");
sleep(5);
$retry++;
if($retry>10) {
exit(4);
}
} $scroll_id = $json['_scroll_id']; foreach($json['hits']['hits'] as $row) {
fwrite(STDOUT, json_encode($row)."\n");
$num++;
} if(count($json['hits']['hits'])==0)
{
break;
} usleep(100000);
} //校验条数是否一致
$query = json_decode($query, true);
unset($query['size'], $query['sort']);
$result = post("{$host}/{$index}/_count", json_encode($query));
$json = json_decode($result, true);
if(!isset($json['count']) or intval($json['count'])!==$num) {
fwrite(STDERR, "校验失败:索引-{$index} 起始-{$start} 截止-{$end} 记录条数-{$json['count']} 导出条数-{$num}\n");
} function get($url)
{
$handle = curl_init();
//curl_setopt($handle, CURLOPT_POST, 1);
curl_setopt($handle, CURLOPT_HEADER, 0);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($handle, CURLOPT_URL, $url);
//curl_setopt($handle, CURLOPT_POSTFIELDS, $data);
return curl_exec($handle);
} function post($url, $data)
{
$handle = curl_init();
curl_setopt($handle, CURLOPT_POST, 1);
curl_setopt($handle, CURLOPT_HEADER, 0);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($handle, CURLOPT_URL, $url);
curl_setopt($handle, CURLOPT_POSTFIELDS, $data);
return curl_exec($handle);
}
备份时执行shell脚本即可备份昨天的增量数据
谢土豪
如果有帮到你的话,请赞赏我吧!
本文为kerwin原创,转载请注明出处。
http://www.cnblogs.com/kerwinC/p/6296675.html
ELASTIC的备份与恢复的更多相关文章
- Elastic认证考试,请先看这一篇
版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明.本文链接:https://blog.csdn.net/wojiushiwo987/article ...
- MySQL 数据库的备份与恢复
一.MySQL 常见的备份方式 1. 直接拷贝数据库文件(物理拷贝) 2. 使用 mysqldump 工具备份 3. 使用 mysqlhotcopy 工具备份 4. 使用 mysql 的主从同步复制, ...
- elastic search使用总结
1. elasticsearch安装 官方下载地址:https://www.elastic.co/downloads/elasticsearch 解压文件 elasticsearch-2.4.0.zi ...
- Assign an Elastic IP Address to Your Instance
By default, an instance in a nondefault VPC is not assigned a public IP address, and is private.You ...
- ZTE交换路由设备配置的备份与恢复
一.TFTP服务器搭建 使用用户计算机搭建TFTP服务器,交换路由设备作为TFTP客户端. 运行TFTPServer.exe,该程序所在的目录为TFTP的根目录. 请保证您的TFTP可以Ping通所要 ...
- mysql数据库的备份与恢复
假定我们的目标数据库是 test, 表是 user.假定mysql的用户名和密码均为 root. 备份与恢复所用的程序分别是mysql软件包提供的 mysqldump 命令和 mysql 命令.思想很 ...
- Oracle逻辑备份与恢复
1. 备份的类型 按照备份方式的不同,可以把备份分为两类: 1.1 逻辑备份:指通过逻辑导出对数据进行备份.将数据库中的用户对象导出到一个二进制文件中,逻辑备份使用导入导出工具:EXPDP/IMP ...
- How to ssh to your Amazon Elastic Beanstalk instance?
Well, if it's ec2 or a digital ocean server, it would be a lot easier- you do what you normally do f ...
- RMAN备份与恢复之初入茅庐
理解数据库备份 所谓备份实际上是把数据库复制到转储设备的过程. 从备份方式来看数据库备份分为物理备份和逻辑备份,物理备份是把构成数据库的所有文件拷贝到指定的位置的过程,而逻辑备份只是利用SQL语言从数 ...
随机推荐
- [整理]Git使用文章整理
http://pcottle.github.io/learnGitBranching/ http://www.ruanyifeng.com/blog/2012/07/git.html
- 利用JS实现图片的缓存
web页面使用HTML的<img>元素来嵌入图片,和所有HTML元素一样,<img>元素也是可以通过脚本来操控的(设置元素的src属性,将其指向一个新的URL会导致浏览器载入并 ...
- leaflet-velocity 参数
本文地址: https://www.cnblogs.com/veinyin/p/10769611.html leaflet-velocity 是 leaflet 绘制风场的一个插件,其控制参数如下所 ...
- perl6 HTTP::UserAgent (2)
http://www.cnblogs.com/perl6/p/6911166.html 之前这里有个小小例子, 这里只要是总结一下. HTTP::UserAgent包含了以下模块: --------- ...
- 搜索引擎ElasticSearchV5.4.2系列三之ES使用
相关博文: 搜索引擎ElasticSearchV5.4.2系列一之ES介绍 搜索引擎ElasticSearchV5.4.2系列二之ElasticSearchV5.4.2+kibanaV5.4.2+x- ...
- springboot自定义SpringApplication启动类
如果默认的SpringApplication不符合你的口味,你可以创建一个本地的实例并自定义它.例如,关闭banner你可以这样写: public static void main(String[] ...
- hashlib和hmac
hashlib hashlib模块用于加密相关的操作,代替了md5和sha模块,主要提供SHA1,SHA224,SHA256,SHA384,SHA512,MD5算法. #!/usr/bin/env p ...
- Ubuntu下SSH安装
step: 1.输入命令: sudo apt-get install openssh-server 2.验证sshserver是否启动了,以下两条命令均可 ps -e | grep ssh netst ...
- npm tls证书报错
Error: unable to verify the first certificate 设置node环境(零时使用,再次使用需要重新使用) set NODE_TLS_REJECT_UNAUTHOR ...
- python实战之原生爬虫(爬取熊猫主播排行榜)
""" this is a module,多行注释 """ import re from urllib import request # B ...