ES scroll(ES游标) 解决深分页
ES scroll(ES游标) 解决深分页。
Why
当Elasticsearch响应请求时,它必须确定docs的顺序,排列响应结果。如果请求的页数较少(假设每页20个docs), Elasticsearch不会有什么问题,但是如果页数较大时,比如请求第20页,Elasticsearch不得不取出第1页到第20页的所有docs,再去除第1页到第19页的docs,得到第20页的docs。
原理
Scrolling allows us to do an initial search and to keep pulling batches of results from Elasticsearch until there are no more results left. It’s a bit like a cursor in a traditional database.
A scrolled search takes a snapshot in time(适时). 中间更新不可见。
1
2
|
<code>By keeping old data files around. </code> |
深分页的代价是全局排序,若禁止排序,sort by _doc,return the next batch of results from every shard that still has results to return.
context keepalive time(当批够用) 和 scroll_id(最新)
Set the scroll value to the length of time we want to keep the scroll window open.
How long it should keep the “search context” alive.
The scroll expiry time is refreshed every time we run a scroll request,所以不宜过长(垃圾)、过短(超时),够处理一批数据即可。
1
2
3
4
5
6
7
8
9
10
11
12
13
|
<code>GET /old_index/_search?scroll=1m //第1次请求 { "query" : { "match_all" : {}}, "sort" : [ "_doc" ], //the most efficient sort order "size" : 1000 } 返回结果包含:_scroll_id ,base- 64 编码的字符串 GET /_search/scroll //后续请求 { "scroll" : "1m" , "scroll_id" : "cXVlcnlUaGVuRmV0Y2g7NTsxMDk5NDpkUmpiR2FjOFNhNnlCM1ZDMWpWYnRROzEwOTk1OmRSamJHYWM4U2E2eUIzVkMxalZidFE7MTA5OTM6ZFJqYkdhYzhTYTZ5QjNWQzFqVmJ0UTsxMTE5MDpBVUtwN2lxc1FLZV8yRGVjWlI2QUVBOzEwOTk2OmRSamJHYWM4U2E2eUIzVkMxalZidFE7MDs=" }</code> |
scroll parameter : how long it should keep the search context alive,long enough to process the previous batch of results, each scroll request sets a new expiry time.
An open search context prevents the old segments from being deleted while they are still in use.
注意:Keeping older segments alive means that more file handles(FD) are needed.
检查有多少search contexts(open_contexts):
1
|
<code>GET _nodes/stats/indices/search</code> |
Clear scroll API
Search context are automatically removed when the scroll timeout has been exceeded.
1
2
|
<code>清所有,可以清部分(无意义): DELETE _search/scroll/_all</code> |
size
When scanning, the size is applied to each shard, 真实size是:size * number_of_primary_shards.
否则(regular scroll),返还总的size。
查询结束
No more hits are returned. Each call to the scroll API returns the next batch of results until there are no more results left to return, ie the hits array is empty.
适用场景
Scrolling is not intended for real time(实时) user requests, but rather for processing large amounts of data.
scroll目的,不是处理实时的用户请求,而是为处理大数据的。
似快照
The results that are returned from a scroll request reflect the state of the index at the time that the initial search request was made, like a snapshot in time. Subsequent changes to documents (index, update or delete) will only affect later search requests.
聚合
If the request specifies aggs, only the initial search response will contain the aggs results.
顺序无关
不关心返回文档的顺序!
Scroll requests have optimizations that make them faster when the sort order is _doc. If you want to iterate over all documents regardless of the order, this is the most efficient option:
1
2
3
4
5
6
|
<code>GET /_search?scroll=1m { "sort" : [ "_doc" ] }</code> |
slice scroll
split the scroll in multiple slices
scanning and standard scroll
scanning scroll与standard scroll 查询几点不同:
1. scanning scroll 结果没有排序,结果顺序是doc入库时的顺序;
2. scanning scroll 不支持聚合
3. scanning scroll 最初查询结果的“hits”列表中不会包含结果
4. scanning scroll 最初查询中如果设定了“size”,是设定每个分片(shard)size的数量,若size=3,有5个shard,每次返回结果的最大值就是3*5=15。
示例
常见问题
scroll_id一样与否
1
2
|
<code><code>the scroll_id may change over the course of multiple calls and so it is required to always pass the most recent scroll_id as the scroll_id for the subsequent request. </code></code> |
异常:SearchContextMissingException
异常:SearchContextMissingException
SearchContextMissingException[No search context found for id [721283]];
原因:scroll设置的时间过短了。
源码212">问源码(2.1.2)
问源码(2.1.2)
scroll_id的生成:
…search.type.TransportSearchHelper#buildScrollId(…) 三个参数,搜索查询类型、结果信息、查询条件参数 TransportSearchQueryThenFetchAction.AsyncAction. finishHim()
ES scroll(ES游标) 解决深分页的更多相关文章
- 查询效率提升10倍!3种优化方案,帮你解决MySQL深分页问题
开发经常遇到分页查询的需求,但是当翻页过多的时候,就会产生深分页,导致查询效率急剧下降. 有没有什么办法,能解决深分页的问题呢? 本文总结了三种优化方案,查询效率直接提升10倍,一起学习一下. 1. ...
- ElasticSearch - 解决ES的深分页问题 (游标 scroll)
https://www.jianshu.com/p/f4d322415d29 1.简介 ES为了避免深分页,不允许使用分页(from&size)查询10000条以后的数据,因此如果要查询第10 ...
- solr深分页,游标操作分页,解决性能问题
solr深分页,游标操作分页,解决性能问题 @Test public void pageByCursor() { try { solrServer.connect(); String query = ...
- 【分页问题】elasticsearch 深分页问题以及解决方法
本文主要参考: 1.https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html ...
- Elasticsearch from/size-浅分页查询-深分页 scroll-深分页search_after深度查询区别使用及应用场景
Elasticsearch调研深度查询 1.from/size 浅分页查询 一般的分页需求我们可以使用from和size的方式实现,但是这种的分页方式在深分页的场景下应该是避免使用的.深分页的页次增加 ...
- MySQL 千万数据库深分页查询优化,拒绝线上故障!
文章首发在公众号(龙台的技术笔记),之后同步到博客园和个人网站:xiaomage.info 优化项目代码过程中发现一个千万级数据深分页问题,缘由是这样的 库里有一张耗材 MCS_PROD 表,通过同步 ...
- Elasticsearch由浅入深(七)搜索引擎:_search含义、_multi-index搜索模式、分页搜索以及深分页性能问题、query string search语法以及_all metadata原理
_search含义 _search查询返回结果数据含义分析 GET _search { , "timed_out": false, "_shards": { , ...
- ElasticSearch解决深度分页性能存在的问题使用scoll来解决
现在我们全局搜索全部的数据,每次返回3条, 从 scroll 请求返回的结果反映了 search 发生时刻的索引状态,就像一个快照.后续的对文档的改动(索引.更新或者删除)都只会影响后面的搜索请求. ...
- 深分页(Deep Pagination)
取回阶段 | Elasticsearch: 权威指南 | Elastic https://www.elastic.co/guide/cn/elasticsearch/guide/current/_fe ...
随机推荐
- Android 怎样实现 焦点图的 无线循环滑动的状态?
參考网址:http://my.oschina.net/xsk/blog/119167 总体的架构:ViewPgaer 中直接嵌套 IamgeView 方案一: 重写Viewpager 这样有局限性 ...
- css3 背景background
Css3背景<background> Css3包含多个新的背景属性,它们提供了对背景更强大的控制.可以自定义背景图的大小,可以规定背景图片的定位区域,css3还允许我们为元素使用多个背景图 ...
- vmware笔试题目
http://discuss.acmcoder.com/topic/58db8e2ebb0f44ba0e94e670 上面是完整的题目,下面一下我自己的想法. http://discuss.acmco ...
- maven使用杂记
maven test使用记录 运行指定的测试类: >mvn test -Dtest=[ClassName] 运行测试类中指定的方法:(这个需要maven-surefire-plugin: ...
- Hadoop MapReduce编程 API入门系列之MapReduce多种输出格式分析(十九)
不多说,直接上代码. 假如这里有一份邮箱数据文件,我们期望统计邮箱出现次数并按照邮箱的类别,将这些邮箱分别输出到不同文件路径下. 代码版本1 package zhouls.bigdata.myMapR ...
- Tomcat 日志切割
一.installing 日志轮训工具 yum install cronolog -y 二.安装.修改tomcat文件 wget http://mirrors.shuosc.org/apache/ ...
- weborm aspx开发基础
ASP.NET - .net开发网站应用程序的技术总称,来源于 ASP 两种方法技术—WebForm: MVC:java用 十二个表单元素: 文本框<input type="text& ...
- 5个对话框和FileStream:文件流
1.private void button1_Click(object sender, EventArgs e) { colorDialog1.ShowDialog();//显示颜色选择器 panel ...
- T对象序列化后T对象中属性字段不见了?
序列化:JsonConvert.SerializeObject(T) 直接在类的上面添加[Table("表名")] 在类上添加属性[DataContract] 在属性上添加属性[D ...
- BZOJ3573: [Hnoi2014]米特运输(树上乱搞)
Time Limit: 20 Sec Memory Limit: 128 MBSubmit: 1669 Solved: 1031[Submit][Status][Discuss] Descript ...