Frame of Reference and Roaring Bitmaps
https://www.elastic.co/cn/blog/frame-of-reference-and-roaring-bitmaps
2015年2月18日Engineering
Frame of Reference and Roaring Bitmaps
Postings lists
While it may surprise you if you are new to search engine internals, one of the most important building blocks of a search engine is the ability to efficiently compress and quickly decode sorted lists of integers. Why is this useful? As you may know, Elasticsearch shards, which are Lucene indices under the hood, split the data that they store into segments which are regularly merged together. Inside each segment, documents are given an identifier between 0 and the number of documents in the segment (up to 231-1). This is conceptually like an index in an array: it is stored nowhere but is enough to identity an item. Segments store data about documents sequentially, and a doc ID is the index of a document in a segment. So the first document in a segment would have a doc ID of 0, the second 1, etc. until the last document, which has a doc ID equal to the total number of documents in the segment minus one.
Why are these doc IDs useful? An inverted index needs to map terms to the list of documents that contain this term, called a postings list, and these doc IDs that we just discussed are a perfect fit since they can be compressed efficiently.
Frame Of Reference
In order to be able to compute intersections and unions efficiently, we require that these postings lists are sorted. A nice side-effect of this decision is that postings lists can be compressed with delta-encoding.
For instance, if your postings list is [73, 300, 302, 332, 343, 372]
, the list of deltas would be [73, 227, 2, 30, 11, 29]
. What is interesting to note here is that all deltas are between 0 and 255, so you only need one byte per value. This is the technique that Lucene is using in order to encode your inverted index on disk: postings lists are split into blocks of 256 doc IDs and then each block is compressed separately using delta-encoding and bit packing: Lucene computes the maximum number of bits required to store deltas in a block, adds this information to the block header, and then encodes all deltas of the block using this number of bits. This encoding technique is known as Frame Of Reference (FOR) in the literature and has been used since Lucene 4.1.
Here is an example with a block size of 3 (instead of 256 in practice):
Frame of Reference and Roaring Bitmaps的更多相关文章
- OD: Register, Stack Frame, Function Reference
几个重要的 Win32 寄存器 EIP 指令寄存器(Extended Instruction Pointer) 存放一个指针,指向下一条等待执行的指令地址 ESP 栈指针寄存器(Extended St ...
- Elasticsearch 通关教程(七): Elasticsearch 的性能优化
硬件选择 Elasticsearch(后文简称 ES)的基础是 Lucene,所有的索引和文档数据是存储在本地的磁盘中,具体的路径可在 ES 的配置文件../config/elasticsearch. ...
- Elasticsearch 技术分析(九):Elasticsearch的使用和原理总结
前言 之前已经分享过Elasticsearch的使用和原理的知识,由于近期在公司内部做了一次内部分享,所以本篇主要是基于之前的博文的一个总结,希望通过这篇文章能让读者大致了解Elasticsearch ...
- 全文搜索引擎Elasticsearch详细介绍
我们生活中的数据总体分为两种:结构化数据 和 非结构化数据. 结构化数据:也称作行数据,是由二维表结构来逻辑表达和实现的数据,严格地遵循数据格式与长度规范,主要通过关系型数据库进行存储和管理.指具有固 ...
- L ==> E · L · K
三剑客:Elastic Stack 在学习ELK前,先对 Lucene作基本了解. 今天才知道关系型数据库的索引是 B-Tree,罪过... 减少磁盘寻道次数 ---> 提高查询性能 Lucen ...
- 带你走进神一样的Elasticsearch索引机制
更多精彩内容请看我的个人博客 前言 相比于大多数人熟悉的MySQL数据库的索引,Elasticsearch的索引机制是完全不同于MySQL的B+Tree结构.索引会被压缩放入内存用于加速搜索过程,这一 ...
- Busting Frame Busting: a Study of Clickjacking Vulnerabilities on Popular Sites
Busting Frame Busting Reference From: http://seclab.stanford.edu/websec/framebusting/framebust.pdf T ...
- Frames of Reference参考框架
Frames of Reference参考框架 When describing the position and orientation of something (for example, your ...
- Elasticsearch索引原理
转载 http://blog.csdn.net/endlu/article/details/51720299 最近在参与一个基于Elasticsearch作为底层数据框架提供大数据量(亿级)的实时统计 ...
随机推荐
- VMware Workstation 16 启动虚拟机失败(vmmon 版本问题)
问题简述 在 Manjora Linux 下,之前更新过系统,这次启动虚拟机居然就是失败了. 报错信息如下: Version mismatch with vmmon module: expecting ...
- 宿主机网络中其它机器与Docker容器网络互通配置
前言 目前项目采用微服务架构进行开发,Nacos和其它服务部署到Docker中,Docker中容器采用的网络默认是桥接模式(Bridge),默认的子网码是172.17.0.1/16:宿主机是192.1 ...
- Daphile 安装手册 -- 官方文档译文 [原创]
Daphile 安装手册(Daphile Installation) 英文原文:https://www.daphile.com/download/DaphileInstallation.pdf 采集日 ...
- 30天自制操作系统-day1
30天自制操作系统(linux环境)--第一天 我是在CentOS的环境上面实现的,使用ubuntu的环境也是类似的 第一步:因为要对二进制文件进行编辑,所以安装二进制编辑器hexedit(当然其他的 ...
- android中VideoView播放sd卡上面的视频
(1)videoView组件只支持MP4和3gp格式的视屏播放,如果想播放其它视屏格式的文件,还得开发能够播放的视屏播放器 (2)videoView组件功能比较单一,如果想开发功能丰富的播放器,还得重 ...
- AjaxControlToolKit CalendarExtender(日历扩展控件)的使用方法
设置CalendarExtender的TargetControlID为需要显示日期的TextBox的ID,textBox控件的readOnly属性设置为 false ,这样就可以点击textbox控件 ...
- uni-app 顶部tabbar切换
完成样式 项目地址:https://gitee.com/jielov/uni-app-tabbar 顶部tabbar代码 <!--顶部导航栏--> <view class=" ...
- 详解Vue中的computed和watch
作者:小土豆 博客园:https://www.cnblogs.com/HouJiao/ 掘金:https://juejin.cn/user/2436173500265335 1. 前言 作为一名Vue ...
- 求素数个数的优化-LeetCode204
问题 计数质数 统计所有小于非负整数 n 的质数的数量. 示例: 输入: 10 输出: 4 解释: 小于 10 的质数一共有 4 个, 它们是 2, 3, 5, 7 . 第一种解法容易想到但是会 超时 ...
- Openstack OCATA 安装环境说明(一) 未完成版本
1 Openstack简介: 2 实验说明: 3 图例: 4 实验环境说明: 4.1 ) 网卡说明: 网卡名 网 段 连接方式 说明 eth0 10.10.5.0/24 仅主机网络 内部网络自动以IP ...