Lucene dvd dvm文件便是docvalues文件——就是针对field value的列存储
public final class Lucene54DocValuesFormat
extends DocValuesFormat
Encodes the five per-document value types (Numeric,Binary,Sorted,SortedSet,SortedNumeric) with these strategies:
- Delta-compressed: per-document integers written as deltas from the minimum value, compressed with bitpacking. For more information, see
DirectWriter
. - Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used (such as
SmallFloat
), a lookup table is written instead. Each per-document entry is instead the ordinal to this table, and those ordinals are compressed with bitpacking (DirectWriter
). - GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
- Monotonic-compressed: when all numbers are monotonically increasing offsets, they are written as blocks of bitpacked integers, encoding the deviation from the expected delta.
- Const-compressed: when there is only one possible non-missing value, only the missing bitset is encoded.
- Sparse-compressed: only documents with a value are stored, and lookups are performed using binary search.
- Fixed-width Binary: one large concatenated byte[] is written, along with the fixed length. Each document's value can be addressed directly with multiplication (
docID * length
). - Variable-width Binary: one large concatenated byte[] is written, along with end addresses for each document. The addresses are written as Monotonic-compressed numerics.
- Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed numerics. A reverse lookup index is written from a portion of every 1024th term.
- Sorted: a mapping of ordinals to deduplicated terms is written as Binary, along with the per-document ordinals written using one of the numeric strategies above.
- Single: if all documents have 0 or 1 value, then data are written like SORTED.
- SortedSet table: when there are few unique sets of values (< 256) then each set is assigned an id, a lookup table is written and the mapping from document to set id is written using the numeric strategies above.
- SortedSet: a mapping of ordinals to deduplicated terms is written as Binary, an ordinal list and per-document index into this list are written using the numeric strategies above.
- Single: if all documents have 0 or 1 value, then data are written like NUMERIC.
- SortedSet table: when there are few unique sets of values (< 256) then each set is assigned an id, a lookup table is written and the mapping from document to set id is written using the numeric strategies above.
- SortedNumeric: a value list and per-document index into this list are written using the numeric strategies above.
Files:
- .dvd: DocValues data
- .dvm: DocValues metadata
转自:http://lucene.apache.org/core/6_4_2/core/org/apache/lucene/codecs/lucene54/Lucene54DocValuesFormat.html
可以看到占用空间非常小!!!
du -sm elasticsearch/nodes/0/indices/hec_test2/0/index/*
299 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fdt
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fdx
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.fnm
148 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.doc
130 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.tim
5 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene50_0.tip
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene54_0.dvd
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e_Lucene54_0.dvm
1 elasticsearch/nodes/0/indices/hec_test2/0/index/_e.si
1 elasticsearch/nodes/0/indices/hec_test2/0/index/segments_7
0 elasticsearch/nodes/0/indices/hec_test2/0/index/write.lock
Lucene dvd dvm文件便是docvalues文件——就是针对field value的列存储的更多相关文章
- 腾讯Hermes设计概要——数据分析用的是列存储,词典文件前缀压缩,倒排文件递增id、变长压缩、依然是跳表-本质是lucene啊
转自:http://data.qq.com/article?id=817 三.Hermes设计概要 架构描述 系统核心进程均采用分散化设计,根据业务发展需求,可随意扩缩容机器; 周期性数据直接通过td ...
- FileShare文件读写锁解决“文件XXX正由另一进程使用,因此该进程无法访问此文件”(转)
开发过程中,我们往往需要大量与文件交互,读文件,写文件已成家常便饭,本地运行完美,但一上到投产环境,往往会出现很多令人措手不及的意外,或开发中的烦恼,因此,我对普通的C#文件操作做了一次总结,问题大部 ...
- .c和.h文件的区别(头文件与之实现文件的的关系~ )
.c和.h文件的区别 一个简单的问题:.c和.h文件的区别 学了几个月的C语言,反而觉得越来越不懂了.同样是子程序,可以定义在.c文件中,也可以定义在.h文件中,那这两个文件到底在用法上有什么区别呢 ...
- [转载]webarchive文件转换成htm文件
原文地址:webarchive文件转换成htm文件作者:xhbaxf Mac OS X系统带有文件转换功能,可以把webarchive文件变成html文件.方法是: Step 1: 建立一个文件夹 ...
- 怎样将word文件转化为Latex文件:word-to-latex-2.56具体解释
首先推荐大家读一读这篇博文:http://blog.csdn.net/ibingow/article/details/8613556 --------------------------------- ...
- PHP上传文件参考配置大文件上传
PHP用超级全局变量数组$_FILES来记录文件上传相关信息的. 1.file_uploads=on/off 是否允许通过http方式上传文件 2.max_execution_time=30 允许脚本 ...
- R8—批量生成文件夹,批量读取文件夹名称+R文件管理系统操作函数
一. 批量生成文件夹,批量读取文件夹名称 今日,工作中遇到这样一个问题:boss给我们提供了200多家公司的ID代码(如6007.7920等),需要根据这些ID号去搜索下载新闻,从而将下载到的新闻存到 ...
- Python:文件操作总结1——文件基本操作
一.文件的操作流程 1.打开文件,得到文件句柄并赋值给一个变量 2.通过句柄对文件进行操作 3.关闭文件 二.文件的打开与关闭 A.文件的打开——open函数 语法:open(file[,mode[, ...
- linux下压缩成zip文件解压zip文件
linux zip命令的基本用法是: zip [参数] [打包后的文件名] [打包的目录路径] linux zip命令参数列表: -a 将文件转成ASCII模式 -F 尝试修复损坏 ...
随机推荐
- NYOJ 239 月老的难题
月老的难题 时间限制:1000 ms | 内存限制:65535 KB 难度:4 描述 月老准备给n个女孩与n个男孩牵红线,成就一对对美好的姻缘. 现在,由于一些原因,部分男孩与女孩可能结成幸福 ...
- xtu DP Training B. Collecting Bugs
B. Collecting Bugs Time Limit: 10000ms Memory Limit: 64000KB 64-bit integer IO format: %lld Jav ...
- XV6第一个进程
第一个进程 本章通过第一个进程的创建来解释 xv6 是如何开始运行的,让我们得以一窥 xv6 提供的各个抽象是如何实现和交互的.xv6 尽量复用了普通操作的代码来建立第一个进程,避免单独为其撰写代码. ...
- Leetcode 319.灯泡开关
灯泡开关 初始时有 n 个灯泡关闭.第 1 轮,你打开所有的灯泡.第 2 轮,每两个灯泡你关闭一次.第 3 轮,每三个灯泡切换一次开关(如果关闭则开启,如果开启则关闭).第 i 轮,每 i 个灯泡切换 ...
- 动手实操:如何用 Python 实现人脸识别,证明这个杨幂是那个杨幂?
当前,人脸识别应用于许多领域,如支付宝的用户认证,许多的能识别人心情的 AI,也就是人的面部表情,还有能分析人的年龄等等,而这里面有着许多的难度,在这里我想要分享的是一个利用七牛 SDK 简单的实现人 ...
- [转]Fedora22添加国内软件源和本地软件源
Fedora22添加国内软件源和本地软件源 Linux系统和Windows系统一个很大的区别就是软件安装方式,windows系统下安软件,我们去相应的网站下载软件安装包离线安装就可以了.虽然Linux ...
- Couriers(bzoj 3524)
Description 给一个长度为n的序列a.1≤a[i]≤n.m组询问,每次询问一个区间[l,r],是否存在一个数在[l,r]中出现的次数大于(r-l+1)/2.如果存在,输出这个数,否则输出0. ...
- 【2018 Multi-University Training Contest 2】
01: 02: 03: 04:https://www.cnblogs.com/myx12345/p/9394511.html 05: 06: 07:https://www.cnblogs.com/my ...
- Topcoder 658 650 point
Topcoder 658 div2 500 加强版 不过给了<=20,暴力肯定不行. 然后想DP方程,先二分可能需要的最大次数mid; 然后根据 mid 构造 DP方程. 假设x[i]需要 x个 ...
- [Bzoj2039]小Z的袜子 (莫队算法模板题)
2038: [2009国家集训队]小Z的袜子(hose) Time Limit: 20 Sec Memory Limit: 259 MBSubmit: 11866 Solved: 5318[Sub ...