fsync sync

fsync/sync
sync is a standard system call in the Unix operating system, which commits to disk all data in the kernel filesystem buffers,data which has been scheduled for writing via low-level I/O system calls.Higher-level I/O layers such as stdio may maintain separate buffers of their own.The related system call fsync() commits just the buffered data relating to a specified file descriptor.[1] fdatasync() is also available to write out just the changes made to the data in the file, and not necessarily the file's related metadata.

inside a shard

inverted index

  1. 词汇列表
  2. 相似度算法
  3. 词汇:大小写,单数复数,同义词

每个被索引的字段都有倒排索引
every indexed field in a JSON document has its own inverted index.

写到磁盘上的inverted index是不可变的。不可变的好处有:
  1. 不需要加锁,不用考虑更新时的多线程问题
  2. 当index被写入filesystem cache的时候,就一直在cache里了,因为他是不变的,从内存里读取而不用访问磁盘,
    提高性能
  3. 其他的缓存,比如filter cache,也将是一直有效的,因为index不变,所以其他的缓存也不需要在数据变化是跟着变化
  4. 允许数据压缩(Writing a single large inverted index allows the data to be compressed, reducing costly disk I/O and the amount of RAM needed to cache the index)
inverted index不可变的缺点
  1. 如果需要让一个新文档可以被搜索,就要重建整个索引,这就限制了一个index的数据容量和索引重建的频率
如何让索引可以更新,任然具有不可变的优势
  1. 使用更多的索引
  2. per-segment search的概念,一个片段是一个倒排索引。
    一个shard有多个segments
  3. 一个分片就是一个 Lucene index,es的一个索引是多个分片的集合
  4. 新文档存在缓存里的indexing buffer,每隔一段时间提交一次
indexing buffer提交时做什么
  1. A Lucene index 的内存缓冲区里的新文档准备提交
  2. 一个新的片段(a supplementary inverted index)写入到磁
  3. 这个new segment加入到commit point,缓冲区清空。
    commit point lists all known segments
  4. 所有在filesystem cache的数据被写入到文件(持久化)

查询时,所有的segment会被轮流查询,

segment是不可变的,没法从老的片段删除或添加文档。所以每个commit point有一个.del文件,
里面记录了哪个片段的哪个文档被删除了,当文档更新时,老版本的文档被表示删除,新版本的文档索引到新segment里

如果让变化的文档更快的searchable

瓶颈在于磁盘,提交一个new segment到磁盘需要fsync,fsync是昂贵的。
在es和磁盘直接的是filesystem cache,new segment先写入filesystemcache,之后在写入磁盘。
这个过程叫refresh,分片默认每秒refresh,配置参数refresh_interval,

PUT /my_logs
{
"settings": {
"refresh_interval": "30s"
}
}

这个参数可以动态的修改。可以在建立索引时关闭refresh,使用时打开

PUT /my_logs/_settings
{ "refresh_interval": -1 } PUT /my_logs/_settings
{ "refresh_interval": "1s" }

持久化

full commit: 将在filesystem cache里的segment写入磁盘,commit point。用在失败后恢复
commit point lists all known segments,es在启动和重新打卡索引时,通过commit point知道segments属于哪个shards.
当full commit时文件改变了怎么办?

translog

translog记录了es发生的每个行为。
文档先添加到in-memory buffer, 再添加到translog.
refresh的时候,buffer清空,translog不变。
The docs in the in-memory buffer are written to a new segment, without an fsync.
The segment is opened to make it visible to search.
The in-memory buffer is cleared.

full commit

flush + create new translog
当translog太大或者一定时间后,index is flushed,创建新的translog.

  1. Any docs in the in-memory buffer are written to a new segment.
  2. The buffer is cleared.
  3. A commit point is written to disk.
  4. The filesystem cache is flushed with an fsync.
  5. The old translog is delete

es启动时,通过last commit point来恢复segments,接着重新执行translog里记录的操作
(When starting up, Elasticsearch will use the last commit point to recover known segments from disk, and will then replay all operations in the translog to add the changes that happened after the last commit.)

translog还被用来做实时的CRUD,当需要通过id retrieve, update, or delete a document,会先检查translog有没有更改,再去segment取文档。这就提供了实时访问最新的文档的方式。

full commit and truncating the translog is called flush.分片默认30分钟flush或当translog太大的时候

translog
  1. index.translog.sync_interval:
    How often the translog is fsynced to disk and committed, regardless of write operations. Defaults to 5s.
  2. index.translog.durability:
    每次索引,删除,更新,或批量请求之后,是否需要fsync和提交translog,
    request: (default) fsync and commit after every write request((e.g. index, delete, update, bulk).). In the event of hardware failure, all acknowledged writes will already have been committed to disk
    async: (lose sync_interval's worth of data )fsync and commit in the background every sync_interval. In the event of hardware failure, all acknowledged writes since the last automatic commit will be discarded.
  3. index.translog.fs.type:
    Whether to buffer writes to the transaction log in memory or not. This setting accepts the following parameters:
    buffered: (default) Translog writes first go to a 64kB buffer in memory, and are only written to the disk when the buffer is full, or when an fsync is triggered by a write request or the sync_interval.
    simple: Translog writes are written to the file system immediately, without buffering. However, these writes will only be persisted to disk when an fsync and commit is triggered by a write request or the sync_interval.

segment merge

自动refresh每秒就创建一个segment,每次搜索都会查询每个segment,so,segment越多查询越慢。
es会在后台合并segment,小的合并到大的,这个时候那些已经删除的旧的文档就会从文件系统清除。删除的文档和旧版本的修改过的文档不会复制到新的大segment里
合并结束之后:

  1. The new segment is flushed to disk.
  2. A new commit point is written that includes the new segment and excludes the old, smaller 3. segments.
  3. The new segment is opened for search.
  4. The old segments are deleted.
optimize api

强制合并的api。强制分片的segment数量小于max_num_segments 参数。不应该在活跃的索引上使用。

POST /logstash-2014-10/_optimize?max_num_segments=1

optimize 出发的merge是完全没有限制的,他们可能用掉所有的I/O, If you plan on optimizing an index, you should use shard allocation (see Migrate Old Indices) to first move the index to a node where it is safe to run.

inside a shard的更多相关文章

  1. 可以执行全文搜索的原因 Elasticsearch full-text search Kibana RESTful API with JSON over HTTP elasticsearch_action es 模糊查询

    https://www.elastic.co/guide/en/elasticsearch/guide/current/getting-started.html Elasticsearch is a ...

  2. cacheed 限制 4节点 3000万 es 批量删除 shell脚本练习 elasticsearch_action

    文件分割 "www.laiwunews.cn/xinxi/25324717.html""www.zznews.cn/xinxi/10411214.html"&q ...

  3. vsftpd:500 OOPS: vsftpd: refusing to run with writable root inside chroot ()错误的解决方法

    ---恢复内容开始--- 最近在安装了vsftpd后 添加了虚拟账户后 新建用户 为新用户创立独立的工作目录 因为虚拟用户在工作目录需要上传文件 所以必须拥有此目录的W权限,但每当给此目录加上W权限后 ...

  4. 解决vsftpd的refusing to run with writable root inside chroot错误

    参考 http://www.cnblogs.com/CSGrandeur/p/3754126.html 在Ubuntu下用 vsftpd 配置FTP服务器,配置 “ sudo chmod a-w /h ...

  5. MongoDBV3.0.7版本(shard+replica)集群的搭建及验证

    集群的模块介绍: 从MongoDB官方给的集群架构了解,整个集群主要有4个模块:Config Server.mongs. shard.replica set: Config Server:用来存放集群 ...

  6. 《Inside UE4》目录

    <Inside UE4>目录 InsideUE4 UE4无疑是非常优秀的世界上最顶尖的引擎之一,性能和效果都非常出众,编辑器工作流也非常的出色,更难得宝贵的是完全的开源让我们有机会去从中吸 ...

  7. 在子线程中new Handler报错--Can't create handler inside thread that has not called Looper.prepare()

    在子线程中new一个Handler为什么会报以下错误? java.lang.RuntimeException:  Can't create handler inside thread that has ...

  8. 《Inside UE4》-0-开篇

    <Inside UE4>-0-开篇 InsideUE4   前言 VR行业发展是越来越火热了,硬件设备上有HTC VIVE,Oculus rift,PS VR,各种魔镜:平台上有Steam ...

  9. 500 OOPS: vsftpd: refusing to run with writable root inside chroot()

    Ubuntu 12.04 64bit系统下安装的vsftpd,在登陆时提示500 OOPS: vsftpd: refusing to run with writable root inside chr ...

随机推荐

  1. BZOJ 1010 HNOI2008 玩具装箱 斜率优化

    题目链接: http://www.lydsy.com/JudgeOnline/problem.php?id=1010 Description P教授要去看奥运,但是他舍不下他的玩具,于是他决定把所有的 ...

  2. HDU 4722 Good Numbers(位数DP)(2013 ACM/ICPC Asia Regional Online ―― Warmup2)

    Description If we sum up every digit of a number and the result can be exactly divided by 10, we say ...

  3. 【iOS开发】iOS开发CGRectGetMidX. CGRectGetMidY.CGRectGetMinY. CGRectGetMaxY. CGRectGetMinX. CGRectGetMaxX的使用

    UILabel *label = [[UILabel alloc]initWithFrame:CGRectMake(10, 10, 110, 150)]; label.backgroundColor ...

  4. systemtap没找到函数变量

    为啥systemtap没找到函数 hon@station6:~/codebox/stap/net$ sudo stap -L 'kernel.function("sock_recvmsg_n ...

  5. RunKit & NPM

    RunKit + NPM Try any Node.js package right in your browser https://npm.runkit.com/segmentit

  6. 51nod 1565模糊搜索(FFT)

    题目大意就是字符串匹配,不过有一个门限k而已 之前有提到过fft做字符串匹配,这里和之前那种有些许不同 因为只有A,C,G,T四种字符,所以就考虑构造4个01序列 例如,模板串a关于'A'的01序列中 ...

  7. 【题解】NOIP2016换教室

    哇好开心啊!写的时候真的全然对于这个加法没有把握,但还是大着胆子试着写了一下——竟然过了样例?于是又调了一下就过啦. 不过想想也觉得是正确的吧,互相独立的事件对于期望的影响自然也是相互独立的,可以把所 ...

  8. [USACO06NOV]玉米田Corn Fields

    题面描述 状压dp. 设\(f[i][sta]\)为第\(i\)层状态为\(sta\)的方案数. 然后每次可以枚举上一层的状态以及本层的状态,然后如果不冲突且满足地图的要求,则转移. 时间复杂度\(O ...

  9. Jsp上传组件Smartupload介绍

    <form action="UploadServlet" enctype="multipart/form-data" method="post& ...

  10. 【ZJ选讲·调整】

    给出n个点,m条有向边(带正权),起点S,终点T.(n<=2000,m<=30000) 再给出一个k,表示可以把最多k条边的权值调整为任意非负整数.(k<=100) 问是否可以通 ...