rsync 文件校验及同步原理

参考:http://rsync.samba.org/how-rsync-works.html

我们关注的是其发送与接收校验文件的算法,这里附上原文和我老婆(^_^)的翻译:

The Sender

The sender process reads the file index numbers and associated block checksum sets one at a time from the generator.

发送进程一次从生成器读取一个文件索引号和关联的块校验集合

For each file id the generator sends it will store the block checksums and build a hash index of them for rapid lookup.

 对于生成器发送的每个文件ID,它会存储数据块校验和并生成它们的哈希索引,以进行快速查找 。

Then the local file is read and a checksum is generated for the block beginning with the first byte of the local file. This block checksum is looked for in the set that was sent by the generator, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum”

 然后会读取本地文件,并为以本地文件的第一个字节开头的数据块生成校验和。此数据块校验和在由生成器发送的集中查找,如果未找到匹配,则会将非匹配字节附加到非匹配数据,并且会比较以下一字节开头的数据块。 这称为“rolling checksum” .

If a block checksum match is found it is considered a matching block and any accumulated non-matching data will be sent to the receiver followed by the offset and length in the receiver's file of the matching block and the block checksum generator will be advanced to the next byte after the matching block.

 如果找到数据块校验和匹配,则会将它视为匹配块,所有累积的非匹配数据将被加上在接收端的文件中的匹配数据块的偏移量和长度之后发送到接收端,并且数据块校验和生成器将提前到匹配块之后的下一字节。

Matching blocks can be identified in this way even if the blocks are reordered or at different offsets. This process is the very heart of the rsync algorithm.

可以以这种方式标识匹配块,即使重新排列数据块的顺序或数据块的偏移量不同。此过程是 rsync 算法的核心。

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfe), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file.

Generating the rolling checksums and searching for matches in the checksum set sent by the generator require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The Receiver

The receiver will read from the sender data for each file identified by the file index number. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance.

将数据从基础文件复制到临时文件会使receiver在所有rsync进程中最耗磁盘。小文件可以仍处于缓解此作用的磁盘缓存中,但对于大型文件,由于生成器已移动到其他文件,并且存在sender引起的进一步延迟,缓存可能会"抖动"(thrash)。 数据可能从一个文件随机读取,写入另一文件,如果工作集大于磁盘缓存,则会发生"寻道风暴"(seek storm),进一步影响性能。

看到这儿可能还是一头雾水,好吧,刚bing一下,前面已经有人栽树了,有图有真相:

http://coolshell.cn/articles/7425.html#more-7425

本文为原创内容,转载请注明出自 梁小白博客(http://biangbiang.cnblogs.com)
 
分类: Linux开源软件
标签: rsync

rsync 文件校验及同步原理的更多相关文章

  1. rsync 文件校验及同步原理及rsync server配置

    参考:http://rsync.samba.org/how-rsync-works.html 我们关注的是其发送与接收校验文件的算法,这里附上原文和我老婆(^_^)的翻译: The Sender Th ...

  2. rsync 文件同步和备份

    rsync 是同步文件的利器,一般用于多个机器之间的文件同步与备份,同时也支持在本地的不同目录之间互相同步文件.在这种场景下,rsync 远比 cp 命令和 ftp 命令更加合适,它只会同步需要更新的 ...

  3. rsync + inotify-tools实现文件的实时同步

    文章摘自:http://lxw66.blog.51cto.com/5547576/1331048 rsync 帮助文档:http://man.linuxde.net/rsync 最近有个想法就是部署一 ...

  4. CentOS系统rsync文件同步 安装配置

    rsync是类unix系统下的数据镜像备份工具,从软件的命名上就可以看出来了——remote sync 它的特性如下: 可以镜像保存整个目录树和文件系统. 可以很容易做到保持原来文件的权限.时间.软硬 ...

  5. sersync + rsync 实现文件的实时同步

    这里有一点要特别注意了,就是在你完成备份之后,先不要把本地的文件都给删除了,先把服务停了之后再删除文件, 因为你已删除,检查到两边不一致,他又会把备份端给删除了.所以特别得注意了.这里吃过一次亏. 还 ...

  6. Rsync文件同步

    Rsync文件同步 本章结构 关于rsync 1.一款增量备份工具,remote sync,远程同步,支持本地复制或者与其他SSH.rsync主机同步,官方网站:http://rsync.samba. ...

  7. inotify 与 rsync文件同步实现与问题

    首先分别介绍inotify 与 rsync的使用,然后用两者实现实时文件同步,最后说一下这样的系统存在什么样的问题. 1. inotify 这个具体使用网上很多,参考 inotify-tools 命令 ...

  8. rsync文件同步详解

    一.  环境和测试说明 rsync(remote sync)是unix及类unix平台下的数据镜像备份软件,它不像FTP那样需要全备份,rsync可以根据数据的变化进行差异备份,从而减少数据流量,提高 ...

  9. Rsync文件同步服务

    Rsync简介 Rsync是一款开源的.快速的.多功能的.可实现全量及增量的本地或远程数据同步备份的优秀工具,适用于Unix/Linux/Windows等多种操作系统. Rsync的特性 支持拷贝特殊 ...

随机推荐

  1. perconaXTRADB Cluster在Redhat Linux上的安装

    installing-perconaXTRADB Cluster 5.6 For Redhat 6.4 一.server版本号查看 Root# cat /etc/redhat-release Red ...

  2. 【Unity 3D】学习笔记四十一:关节

    关节 关节组件能够加入至多个游戏对象中,而加入关节的游戏对象将通过关节连接在一起而且感觉连带的物理效果.须要注意的是:关节必须依赖于刚体组件. 关节介绍 关节一共分为5大类:链条关节,固定关节,弹簧关 ...

  3. java设计模式之二抽象工厂模式(Abstract Factory)

    工厂方法模式有一个问题就是,类的创建依赖工厂类,也就是说,如果想要拓展程序,必须对工厂类进行修改,这违背了闭包原则,所以,从设计角度考虑,有一定的问题,如何解决?就用到抽象工厂模式,创建多个工厂类,这 ...

  4. javascript5

    调用对象call object: 声明上下文对象declarative environment record; 作用域链scopechain: 变量解析:variable resolution: 引用 ...

  5. android意图传递参数返回结果(六)

    例如,有两页a,b. a参数传递到页面b页面,b后,将获得的参数的处理页,然后将结果传回与参数的页面a. 1.a主页MainActivity的代码类型,如以下: private Button butt ...

  6. Linux忘记rootpassword

    我们常常会碰到忘记rootpassword的情况,以下是解决之道,  此方法使用绝大多数的Linux发行版:  1. 首先进入grub  2. 在须要编辑的入口处,按下e,在quite后增加     ...

  7. ASP.NET MVC局部视图

    使用ASP.NET MVC局部视图避免JS拼接HTML,编写易于维护的HTML页面   以前使用ASP.NET WebForm开发时,喜欢使用Repeater控件嵌套的方式开发前台页面,这样就不用JS ...

  8. 用CQRS+ES实现DDD

    用CQRS+ES实现DDD 这篇文章应该算是对前三篇的一个补充,在写之前说个题外话,有园友评论这是在用三层架构在写DDD,我的个人理解DDD是一种设计思想,跟具体用什么架构应该没有什么关系,DDD也需 ...

  9. .NET的DTO映射工具AutoMapper

    .NET的DTO映射工具AutoMapper 原文:https://github.com/AutoMapper/AutoMapper/wiki/Getting-started 参考:http://ww ...

  10. SQL点滴4—筛选数据列的类型,字段大小,是否可为空,是否是主键,约束等等信息

    原文:SQL点滴4-筛选数据列的类型,字段大小,是否可为空,是否是主键,约束等等信息 项目需要将Access数据库中的数据导入到SQL Server中,需要检验导入后的数据完整性,数据值是否正确.我们 ...