The Hadoop Distributed File System has been great in providing a cloud-type file system. It is robust (when administered correctly :-)) and highly scalable. However, one of the main drawbacks of HDFS is that each piece of data is replicated in three places. This is acceptable because disk storage is cheap and is becoming cheaper by the day; this isn't a problem if you have a relatively small to medium size cluster. The price difference (in absolute terms) is not much whether you use 15 disks or whether you use 10 disks. If we consider the cost of $1 per GByte, the price difference between fifteen 1 TB disk and ten 1 TB disk is only $5K. But when the total size of your cluster is 10 PBytes, then the costs savings in storing the data in two places versus three is a huge ten million dollars!

The reason HDFS stores disk blocks in triplicate is because it uses commodity hardware and there is non-negligible probability of a disk failure. It has been observed that a replication factor of 3 and the fact the HDFS aggressively detects failures and immediately replicates failed -block-replicas is sufficient to never lose any data in practice. The challenge now is to achieve an effective replication factor of 3 while keeping the real physical replication factor at close to 2! How best to do it than by usingErasure Codes.

I heard about this idea called DiskReduce from the folks at CMU. The CMU PDL Labs has been a powerhouse of research in file systems and it is no surprise that they proposed a elegant way of implementing erasure codes in HDFS. I borrowed heavily from their idea in my implementation of Erasure Codes in HDFS described in HDFS-503. One of the main motivation of my design is to keep the HDFS Erasure Coding as a software layer above HDFS rather than inter-twining it inside of HDFS code. The HDFS code is complex by itself and it is really nice to not have to make it more complex and heavyweight.

Distributed Raid File System consists of two main software components. The first component is the RaidNode, a daemon that creates parity files from specified HDFS files. The second component "raidfs" is a software that is layered over a HDFS client and it intercepts all calls that an application makes to the HDFS client. If the HDFS client encounters corrupted data while reading a file, the raidfs client detects it; it uses the relevant parity blocks to recover the corrupted data (if possible) and returns the data to the application. The application is completely transparent to the fact that parity data was used to satisfy it's read request. The Distributed Raid File System can be configured in such a way that a set of data blocks of a file are combined together to form one or more parity blocks. This allows one to reduce the replication factor of a HDFS file from 3 to 2 while keeping the failure probabilty relatively same as before.

I have seen that using a stripe size of 10 blocks decreases the physical replication factor of a file to 2.2 while keeping the effective replication factor of a file at 3. This typically results in saving 25% to 30% of storage space in a HDFS cluster.

One of the shortcoming of this implementation is that we need a parity file for every file in HDFS. This potentially increases the number of files in the NameNode. To alleviate this problem, I will enhance this implementation (in future) to use the Hadoop Archive feature to archive all the parity files together in larger containers so that the NameNode does not have to support additional files when the HDFS Erasure Coding is switched on. This works reasonably well because it is a very very rare case that the parity files are ever used to satisfy a read request.

ref: http://hadoopblog.blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html

[转载] HDFS and Erasure Codes (HDFS-RAID)的更多相关文章

  1. HDFS Federation(转HDFS Federation(HDFS 联盟)介绍 CSDN)

    转载地址:http://blog.csdn.net/strongerbit/article/details/7013221 HDFS Federation(HDFS 联盟)介绍 1. 当前HDFS架构 ...

  2. vivo 万台规模 HDFS 集群升级 HDFS 3.x 实践

    vivo 互联网大数据团队-Lv Jia Hadoop 3.x的第一个稳定版本在2017年底就已经发布了,有很多重大的改进. 在HDFS方面,支持了Erasure Coding.More than 2 ...

  3. HDFS shell操作及HDFS Java API编程

    HDFS shell操作及HDFS Java API编程 1.熟悉Hadoop文件结构. 2.进行HDFS shell操作. 3.掌握通过Hadoop Java API对HDFS操作. 4.了解Had ...

  4. (第3篇)HDFS是什么?HDFS适合做什么?我们应该怎样操作HDFS系统?

    摘要: 这篇文章会详细介绍HDFS是什么,HDFS的作用,适合和不适合的场景,我们该如何操作HDFS?   HDFS文件系统 Hadoop 附带了一个名为 HDFS(Hadoop分布式文件系统)的分布 ...

  5. Hadoop之HDFS(一)HDFS入门及基本Shell命令操作

    1 . HDFS 基本概念 1.1  HDFS 介绍 HDFS 是 Hadoop Distribute File System 的简称,意为:Hadoop 分布式文件系统.是 Hadoop 核心组件之 ...

  6. HDFS之二:HDFS文件系统JavaAPI接口

    HDFS是存取数据的分布式文件系统,HDFS文件操作常有两种方式,一种是命令行方式,即Hadoop提供了一套与Linux文件命令类似的命令行工具.HDFS操作之一:hdfs命令行操作 另一种是Java ...

  7. hdfs的文件个数 HDFS Quotas Guide

    HDFS Quotas Guide Overview HDFS允许管理员为多个每个目录设置使用的命名空间和空间的配额.命名空间配额和空间配额独立操作,但是这两种类型的配额的管理和实现非常类似. Nam ...

  8. 【转载】Hadoop分布式文件系统HDFS的工作原理详述

    转载请注明来自36大数据(36dsj.com):36大数据 » Hadoop分布式文件系统HDFS的工作原理详述 转注:读了这篇文章以后,觉得内容比较易懂,所以分享过来支持一下. Hadoop分布式文 ...

  9. 【转载】经典漫画讲解HDFS原理

    分布式文件系统比较出名的有HDFS  和 GFS,其中HDFS比较简单一点.本文是一篇描述非常简洁易懂的漫画形式讲解HDFS的原理.比一般PPT要通俗易懂很多.不难得的学习资料. 1.三个部分: 客户 ...

随机推荐

  1. 转:快乐Node码农的十个习惯

    文章来源于:http://www.infoq.com/cn/articles/node.js-habits 从问世到现在将近20年,JavaScript一直缺乏其它有吸引力的编程语言,比如Python ...

  2. 初识Java反射机制

    1.ClassLoader的类加载机制:并非一次性加载,而是需要的时候加载(运行期间动态加载)(Class文件加载到内存的代码段),static语句块在加载后执行一次.dynamic语句块(就是一个语 ...

  3. zoj 3787 Access System

    http://acm.zju.edu.cn/onlinejudge/showProblem.do?problemId=5274 #include <cstdio> #include < ...

  4. 树形dp+MST-hdu-4126-Genghis Khan the Conqueror

    题目链接: http://acm.hdu.edu.cn/showproblem.php?pid=4126 题目意思: 给一图,n个点,m条边,每条边有个花费,给出q条可疑的边,每条边有新的花费,每条可 ...

  5. POJ1789 Truck History(prim)

    题目链接. 分析: 最大的敌人果然不是别人,就是她(英语). 每种代表车型的串,他们的distance就是串中不同字符的个数,要求算出所有串的distance's 最小 sum. AC代码如下: #i ...

  6. -_-#【Canvas】FPS

    <!DOCTYPE html> <html> <head> <meta charset="utf-8"> <title> ...

  7. Robot Framework安装配置 Linux

    Simple introduction Robot Framework is a generic test automation framework for acceptance testing an ...

  8. Controlling How NSThread and NSRunLoop Exit

    http://shaheengandhi.com/controlling-thread-exit/ While most concurrency can be dealt with by using ...

  9. URAL 1081

    题目大意:求词典序下第K个长度为N且无相邻位置都为1的0.1序列.无解时输出-1. 例如: input:     output: 3 1        000(所有符合条件的序列有:000,001,0 ...

  10. [置顶] Android开发之ProcessState和IPCThreadState类分析

    在Android中ProcessState是客户端和服务端公共的部分,作为Binder通信的基础,ProcessState是一个singleton类,每个 进程只有一个对象,这个对象负责打开Binde ...