[转载] HDFS and Erasure Codes (HDFS-RAID)
The Hadoop Distributed File System has been great in providing a cloud-type file system. It is robust (when administered correctly :-)) and highly scalable. However, one of the main drawbacks of HDFS is that each piece of data is replicated in three places. This is acceptable because disk storage is cheap and is becoming cheaper by the day; this isn't a problem if you have a relatively small to medium size cluster. The price difference (in absolute terms) is not much whether you use 15 disks or whether you use 10 disks. If we consider the cost of $1 per GByte, the price difference between fifteen 1 TB disk and ten 1 TB disk is only $5K. But when the total size of your cluster is 10 PBytes, then the costs savings in storing the data in two places versus three is a huge ten million dollars!
The reason HDFS stores disk blocks in triplicate is because it uses commodity hardware and there is non-negligible probability of a disk failure. It has been observed that a replication factor of 3 and the fact the HDFS aggressively detects failures and immediately replicates failed -block-replicas is sufficient to never lose any data in practice. The challenge now is to achieve an effective replication factor of 3 while keeping the real physical replication factor at close to 2! How best to do it than by usingErasure Codes.
I heard about this idea called DiskReduce from the folks at CMU. The CMU PDL Labs has been a powerhouse of research in file systems and it is no surprise that they proposed a elegant way of implementing erasure codes in HDFS. I borrowed heavily from their idea in my implementation of Erasure Codes in HDFS described in HDFS-503. One of the main motivation of my design is to keep the HDFS Erasure Coding as a software layer above HDFS rather than inter-twining it inside of HDFS code. The HDFS code is complex by itself and it is really nice to not have to make it more complex and heavyweight.
Distributed Raid File System consists of two main software components. The first component is the RaidNode, a daemon that creates parity files from specified HDFS files. The second component "raidfs" is a software that is layered over a HDFS client and it intercepts all calls that an application makes to the HDFS client. If the HDFS client encounters corrupted data while reading a file, the raidfs client detects it; it uses the relevant parity blocks to recover the corrupted data (if possible) and returns the data to the application. The application is completely transparent to the fact that parity data was used to satisfy it's read request. The Distributed Raid File System can be configured in such a way that a set of data blocks of a file are combined together to form one or more parity blocks. This allows one to reduce the replication factor of a HDFS file from 3 to 2 while keeping the failure probabilty relatively same as before.
I have seen that using a stripe size of 10 blocks decreases the physical replication factor of a file to 2.2 while keeping the effective replication factor of a file at 3. This typically results in saving 25% to 30% of storage space in a HDFS cluster.
One of the shortcoming of this implementation is that we need a parity file for every file in HDFS. This potentially increases the number of files in the NameNode. To alleviate this problem, I will enhance this implementation (in future) to use the Hadoop Archive feature to archive all the parity files together in larger containers so that the NameNode does not have to support additional files when the HDFS Erasure Coding is switched on. This works reasonably well because it is a very very rare case that the parity files are ever used to satisfy a read request.
ref: http://hadoopblog.blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html
[转载] HDFS and Erasure Codes (HDFS-RAID)的更多相关文章
- HDFS Federation(转HDFS Federation(HDFS 联盟)介绍 CSDN)
转载地址:http://blog.csdn.net/strongerbit/article/details/7013221 HDFS Federation(HDFS 联盟)介绍 1. 当前HDFS架构 ...
- vivo 万台规模 HDFS 集群升级 HDFS 3.x 实践
vivo 互联网大数据团队-Lv Jia Hadoop 3.x的第一个稳定版本在2017年底就已经发布了,有很多重大的改进. 在HDFS方面,支持了Erasure Coding.More than 2 ...
- HDFS shell操作及HDFS Java API编程
HDFS shell操作及HDFS Java API编程 1.熟悉Hadoop文件结构. 2.进行HDFS shell操作. 3.掌握通过Hadoop Java API对HDFS操作. 4.了解Had ...
- (第3篇)HDFS是什么?HDFS适合做什么?我们应该怎样操作HDFS系统?
摘要: 这篇文章会详细介绍HDFS是什么,HDFS的作用,适合和不适合的场景,我们该如何操作HDFS? HDFS文件系统 Hadoop 附带了一个名为 HDFS(Hadoop分布式文件系统)的分布 ...
- Hadoop之HDFS(一)HDFS入门及基本Shell命令操作
1 . HDFS 基本概念 1.1 HDFS 介绍 HDFS 是 Hadoop Distribute File System 的简称,意为:Hadoop 分布式文件系统.是 Hadoop 核心组件之 ...
- HDFS之二:HDFS文件系统JavaAPI接口
HDFS是存取数据的分布式文件系统,HDFS文件操作常有两种方式,一种是命令行方式,即Hadoop提供了一套与Linux文件命令类似的命令行工具.HDFS操作之一:hdfs命令行操作 另一种是Java ...
- hdfs的文件个数 HDFS Quotas Guide
HDFS Quotas Guide Overview HDFS允许管理员为多个每个目录设置使用的命名空间和空间的配额.命名空间配额和空间配额独立操作,但是这两种类型的配额的管理和实现非常类似. Nam ...
- 【转载】Hadoop分布式文件系统HDFS的工作原理详述
转载请注明来自36大数据(36dsj.com):36大数据 » Hadoop分布式文件系统HDFS的工作原理详述 转注:读了这篇文章以后,觉得内容比较易懂,所以分享过来支持一下. Hadoop分布式文 ...
- 【转载】经典漫画讲解HDFS原理
分布式文件系统比较出名的有HDFS 和 GFS,其中HDFS比较简单一点.本文是一篇描述非常简洁易懂的漫画形式讲解HDFS的原理.比一般PPT要通俗易懂很多.不难得的学习资料. 1.三个部分: 客户 ...
随机推荐
- 创建局域网内远程git仓库,并将本地仓库push推到远程仓库中
转载请注明出处 http://www.goteny.com/articles/2014/06/136.html http://www.cnblogs.com/zjjne/p/3778640.html ...
- 转:Node.js异步处理CPU密集型任务的新思路
原文来自于:http://www.infoq.com/cn/articles/new-idea-of-nodejs-asynchronous-processing-tasks?utm_source=i ...
- Ubuntu 启用root账户
设置分配很简单,只要为root设置一个root密码就行了: $ sudo passwd root 之后会提示要输入root用户的密码,连续输入root密码,再使用:$ su 就可以切换成超级管理员用户 ...
- Facebook 开源三款图像识别人工智能软件
Facebook今天开源了三款人工智能图像分割(Image Segmentation)软件,分别是DeepMask.SharpMask和MultiPathNet,三款工具相互配合完成一个完整的图像识别 ...
- JavaScript权威指南学习笔记6
这两天主要翻看了书中的第18-22章,重点看了第17章:事件化处理,其它几章节主要是翻了下书知道有相关的概念,没有真正理解其中的内容,或者没有考虑究竟如何能把里面的内容应用到实际的项目中.说的讽刺一点 ...
- HDOJ 1285 确定比赛名次(拓扑排序)
Problem Description 有N个比赛队(1<=N<=500),编号依次为1,2,3,....,N进行比赛,比赛结束后,裁判委员会要将所有参赛队伍从前往后依次排名,但现在裁判委 ...
- openSource clouds
学习当前较主流的开源云基础设施管理软件by Ruiy summarize publish; 我擦,有不时候Ruiy干事也就那吊风格,啥事也就那么随口一说,你们太sensitivity,同网络延迟对存储 ...
- Java中直接输出一个类的对象
例如 package com.atguigu.java.fanshe; public class Person { String name; private int age; public Strin ...
- 使用Jekyll搭建免费的Github Pages个人博客
一.Git 1.Git概述 Git is a free and open source distributed version control system designed to handle ev ...
- SpringMVC 之文件上传 MultipartResolver
基于前面文章的基础上. 一.准备 需要的jar 二.配置 1. spmvc-servlet.xml <?xml version="1.0" encoding=" ...