Formatting HDFS
Working on hadoop, especially on test clusters, I have managed to break my HDFS layer and sometimes with no possible redemption, or at least none that I wanted to invest time in. For whatever other reason sometimes you just want to scratch your HDFS and start anew.
Without going on too much details, which is outside the point of this blog post. HDFS is mainly composed of 2 types of elements:
- Namenode: At high level the namenode stores the HDFS namespace, think of it as your file system tree.
- Datanode: this is where your data is actually stored
The Namenode: /hadoop/hdfs/namenode/current
All new edits are written to the the edit log and regularly merged out to an FSImage file, for more concise management. An fsimage file represents the file system state after all modifications up to a specific transaction ID. The seen_txid file, has the last seen transaction. VERSION: contains cluster and hdfs IDs.
For a more detailled explanation: Hdfs metadata
The Datanode: /hadoop/hdfs/data/current
In our example we will only focus on VERSIOn very close to the namenode VERSION.
Hdfs non HA formatting
In non HA everything is simple enough.
- Stop the HDFS Service
- run hadoop namenode -format (as user hdfs)
- clear the data directory on all datanodes
- restart hdfs
At this point your HDFS layer is empty and if you check the VERSION of namenodes and datanodes they should coincide
Hdfs HA formatting
In HA things get a little more complicated. In HA Standby and Active namenodes have a shared storage managed by the journal node service. HA relies on a failover scenario to swap from StandBy to Active Namenode and as any other system in hadoop this uses zookeeper. As you can see a couple more pieces need to made aware of a formatting action.
The initial steps are very close
- Stop the Hdfs service
- Start only the journal nodes (as they will need to be made aware of the formatting)
- On the first namenode (as user hdfs)
- hadoop namenode -format
- hdfs namenode -initializeSharedEdits -force (for the journal nodes)
- hdfs zkfc -formatZK -force (to force zookeeper to reinitialise)
- restart that first namenode
- On the second namenode
- hdfs namenode -bootstrapStandby -force (force synch with first namenode)
- On every datanode clear the data directory
- Restart the HDFS service
This was a very simple step by step guide to formatting. In a later article we will cover actually repairing common errors in HDFS
Formatting HDFS的更多相关文章
- HDFS中namenode启动失败
1.环境配置: -1.core-site.xml文件 <configuration> <property> <name>fs.defaultFS</name& ...
- Hadoop 2.7.4 HDFS+YRAN HA部署
实验环境 主机名称 IP地址 角色 统一安装目录 统一安装用户 sht-sgmhadoopnn-01 172.16.101.55 namenode,resourcemanager /usr/local ...
- Hadoop集群-HDFS集群中大数据运维常用的命令总结
Hadoop集群-HDFS集群中大数据运维常用的命令总结 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 本篇博客会简单涉及到滚动编辑,融合镜像文件,目录的空间配额等运维操作简介.话 ...
- Hadoop集群(二) HDFS搭建
HDFS只是Hadoop最基本的一个服务,很多其他服务,都是基于HDFS展开的.所以部署一个HDFS集群,是很核心的一个动作,也是大数据平台的开始. 安装Hadoop集群,首先需要有Zookeeper ...
- Apache hadoop namenode ha和yarn ha ---HDFS高可用性
HDFS高可用性Hadoop HDFS 的两大问题:NameNode单点:虽然有StandbyNameNode,但是冷备方案,达不到高可用--阶段性的合并edits和fsimage,以缩短集群启动的时 ...
- HDFS ha 格式化报错:a shared edits dir must not be specified if HA is not enabled.
错误内容: Formatting using clusterid: CID-19921335-620f-4e72-a056-899702613a6b2019-01-12 07:28:46,986 IN ...
- hadoop 2.7.3本地环境运行官方wordcount-基于HDFS
接上篇<hadoop 2.7.3本地环境运行官方wordcount>.继续在本地模式下测试,本次使用hdfs. 2 本地模式使用fs计数wodcount 上面是直接使用的是linux的文件 ...
- Hadoop学习之旅二:HDFS
本文基于Hadoop1.X 概述 分布式文件系统主要用来解决如下几个问题: 读写大文件 加速运算 对于某些体积巨大的文件,比如其大小超过了计算机文件系统所能存放的最大限制或者是其大小甚至超过了计算机整 ...
- python基础操作以及hdfs操作
目录 前言 基础操作 hdfs操作 总结 一.前言 作为一个全栈工程师,必须要熟练掌握各种语言...HelloWorld.最近就被"逼着"走向了python开发之路, ...
随机推荐
- python 命令行工具 fire
简介 A library for automatically generating command line interfaces. Python Fire is a library for auto ...
- 布局分析002:入门级的CSS导航弹出菜单
这种弹出菜单非常有意思,也有记录的意义,甚至可以说,掌握了这种弹出菜单,基本上CSS掌握的差不多. 主要涉及下面三个重要知识: CSS的继承性质. relative absolute定位. 子选择符& ...
- 团体程序设计天梯赛L2-021 点赞狂魔 2017-04-18 11:39 154人阅读 评论(0) 收藏
L2-021. 点赞狂魔 时间限制 200 ms 内存限制 65536 kB 代码长度限制 8000 B 判题程序 Standard 作者 陈越 微博上有个"点赞"功能,你可以为你 ...
- 洛谷 P1967 货车运输(克鲁斯卡尔重构树)
题目描述 AAA国有nn n座城市,编号从 11 1到n nn,城市之间有 mmm 条双向道路.每一条道路对车辆都有重量限制,简称限重.现在有 qqq 辆货车在运输货物, 司机们想知道每辆车在不超过车 ...
- handsontable-cell type
在单元格中呈现自定义的元素:不能使用html元素 var data = [ { title: "<a href='http://www.amazon.com/Professional- ...
- osx上使用'cd'命令跳转到别名(alias)目录
在mac上使用windows的共享目录时,在terminal中时法使用cd的,会提示"xxx 不是目录",经过一番的查找,发现了Mac Terminal 'cd' to folde ...
- roadflow作为工作流引擎服务中心webapi说明
将RoadFlow作为工作流引擎服务中心,其它第三方系统如OA,ERP等通过调用RoadFlow对外提供的标准WebApi接口来实现流程发送.退回.查询待办事项.已办事项.查看流转审批过程等操作.实现 ...
- Exp3 免杀原理与实践 20164323段钊阳
网络对抗技术 20164323 Exp3 免杀原理与实践 免杀 一般是对恶意软件做处理,让它不被杀毒软件所检测.也是渗透测试中需要使用到的技术. 要做好免杀,就时清楚杀毒软件(恶意软件检测工具)是如何 ...
- 467. Unique Substrings in Wraparound String
Consider the string s to be the infinite wraparound string of "abcdefghijklmnopqrstuvwxyz" ...
- NOI2019省选模拟赛 第三场
传送门 明明没参加过却因为点进去结果狂掉\(rating\)-- \(A\) 集合 如果我们记 \[f_k=\sum_{i=1}^nT^i{n-i\choose k}\] 那么答案显然就是\(f_{k ...