原文链接：lxw的大数据田地 » hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)

关键字：hdfs fsck、block、locations

在HDFS中，提供了fsck命令，用于检查HDFS上文件和目录的健康状态、获取文件的block信息和位置信息等。

fsck命令必须由HDFS超级用户来执行，普通用户无权限。

  1. [hadoop@dev ~]$ hdfs fsck
  2. Usage: DFSck [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]]
  3. start checking from this path
  4. -move move corrupted files to /lost+found
  5. -delete delete corrupted files
  6. -files print out files being checked
  7. -openforwrite print out files opened for write
  8. -includeSnapshots include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
  9. -list-corruptfileblocks print out list of missing blocks and files they belong to
  10. -blocks print out block report
  11. -locations print out locations for every block
  12. -racks print out network topology for data-node locations

下面介绍每一个选项的含义及用法。

查看文件中损坏的块（-list-corruptfileblocks）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -list-corruptfileblocks
  2. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' has 0 CORRUPT files

将损坏的文件移动至/lost+found目录（-move）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -move
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:36:35 CST 2015
  3. .Status: HEALTHY
  4. Total size: 13497058 B
  5. Total dirs: 0
  6. Total files: 1
  7. Total symlinks: 0
  8. Total blocks (validated): 1 (avg. block size 13497058 B)
  9. Minimally replicated blocks: 1 (100.0 %)
  10. Over-replicated blocks: 0 (0.0 %)
  11. Under-replicated blocks: 0 (0.0 %)
  12. Mis-replicated blocks: 0 (0.0 %)
  13. Default replication factor: 2
  14. Average block replication: 2.0
  15. Corrupt blocks: 0
  16. Missing replicas: 0 (0.0 %)
  17. Number of data-nodes: 15
  18. Number of racks: 1
  19. FSCK ended at Thu Aug 13 09:36:35 CST 2015 in 1 milliseconds
  20. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

删除损坏的文件（-delete）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 -delete
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168 at Thu Aug 13 09:37:58 CST 2015
  3. .Status: HEALTHY
  4. Total size: 13497058 B
  5. Total dirs: 0
  6. Total files: 1
  7. Total symlinks: 0
  8. Total blocks (validated): 1 (avg. block size 13497058 B)
  9. Minimally replicated blocks: 1 (100.0 %)
  10. Over-replicated blocks: 0 (0.0 %)
  11. Under-replicated blocks: 0 (0.0 %)
  12. Mis-replicated blocks: 0 (0.0 %)
  13. Default replication factor: 2
  14. Average block replication: 2.0
  15. Corrupt blocks: 0
  16. Missing replicas: 0 (0.0 %)
  17. Number of data-nodes: 15
  18. Number of racks: 1
  19. FSCK ended at Thu Aug 13 09:37:58 CST 2015 in 1 milliseconds
  20. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00168' is HEALTHY

检查并列出所有文件状态（-files）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -files
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:39:38 CST 2015
  3. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ dir
  4. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/_SUCCESS 0 bytes, 0 block(s): OK
  5. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00000 13583807 bytes, 1 block(s): OK
  6. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00001 13577427 bytes, 1 block(s): OK
  7. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00002 13588601 bytes, 1 block(s): OK
  8. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00003 13479213 bytes, 1 block(s): OK
  9. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00004 13497012 bytes, 1 block(s): OK
  10. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00005 13557451 bytes, 1 block(s): OK
  11. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00006 13580267 bytes, 1 block(s): OK
  12. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00007 13486035 bytes, 1 block(s): OK
  13. /hivedata/warehouse/liuxiaowen.db/lxw_product_names/part-00008 13481498 bytes, 1 block(s): OK
  14. ...

检查并打印正在被打开执行写操作的文件（-openforwrite）

  1. [hadoop@dev ~]$ hdfs fsck /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ -openforwrite
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /hivedata/warehouse/liuxiaowen.db/lxw_product_names/ at Thu Aug 13 09:41:28 CST 2015
  3. ....................................................................................................
  4. ....................................................................................................
  5. .Status: HEALTHY
  6. Total size: 2704782548 B
  7. Total dirs: 1
  8. Total files: 201
  9. Total symlinks: 0
  10. Total blocks (validated): 200 (avg. block size 13523912 B)
  11. Minimally replicated blocks: 200 (100.0 %)
  12. Over-replicated blocks: 0 (0.0 %)
  13. Under-replicated blocks: 0 (0.0 %)
  14. Mis-replicated blocks: 0 (0.0 %)
  15. Default replication factor: 2
  16. Average block replication: 2.0
  17. Corrupt blocks: 0
  18. Missing replicas: 0 (0.0 %)
  19. Number of data-nodes: 15
  20. Number of racks: 1
  21. FSCK ended at Thu Aug 13 09:41:28 CST 2015 in 10 milliseconds
  22. The filesystem under path '/hivedata/warehouse/liuxiaowen.db/lxw_product_names/' is HEALTHY

打印文件的Block报告（-blocks）

需要和-files一起使用。

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2
  10. 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2
  11. 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2
  12. 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2
  13. 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2
  14. 10. BP-1034052771-172.16.212.130-1405595752491:blk_1075893003_2152402 len=134217728 repl=2
  15. 11. BP-1034052771-172.16.212.130-1405595752491:blk_1075893004_2152403 len=134217728 repl=2
  16. 12. BP-1034052771-172.16.212.130-1405595752491:blk_1075893005_2152404 len=134217728 repl=2
  17. 13. BP-1034052771-172.16.212.130-1405595752491:blk_1075893006_2152405 len=134217728 repl=2
  18. 14. BP-1034052771-172.16.212.130-1405595752491:blk_1075893007_2152406 len=134217728 repl=2
  19. ...

其中，/logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): 表示文件的总大小和block数；

0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2

1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2

2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2

前面的0. 1. 2.代表该文件的block索引，56的文件块，就从0-55;

BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381表示block id；

len=134217728 表示该文件块大小；

repl=2 表示该文件块副本数；

打印文件块的位置信息（-locations）

需要和-files -blocks一起使用。

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [172.16.212.139:50010, 172.16.212.135:50010]
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.133:50010]
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [172.16.212.136:50010, 172.16.212.141:50010]
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.135:50010]
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [172.16.212.142:50010, 172.16.212.139:50010]
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.139:50010]
  10. 6. BP-1034052771-172.16.212.130-1405595752491:blk_1075892999_2152398 len=134217728 repl=2 [172.16.212.141:50010, 172.16.212.135:50010]
  11. 7. BP-1034052771-172.16.212.130-1405595752491:blk_1075893000_2152399 len=134217728 repl=2 [172.16.212.144:50010, 172.16.212.142:50010]
  12. 8. BP-1034052771-172.16.212.130-1405595752491:blk_1075893001_2152400 len=134217728 repl=2 [172.16.212.133:50010, 172.16.212.138:50010]
  13. 9. BP-1034052771-172.16.212.130-1405595752491:blk_1075893002_2152401 len=134217728 repl=2 [172.16.212.140:50010, 172.16.212.134:50010]
  14. ...

和打印出的文件块信息相比，多了一个文件块的位置信息：[172.16.212.139:50010, 172.16.212.135:50010]

打印文件块位置所在的机架信息（-racks）

  1. [hadoop@dev ~]$ hdfs fsck /logs/site/2015-08-08/lxw1234.log -files -blocks -locations -racks
  2. FSCK started by hadoop (auth:SIMPLE) from /172.16.212.17 for path /logs/site/2015-08-08/lxw1234.log at Thu Aug 13 09:45:59 CST 2015
  3. /logs/site/2015-08-08/lxw1234.log 7408754725 bytes, 56 block(s): OK
  4. 0. BP-1034052771-172.16.212.130-1405595752491:blk_1075892982_2152381 len=134217728 repl=2 [/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]
  5. 1. BP-1034052771-172.16.212.130-1405595752491:blk_1075892983_2152382 len=134217728 repl=2 [/default-rack/172.16.212.140:50010, /default-rack/172.16.212.133:50010]
  6. 2. BP-1034052771-172.16.212.130-1405595752491:blk_1075892984_2152383 len=134217728 repl=2 [/default-rack/172.16.212.136:50010, /default-rack/172.16.212.141:50010]
  7. 3. BP-1034052771-172.16.212.130-1405595752491:blk_1075892985_2152384 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.135:50010]
  8. 4. BP-1034052771-172.16.212.130-1405595752491:blk_1075892997_2152396 len=134217728 repl=2 [/default-rack/172.16.212.142:50010, /default-rack/172.16.212.139:50010]
  9. 5. BP-1034052771-172.16.212.130-1405595752491:blk_1075892998_2152397 len=134217728 repl=2 [/default-rack/172.16.212.133:50010, /default-rack/172.16.212.139:50010]
  10. ...

和前面打印出的信息相比，多了机架信息：[/default-rack/172.16.212.139:50010, /default-rack/172.16.212.135:50010]

fsck获取文件的block信息和位置信息的更多相关文章

  1. PHP获取访客IP、地区位置信息、浏览器、来源页面

    不多说了,每个方法都注释了,可以直接用: <?php //这个类似用来获取访客信息的 //方便统计 class visitorInfo { //获取访客ip public function ge ...

  2. JAVA获取文件本身所在的磁盘位置

    我们在做java开发(纯java程序,或者java web开发)时,经常会遇到需要读取配置文件的需求,如果我们将文件所在位置的信息直接写到程序中,例如:e:\workspace\javagui\bin ...

  3. 获取访客IP、地区位置信息、浏览器、来源页面

    <?php //这个类似用来获取访客信息的 //方便统计 class visitorInfo { //获取访客ip public function getIp() { $ip=false; if ...

  4. hdfs fsck命令查看HDFS文件对应的文件块信息(Block)和位置信息(Locations)

    关键字:hdfs fsck.block.locations 在HDFS中,提供了fsck命令,用于检查HDFS上文件和目录的健康状态.获取文件的block信息和位置信息等. fsck命令必须由HDFS ...

  5. C# 图像处理:获取鼠标位置信息(全局)

    Point ms = Control.MousePosition; //获取鼠标位置 this.label2.Text = string.Format("{0}:{1}", ms. ...

  6. jquery获取元素在文档中的位置信息以及滚动条位置（转）

    jquery获取元素在文档中的位置信息以及滚动条位置 http://blog.csdn.net/qq_34095777/article/details/78750886     原文链接 原创 201 ...

  7. 定位 - CoreLocation - 打印位置信息

    1. 导入框架 <CoreLocation.framework>, 引入头文件 import <CoreLocation/CoreLocation.h>; 2. 创建管理者对象 ...

  8. lucene反向索引——倒排表无论是文档号及词频，还是位置信息，都是以跳跃表的结构存在的

    转自:http://www.cnblogs.com/forfuture1978/archive/2010/02/02/1661436.html 4.2. 反向信息 反向信息是索引文件的核心,也即反向索 ...

  9. Android获取位置信息的方法总结

    1.位置服务的简介:位置服务,英文翻译为Location-Based Services,缩写为LBS,又称为定位服务或基于位置的服务,融合了GPS定位.移动通信.导航等多种技术,提供与空间位置相关的综 ...

随机推荐

  1. Appium 介绍及环境安装

    Appium是一个可用于测试iOS. Android操作系统和Windows桌面平台原生应用,移动网页应用和混合应用的自动化测试框架. 原生应用(Native App):用 android.iOS或者 ...

  2. Vue基础语法与指令

    项目初始化 用vscode打开终端,输入npm init -y生成package.json 然后安装vue npm install vue 需要注意的是,我遇到了这个问题 出现原因:文件夹名和生成的p ...

  3. maven 报的一堆错

    今天初学maven,刚开始下载的是Apache-maven-3.6.2然后配置运行一个servlet,但是在pom.xml中写jar包坐标时一直报错显示红色,本地仓库和官网上的中央仓库都试过了就是依赖 ...

  4. 【Software Test】Basic Of ST

    文章目录 Learning Objective Introduction Software Applications Before Software Testing What is testing? ...

  5. top有用的开关控制命令

    [原创]本文为原创博文,转发请注明出处:https://www.cnblogs.com/dingbj/p/top_command.html 今天偶然用到top命令,在动态刷新的界面上输入h顺便看了下帮 ...

  6. 记录Js动态加载页面.append、html、appendChild、repend添加元素节点不生效以及解决办法

    今天再优化blog页面的时候添加了个关注按钮和图片,但是页面上这个按钮和图片时有时无,本来是搞后端的,被这个前端的小问题搞得抓耳挠腮的! 网上各种查询解决方案,把我解决问题的艰辛历程分享出来,希望大家 ...

  7. cursor pin s和cursor pin s wait on x

    1.cursor pin s是一个共享锁,一般情况下是因为发生在SQL短时间内大量执行 案例:在生产库中,突然出现大量的cursor pin s的等待,询问是否有动作后,同事说有编译存储过程(被误导了 ...

  8. oracle编译表上失效USERDBY脚本

    对表进行DLL操作之后,依赖这个表的一些存储过程,触发器等会失效,可以用下边的脚本进行重编译 /* Formatted on 2020/7/8 上午 09:31:31 (QP5 v5.163.1008 ...

  9. D2Admin 登录用户重新初始话右侧菜单

    背景 最近用到D2Admin开发项目,用户登录菜单要根据用户角色权限获取,但是又不想用官网的方案(vue基于d2-admin的RBAC权限管理解决方案),所以自己加了个只修改 menuAside的方案 ...

  10. SwiftUI 中一些和响应式状态有关的属性包装器的用途

    SwiftUI 借鉴了 React 等 UI 框架的概念,通过 state 的变化,对 View 进行响应式的渲染.主要通过 @State, @StateObject, @ObservedObject ...