HDFS中大数据常见运维指令总结

一、查看HDFS下的参数信息

[root@master ~]# hdfs
Usage: hdfs [--config confdir] COMMAND
       where COMMAND is one of:
  dfs                  run a filesystem command on the file systems supported in Hadoop.
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  journalnode          run the DFS journalnode
  zkfc                 run the ZK Failover Controller daemon
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  haadmin              run a DFS HA admin client
  fsck                 run a DFS filesystem checking utility
  balancer             run a cluster balancing utility
  jmxget               get JMX exported values from NameNode or DataNode.
  mover                run a utility to move block replicas across
                       storage types
  oiv                  apply the offline fsimage viewer to an fsimage
  oiv_legacy           apply the offline fsimage viewer to an legacy fsimage
  oev                  apply the offline edits viewer to an edits file
  fetchdt              fetch a delegation token from the NameNode
  getconf              get config values from configuration
  groups               get the groups which users belong to
  snapshotDiff         diff two snapshots of a directory or diff the
                       current directory contents with a snapshot
  lsSnapshottableDir   list all snapshottable dirs owned by the current user
                                                Use -help to see options
  portmap              run a portmap service
  nfs3                 run an NFS version 3 gateway
  cacheadmin           configure the HDFS cache
  crypto               configure HDFS encryption zones
  storagepolicies      get all the existing block storage policies
  version              print the version

Most commands print help when invoked w/o parameters.

二、hdfs与dfs结合使用的参数信息

[root@master ~]# hdfs dfs
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
        [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-count [-q] [-h] <path> ...]
        [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] <path> ...]
        [-expunge]
        [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] <src> <localdst>]
        [-help [cmd ...]]
        [-ls [-d] [-h] [-R] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
        [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
        [-setfattr {-n name [-v value] | -x name} <path>]
        [-setrep [-R] [-w] <rep> <path> ...]
        [-stat [format] <path> ...]
        [-tail [-f] <file>]
        [-test -[defsz] <path>]
        [-text [-ignoreCrc] <src> ...]
        [-touchz <path> ...]
        [-usage [cmd ...]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

其他一些操作命令

说明：仅记录用于学习的指令。

1、追加文件内容到hdfs文件系统中的文件

hdfs dfs -appendToFile testc.sh /top.sh

2、查看hadoop的Sequencefile文件内容

[root@master ~]# hdfs dfs -text /sparktest.squence
11 aa
22 bb
11 cc

3、使用df命令查看可用空间

[root@master ~]# hdfs dfs -df -h
Filesystem Size Used Available Use%
hdfs://master:9000 64.5 G 812.7 M 49.4 G 1%

4、降低复制因子（默认3副本）

[root@master ~]# hdfs dfs -setrep -w 2 /sparktest.txt

5、使用du命令查看已用空间

[root@master ~]# hdfs dfs -du -s -h /hbase
240.3 K  /hbase

三、hdfs与getconf结合使用

[root@master ~]# hdfs getconf
hdfs getconf is utility for getting configuration information from the config file.

hadoop getconf
        [-namenodes]                    gets list of namenodes in the cluster.
        [-secondaryNameNodes]                   gets list of secondary namenodes in the cluster.
        [-backupNodes]                  gets list of backup nodes in the cluster.
        [-includeFile]                  gets the include file path that defines the datanodes that can join the cluster.
        [-excludeFile]                  gets the exclude file path that defines the datanodes that need to decommissioned.
        [-nnRpcAddresses]                       gets the namenode rpc addresses
        [-confKey [key]]                        gets a specific key from the configuration

1、获取NameNode的节点名称

[root@master ~]# hdfs getconf -namenodes
master

2、获取hdfs最小块信息（默认大小为1M,即1048576字节，如果想要修改的话必须为512的倍数，因为HDFS底层传输数据是每512字节进行校验）

[root@master ~]# hdfs getconf -confKey dfs.namenode.fs-limits.min-block-size
1048576

3、查找hdfs的NameNode的RPC地址

[root@master ~]# hdfs getconf -nnRpcAddresses
master:9000

四、hdfs与dfsadmin结合使用

[root@master ~]# hdfs dfsadmin
Usage: hdfs dfsadmin
Note: Administrative commands can only be run as the HDFS superuser.
        [-report [-live] [-dead] [-decommissioning]]
        [-safemode <enter | leave | get | wait>]
        [-saveNamespace]
        [-rollEdits]
        [-restoreFailedStorage true|false|check]
        [-refreshNodes]
        [-setQuota <quota> <dirname>...<dirname>]
        [-clrQuota <dirname>...<dirname>]
        [-setSpaceQuota <quota> <dirname>...<dirname>]
        [-clrSpaceQuota <dirname>...<dirname>]
        [-finalizeUpgrade]
        [-rollingUpgrade [<query|prepare|finalize>]]
        [-refreshServiceAcl]
        [-refreshUserToGroupsMappings]
        [-refreshSuperUserGroupsConfiguration]
        [-refreshCallQueue]
        [-refresh <host:ipc_port> <key> [arg1..argn]
        [-reconfig <datanode|...> <host:ipc_port> <start|status>]
        [-printTopology]
        [-refreshNamenodes datanode_host:ipc_port]
        [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]]
        [-setBalancerBandwidth <bandwidth in bytes per second>]
        [-fetchImage <local directory>]
        [-allowSnapshot <snapshotDir>]
        [-disallowSnapshot <snapshotDir>]
        [-shutdownDatanode <datanode_host:ipc_port> [upgrade]]
        [-getDatanodeInfo <datanode_host:ipc_port>]
        [-metasave filename]
        [-setStoragePolicy path policyName]
        [-getStoragePolicy path]
        [-triggerBlockReport [-incremental] <datanode_host:ipc_port>]
        [-help [cmd]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

1、查看指定命令的帮助信息

[root@master ~]# hdfs dfsadmin -help safemode
-safemode <enter|leave|get|wait>:  Safe mode maintenance command.
                Safe mode is a Namenode state in which it
                        1.  does not accept changes to the name space (read-only)
                        2.  does not replicate or delete blocks.
                Safe mode is entered automatically at Namenode startup, and
                leaves safe mode automatically when the configured minimum
                percentage of blocks satisfies the minimum replication
                condition.  Safe mode can also be entered manually, but then
                it can only be turned off manually as well.

2、查看当前的模式

[root@master ~]# hdfs dfsadmin -safemode get
Safe mode is OFF

3、进入安全模式

[root@master ~]# hdfs dfsadmin -safemode enter

4、离开安全模式

[root@master ~]# hdfs dfsadmin -safemode leave

5、安全模式的wait状态

[root@master ~]# hdfs dfsadmin -safemode wait

6、检查HDFS集群的状态

[root@master ~]# hdfs dfsadmin -report
Configured Capacity: 69209960448 (64.46 GB)     #此集群中HDFS已配置的容量
Present Capacity: 53855645696 (50.16 GB)　　　　　#现有的HFDS容量
DFS Remaining: 53003517952 (49.36 GB)　　　　　　　#剩余的HDFS容量
DFS Used: 852127744 (812.65 MB)　　　　　　　　　　#HDFS使用存储的统计信息，按照文件大小统计
DFS Used%: 1.58%　　　　　　　　　　　　　　　　　　　　#同上，这里按照的是百分比统计
Under replicated blocks: 156　　　　　　　　　　　　##显示是否有任何未充分复制的块
Blocks with corrupt replicas: 0　　　　　　　　　　　#显示是否有损坏的块　
Missing blocks: 0　　　　　　　　　　　　　　　　　　　　#显示是否有丢失的块

-------------------------------------------------
Live datanodes (3):　　　　　　　　　　　　　　　　　　#显示集群中有多少个DataNode是活动的并可用
Name: 192.168.200.102:50010 (slave02)
Hostname: slave02
Decommission Status : Normal　　　　　　　　　　　　　#当前节点的DataNode的状态（Normal表示正常）
Configured Capacity: 23069986816 (21.49 GB)　　　　#DataNOde的配置和使用的容量
DFS Used: 284041216 (270.88 MB)
Non DFS Used: 3754188800 (3.50 GB)
DFS Remaining: 19031756800 (17.72 GB)
DFS Used%: 1.23%
DFS Remaining%: 82.50%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)　　　　　　　　　　　　　　　　　　　　#缓存使用情况统计信息
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Aug 12 10:30:19 CST 2019

Name: 192.168.200.100:50010 (master)
Hostname: master
Decommission Status : Normal
Configured Capacity: 23069986816 (21.49 GB)
DFS Used: 284045312 (270.89 MB)
Non DFS Used: 7988813824 (7.44 GB)
DFS Remaining: 14797127680 (13.78 GB)
DFS Used%: 1.23%
DFS Remaining%: 64.14%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Aug 12 10:30:18 CST 2019

Name: 192.168.200.101:50010 (slave01)
Hostname: slave01
Decommission Status : Normal
Configured Capacity: 23069986816 (21.49 GB)
DFS Used: 284041216 (270.88 MB)
Non DFS Used: 3611312128 (3.36 GB)
DFS Remaining: 19174633472 (17.86 GB)
DFS Used%: 1.23%
DFS Remaining%: 83.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Aug 12 10:30:19 CST 2019

7、获取某个namenode的节点状态

hdfs haadmin -getServiceState master

五、hdfs与fsck结合使用

1、查看hdfs文件系统信息

[root@master hadoop]# hdfs fsck /
 .........................................
 ........................................

 Total size:    279242984 B　　　　　　　　　　　　　　　　　　　　　　　　#代表根目录下文件总大小
 Total dirs:    342　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#根目录下总共有多少目录
 Total files:   460　　　　　　　　　　　　　　　　　　　　　　　　　　　　　#代表检测的目录下总共有多少文件
 Total symlinks:                0　　　　　　　　　　　　　　　　　　　　　#代表检测下目录下有多少个符号链接
 Total blocks (validated):      434 (avg. block size 643417 B)      #代表检测的目录下有多少的block是有效的
 Minimally replicated blocks:   434 (100.0 %)                      #代表拷贝的最小block块数
 Over-replicated blocks:        0 (0.0 %)　　　　　　　　　　　　　　　　#代表当前副本数大于指定副本数的block数量
 Under-replicated blocks:       156 (35.944702 %)　　　　　　　　　　　　#代表当前副本数小于指定副本数的block数量
 Mis-replicated blocks:         0 (0.0 %)　　　　　　　　　　　　　　　　#代表丢失的block数量
 Default replication factor:    3　　　　　　　　　　　　　　　　　　　　　#代表默认的副本数（自身一份，默认拷贝两份）
 Average block replication:     3.0　　　　　　　　　　　　　　　　　　　　#代表块平均的副本数
 Corrupt blocks:                0　　　　　　　　　　　　　　　　　　　　　　#代表坏的block数，这个值不为0，说明当前集群有不可恢复的块，即数据丢失
 Missing replicas:              1092 (45.614037 %)　　　　　　　　　　　　#代表丢失的副本数　　

 Number of data-nodes:          3　　　　　　　　　　　　　　　　　　　　　　代表有多少个Datanode节点
 Number of racks:               1　　　　　　　　　　　　　　　　　　　　　　#代表有多少个机架
FSCK ended at Mon Aug 12 10:44:46 CST 2019 in 217 milliseconds

The filesystem under path '/' is HEALTHY　　　　　　　　　　　　　　#检测状态

2、fsck指令显示HDFS块信息

Status: HEALTHY
 Total size:    279242984 B
 Total dirs:    342
 Total files:   460
 Total symlinks:                0

[root@master hadoop]# hdfs fsck / -files -blocks
.............................................................
..................................................
 Total blocks (validated):      434 (avg. block size 643417 B)
 Minimally replicated blocks:   434 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       156 (35.944702 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              1092 (45.614037 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Mon Aug 12 10:54:27 CST 2019 in 415 milliseconds

The filesystem under path '/' is HEALTHY

六、快照

快照可以迅速对文件(夹)进行备份，不产生新文件，使用差值存储，默认是禁用状态。因此，想要使用快照功能的话得先启用该功能！我们可以通过“hdfs dfsadmin” 命令来启动或者禁止快照管理。

1、启用快照

[root@master hadoop]# hdfs dfsadmin -allowSnapshot /sparkTest2
Allowing snaphot on /sparkTest2 succeeded

2、禁用快照

[root@master hadoop]# hdfs dfsadmin -disallowSnapShot /sparkTest2
Disallowing snaphot on /sparkTest2 succeeded

3、创建快照

[root@master hadoop]# hdfs dfs -createSnapshot  /sparkTest2 sparkTest2Snapshot
Created snapshot /sparkTest2/.snapshot/sparkTest2Snapshot

4、快照的重命名操作

[root@master hadoop]# hdfs dfs -renameSnapshot /sparkTest2 sparkTest2Snapshot  newSnapshot

5、快照的删除操作

[root@master hadoop]# hdfs dfs -deleteSnapshot /sparkTest2 newSnapshot

6、快照模块的注意点

1、创建快照时，会在对应目录下生成一个.snapshot的隐藏目录，该目录下生成了一个子目录，这个目录就是 快照的名称，该目录下存放的都是创建快照时间节点的数据
2、快照并不产生新的文件
    这个不产生新的文件指的是不完全克隆一份数据出来，而是将数据都指向了同一个存储的ID
3、修改数据源文件与快照无
    当我们修改源数据文件时，快照中保存的数据并不会受到影响，快照中保存的数据还是当时创建快照时的数据