Hadoop学习笔记（两）设置单节点集群
本文描写叙述怎样设置一个单一节点的 Hadoop 安装。以便您能够高速运行简单的操作，使用 Hadoop MapReduce 和 Hadoop 分布式文件系统 (HDFS)。
參考官方文档：Hadoop MapReduce Next Generation - Setting up a Single Node Cluster.
Hadoop版本号：Apache Hadoop 2.5.1
系统版本号：CentOS 6.5。内核（uname -r）：2.6.32-431.el6.x86_64
系统必备组件
支持的系统平台
GNU/Linux 作为开发和生产的平台。毫无疑问。Windows 也是受支持的平台，可是下面步骤仅用于 Linux。
依赖的软件
在Linux系统上安装所须要的软件包
1、JAVA（JDK）必须安装，推荐的版本号请參考Hadoop JAVA Version，我这里安装的是1.7。
2、ssh 必须安装，必须运行 sshd 才干使用管理远程 Hadoop 守护程序的 Hadoop 脚本。
安装依赖的软件
假设您的系统没有所需的软件，您将须要安装它。
比如在Ubuntu Linux上使用下面命令:
$ sudo apt-get install ssh
$ sudo apt-get install rsync
CentOS应该是即使是最小安装也带了ssh（Secure Shell），刚開始我给弄混了，以为是JAVA的SSH（Spring + Struts +Hibernate），汗！
安装JDK，參考：CentOS下安装JDK7
下载
就不多说了，上一篇下过了。
准备启动 Hadoop 集群
解压文件hadoop-2.5.1.tar.gz，运行：tar xvf hadoop-2.5.1.tar.gz，会将文件解压到hadoop-2.5.1文件夹下；
切换文件夹:cd hadoop-2.5.1/etc/hadoop/
编辑“hadoop-env.sh”文件，加入參考定义；
vi hadoop-env.sh
个人认为比較好的习惯是编辑文件之前先做个备份（cp hadoop-env.sh hadoop-env.sh.bak）。
找到下面位置：
# The java implementation to use.
export JAVA_HOME={JAVA_HOME}
将其改为：
# The java implementation to use.
export JAVA_HOME=/usr/java/latest
在下面再加入一句：
# Assuming your installation directory is /usr/local/hadoop
export HADOOP_PREFIX=/usr/local/hadoop
保存并退出。ESC,:wq
切换文件夹（cd ../..），返回“/opt/hadoop-2.5.1”。
尝试运行下面命令：
./bin/hadoop
这将显示 hadoop 脚本的使用文档。输出例如以下：
Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
fs run a generic filesystem user client
version print the version
jar <jar> run a jar file
checknative [-a|-h] check native hadoop and compression libraries availability
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath prints the class path needed to get the
Hadoop jar and the required libraries
daemonlog get/set the log level for each daemon
or
CLASSNAME run the class named CLASSNAME Most commands print help when invoked w/o parameters.
你如今准备好開始您的 Hadoop 集群三个受支持的模式之中的一个：
- 本地 （独立） 模式
- 伪分布的模式
- 全然分布式模式
本地模式操作方法
默认情况下。Hadoop 被配置为运行在非分布式模式下，作为一个单一的 Java 进程。这比較适合用于调试。
下面的演示样例复制要使用作为输入的解压缩的 conf 文件夹。然后查找并显示给定正則表達式的每一场比赛。输出被写入给定的输出文件夹。
$ mkdir input
$ cp etc/hadoop/*.xml input
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'
$ cat output/*
运行“bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'”时
却出现错误：Error: Could not find or load main class org.apache.hadoop.util.RunJar
此问题仅仅在Stack Overflow上见到
What does “Error: Could not find or load main class org.apache.hadoop.util.RunJar”?
可是也没能找到解决的办法；还是自己摸索吧！
解决步骤：
刚刚备份的“hadoop-env.sh”文件如今用上了。还原它。
再运行“bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'”。
提示：
./bin/hadoop: line 133: /usr/java/jdk1.7.0/bin/java: No such file or directory
./bin/hadoop: line 133: exec: /usr/java/jdk1.7.0/bin/java: cannot execute: No such file or directory
按提示应该还是JAVA（JDK）的安装的问题，我安装JDK的时候仅仅运行到
rpm -ivh /文件夹/jdk-7-linux-x64.rpm
再没运行其他操作。将兴许的步骤运行完毕后，再运行“bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'”，
输出：
14/10/07 03:35:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/07 03:35:58 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/10/07 03:35:58 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/10/07 03:35:59 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
14/10/07 03:35:59 INFO input.FileInputFormat: Total input paths to process : 6
14/10/07 03:35:59 INFO mapreduce.JobSubmitter: number of splits:6
14/10/07 03:36:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1185570365_0001
14/10/07 03:36:00 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root1185570365/.staging/job_local1185570365_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/07 03:36:01 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root1185570365/.staging/job_local1185570365_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/07 03:36:01 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1185570365_0001/job_local1185570365_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/07 03:36:01 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1185570365_0001/job_local1185570365_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/07 03:36:01 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/10/07 03:36:01 INFO mapreduce.Job: Running job: job_local1185570365_0001
14/10/07 03:36:01 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/10/07 03:36:01 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Waiting for map tasks
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000000_0
14/10/07 03:36:02 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:02 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/hadoop-policy.xml:0+9201
14/10/07 03:36:02 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:02 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:02 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:02 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:02 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:02 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:02 INFO mapred.LocalJobRunner:
14/10/07 03:36:02 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:02 INFO mapred.MapTask: Spilling map output
14/10/07 03:36:02 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
14/10/07 03:36:02 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
14/10/07 03:36:02 INFO mapreduce.Job: Job job_local1185570365_0001 running in uber mode : false
14/10/07 03:36:02 INFO mapred.MapTask: Finished spill 0
14/10/07 03:36:02 INFO mapreduce.Job: map 0% reduce 0%
14/10/07 03:36:02 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000000_0 is done. And is in the process of committing
14/10/07 03:36:02 INFO mapred.LocalJobRunner: map
14/10/07 03:36:02 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000000_0' done.
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000000_0
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000001_0
14/10/07 03:36:02 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:02 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/capacity-scheduler.xml:0+3589
14/10/07 03:36:02 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:02 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:02 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:02 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:02 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:02 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:02 INFO mapred.LocalJobRunner:
14/10/07 03:36:02 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:02 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000001_0 is done. And is in the process of committing
14/10/07 03:36:02 INFO mapred.LocalJobRunner: map
14/10/07 03:36:02 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000001_0' done.
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000001_0
14/10/07 03:36:02 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000002_0
14/10/07 03:36:02 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:02 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/hdfs-site.xml:0+775
14/10/07 03:36:02 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:03 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:03 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:03 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:03 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:03 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:03 INFO mapred.LocalJobRunner:
14/10/07 03:36:03 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:03 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000002_0 is done. And is in the process of committing
14/10/07 03:36:03 INFO mapred.LocalJobRunner: map
14/10/07 03:36:03 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000002_0' done.
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000002_0
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000003_0
14/10/07 03:36:03 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:03 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/core-site.xml:0+774
14/10/07 03:36:03 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:03 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:03 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:03 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:03 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:03 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:03 INFO mapred.LocalJobRunner:
14/10/07 03:36:03 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:03 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000003_0 is done. And is in the process of committing
14/10/07 03:36:03 INFO mapred.LocalJobRunner: map
14/10/07 03:36:03 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000003_0' done.
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000003_0
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000004_0
14/10/07 03:36:03 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:03 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/yarn-site.xml:0+690
14/10/07 03:36:03 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:03 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:03 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:03 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:03 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:03 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:03 INFO mapred.LocalJobRunner:
14/10/07 03:36:03 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:03 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000004_0 is done. And is in the process of committing
14/10/07 03:36:03 INFO mapred.LocalJobRunner: map
14/10/07 03:36:03 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000004_0' done.
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000004_0
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_m_000005_0
14/10/07 03:36:03 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:03 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/httpfs-site.xml:0+620
14/10/07 03:36:03 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/07 03:36:03 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/07 03:36:03 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/07 03:36:03 INFO mapred.MapTask: soft limit at 83886080
14/10/07 03:36:03 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/07 03:36:03 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/07 03:36:03 INFO mapred.LocalJobRunner:
14/10/07 03:36:03 INFO mapred.MapTask: Starting flush of map output
14/10/07 03:36:03 INFO mapred.Task: Task:attempt_local1185570365_0001_m_000005_0 is done. And is in the process of committing
14/10/07 03:36:03 INFO mapred.LocalJobRunner: map
14/10/07 03:36:03 INFO mapred.Task: Task 'attempt_local1185570365_0001_m_000005_0' done.
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_m_000005_0
14/10/07 03:36:03 INFO mapred.LocalJobRunner: map task executor complete.
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Waiting for reduce tasks
14/10/07 03:36:03 INFO mapred.LocalJobRunner: Starting task: attempt_local1185570365_0001_r_000000_0
14/10/07 03:36:03 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/07 03:36:03 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@57931be2
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
14/10/07 03:36:03 INFO reduce.EventFetcher: attempt_local1185570365_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1185570365_0001_m_000001_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
14/10/07 03:36:03 INFO mapreduce.Job: map 100% reduce 0%
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1185570365_0001_m_000004_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1185570365_0001_m_000005_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1185570365_0001_m_000002_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1185570365_0001_m_000003_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
14/10/07 03:36:03 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1185570365_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
14/10/07 03:36:03 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1185570365_0001_m_000000_0
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->31
14/10/07 03:36:03 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
14/10/07 03:36:03 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: finalMerge called with 6 in-memory map-outputs and 0 on-disk map-outputs
14/10/07 03:36:03 INFO mapred.Merger: Merging 6 sorted segments
14/10/07 03:36:03 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: Merged 6 segments, 31 bytes to disk to satisfy reduce memory limit
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
14/10/07 03:36:03 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
14/10/07 03:36:03 INFO mapred.Merger: Merging 1 sorted segments
14/10/07 03:36:03 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
14/10/07 03:36:03 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/07 03:36:04 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
14/10/07 03:36:04 INFO mapred.Task: Task:attempt_local1185570365_0001_r_000000_0 is done. And is in the process of committing
14/10/07 03:36:04 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/07 03:36:04 INFO mapred.Task: Task attempt_local1185570365_0001_r_000000_0 is allowed to commit now
14/10/07 03:36:04 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1185570365_0001_r_000000_0' to file:/opt/hadoop-2.5.1/grep-temp-767563685/_temporary/0/task_local1185570365_0001_r_000000
14/10/07 03:36:04 INFO mapred.LocalJobRunner: reduce > reduce
14/10/07 03:36:04 INFO mapred.Task: Task 'attempt_local1185570365_0001_r_000000_0' done.
14/10/07 03:36:04 INFO mapred.LocalJobRunner: Finishing task: attempt_local1185570365_0001_r_000000_0
14/10/07 03:36:04 INFO mapred.LocalJobRunner: reduce task executor complete.
14/10/07 03:36:04 INFO mapreduce.Job: map 100% reduce 100%
14/10/07 03:36:04 INFO mapreduce.Job: Job job_local1185570365_0001 completed successfully
14/10/07 03:36:04 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=114663
FILE: Number of bytes written=1613316
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=405
Map output records=1
Map output bytes=17
Map output materialized bytes=55
Input split bytes=657
Combine input records=1
Combine output records=1
Reduce input groups=1
Reduce shuffle bytes=55
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =6
Failed Shuffles=0
Merged Map outputs=6
GC time elapsed (ms)=225
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1106100224
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=15649
File Output Format Counters
Bytes Written=123
14/10/07 03:36:04 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/opt/hadoop-2.5.1/output already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.examples.Grep.run(Grep.java:92)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Output directory file:/opt/hadoop-2.5.1/output already exists。噢，原因是output文件夹已经存在了（之前我排查问题的时候创建的）。
删除output文件夹（rm -rf output）；
再运行“bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar grep input output 'dfs[a-z.]+'”命令，输出例如以下：
14/10/08 05:57:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/08 05:57:35 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/10/08 05:57:35 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/10/08 05:57:36 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
14/10/08 05:57:36 INFO input.FileInputFormat: Total input paths to process : 6
14/10/08 05:57:36 INFO mapreduce.JobSubmitter: number of splits:6
14/10/08 05:57:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local380762736_0001
14/10/08 05:57:37 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root380762736/.staging/job_local380762736_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/08 05:57:37 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root380762736/.staging/job_local380762736_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/08 05:57:38 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local380762736_0001/job_local380762736_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/08 05:57:38 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local380762736_0001/job_local380762736_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/08 05:57:38 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/10/08 05:57:38 INFO mapreduce.Job: Running job: job_local380762736_0001
14/10/08 05:57:38 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/10/08 05:57:38 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/10/08 05:57:38 INFO mapred.LocalJobRunner: Waiting for map tasks
14/10/08 05:57:38 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000000_0
14/10/08 05:57:39 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:39 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/hadoop-policy.xml:0+9201
14/10/08 05:57:39 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:39 INFO mapreduce.Job: Job job_local380762736_0001 running in uber mode : false
14/10/08 05:57:39 INFO mapreduce.Job: map 0% reduce 0%
14/10/08 05:57:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:43 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:44 INFO mapred.LocalJobRunner:
14/10/08 05:57:44 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:44 INFO mapred.MapTask: Spilling map output
14/10/08 05:57:44 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
14/10/08 05:57:44 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
14/10/08 05:57:44 INFO mapred.MapTask: Finished spill 0
14/10/08 05:57:44 INFO mapred.Task: Task:attempt_local380762736_0001_m_000000_0 is done. And is in the process of committing
14/10/08 05:57:45 INFO mapred.LocalJobRunner: map
14/10/08 05:57:45 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000000_0' done.
14/10/08 05:57:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000000_0
14/10/08 05:57:45 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000001_0
14/10/08 05:57:45 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:45 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/capacity-scheduler.xml:0+3589
14/10/08 05:57:45 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:45 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:45 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:45 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:45 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:45 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:45 INFO mapred.LocalJobRunner:
14/10/08 05:57:45 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:45 INFO mapred.Task: Task:attempt_local380762736_0001_m_000001_0 is done. And is in the process of committing
14/10/08 05:57:45 INFO mapred.LocalJobRunner: map
14/10/08 05:57:45 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000001_0' done.
14/10/08 05:57:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000001_0
14/10/08 05:57:45 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000002_0
14/10/08 05:57:45 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:45 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/hdfs-site.xml:0+775
14/10/08 05:57:45 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:46 INFO mapreduce.Job: map 100% reduce 0%
14/10/08 05:57:46 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:46 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:46 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:46 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:46 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:46 INFO mapred.LocalJobRunner:
14/10/08 05:57:46 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:46 INFO mapred.Task: Task:attempt_local380762736_0001_m_000002_0 is done. And is in the process of committing
14/10/08 05:57:46 INFO mapred.LocalJobRunner: map
14/10/08 05:57:46 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000002_0' done.
14/10/08 05:57:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000002_0
14/10/08 05:57:46 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000003_0
14/10/08 05:57:46 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:46 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/core-site.xml:0+774
14/10/08 05:57:46 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:47 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:47 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:47 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:47 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:47 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:47 INFO mapred.LocalJobRunner:
14/10/08 05:57:47 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:47 INFO mapred.Task: Task:attempt_local380762736_0001_m_000003_0 is done. And is in the process of committing
14/10/08 05:57:47 INFO mapred.LocalJobRunner: map
14/10/08 05:57:47 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000003_0' done.
14/10/08 05:57:47 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000003_0
14/10/08 05:57:47 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000004_0
14/10/08 05:57:47 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:47 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/yarn-site.xml:0+690
14/10/08 05:57:47 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:49 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:49 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:49 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:49 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:49 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:49 INFO mapred.LocalJobRunner:
14/10/08 05:57:49 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:49 INFO mapred.Task: Task:attempt_local380762736_0001_m_000004_0 is done. And is in the process of committing
14/10/08 05:57:49 INFO mapred.LocalJobRunner: map
14/10/08 05:57:49 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000004_0' done.
14/10/08 05:57:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000004_0
14/10/08 05:57:49 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_m_000005_0
14/10/08 05:57:49 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:49 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/input/httpfs-site.xml:0+620
14/10/08 05:57:49 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:49 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:49 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:49 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:49 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:49 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:49 INFO mapred.LocalJobRunner:
14/10/08 05:57:49 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:49 INFO mapred.Task: Task:attempt_local380762736_0001_m_000005_0 is done. And is in the process of committing
14/10/08 05:57:49 INFO mapred.LocalJobRunner: map
14/10/08 05:57:49 INFO mapred.Task: Task 'attempt_local380762736_0001_m_000005_0' done.
14/10/08 05:57:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_m_000005_0
14/10/08 05:57:49 INFO mapred.LocalJobRunner: map task executor complete.
14/10/08 05:57:49 INFO mapred.LocalJobRunner: Waiting for reduce tasks
14/10/08 05:57:49 INFO mapred.LocalJobRunner: Starting task: attempt_local380762736_0001_r_000000_0
14/10/08 05:57:49 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:49 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6d36df08
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
14/10/08 05:57:50 INFO reduce.EventFetcher: attempt_local380762736_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local380762736_0001_m_000000_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local380762736_0001_m_000004_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 21, usedMemory ->23
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local380762736_0001_m_000003_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local380762736_0001_m_000005_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local380762736_0001_m_000001_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29
14/10/08 05:57:50 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local380762736_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
14/10/08 05:57:50 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local380762736_0001_m_000002_0
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31
14/10/08 05:57:50 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
14/10/08 05:57:50 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: finalMerge called with 6 in-memory map-outputs and 0 on-disk map-outputs
14/10/08 05:57:50 INFO mapred.Merger: Merging 6 sorted segments
14/10/08 05:57:50 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: Merged 6 segments, 31 bytes to disk to satisfy reduce memory limit
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
14/10/08 05:57:50 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
14/10/08 05:57:50 INFO mapred.Merger: Merging 1 sorted segments
14/10/08 05:57:50 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
14/10/08 05:57:50 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/08 05:57:50 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
14/10/08 05:57:50 INFO mapred.Task: Task:attempt_local380762736_0001_r_000000_0 is done. And is in the process of committing
14/10/08 05:57:50 INFO mapred.LocalJobRunner: 6 / 6 copied.
14/10/08 05:57:50 INFO mapred.Task: Task attempt_local380762736_0001_r_000000_0 is allowed to commit now
14/10/08 05:57:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local380762736_0001_r_000000_0' to file:/opt/hadoop-2.5.1/grep-temp-913340630/_temporary/0/task_local380762736_0001_r_000000
14/10/08 05:57:50 INFO mapred.LocalJobRunner: reduce > reduce
14/10/08 05:57:50 INFO mapred.Task: Task 'attempt_local380762736_0001_r_000000_0' done.
14/10/08 05:57:50 INFO mapred.LocalJobRunner: Finishing task: attempt_local380762736_0001_r_000000_0
14/10/08 05:57:50 INFO mapred.LocalJobRunner: reduce task executor complete.
14/10/08 05:57:51 INFO mapreduce.Job: map 100% reduce 100%
14/10/08 05:57:51 INFO mapreduce.Job: Job job_local380762736_0001 completed successfully
14/10/08 05:57:51 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=114663
FILE: Number of bytes written=1604636
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=405
Map output records=1
Map output bytes=17
Map output materialized bytes=55
Input split bytes=657
Combine input records=1
Combine output records=1
Reduce input groups=1
Reduce shuffle bytes=55
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =6
Failed Shuffles=0
Merged Map outputs=6
GC time elapsed (ms)=2359
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1106096128
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=15649
File Output Format Counters
Bytes Written=123
14/10/08 05:57:51 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
14/10/08 05:57:51 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
14/10/08 05:57:51 INFO input.FileInputFormat: Total input paths to process : 1
14/10/08 05:57:51 INFO mapreduce.JobSubmitter: number of splits:1
14/10/08 05:57:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local571678604_0002
14/10/08 05:57:51 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root571678604/.staging/job_local571678604_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/08 05:57:51 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/staging/root571678604/.staging/job_local571678604_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/08 05:57:52 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local571678604_0002/job_local571678604_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/10/08 05:57:52 WARN conf.Configuration: file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local571678604_0002/job_local571678604_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
14/10/08 05:57:52 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/10/08 05:57:52 INFO mapreduce.Job: Running job: job_local571678604_0002
14/10/08 05:57:52 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/10/08 05:57:52 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Waiting for map tasks
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Starting task: attempt_local571678604_0002_m_000000_0
14/10/08 05:57:52 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:52 INFO mapred.MapTask: Processing split: file:/opt/hadoop-2.5.1/grep-temp-913340630/part-r-00000:0+111
14/10/08 05:57:52 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/10/08 05:57:52 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/10/08 05:57:52 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/10/08 05:57:52 INFO mapred.MapTask: soft limit at 83886080
14/10/08 05:57:52 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/10/08 05:57:52 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/10/08 05:57:52 INFO mapred.LocalJobRunner:
14/10/08 05:57:52 INFO mapred.MapTask: Starting flush of map output
14/10/08 05:57:52 INFO mapred.MapTask: Spilling map output
14/10/08 05:57:52 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
14/10/08 05:57:52 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
14/10/08 05:57:52 INFO mapred.MapTask: Finished spill 0
14/10/08 05:57:52 INFO mapred.Task: Task:attempt_local571678604_0002_m_000000_0 is done. And is in the process of committing
14/10/08 05:57:52 INFO mapred.LocalJobRunner: map
14/10/08 05:57:52 INFO mapred.Task: Task 'attempt_local571678604_0002_m_000000_0' done.
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local571678604_0002_m_000000_0
14/10/08 05:57:52 INFO mapred.LocalJobRunner: map task executor complete.
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Waiting for reduce tasks
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Starting task: attempt_local571678604_0002_r_000000_0
14/10/08 05:57:52 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
14/10/08 05:57:52 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@63ae8b5c
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
14/10/08 05:57:52 INFO reduce.EventFetcher: attempt_local571678604_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
14/10/08 05:57:52 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local571678604_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
14/10/08 05:57:52 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local571678604_0002_m_000000_0
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
14/10/08 05:57:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
14/10/08 05:57:52 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
14/10/08 05:57:52 INFO mapred.Merger: Merging 1 sorted segments
14/10/08 05:57:52 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
14/10/08 05:57:52 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
14/10/08 05:57:52 INFO mapred.Merger: Merging 1 sorted segments
14/10/08 05:57:52 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
14/10/08 05:57:52 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/10/08 05:57:52 INFO mapred.Task: Task:attempt_local571678604_0002_r_000000_0 is done. And is in the process of committing
14/10/08 05:57:52 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/10/08 05:57:52 INFO mapred.Task: Task attempt_local571678604_0002_r_000000_0 is allowed to commit now
14/10/08 05:57:52 INFO output.FileOutputCommitter: Saved output of task 'attempt_local571678604_0002_r_000000_0' to file:/opt/hadoop-2.5.1/output/_temporary/0/task_local571678604_0002_r_000000
14/10/08 05:57:52 INFO mapred.LocalJobRunner: reduce > reduce
14/10/08 05:57:52 INFO mapred.Task: Task 'attempt_local571678604_0002_r_000000_0' done.
14/10/08 05:57:52 INFO mapred.LocalJobRunner: Finishing task: attempt_local571678604_0002_r_000000_0
14/10/08 05:57:52 INFO mapred.LocalJobRunner: reduce task executor complete.
14/10/08 05:57:53 INFO mapreduce.Job: Job job_local571678604_0002 running in uber mode : false
14/10/08 05:57:53 INFO mapreduce.Job: map 100% reduce 100%
14/10/08 05:57:53 INFO mapreduce.Job: Job job_local571678604_0002 completed successfully
14/10/08 05:57:53 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=39892
FILE: Number of bytes written=913502
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=120
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=25
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=37
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=250560512
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=123
File Output Format Counters
Bytes Written=23
OK，终于上了。
版权声明：本文博客原创文章。博客，未经同意，不得转载。
Hadoop学习笔记（两）设置单节点集群的更多相关文章
- Ambari安装之部署单节点集群
前期博客 大数据领域两大最主流集群管理工具Ambari和Cloudera Manger Ambari架构原理 Ambari安装之Ambari安装前准备(CentOS6.5)(一) Ambari安装之部 ...
- redis单节点集群
一.概念 redis是一种支持Key-Value等多种数据结构的存储系统.可用于缓存.事件发布或订阅.高速队列等场景.该数据库使用ANSI C语言编写,支持网络,提供字符串.哈希.列表.队列.集合结构 ...
- kubernetes系列：（一）、kubeadm搭建kubernetes（v1.13.1）单节点集群
kubeadm是Kubernetes官方提供的用于快速部署Kubernetes集群的工具,本篇文章使用kubeadm搭建一个单master节点的k8s集群. 节点部署信息 节点主机名 节点IP 节点角 ...
- Hadoop学习笔记(3)——分布式环境搭建
Hadoop学习笔记(3) ——分布式环境搭建 前面,我们已经在单机上把Hadoop运行起来了,但我们知道Hadoop支持分布式的,而它的优点就是在分布上突出的,所以我们得搭个环境模拟一下. 在这里, ...
- Hadoop学习笔记(8) ——实战 做个倒排索引
Hadoop学习笔记(8) ——实战 做个倒排索引 倒排索引是文档检索系统中最常用数据结构.根据单词反过来查在文档中出现的频率,而不是根据文档来,所以称倒排索引(Inverted Index).结构如 ...
- Hadoop学习笔记—22.Hadoop2.x环境搭建与配置
自从2015年花了2个多月时间把Hadoop1.x的学习教程学习了一遍,对Hadoop这个神奇的小象有了一个初步的了解,还对每次学习的内容进行了总结,也形成了我的一个博文系列<Hadoop学习笔 ...
- Hadoop学习笔记(7) ——高级编程
Hadoop学习笔记(7) ——高级编程 从前面的学习中,我们了解到了MapReduce整个过程需要经过以下几个步骤: 1.输入(input):将输入数据分成一个个split,并将split进一步拆成 ...
- Hadoop学习笔记(6) ——重新认识Hadoop
Hadoop学习笔记(6) ——重新认识Hadoop 之前,我们把hadoop从下载包部署到编写了helloworld,看到了结果.现是得开始稍微更深入地了解hadoop了. Hadoop包含了两大功 ...
- Hadoop学习笔记(2)
Hadoop学习笔记(2) ——解读Hello World 上一章中,我们把hadoop下载.安装.运行起来,最后还执行了一个Hello world程序,看到了结果.现在我们就来解读一下这个Hello ...
随机推荐
- MyBATIS使用CRUD
MyEclipse不提供自己主动生成,这里提供mybatis文件包和开发文档 http://download.csdn.net/detail/u010026901/7489319 自己建立配置文件, ...
- html怎么引用css
<head> <title>统一站内风格</title> <link rel="stylesheet" type="text/c ...
- NetAnalyzer2016使用方法
NetAnalyzer笔记 之 八 NetAnalyzer2016使用方法(2) [创建时间:2016-05-06 22:07:00] NetAnalyzer下载地址 在写本篇的时候,NetAna ...
- 旧发票要保留SIRET等信息，或者整个PDF
查看旧发票时,每次都实时生成发票是不行的,因为公司的SIRET居然会是变的!!
- IM应用中的优化
以前做过IM应用中考虑过的一些关于应用优化的问题. 优化重心主要放在了电量优化和流量优化上: 电量优化 如何測试耗电量(前台耗电.后台耗电) 1.专用设备 2.性能相关的app 3.自己动手去写电量消 ...
- 欧舒丹 L'Occitane 活力清泉保湿面霜 - 男士护肤 - 香港草莓网StrawberryNET.com
欧舒丹 L'Occitane 活力清泉保湿面霜 - 男士护肤 - 香港草莓网StrawberryNET.com 欧舒丹 活力清泉保湿面霜 50ml/1.7oz
- Codeforces Round #296 (Div. 2) A B C D
A:模拟辗转相除法时记录答案 B:3种情况:能降低2,能降低1.不能降低分别考虑清楚 C:利用一个set和一个multiset,把行列分开考虑.利用set自带的排序和查询.每次把对应的块拿出来分成两块 ...
- Object.Instantiate 实例
static function Instantiate (original : Object, position : Vector3, rotation : Quaternion) : Object ...
- MySQL在一台db服务器上面如何启动多个实例
安装过程省略过,源码安装请参考http://write.blog.csdn.net/postlist/1609043/all 整理自己的文档,发现以前做的例子,share下,欢迎大家提出改进意见. 一 ...
- SimpleDateFormat 的线程安全问题与解决方式
SimpleDateFormat 的线程安全问题 SimpleDateFormat 是一个以国别敏感的方式格式化和分析数据的详细类. 它同意格式化 (date -> text).语法分析 (te ...