Hadoop MapReduce编程 API入门系列之网页排序（二十八）

　　不多说，直接上代码。

Map output bytes=247
Map output materialized bytes=275
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=275
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1439694848
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=119
File Output Format Counters
Bytes Written=113
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=17
success.
17
2016-12-13 19:07:44,783 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:44,796 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:44,799 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:45,231 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:45,245 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:45,266 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local455505407_0021
2016-12-13 19:07:45,483 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:45,484 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local455505407_0021
2016-12-13 19:07:45,484 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:45,485 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:45,495 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:45,495 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local455505407_0021_m_000000_0
2016-12-13 19:07:45,500 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:45,559 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@223b788b
2016-12-13 19:07:45,565 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr20/part-r-00000:0+101
2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:45,597 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:45,598 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:45,600 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:45,608 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:45,609 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 255; bufvoid = 104857600
2016-12-13 19:07:45,610 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:45,625 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:45,631 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local455505407_0021_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:45,638 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr20/part-r-00000:0+101
2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local455505407_0021_m_000000_0' done.
2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local455505407_0021_m_000000_0
2016-12-13 19:07:45,639 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:45,640 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:45,641 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local455505407_0021_r_000000_0
2016-12-13 19:07:45,645 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:45,690 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7d5e1574
2016-12-13 19:07:45,691 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2bd2b2f9
2016-12-13 19:07:45,703 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:45,704 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local455505407_0021_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:45,709 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#21 about to shuffle output of map attempt_local455505407_0021_m_000000_0 decomp: 279 len: 283 to MEMORY
2016-12-13 19:07:45,710 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 279 bytes from map-output for attempt_local455505407_0021_m_000000_0
2016-12-13 19:07:45,711 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 279, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->279
2016-12-13 19:07:45,712 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:45,714 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:45,715 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:45,729 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:45,730 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 275 bytes
2016-12-13 19:07:45,732 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 279 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 283 bytes from disk
2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:45,734 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:45,736 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 275 bytes
2016-12-13 19:07:45,737 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.21104025855884012
3___________
*********** new pageRank value is 0.3575070100293671
5___________
*********** new pageRank value is 0.4016854782039228
6___________
*********** new pageRank value is 0.1286040574733812
1___________
2016-12-13 19:07:45,749 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local455505407_0021_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:45,753 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:45,753 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local455505407_0021_r_000000_0 is allowed to commit now
2016-12-13 19:07:45,762 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local455505407_0021_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/_temporary/0/task_local455505407_0021_r_000000
2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local455505407_0021_r_000000_0' done.
2016-12-13 19:07:45,764 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local455505407_0021_r_000000_0
2016-12-13 19:07:45,765 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:46,485 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local455505407_0021 running in uber mode : false
2016-12-13 19:07:46,486 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:46,487 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local455505407_0021 completed successfully
2016-12-13 19:07:46,498 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=35976
FILE: Number of bytes written=8181702
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=255
Map output materialized bytes=283
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=283
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1588592640
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=121
File Output Format Counters
Bytes Written=111
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=15
success.
15
2016-12-13 19:07:46,508 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:46,516 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:46,519 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:46,868 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:46,879 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:46,896 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1152686492_0022
2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1152686492_0022
2016-12-13 19:07:47,037 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:47,039 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:47,045 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:47,045 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1152686492_0022_m_000000_0
2016-12-13 19:07:47,048 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:47,105 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@35f8fa7b
2016-12-13 19:07:47,109 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/part-r-00000:0+99
2016-12-13 19:07:47,131 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:47,132 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:47,133 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:47,139 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 248; bufvoid = 104857600
2016-12-13 19:07:47,140 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:47,155 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:47,160 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1152686492_0022_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr21/part-r-00000:0+99
2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1152686492_0022_m_000000_0' done.
2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1152686492_0022_m_000000_0
2016-12-13 19:07:47,165 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:47,166 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:47,167 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1152686492_0022_r_000000_0
2016-12-13 19:07:47,169 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:47,215 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@6df4957a
2016-12-13 19:07:47,215 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5481b4fb
2016-12-13 19:07:47,217 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:47,219 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1152686492_0022_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:47,224 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#22 about to shuffle output of map attempt_local1152686492_0022_m_000000_0 decomp: 272 len: 276 to MEMORY
2016-12-13 19:07:47,225 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 272 bytes from map-output for attempt_local1152686492_0022_m_000000_0
2016-12-13 19:07:47,225 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 272, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->272
2016-12-13 19:07:47,227 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:47,228 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:47,228 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:47,237 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:47,238 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 268 bytes
2016-12-13 19:07:47,240 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 272 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:47,241 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 276 bytes from disk
2016-12-13 19:07:47,241 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:47,242 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:47,243 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 268 bytes
2016-12-13 19:07:47,244 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.2082163282366672
2___________
*********** new pageRank value is 0.3525651625503612
4___________
*********** new pageRank value is 0.396037682951149
5___________
*********** new pageRank value is 0.12719210988750704
1___________
2016-12-13 19:07:47,256 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1152686492_0022_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:47,260 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:47,261 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1152686492_0022_r_000000_0 is allowed to commit now
2016-12-13 19:07:47,270 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1152686492_0022_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/_temporary/0/task_local1152686492_0022_r_000000
2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1152686492_0022_r_000000_0' done.
2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1152686492_0022_r_000000_0
2016-12-13 19:07:47,273 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:48,038 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1152686492_0022 running in uber mode : false
2016-12-13 19:07:48,039 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:48,040 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1152686492_0022 completed successfully
2016-12-13 19:07:48,048 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=37782
FILE: Number of bytes written=8572508
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=248
Map output materialized bytes=276
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=276
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=13
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1588592640
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=119
File Output Format Counters
Bytes Written=110
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=12
success.
12
2016-12-13 19:07:48,059 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:48,070 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:48,073 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:48,425 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:48,446 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:48,465 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1854065889_0023
2016-12-13 19:07:48,612 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:48,613 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:48,613 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1854065889_0023
2016-12-13 19:07:48,615 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:48,621 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:48,622 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1854065889_0023_m_000000_0
2016-12-13 19:07:48,624 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:48,668 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@135c5527
2016-12-13 19:07:48,672 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/part-r-00000:0+98
2016-12-13 19:07:48,709 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:48,710 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:48,711 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:48,720 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 247; bufvoid = 104857600
2016-12-13 19:07:48,722 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:48,738 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:48,742 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1854065889_0023_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr22/part-r-00000:0+98
2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1854065889_0023_m_000000_0' done.
2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1854065889_0023_m_000000_0
2016-12-13 19:07:48,746 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:48,747 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:48,748 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1854065889_0023_r_000000_0
2016-12-13 19:07:48,751 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:48,798 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@21287ab4
2016-12-13 19:07:48,798 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@688fd06d
2016-12-13 19:07:48,801 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:48,802 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1854065889_0023_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:48,807 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#23 about to shuffle output of map attempt_local1854065889_0023_m_000000_0 decomp: 271 len: 275 to MEMORY
2016-12-13 19:07:48,809 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 271 bytes from map-output for attempt_local1854065889_0023_m_000000_0
2016-12-13 19:07:48,811 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 271, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->271
2016-12-13 19:07:48,815 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:48,817 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:48,818 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:48,831 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:48,832 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 267 bytes
2016-12-13 19:07:48,834 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 271 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 275 bytes from disk
2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:48,836 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:48,837 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 267 bytes
2016-12-13 19:07:48,839 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.20581601525423832
2___________
*********** new pageRank value is 0.3483646014570123
4___________
*********** new pageRank value is 0.3912370348699975
4___________
*********** new pageRank value is 0.12599193950058354
1___________
2016-12-13 19:07:48,854 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1854065889_0023_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:48,858 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:48,859 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1854065889_0023_r_000000_0 is allowed to commit now
2016-12-13 19:07:48,870 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1854065889_0023_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/_temporary/0/task_local1854065889_0023_r_000000
2016-12-13 19:07:48,879 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:48,880 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1854065889_0023_r_000000_0' done.
2016-12-13 19:07:48,882 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1854065889_0023_r_000000_0
2016-12-13 19:07:48,883 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:49,614 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1854065889_0023 running in uber mode : false
2016-12-13 19:07:49,615 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:49,617 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1854065889_0023 completed successfully
2016-12-13 19:07:49,637 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=39570
FILE: Number of bytes written=8963305
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=247
Map output materialized bytes=275
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=275
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=13
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1755316224
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=118
File Output Format Counters
Bytes Written=112
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=11
success.
11
2016-12-13 19:07:49,648 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:49,656 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:49,659 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:50,057 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:50,073 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:50,093 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local2101720639_0024
2016-12-13 19:07:50,248 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:50,249 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local2101720639_0024
2016-12-13 19:07:50,251 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:50,252 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:50,262 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:50,263 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local2101720639_0024_m_000000_0
2016-12-13 19:07:50,267 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:50,324 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@63a803dc
2016-12-13 19:07:50,329 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/part-r-00000:0+100
2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:50,405 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:50,406 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:50,407 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:50,417 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 253; bufvoid = 104857600
2016-12-13 19:07:50,418 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:50,437 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:50,442 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local2101720639_0024_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:50,447 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr23/part-r-00000:0+100
2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local2101720639_0024_m_000000_0' done.
2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local2101720639_0024_m_000000_0
2016-12-13 19:07:50,448 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:50,450 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:50,450 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local2101720639_0024_r_000000_0
2016-12-13 19:07:50,454 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:50,523 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7e37d95
2016-12-13 19:07:50,524 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6849c53e
2016-12-13 19:07:50,526 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:50,535 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local2101720639_0024_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:50,539 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#24 about to shuffle output of map attempt_local2101720639_0024_m_000000_0 decomp: 277 len: 281 to MEMORY
2016-12-13 19:07:50,541 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 277 bytes from map-output for attempt_local2101720639_0024_m_000000_0
2016-12-13 19:07:50,541 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 277, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->277
2016-12-13 19:07:50,542 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:50,544 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:50,545 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:50,559 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:50,559 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 273 bytes
2016-12-13 19:07:50,562 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 277 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 281 bytes from disk
2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:50,563 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:50,564 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 273 bytes
2016-12-13 19:07:50,566 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.20377573981974895
2___________
*********** new pageRank value is 0.3447941205905482
3___________
*********** new pageRank value is 0.3871564855262084
4___________
*********** new pageRank value is 0.12497180648305128
1___________
2016-12-13 19:07:50,580 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local2101720639_0024_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:50,584 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:50,585 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local2101720639_0024_r_000000_0 is allowed to commit now
2016-12-13 19:07:50,595 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local2101720639_0024_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/_temporary/0/task_local2101720639_0024_r_000000
2016-12-13 19:07:50,597 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:50,597 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local2101720639_0024_r_000000_0' done.
2016-12-13 19:07:50,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local2101720639_0024_r_000000_0
2016-12-13 19:07:50,598 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:51,250 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local2101720639_0024 running in uber mode : false
2016-12-13 19:07:51,252 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:51,253 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local2101720639_0024 completed successfully
2016-12-13 19:07:51,263 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=41372
FILE: Number of bytes written=9354121
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=253
Map output materialized bytes=281
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=281
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1755316224
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=120
File Output Format Counters
Bytes Written=112
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=10
success.
10
2016-12-13 19:07:51,279 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:51,292 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:51,295 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:51,881 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:51,893 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:51,913 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local756232932_0025
2016-12-13 19:07:52,079 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:52,080 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local756232932_0025
2016-12-13 19:07:52,080 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:52,081 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:52,087 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:52,087 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local756232932_0025_m_000000_0
2016-12-13 19:07:52,090 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:52,137 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@2bbddda
2016-12-13 19:07:52,141 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/part-r-00000:0+100
2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:52,165 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:52,166 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:52,167 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:52,175 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 251; bufvoid = 104857600
2016-12-13 19:07:52,176 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:52,193 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:52,198 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local756232932_0025_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:52,203 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr24/part-r-00000:0+100
2016-12-13 19:07:52,203 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local756232932_0025_m_000000_0' done.
2016-12-13 19:07:52,204 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local756232932_0025_m_000000_0
2016-12-13 19:07:52,204 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:52,205 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:52,205 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local756232932_0025_r_000000_0
2016-12-13 19:07:52,209 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:52,264 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@74f508b9
2016-12-13 19:07:52,264 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@30fe5ac
2016-12-13 19:07:52,266 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:52,268 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local756232932_0025_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:52,274 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#25 about to shuffle output of map attempt_local756232932_0025_m_000000_0 decomp: 275 len: 279 to MEMORY
2016-12-13 19:07:52,276 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 275 bytes from map-output for attempt_local756232932_0025_m_000000_0
2016-12-13 19:07:52,277 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 275, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->275
2016-12-13 19:07:52,278 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:52,280 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:52,280 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:52,294 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:52,295 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes
2016-12-13 19:07:52,298 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 275 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:52,299 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 279 bytes from disk
2016-12-13 19:07:52,299 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:52,300 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:52,301 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes
2016-12-13 19:07:52,303 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.20204150634863857
1___________
*********** new pageRank value is 0.34175921352732863
3___________
*********** new pageRank value is 0.38368802025726273
3___________
*********** new pageRank value is 0.12410468942339331
0___________
2016-12-13 19:07:52,321 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local756232932_0025_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:52,326 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:52,327 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local756232932_0025_r_000000_0 is allowed to commit now
2016-12-13 19:07:52,339 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local756232932_0025_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/_temporary/0/task_local756232932_0025_r_000000
2016-12-13 19:07:52,343 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:52,344 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local756232932_0025_r_000000_0' done.
2016-12-13 19:07:52,344 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local756232932_0025_r_000000_0
2016-12-13 19:07:52,346 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:53,080 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local756232932_0025 running in uber mode : false
2016-12-13 19:07:53,081 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:53,082 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local756232932_0025 completed successfully
2016-12-13 19:07:53,092 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=43182
FILE: Number of bytes written=9742963
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=251
Map output materialized bytes=279
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=279
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1774190592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=120
File Output Format Counters
Bytes Written=114
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=7
success.
7
2016-12-13 19:07:53,109 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:53,116 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:53,119 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:53,539 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:53,553 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:53,574 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local525031809_0026
2016-12-13 19:07:53,728 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:53,729 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local525031809_0026
2016-12-13 19:07:53,729 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:53,730 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:53,736 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:53,737 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local525031809_0026_m_000000_0
2016-12-13 19:07:53,739 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:53,787 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4190fa1c
2016-12-13 19:07:53,793 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/part-r-00000:0+102
2016-12-13 19:07:53,825 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:53,825 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:53,826 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:53,827 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:53,835 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:53,836 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:53,836 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:53,837 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 258; bufvoid = 104857600
2016-12-13 19:07:53,837 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:53,859 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:53,864 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local525031809_0026_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:53,868 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr25/part-r-00000:0+102
2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local525031809_0026_m_000000_0' done.
2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local525031809_0026_m_000000_0
2016-12-13 19:07:53,869 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:53,870 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:53,871 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local525031809_0026_r_000000_0
2016-12-13 19:07:53,874 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:53,929 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4ab0342e
2016-12-13 19:07:53,930 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7631de50
2016-12-13 19:07:53,932 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:53,934 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local525031809_0026_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:53,940 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#26 about to shuffle output of map attempt_local525031809_0026_m_000000_0 decomp: 282 len: 286 to MEMORY
2016-12-13 19:07:53,942 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 282 bytes from map-output for attempt_local525031809_0026_m_000000_0
2016-12-13 19:07:53,942 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 282, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->282
2016-12-13 19:07:53,944 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:53,946 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:53,947 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:53,959 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:53,960 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes
2016-12-13 19:07:53,962 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 282 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 286 bytes from disk
2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:53,963 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:53,964 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes
2016-12-13 19:07:53,965 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.20056740860933667
1___________
*********** new pageRank value is 0.3391795418124502
2___________
*********** new pageRank value is 0.38073982450317145
2___________
*********** new pageRank value is 0.1233676401981714
0___________
2016-12-13 19:07:53,977 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local525031809_0026_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:53,981 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:53,981 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local525031809_0026_r_000000_0 is allowed to commit now
2016-12-13 19:07:53,988 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local525031809_0026_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/_temporary/0/task_local525031809_0026_r_000000
2016-12-13 19:07:53,990 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:53,990 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local525031809_0026_r_000000_0' done.
2016-12-13 19:07:53,991 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local525031809_0026_r_000000_0
2016-12-13 19:07:53,991 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:54,729 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local525031809_0026 running in uber mode : false
2016-12-13 19:07:54,730 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:54,730 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local525031809_0026 completed successfully
2016-12-13 19:07:54,739 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=45006
FILE: Number of bytes written=10131824
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=258
Map output materialized bytes=286
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=286
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1774190592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=122
File Output Format Counters
Bytes Written=112
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=5
success.
5
2016-12-13 19:07:54,749 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:54,756 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:54,759 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:55,178 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:55,190 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:55,211 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local582978068_0027
2016-12-13 19:07:55,356 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:55,357 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local582978068_0027
2016-12-13 19:07:55,357 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:55,358 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:55,366 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:55,367 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local582978068_0027_m_000000_0
2016-12-13 19:07:55,371 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:55,428 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4b81e280
2016-12-13 19:07:55,432 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/part-r-00000:0+100
2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:55,459 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:55,461 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:55,469 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:55,470 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 251; bufvoid = 104857600
2016-12-13 19:07:55,471 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:55,486 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:55,493 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local582978068_0027_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:55,499 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr26/part-r-00000:0+100
2016-12-13 19:07:55,500 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local582978068_0027_m_000000_0' done.
2016-12-13 19:07:55,500 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local582978068_0027_m_000000_0
2016-12-13 19:07:55,501 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:55,502 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:55,502 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local582978068_0027_r_000000_0
2016-12-13 19:07:55,507 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:55,570 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@314779f4
2016-12-13 19:07:55,571 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4b979b72
2016-12-13 19:07:55,573 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:55,577 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local582978068_0027_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:55,587 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#27 about to shuffle output of map attempt_local582978068_0027_m_000000_0 decomp: 275 len: 279 to MEMORY
2016-12-13 19:07:55,589 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 275 bytes from map-output for attempt_local582978068_0027_m_000000_0
2016-12-13 19:07:55,590 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 275, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->275
2016-12-13 19:07:55,591 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:55,595 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:55,595 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:55,609 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:55,609 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes
2016-12-13 19:07:55,611 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 275 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 279 bytes from disk
2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:55,613 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:55,614 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 271 bytes
2016-12-13 19:07:55,616 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.19931442541384786
1___________
*********** new pageRank value is 0.33698682115703876
2___________
*********** new pageRank value is 0.37823385762480544
2___________
*********** new pageRank value is 0.12274114865896807
0___________
2016-12-13 19:07:55,636 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local582978068_0027_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:55,640 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:55,640 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local582978068_0027_r_000000_0 is allowed to commit now
2016-12-13 19:07:55,649 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local582978068_0027_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/_temporary/0/task_local582978068_0027_r_000000
2016-12-13 19:07:55,657 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:55,657 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local582978068_0027_r_000000_0' done.
2016-12-13 19:07:55,658 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local582978068_0027_r_000000_0
2016-12-13 19:07:55,658 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:56,357 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local582978068_0027 running in uber mode : false
2016-12-13 19:07:56,358 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:56,359 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local582978068_0027 completed successfully
2016-12-13 19:07:56,366 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=46826
FILE: Number of bytes written=10520671
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=251
Map output materialized bytes=279
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=279
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1889533952
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=120
File Output Format Counters
Bytes Written=114
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=5
success.
5
2016-12-13 19:07:56,374 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:56,384 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:56,386 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:56,854 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:56,866 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:56,880 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1006109257_0028
2016-12-13 19:07:57,050 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:57,051 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1006109257_0028
2016-12-13 19:07:57,051 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:57,052 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:57,059 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:57,059 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1006109257_0028_m_000000_0
2016-12-13 19:07:57,062 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:57,114 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@753f2425
2016-12-13 19:07:57,118 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/part-r-00000:0+102
2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:57,152 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:57,153 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:57,153 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:57,154 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:57,162 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 258; bufvoid = 104857600
2016-12-13 19:07:57,163 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:57,175 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:57,179 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1006109257_0028_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:57,183 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr27/part-r-00000:0+102
2016-12-13 19:07:57,183 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1006109257_0028_m_000000_0' done.
2016-12-13 19:07:57,184 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1006109257_0028_m_000000_0
2016-12-13 19:07:57,184 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:57,185 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:57,185 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1006109257_0028_r_000000_0
2016-12-13 19:07:57,188 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:57,237 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@b901239
2016-12-13 19:07:57,237 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1f9c621d
2016-12-13 19:07:57,239 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:57,242 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local1006109257_0028_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:57,246 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#28 about to shuffle output of map attempt_local1006109257_0028_m_000000_0 decomp: 282 len: 286 to MEMORY
2016-12-13 19:07:57,247 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 282 bytes from map-output for attempt_local1006109257_0028_m_000000_0
2016-12-13 19:07:57,247 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 282, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->282
2016-12-13 19:07:57,248 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:57,250 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:57,250 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:57,259 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:57,260 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes
2016-12-13 19:07:57,263 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 282 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:57,263 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 286 bytes from disk
2016-12-13 19:07:57,264 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:57,264 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:57,265 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 278 bytes
2016-12-13 19:07:57,266 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.1982493894905423
1___________
*********** new pageRank value is 0.33512300847148907
1___________
*********** new pageRank value is 0.3761037861635444
2___________
*********** new pageRank value is 0.12220863080088534
0___________
2016-12-13 19:07:57,280 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local1006109257_0028_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:57,284 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:57,284 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local1006109257_0028_r_000000_0 is allowed to commit now
2016-12-13 19:07:57,295 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local1006109257_0028_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/_temporary/0/task_local1006109257_0028_r_000000
2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local1006109257_0028_r_000000_0' done.
2016-12-13 19:07:57,299 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local1006109257_0028_r_000000_0
2016-12-13 19:07:57,300 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:58,051 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1006109257_0028 running in uber mode : false
2016-12-13 19:07:58,052 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:58,053 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1006109257_0028 completed successfully
2016-12-13 19:07:58,057 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=48650
FILE: Number of bytes written=10911508
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=258
Map output materialized bytes=286
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=286
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1889533952
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=122
File Output Format Counters
Bytes Written=112
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=4
success.
4
2016-12-13 19:07:58,066 INFO [org.apache.hadoop.metrics.jvm.JvmMetrics] - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-13 19:07:58,073 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-12-13 19:07:58,076 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-13 19:07:58,528 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input paths to process : 1
2016-12-13 19:07:58,540 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1
2016-12-13 19:07:58,559 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local626952993_0029
2016-12-13 19:07:58,716 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/
2016-12-13 19:07:58,717 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local626952993_0029
2016-12-13 19:07:58,717 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null
2016-12-13 19:07:58,718 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-12-13 19:07:58,725 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks
2016-12-13 19:07:58,726 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local626952993_0029_m_000000_0
2016-12-13 19:07:58,728 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:58,780 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3e0a19cb
2016-12-13 19:07:58,784 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/part-r-00000:0+100
2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - mapreduce.task.io.sort.mb: 100
2016-12-13 19:07:58,831 INFO [org.apache.hadoop.mapred.MapTask] - soft limit at 83886080
2016-12-13 19:07:58,832 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufvoid = 104857600
2016-12-13 19:07:58,832 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396; length = 6553600
2016-12-13 19:07:58,833 INFO [org.apache.hadoop.mapred.MapTask] - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-13 19:07:58,839 INFO [org.apache.hadoop.mapred.LocalJobRunner] -
2016-12-13 19:07:58,839 INFO [org.apache.hadoop.mapred.MapTask] - Starting flush of map output
2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - Spilling map output
2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - bufstart = 0; bufend = 252; bufvoid = 104857600
2016-12-13 19:07:58,840 INFO [org.apache.hadoop.mapred.MapTask] - kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
2016-12-13 19:07:58,853 INFO [org.apache.hadoop.mapred.MapTask] - Finished spill 0
2016-12-13 19:07:58,858 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local626952993_0029_m_000000_0 is done. And is in the process of committing
2016-12-13 19:07:58,863 INFO [org.apache.hadoop.mapred.LocalJobRunner] - file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr28/part-r-00000:0+100
2016-12-13 19:07:58,863 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local626952993_0029_m_000000_0' done.
2016-12-13 19:07:58,864 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local626952993_0029_m_000000_0
2016-12-13 19:07:58,864 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete.
2016-12-13 19:07:58,865 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for reduce tasks
2016-12-13 19:07:58,866 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local626952993_0029_r_000000_0
2016-12-13 19:07:58,870 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux.
2016-12-13 19:07:58,926 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@6f36ac8a
2016-12-13 19:07:58,926 INFO [org.apache.hadoop.mapred.ReduceTask] - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6e79d110
2016-12-13 19:07:58,928 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - MergerManager: memoryLimit=1327077760, maxSingleShuffleLimit=331769440, mergeThreshold=875871360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-13 19:07:58,930 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - attempt_local626952993_0029_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-13 19:07:58,935 INFO [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] - localfetcher#29 about to shuffle output of map attempt_local626952993_0029_m_000000_0 decomp: 276 len: 280 to MEMORY
2016-12-13 19:07:58,936 INFO [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] - Read 276 bytes from map-output for attempt_local626952993_0029_m_000000_0
2016-12-13 19:07:58,937 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - closeInMemoryFile -> map-output of size: 276, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->276
2016-12-13 19:07:58,938 INFO [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] - EventFetcher is interrupted.. Returning
2016-12-13 19:07:58,940 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:58,940 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-13 19:07:58,949 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:58,949 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 272 bytes
2016-12-13 19:07:58,952 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merged 1 segments, 276 bytes to disk to satisfy reduce memory limit
2016-12-13 19:07:58,953 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 1 files, 280 bytes from disk
2016-12-13 19:07:58,953 INFO [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] - Merging 0 segments, 0 bytes from memory into reduce
2016-12-13 19:07:58,954 INFO [org.apache.hadoop.mapred.Merger] - Merging 1 sorted segments
2016-12-13 19:07:58,954 INFO [org.apache.hadoop.mapred.Merger] - Down to the last merge-pass, with 1 segments left of total size: 272 bytes
2016-12-13 19:07:58,956 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
*********** new pageRank value is 0.19734410911950637
0___________
*********** new pageRank value is 0.33353876774336305
1___________
*********** new pageRank value is 0.37429322529114195
1___________
*********** new pageRank value is 0.12175599053348046
0___________
2016-12-13 19:07:58,969 INFO [org.apache.hadoop.mapred.Task] - Task:attempt_local626952993_0029_r_000000_0 is done. And is in the process of committing
2016-12-13 19:07:58,978 INFO [org.apache.hadoop.mapred.LocalJobRunner] - 1 / 1 copied.
2016-12-13 19:07:58,979 INFO [org.apache.hadoop.mapred.Task] - Task attempt_local626952993_0029_r_000000_0 is allowed to commit now
2016-12-13 19:07:58,996 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - Saved output of task 'attempt_local626952993_0029_r_000000_0' to file:/D:/Code/MyEclipseJavaCode/myMapReduce/out/pagerank/pr29/_temporary/0/task_local626952993_0029_r_000000
2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce > reduce
2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.Task] - Task 'attempt_local626952993_0029_r_000000_0' done.
2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Finishing task: attempt_local626952993_0029_r_000000_0
2016-12-13 19:07:58,999 INFO [org.apache.hadoop.mapred.LocalJobRunner] - reduce task executor complete.
2016-12-13 19:07:59,717 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local626952993_0029 running in uber mode : false
2016-12-13 19:07:59,718 INFO [org.apache.hadoop.mapreduce.Job] - map 100% reduce 100%
2016-12-13 19:07:59,719 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local626952993_0029 completed successfully
2016-12-13 19:07:59,728 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 34
File System Counters
FILE: Number of bytes read=50472
FILE: Number of bytes written=11300358
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=4
Map output records=11
Map output bytes=252
Map output materialized bytes=280
Input split bytes=139
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=280
Reduce input records=11
Reduce output records=4
Spilled Records=22
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=21
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=1901068288
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=120
File Output Format Counters
Bytes Written=114
zhouls.bigdata.myMapReduce.pagerank.RunJob$Mycounter
my=2
success.
2

代码

package zhouls.bigdata.myMapReduce.pagerank; //pr值

import java.io.IOException;
import java.util.Arrays;

import org.apache.commons.lang.StringUtils;

public class Node {

private double pageRank=1.0;
private String[] adjacentNodeNames;

public static final char fieldSeparator = '\t';//分隔符

public double getPageRank() {
return pageRank;
}

public Node setPageRank(double pageRank) {
this.pageRank = pageRank;
return this;
}

public String[] getAdjacentNodeNames() {
return adjacentNodeNames;
}

public Node setAdjacentNodeNames(String[] adjacentNodeNames) {
this.adjacentNodeNames = adjacentNodeNames;
return this;
}//这一大段的get和set，可以右键，source，产生get和set，自动生成。

public boolean containsAdjacentNodes() {
return adjacentNodeNames != null && adjacentNodeNames.length>0;
}

@Override
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append(pageRank);

if (getAdjacentNodeNames() != null) {
sb.append(fieldSeparator)
.append(StringUtils
.join(getAdjacentNodeNames(), fieldSeparator));
}
return sb.toString();
}

//value =1.0 B D
public static Node fromMR(String value) throws IOException {
String[] parts = StringUtils.splitPreserveAllTokens(
value, fieldSeparator);
if (parts.length < 1) {
throw new IOException(
"Expected 1 or more parts but received " + parts.length);
}
Node node = new Node()
.setPageRank(Double.valueOf(parts[0]));
if (parts.length > 1) {
node.setAdjacentNodeNames(Arrays.copyOfRange(parts, 1,
parts.length));
}
return node;
}
}

package zhouls.bigdata.myMapReduce.pagerank;//网页排名

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class RunJob {

public static enum Mycounter{
my
}

public static void main(String[] args) {
Configuration config =new Configuration();
// config.set("fs.defaultFS", "hdfs://HadoopMaster:9000");
// config.set("yarn.resourcemanager.hostname", "HadoopMaster");
double d =0.001;
int i=0;
while(true){
i++;
try {
config.setInt("runCount", i);
FileSystem fs =FileSystem.get(config);
Job job =Job.getInstance(config);
job.setJarByClass(RunJob.class);
job.setJobName("pr"+i);
job.setMapperClass(PageRankMapper.class);
job.setReducerClass(PageRankReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setInputFormatClass(KeyValueTextInputFormat.class);
// Path inputPath =new Path("hdfs://HadoopMaster:9000/pagerank/pagerank.txt");
Path inputPath =new Path("./data/pagerank/pagerank.txt");
if(i>1){
// inputPath =new Path("hdfs://HadoopMaster:9000/out/pagerank/pr"+(i-1));
inputPath =new Path("./out/pagerank/pr"+(i-1));
}
FileInputFormat.addInputPath(job, inputPath);

// Path outpath =new Path("hdfs://HadoopMaster:9000/out/pagerank/pr"+i);
Path outpath =new Path("./out/pagerank/pr"+i);
if(fs.exists(outpath)){
fs.delete(outpath, true);
}
FileOutputFormat.setOutputPath(job, outpath);

boolean f= job.waitForCompletion(true);
if(f){
System.out.println("success.");
long sum= job.getCounters().findCounter(Mycounter.my).getValue();
System.out.println(sum);
double avgd= sum/4000.0;
if(avgd<d){
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}

}

static class PageRankMapper extends Mapper<Text, Text, Text, Text>{
protected void map(Text key, Text value,
Context context)
throws IOException, InterruptedException {
int runCount= context.getConfiguration().getInt("runCount", 1);
String page =key.toString();
Node node =null;
if(runCount==1){
node =Node.fromMR("1.0"+"\t"+value.toString());
}else{
node =Node.fromMR(value.toString());
}
context.write(new Text(page), new Text(node.toString()));//A:1.0 B D
if(node.containsAdjacentNodes()){
double outValue =node.getPageRank()/node.getAdjacentNodeNames().length;
for (int i = 0; i < node.getAdjacentNodeNames().length; i++) {
String outPage = node.getAdjacentNodeNames()[i];
context.write(new Text(outPage), new Text(outValue+""));//B:0.5 D:0.5
}
}
}
}

static class PageRankReducer extends Reducer<Text, Text, Text, Text>{
protected void reduce(Text arg0, Iterable<Text> arg1,
Context arg2)
throws IOException, InterruptedException {
double sum =0.0;
Node sourceNode =null;
for(Text i:arg1){
Node node =Node.fromMR(i.toString());
if(node.containsAdjacentNodes()){
sourceNode =node;
}else{
sum=sum+node.getPageRank();
}
}

double newPR=(0.15/4)+(0.85*sum);
System.out.println("*********** new pageRank value is "+newPR);

//把新的pr值和计算之前的pr比较
double d= newPR -sourceNode.getPageRank();

int j=(int)( d*1000.0);
j=Math.abs(j);
System.out.println(j+"___________");
arg2.getCounter(Mycounter.my).increment(j);;

sourceNode.setPageRank(newPR);
arg2.write(arg0, new Text(sourceNode.toString()));
}
}
}