原博文出自于：　　http://blog.fens.me/hadoop-mahout-kmeans/　　　　　　　　感谢！

Mahout分步式程序开发 聚类Kmeans

Hadoop家族系列文章，主要介绍Hadoop家族产品，常用的项目包括Hadoop, Hive, Pig, HBase, Sqoop, Mahout, Zookeeper, Avro, Ambari, Chukwa，新增加的项目包括，YARN, Hcatalog, Oozie, Cassandra, Hama, Whirr, Flume, Bigtop, Crunch, Hue等。

从2011年开始，中国进入大数据风起云涌的时代，以Hadoop为代表的家族软件，占据了大数据处理的广阔地盘。开源界及厂商，所有数据软件，无一不向Hadoop靠拢。Hadoop也从小众的高富帅领域，变成了大数据开发的标准。在Hadoop原有技术基础之上，出现了Hadoop家族产品，通过“大数据”概念不断创新，推出科技进步。

作为IT界的开发人员，我们也要跟上节奏，抓住机遇，跟着Hadoop一起雄起！

关于作者：

  • 张丹(Conan), 程序员Java,R,PHP,Javascript
  • weibo：@Conan_Z
  • blog: http://blog.fens.me
  • email: bsspirit@gmail.com

转载请注明出处：
http://blog.fens.me/hadoop-mahout-kmeans/

前言

Mahout是基于Hadoop用于机器学习的程序开发框架，Mahout封装了3大类的机器学习算法，其中包括聚类算法。kmeans是我们经常会提到用到的聚类算法之一，特别处理未知数据集的时，都会先聚类一下，看看数据集会有一些什么样的规则。

本文主要讲解，基于Mahout程序开发,实现分步式的kmeans算法。

目录

  1. 聚类算法kmeans
  2. Mahout开发环境介绍
  3. 用Mahout实现聚类算法kmeans
  4. 用R语言可视化结果
  5. 模板项目上传github

1. 聚类算法kmeans

聚类分析是数据挖掘及机器学习领域内的重点问题之一，在数据挖掘、模式识别、决策支持、机器学习及图像分割等领域有广泛的应用，是最重要的数据分析方法之一。聚类是在给定的数据集合中寻找同类的数据子集合，每一个子集合形成一个类簇，同类簇中的数据具有更大的相似性。聚类算法大体上可分为基于划分的方法、基于层次的方法、基于密度的方法、基于网格的方法以及基于模型的方法。

k-means algorithm算法是一种得到最广泛使用的基于划分的聚类算法，把n个对象分为k个簇，以使簇内具有较高的相似度。相似度的计算根据一个簇中对象的平均值来进行。它与处理混合正态分布的最大期望算法很相似，因为他们都试图找到数据中自然聚类的中心。

算法首先随机地选择k个对象，每个对象初始地代表了一个簇的平均值或中心。对剩余的每个对象根据其与各个簇中心的距离，将它赋给最近的簇，然后重新计算每个簇的平均值。这个过程不断重复，直到准则函数收敛。

kmeans介绍摘自：http://zh.wikipedia.org/wiki/K平均算法

2. Mahout开发环境介绍

接上一篇文章：Mahout分步式程序开发 基于物品的协同过滤ItemCF

所有环境变量 和 系统配置 与上文一致！

3. 用Mahout实现聚类算法kmeans

实现步骤:

  • 1. 准备数据文件: randomData.csv
  • 2. Java程序：KmeansHadoop.java
  • 3. 运行程序
  • 4. 聚类结果解读
  • 5. HDFS产生的目录

1). 准备数据文件: randomData.csv
数据文件randomData.csv，由R语言通过“随机正太分布函数”程序生成，单机内存实验请参考文章：
用Maven构建Mahout项目

原始数据文件：这里只截取了一部分数据。


  2. ~ vi datafile/randomData.csv
  4. -0.883033363823402 -3.31967192630249
  5. -2.39312626419456 3.34726861118871
  6. 2.66976353341256 1.85144276077058
  7. -1.09922906899594 -6.06261735207489
  8. -4.36361936997216 1.90509905380532
  9. -0.00351835125495037 -0.610105996559153
  10. -2.9962958796338 -3.60959839525735
  11. -3.27529418132066 0.0230099799641799
  12. 2.17665594420569 6.77290756817957
  13. -2.47862038335637 2.53431833167278
  14. 5.53654901906814 2.65089785582474
  15. 5.66257474538338 6.86783609641077
  16. -0.558946883114376 1.22332819416237
  17. 5.11728525486132 3.74663871584768
  18. 1.91240516693351 2.95874731384062
  19. -2.49747101306535 2.05006504756875
  20. 3.98781883213459 1.00780938946366
  21. 5.47470532716682 5.35084411045171

注：由于Mahout中kmeans算法，默认的分融符是” “(空格)，因些我把逗号分隔的数据文件，改成以空格分隔。

2). Java程序：KmeansHadoop.java

kmeans的算法实现，请查看Mahout in Action。


  2. package org.conan.mymahout.cluster08;
  4. import org.apache.hadoop.fs.Path;
  5. import org.apache.hadoop.mapred.JobConf;
  6. import org.apache.mahout.clustering.conversion.InputDriver;
  7. import org.apache.mahout.clustering.kmeans.KMeansDriver;
  8. import org.apache.mahout.clustering.kmeans.RandomSeedGenerator;
  9. import org.apache.mahout.common.distance.DistanceMeasure;
  10. import org.apache.mahout.common.distance.EuclideanDistanceMeasure;
  11. import org.apache.mahout.utils.clustering.ClusterDumper;
  12. import org.conan.mymahout.hdfs.HdfsDAO;
  13. import org.conan.mymahout.recommendation.ItemCFHadoop;
  15. public class KmeansHadoop {
  16.     private static final String HDFS = "hdfs://192.168.1.210:9000";
  18.     public static void main(String[] args) throws Exception {
  19.         String localFile = "datafile/randomData.csv";
  20.         String inPath = HDFS + "/user/hdfs/mix_data";
  21.         String seqFile = inPath + "/seqfile";
  22.         String seeds = inPath + "/seeds";
  23.         String outPath = inPath + "/result/";
  24.         String clusteredPoints = outPath + "/clusteredPoints";
  26.         JobConf conf = config();
  27.         HdfsDAO hdfs = new HdfsDAO(HDFS, conf);
  28.         hdfs.rmr(inPath);
  29.         hdfs.mkdirs(inPath);
  30.         hdfs.copyFile(localFile, inPath);
  31.         hdfs.ls(inPath);
  33.         InputDriver.runJob(new Path(inPath), new Path(seqFile), "org.apache.mahout.math.RandomAccessSparseVector");
  35.         int k = 3;
  36.         Path seqFilePath = new Path(seqFile);
  37.         Path clustersSeeds = new Path(seeds);
  38.         DistanceMeasure measure = new EuclideanDistanceMeasure();
  39.         clustersSeeds = RandomSeedGenerator.buildRandom(conf, seqFilePath, clustersSeeds, k, measure);
  40.         KMeansDriver.run(conf, seqFilePath, clustersSeeds, new Path(outPath), measure, 0.01, 10, true, 0.01, false);
  42.         Path outGlobPath = new Path(outPath, "clusters-*-final");
  43.         Path clusteredPointsPath = new Path(clusteredPoints);
  44.         System.out.printf("Dumping out clusters from clusters: %s and clusteredPoints: %s\n", outGlobPath, clusteredPointsPath);
  46.         ClusterDumper clusterDumper = new ClusterDumper(outGlobPath, clusteredPointsPath);
  47.         clusterDumper.printClusters(null);
  48.     }
  50.     public static JobConf config() {
  51.         JobConf conf = new JobConf(ItemCFHadoop.class);
  52.         conf.setJobName("ItemCFHadoop");
  53.         conf.addResource("classpath:/hadoop/core-site.xml");
  54.         conf.addResource("classpath:/hadoop/hdfs-site.xml");
  55.         conf.addResource("classpath:/hadoop/mapred-site.xml");
  56.         return conf;
  57.     }
  59. }

3). 运行程序
控制台输出:


  2. Delete: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  3. Create: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  4. copy from: datafile/randomData.csv to hdfs://192.168.1.210:9000/user/hdfs/mix_data
  5. ls: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  6. ==========================================================
  7. name: hdfs://192.168.1.210:9000/user/hdfs/mix_data/randomData.csv, folder: false, size: 36655
  8. ==========================================================
  9. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  10. SLF4J: Defaulting to no-operation (NOP) logger implementation
  11. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
  12. 2013-10-14 15:39:31 org.apache.hadoop.util.NativeCodeLoader
  13. 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  14. 2013-10-14 15:39:31 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  15. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  16. 2013-10-14 15:39:31 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  17. 信息: Total input paths to process : 1
  18. 2013-10-14 15:39:31 org.apache.hadoop.io.compress.snappy.LoadSnappy
  19. 警告: Snappy native library not loaded
  20. 2013-10-14 15:39:31 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  21. 信息: Running job: job_local_0001
  22. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task initialize
  23. 信息:  Using ResourceCalculatorPlugin : null
  24. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task done
  25. 信息: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  26. 2013-10-14 15:39:31 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  27. 信息:
  28. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task commit
  29. 信息: Task attempt_local_0001_m_000000_0 is allowed to commit now
  30. 2013-10-14 15:39:31 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  31. 信息: Saved output of task 'attempt_local_0001_m_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/seqfile
  32. 2013-10-14 15:39:31 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  33. 信息:
  34. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task sendDone
  35. 信息: Task 'attempt_local_0001_m_000000_0' done.
  36. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  37. 信息:  map 100% reduce 0%
  38. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  39. 信息: Job complete: job_local_0001
  40. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  41. 信息: Counters: 11
  42. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  43. 信息:   File Output Format Counters
  44. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  45. 信息:     Bytes Written=31390
  46. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  47. 信息:   File Input Format Counters
  48. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  49. 信息:     Bytes Read=36655
  50. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  51. 信息:   FileSystemCounters
  52. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  53. 信息:     FILE_BYTES_READ=475910
  54. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  55. 信息:     HDFS_BYTES_READ=36655
  56. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  57. 信息:     FILE_BYTES_WRITTEN=506350
  58. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  59. 信息:     HDFS_BYTES_WRITTEN=68045
  60. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  61. 信息:   Map-Reduce Framework
  62. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  63. 信息:     Map input records=1000
  64. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  65. 信息:     Spilled Records=0
  66. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  67. 信息:     Total committed heap usage (bytes)=188284928
  68. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  69. 信息:     SPLIT_RAW_BYTES=124
  70. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Counters log
  71. 信息:     Map output records=1000
  72. 2013-10-14 15:39:32 org.apache.hadoop.io.compress.CodecPool getCompressor
  73. 信息: Got brand-new compressor
  74. 2013-10-14 15:39:32 org.apache.hadoop.io.compress.CodecPool getDecompressor
  75. 信息: Got brand-new decompressor
  76. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  77. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  78. 2013-10-14 15:39:32 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  79. 信息: Total input paths to process : 1
  80. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  81. 信息: Running job: job_local_0002
  82. 2013-10-14 15:39:32 org.apache.hadoop.mapred.Task initialize
  83. 信息:  Using ResourceCalculatorPlugin : null
  84. 2013-10-14 15:39:32 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  85. 信息: io.sort.mb = 100
  86. 2013-10-14 15:39:32 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  87. 信息: data buffer = 79691776/99614720
  88. 2013-10-14 15:39:32 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  89. 信息: record buffer = 262144/327680
  90. 2013-10-14 15:39:33 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  91. 信息: Starting flush of map output
  92. 2013-10-14 15:39:33 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  93. 信息: Finished spill 0
  94. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task done
  95. 信息: Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
  96. 2013-10-14 15:39:33 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  97. 信息:
  98. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task sendDone
  99. 信息: Task 'attempt_local_0002_m_000000_0' done.
  100. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task initialize
  101. 信息:  Using ResourceCalculatorPlugin : null
  102. 2013-10-14 15:39:33 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  103. 信息:
  104. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Merger$MergeQueue merge
  105. 信息: Merging 1 sorted segments
  106. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Merger$MergeQueue merge
  107. 信息: Down to the last merge-pass, with 1 segments left of total size: 623 bytes
  108. 2013-10-14 15:39:33 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  109. 信息:
  110. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task done
  111. 信息: Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
  112. 2013-10-14 15:39:33 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  113. 信息:
  114. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task commit
  115. 信息: Task attempt_local_0002_r_000000_0 is allowed to commit now
  116. 2013-10-14 15:39:33 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  117. 信息: Saved output of task 'attempt_local_0002_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-1
  118. 2013-10-14 15:39:33 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  119. 信息: reduce > reduce
  120. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Task sendDone
  121. 信息: Task 'attempt_local_0002_r_000000_0' done.
  122. 2013-10-14 15:39:33 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  123. 信息:  map 100% reduce 100%
  124. 2013-10-14 15:39:33 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  125. 信息: Job complete: job_local_0002
  126. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  127. 信息: Counters: 19
  128. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  129. 信息:   File Output Format Counters
  130. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  131. 信息:     Bytes Written=695
  132. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  133. 信息:   FileSystemCounters
  134. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  135. 信息:     FILE_BYTES_READ=4239303
  136. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  137. 信息:     HDFS_BYTES_READ=203963
  138. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  139. 信息:     FILE_BYTES_WRITTEN=4457168
  140. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  141. 信息:     HDFS_BYTES_WRITTEN=140321
  142. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  143. 信息:   File Input Format Counters
  144. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  145. 信息:     Bytes Read=31390
  146. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  147. 信息:   Map-Reduce Framework
  148. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  149. 信息:     Map output materialized bytes=627
  150. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  151. 信息:     Map input records=1000
  152. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  153. 信息:     Reduce shuffle bytes=0
  154. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  155. 信息:     Spilled Records=6
  156. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  157. 信息:     Map output bytes=612
  158. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  159. 信息:     Total committed heap usage (bytes)=376569856
  160. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  161. 信息:     SPLIT_RAW_BYTES=130
  162. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  163. 信息:     Combine input records=0
  164. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  165. 信息:     Reduce input records=3
  166. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  167. 信息:     Reduce input groups=3
  168. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  169. 信息:     Combine output records=0
  170. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  171. 信息:     Reduce output records=3
  172. 2013-10-14 15:39:33 org.apache.hadoop.mapred.Counters log
  173. 信息:     Map output records=3
  174. 2013-10-14 15:39:34 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  175. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  176. 2013-10-14 15:39:34 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  177. 信息: Total input paths to process : 1
  178. 2013-10-14 15:39:34 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  179. 信息: Running job: job_local_0003
  180. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task initialize
  181. 信息:  Using ResourceCalculatorPlugin : null
  182. 2013-10-14 15:39:34 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  183. 信息: io.sort.mb = 100
  184. 2013-10-14 15:39:34 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  185. 信息: data buffer = 79691776/99614720
  186. 2013-10-14 15:39:34 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  187. 信息: record buffer = 262144/327680
  188. 2013-10-14 15:39:34 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  189. 信息: Starting flush of map output
  190. 2013-10-14 15:39:34 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  191. 信息: Finished spill 0
  192. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task done
  193. 信息: Task:attempt_local_0003_m_000000_0 is done. And is in the process of commiting
  194. 2013-10-14 15:39:34 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  195. 信息:
  196. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task sendDone
  197. 信息: Task 'attempt_local_0003_m_000000_0' done.
  198. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task initialize
  199. 信息:  Using ResourceCalculatorPlugin : null
  200. 2013-10-14 15:39:34 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  201. 信息:
  202. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Merger$MergeQueue merge
  203. 信息: Merging 1 sorted segments
  204. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Merger$MergeQueue merge
  205. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  206. 2013-10-14 15:39:34 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  207. 信息:
  208. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task done
  209. 信息: Task:attempt_local_0003_r_000000_0 is done. And is in the process of commiting
  210. 2013-10-14 15:39:34 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  211. 信息:
  212. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task commit
  213. 信息: Task attempt_local_0003_r_000000_0 is allowed to commit now
  214. 2013-10-14 15:39:34 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  215. 信息: Saved output of task 'attempt_local_0003_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-2
  216. 2013-10-14 15:39:34 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  217. 信息: reduce > reduce
  218. 2013-10-14 15:39:34 org.apache.hadoop.mapred.Task sendDone
  219. 信息: Task 'attempt_local_0003_r_000000_0' done.
  220. 2013-10-14 15:39:35 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  221. 信息:  map 100% reduce 100%
  222. 2013-10-14 15:39:35 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  223. 信息: Job complete: job_local_0003
  224. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  225. 信息: Counters: 19
  226. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  227. 信息:   File Output Format Counters
  228. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  229. 信息:     Bytes Written=695
  230. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  231. 信息:   FileSystemCounters
  232. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  233. 信息:     FILE_BYTES_READ=7527467
  234. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  235. 信息:     HDFS_BYTES_READ=271193
  236. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  237. 信息:     FILE_BYTES_WRITTEN=7901744
  238. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  239. 信息:     HDFS_BYTES_WRITTEN=142099
  240. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  241. 信息:   File Input Format Counters
  242. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  243. 信息:     Bytes Read=31390
  244. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  245. 信息:   Map-Reduce Framework
  246. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  247. 信息:     Map output materialized bytes=681
  248. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  249. 信息:     Map input records=1000
  250. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  251. 信息:     Reduce shuffle bytes=0
  252. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  253. 信息:     Spilled Records=6
  254. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  255. 信息:     Map output bytes=666
  256. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  257. 信息:     Total committed heap usage (bytes)=575930368
  258. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  259. 信息:     SPLIT_RAW_BYTES=130
  260. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  261. 信息:     Combine input records=0
  262. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  263. 信息:     Reduce input records=3
  264. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  265. 信息:     Reduce input groups=3
  266. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  267. 信息:     Combine output records=0
  268. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  269. 信息:     Reduce output records=3
  270. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Counters log
  271. 信息:     Map output records=3
  272. 2013-10-14 15:39:35 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  273. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  274. 2013-10-14 15:39:35 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  275. 信息: Total input paths to process : 1
  276. 2013-10-14 15:39:35 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  277. 信息: Running job: job_local_0004
  278. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task initialize
  279. 信息:  Using ResourceCalculatorPlugin : null
  280. 2013-10-14 15:39:35 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  281. 信息: io.sort.mb = 100
  282. 2013-10-14 15:39:35 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  283. 信息: data buffer = 79691776/99614720
  284. 2013-10-14 15:39:35 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  285. 信息: record buffer = 262144/327680
  286. 2013-10-14 15:39:35 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  287. 信息: Starting flush of map output
  288. 2013-10-14 15:39:35 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  289. 信息: Finished spill 0
  290. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task done
  291. 信息: Task:attempt_local_0004_m_000000_0 is done. And is in the process of commiting
  292. 2013-10-14 15:39:35 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  293. 信息:
  294. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task sendDone
  295. 信息: Task 'attempt_local_0004_m_000000_0' done.
  296. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task initialize
  297. 信息:  Using ResourceCalculatorPlugin : null
  298. 2013-10-14 15:39:35 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  299. 信息:
  300. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Merger$MergeQueue merge
  301. 信息: Merging 1 sorted segments
  302. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Merger$MergeQueue merge
  303. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  304. 2013-10-14 15:39:35 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  305. 信息:
  306. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task done
  307. 信息: Task:attempt_local_0004_r_000000_0 is done. And is in the process of commiting
  308. 2013-10-14 15:39:35 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  309. 信息:
  310. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task commit
  311. 信息: Task attempt_local_0004_r_000000_0 is allowed to commit now
  312. 2013-10-14 15:39:35 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  313. 信息: Saved output of task 'attempt_local_0004_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-3
  314. 2013-10-14 15:39:35 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  315. 信息: reduce > reduce
  316. 2013-10-14 15:39:35 org.apache.hadoop.mapred.Task sendDone
  317. 信息: Task 'attempt_local_0004_r_000000_0' done.
  318. 2013-10-14 15:39:36 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  319. 信息:  map 100% reduce 100%
  320. 2013-10-14 15:39:36 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  321. 信息: Job complete: job_local_0004
  322. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  323. 信息: Counters: 19
  324. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  325. 信息:   File Output Format Counters
  326. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  327. 信息:     Bytes Written=695
  328. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  329. 信息:   FileSystemCounters
  330. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  331. 信息:     FILE_BYTES_READ=10815685
  332. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  333. 信息:     HDFS_BYTES_READ=338143
  334. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  335. 信息:     FILE_BYTES_WRITTEN=11346320
  336. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  337. 信息:     HDFS_BYTES_WRITTEN=143877
  338. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  339. 信息:   File Input Format Counters
  340. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  341. 信息:     Bytes Read=31390
  342. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  343. 信息:   Map-Reduce Framework
  344. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  345. 信息:     Map output materialized bytes=681
  346. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  347. 信息:     Map input records=1000
  348. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  349. 信息:     Reduce shuffle bytes=0
  350. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  351. 信息:     Spilled Records=6
  352. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  353. 信息:     Map output bytes=666
  354. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  355. 信息:     Total committed heap usage (bytes)=775290880
  356. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  357. 信息:     SPLIT_RAW_BYTES=130
  358. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  359. 信息:     Combine input records=0
  360. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  361. 信息:     Reduce input records=3
  362. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  363. 信息:     Reduce input groups=3
  364. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  365. 信息:     Combine output records=0
  366. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  367. 信息:     Reduce output records=3
  368. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Counters log
  369. 信息:     Map output records=3
  370. 2013-10-14 15:39:36 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  371. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  372. 2013-10-14 15:39:36 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  373. 信息: Total input paths to process : 1
  374. 2013-10-14 15:39:36 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  375. 信息: Running job: job_local_0005
  376. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task initialize
  377. 信息:  Using ResourceCalculatorPlugin : null
  378. 2013-10-14 15:39:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  379. 信息: io.sort.mb = 100
  380. 2013-10-14 15:39:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  381. 信息: data buffer = 79691776/99614720
  382. 2013-10-14 15:39:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  383. 信息: record buffer = 262144/327680
  384. 2013-10-14 15:39:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  385. 信息: Starting flush of map output
  386. 2013-10-14 15:39:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  387. 信息: Finished spill 0
  388. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task done
  389. 信息: Task:attempt_local_0005_m_000000_0 is done. And is in the process of commiting
  390. 2013-10-14 15:39:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  391. 信息:
  392. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task sendDone
  393. 信息: Task 'attempt_local_0005_m_000000_0' done.
  394. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task initialize
  395. 信息:  Using ResourceCalculatorPlugin : null
  396. 2013-10-14 15:39:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  397. 信息:
  398. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Merger$MergeQueue merge
  399. 信息: Merging 1 sorted segments
  400. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Merger$MergeQueue merge
  401. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  402. 2013-10-14 15:39:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  403. 信息:
  404. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task done
  405. 信息: Task:attempt_local_0005_r_000000_0 is done. And is in the process of commiting
  406. 2013-10-14 15:39:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  407. 信息:
  408. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task commit
  409. 信息: Task attempt_local_0005_r_000000_0 is allowed to commit now
  410. 2013-10-14 15:39:36 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  411. 信息: Saved output of task 'attempt_local_0005_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-4
  412. 2013-10-14 15:39:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  413. 信息: reduce > reduce
  414. 2013-10-14 15:39:36 org.apache.hadoop.mapred.Task sendDone
  415. 信息: Task 'attempt_local_0005_r_000000_0' done.
  416. 2013-10-14 15:39:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  417. 信息:  map 100% reduce 100%
  418. 2013-10-14 15:39:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  419. 信息: Job complete: job_local_0005
  420. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  421. 信息: Counters: 19
  422. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  423. 信息:   File Output Format Counters
  424. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  425. 信息:     Bytes Written=695
  426. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  427. 信息:   FileSystemCounters
  428. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  429. 信息:     FILE_BYTES_READ=14103903
  430. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  431. 信息:     HDFS_BYTES_READ=405093
  432. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  433. 信息:     FILE_BYTES_WRITTEN=14790888
  434. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  435. 信息:     HDFS_BYTES_WRITTEN=145655
  436. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  437. 信息:   File Input Format Counters
  438. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  439. 信息:     Bytes Read=31390
  440. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  441. 信息:   Map-Reduce Framework
  442. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  443. 信息:     Map output materialized bytes=681
  444. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  445. 信息:     Map input records=1000
  446. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  447. 信息:     Reduce shuffle bytes=0
  448. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  449. 信息:     Spilled Records=6
  450. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  451. 信息:     Map output bytes=666
  452. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  453. 信息:     Total committed heap usage (bytes)=974651392
  454. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  455. 信息:     SPLIT_RAW_BYTES=130
  456. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  457. 信息:     Combine input records=0
  458. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  459. 信息:     Reduce input records=3
  460. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  461. 信息:     Reduce input groups=3
  462. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  463. 信息:     Combine output records=0
  464. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  465. 信息:     Reduce output records=3
  466. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Counters log
  467. 信息:     Map output records=3
  468. 2013-10-14 15:39:37 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  469. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  470. 2013-10-14 15:39:37 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  471. 信息: Total input paths to process : 1
  472. 2013-10-14 15:39:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  473. 信息: Running job: job_local_0006
  474. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task initialize
  475. 信息:  Using ResourceCalculatorPlugin : null
  476. 2013-10-14 15:39:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  477. 信息: io.sort.mb = 100
  478. 2013-10-14 15:39:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  479. 信息: data buffer = 79691776/99614720
  480. 2013-10-14 15:39:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  481. 信息: record buffer = 262144/327680
  482. 2013-10-14 15:39:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  483. 信息: Starting flush of map output
  484. 2013-10-14 15:39:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  485. 信息: Finished spill 0
  486. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task done
  487. 信息: Task:attempt_local_0006_m_000000_0 is done. And is in the process of commiting
  488. 2013-10-14 15:39:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  489. 信息:
  490. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task sendDone
  491. 信息: Task 'attempt_local_0006_m_000000_0' done.
  492. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task initialize
  493. 信息:  Using ResourceCalculatorPlugin : null
  494. 2013-10-14 15:39:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  495. 信息:
  496. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Merger$MergeQueue merge
  497. 信息: Merging 1 sorted segments
  498. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Merger$MergeQueue merge
  499. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  500. 2013-10-14 15:39:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  501. 信息:
  502. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task done
  503. 信息: Task:attempt_local_0006_r_000000_0 is done. And is in the process of commiting
  504. 2013-10-14 15:39:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  505. 信息:
  506. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task commit
  507. 信息: Task attempt_local_0006_r_000000_0 is allowed to commit now
  508. 2013-10-14 15:39:37 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  509. 信息: Saved output of task 'attempt_local_0006_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-5
  510. 2013-10-14 15:39:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  511. 信息: reduce > reduce
  512. 2013-10-14 15:39:37 org.apache.hadoop.mapred.Task sendDone
  513. 信息: Task 'attempt_local_0006_r_000000_0' done.
  514. 2013-10-14 15:39:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  515. 信息:  map 100% reduce 100%
  516. 2013-10-14 15:39:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  517. 信息: Job complete: job_local_0006
  518. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  519. 信息: Counters: 19
  520. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  521. 信息:   File Output Format Counters
  522. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  523. 信息:     Bytes Written=695
  524. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  525. 信息:   FileSystemCounters
  526. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  527. 信息:     FILE_BYTES_READ=17392121
  528. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  529. 信息:     HDFS_BYTES_READ=472043
  530. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  531. 信息:     FILE_BYTES_WRITTEN=18235456
  532. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  533. 信息:     HDFS_BYTES_WRITTEN=147433
  534. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  535. 信息:   File Input Format Counters
  536. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  537. 信息:     Bytes Read=31390
  538. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  539. 信息:   Map-Reduce Framework
  540. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  541. 信息:     Map output materialized bytes=681
  542. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  543. 信息:     Map input records=1000
  544. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  545. 信息:     Reduce shuffle bytes=0
  546. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  547. 信息:     Spilled Records=6
  548. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  549. 信息:     Map output bytes=666
  550. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  551. 信息:     Total committed heap usage (bytes)=1174011904
  552. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  553. 信息:     SPLIT_RAW_BYTES=130
  554. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  555. 信息:     Combine input records=0
  556. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  557. 信息:     Reduce input records=3
  558. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  559. 信息:     Reduce input groups=3
  560. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  561. 信息:     Combine output records=0
  562. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  563. 信息:     Reduce output records=3
  564. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Counters log
  565. 信息:     Map output records=3
  566. 2013-10-14 15:39:38 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  567. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  568. 2013-10-14 15:39:38 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  569. 信息: Total input paths to process : 1
  570. 2013-10-14 15:39:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  571. 信息: Running job: job_local_0007
  572. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task initialize
  573. 信息:  Using ResourceCalculatorPlugin : null
  574. 2013-10-14 15:39:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  575. 信息: io.sort.mb = 100
  576. 2013-10-14 15:39:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  577. 信息: data buffer = 79691776/99614720
  578. 2013-10-14 15:39:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  579. 信息: record buffer = 262144/327680
  580. 2013-10-14 15:39:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  581. 信息: Starting flush of map output
  582. 2013-10-14 15:39:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  583. 信息: Finished spill 0
  584. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task done
  585. 信息: Task:attempt_local_0007_m_000000_0 is done. And is in the process of commiting
  586. 2013-10-14 15:39:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  587. 信息:
  588. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task sendDone
  589. 信息: Task 'attempt_local_0007_m_000000_0' done.
  590. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task initialize
  591. 信息:  Using ResourceCalculatorPlugin : null
  592. 2013-10-14 15:39:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  593. 信息:
  594. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Merger$MergeQueue merge
  595. 信息: Merging 1 sorted segments
  596. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Merger$MergeQueue merge
  597. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  598. 2013-10-14 15:39:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  599. 信息:
  600. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task done
  601. 信息: Task:attempt_local_0007_r_000000_0 is done. And is in the process of commiting
  602. 2013-10-14 15:39:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  603. 信息:
  604. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task commit
  605. 信息: Task attempt_local_0007_r_000000_0 is allowed to commit now
  606. 2013-10-14 15:39:38 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  607. 信息: Saved output of task 'attempt_local_0007_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-6
  608. 2013-10-14 15:39:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  609. 信息: reduce > reduce
  610. 2013-10-14 15:39:38 org.apache.hadoop.mapred.Task sendDone
  611. 信息: Task 'attempt_local_0007_r_000000_0' done.
  612. 2013-10-14 15:39:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  613. 信息:  map 100% reduce 100%
  614. 2013-10-14 15:39:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  615. 信息: Job complete: job_local_0007
  616. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  617. 信息: Counters: 19
  618. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  619. 信息:   File Output Format Counters
  620. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  621. 信息:     Bytes Written=695
  622. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  623. 信息:   FileSystemCounters
  624. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  625. 信息:     FILE_BYTES_READ=20680339
  626. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  627. 信息:     HDFS_BYTES_READ=538993
  628. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  629. 信息:     FILE_BYTES_WRITTEN=21680040
  630. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  631. 信息:     HDFS_BYTES_WRITTEN=149211
  632. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  633. 信息:   File Input Format Counters
  634. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  635. 信息:     Bytes Read=31390
  636. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  637. 信息:   Map-Reduce Framework
  638. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  639. 信息:     Map output materialized bytes=681
  640. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  641. 信息:     Map input records=1000
  642. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  643. 信息:     Reduce shuffle bytes=0
  644. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  645. 信息:     Spilled Records=6
  646. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  647. 信息:     Map output bytes=666
  648. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  649. 信息:     Total committed heap usage (bytes)=1373372416
  650. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  651. 信息:     SPLIT_RAW_BYTES=130
  652. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  653. 信息:     Combine input records=0
  654. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  655. 信息:     Reduce input records=3
  656. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  657. 信息:     Reduce input groups=3
  658. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  659. 信息:     Combine output records=0
  660. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  661. 信息:     Reduce output records=3
  662. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Counters log
  663. 信息:     Map output records=3
  664. 2013-10-14 15:39:39 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  665. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  666. 2013-10-14 15:39:39 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  667. 信息: Total input paths to process : 1
  668. 2013-10-14 15:39:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  669. 信息: Running job: job_local_0008
  670. 2013-10-14 15:39:39 org.apache.hadoop.mapred.Task initialize
  671. 信息:  Using ResourceCalculatorPlugin : null
  672. 2013-10-14 15:39:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  673. 信息: io.sort.mb = 100
  674. 2013-10-14 15:39:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  675. 信息: data buffer = 79691776/99614720
  676. 2013-10-14 15:39:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  677. 信息: record buffer = 262144/327680
  678. 2013-10-14 15:39:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  679. 信息: Starting flush of map output
  680. 2013-10-14 15:39:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  681. 信息: Finished spill 0
  682. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task done
  683. 信息: Task:attempt_local_0008_m_000000_0 is done. And is in the process of commiting
  684. 2013-10-14 15:39:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  685. 信息:
  686. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task sendDone
  687. 信息: Task 'attempt_local_0008_m_000000_0' done.
  688. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task initialize
  689. 信息:  Using ResourceCalculatorPlugin : null
  690. 2013-10-14 15:39:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  691. 信息:
  692. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Merger$MergeQueue merge
  693. 信息: Merging 1 sorted segments
  694. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Merger$MergeQueue merge
  695. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  696. 2013-10-14 15:39:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  697. 信息:
  698. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task done
  699. 信息: Task:attempt_local_0008_r_000000_0 is done. And is in the process of commiting
  700. 2013-10-14 15:39:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  701. 信息:
  702. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task commit
  703. 信息: Task attempt_local_0008_r_000000_0 is allowed to commit now
  704. 2013-10-14 15:39:40 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  705. 信息: Saved output of task 'attempt_local_0008_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-7
  706. 2013-10-14 15:39:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  707. 信息: reduce > reduce
  708. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Task sendDone
  709. 信息: Task 'attempt_local_0008_r_000000_0' done.
  710. 2013-10-14 15:39:40 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  711. 信息:  map 100% reduce 100%
  712. 2013-10-14 15:39:40 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  713. 信息: Job complete: job_local_0008
  714. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  715. 信息: Counters: 19
  716. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  717. 信息:   File Output Format Counters
  718. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  719. 信息:     Bytes Written=695
  720. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  721. 信息:   FileSystemCounters
  722. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  723. 信息:     FILE_BYTES_READ=23968557
  724. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  725. 信息:     HDFS_BYTES_READ=605943
  726. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  727. 信息:     FILE_BYTES_WRITTEN=25124624
  728. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  729. 信息:     HDFS_BYTES_WRITTEN=150989
  730. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  731. 信息:   File Input Format Counters
  732. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  733. 信息:     Bytes Read=31390
  734. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  735. 信息:   Map-Reduce Framework
  736. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  737. 信息:     Map output materialized bytes=681
  738. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  739. 信息:     Map input records=1000
  740. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  741. 信息:     Reduce shuffle bytes=0
  742. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  743. 信息:     Spilled Records=6
  744. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  745. 信息:     Map output bytes=666
  746. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  747. 信息:     Total committed heap usage (bytes)=1572732928
  748. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  749. 信息:     SPLIT_RAW_BYTES=130
  750. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  751. 信息:     Combine input records=0
  752. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  753. 信息:     Reduce input records=3
  754. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  755. 信息:     Reduce input groups=3
  756. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  757. 信息:     Combine output records=0
  758. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  759. 信息:     Reduce output records=3
  760. 2013-10-14 15:39:40 org.apache.hadoop.mapred.Counters log
  761. 信息:     Map output records=3
  762. 2013-10-14 15:39:41 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  763. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  764. 2013-10-14 15:39:41 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  765. 信息: Total input paths to process : 1
  766. 2013-10-14 15:39:41 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  767. 信息: Running job: job_local_0009
  768. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task initialize
  769. 信息:  Using ResourceCalculatorPlugin : null
  770. 2013-10-14 15:39:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  771. 信息: io.sort.mb = 100
  772. 2013-10-14 15:39:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  773. 信息: data buffer = 79691776/99614720
  774. 2013-10-14 15:39:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  775. 信息: record buffer = 262144/327680
  776. 2013-10-14 15:39:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  777. 信息: Starting flush of map output
  778. 2013-10-14 15:39:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  779. 信息: Finished spill 0
  780. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task done
  781. 信息: Task:attempt_local_0009_m_000000_0 is done. And is in the process of commiting
  782. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  783. 信息:
  784. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task sendDone
  785. 信息: Task 'attempt_local_0009_m_000000_0' done.
  786. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task initialize
  787. 信息:  Using ResourceCalculatorPlugin : null
  788. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  789. 信息:
  790. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Merger$MergeQueue merge
  791. 信息: Merging 1 sorted segments
  792. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Merger$MergeQueue merge
  793. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  794. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  795. 信息:
  796. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task done
  797. 信息: Task:attempt_local_0009_r_000000_0 is done. And is in the process of commiting
  798. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  799. 信息:
  800. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task commit
  801. 信息: Task attempt_local_0009_r_000000_0 is allowed to commit now
  802. 2013-10-14 15:39:41 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  803. 信息: Saved output of task 'attempt_local_0009_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-8
  804. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  805. 信息: reduce > reduce
  806. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task sendDone
  807. 信息: Task 'attempt_local_0009_r_000000_0' done.
  808. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  809. 信息:  map 100% reduce 100%
  810. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  811. 信息: Job complete: job_local_0009
  812. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  813. 信息: Counters: 19
  814. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  815. 信息:   File Output Format Counters
  816. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  817. 信息:     Bytes Written=695
  818. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  819. 信息:   FileSystemCounters
  820. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  821. 信息:     FILE_BYTES_READ=27256775
  822. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  823. 信息:     HDFS_BYTES_READ=673669
  824. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  825. 信息:     FILE_BYTES_WRITTEN=28569192
  826. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  827. 信息:     HDFS_BYTES_WRITTEN=152767
  828. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  829. 信息:   File Input Format Counters
  830. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  831. 信息:     Bytes Read=31390
  832. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  833. 信息:   Map-Reduce Framework
  834. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  835. 信息:     Map output materialized bytes=681
  836. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  837. 信息:     Map input records=1000
  838. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  839. 信息:     Reduce shuffle bytes=0
  840. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  841. 信息:     Spilled Records=6
  842. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  843. 信息:     Map output bytes=666
  844. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  845. 信息:     Total committed heap usage (bytes)=1772093440
  846. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  847. 信息:     SPLIT_RAW_BYTES=130
  848. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  849. 信息:     Combine input records=0
  850. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  851. 信息:     Reduce input records=3
  852. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  853. 信息:     Reduce input groups=3
  854. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  855. 信息:     Combine output records=0
  856. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  857. 信息:     Reduce output records=3
  858. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log
  859. 信息:     Map output records=3
  860. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  861. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  862. 2013-10-14 15:39:42 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  863. 信息: Total input paths to process : 1
  864. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  865. 信息: Running job: job_local_0010
  866. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task initialize
  867. 信息:  Using ResourceCalculatorPlugin : null
  868. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  869. 信息: io.sort.mb = 100
  870. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  871. 信息: data buffer = 79691776/99614720
  872. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  873. 信息: record buffer = 262144/327680
  874. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  875. 信息: Starting flush of map output
  876. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  877. 信息: Finished spill 0
  878. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task done
  879. 信息: Task:attempt_local_0010_m_000000_0 is done. And is in the process of commiting
  880. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  881. 信息:
  882. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task sendDone
  883. 信息: Task 'attempt_local_0010_m_000000_0' done.
  884. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task initialize
  885. 信息:  Using ResourceCalculatorPlugin : null
  886. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  887. 信息:
  888. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Merger$MergeQueue merge
  889. 信息: Merging 1 sorted segments
  890. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Merger$MergeQueue merge
  891. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  892. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  893. 信息:
  894. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task done
  895. 信息: Task:attempt_local_0010_r_000000_0 is done. And is in the process of commiting
  896. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  897. 信息:
  898. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task commit
  899. 信息: Task attempt_local_0010_r_000000_0 is allowed to commit now
  900. 2013-10-14 15:39:42 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  901. 信息: Saved output of task 'attempt_local_0010_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-9
  902. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  903. 信息: reduce > reduce
  904. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task sendDone
  905. 信息: Task 'attempt_local_0010_r_000000_0' done.
  906. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  907. 信息:  map 100% reduce 100%
  908. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  909. 信息: Job complete: job_local_0010
  910. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  911. 信息: Counters: 19
  912. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  913. 信息:   File Output Format Counters
  914. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  915. 信息:     Bytes Written=695
  916. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  917. 信息:   FileSystemCounters
  918. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  919. 信息:     FILE_BYTES_READ=30544993
  920. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  921. 信息:     HDFS_BYTES_READ=741007
  922. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  923. 信息:     FILE_BYTES_WRITTEN=32013760
  924. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  925. 信息:     HDFS_BYTES_WRITTEN=154545
  926. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  927. 信息:   File Input Format Counters
  928. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  929. 信息:     Bytes Read=31390
  930. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  931. 信息:   Map-Reduce Framework
  932. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  933. 信息:     Map output materialized bytes=681
  934. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  935. 信息:     Map input records=1000
  936. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  937. 信息:     Reduce shuffle bytes=0
  938. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  939. 信息:     Spilled Records=6
  940. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  941. 信息:     Map output bytes=666
  942. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  943. 信息:     Total committed heap usage (bytes)=1966735360
  944. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  945. 信息:     SPLIT_RAW_BYTES=130
  946. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  947. 信息:     Combine input records=0
  948. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  949. 信息:     Reduce input records=3
  950. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  951. 信息:     Reduce input groups=3
  952. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  953. 信息:     Combine output records=0
  954. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  955. 信息:     Reduce output records=3
  956. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log
  957. 信息:     Map output records=3
  958. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  959. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  960. 2013-10-14 15:39:43 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  961. 信息: Total input paths to process : 1
  962. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  963. 信息: Running job: job_local_0011
  964. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task initialize
  965. 信息:  Using ResourceCalculatorPlugin : null
  966. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  967. 信息: io.sort.mb = 100
  968. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  969. 信息: data buffer = 79691776/99614720
  970. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  971. 信息: record buffer = 262144/327680
  972. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  973. 信息: Starting flush of map output
  974. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  975. 信息: Finished spill 0
  976. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task done
  977. 信息: Task:attempt_local_0011_m_000000_0 is done. And is in the process of commiting
  978. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  979. 信息:
  980. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task sendDone
  981. 信息: Task 'attempt_local_0011_m_000000_0' done.
  982. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task initialize
  983. 信息:  Using ResourceCalculatorPlugin : null
  984. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  985. 信息:
  986. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Merger$MergeQueue merge
  987. 信息: Merging 1 sorted segments
  988. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Merger$MergeQueue merge
  989. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes
  990. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  991. 信息:
  992. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task done
  993. 信息: Task:attempt_local_0011_r_000000_0 is done. And is in the process of commiting
  994. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  995. 信息:
  996. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task commit
  997. 信息: Task attempt_local_0011_r_000000_0 is allowed to commit now
  998. 2013-10-14 15:39:43 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  999. 信息: Saved output of task 'attempt_local_0011_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-10
  1000. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  1001. 信息: reduce > reduce
  1002. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task sendDone
  1003. 信息: Task 'attempt_local_0011_r_000000_0' done.
  1004. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  1005. 信息:  map 100% reduce 100%
  1006. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  1007. 信息: Job complete: job_local_0011
  1008. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1009. 信息: Counters: 19
  1010. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1011. 信息:   File Output Format Counters
  1012. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1013. 信息:     Bytes Written=695
  1014. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1015. 信息:   FileSystemCounters
  1016. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1017. 信息:     FILE_BYTES_READ=33833211
  1018. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1019. 信息:     HDFS_BYTES_READ=808345
  1020. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1021. 信息:     FILE_BYTES_WRITTEN=35458320
  1022. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1023. 信息:     HDFS_BYTES_WRITTEN=156323
  1024. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1025. 信息:   File Input Format Counters
  1026. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1027. 信息:     Bytes Read=31390
  1028. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1029. 信息:   Map-Reduce Framework
  1030. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1031. 信息:     Map output materialized bytes=681
  1032. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1033. 信息:     Map input records=1000
  1034. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1035. 信息:     Reduce shuffle bytes=0
  1036. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1037. 信息:     Spilled Records=6
  1038. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1039. 信息:     Map output bytes=666
  1040. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1041. 信息:     Total committed heap usage (bytes)=2166095872
  1042. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1043. 信息:     SPLIT_RAW_BYTES=130
  1044. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1045. 信息:     Combine input records=0
  1046. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1047. 信息:     Reduce input records=3
  1048. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1049. 信息:     Reduce input groups=3
  1050. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1051. 信息:     Combine output records=0
  1052. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1053. 信息:     Reduce output records=3
  1054. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log
  1055. 信息:     Map output records=3
  1056. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles
  1057. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  1058. 2013-10-14 15:39:44 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  1059. 信息: Total input paths to process : 1
  1060. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  1061. 信息: Running job: job_local_0012
  1062. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task initialize
  1063. 信息:  Using ResourceCalculatorPlugin : null
  1064. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task done
  1065. 信息: Task:attempt_local_0012_m_000000_0 is done. And is in the process of commiting
  1066. 2013-10-14 15:39:44 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  1067. 信息:
  1068. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task commit
  1069. 信息: Task attempt_local_0012_m_000000_0 is allowed to commit now
  1070. 2013-10-14 15:39:44 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  1071. 信息: Saved output of task 'attempt_local_0012_m_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints
  1072. 2013-10-14 15:39:44 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  1073. 信息:
  1074. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task sendDone
  1075. 信息: Task 'attempt_local_0012_m_000000_0' done.
  1076. 2013-10-14 15:39:45 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  1077. 信息:  map 100% reduce 0%
  1078. 2013-10-14 15:39:45 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  1079. 信息: Job complete: job_local_0012
  1080. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1081. 信息: Counters: 11
  1082. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1083. 信息:   File Output Format Counters
  1084. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1085. 信息:     Bytes Written=41520
  1086. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1087. 信息:   File Input Format Counters
  1088. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1089. 信息:     Bytes Read=31390
  1090. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1091. 信息:   FileSystemCounters
  1092. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1093. 信息:     FILE_BYTES_READ=18560374
  1094. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1095. 信息:     HDFS_BYTES_READ=437203
  1096. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1097. 信息:     FILE_BYTES_WRITTEN=19450325
  1098. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1099. 信息:     HDFS_BYTES_WRITTEN=120417
  1100. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1101. 信息:   Map-Reduce Framework
  1102. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1103. 信息:     Map input records=1000
  1104. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1105. 信息:     Spilled Records=0
  1106. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1107. 信息:     Total committed heap usage (bytes)=1083047936
  1108. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1109. 信息:     SPLIT_RAW_BYTES=130
  1110. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log
  1111. 信息:     Map output records=1000
  1112. Dumping out clusters from clusters: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-*-final and clusteredPoints: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints
  1113. CL-552{n=443 c=[1.631, -0.412] r=[1.563, 1.407]}
  1114. 	Weight : [props - optional]:  Point:
  1115. 	1.0: [-2.393, 3.347]
  1116. 	1.0: [-4.364, 1.905]
  1117. 	1.0: [-3.275, 0.023]
  1118. 	1.0: [-2.479, 2.534]
  1119. 	1.0: [-0.559, 1.223]
  1120. 	...
  1122. CL-847{n=77 c=[-2.953, -0.971] r=[1.767, 2.189]}
  1123. 	Weight : [props - optional]:  Point:
  1124. 	1.0: [-0.883, -3.320]
  1125. 	1.0: [-1.099, -6.063]
  1126. 	1.0: [-0.004, -0.610]
  1127. 	1.0: [-2.996, -3.610]
  1128. 	1.0: [3.988, 1.008]
  1129. 	...
  1131. CL-823{n=480 c=[0.219, 2.600] r=[1.479, 1.385]}
  1132. 	Weight : [props - optional]:  Point:
  1133. 	1.0: [2.670, 1.851]
  1134. 	1.0: [2.177, 6.773]
  1135. 	1.0: [5.537, 2.651]
  1136. 	1.0: [5.663, 6.868]
  1137. 	1.0: [5.117, 3.747]
  1138. 	1.0: [1.912, 2.959]
  1139. 	...

4). 聚类结果解读
我们可以把上面的日志分解析成3个部分解读

  • a. 初始化环境
  • b. 算法执行
  • c. 打印聚类结果

a. 初始化环境
出初HDFS的数据目录和工作目录，并上传数据文件。


  2. Delete: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  3. Create: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  4. copy from: datafile/randomData.csv to hdfs://192.168.1.210:9000/user/hdfs/mix_data
  5. ls: hdfs://192.168.1.210:9000/user/hdfs/mix_data
  6. ==========================================================
  7. name: hdfs://192.168.1.210:9000/user/hdfs/mix_data/randomData.csv, folder: false, size: 36655

b. 算法执行
算法执行，有3个步骤。

  • 1):把原始数据randomData.csv，转成Mahout sequence files of VectorWritable。
  • 2):通过随机的方法，选中kmeans的3个中心，做为初始集群
  • 3):根据迭代次数的设置，执行MapReduce，进行计算

1):把原始数据randomData.csv，转成Mahout sequence files of VectorWritable。

程序源代码:


  2.       InputDriver.runJob(new Path(inPath), new Path(seqFile), "org.apache.mahout.math.RandomAccessSparseVector");

日志输出：

  1. Job complete: job_local_0001

2):通过随机的方法，选中kmeans的3个中心，做为初始集群

程序源代码:


  2.         int k = 3;
  3.         Path seqFilePath = new Path(seqFile);
  4.         Path clustersSeeds = new Path(seeds);
  5.         DistanceMeasure measure = new EuclideanDistanceMeasure();
  6.         clustersSeeds = RandomSeedGenerator.buildRandom(conf, seqFilePath, clustersSeeds, k, measure);

日志输出：

  1. Job complete: job_local_0002

3):根据迭代次数的设置，执行MapReduce，进行计算
程序源代码:


  2.         KMeansDriver.run(conf, seqFilePath, clustersSeeds, new Path(outPath), measure, 0.01, 10, true, 0.01, false);

日志输出：


  2. Job complete: job_local_0003
  3. Job complete: job_local_0004
  4. Job complete: job_local_0005
  5. Job complete: job_local_0006
  6. Job complete: job_local_0007
  7. Job complete: job_local_0008
  8. Job complete: job_local_0009
  9. Job complete: job_local_0010
  10. Job complete: job_local_0011
  11. Job complete: job_local_0012

c. 打印聚类结果


  2. Dumping out clusters from clusters: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-*-final and clusteredPoints: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints
  3. CL-552{n=443 c=[1.631, -0.412] r=[1.563, 1.407]}
  4. CL-847{n=77 c=[-2.953, -0.971] r=[1.767, 2.189]}
  5. CL-823{n=480 c=[0.219, 2.600] r=[1.479, 1.385]}

运行结果：有3个中心。

  • Cluster1, 包括443个点，中心坐标[1.631, -0.412]
  • Cluster2, 包括77个点，中心坐标[-2.953, -0.971]
  • Cluster3, 包括480 个点，中心坐标[0.219, 2.600]

5). HDFS产生的目录


  2. # 根目录
  3. ~ hadoop fs -ls /user/hdfs/mix_data
  4. Found 4 items
  5. -rw-r--r--   3 Administrator supergroup      36655 2013-10-04 15:31 /user/hdfs/mix_data/randomData.csv
  6. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result
  7. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/seeds
  8. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/seqfile
  10. # 输出目录
  11. ~ hadoop fs -ls /user/hdfs/mix_data/result
  12. Found 13 items
  13. -rw-r--r--   3 Administrator supergroup        194 2013-10-04 15:31 /user/hdfs/mix_data/result/_policy
  14. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusteredPoints
  15. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-0
  16. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-1
  17. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-10-final
  18. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-2
  19. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-3
  20. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-4
  21. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-5
  22. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-6
  23. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-7
  24. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-8
  25. drwxr-xr-x   - Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-9
  27. # 产生的随机中心种子目录
  28. ~ hadoop fs -ls /user/hdfs/mix_data/seeds
  29. Found 1 items
  30. -rw-r--r--   3 Administrator supergroup        599 2013-10-04 15:31 /user/hdfs/mix_data/seeds/part-randomSeed
  32. # 输入文件换成Mahout格式文件的目录
  33. ~ hadoop fs -ls /user/hdfs/mix_data/seqfile
  34. Found 2 items
  35. -rw-r--r--   3 Administrator supergroup          0 2013-10-04 15:31 /user/hdfs/mix_data/seqfile/_SUCCESS
  36. -rw-r--r--   3 Administrator supergroup      31390 2013-10-04 15:31 /user/hdfs/mix_data/seqfile/part-m-00000

4. 用R语言可视化结果

分别把聚类后的点，保存到不同的cluster*.csv文件，然后用R语言画图。


  2. c1<-read.csv(file="cluster1.csv",sep=",",header=FALSE)
  3. c2<-read.csv(file="cluster2.csv",sep=",",header=FALSE)
  4. c3<-read.csv(file="cluster3.csv",sep=",",header=FALSE)
  5. y<-rbind(c1,c2,c3)
  6. cols<-c(rep(1,nrow(c1)),rep(2,nrow(c2)),rep(3,nrow(c3)))
  7. plot(y, col=c("black","blue","green")[cols])
  8. center<-matrix(c(1.631, -0.412,-2.953, -0.971,0.219, 2.600),ncol=2,byrow=TRUE)
  9. points(center, col="violetred", pch = 19)

从上图中，我们看到有 黑，蓝，绿，三种颜色的空心点，这些点就是原始数据。
3个紫色实点，是Mahout的kmeans后生成的3个中心。

对比文章中用R语言实现的kmeans的分类和中心，都不太一样。 用Maven构建Mahout项目

简单总结一下，在使用kmeans时，根据距离算法，阈值，出始中心，迭代次数的不同，kmeans计算的结果是不相同的。因此，用kmeans算法，我们一般只能得到一个模糊的分类标准，这个标准对于我们认识未知领域的数据集是很有帮助的。不能做为精确衡量数据的指标。

5. 模板项目上传github

https://github.com/bsspirit/maven_mahout_template/tree/mahout-0.8

大家可以下载这个项目，做为开发的起点。


  2. ~ git clone https://github.com/bsspirit/maven_mahout_template
  3. ~ git checkout mahout-0.8

这样，我们完成了Mahout的聚类算法Kmeans的分步式实现。接下来，我们会继续做关于Mahout中分类的实验!

转载请注明出处：
http://blog.fens.me/hadoop-mahout-kmeans/

转】Mahout分步式程序开发 聚类Kmeans的更多相关文章

  1. Mahout分步式程序开发 聚类Kmeans（转）

    Posted: Oct 14, 2013 Tags: clusterHadoopkmeansMahoutR聚类 Comments: 13 Comments Mahout分步式程序开发 聚类Kmeans ...

  2. 转】Mahout分步式程序开发 基于物品的协同过滤ItemCF

    原博文出自于: http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/ 感谢! Posted: Oct 14, 2013 Tags: Hadoopite ...

  3. Mahout分步式程序开发 基于物品的协同过滤ItemCF

    http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/ Hadoop家族系列文章,主要介绍Hadoop家族产品,常用的项目包括Hadoop, Hive, ...

  4. 插件式程序开发及其应用（C#）

    1.  定义 所谓“插件模型”,指应用程序由一些动态的独立模块构成,每个模块均具有一个或多个服务,并满足一定的插件协议,能够借助主程序实现主程序-插件:插件-插件之间的通讯. 应用该模型的系统,具有以 ...

  5. C# 插件式程序开发

    在网上找了下插件式编程的资料,这里自己先借鉴下别人的,同时发现有自己的看法,不过由于本人水平有限,不一定有参考价值,写出来一方面是为了总结自己,以求提高,另一方面也希望各为朋友看到我的不足,给我提出宝 ...

  6. 程序开发心理学阅读笔记——第II篇

    作为社会行为的软件开发程序开发组->程序开发团队->程序开发项目1.要判断程序员的某个集体是否构成一支团队,要看其中的成员以何种方式相互协作,以共同开发软件产品.2.健康的团队要始终能够保 ...

  7. mahout in Action2.2-聚类介绍-K-means聚类算法

    聚类介绍 本章包含 1 实战操作了解聚类 2.了解相似性概念 3 使用mahout执行一个简单的聚类实例 4.用于聚类的各种不同的距离測算方法 作为人类,我们倾向于与志同道合的人合作-"鸟的 ...

  8. 基于Flask的Web应用程序插件式结构开发

    事实上,很多应用程序基于插件式结构开发,可以很方便了扩展软件的功能,并且这些功能完全可以依托于第三方开发者,只要提供好接口和完备文档,比如wordpress.谷歌火狐浏览器等. Python这样的动态 ...

  9. 基于Zookeeper的分步式队列系统集成案例

    基于Zookeeper的分步式队列系统集成案例 Hadoop家族系列文章,主要介绍Hadoop家族产品,常用的项目包括Hadoop, Hive, Pig, HBase, Sqoop, Mahout, ...

随机推荐

  1. 等额本息Vs等额本金

    1:贷款种类一旦选择不能改变.2:你提前还款的全部属于本金部分,若能一次性归还本金只需付清当月月息即可[不按年利率计算而是月利率],与你归还的本金违约金[设:提前还款10万*X.XXX%=违约金,具体 ...

  2. Hadoop集群（第8期）_HDFS初探之旅

    1.HDFS简介 HDFS(Hadoop Distributed File System)是Hadoop项目的核心子项目,是分布式计算中数据存储管理的基础,是基于流数据模式访问和处理超大文件的需求而开 ...

  3. Java NIO读书笔记2

    一.选择器(Selector) Selector(选择器)是Java NIO中能够检测一到多个NIO通道,并能够知晓通道是否为诸如读写事件做好准备的组件.这样,一个单独的线程可以管理多个channel ...

  4. 浅析JavaScript引用类型之--Object、Array

    1.Object类型 对象是某个特定引用类型的实例,新对象有两种创建方式: i.使用new操作符调用构造函数来创建. var person = new Object(); person.name = ...

  5. jquery ajax 开发手记

    1.json解析的格式要求更严格了,必须全部加引号,否则无法识别 {"result":"false"} 2.ashx如果要使用Session需要继承接口IReq ...

  6. design pattern及其使用

    什么是设计模式? design pattern是一个通用的,可以被重用的关于一个常见的问题的解决方案. 为什么要用设计模式? 引入设计模式的理论基础非常简单.我们每天都会碰到问题.我们可能碰到决定使用 ...

  7. SQL group by分组查询（转）

    本文导读:在实际SQL应用中,经常需要进行分组聚合,即将查询对象按一定条件分组,然后对每一个组进行聚合分析.创建分组是通过GROUP BY子句实现的.与WHERE子句不同,GROUP BY子句用于归纳 ...

  8. python练习程序（c100经典例12）

    题目: 判断101-200之间有多少个素数,并输出所有素数. for i in range(101,201): flag=0; for j in range(2,int(i**(1.0/2))): i ...

  9. setTimeout/setInterval

    //使用 setTimeout 时需注意,当该代码执行时,JS 会立即编译函数第一个参数“code” //所以该函数的第一个参数应该为:需要编译的代码.或者一个函数 //例1:setTimeout(& ...

  10. 【Android】跟着教程做の学习笔记

    教程 + <第一行代码 - Android> //尽量在十二月底学完吧(同步学习java基础)