http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/

Hadoop家族系列文章，主要介绍Hadoop家族产品，常用的项目包括Hadoop, Hive, Pig, HBase, Sqoop, Mahout, Zookeeper, Avro, Ambari, Chukwa，新增加的项目包括，YARN, Hcatalog, Oozie, Cassandra, Hama, Whirr, Flume, Bigtop, Crunch, Hue等。

从2011年开始，中国进入大数据风起云涌的时代，以Hadoop为代表的家族软件，占据了大数据处理的广阔地盘。开源界及厂商，所有数据软件，无一不向Hadoop靠拢。Hadoop也从小众的高富帅领域，变成了大数据开发的标准。在Hadoop原有技术基础之上，出现了Hadoop家族产品，通过“大数据”概念不断创新，推出科技进步。

作为IT界的开发人员，我们也要跟上节奏，抓住机遇，跟着Hadoop一起雄起！

关于作者：

  • 张丹(Conan), 程序员Java,R,PHP,Javascript
  • weibo：@Conan_Z
  • blog: http://blog.fens.me
  • email: bsspirit@gmail.com

转载请注明出处：
http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/

前言

Mahout是Hadoop家族一员，从血缘就继承了Hadoop程序的特点，支持HDFS访问和MapReduce分步式算法。随着Mahout的发展，从0.7版本开始，Mahout做了重大的升级。移除了部分算法的单机内存计算，只支持基于Hadoop的MapReduce平行计算。

从这点上，我们能看出Mahout走向大数据，坚持并行化的决心！相信在Hadoop的大框架下，Mahout最终能成为一个大数据的明星产品！

目录

  1. Mahout开发环境介绍
  2. Mahout基于Hadoop的分步环境介绍
  3. 用Mahout实现协同过滤ItemCF
  4. 模板项目上传github

1. Mahout开发环境介绍

在 用Maven构建Mahout项目 文章中，我们已经配置好了基于Maven的Mahout的开发环境，我们将继续完成Mahout的分步式的程序开发。

本文的mahout版本为0.8。

开发环境：

  • Win7 64bit
  • Java 1.6.0_45
  • Maven 3
  • Eclipse Juno Service Release 2
  • Mahout 0.8
  • Hadoop 1.1.2

找到pom.xml，修改mahout版本为0.8

  1. <mahout.version>0.8</mahout.version>

然后，下载依赖库。

  1. ~ mvn clean install

由于 org.conan.mymahout.cluster06.Kmeans.java 类代码，是基于mahout-0.6的，所以会报错。我们可以先注释这个文件。

2. Mahout基于Hadoop的分步环境介绍

如上图所示，我们可以选择在win7中开发，也可以在linux中开发，开发过程我们可以在本地环境进行调试，标配的工具都是Maven和Eclipse。

Mahout在运行过程中，会把MapReduce的算法程序包，自动发布的Hadoop的集群环境中，这种开发和运行模式，就和真正的生产环境差不多了。

3. 用Mahout实现协同过滤ItemCF

实现步骤:

  • 1. 准备数据文件: item.csv
  • 2. Java程序：HdfsDAO.java
  • 3. Java程序：ItemCFHadoop.java
  • 4. 运行程序
  • 5. 推荐结果解读

1). 准备数据文件: item.csv
上传测试数据到HDFS，单机内存实验请参考文章：用Maven构建Mahout项目


  2. ~ hadoop fs -mkdir /user/hdfs/userCF
  3. ~ hadoop fs -copyFromLocal /home/conan/datafiles/item.csv /user/hdfs/userCF
  5. ~ hadoop fs -cat /user/hdfs/userCF/item.csv
  6. 1,101,5.0
  7. 1,102,3.0
  8. 1,103,2.5
  9. 2,101,2.0
  10. 2,102,2.5
  11. 2,103,5.0
  12. 2,104,2.0
  13. 3,101,2.5
  14. 3,104,4.0
  15. 3,105,4.5
  16. 3,107,5.0
  17. 4,101,5.0
  18. 4,103,3.0
  19. 4,104,4.5
  20. 4,106,4.0
  21. 5,101,4.0
  22. 5,102,3.0
  23. 5,103,2.0
  24. 5,104,4.0
  25. 5,105,3.5
  26. 5,106,4.0

2). Java程序：HdfsDAO.java
HdfsDAO.java，是一个HDFS操作的工具，用API实现Hadoop的各种HDFS命令，请参考文章：Hadoop编程调用HDFS

我们这里会用到HdfsDAO.java类中的一些方法：


  2.         HdfsDAO hdfs = new HdfsDAO(HDFS, conf);
  3.         hdfs.rmr(inPath);
  4.         hdfs.mkdirs(inPath);
  5.         hdfs.copyFile(localFile, inPath);
  6.         hdfs.ls(inPath);
  7.         hdfs.cat(inFile);

3). Java程序：ItemCFHadoop.java
用Mahout实现分步式算法，我们看到Mahout in Action中的解释。

实现程序：


  2. package org.conan.mymahout.recommendation;
  4. import org.apache.hadoop.mapred.JobConf;
  5. import org.apache.mahout.cf.taste.hadoop.item.RecommenderJob;
  6. import org.conan.mymahout.hdfs.HdfsDAO;
  8. public class ItemCFHadoop {
  10.     private static final String HDFS = "hdfs://192.168.1.210:9000";
  12.     public static void main(String[] args) throws Exception {
  13.         String localFile = "datafile/item.csv";
  14.         String inPath = HDFS + "/user/hdfs/userCF";
  15.         String inFile = inPath + "/item.csv";
  16.         String outPath = HDFS + "/user/hdfs/userCF/result/";
  17.         String outFile = outPath + "/part-r-00000";
  18.         String tmpPath = HDFS + "/tmp/" + System.currentTimeMillis();
  20.         JobConf conf = config();
  21.         HdfsDAO hdfs = new HdfsDAO(HDFS, conf);
  22.         hdfs.rmr(inPath);
  23.         hdfs.mkdirs(inPath);
  24.         hdfs.copyFile(localFile, inPath);
  25.         hdfs.ls(inPath);
  26.         hdfs.cat(inFile);
  28.         StringBuilder sb = new StringBuilder();
  29.         sb.append("--input ").append(inPath);
  30.         sb.append(" --output ").append(outPath);
  31.         sb.append(" --booleanData true");
  32.         sb.append(" --similarityClassname org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.EuclideanDistanceSimilarity");
  33.         sb.append(" --tempDir ").append(tmpPath);
  34.         args = sb.toString().split(" ");
  36.         RecommenderJob job = new RecommenderJob();
  37.         job.setConf(conf);
  38.         job.run(args);
  40.         hdfs.cat(outFile);
  41.     }
  43.     public static JobConf config() {
  44.         JobConf conf = new JobConf(ItemCFHadoop.class);
  45.         conf.setJobName("ItemCFHadoop");
  46.         conf.addResource("classpath:/hadoop/core-site.xml");
  47.         conf.addResource("classpath:/hadoop/hdfs-site.xml");
  48.         conf.addResource("classpath:/hadoop/mapred-site.xml");
  49.         return conf;
  50.     }
  51. }

RecommenderJob.java，实际上就是封装了，上面整个图的分步式并行算法的执行过程！如果没有这层封装，我们需要自己去实现图中8个步骤MapReduce算法。

关于上面算法的深度剖析，请参考文章：R实现MapReduce的协同过滤算法

4). 运行程序
控制台输出：


  2. Delete: hdfs://192.168.1.210:9000/user/hdfs/userCF
  3. Create: hdfs://192.168.1.210:9000/user/hdfs/userCF
  4. copy from: datafile/item.csv to hdfs://192.168.1.210:9000/user/hdfs/userCF
  5. ls: hdfs://192.168.1.210:9000/user/hdfs/userCF
  6. ==========================================================
  7. name: hdfs://192.168.1.210:9000/user/hdfs/userCF/item.csv, folder: false, size: 229
  8. ==========================================================
  9. cat: hdfs://192.168.1.210:9000/user/hdfs/userCF/item.csv
  10. 1,101,5.0
  11. 1,102,3.0
  12. 1,103,2.5
  13. 2,101,2.0
  14. 2,102,2.5
  15. 2,103,5.0
  16. 2,104,2.0
  17. 3,101,2.5
  18. 3,104,4.0
  19. 3,105,4.5
  20. 3,107,5.0
  21. 4,101,5.0
  22. 4,103,3.0
  23. 4,104,4.5
  24. 4,106,4.0
  25. 5,101,4.0
  26. 5,102,3.0
  27. 5,103,2.0
  28. 5,104,4.0
  29. 5,105,3.5
  30. 5,106,4.0SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  31. SLF4J: Defaulting to no-operation (NOP) logger implementation
  32. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
  33. 2013-10-14 10:26:35 org.apache.hadoop.util.NativeCodeLoader
  34. 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  35. 2013-10-14 10:26:35 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  36. 信息: Total input paths to process : 1
  37. 2013-10-14 10:26:35 org.apache.hadoop.io.compress.snappy.LoadSnappy
  38. 警告: Snappy native library not loaded
  39. 2013-10-14 10:26:36 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  40. 信息: Running job: job_local_0001
  41. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task initialize
  42. 信息:  Using ResourceCalculatorPlugin : null
  43. 2013-10-14 10:26:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  44. 信息: io.sort.mb = 100
  45. 2013-10-14 10:26:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  46. 信息: data buffer = 79691776/99614720
  47. 2013-10-14 10:26:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  48. 信息: record buffer = 262144/327680
  49. 2013-10-14 10:26:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  50. 信息: Starting flush of map output
  51. 2013-10-14 10:26:36 org.apache.hadoop.io.compress.CodecPool getCompressor
  52. 信息: Got brand-new compressor
  53. 2013-10-14 10:26:36 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  54. 信息: Finished spill 0
  55. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task done
  56. 信息: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
  57. 2013-10-14 10:26:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  58. 信息:
  59. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task sendDone
  60. 信息: Task 'attempt_local_0001_m_000000_0' done.
  61. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task initialize
  62. 信息:  Using ResourceCalculatorPlugin : null
  63. 2013-10-14 10:26:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  64. 信息:
  65. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Merger$MergeQueue merge
  66. 信息: Merging 1 sorted segments
  67. 2013-10-14 10:26:36 org.apache.hadoop.io.compress.CodecPool getDecompressor
  68. 信息: Got brand-new decompressor
  69. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Merger$MergeQueue merge
  70. 信息: Down to the last merge-pass, with 1 segments left of total size: 42 bytes
  71. 2013-10-14 10:26:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  72. 信息:
  73. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task done
  74. 信息: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
  75. 2013-10-14 10:26:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  76. 信息:
  77. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task commit
  78. 信息: Task attempt_local_0001_r_000000_0 is allowed to commit now
  79. 2013-10-14 10:26:36 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  80. 信息: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/preparePreferenceMatrix/itemIDIndex
  81. 2013-10-14 10:26:36 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  82. 信息: reduce > reduce
  83. 2013-10-14 10:26:36 org.apache.hadoop.mapred.Task sendDone
  84. 信息: Task 'attempt_local_0001_r_000000_0' done.
  85. 2013-10-14 10:26:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  86. 信息:  map 100% reduce 100%
  87. 2013-10-14 10:26:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  88. 信息: Job complete: job_local_0001
  89. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  90. 信息: Counters: 19
  91. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  92. 信息:   File Output Format Counters
  93. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  94. 信息:     Bytes Written=187
  95. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  96. 信息:   FileSystemCounters
  97. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  98. 信息:     FILE_BYTES_READ=3287330
  99. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  100. 信息:     HDFS_BYTES_READ=916
  101. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  102. 信息:     FILE_BYTES_WRITTEN=3443292
  103. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  104. 信息:     HDFS_BYTES_WRITTEN=645
  105. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  106. 信息:   File Input Format Counters
  107. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  108. 信息:     Bytes Read=229
  109. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  110. 信息:   Map-Reduce Framework
  111. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  112. 信息:     Map output materialized bytes=46
  113. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  114. 信息:     Map input records=21
  115. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  116. 信息:     Reduce shuffle bytes=0
  117. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  118. 信息:     Spilled Records=14
  119. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  120. 信息:     Map output bytes=84
  121. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  122. 信息:     Total committed heap usage (bytes)=376569856
  123. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  124. 信息:     SPLIT_RAW_BYTES=116
  125. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  126. 信息:     Combine input records=21
  127. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  128. 信息:     Reduce input records=7
  129. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  130. 信息:     Reduce input groups=7
  131. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  132. 信息:     Combine output records=7
  133. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  134. 信息:     Reduce output records=7
  135. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Counters log
  136. 信息:     Map output records=21
  137. 2013-10-14 10:26:37 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  138. 信息: Total input paths to process : 1
  139. 2013-10-14 10:26:37 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  140. 信息: Running job: job_local_0002
  141. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task initialize
  142. 信息:  Using ResourceCalculatorPlugin : null
  143. 2013-10-14 10:26:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  144. 信息: io.sort.mb = 100
  145. 2013-10-14 10:26:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  146. 信息: data buffer = 79691776/99614720
  147. 2013-10-14 10:26:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  148. 信息: record buffer = 262144/327680
  149. 2013-10-14 10:26:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  150. 信息: Starting flush of map output
  151. 2013-10-14 10:26:37 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  152. 信息: Finished spill 0
  153. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task done
  154. 信息: Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
  155. 2013-10-14 10:26:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  156. 信息:
  157. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task sendDone
  158. 信息: Task 'attempt_local_0002_m_000000_0' done.
  159. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task initialize
  160. 信息:  Using ResourceCalculatorPlugin : null
  161. 2013-10-14 10:26:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  162. 信息:
  163. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Merger$MergeQueue merge
  164. 信息: Merging 1 sorted segments
  165. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Merger$MergeQueue merge
  166. 信息: Down to the last merge-pass, with 1 segments left of total size: 68 bytes
  167. 2013-10-14 10:26:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  168. 信息:
  169. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task done
  170. 信息: Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
  171. 2013-10-14 10:26:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  172. 信息:
  173. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task commit
  174. 信息: Task attempt_local_0002_r_000000_0 is allowed to commit now
  175. 2013-10-14 10:26:37 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  176. 信息: Saved output of task 'attempt_local_0002_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/preparePreferenceMatrix/userVectors
  177. 2013-10-14 10:26:37 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  178. 信息: reduce > reduce
  179. 2013-10-14 10:26:37 org.apache.hadoop.mapred.Task sendDone
  180. 信息: Task 'attempt_local_0002_r_000000_0' done.
  181. 2013-10-14 10:26:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  182. 信息:  map 100% reduce 100%
  183. 2013-10-14 10:26:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  184. 信息: Job complete: job_local_0002
  185. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  186. 信息: Counters: 20
  187. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  188. 信息:   org.apache.mahout.cf.taste.hadoop.item.ToUserVectorsReducer$Counters
  189. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  190. 信息:     USERS=5
  191. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  192. 信息:   File Output Format Counters
  193. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  194. 信息:     Bytes Written=288
  195. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  196. 信息:   FileSystemCounters
  197. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  198. 信息:     FILE_BYTES_READ=6574274
  199. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  200. 信息:     HDFS_BYTES_READ=1374
  201. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  202. 信息:     FILE_BYTES_WRITTEN=6887592
  203. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  204. 信息:     HDFS_BYTES_WRITTEN=1120
  205. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  206. 信息:   File Input Format Counters
  207. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  208. 信息:     Bytes Read=229
  209. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  210. 信息:   Map-Reduce Framework
  211. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  212. 信息:     Map output materialized bytes=72
  213. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  214. 信息:     Map input records=21
  215. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  216. 信息:     Reduce shuffle bytes=0
  217. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  218. 信息:     Spilled Records=42
  219. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  220. 信息:     Map output bytes=63
  221. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  222. 信息:     Total committed heap usage (bytes)=575930368
  223. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  224. 信息:     SPLIT_RAW_BYTES=116
  225. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  226. 信息:     Combine input records=0
  227. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  228. 信息:     Reduce input records=21
  229. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  230. 信息:     Reduce input groups=5
  231. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  232. 信息:     Combine output records=0
  233. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  234. 信息:     Reduce output records=5
  235. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Counters log
  236. 信息:     Map output records=21
  237. 2013-10-14 10:26:38 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  238. 信息: Total input paths to process : 1
  239. 2013-10-14 10:26:38 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  240. 信息: Running job: job_local_0003
  241. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task initialize
  242. 信息:  Using ResourceCalculatorPlugin : null
  243. 2013-10-14 10:26:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  244. 信息: io.sort.mb = 100
  245. 2013-10-14 10:26:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  246. 信息: data buffer = 79691776/99614720
  247. 2013-10-14 10:26:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  248. 信息: record buffer = 262144/327680
  249. 2013-10-14 10:26:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  250. 信息: Starting flush of map output
  251. 2013-10-14 10:26:38 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  252. 信息: Finished spill 0
  253. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task done
  254. 信息: Task:attempt_local_0003_m_000000_0 is done. And is in the process of commiting
  255. 2013-10-14 10:26:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  256. 信息:
  257. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task sendDone
  258. 信息: Task 'attempt_local_0003_m_000000_0' done.
  259. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task initialize
  260. 信息:  Using ResourceCalculatorPlugin : null
  261. 2013-10-14 10:26:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  262. 信息:
  263. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Merger$MergeQueue merge
  264. 信息: Merging 1 sorted segments
  265. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Merger$MergeQueue merge
  266. 信息: Down to the last merge-pass, with 1 segments left of total size: 89 bytes
  267. 2013-10-14 10:26:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  268. 信息:
  269. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task done
  270. 信息: Task:attempt_local_0003_r_000000_0 is done. And is in the process of commiting
  271. 2013-10-14 10:26:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  272. 信息:
  273. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task commit
  274. 信息: Task attempt_local_0003_r_000000_0 is allowed to commit now
  275. 2013-10-14 10:26:38 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  276. 信息: Saved output of task 'attempt_local_0003_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/preparePreferenceMatrix/ratingMatrix
  277. 2013-10-14 10:26:38 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  278. 信息: reduce > reduce
  279. 2013-10-14 10:26:38 org.apache.hadoop.mapred.Task sendDone
  280. 信息: Task 'attempt_local_0003_r_000000_0' done.
  281. 2013-10-14 10:26:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  282. 信息:  map 100% reduce 100%
  283. 2013-10-14 10:26:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  284. 信息: Job complete: job_local_0003
  285. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  286. 信息: Counters: 21
  287. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  288. 信息:   File Output Format Counters
  289. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  290. 信息:     Bytes Written=335
  291. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  292. 信息:   org.apache.mahout.cf.taste.hadoop.preparation.ToItemVectorsMapper$Elements
  293. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  294. 信息:     USER_RATINGS_NEGLECTED=0
  295. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  296. 信息:     USER_RATINGS_USED=21
  297. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  298. 信息:   FileSystemCounters
  299. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  300. 信息:     FILE_BYTES_READ=9861349
  301. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  302. 信息:     HDFS_BYTES_READ=1950
  303. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  304. 信息:     FILE_BYTES_WRITTEN=10331958
  305. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  306. 信息:     HDFS_BYTES_WRITTEN=1751
  307. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  308. 信息:   File Input Format Counters
  309. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  310. 信息:     Bytes Read=288
  311. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  312. 信息:   Map-Reduce Framework
  313. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  314. 信息:     Map output materialized bytes=93
  315. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  316. 信息:     Map input records=5
  317. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  318. 信息:     Reduce shuffle bytes=0
  319. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  320. 信息:     Spilled Records=14
  321. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  322. 信息:     Map output bytes=336
  323. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  324. 信息:     Total committed heap usage (bytes)=775290880
  325. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  326. 信息:     SPLIT_RAW_BYTES=157
  327. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  328. 信息:     Combine input records=21
  329. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  330. 信息:     Reduce input records=7
  331. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  332. 信息:     Reduce input groups=7
  333. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  334. 信息:     Combine output records=7
  335. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  336. 信息:     Reduce output records=7
  337. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Counters log
  338. 信息:     Map output records=21
  339. 2013-10-14 10:26:39 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  340. 信息: Total input paths to process : 1
  341. 2013-10-14 10:26:39 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  342. 信息: Running job: job_local_0004
  343. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task initialize
  344. 信息:  Using ResourceCalculatorPlugin : null
  345. 2013-10-14 10:26:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  346. 信息: io.sort.mb = 100
  347. 2013-10-14 10:26:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  348. 信息: data buffer = 79691776/99614720
  349. 2013-10-14 10:26:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  350. 信息: record buffer = 262144/327680
  351. 2013-10-14 10:26:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  352. 信息: Starting flush of map output
  353. 2013-10-14 10:26:39 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  354. 信息: Finished spill 0
  355. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task done
  356. 信息: Task:attempt_local_0004_m_000000_0 is done. And is in the process of commiting
  357. 2013-10-14 10:26:39 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  358. 信息:
  359. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task sendDone
  360. 信息: Task 'attempt_local_0004_m_000000_0' done.
  361. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task initialize
  362. 信息:  Using ResourceCalculatorPlugin : null
  363. 2013-10-14 10:26:39 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  364. 信息:
  365. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Merger$MergeQueue merge
  366. 信息: Merging 1 sorted segments
  367. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Merger$MergeQueue merge
  368. 信息: Down to the last merge-pass, with 1 segments left of total size: 118 bytes
  369. 2013-10-14 10:26:39 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  370. 信息:
  371. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task done
  372. 信息: Task:attempt_local_0004_r_000000_0 is done. And is in the process of commiting
  373. 2013-10-14 10:26:39 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  374. 信息:
  375. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task commit
  376. 信息: Task attempt_local_0004_r_000000_0 is allowed to commit now
  377. 2013-10-14 10:26:39 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  378. 信息: Saved output of task 'attempt_local_0004_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/weights
  379. 2013-10-14 10:26:39 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  380. 信息: reduce > reduce
  381. 2013-10-14 10:26:39 org.apache.hadoop.mapred.Task sendDone
  382. 信息: Task 'attempt_local_0004_r_000000_0' done.
  383. 2013-10-14 10:26:40 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  384. 信息:  map 100% reduce 100%
  385. 2013-10-14 10:26:40 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  386. 信息: Job complete: job_local_0004
  387. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  388. 信息: Counters: 20
  389. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  390. 信息:   File Output Format Counters
  391. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  392. 信息:     Bytes Written=381
  393. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  394. 信息:   FileSystemCounters
  395. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  396. 信息:     FILE_BYTES_READ=13148476
  397. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  398. 信息:     HDFS_BYTES_READ=2628
  399. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  400. 信息:     FILE_BYTES_WRITTEN=13780408
  401. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  402. 信息:     HDFS_BYTES_WRITTEN=2551
  403. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  404. 信息:   File Input Format Counters
  405. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  406. 信息:     Bytes Read=335
  407. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  408. 信息:   org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob$Counters
  409. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  410. 信息:     ROWS=7
  411. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  412. 信息:   Map-Reduce Framework
  413. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  414. 信息:     Map output materialized bytes=122
  415. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  416. 信息:     Map input records=7
  417. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  418. 信息:     Reduce shuffle bytes=0
  419. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  420. 信息:     Spilled Records=16
  421. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  422. 信息:     Map output bytes=516
  423. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  424. 信息:     Total committed heap usage (bytes)=974651392
  425. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  426. 信息:     SPLIT_RAW_BYTES=158
  427. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  428. 信息:     Combine input records=24
  429. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  430. 信息:     Reduce input records=8
  431. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  432. 信息:     Reduce input groups=8
  433. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  434. 信息:     Combine output records=8
  435. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  436. 信息:     Reduce output records=5
  437. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Counters log
  438. 信息:     Map output records=24
  439. 2013-10-14 10:26:40 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  440. 信息: Total input paths to process : 1
  441. 2013-10-14 10:26:40 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  442. 信息: Running job: job_local_0005
  443. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task initialize
  444. 信息:  Using ResourceCalculatorPlugin : null
  445. 2013-10-14 10:26:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  446. 信息: io.sort.mb = 100
  447. 2013-10-14 10:26:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  448. 信息: data buffer = 79691776/99614720
  449. 2013-10-14 10:26:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  450. 信息: record buffer = 262144/327680
  451. 2013-10-14 10:26:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  452. 信息: Starting flush of map output
  453. 2013-10-14 10:26:40 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  454. 信息: Finished spill 0
  455. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task done
  456. 信息: Task:attempt_local_0005_m_000000_0 is done. And is in the process of commiting
  457. 2013-10-14 10:26:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  458. 信息:
  459. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task sendDone
  460. 信息: Task 'attempt_local_0005_m_000000_0' done.
  461. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task initialize
  462. 信息:  Using ResourceCalculatorPlugin : null
  463. 2013-10-14 10:26:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  464. 信息:
  465. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Merger$MergeQueue merge
  466. 信息: Merging 1 sorted segments
  467. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Merger$MergeQueue merge
  468. 信息: Down to the last merge-pass, with 1 segments left of total size: 121 bytes
  469. 2013-10-14 10:26:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  470. 信息:
  471. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task done
  472. 信息: Task:attempt_local_0005_r_000000_0 is done. And is in the process of commiting
  473. 2013-10-14 10:26:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  474. 信息:
  475. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task commit
  476. 信息: Task attempt_local_0005_r_000000_0 is allowed to commit now
  477. 2013-10-14 10:26:40 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  478. 信息: Saved output of task 'attempt_local_0005_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/pairwiseSimilarity
  479. 2013-10-14 10:26:40 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  480. 信息: reduce > reduce
  481. 2013-10-14 10:26:40 org.apache.hadoop.mapred.Task sendDone
  482. 信息: Task 'attempt_local_0005_r_000000_0' done.
  483. 2013-10-14 10:26:41 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  484. 信息:  map 100% reduce 100%
  485. 2013-10-14 10:26:41 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  486. 信息: Job complete: job_local_0005
  487. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  488. 信息: Counters: 21
  489. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  490. 信息:   File Output Format Counters
  491. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  492. 信息:     Bytes Written=392
  493. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  494. 信息:   FileSystemCounters
  495. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  496. 信息:     FILE_BYTES_READ=16435577
  497. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  498. 信息:     HDFS_BYTES_READ=3488
  499. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  500. 信息:     FILE_BYTES_WRITTEN=17230010
  501. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  502. 信息:     HDFS_BYTES_WRITTEN=3408
  503. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  504. 信息:   File Input Format Counters
  505. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  506. 信息:     Bytes Read=381
  507. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  508. 信息:   org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob$Counters
  509. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  510. 信息:     PRUNED_COOCCURRENCES=0
  511. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  512. 信息:     COOCCURRENCES=57
  513. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  514. 信息:   Map-Reduce Framework
  515. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  516. 信息:     Map output materialized bytes=125
  517. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  518. 信息:     Map input records=5
  519. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  520. 信息:     Reduce shuffle bytes=0
  521. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  522. 信息:     Spilled Records=14
  523. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  524. 信息:     Map output bytes=744
  525. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  526. 信息:     Total committed heap usage (bytes)=1174011904
  527. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  528. 信息:     SPLIT_RAW_BYTES=129
  529. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  530. 信息:     Combine input records=21
  531. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  532. 信息:     Reduce input records=7
  533. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  534. 信息:     Reduce input groups=7
  535. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  536. 信息:     Combine output records=7
  537. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  538. 信息:     Reduce output records=7
  539. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Counters log
  540. 信息:     Map output records=21
  541. 2013-10-14 10:26:41 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  542. 信息: Total input paths to process : 1
  543. 2013-10-14 10:26:41 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  544. 信息: Running job: job_local_0006
  545. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task initialize
  546. 信息:  Using ResourceCalculatorPlugin : null
  547. 2013-10-14 10:26:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  548. 信息: io.sort.mb = 100
  549. 2013-10-14 10:26:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  550. 信息: data buffer = 79691776/99614720
  551. 2013-10-14 10:26:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  552. 信息: record buffer = 262144/327680
  553. 2013-10-14 10:26:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  554. 信息: Starting flush of map output
  555. 2013-10-14 10:26:41 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  556. 信息: Finished spill 0
  557. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task done
  558. 信息: Task:attempt_local_0006_m_000000_0 is done. And is in the process of commiting
  559. 2013-10-14 10:26:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  560. 信息:
  561. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task sendDone
  562. 信息: Task 'attempt_local_0006_m_000000_0' done.
  563. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task initialize
  564. 信息:  Using ResourceCalculatorPlugin : null
  565. 2013-10-14 10:26:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  566. 信息:
  567. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Merger$MergeQueue merge
  568. 信息: Merging 1 sorted segments
  569. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Merger$MergeQueue merge
  570. 信息: Down to the last merge-pass, with 1 segments left of total size: 158 bytes
  571. 2013-10-14 10:26:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  572. 信息:
  573. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task done
  574. 信息: Task:attempt_local_0006_r_000000_0 is done. And is in the process of commiting
  575. 2013-10-14 10:26:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  576. 信息:
  577. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task commit
  578. 信息: Task attempt_local_0006_r_000000_0 is allowed to commit now
  579. 2013-10-14 10:26:41 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  580. 信息: Saved output of task 'attempt_local_0006_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/similarityMatrix
  581. 2013-10-14 10:26:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  582. 信息: reduce > reduce
  583. 2013-10-14 10:26:41 org.apache.hadoop.mapred.Task sendDone
  584. 信息: Task 'attempt_local_0006_r_000000_0' done.
  585. 2013-10-14 10:26:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  586. 信息:  map 100% reduce 100%
  587. 2013-10-14 10:26:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  588. 信息: Job complete: job_local_0006
  589. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  590. 信息: Counters: 19
  591. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  592. 信息:   File Output Format Counters
  593. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  594. 信息:     Bytes Written=554
  595. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  596. 信息:   FileSystemCounters
  597. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  598. 信息:     FILE_BYTES_READ=19722740
  599. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  600. 信息:     HDFS_BYTES_READ=4342
  601. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  602. 信息:     FILE_BYTES_WRITTEN=20674772
  603. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  604. 信息:     HDFS_BYTES_WRITTEN=4354
  605. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  606. 信息:   File Input Format Counters
  607. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  608. 信息:     Bytes Read=392
  609. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  610. 信息:   Map-Reduce Framework
  611. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  612. 信息:     Map output materialized bytes=162
  613. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  614. 信息:     Map input records=7
  615. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  616. 信息:     Reduce shuffle bytes=0
  617. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  618. 信息:     Spilled Records=14
  619. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  620. 信息:     Map output bytes=599
  621. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  622. 信息:     Total committed heap usage (bytes)=1373372416
  623. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  624. 信息:     SPLIT_RAW_BYTES=140
  625. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  626. 信息:     Combine input records=25
  627. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  628. 信息:     Reduce input records=7
  629. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  630. 信息:     Reduce input groups=7
  631. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  632. 信息:     Combine output records=7
  633. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  634. 信息:     Reduce output records=7
  635. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Counters log
  636. 信息:     Map output records=25
  637. 2013-10-14 10:26:42 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  638. 信息: Total input paths to process : 1
  639. 2013-10-14 10:26:42 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  640. 信息: Total input paths to process : 1
  641. 2013-10-14 10:26:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  642. 信息: Running job: job_local_0007
  643. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task initialize
  644. 信息:  Using ResourceCalculatorPlugin : null
  645. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  646. 信息: io.sort.mb = 100
  647. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  648. 信息: data buffer = 79691776/99614720
  649. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  650. 信息: record buffer = 262144/327680
  651. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  652. 信息: Starting flush of map output
  653. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  654. 信息: Finished spill 0
  655. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task done
  656. 信息: Task:attempt_local_0007_m_000000_0 is done. And is in the process of commiting
  657. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  658. 信息:
  659. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task sendDone
  660. 信息: Task 'attempt_local_0007_m_000000_0' done.
  661. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task initialize
  662. 信息:  Using ResourceCalculatorPlugin : null
  663. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  664. 信息: io.sort.mb = 100
  665. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  666. 信息: data buffer = 79691776/99614720
  667. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  668. 信息: record buffer = 262144/327680
  669. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  670. 信息: Starting flush of map output
  671. 2013-10-14 10:26:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  672. 信息: Finished spill 0
  673. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task done
  674. 信息: Task:attempt_local_0007_m_000001_0 is done. And is in the process of commiting
  675. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  676. 信息:
  677. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task sendDone
  678. 信息: Task 'attempt_local_0007_m_000001_0' done.
  679. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task initialize
  680. 信息:  Using ResourceCalculatorPlugin : null
  681. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  682. 信息:
  683. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Merger$MergeQueue merge
  684. 信息: Merging 2 sorted segments
  685. 2013-10-14 10:26:42 org.apache.hadoop.io.compress.CodecPool getDecompressor
  686. 信息: Got brand-new decompressor
  687. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Merger$MergeQueue merge
  688. 信息: Down to the last merge-pass, with 2 segments left of total size: 233 bytes
  689. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  690. 信息:
  691. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task done
  692. 信息: Task:attempt_local_0007_r_000000_0 is done. And is in the process of commiting
  693. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  694. 信息:
  695. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task commit
  696. 信息: Task attempt_local_0007_r_000000_0 is allowed to commit now
  697. 2013-10-14 10:26:42 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  698. 信息: Saved output of task 'attempt_local_0007_r_000000_0' to hdfs://192.168.1.210:9000/tmp/1381717594500/partialMultiply
  699. 2013-10-14 10:26:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  700. 信息: reduce > reduce
  701. 2013-10-14 10:26:42 org.apache.hadoop.mapred.Task sendDone
  702. 信息: Task 'attempt_local_0007_r_000000_0' done.
  703. 2013-10-14 10:26:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  704. 信息:  map 100% reduce 100%
  705. 2013-10-14 10:26:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  706. 信息: Job complete: job_local_0007
  707. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  708. 信息: Counters: 19
  709. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  710. 信息:   File Output Format Counters
  711. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  712. 信息:     Bytes Written=572
  713. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  714. 信息:   FileSystemCounters
  715. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  716. 信息:     FILE_BYTES_READ=34517913
  717. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  718. 信息:     HDFS_BYTES_READ=8751
  719. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  720. 信息:     FILE_BYTES_WRITTEN=36182630
  721. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  722. 信息:     HDFS_BYTES_WRITTEN=7934
  723. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  724. 信息:   File Input Format Counters
  725. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  726. 信息:     Bytes Read=0
  727. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  728. 信息:   Map-Reduce Framework
  729. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  730. 信息:     Map output materialized bytes=241
  731. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  732. 信息:     Map input records=12
  733. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  734. 信息:     Reduce shuffle bytes=0
  735. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  736. 信息:     Spilled Records=56
  737. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  738. 信息:     Map output bytes=453
  739. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  740. 信息:     Total committed heap usage (bytes)=2558459904
  741. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  742. 信息:     SPLIT_RAW_BYTES=665
  743. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  744. 信息:     Combine input records=0
  745. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  746. 信息:     Reduce input records=28
  747. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  748. 信息:     Reduce input groups=7
  749. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  750. 信息:     Combine output records=0
  751. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  752. 信息:     Reduce output records=7
  753. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Counters log
  754. 信息:     Map output records=28
  755. 2013-10-14 10:26:43 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus
  756. 信息: Total input paths to process : 1
  757. 2013-10-14 10:26:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  758. 信息: Running job: job_local_0008
  759. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task initialize
  760. 信息:  Using ResourceCalculatorPlugin : null
  761. 2013-10-14 10:26:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  762. 信息: io.sort.mb = 100
  763. 2013-10-14 10:26:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  764. 信息: data buffer = 79691776/99614720
  765. 2013-10-14 10:26:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer
  766. 信息: record buffer = 262144/327680
  767. 2013-10-14 10:26:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush
  768. 信息: Starting flush of map output
  769. 2013-10-14 10:26:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill
  770. 信息: Finished spill 0
  771. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task done
  772. 信息: Task:attempt_local_0008_m_000000_0 is done. And is in the process of commiting
  773. 2013-10-14 10:26:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  774. 信息:
  775. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task sendDone
  776. 信息: Task 'attempt_local_0008_m_000000_0' done.
  777. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task initialize
  778. 信息:  Using ResourceCalculatorPlugin : null
  779. 2013-10-14 10:26:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  780. 信息:
  781. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Merger$MergeQueue merge
  782. 信息: Merging 1 sorted segments
  783. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Merger$MergeQueue merge
  784. 信息: Down to the last merge-pass, with 1 segments left of total size: 206 bytes
  785. 2013-10-14 10:26:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  786. 信息:
  787. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task done
  788. 信息: Task:attempt_local_0008_r_000000_0 is done. And is in the process of commiting
  789. 2013-10-14 10:26:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  790. 信息:
  791. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task commit
  792. 信息: Task attempt_local_0008_r_000000_0 is allowed to commit now
  793. 2013-10-14 10:26:43 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask
  794. 信息: Saved output of task 'attempt_local_0008_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/userCF/result
  795. 2013-10-14 10:26:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate
  796. 信息: reduce > reduce
  797. 2013-10-14 10:26:43 org.apache.hadoop.mapred.Task sendDone
  798. 信息: Task 'attempt_local_0008_r_000000_0' done.
  799. 2013-10-14 10:26:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  800. 信息:  map 100% reduce 100%
  801. 2013-10-14 10:26:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob
  802. 信息: Job complete: job_local_0008
  803. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  804. 信息: Counters: 19
  805. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  806. 信息:   File Output Format Counters
  807. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  808. 信息:     Bytes Written=217
  809. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  810. 信息:   FileSystemCounters
  811. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  812. 信息:     FILE_BYTES_READ=26299802
  813. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  814. 信息:     HDFS_BYTES_READ=7357
  815. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  816. 信息:     FILE_BYTES_WRITTEN=27566408
  817. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  818. 信息:     HDFS_BYTES_WRITTEN=6269
  819. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  820. 信息:   File Input Format Counters
  821. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  822. 信息:     Bytes Read=572
  823. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  824. 信息:   Map-Reduce Framework
  825. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  826. 信息:     Map output materialized bytes=210
  827. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  828. 信息:     Map input records=7
  829. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  830. 信息:     Reduce shuffle bytes=0
  831. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  832. 信息:     Spilled Records=42
  833. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  834. 信息:     Map output bytes=927
  835. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  836. 信息:     Total committed heap usage (bytes)=1971453952
  837. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  838. 信息:     SPLIT_RAW_BYTES=137
  839. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  840. 信息:     Combine input records=0
  841. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  842. 信息:     Reduce input records=21
  843. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  844. 信息:     Reduce input groups=5
  845. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  846. 信息:     Combine output records=0
  847. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  848. 信息:     Reduce output records=5
  849. 2013-10-14 10:26:44 org.apache.hadoop.mapred.Counters log
  850. 信息:     Map output records=21
  851. cat: hdfs://192.168.1.210:9000/user/hdfs/userCF/result//part-r-00000
  852. 1	[104:1.280239,106:1.1462644,105:1.0653841,107:0.33333334]
  853. 2	[106:1.560478,105:1.4795978,107:0.69935876]
  854. 3	[103:1.2475469,106:1.1944525,102:1.1462644]
  855. 4	[102:1.6462644,105:1.5277859,107:0.69935876]
  856. 5	[107:1.1993587]

5). 推荐结果解读
我们可以把上面的日志分解析成3个部分解读

  • a. 初始化环境
  • b. 算法执行
  • c. 打印推荐结果

a. 初始化环境
出初HDFS的数据目录和工作目录，并上传数据文件。


  2. Delete: hdfs://192.168.1.210:9000/user/hdfs/userCF
  3. Create: hdfs://192.168.1.210:9000/user/hdfs/userCF
  4. copy from: datafile/item.csv to hdfs://192.168.1.210:9000/user/hdfs/userCF
  5. ls: hdfs://192.168.1.210:9000/user/hdfs/userCF
  6. ==========================================================
  7. name: hdfs://192.168.1.210:9000/user/hdfs/userCF/item.csv, folder: false, size: 229
  8. ==========================================================
  9. cat: hdfs://192.168.1.210:9000/user/hdfs/userCF/item.csv

b. 算法执行
分别执行，上图中对应的8种MapReduce算法。
Job complete: job_local_0001
Job complete: job_local_0002
Job complete: job_local_0003
Job complete: job_local_0004
Job complete: job_local_0005
Job complete: job_local_0006
Job complete: job_local_0007
Job complete: job_local_0008

c. 打印推荐结果

方便我们看到计算后的推荐结果


  2. cat: hdfs://192.168.1.210:9000/user/hdfs/userCF/result//part-r-00000
  3. 1	[104:1.280239,106:1.1462644,105:1.0653841,107:0.33333334]
  4. 2	[106:1.560478,105:1.4795978,107:0.69935876]
  5. 3	[103:1.2475469,106:1.1944525,102:1.1462644]
  6. 4	[102:1.6462644,105:1.5277859,107:0.69935876]
  7. 5	[107:1.1993587]

4. 模板项目上传github

https://github.com/bsspirit/maven_mahout_template/tree/mahout-0.8

大家可以下载这个项目，做为开发的起点。


  2. ~ git clone https://github.com/bsspirit/maven_mahout_template
  3. ~ git checkout mahout-0.8

我们完成了基于物品的协同过滤分步式算法实现，下面将继续介绍Mahout的Kmeans的分步式算法实现，请参考文章：Mahout分步式程序开发 聚类Kmeans

转载请注明出处：
http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/

 

This entry was posted in Hadoop实践, JAVA语言实践, 数据挖掘, 架构设计

Mahout分步式程序开发 基于物品的协同过滤ItemCF的更多相关文章

  1. 转】Mahout分步式程序开发 基于物品的协同过滤ItemCF

    原博文出自于: http://blog.fens.me/hadoop-mahout-mapreduce-itemcf/ 感谢! Posted: Oct 14, 2013 Tags: Hadoopite ...

  2. 转】Mahout分步式程序开发 聚类Kmeans

    原博文出自于: http://blog.fens.me/hadoop-mahout-kmeans/ 感谢! Mahout分步式程序开发 聚类Kmeans Hadoop家族系列文章,主要介绍Hadoop ...

  3. Mahout分步式程序开发 聚类Kmeans（转）

    Posted: Oct 14, 2013 Tags: clusterHadoopkmeansMahoutR聚类 Comments: 13 Comments Mahout分步式程序开发 聚类Kmeans ...

  4. 基于物品的协同过滤ItemCF的mapreduce实现

    文章的UML图比较好看..... 原文链接:www.cnblogs.com/anny-1980/articles/3519555.html 基于物品的协同过滤ItemCF 数据集字段: 1．  Use ...

  5. 基于物品的协同过滤item-CF 之电影推荐 python

    推荐算法有基于协同的Collaboration Filtering:包括 user Based和item Based:基于内容 : Content Based 协同过滤包括基于物品的协同过滤和基于用户 ...

  6. 推荐召回--基于物品的协同过滤：ItemCF

    目录 1. 前言 2. 原理&计算&改进 3. 总结 1. 前言 说完基于用户的协同过滤后,趁热打铁,我们来说说基于物品的协同过滤:"看了又看","买了又 ...

  7. 基于物品的协同过滤推荐算法——读“Item-Based Collaborative Filtering Recommendation Algorithms” .

    ligh@local-host$ ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.0.3 基于物品的协同过滤推荐算法--读"Item-Based ...

  8. ItemCF_基于物品的协同过滤_MapReduceJava代码实现思路

    ItemCF_基于物品的协同过滤 1.    概念 2.    原理 如何给用户推荐? 给用户推荐他没有买过的物品--103 3.    java代码实现思路 数据集: 第一步:构建物品的同现矩阵 第 ...

  9. Music Recommendation System with User-based and Item-based Collaborative Filtering Technique(使用基于用户及基于物品的协同过滤技术的音乐推荐系统)【更新】

    摘要: 大数据催生了互联网,电子商务,也导致了信息过载.信息过载的问题可以由推荐系统来解决.推荐系统可以提供选择新产品(电影,音乐等)的建议.这篇论文介绍了一个音乐推荐系统,它会根据用户的历史行为和口 ...

随机推荐

  1. 16、Redis手动创建集群

    写在前面的话:读书破万卷,编码如有神 --------------------------------------------------------------------------------- ...

  2. Java Web c3p0 pool池泄漏优化与日志分析

    问题跟踪: 近期在整合SSH(spring.springmvc.hibernate)项目,提供给第三方服务.每当调用内存池达到上限之后,外界调用服务直接失败,提示[cannot open connec ...

  3. 解决 MyEclipse build workspace 慢，validation javascript 更慢的问题

    自从升级了MyEclipse到7.0,项目Build的时候总是很慢,显示Validating 那些js,html文件.不管我怎么调整 Windows > Preference > MyEc ...

  4. OpenVPN搭建中tap与tun的实际使用区别

    tap俗称网桥模式,tun俗称路由模式,tap在二层,tun在三层,在实际应用中,其实以上这些知识概念,我是抄来的,具体的解释可以看以下参考链接. 下面将介绍在实际使用中的区别: 1.tap可以直接使 ...

  5. @Transactional导致AbstractRoutingDataSource动态数据源无法切换的解决办法

    上午花了大半天排查一个多数据源主从切换的问题,记录一下: 背景: 项目的数据库采用了读写分离多数据源,采用AOP进行拦截,利用ThreadLocal及AbstractRoutingDataSource ...

  6. Unity IOC容器通过配置实现类型映射的几种基本使用方法

    网上关于Unity IOC容器使用的方法已很多,但未能做一个总结,故我这里总结一下,方便大家选择. 首先讲一下通过代码来进行类型映射,很简单,代码如下 unityContainer = new Uni ...

  7. GO环境变量设置

    GOROOT就是go的安装路径在~/.bash_profile中添加下面语句: GOROOT=/usr/local/go export GOROOT 当然, 要执行go命令和go工具, 就要配置go的 ...

  8. Delphi如何在Form的标题栏绘制自定义文字

    Delphi中Form窗体的标题被设计成绘制在系统菜单的旁边,如果你想要在标题栏绘制自定义文本又不想改变Caption属性,你需要处理特定的Windows消息:WM_NCPAINT.. WM_NCPA ...

  9. MongoDB C# 驱动的各种版本下载地址

    https://github.com/mongodb/mongo-csharp-driver/releases

  10. A股和B股票的区别？

    一.A股和B股的区别——概念不同 (一).A股的概念A股是由中国境内的公司发行,正式名称是人民币普通股票,供境内机构.组织或个人(从4月1日起,境内港.澳.台居民可开立A股账户)以人民币认购和交易的普 ...