mahout学习

参考：http://www.360doc.com/content/14/0117/09/1200324_345883534.shtml

Precondition: 启动Hadoop集群

bin/hdfs namenode -format

sbin/start-dfs.sh(启动Namenode,nodenode相关节点)

sbin/start-yarn.sh(启动ResourceManager，nodeManager相关资源)

bin/hdfs dfsadmin -safemode leave(关闭安全模式)

Note：所有bin/mahout下对应的输入文件，输入文件夹均在HDFS文件目录下

In addition，Mahout下处理的文件必须是SequenceFile文件格式的，故需将txt格式文件转化为SequenceFile文件，如下：

kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0$ bin/mahout seqdirectory -input（输入） /TagOutput/txtFile.txt -output（输出）/TagOutput/seqFile.txt --charset UTF-8

相关实践过程如下：

(将sequenceFile文件转化为可读的txt文件)

kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0$ bin/mahout seqdumper -i（输入） /TagOutput/clusteredPoints/part-m-00000 -o（输出） ./TagOutput/clusterPoints.txt
Running on hadoop, using /home/kelvin/UntarFile/hadoop2CDH4//bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: /home/kelvin/UntarFile/mahout-0.7-cdh4.5.0/mahout-examples-0.7-cdh4.5.0-job.jar
14/06/06 02:18:07 INFO common.AbstractJob: Command line arguments: {--endPhase=[2147483647], --input=[/TagOutput/clusteredPoints/part-m-00000], --output=[./TagOutput/clusterPoints.txt], --startPhase=[0], --tempDir=[temp]}
14/06/06 02:18:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/06 02:18:08 INFO driver.MahoutDriver: Program took 1162 ms (Minutes: 0.019366666666666667)
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0$ cd TagOutput/
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0/TagOutput$ ll
total 12
drwxrwxr-x 2 kelvin kelvin 4096 6? 6 02:12 ./
drwxr-xr-x 16 kelvin kelvin 4096 6? 6 02:15 ../
-rw-rw-r-- 1 kelvin kelvin 2767 6? 6 02:18 clusterPoints.txt
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0/TagOutput$ cat clusterPoints.txt
Input Path: /TagOutput/clusteredPoints/part-m-00000
Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.mahout.clustering.classify.WeightedPropertyVectorWritable
Key: 14: Value: wt: 1.0 distance: 13.357142857142858 vec: 2012000317 = [24.000]
Key: 14: Value: wt: 1.0 distance: 6.642857142857143 vec: 2012000318 = [4.000]
Key: 25: Value: wt: 1.0 distance: 11.615384615384592 vec: 2012000319 = [56.000]
Key: 14: Value: wt: 1.0 distance: 7.642857142857142 vec: 2012000320 = [3.000]
Key: 14: Value: wt: 1.0 distance: 11.357142857142858 vec: 2012000321 = [22.000]
Key: 25: Value: wt: 1.0 distance: 0.6153846153846132 vec: 2012000322 = [45.000]
Key: 14: Value: wt: 1.0 distance: 13.357142857142858 vec: 2012000323 = [24.000]
Key: 14: Value: wt: 1.0 distance: 6.642857142857143 vec: 2012000324 = [4.000]
Key: 25: Value: wt: 1.0 distance: 11.615384615384592 vec: 2012000325 = [56.000]
Key: 14: Value: wt: 1.0 distance: 7.642857142857142 vec: 2012000326 = [3.000]
Key: 14: Value: wt: 1.0 distance: 11.357142857142858 vec: 2012000327 = [22.000]
Key: 25: Value: wt: 1.0 distance: 2.384615384615403 vec: 2012000328 = [42.000]
Key: 25: Value: wt: 1.0 distance: 4.61538461538464 vec: 2012000329 = [49.000]
Key: 25: Value: wt: 1.0 distance: 3.384615384615356 vec: 2012000330 = [41.000]
Key: 14: Value: wt: 1.0 distance: 5.642857142857143 vec: 2012000331 = [5.000]
Key: 14: Value: wt: 1.0 distance: 7.642857142857142 vec: 2012000332 = [3.000]
Key: 14: Value: wt: 1.0 distance: 10.357142857142858 vec: 2012000333 = [21.000]
Key: 25: Value: wt: 1.0 distance: 10.38461538461539 vec: 2012000334 = [34.000]
Key: 25: Value: wt: 1.0 distance: 15.384615384615387 vec: 2012000335 = [29.000]
Key: 25: Value: wt: 1.0 distance: 1.3846153846153868 vec: 2012000336 = [43.000]
Key: 25: Value: wt: 1.0 distance: 9.615384615384606 vec: 2012000337 = [54.000]
Key: 25: Value: wt: 1.0 distance: 8.384615384615397 vec: 2012000338 = [36.000]
Key: 14: Value: wt: 1.0 distance: 8.642857142857142 vec: 2012000339 = [2.000]
Key: 14: Value: wt: 1.0 distance: 5.642857142857143 vec: 2012000340 = [5.000]
Key: 20: Value: wt: 1.0 distance: 1.7999999999999972 vec: 2012000341 = [78.000]
Key: 25: Value: wt: 1.0 distance: 9.615384615384606 vec: 2012000342 = [54.000]
Key: 20: Value: wt: 1.0 distance: 13.800000000000018 vec: 2012000343 = [66.000]
Key: 14: Value: wt: 1.0 distance: 3.6428571428571423 vec: 2012000344 = [7.000]
Key: 20: Value: wt: 1.0 distance: 29.200000000000053 vec: 2012000345 = [109.000]
Key: 20: Value: wt: 1.0 distance: 11.800000000000068 vec: 2012000346 = [68.000]
Key: 20: Value: wt: 1.0 distance: 1.7999999999999972 vec: 2012000347 = [78.000]
Key: 25: Value: wt: 1.0 distance: 6.384615384615375 vec: 2012000348 = [38.000]
Count: 32
（将相应的数据结点聚类后输出）

kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0$ bin/mahout clusterdump --input /TagOutput/*final(:目录，非文件，最后一次迭代的clusters) --pointsDir /TagOutput/clusteredPoints（最后一次聚类后的点） --output ./TagOutput/clusterResult.txt
Running on hadoop, using /home/kelvin/UntarFile/hadoop2CDH4//bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: /home/kelvin/UntarFile/mahout-0.7-cdh4.5.0/mahout-examples-0.7-cdh4.5.0-job.jar
14/06/06 02:50:11 INFO common.AbstractJob: Command line arguments: {--dictionaryType=[text], --distanceMeasure=[org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure], --endPhase=[2147483647], --input=[/TagOutput/*final], --output=[./TagOutput/clusterResult.txt], --outputFormat=[TEXT], --pointsDir=[/TagOutput/clusteredPoints], --startPhase=[0], --tempDir=[temp]}
14/06/06 02:50:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/06 02:50:14 INFO clustering.ClusterDumper: Wrote 3 clusters
14/06/06 02:50:14 INFO driver.MahoutDriver: Program took 2809 ms (Minutes: 0.046816666666666666)
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0$ cd TagOutput/
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0/TagOutput$ ls
clusterPoints.txt clusterResult.txt
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0/TagOutput$ ll
total 16
drwxrwxr-x 2 kelvin kelvin 4096 6? 6 02:30 ./
drwxr-xr-x 16 kelvin kelvin 4096 6? 6 02:15 ../
-rw-rw-r-- 1 kelvin kelvin 2767 6? 6 02:18 clusterPoints.txt
-rw-rw-r-- 1 kelvin kelvin 2108 6? 6 02:50 clusterResult.txt
kelvin@Master:~/UntarFile/mahout-0.7-cdh4.5.0/TagOutput$ cat clusterResult.txt
VL-14{n=14 c=[10.643] r=[9.013]}
        Weight : [props - optional]: Point:
        1.0 : [distance=13.357142857142858]: 2012000317 = [24.000]
        1.0 : [distance=6.642857142857143]: 2012000318 = [4.000]
        1.0 : [distance=7.642857142857142]: 2012000320 = [3.000]
        1.0 : [distance=11.357142857142858]: 2012000321 = [22.000]
        1.0 : [distance=13.357142857142858]: 2012000323 = [24.000]
        1.0 : [distance=6.642857142857143]: 2012000324 = [4.000]
        1.0 : [distance=7.642857142857142]: 2012000326 = [3.000]
        1.0 : [distance=11.357142857142858]: 2012000327 = [22.000]
        1.0 : [distance=5.642857142857143]: 2012000331 = [5.000]
        1.0 : [distance=7.642857142857142]: 2012000332 = [3.000]
        1.0 : [distance=10.357142857142858]: 2012000333 = [21.000]
        1.0 : [distance=8.642857142857142]: 2012000339 = [2.000]
        1.0 : [distance=5.642857142857143]: 2012000340 = [5.000]
        1.0 : [distance=3.6428571428571423]: 2012000344 = [7.000]
VL-20{n=5 c=[79.800] r=[15.419]}
        Weight : [props - optional]: Point:
        1.0 : [distance=1.7999999999999972]: 2012000341 = [78.000]
        1.0 : [distance=13.800000000000018]: 2012000343 = [66.000]
        1.0 : [distance=29.200000000000053]: 2012000345 = [109.000]
        1.0 : [distance=11.800000000000068]: 2012000346 = [68.000]
        1.0 : [distance=1.7999999999999972]: 2012000347 = [78.000]
VL-25{n=13 c=[44.385] r=[8.553]}
        Weight : [props - optional]: Point:
        1.0 : [distance=11.615384615384592]: 2012000319 = [56.000]
        1.0 : [distance=0.6153846153846132]: 2012000322 = [45.000]
        1.0 : [distance=11.615384615384592]: 2012000325 = [56.000]
        1.0 : [distance=2.384615384615403]: 2012000328 = [42.000]
        1.0 : [distance=4.61538461538464]: 2012000329 = [49.000]
        1.0 : [distance=3.384615384615356]: 2012000330 = [41.000]
        1.0 : [distance=10.38461538461539]: 2012000334 = [34.000]
        1.0 : [distance=15.384615384615387]: 2012000335 = [29.000]
        1.0 : [distance=1.3846153846153868]: 2012000336 = [43.000]
        1.0 : [distance=9.615384615384606]: 2012000337 = [54.000]
        1.0 : [distance=8.384615384615397]: 2012000338 = [36.000]
        1.0 : [distance=9.615384615384606]: 2012000342 = [54.000]
        1.0 : [distance=6.384615384615375]: 2012000348 = [38.000]
上述结果的输出等同于在Eclipse中编写的ClusterDumper，如下：

public static void run(Configuration conf, Path input, Path output, DistanceMeasure measure, double t1, double t2,

      double convergenceDelta, int maxIterations) throws Exception {

    Path directoryContainingConvertedInput = new Path(output, DIRECTORY_CONTAINING_CONVERTED_INPUT);

    log.info("Preparing Input");

    TagInputDriver.runJob(input, directoryContainingConvertedInput, "org.apache.mahout.math.RandomAccessSparseVector");

    log.info("Running Canopy to get initial clusters");

    Path canopyOutput = new Path(output, "canopies");

    CanopyDriver.run(new Configuration(), directoryContainingConvertedInput, canopyOutput, measure, t1, t2, false, 0.0,

        false);

    log.info("Running KMeans");

    TagKMeansDriver.run(conf, directoryContainingConvertedInput, new Path(canopyOutput, Cluster.INITIAL_CLUSTERS_DIR

        + "-final"), output, convergenceDelta, maxIterations, true, 0.0, false);

    // run ClusterDumper

    ClusterDumper clusterDumper = new ClusterDumper(new Path(output, "clusters-*-final"), new Path(output,

        "clusteredPoints"));

    clusterDumper.printClusters(null);

  }

其他前提操作：

kelvin@Master:~/UntarFile/hadoop2CDH4$ bin/hadoop fs -put ./../mahout-0.7-cdh4.5.0/output/* /TagOutput
14/06/06 01:53:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
kelvin@Master:~/UntarFile/hadoop2CDH4$ bin/hadoop fs -ls /TagOutput
14/06/06 01:53:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 15 items
-rw-r--r--   1 kelvin supergroup        194 2014-06-06 01:53 /TagOutput/_policy
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusteredPoints
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-0
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-1
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-10-final
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-2
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-3
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-4
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-5
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-6
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-7
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-8
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/clusters-9
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/data
drwxr-xr-x   - kelvin supergroup          0 2014-06-06 01:53 /TagOutput/random-seeds

mahout学习的更多相关文章

转】Mahout学习路线图
原博文出自于: http://blog.fens.me/hadoop-mahout-roadmap/ 感谢! Mahout学习路线图 Hadoop家族系列文章,主要介绍Hadoop家族产品,常用的项目 ...
mahout第一篇-----Mahout学习路线图
Mahout学习路线图前言 Mahout是Hadoop家族中与众不同的一个成员,是基于一个Hadoop的机器学习和数据挖掘的分布式计算框架.Mahout是一个跨学科产品,同时也是我认为Hadoop家 ...
Mahout学习路线图
转自:http://blog.fens.me/hadoop-mahout-roadmap/ Mahout学习路线图 Hadoop家族系列文章,主要介绍Hadoop家族产品,常用的项目包括Hadoop, ...
Mahout学习路线图-张丹老师
前言 Mahout是Hadoop家族中与众不同的一个成员,是基于一个Hadoop的机器学习和数据挖掘的分布式计算框架.Mahout是一个跨学科产品,同时也是我认为Hadoop家族中,最有竞争力,最难掌 ...
Hadoop里的数据挖掘应用-Mahout——学习笔记<三>
之前有幸在MOOC学院抽中小象学院hadoop体验课. 这是小象学院hadoop2.X的笔记由于平时对数据挖掘做的比较多,所以优先看Mahout方向视频. Mahout有很好的扩展性与容错性(基于H ...
Mahout学习之Mahout简介、安装、配置、入门程序测试
一.Mahout简介查了Mahout的中文意思——驭象的人,再看看Mahout的logo,好吧,想和小黄象happy地玩耍,得顺便陪陪这位驭象人耍耍了... 附logo: (就是他,骑在象头上的那个 ...
Mahout学习之Mahout简单介绍、安装、配置、入门程序測试
一.Mahout简单介绍查了Mahout的中文意思--驭象的人,再看看Mahout的logo,好吧,想和小黄象happy地玩耍,得顺便陪陪这位驭象人耍耍了... 附logo: (就是他,骑在象头上的 ...
mahout学习-1
一. 安装软件需要安装如下文件: java, Eclipse, Maven,Hadoop,mahout 二. 推荐系统简介每天,我们都会对一些事物表达自己的看法,喜欢,或不喜欢,或不在乎.这些都在 ...
Mahout学习资料
Apache Mahout 简介 http://www.ibm.com/developerworks/cn/java/j-mahout/ 从源代码剖析Mahout推荐引擎 http://blog.fe ...

随机推荐

2018.08.11 洛谷P3224 [HNOI2012]永无乡（线段树合并）
传送门给出n个带点权的点,支持连边和查询连通块第k大. 这个貌似就是一道线段树合并的裸板啊... 代码: #include<bits/stdc++.h> #define N 100005 ...
常见XML解析器
xpp3 官网 http://www.extreme.indiana.edu/xgws/xsoap/xpp/ 简介 Xml Pull Parser (in short XPP) is a stream ...
关于React的入门级安装和最浅显解释
春节临近,办公室里半片空位,半片浮嚣. 想到将放假,屏幕上的代码也都变成了雀跃的小虫. 无法专心了. 终于还是强迫自己读了半篇文档,写了几坨程序. 这次记录的是关于React,最浅显的内容. ———— ...
（匹配最小路径覆盖）Air Raid --hdu --1151
链接: http://acm.hdu.edu.cn/showproblem.php?pid=1151 http://acm.hust.edu.cn/vjudge/contest/view.action ...
STL中的Vector相关用法
STL中的Vector相关用法标准库vector类型使用需要的头文件:#include <vector>. vector 是一个类模板,不是一种数据类型,vector<int> ...
iterm2 学习笔记
itrem 笔记选中即复制,有两种方式. 在新Tab中自动使用前一Tab路径,该怎么用? 系统热键:option+space 自动完成:输入打头几个字母,然后输入command+“;” iterm2 ...
比较分析 Spring AOP 和 AspectJ 之间的差别
面向方面的编程(AOP) 是一种编程范式,旨在通过允许横切关注点的分离,提高模块化.AOP提供方面来将跨越对象关注点模块化.虽然现在可以获得许多AOP框架,但在这里我们要区分的只有两个流行的框架:Sp ...
element ui aside — 侧栏导航菜单移入移出折叠效果
效果如图,移入移出控制折叠,点击按钮后移入移出不可控制折叠. 功能是很简单的功能,不过昨天这块还是弄了小一个小时,所以记录下来. 发现的问题: 模版上el-aside写上@mouseenter无效解 ...
RabbitMQ基础入门篇
下载安装 Erlang RabbitMQ 启动RabbitMQ管理平台插件 DOS下进入到安装目录\sbin,执行以下命令 rabbitmq-plugins enable rabbitmq_manag ...
wpf 右下角弹出窗
自己写的wpf 弹出框,欢迎拍砖,动画都写在了后台代码,前台代码不太重要,用了一下iconfont,具体样式我就不贴出来了,本次主要是后台代码的动画需要有父级窗口才可以使用. 前台代码: <W ...

mahout学习

mahout学习的更多相关文章

随机推荐

热门专题