AAS代码运行-第4章
[root@node1 aas]# ls
ch02 ch03 spark-1.2.-bin-hadoop2. spark-1.2.-bin-hadoop2..tgz
[root@node1 aas]# cd spark-1.2.-bin-hadoop2.
[root@node1 spark-1.2.-bin-hadoop2.]# cd ..
[root@node1 aas]# mkdir ch04
[root@node1 aas]# cd ch04
[root@node1 ch04]# ls
[root@node1 ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz
---- ::-- https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:... connected.
HTTP request sent, awaiting response... OK
Length: (11M) [application/x-gzip]
Saving to: ?.ovtype.data.gz? %[===============================================================================================================================================================>] ,, 2.62M/s in .2s -- :: (2.53 MB/s) - ?.ovtype.data.gz?.saved [/] [root@node1 ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.info
---- ::-- https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.info
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:... connected.
HTTP request sent, awaiting response... OK
Length: (14K) [text/plain]
Saving to: ?.ovtype.info? %[===============================================================================================================================================================>] , --.-K/s in .001s -- :: (15.6 MB/s) - ?.ovtype.info?.saved [/] [root@node1 ch04]# wget https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/old_covtype.info
---- ::-- https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/old_covtype.info
Resolving archive.ics.uci.edu... 128.195.10.249
Connecting to archive.ics.uci.edu|128.195.10.249|:... connected.
HTTP request sent, awaiting response... OK
Length: (.7K) [text/plain]
Saving to: ?.ld_covtype.info? %[===============================================================================================================================================================>] , --.-K/s in 0s -- :: (12.7 MB/s) - ?.ld_covtype.info?.saved [/]
将数据放到HDFS上
[root@node1 ch04]# ls
covtype.data.gz covtype.info old_covtype.info
[root@node1 ch04]# gunzip -d covtype.data.gz
[root@node1 ch04]# ll
total
-rw-r--r-- root root Sep covtype.data
-rw-r--r-- root root Apr covtype.info
-rw-r--r-- root root Sep old_covtype.info
[root@node1 ch04]# hdfs dfs -mkdir /user/root/covtype
[root@node1 ch04]# hdfs dfs -put * /user/root/covtype
[root@node1 ch04]# hdfs dfs -ls /user/root/covtype
Found items
-rw-r--r-- root supergroup -- : /user/root/covtype/covtype.data
-rw-r--r-- root supergroup -- : /user/root/covtype/covtype.info
-rw-r--r-- root supergroup -- : /user/root/covtype/old_covtype.info
启动spark-shell
[root@node1 ch04]# ../spark-1.2.-bin-hadoop2./bin/spark-shell --master yarn-client
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.2.
/_/ Using Scala version 2.10. (OpenJDK -Bit Server VM, Java 1.7.0_09-icedtea)
Type in expressions to have them evaluated.
Type :help for more information.
// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context available as sc.
运行代码4.7节代码：
scala> import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg._ scala> import org.apache.spark.mllib.regression._
import org.apache.spark.mllib.regression._ scala> scala> val rawData = sc.textFile("hdfs:///user/ds/covtype.data" )
rawData: org.apache.spark.rdd.RDD[String] = hdfs:///user/ds/covtype.data MappedRDD[1] at textFile at <console>:18 scala> val data = rawData.map { line =>
| val values = line.split(',' ).map(_. toDouble)
| val featureVector = Vectors.dense(values. init)
| val label = values.last -
| LabeledPoint(label, featureVector)
| }
data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = MappedRDD[] at map at <console>: scala> val Array(trainData, cvData, testData) = data. randomSplit(Array(0.8, 0.1, 0.1))
trainData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[] at randomSplit at <console>:
cvData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[] at randomSplit at <console>:
testData: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = PartitionwiseSampledRDD[] at randomSplit at <console>: scala> trainData.cache()
res0: trainData.type = PartitionwiseSampledRDD[] at randomSplit at <console>: scala> cvData.cache()
res1: cvData.type = PartitionwiseSampledRDD[] at randomSplit at <console>: scala> testData.cache()
res2: testData.type = PartitionwiseSampledRDD[] at randomSplit at <console>:22
scala> import org.apache.spark.mllib.evaluation._
import org.apache.spark.mllib.evaluation._
scala> import org.apache.spark.mllib.tree._
import org.apache.spark.mllib.tree._
scala> import org.apache.spark.mllib.tree.model._
import org.apache.spark.mllib.tree.model._
scala> import org.apache.spark.rdd._
import org.apache.spark.rdd._
scala> def getMetrics(model: DecisionTreeModel, data: RDD[LabeledPoint]):
| MulticlassMetrics = {
| val predictionsAndLabels = data. map(example =>(model. predict(example. features), example. label))
| new MulticlassMetrics(predictionsAndLabels)
| }
getMetrics: (model: org.apache.spark.mllib.tree.model.DecisionTreeModel, data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint])org.apache.spark.mllib.evaluation.MulticlassMetrics
scala> val model = DecisionTree.trainClassifier(trainData, 7, Map[Int, Int](), "gini" , 4, 100)
model: org.apache.spark.mllib.tree.model.DecisionTreeModel = DecisionTreeModel classifier of depth 4 with 31 nodes
scala> val metrics = getMetrics(model, cvData)
metrics: org.apache.spark.mllib.evaluation.MulticlassMetrics = org.apache.spark.mllib.evaluation.MulticlassMetrics@5d574c23
scala> metrics. confusionMatrix
res6: org.apache.spark.mllib.linalg.Matrix =
15535.0 5345.0 21.0 0.0 0.0 0.0 392.0
6669.0 20855.0 688.0 0.0 5.0 0.0 47.0
0.0 610.0 2942.0 0.0 0.0 0.0 0.0
0.0 0.0 274.0 0.0 0.0 0.0 0.0
12.0 874.0 57.0 0.0 15.0 0.0 0.0
0.0 446.0 1318.0 0.0 0.0 0.0 0.0
1150.0 19.0 8.0 0.0 0.0 0.0 905.0
scala> metrics.precision
res7: Double = 0.6917696392665028
scala> metrics.precision
res7: Double = 0.6917696392665028
scala> (0 until 7).map(
| cat => (metrics.precision(cat), metrics.recall(cat))
| ).foreach(println)
(0.6648549174013524,0.729582491898746)
(0.7408788944545099,0.7378644211718086)
(0.554257724189902,0.8282657657657657)
(0.0,0.0)
(0.75,0.015657620041753653)
(0.0,0.0)
(0.6733630952380952,0.4346781940441883)
scala> import org.apache.spark.rdd._
import org.apache.spark.rdd._
scala> def classProbabilities(data: RDD[LabeledPoint]): Array[Double] = {
| val countsByCategory = data.map(_.label).countByValue()
| val counts = countsByCategory.toArray.sortBy(_. _1).map(_. _2)
| counts.map(_.toDouble / counts.sum)
| }
classProbabilities: (data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint])Array[Double]
scala> val trainPriorProbabilities = classProbabilities(trainData)
trainPriorProbabilities: Array[Double] = Array(0.3644680841907762, 0.48778063233452534, 0.06163475731247069, 0.004682046846288574, 0.0163893156379504, 0.029860958700732, 0.035184204977256786)
scala> val cvPriorProbabilities = classProbabilities(cvData)
cvPriorProbabilities: Array[Double] = Array(0.36594084589341264, 0.4857442384037672, 0.061044563218588345, 0.004708955608641105, 0.016464158660869265, 0.03031604997679894, 0.03578118823792256)
scala> trainPriorProbabilities.zip(cvPriorProbabilities).map {
| case (trainProb, cvProb) => trainProb * cvProb
| }.sum
res9: Double = 0.3765289404519721
scala> val evaluations =
| for (impurity <- Array("gini" , "entropy" );
| depth <- Array(1, 20);
| bins <- Array(10, 300))
| yield {
| val model = DecisionTree. trainClassifier(trainData, 7, Map[Int, Int](), impurity, depth, bins)
| val predictionsAndLabels = cvData. map(example =>(model. predict(example. features), example. label))
| val accuracy = new MulticlassMetrics(predictionsAndLabels). precision
| ((impurity, depth, bins), accuracy)
| }
evaluations: Array[((String, Int, Int), Double)] = Array(((gini,1,10),0.6319968377816351), ((gini,1,300),0.6323577431385017), ((gini,20,10),0.889253613350061), ((gini,20,300),0.9074191829790159), ((entropy,1,10),0.4857442384037672), ((entropy,1,300),0.4857442384037672), ((entropy,20,10),0.8946500077336862), ((entropy,20,300),0.9099455204770825))
scala> evaluations.sortBy(_. _2).reverse.foreach(println)
((entropy,20,300),0.9099455204770825)
((gini,20,300),0.9074191829790159)
((entropy,20,10),0.8946500077336862)
((gini,20,10),0.889253613350061)
((gini,1,300),0.6323577431385017)
((gini,1,10),0.6319968377816351)
((entropy,1,300),0.4857442384037672)
((entropy,1,10),0.4857442384037672)
scala> val data = rawData.map { line =>
| val values = line.split(',' ).map(_.toDouble)
| val wilderness = values.slice(10, 14).indexOf(1.0).toDouble
| val soil = values.slice(14, 54).indexOf(1.0).toDouble
| val featureVector =
| Vectors.dense(values.slice(0, 10) :+ wilderness :+ soil)
| val label = values.last - 1
| LabeledPoint(label, featureVector)
| }
data: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint] = MappedRDD[391] at map at <console>:47
scala>
scala> val evaluations =
| for (impurity <- Array("gini" , "entropy" );
| depth <- Array(10, 20, 30);
| bins <- Array(40, 300))
| yield {
| val model = DecisionTree. trainClassifier(trainData, 7, Map(10 -> 4, 11 -> 40),impurity, depth, bins)
| val trainAccuracy = getMetrics(model, trainData). precision
| val cvAccuracy = getMetrics(model, cvData). precision
| ((impurity, depth, bins), (trainAccuracy, cvAccuracy))
| }
evaluations: Array[((String, Int, Int), (Double, Double))] = Array(((gini,10,40),(0.7772542032989496,0.7730420884389984)), ((gini,10,300),(0.7849615065174265,0.7793665251688522)), ((gini,20,40),(0.9393033733975393,0.904480382215959)), ((gini,20,300),(0.9421715574260792,0.904480382215959)), ((gini,30,40),(0.9972329447406585,0.9341089934177738)), ((gini,30,300),(0.9974352022790551,0.9347964321927579)), ((entropy,10,40),(0.7768755083334409,0.7716672108890302)), ((entropy,10,300),(0.7715307452975122,0.7655318198222971)), ((entropy,20,40),(0.9487578374796128,0.9103407977726984)), ((entropy,20,300),(0.9484781196073622,0.9088971763452317)), ((entropy,30,40),(0.998582045555283,0.9374430714764467)), ((entropy,30,300),(0.9990833860493938,0.9413786584632307)))
scala> val forest = RandomForest. trainClassifier(
| trainData, 7, Map(10 -> 4, 11 -> 40), 20,
| "auto" , "entropy" , 30, 300)
forest: org.apache.spark.mllib.tree.model.RandomForestModel =
TreeEnsembleModel classifier with 20 trees
AAS代码运行-第4章的更多相关文章
- AAS代码运行-第11章-2
hdfs dfs -ls /user/littlesuccess/AdvancedAnalysisWithSparkhdfs dfs -mkdir /user/littlesuccess/Advanc ...
- AAS代码运行-第11章-1
启动PySpark export IPYTHON= # PySpark也可使用IPython shell pyspark --master yarn --num-executors 发生如下错误: / ...
- 20172327 2018-2019-1 《第一行代码Android》第一章学习总结
学号 2018-2019-1 <第一行代码Android>第一章学习总结 教材学习内容总结 - Android系统架构: 1.Linux内核层 Android系统是基于Linux内核的,这 ...
- Hbase集群搭建及所有配置调优参数整理及API代码运行
最近为了方便开发,在自己的虚拟机上搭建了三节点的Hadoop集群与Hbase集群,hadoop集群的搭建与zookeeper集群这里就不再详细说明,原来的笔记中记录过.这里将hbase配置参数进行相应 ...
- 第十章实践——系统级I/O代码运行
第十章实践——系统级I/O代码运行 实验代码清单如下: 1. cp1——复制一个文件到另一个文件中(两个已经存在的文件) 复制前: 执行后结果 2. setecho.echostate——改变.显示输 ...
- 如何加速MATLAB代码运行
学习笔记 V1.0 2015/4/17 如何加速MATLAB代码运行 概述 本文源于LDPCC的MATLAB代码,即<CCSDS标准的LDPC编译码仿真>.由于代码的问题,在信息位长度很长 ...
- Spark菜鸟学习营Day6 分布式代码运行调试
Spark菜鸟学习营Day6 分布式代码运行调试 作为代码调试,一般会分成两个部分 语法调试,也就是确定能够运行 结果调试,也就是确定程序逻辑的正确 其实这个都离不开运行,所以我们说一下如何让开发的S ...
- 监控代码运行时长 -- StopWatch用法例程
在.net环境下,精确的测量出某段代码运行的时长,在网络通信.串口通信以及异步操作中很有意义.现在做了简单的总结.具体代码如下: (1).首先 using System.Diagnostics; (2 ...
- 解决“无法连接到Python代码运行助手。请检查本机的设置”问题
廖雪峰老师python课程里有个代码运行助手,可以让你在线输入Python代码,然后通过本机运行的一个Python脚本来执行代码,很方便的一个脚本工具,但是很多人用过之后出现了这样的提示:“无法连接到 ...
随机推荐
- hihoCoder 1432 : JiLi Number（吉利数）
hihoCoder #1432 : JiLi Number(吉利数) 时间限制:1000ms 单点时限:1000ms 内存限制:256MB Description - 题目描述 Driver Ji l ...
- 如何为CriteriaOperator过滤对象转换为lambda表达式，即：linq to xpo的动态where语句
How to convert the CriteriaOperator to a lambda expression, so, the latter expression can be used in ...
- storm中DAU实时计算方案
所就职的公司是一家互联网视频公司,存在大量的实时计算需求,计算uv,pv等一些经典的实时指标统计.由于要统计当天的实时 UV,当天的uv由于要存储当天的所有的key,面临本地内存不够用的问题,异常重启 ...
- codeigniter nginx配置
转载:http://www.nginx.cn/1134.html server{ listen 80; server_name www.ci.oa.com; access_log /usr/local ...
- WordPress 插件推荐
1.电商类: Woocommerce 2.幻灯片: Reslider 3.网页编写类: js_composer
- 【转】linux中do{...} while(0)的解释
在看ldlm的代码过程中遇到了一个很奇怪的问题,有很多宏定义使用了do while(0)这种看起来好像没啥用的代码.然后我就问问师兄,才得知,这种用法很常见,自己又查了一下资料,原来在linux内核代 ...
- iOS runtime 知识点整理
// ------ 动态创建类, 添加成员变量, 赋值并调用动态添加的方法 ------- @implementation ViewController - (void)viewDidLoad { [ ...
- Java分别与MySQL、Oracle、SQL Server数据库建立连接
1.与MySQL连接 jar包下载地址: Class.forName("com.mysql.jdbc.Driver");//加载数据库驱动(MySQL的jar包) String u ...
- Python递归报错：RuntimeError: maximum recursion depth exceeded in comparison
Python中默认的最大递归深度是989,当尝试递归第990时便出现递归深度超限的错误: RuntimeError: maximum recursion depth exceeded in compa ...
- 命名空间$.fn
$.fn.xxx是可以用对象来调用的命名空间,例如 $.fn.input() 在声明时就可以用 $('abc').input() $.fx是指jquery的特效. 如果使用显示.滑动.淡入淡出. ...