python spark 决策树 入门demo
Refer to the DecisionTree
Python docs and DecisionTreeModel
Python docs for more details on the API.
from pyspark.mllib.tree import DecisionTree, DecisionTreeModel
from pyspark.mllib.util import MLUtils # Load and parse the data file into an RDD of LabeledPoint.
data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt')
# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3]) # Train a DecisionTree model.
# Empty categoricalFeaturesInfo indicates all features are continuous.
model = DecisionTree.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo={},
impurity='gini', maxDepth=5, maxBins=32) # Evaluate model on test instances and compute test error
predictions = model.predict(testData.map(lambda x: x.features))
labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions)
testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
print('Test Error = ' + str(testErr))
print('Learned classification tree model:')
print(model.toDebugString()) # Save and load model
model.save(sc, "target/tmp/myDecisionTreeClassificationModel")
sameModel = DecisionTreeModel.load(sc, "target/tmp/myDecisionTreeClassificationModel")
class pyspark.mllib.tree.DecisionTree[source]
Learning algorithm for a decision tree model for classification or regression.
New in version 1.1.0.
- classmethod trainClassifier(data, numClasses, categoricalFeaturesInfo, impurity='gini', maxDepth=5, maxBins=32, minInstancesPerNode=1, minInfoGain=0.0)[source]
-
Train a decision tree model for classification.
Parameters: - data – Training data: RDD of LabeledPoint. Labels should take values {0, 1, ..., numClasses-1}.
- numClasses – Number of classes for classification.
- categoricalFeaturesInfo – Map storing arity of categorical features. An entry (n -> k) indicates that feature n is categorical with k categories indexed from 0: {0, 1, ..., k-1}.
- impurity – Criterion used for information gain calculation. Supported values: “gini” or “entropy”. (default: “gini”)
- maxDepth – Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1 means 1 internal node + 2 leaf nodes). (default: 5)
- maxBins – Number of bins used for finding splits at each node. (default: 32)
- minInstancesPerNode – Minimum number of instances required at child nodes to create the parent split. (default: 1)
- minInfoGain – Minimum info gain required to create a split. (default: 0.0)
Returns: DecisionTreeModel.
Example usage:
>>> from numpy import array
>>> from pyspark.mllib.regression import LabeledPoint
>>> from pyspark.mllib.tree import DecisionTree
>>>
>>> data = [
... LabeledPoint(0.0, [0.0]),
... LabeledPoint(1.0, [1.0]),
... LabeledPoint(1.0, [2.0]),
... LabeledPoint(1.0, [3.0])
... ]
>>> model = DecisionTree.trainClassifier(sc.parallelize(data), 2, {})
>>> print(model)
DecisionTreeModel classifier of depth 1 with 3 nodes>>> print(model.toDebugString())
DecisionTreeModel classifier of depth 1 with 3 nodes
If (feature 0 <= 0.0)
Predict: 0.0
Else (feature 0 > 0.0)
Predict: 1.0 >>> model.predict(array([1.0]))
1.0
>>> model.predict(array([0.0]))
0.0
>>> rdd = sc.parallelize([[1.0], [0.0]])
>>> model.predict(rdd).collect()
[1.0, 0.0]-
摘自:https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.tree.DecisionTree
python spark 决策树 入门demo的更多相关文章
- Spark快速入门 - Spark 1.6.0
Spark快速入门 - Spark 1.6.0 转载请注明出处:http://www.cnblogs.com/BYRans/ 快速入门(Quick Start) 本文简单介绍了Spark的使用方式.首 ...
- Spark快速入门
Spark 快速入门 本教程快速介绍了Spark的使用. 首先我们介绍了通过Spark 交互式shell调用API( Python或者scala代码),然后演示如何使用Java, Scala或者P ...
- spark streaming 入门例子
spark streaming 入门例子: spark shell import org.apache.spark._ import org.apache.spark.streaming._ sc.g ...
- 转-Python自然语言处理入门
Python自然语言处理入门 原文链接:http://python.jobbole.com/85094/ 分享到:20 本文由 伯乐在线 - Ree Ray 翻译,renlytime 校稿.未经许 ...
- Spark高速入门指南(Quick Start Spark)
版权声明:本博客已经不再更新.请移步到Hadoop技术博客:https://www.iteblog.com https://blog.csdn.net/w397090770/article/detai ...
- [转] Spark快速入门指南 – Spark安装与基础使用
[From] https://blog.csdn.net/w405722907/article/details/77943331 Spark快速入门指南 – Spark安装与基础使用 2017年09月 ...
- storm入门demo
一.storm入门demo的介绍 storm的入门helloworld有2种方式,一种是本地的,另一种是远程. 本地实现: 本地写好demo之后,不用搭建storm集群,下载storm的相关jar包即 ...
- Python学习--01入门
Python学习--01入门 Python是一种解释型.面向对象.动态数据类型的高级程序设计语言.和PHP一样,它是后端开发语言. 如果有C语言.PHP语言.JAVA语言等其中一种语言的基础,学习Py ...
- Python简单爬虫入门三
我们继续研究BeautifulSoup分类打印输出 Python简单爬虫入门一 Python简单爬虫入门二 前两部主要讲述我们如何用BeautifulSoup怎去抓取网页信息以及获取相应的图片标题等信 ...
随机推荐
- php实时推送系统消息给客户端
在我们实际开发过程中,有些数据需要实时获取:比如erp系统中的订单信息,OA系统中的流程审批等都需要及时处理,这时我们就不能再使用http协议了:当然也可以使用轮询的机制.但是轮询请求中有大半是无用, ...
- Android使用charles抓包
1.下载并安状软件,官网在此: 2.前题条件,电脑和手机必须在同一网段 3.在Charles界面选择菜单 proxy->proxy settings 勾选"Enable transpa ...
- 浅谈ByteBuffer转换成byte[]时遇到的问题
有些时候我们要把ByteBuffer转换成byte[]来使用.于是很多时候会用以下代码来转换: ByteBuffer buf; .....(一些往buffer写数据的操作) byte[] bs= ne ...
- VTK初始化New返回Null问题
原文链接:http://www.cppblog.com/mythma/archive/2013/08/02/vtk-6-new-null.html 在使用VTK6.0时候,会遇到X::New()返回为 ...
- 06--谈谈:C++类的“包含”机制
谈谈:C++类的“包含”机制 本人在学习Qt的时候发现了一个非常有趣的现象.有很多函数的调用方法都写成了如下的形式: object.func().func2(); 这令小弟着实不懂.在上面这段代码 ...
- Arduino LM35温度检测
一. 接线原理图 二.实物图 三.代码例子
- 如何将一个已有的项目托管到github或是码云上?git的配置
场景一:已有的一个项目,要把它托管到Git上去,步骤和方法如下: 方法一: ①在工程的路径下 : git init 建一个裸仓库. ②远程仓库地址 :将本地的仓库和远程仓库关联 git remote ...
- PAT_A1087#All Roads Lead to Rome
Source: PAT A1087 All Roads Lead to Rome (30 分) Description: Indeed there are many different tourist ...
- eas之视图冻结与解冻
// 冻结视图 table.getViewManager().freeze(verticalIndex, horizonIndex); //冻结视图:该方法在table还没显示的时候使用,也就是该方法 ...
- PKU 1019 Number Sequence(模拟,思维)
题目 以下思路参考自discuss:http://poj.org/showmessage?message_id=176353 /*我的思路: 1.将长串数分成一个个部分,每个部分是从1到x的无重复的数 ...