python spark 随机森林入门demo
class pyspark.mllib.tree.RandomForest[source]
Learning algorithm for a random forest model for classification or regression.
New in version 1.2.0.
- supportedFeatureSubsetStrategies = ('auto', 'all', 'sqrt', 'log2', 'onethird')
- classmethod trainClassifier(data, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy='auto', impurity='gini', maxDepth=4, maxBins=32, seed=None)[source]
-
Train a random forest model for binary or multiclass classification.
Parameters: - data – Training dataset: RDD of LabeledPoint. Labels should take values {0, 1, ..., numClasses-1}.
- numClasses – Number of classes for classification.
- categoricalFeaturesInfo – Map storing arity of categorical features. An entry (n -> k) indicates that feature n is categorical with k categories indexed from 0: {0, 1, ..., k-1}.
- numTrees – Number of trees in the random forest.
- featureSubsetStrategy – Number of features to consider for splits at each node. Supported values: “auto”, “all”, “sqrt”, “log2”, “onethird”. If “auto” is set, this parameter is set based on numTrees: if numTrees == 1, set to “all”; if numTrees > 1 (forest) set to “sqrt”. (default: “auto”)
- impurity – Criterion used for information gain calculation. Supported values: “gini” or “entropy”. (default: “gini”)
- maxDepth – Maximum depth of tree (e.g. depth 0 means 1 leaf node, depth 1 means 1 internal node + 2 leaf nodes). (default: 4)
- maxBins – Maximum number of bins used for splitting features. (default: 32)
- seed – Random seed for bootstrapping and choosing feature subsets. Set as None to generate seed based on system time. (default: None)
Returns: RandomForestModel that can be used for prediction.
Example usage:
>>> from pyspark.mllib.regression import LabeledPoint
>>> from pyspark.mllib.tree import RandomForest
>>>
>>> data = [
... LabeledPoint(0.0, [0.0]),
... LabeledPoint(0.0, [1.0]),
... LabeledPoint(1.0, [2.0]),
... LabeledPoint(1.0, [3.0])
... ]
>>> model = RandomForest.trainClassifier(sc.parallelize(data), 2, {}, 3, seed=42)
>>> model.numTrees()
3
>>> model.totalNumNodes()
7
>>> print(model)
TreeEnsembleModel classifier with 3 trees >>> print(model.toDebugString())
TreeEnsembleModel classifier with 3 trees Tree 0:
Predict: 1.0
Tree 1:
If (feature 0 <= 1.0)
Predict: 0.0
Else (feature 0 > 1.0)
Predict: 1.0
Tree 2:
If (feature 0 <= 1.0)
Predict: 0.0
Else (feature 0 > 1.0)
Predict: 1.0 >>> model.predict([2.0])
1.0
>>> model.predict([0.0])
0.0
>>> rdd = sc.parallelize([[3.0], [1.0]])
>>> model.predict(rdd).collect()
[1.0, 0.0]New in version 1.2.0.
摘自:https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.tree.DecisionTree
python spark 随机森林入门demo的更多相关文章
- Spark随机森林实现学习
前言 最近阅读了spark mllib(版本:spark 1.3)中Random Forest的实现,发现在分布式的数据结构上实现迭代算法时,有些地方与单机环境不一样.单机上一些直观的操作(递归),在 ...
- 用Python实现随机森林算法,深度学习
用Python实现随机森林算法,深度学习 拥有高方差使得决策树(secision tress)在处理特定训练数据集时其结果显得相对脆弱.bagging(bootstrap aggregating 的缩 ...
- 随机森林算法demo python spark
关键参数 最重要的,常常需要调试以提高算法效果的有两个参数:numTrees,maxDepth. numTrees(决策树的个数):增加决策树的个数会降低预测结果的方差,这样在测试时会有更高的accu ...
- Python中随机森林的实现与解释
使用像Scikit-Learn这样的库,现在很容易在Python中实现数百种机器学习算法.这很容易,我们通常不需要任何关于模型如何工作的潜在知识来使用它.虽然不需要了解所有细节,但了解机器学习模型是如 ...
- python实现随机森林、逻辑回归和朴素贝叶斯的新闻文本分类
实现本文的文本数据可以在THUCTC下载也可以自己手动爬虫生成, 本文主要参考:https://blog.csdn.net/hao5335156/article/details/82716923 nb ...
- Spark随机森林实战
package big.data.analyse.ml.randomforest import org.apache.spark.ml.Pipeline import org.apache.spark ...
- spark 随机森林算法案例实战
随机森林算法 由多个决策树构成的森林,算法分类结果由这些决策树投票得到,决策树在生成的过程当中分别在行方向和列方向上添加随机过程,行方向上构建决策树时采用放回抽样(bootstraping)得到训练数 ...
- python的随机森林模型调参
一.一般的模型调参原则 1.调参前提:模型调参其实是没有定论,需要根据不同的数据集和不同的模型去调.但是有一些调参的思想是有规律可循的,首先我们可以知道,模型不准确只有两种情况:一是过拟合,而是欠拟合 ...
- Python之随机森林实战
代码实现: # -*- coding: utf-8 -*- """ Created on Tue Sep 4 09:38:57 2018 @author: zhen &q ...
随机推荐
- “发布后tomcat中的classes目录为空”问题
办法:Project-clean,ok,问题解决.
- 【Oracle】删除手工创建的数据库
众所周知,DBCA创建的数据库可以通过DBCA命令删除,但是手工创建的数据库却不能用此方式删除,下面给出删除方式: SQL> startup mount exclusive SQL> al ...
- WEB笔记-让HTML5向下兼容的策略
//给新标签增加块级元素声明 article,aside,dialog,figure,fotter,header,legend,nav,section{display:block} //添加css兹瓷 ...
- Embedded之Makefile
1 Header files The header files are empty, so you can create them with touch: $ touch a.h $ touch b. ...
- java多线线程停止正确方法
//军队线程 //模拟作战双方的行为 public class ArmyRunnable implements Runnable { //volatile保证了线程可以正确的读取其他线程写入的值 // ...
- HTML编辑器ASP.NET
<%@ Page Language="C#" AutoEventWireup="true" CodeFile="kindeditor.aspx. ...
- PS通道的界面颜色设置
编辑--首选项---界面--界面---选项---(勾选)以彩色显示通道(彩色显示)或者不勾选(为黑白色显示)
- 重写servlet,可以获取不同的方法
public class BaseServlet extends HttpServlet { @Override public void service(HttpServletRequest requ ...
- Android 性能测试初探(六)
书接前文 Android 性能测试初探之功耗(五) 本节聊聊性能测试的最后一项- 流量,当然我所指的性能测试是针对大部分应用而言的,可能还有部分应用会关注网速.弱网之类的测试,但本系列文章都不去一一探 ...
- 算法87-----DAG有向无环图的拓扑排序
一.题目:课程排表---210 课程表上有一些课,是必须有修学分的先后顺序的,必须要求在上完某些课的情况下才能上下一门.问是否有方案修完所有的课程?如果有的话请返回其中一个符合要求的路径,否则返回[] ...