Decision Tree
Decision Tree builds classification or regression models in the form of a tree structure. It break down dataset into smaller and smaller subsets while an associated decision tree in incrementally developed at the same time.
Decision Tree learning use top-down recursive method. The basic idea is to construct one tree with a fastest declines of information entropy, the entropy value of all instance in each leaf nodes is zero. Each internal node of the tree corresponding to an attribute, and each leaf node corresponding to a class label.
Advantages:
- Decision is easy to explain. It results in a set of rules. It is the same approach as humans generally follow while making decisions.
- Interpretation of a complex Decision Tree can be simplified into visualization.It can be understood by everyone.
- It almost have no hyper-parameter.
Infomation Gain
- The entropy is:
- By the information entropy, we can calculate their Experience entropy:
where:
- we can also calculate their Experience conditions entropy:
- By the information entropy, we can calculate their information gain:
- Information gain ratio:
- Gini index:
For binary classification:
For binary classification and on the condition of feature A:
Three Building Algorithm
- ID3: maximizing information gain
- C4.5: maximizing the ratio of information gain
- CART
- Regression Tree: minimizing the square error.
- Classification Tree: minimizing the Gini index.
Decision Tree Algorithm Pseudocode
- Place the best attribute of the dataset at the root of tree.The way to the selection of best attribute is shown in Three Building Algorithm above.
- Split the train set into subset by the best attribute.
- Repeat Step 1 and Step 2 on each subset until you find leaf nodes in all the branches of the tree.
Random Forest
Random Forest classifiers work around that limitation by creating a whole bunch of decision trees(hence 'forest'), each trained on random subsets of training samples
(bagging
, drawn with replacement) and features
(drawn without replacement).Make the decision tree work together to get result.
In one word, it build on CART with randomness.
Randomness 1:train the tree on the subsets of train set selected by
bagging
(sampling with replacement).
- Randomness 2:train the tree on the subsets of features(sampling without replacement). For example, select 10 features from 100 features in dataset.
Randomness 3:add new feature by low-dimensional projection.
后记
装逼想用英文写博客,想借此锻炼自己的写作能力,无情打脸( ̄ε(# ̄)
Ref:https://clyyuanzi.gitbooks.io/julymlnotes/content/rf.html
http://www.saedsayad.com/decision_tree.htm
http://dataaspirant.com/2017/01/30/how-decision-tree-algorithm-works/
统计学习方法(李航)
Decision Tree的更多相关文章
- Spark MLlib - Decision Tree源码分析
http://spark.apache.org/docs/latest/mllib-decision-tree.html 以决策树作为开始,因为简单,而且也比较容易用到,当前的boosting或ran ...
- 决策树Decision Tree 及实现
Decision Tree 及实现 标签: 决策树熵信息增益分类有监督 2014-03-17 12:12 15010人阅读 评论(41) 收藏 举报 分类: Data Mining(25) Pyt ...
- Gradient Boosting Decision Tree学习
Gradient Boosting Decision Tree,即梯度提升树,简称GBDT,也叫GBRT(Gradient Boosting Regression Tree),也称为Multiple ...
- 使用Decision Tree对MNIST数据集进行实验
使用的Decision Tree中,对MNIST中的灰度值进行了0/1处理,方便来进行分类和计算熵. 使用较少的测试数据测试了在对灰度值进行多分类的情况下,分类结果的正确率如何.实验结果如下. #Te ...
- Sklearn库例子1:Sklearn库中AdaBoost和Decision Tree运行结果的比较
DisCrete Versus Real AdaBoost 关于Discrete 和Real AdaBoost 可以参考博客:http://www.cnblogs.com/jcchen1987/p/4 ...
- 用于分类的决策树(Decision Tree)-ID3 C4.5
决策树(Decision Tree)是一种基本的分类与回归方法(ID3.C4.5和基于 Gini 的 CART 可用于分类,CART还可用于回归).决策树在分类过程中,表示的是基于特征对实例进行划分, ...
- OpenCV码源笔记——Decision Tree决策树
来自OpenCV2.3.1 sample/c/mushroom.cpp 1.首先读入agaricus-lepiota.data的训练样本. 样本中第一项是e或p代表有毒或无毒的标志位:其他是特征,可以 ...
- GBDT(Gradient Boosting Decision Tree)算法&协同过滤算法
GBDT(Gradient Boosting Decision Tree)算法参考:http://blog.csdn.net/dark_scope/article/details/24863289 理 ...
- Gradient Boost Decision Tree(&Treelink)
http://www.cnblogs.com/joneswood/archive/2012/03/04/2379615.html 1. 什么是Treelink Treelink是阿里集团内部 ...
- (转)Decision Tree
Decision Tree:Analysis 大家有没有玩过猜猜看(Twenty Questions)的游戏?我在心里想一件物体,你可以用一些问题来确定我心里想的这个物体:如是不是植物?是否会飞?能游 ...
随机推荐
- HDU 树型dp
HDU 4123 Bob's Race 题意:定义每个点的值为它到树上最远点的距离,每次询问q,回答最长的极值差小于等于q且编号连续的一段点的长度. 题解:求距离两次dp,求极值ST表+尺取法. HD ...
- 添加cordova-plugin-file-opener2后,打包出错
原文:添加cordova-plugin-file-opener2后,打包出错 报错如下: * What went wrong: Execution failed for task ':app:proc ...
- yii2.0预先处理方法
public function beforeAction($action){ return $action; }
- Leetcode 617 Merge Two Binary Trees 二叉树
题意: 给定两棵树,将两棵树合并成一颗树 输入 Tree 1 Tree 2 1 2 / \ / \ 3 2 1 3 / \ \ 5 4 7 输出 合并的树 3 / \ 4 5 / \ \ 5 4 7 ...
- 机器学习:scikit-learn 做笑脸识别 (SVM, KNN, Logisitc regression)
scikit-learn 是 Python 非常强大的一个做机器学习的包,今天介绍scikit-learn 里几个常用的分类器 SVM, KNN 和 logistic regression,用来做笑脸 ...
- Enabling granular discretionary access control for data stored in a cloud computing environment
Enabling discretionary data access control in a cloud computing environment can begin with the obtai ...
- Qt 绘制平滑曲线
本文介绍在 Qt 中绘制平滑曲线的实现,调用下面的函数 SmoothCurveGenerator::generateSmoothCurve(points) 即可.默认曲线的 2 个顶点之间被分割为 1 ...
- 三:Java之Applet
首先我要说的是Applet是一种应用程序,它是一种由JAVA编写的小应用程序,通常这样的应用程序都像他的名字一样,是一个非常小的程序,或许有些朋友就会问了,那么它是用来干什么的呢?JAVA程序就是JA ...
- 人猿方案Ubuntu这些软件的安装
鄙人程序员一枚,Android开发,常年使用Ubuntu(主要是买不起Mac.O(∩_∩)O哈哈~).分享一下自己使用的那些软件.假设你有什么好的软件.欢迎与我交流. 输入法:開始的时候是用的fcit ...
- Android Studio右键选项中没有Git?
从Git clone一个Project并打开后,都会习惯性的像使用Eclipse一样,选中project右键,选择Git的相应版本号控制选项. 例如以下图,你仅仅看到了svn. 怎样配置才干在右键选项 ...