Your Prediction Gets As Good As Your Data
Your Prediction Gets As Good As Your Data
May 5, 2015 by Kazem
In the past, we have seen software engineers and data scientists assume that they can keep increasing their prediction accuracy by improving their machine learning algorithm. Here, we want to approach the classification problem from a different angle where we recommend data scientists should analyze the distribution of their data first to measure information level in data. This approach can givesus an upper bound for how far one can improve the accuracy of a predictive algorithm and make sure our optimization efforts are not wasted!
Entropy and Information
In information theory, mathematician have developed a few useful techniques such as entropy to measure information level in data in process. Let's think of a random coin with a head probability of 1%.
If one filps such a coin, we will get more information when we see the head event since it's a rare event compared to tail which is more likely to happen. We can formualte the amount of information in a random variable with the negative logarithm of the event probability. This captures the described intuition. Mathmatician also formulated another measure called entropy by which they capture the average information in a random process in bits. Below we have shown the entropy formula for a discrete random variable:
For the first example, let's assume we have a coin with P(H)=0% and P(T)=100%. We can compute the entropy of the coin as follows:
For the second example, let's consider a coin where P(H)=1% and P(T)=1-P(H)=99%. Plugging numbers one can find that the entropy of such a coin is:
Finally, if the coin has P(H) = P(T) = 0.5 (i.e. a fair coin), its entropy is calculated as follows:
Entropy and Predictability
So, what these examples tell us? If we have a coin with head probability of zero, the coin's entropy is zero meaning that the average information in the coin is zero. This makes sense because flipping such a coin always comes as tail. Thus, the prediction accuracy is 100%. In other words, when the entropy is zero, we have the maximum predictibility.
In the second example, head probability is not zero but still very close to zero which again makes the coin to be very predictable with a low entropy.
Finally, in the last example we have 50/50 chance of seeing head/tail events which maximizes the entropy and consequently minimizes the predictability. In words, one can show that a fair coin has the meaximum entropy of 1 bit making the prediction as good as a random guess.
Kullback–Leibler divergence
As last example, it's important to give another example of how we can borrow ideas from information theory to measure the distance between two probability distributions. Let's assume we are modeling two random processes by their pmf's: P(.) and Q(.). One can use entropy measure to compute the distance between two pmf's as follows:
Above distance function is known as KL divergence which measures the distance of Q's pmf from P's pmf. The KL divergence can come handy in various problems such as NLP problems where we'd like to measure the distance between two sets of data (e.g. bag of words).
Wrap-up
In this post, we showed that the entropy from information theory provides a way to measure how much information exists in our data. We also highlighted the inverse relationship between the entropy and the predictability. This shows that we can use the entropy measure to calculate an upper bound for the accuracy of the prediction problem in hand.
Feel free to share with us if you have any comments or questions in the comment section below.
You can also reach us at info@AIOptify.com
Your Prediction Gets As Good As Your Data的更多相关文章
- Lessons Learned from Developing a Data Product
Lessons Learned from Developing a Data Product For an assignment I was asked to develop a visual ‘da ...
- A Brief Review of Supervised Learning
There are a number of algorithms that are typically used for system identification, adaptive control ...
- 微软职位内部推荐-Software Engineer II
微软近期Open的职位: Job Description Group: Search Technology Center Asia (STCA)/Search Ads Title: SDEII-Sen ...
- 在opencv3中实现机器学习算法之:利用最近邻算法(knn)实现手写数字分类
手写数字digits分类,这可是深度学习算法的入门练习.而且还有专门的手写数字MINIST库.opencv提供了一张手写数字图片给我们,先来看看 这是一张密密麻麻的手写数字图:图片大小为1000*20 ...
- 在opencv3中的机器学习算法练习:对OCR进行分类
OCR (Optical Character Recognition,光学字符识别),我们这个练习就是对OCR英文字母进行识别.得到一张OCR图片后,提取出字符相关的ROI图像,并且大小归一化,整个图 ...
- Libsvm:脚本(subset.py、grid.py、checkdata.py) | MATLAB/OCTAVE interface | Python interface
1.脚本 This directory includes some useful codes: 1. subset selection tools. (子集抽取工具) subset.py 2. par ...
- Nagios工作原理
图解Nagios的工作原理 Nagios的主动模式和被动模式 被动模式:就如同上图所显示的那样,客户端起nrpe进程,服务端通过check_nrpe插件向客户端发送命令,客户端根据服务端的指示来调用相 ...
- 学习笔记TF020:序列标注、手写小写字母OCR数据集、双向RNN
序列标注(sequence labelling),输入序列每一帧预测一个类别.OCR(Optical Character Recognition 光学字符识别). MIT口语系统研究组Rob Kass ...
- OpenCV OpenGL手写字符识别
另外一篇文章地址:这个比较详细,但是程序略显简单,现在这个程序是比较复杂的 http://blog.csdn.net/wangyaninglm/article/details/17091901 整个项 ...
随机推荐
- SVN回退版本
执行svn up 命令 保证当前本地版本是最新的版本. svn up 执行svn log 命令,查看历史修改,确定需要回复的版本,如果想要对比2个不同版本的文件差异 可以使用命令 svn diff - ...
- 机器视觉及图像处理系列之二(C++,VS2015)——图像级的人脸识别(1)
接上一篇,一切顺利的话,你从github上clone下来的整个工程应该已经成功编译并生成dll和exe文件了:同时,ImageMagic程序亦能够打开并编辑图像了,如此,证明接下来的操练你不会有任何障 ...
- EOS 权限管理之-权限的使用
首先,跟大家说声抱歉,由于之前一直在准备EOS上线的一些工作,所以,很长时间没有更新内容.今天正好有时间,也想到了一些题材,就来说一下这个话题.本文完全是个人见解,如有不当之处,欢迎指出. 前提回顾: ...
- PAT甲题题解-1024. Palindromic Number (25)-大数运算
大数据加法给一个数num和最大迭代数k每次num=num+num的倒序,判断此时的num是否是回文数字,是则输出此时的数字和迭代次数如果k次结束还没找到回文数字,输出此时的数字和k 如果num一开始是 ...
- 20181204-2 Final发布
此作业要求参见:https://edu.cnblogs.com/campus/nenu/2018fall/homework/2476 小组介绍 组长:付佳 组员:张俊余 李文涛 孙赛佳 田良 于洋 段 ...
- Redis学习笔记之入门基础知识——简介
非关系型数据库,存储的数据类型:字符串(STRING).列表(LIST).集合(SET).散列表(HASH).有序集合(ZSET) 持久化:时间点转储(point-in-time-dump)(快照). ...
- 【读书笔记】Linux内核设计与实现(第十八章)
18.1 准备开始 需要: 1.一个确定的bug.但是,大部分bug通常都不是行为可靠定义明确的. 2.一个藏匿bug的内核版本. 18.2 内核中的bug bug发作时的症状: 明白无误的错误代码( ...
- 实验---反汇编一个简单的C程序(杨光)
反汇编一个简单的C程序 攥写人:杨光 学号:20135233 ( *原创作品转载请注明出处*) ( 学习课程:<Linux内核分析>MOOC课程http://mooc.study.163 ...
- Linux内核分析(第三周)
构造一个简单的linux系统menuOS. 一.简介 1.两把宝剑:中断-上下文的切换(保存现场和恢复现场) 进程-上下文的切换 2.linux-3.18.6 arch/x86目录下的代码是我们重点关 ...
- php插入中文数据到MySQL乱码
事情是这样的:我在本地的测试成功了,放到服务器测试,发现服务器的数据库里的中文竟然乱码了. 我进行了以下几步基本的做法: PHP文件改为utf-8的格式. 加入header("Content ...