Your Prediction Gets As Good As Your Data

May 5, 2015 by Kazem

In the past, we have seen software engineers and data scientists assume that they can keep increasing their prediction accuracy by improving their machine learning algorithm. Here, we want to approach the classification problem from a different angle where we recommend data scientists should analyze the distribution of their data first to measure information level in data. This approach can givesus an upper bound for how far one can improve the accuracy of a predictive algorithm and make sure our optimization efforts are not wasted!

Entropy and Information

In information theory, mathematician have developed a few useful techniques such as entropy to measure information level in data in process. Let's think of a random coin with a head probability of 1%.

If one filps such a coin, we will get more information when we see the head event since it's a rare event compared to tail which is more likely to happen. We can formualte the amount of information in a random variable with the negative logarithm of the event probability. This captures the described intuition. Mathmatician also formulated another measure called entropy by which they capture the average information in a random process in bits. Below we have shown the entropy formula for a discrete random variable:



For the first example, let's assume we have a coin with P(H)=0% and P(T)=100%. We can compute the entropy of the coin as follows:



For the second example, let's consider a coin where P(H)=1% and P(T)=1-P(H)=99%. Plugging numbers one can find that the entropy of such a coin is:



Finally, if the coin has P(H) = P(T) = 0.5 (i.e. a fair coin), its entropy is calculated as follows:



Entropy and Predictability

So, what these examples tell us? If we have a coin with head probability of zero, the coin's entropy is zero meaning that the average information in the coin is zero. This makes sense because flipping such a coin always comes as tail. Thus, the prediction accuracy is 100%. In other words, when the entropy is zero, we have the maximum predictibility.

In the second example, head probability is not zero but still very close to zero which again makes the coin to be very predictable with a low entropy.

Finally, in the last example we have 50/50 chance of seeing head/tail events which maximizes the entropy and consequently minimizes the predictability. In words, one can show that a fair coin has the meaximum entropy of 1 bit making the prediction as good as a random guess.

Kullback–Leibler divergence

As last example, it's important to give another example of how we can borrow ideas from information theory to measure the distance between two probability distributions. Let's assume we are modeling two random processes by their pmf's: P(.) and Q(.). One can use entropy measure to compute the distance between two pmf's as follows:



Above distance function is known as KL divergence which measures the distance of Q's pmf from P's pmf. The KL divergence can come handy in various problems such as NLP problems where we'd like to measure the distance between two sets of data (e.g. bag of words).

Wrap-up

In this post, we showed that the entropy from information theory provides a way to measure how much information exists in our data. We also highlighted the inverse relationship between the entropy and the predictability. This shows that we can use the entropy measure to calculate an upper bound for the accuracy of the prediction problem in hand.

Feel free to share with us if you have any comments or questions in the comment section below.

You can also reach us at info@AIOptify.com

Your Prediction Gets As Good As Your Data的更多相关文章

  1. Lessons Learned from Developing a Data Product

    Lessons Learned from Developing a Data Product For an assignment I was asked to develop a visual ‘da ...

  2. A Brief Review of Supervised Learning

    There are a number of algorithms that are typically used for system identification, adaptive control ...

  3. 微软职位内部推荐-Software Engineer II

    微软近期Open的职位: Job Description Group: Search Technology Center Asia (STCA)/Search Ads Title: SDEII-Sen ...

  4. 在opencv3中实现机器学习算法之:利用最近邻算法(knn)实现手写数字分类

    手写数字digits分类,这可是深度学习算法的入门练习.而且还有专门的手写数字MINIST库.opencv提供了一张手写数字图片给我们,先来看看 这是一张密密麻麻的手写数字图:图片大小为1000*20 ...

  5. 在opencv3中的机器学习算法练习:对OCR进行分类

    OCR (Optical Character Recognition,光学字符识别),我们这个练习就是对OCR英文字母进行识别.得到一张OCR图片后,提取出字符相关的ROI图像,并且大小归一化,整个图 ...

  6. Libsvm:脚本(subset.py、grid.py、checkdata.py) | MATLAB/OCTAVE interface | Python interface

    1.脚本 This directory includes some useful codes: 1. subset selection tools. (子集抽取工具) subset.py 2. par ...

  7. Nagios工作原理

    图解Nagios的工作原理 Nagios的主动模式和被动模式 被动模式:就如同上图所显示的那样,客户端起nrpe进程,服务端通过check_nrpe插件向客户端发送命令,客户端根据服务端的指示来调用相 ...

  8. 学习笔记TF020:序列标注、手写小写字母OCR数据集、双向RNN

    序列标注(sequence labelling),输入序列每一帧预测一个类别.OCR(Optical Character Recognition 光学字符识别). MIT口语系统研究组Rob Kass ...

  9. OpenCV OpenGL手写字符识别

    另外一篇文章地址:这个比较详细,但是程序略显简单,现在这个程序是比较复杂的 http://blog.csdn.net/wangyaninglm/article/details/17091901 整个项 ...

随机推荐

  1. MYSQL数据库封装类

    <?phpclass DBDA{    public $host="localhost";    public $uid="root";    publi ...

  2. C#代码分析--阅读程序,回答问题

    阅读下面程序,请回答如下问题: 问题1:这个程序要找的是符合什么条件的数? 问题2:这样的数存在么?符合这一条件的最小的数是什么? 问题3:在电脑上运行这一程序,你估计多长时间才能输出第一个结果?时间 ...

  3. Alpha 冲刺九

    团队成员 051601135 岳冠宇 051604103 陈思孝 031602629 刘意晗 031602248 郑智文 031602234 王淇 会议照片 项目燃尽图 项目进展 完善各自部分 项目描 ...

  4. JMeter学习笔记——认识JMeter(1)

    拿到一个自动化测试工具,我们第一步就应该了解它能提供我们哪方面的功能(最直接的方法就是从官网获取),接下来就是简单的对这个工具进行“功能测试”了,当然这里的功能测试不是让你找它存在的bug,而是让自己 ...

  5. 顺序表的C、C++实现

    一个线性表表实现,函数声明放在 line_list.h 头文件汇总,函数定义放在line_list.c 中,main.c 用来测试各个函数. 1.文件 line_list.h // line_list ...

  6. [转帖]linux namespace 和cgroup lxc

    https://blog.csdn.net/xiaoliuliu2050/article/details/53443863 5.1 linux namespace 和cgroup lxc 2016年1 ...

  7. 微信小程序填坑之旅一(接入)

    一.小程序简介 小程序是什么? 首先“程序”这两个字我们不陌生.看看你手机上的各个软件,那就是程序.平时的程序是直接跑在我们原生的操作系统上面的.小程序是间接跑在原生系统上的.因为它嵌入在微信中,受微 ...

  8. wordpress WP-PageNavi分页

    1.安装WP-PageNavi分页插件: 这个就没什么好介绍直接安装插件. 2.在需要分页的页面按下面的方式加上相应代码: 插入的位置在以 <?php if (have_posts()) : ? ...

  9. 如何让搜索引擎抓取AJAX内容?

    越来越多的网站,开始采用"单页面结构"(Single-page application). 整个网站只有一张网页,采用Ajax技术,根据用户的输入,加载不同的内容. 这种做法的好处 ...

  10. 【笔记】自学ST表笔记

    自学ST表笔记 说实话原先QBXT学的ST表忘的差不多了吧...... 我重新自学巩固一下(回忆一下) 顺便把原先一些思想来源的原博发上来 一.ST表简介 ST表,建表时间\(O(n\cdot log ...