In machine learning, is more data always better than better algorithms?

No. There are times when more data helps, there are times when it doesn't.

Probably one of the most famous quotes defending the power of data is that of Google's Research Director Peter Norvig claiming that "We don’t have better algorithms. We just have more data.". This quote is usually linked to the article on "The Unreasonable Effectiveness of Data", co-authored by Norvig  himself (you should probably be able to find the pdf on the web although the originalis behind the IEEE paywall). The last nail on the coffin of better models is when Norvig is misquoted as saying that "All models are wrong, and you don't need them anyway" (read here for the author's own clarifications on how he was misquoted).

The effect that Norvig et. al were referring to in their article, had already been captured years before in the famous paper by Microsoft Researchers Banko and Brill [2001] "Scaling to Very Very Large Corpora for Natural Language Disambiguation". In that paper, the authors included the plot below.

That figure shows that, for the given problem, very different algorithms perform virtually the same. however, adding more examples (words) to the training set monotonically increases the accuracy of the model.

So, case closed, you might think. Well... not so fast. The reality is that both Norvig's assertions and Banko and Brill's paper are right... in a context. But, they are now and again misquoted in contexts that are completely different than the original ones. But, in order to understand why, we need to get slightly technical.  (I don't plan on giving a full machine learning tutorial in this post. If you don't understand what I explain below, read my answer to How do I learn machine learning?)

Variance or Bias?

The basic idea is that there are two possible (and almost opposite) reasons a model might not perform well.

In the first case, we might have a model that is too complicated for the amount of data we have. This situation, known ashigh variance, leads to model overfitting. We know that we are facing a high variance issue when the training error is much lower than the test error. High variance problems can be addressed by reducing the number of features, and... yes, by increasing the number of data points. So, what kind of models were Banko & Brill's, and Norvig dealing with? Yes, you got it right: high variance. In both cases, the authors were  working on language models in which roughly every word in the vocabulary makes a feature. These are models with many features as compared to the training examples. Therefore, they are likely to overfit. And, yes, in this case adding more examples will help.

But, in the opposite case, we might have a model that is too simple to explain the data we have. In that case, known as high bias, adding more data will not help.  See below a plot of a real production system at Netflix and its performance as we add more training examples.

So, no, more data does not always help. As we have just seen there can be many cases in which adding more examples to our training set will not improve the model performance.

More features to the rescue

If you are with me so far, and you have done your homework in understanding high variance and high bias problems, you might be thinking that I have deliberately left something out of the discussion. Yes, high bias models will not benefit from more training examples, but they might very well benefit from more features. So, in the end, it is all about adding "more" data, right? Well, again, it depends.

Let's take the Netflix Prize, for example. Pretty early on in the game, there wasa blog post by serial entrepreneur and Stanford professor Anand Rajaraman commenting on the use of extra features to solve the problem. The post explains how a team of students got an improvement on the prediction accuracy by adding content features from IMDB.

In retrospect, it is easy to criticize the post for making a gross over-generalization from a single data point. Even more, the follow-up postreferences SVD as one of the "complex" algorithms not worth trying because it limits the ability of scaling up to larger number of features. Clearly, Anand's students did not win the Netflix Prize, and they probably now realize that SVD did have a major role in the winning entry.

As a matter of fact, many teams showed later that adding content features from IMDB or the like to an optimized algorithm had little to no improvement. Some of the members of the Gravity team, one of the top contenders for the Prize, published a detailed paper in which they showed how those content-based features would add no improvement to the highly optimized collaborative filtering matrix factorization approach. The paper was entitled "Recommending New Movies: Even a Few Ratings Are More Valuable Than Metadata".

To be fair, the title of the paper is also an over-generalization. Content-based features (or different features in general) might be able to improve accuracy in many cases. But, you get my point again: More data does not always help.

Better Data != More Data (Added this section in response to a comment)

It is important to point out that, in my opinion, better data is always better. There is no arguing against that. So any effort you can direct towards "improving" your data is always well invested. The issue is that better data does not mean more data. As a matter of fact, sometimes it might mean less!

Think of data cleansing or outlier removal as one trivial illustration of my point. But, there are many other examples that are more subtle. For example, I have seen people invest a lot of effort in implementing distributed Matrix Factorization when the truth is that they could have probably gotten by with sampling their data and gotten to very similar results. In fact, doing some form of smart sampling on your population the right way (e.g. using stratified sampling) can get you to better results than if you used the whole unfiltered data set.

The End of the Scientific Method?

Of course, whenever there is a heated debate about a possible paradigm change, there are people like Malcolm Gladwell or Chris Anderson that make a living out of heating it even more (don't get me wrong, I am a fan of both, and have read most of their books). In this case, Anderson picked on some of Norvig's comments, and misquoted them in an article entitled: "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete".

The article explains several examples of how the abundance of data helps people and companies take decision without even having to understand the meaning of the data itself. As Norvig himself points out in his rebuttal, Anderson has a few points right, but goes above and beyond to try to make them. And the result is a set of false statements, starting from the title: the data deluge does not make the scientific method obsolete. I would argue it is rather the other way around.

Data Without a Sound Approach = Noise

So, am I trying to make the point that the Big Data revolution is only hype? No way. Having more data, both in terms of more examples or more features, is a blessing. The availability of data enables more and better insights and applications. More data indeed enables better approaches. More than that, itrequires better approaches.

In summary, we should dismiss simplistic voices that proclaim the uselessness of theory or models, or the triumph of data over these. As much as data is needed, so are good models and theory that explains them. But, overall, what we need is good approaches that help us understand how to interpret data, models, and the limitations of both in order to produce the best possible output.

In other words, data is important. But, data without a sound approach becomes noise.

(Note: This answer is based on a post that I  previously published on my blog:More data or better models?)

In machine learning, is more data always better than better algorithms?的更多相关文章

  1. Coursera, Big Data 4, Machine Learning With Big Data (week 1/2)

    Week 1 Machine Learning with Big Data KNime - GUI based Spark MLlib - inside Spark CRISP-DM Week 2, ...

  2. [Machine Learning with Python] Data Preparation through Transformation Pipeline

    In the former article "Data Preparation by Pandas and Scikit-Learn", we discussed about a ...

  3. [Machine Learning with Python] Data Preparation by Pandas and Scikit-Learn

    In this article, we dicuss some main steps in data preparation. Drop Labels Firstly, we drop labels ...

  4. Coursera, Big Data 4, Machine Learning With Big Data (week 3/4/5)

    week 3 Classification KNN :基本思想是 input value 类似,就可能是同一类的 Decision Tree Naive Bayes Week 4 Evaluating ...

  5. 斯坦福大学公开课机器学习:machine learning system design | data for machine learning(数据量很大时,学习算法表现比较好的原理)

    下图为四种不同算法应用在不同大小数据量时的表现,可以看出,随着数据量的增大,算法的表现趋于接近.即不管多么糟糕的算法,数据量非常大的时候,算法表现也可以很好. 数据量很大时,学习算法表现比较好的原理: ...

  6. [Machine Learning with Python] Data Visualization by Matplotlib Library

    Before you can plot anything, you need to specify which backend Matplotlib should use. The simplest ...

  7. Machine Learning and Data Mining(机器学习与数据挖掘)

    Problems[show] Classification Clustering Regression Anomaly detection Association rules Reinforcemen ...

  8. 机器学习(Machine Learning)&深度学习(Deep Learning)资料(Chapter 2)

    ##机器学习(Machine Learning)&深度学习(Deep Learning)资料(Chapter 2)---#####注:机器学习资料[篇目一](https://github.co ...

  9. How do I learn machine learning?

    https://www.quora.com/How-do-I-learn-machine-learning-1?redirected_qid=6578644   How Can I Learn X? ...

随机推荐

  1. JAVA IO之管道流总结大全(转)

    要在文本框中显示控制台输出,我们必须用某种方法“截取”控制台流.换句话说,我们要有一种高效地读取写入到System.out和 System.err 所有内容的方法.如果你熟悉Java的管道流Piped ...

  2. park、unpark、ord 函数使用方法(转)

    park,unpark,ord这3个函数,在我们工作中,用到它们的估计不多. 我在最近一个工作中,因为通讯需要用到二进制流,然后接口用php接收.当时在处理时候,查阅不少资料.因为它们使用确实比较少, ...

  3. mysql_convert_table_format 批量修改表引擎

    [root@server-mysql bin]# mysql_convert_table_format --help Conversion of a MySQL tables to other sto ...

  4. SOA体系结构之基础培训教程-大纲篇

    引言: 最近受邀做了一个企业的SOA体系结构的内训,本文是内训课程的培训大纲,分享一下吧,希望大家能够喜欢.同时也想针对大纲中列出的内容对SOA架构体系做一次回顾,如果时间允许把完整的课件也想放上来共 ...

  5. android使用属性动画代替补间动画

    本文参考Android属性动画完全解析(上),初识属性动画的基本用法 android3.0之前一共有两种动画,分别是frame动画和tween动画,关于这两种动画如果不了解可以查看我之前的文章andr ...

  6. Spring Mvc和Mybatis的多数据库访问配置过程

    Spring Mvc 加Mybatis的多数据库访问源配置访问过程如下: 在applicationContext.xml进行配置 <?xml version="1.0" en ...

  7. [C# 基础知识系列]专题四:事件揭秘

    转自http://www.cnblogs.com/zhili/archive/2012/10/27/Event.html 引言: 前面几个专题对委托进行了详细的介绍的,然后我们在编写代码过程中经常会听 ...

  8. js - 多个函数动态加载

    //动态添加物流锁的IEMI列表. function createLi() { var r = '<s:property value="#session.locks"/> ...

  9. iOS 获取URL中的参数

    - (NSString *)getParamByName:(NSString *)name URLString:(NSString *)url { NSError *error; NSString * ...

  10. CSV文件规则

    CSV文件规则 1 开头是不留空,以行为单位.2 可含或不含列名,含列名则居文件第一行.3 一行数据不跨行,无空行.4 以半角逗号(即,)作分隔符,列为空也要表达其存在.5 列内容如存在半角逗号(即, ...