Deep Learning: Assuming a deep neural network is properly regulated, can adding more layers actually make the performance degrade?

I found this to be really puzzling. A deeper NN is supposed to be more powerful or at least equal to a shallower NN. I have already used dropout to prevent overfitting. How can the performance be degraded?
Yoshua's Answer
 

Yoshua Bengio, My lab has been one of the three that started the deep learning approach, bac...

Upvoted by Prateek Tandon, Robotics and Strong Artificial Intelligence Researcher• Paul King, Computational Neuroscientist, Technology Entrepreneur • Jack Rae,Google DeepMind Research Engineer
 
If you do not change the size of the layers and just add more layers, capacity should increase, so you could be overfitting. However, you should check whether training error increases or decreases. If it increases (which is also very plausible), it means that adding the layer made the optimization harder, with the optimization methods and initialization that you are using.  That could also explain your problem. However, if training error decreases and test error increases, you are overfitting.

Deep Learning: Assuming a deep neural network is properly regulated, can adding more layers actually make the performance degrade?的更多相关文章

  1. 【RS】Automatic recommendation technology for learning resources with convolutional neural network - 基于卷积神经网络的学习资源自动推荐技术

    [论文标题]Automatic recommendation technology for learning resources with convolutional neural network ( ...

  2. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...

  3. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Initialization)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Initialization Welcome to the first assignment of "Improving D ...

  4. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Regularization)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Regularization Welcome to the second assignment of this week. Deep ...

  5. [C1W1] Neural Networks and Deep Learning - Introduction to Deep Learning

    第一周:深度学习引言(Introduction to Deep Learning) 欢迎(Welcome) 深度学习改变了传统互联网业务,例如如网络搜索和广告.但是深度学习同时也使得许多新产品和企业以 ...

  6. Coursera, Deep Learning 2, Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Course

    Train/Dev/Test set Bias/Variance Regularization  有下面一些regularization的方法. L2 regularation drop out da ...

  7. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Assignment(Optimization Methods)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. 请不要ctrl+c/ctrl+v作业. Optimization Methods Until now, you've always u ...

  8. 吴恩达《深度学习》-课后测验-第一门课 (Neural Networks and Deep Learning)-Week 3 - Shallow Neural Networks（第三周测验 - 浅层神 经网络）

    Week 3 Quiz - Shallow Neural Networks(第三周测验 - 浅层神经网络) \1. Which of the following are true? (Check al ...

  9. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week3, Hyperparameter tuning, Batch Normalization and Programming Frameworks

    Tuning process 下图中的需要tune的parameter的先后顺序, 红色>黄色>紫色,其他基本不会tune. 先讲到怎么选hyperparameter, 需要随机选取(sa ...

随机推荐

  1. phpcms后台部分修改

    1.后台登陆前提示信息取消及成功后提示信息取消.    (1)后台登陆前提示信息取消               phpcms\modules\admin\classes\admin.class.ph ...

  2. 发布阿里云OSS for phpcmsV9整合教程

    说明:这个算不上是插件,因为没有安装包,需要手工修改代码. 还有一点就是后台发布文章时上传的附件还是会保存在你的服务器上,基于以下原因: 1.个人的需求是前台页面需要使用thumb函数生成多个缩略图大 ...

  3. 表格控件表头栏目（Column）与数据表头步

    不用手工增加栏目的列,也就是Column,由数据库的查询结果自动创建. 用的是Delphi2010,安装了Dev,用CxGrid显示数据库查询结果.用什么控件没有关键,道理相同的.

  4. POJ--1416

    #include <stdio.h> #include <stdlib.h> ]={};//代表有没有切割的数组 ;//输入的要被切割的数字 ]={};//切完输出的数组成的数 ...

  5. kettle插入/更新

    1.数据库环境 --------------------实时表 ),Info )); ,'张启山','长沙'); ,'尹新月','长沙'); ,'二月红','长沙'); --------------- ...

  6. Linux 虚拟机和物理机配互信出现无法连接

    配置文件位置:[root@hank-yoon data]# vi /etc/ssh/sshd_configPermitRootLogin yes 在物理机中,装完系统,默认情况下PermitRootL ...

  7. Android--PullToRefreshListView 的简单使用

    原文:  http://blog.csdn.net/lmj623565791/article/details/38238749; pull-to-refresh对ListView进行了封装,叫做:Pu ...

  8. poj 2887 Big String

    题目连接 http://poj.org/problem?id=2887 Big String Description You are given a string and supposed to do ...

  9. Java之XStream之下划线问题

    XStream将Java对象序列为XML, 如果对象属性带有下划线,输出到XML时会转成"__"双下划. 网上很多资料说是Bug, XStream开发的大神没有哪么笨吧? 查了一下 ...

  10. golang构造函数与转换函数

    golang的每种类型, 凡是用type定义的类型, 其类型名既是其构造函数,也是其转换函数. 其中,构造函数后眼{...}, 转换函数后跟(...)