Gradient descent

Batch Gradient Decent, Mini-batch gradient descent, Stochastic gradient descent

  

  

  

还有很多比gradient decent 更优化的算法,在了解这些算法前,需要先理解  Exponentially weighted averages 这个概念

Exponentially weighted average 是一种计算平均值的方法,非常省storage 和 memory, 但是不是很精确。 然后引出一个bias correction 的概念,就是为了能使得 Exponentially weighted average 更加精确.

  

momentum (or called Gradient descent with momentum)

传统的Gradient descent 算法有如下图所示的问题 - 每次迭代都会来回跳动,不直接指向optimum, 在没有做feature scaling 的时候尤其明显。所以引出一个修正的算法 - Gradient descent with momentum.

  

  

RMSprop

目的和上面讲到的Momentum是一样的,就是使得每次迭代都尽量指向optimum而不是来回跳动. 算法实现如下. RMSprop带来的好处是迭代更快,和可以选用更大的learning rate.

  

Adam optimation algorithm:

  结合了Momentum 和 RMSprop 两种算法. Adam stands for Adaptive mement estimation.

  

  

Learning rate decay

why? to reduce the oscillation near the central point.

  

有哪些实现方式呢?

  

Local optima and saddle point

在大型神经网络里,saddle point 可能比local optima更常见.

  

  

Ref:

Coursera, Deep leaning, Andrew Ng

Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Optimization algorithms的更多相关文章

  1. 《Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization》课堂笔记

    Lesson 2 Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization 这篇文章其 ...

  2. [C4] Andrew Ng - Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

    About this Course This course will teach you the "magic" of getting deep learning to work ...

  3. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Initialization)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Initialization Welcome to the first assignment of "Improving D ...

  4. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Gradient Checking)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Gradient Checking Welcome to the final assignment for this week! In ...

  5. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week1, Assignment(Regularization)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. Regularization Welcome to the second assignment of this week. Deep ...

  6. Coursera Deep Learning 2 Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - week2, Assignment(Optimization Methods)

    声明:所有内容来自coursera,作为个人学习笔记记录在这里. 请不要ctrl+c/ctrl+v作业. Optimization Methods Until now, you've always u ...

  7. 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第一周(Practical aspects of Deep Learning) —— 4.Programming assignments:Gradient Checking

    Gradient Checking Welcome to this week's third programming assignment! You will be implementing grad ...

  8. 吴恩达《深度学习》-课后测验-第二门课 (Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization)-Week 1 - Practical aspects of deep learning(第一周测验 - 深度学习的实践)

    Week 1 Quiz - Practical aspects of deep learning(第一周测验 - 深度学习的实践) \1. If you have 10,000,000 example ...

  9. 吴恩达《深度学习》-第二门课 (Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization)-第一周:深度学习的实践层面 (Practical aspects of Deep Learning) -课程笔记

    第一周:深度学习的实践层面 (Practical aspects of Deep Learning) 1.1 训练,验证,测试集(Train / Dev / Test sets) 创建新应用的过程中, ...

  10. 课程二(Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization),第三周(Hyperparameter tuning, Batch Normalization and Programming Frameworks) —— 2.Programming assignments

    Tensorflow Welcome to the Tensorflow Tutorial! In this notebook you will learn all the basics of Ten ...

随机推荐

  1. A1018. Public Bike Management

    There is a public bike service in Hangzhou City which provides great convenience to the tourists fro ...

  2. Fiddler 只取所需

    Fiddler每次打开的时候都会打开十多个会话,期望只想抓取自己想要的请求.   1)User Filters:启用过滤器 2)在Filers面板中勾选“Use Filters”,并在Hosts区域, ...

  3. ElasticSearch6.5.0【Java客户端之TransportClient】

    说明 TransportClient:网上流传最多的客户端,目前最新版本 Java REST Client:官方推荐的客户端, 官方:我们要在Elasticsearch 7.0的版本中不赞成使用Tra ...

  4. (选择不相交区间)今年暑假不AC hdu2037

    今年暑假不AC Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 65536/32768 K (Java/Others)Total Sub ...

  5. Linux 中用 dd 命令来测试硬盘读写速度

    dd 是 Linux/UNIX 下的一个非常有用的命令,作用是用指定大小的块拷贝一个文件,并在拷贝的同时进行指定的转换. dd 命令通用语法格式如下: dd if=path/to/input_file ...

  6. Win7无法安装程序提示Installer integrity check has failed的解决方法

    最近小明在Win7系统中下载了一款软件,但是在下载的时候弹出了NSIS Error:Installer integrity check has failed.Common causer include ...

  7. TF报错解决

    一.import tensorflow #h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubd ...

  8. OS + Windows 10 / office excel vlookup / CredSSP

    s https://support.microsoft.com/zh-cn/help/10749/windows-10-find-product-key 查找 Windows 7 或 Windows ...

  9. jdk和jvm基本介绍

    一.JDK和JRE? 在刚入门java开发的时候,第一步都要从官网下载JDK来帮助开发,下载下来安装之后看到安装目录结构如下 bin目录:存放Java的编译器.解释器等工具(可执行文件). db目录: ...

  10. 设计模式---接口隔离模式之门面模式(Façade)

    前提:接口隔离模式 在组建构建过程中,某些接口之间直接的依赖常常会带来很多问题.甚至根本无法实现.采用添加一层间接接口(稳定的),来隔离本来相互紧密关联的接口是一种常见的解决方案. 典型模式: 门面模 ...