RNN(Recurrent Neural Network)的几个难点
1. vanish of gradient
RNN的error相对于某个时间点t的梯度为:
\(\frac{\partial E_t}{\partial W}=\sum_{k=1}^{t}\frac{\partial E_t}{\partial y_t}\frac{\partial y_t}{\partial h_i}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W}\) (公式1),
其中\(h\)是hidden node的输出,\(y_t\)是网络在t时刻的output,\(W\)是hidden nodes 到hidden nodes的weight,而\(\frac{\partial h_t}{\partial h_k}\)是导数在时间段[k,t]上的链式展开,这段时间可能很长,会造成vanish或者explosion gradiant。将\(\frac{\partial h_t}{\partial h_k}\)沿时间展开:\(\frac{\partial h_t}{\partial h_k}=\prod_{j=k+1}^{t}\frac{\partial h_j}{\partial h_{j-1}}=\prod_{j=k+1}^{t}W^T \times diag [\frac{\partial\sigma(h_{j-1})}{\partial h_{j-1}}]\)。上式中的diag矩阵是个什么鬼?我来举个例子,你就明白了。假设现在要求解\(\frac{\partial h_5}{\partial h_4}\),回忆向前传播时\(h_5\)是怎么得到的:\(h_5=W\sigma(h_4)+W^{hx}x_4\),则\(\frac{\partial h_5}{\partial h_4}=W\frac{\partial \sigma(h_4)}{\partial h_4}\),注意到\(\sigma(h_4)\)和\(h_4\)都是向量(维度为D),所以\(\frac{\partial \sigma(h_4)}{\partial h_4}\)是Jacobian矩阵也即:\(\frac{\partial \sigma(h_4)}{\partial h_4}=\) \(\begin{bmatrix} \frac{\partial\sigma_1(h_{41})}{\partial h_{41}}&\cdots&\frac{\partial\sigma_1(h_{41})}{\partial h_{4D}} \\ \vdots&\cdots&\vdots \\ \frac{\partial\sigma_D(h_{4D})}{\partial h_{41}}&\cdots&\frac{\partial\sigma_D(h_{4D})}{\partial h_{4D}}\end{bmatrix}\),明显的,非对角线上的值都是0。这是因为sigmoid logistic function \(\sigma\)是element-wise的操作。
后面推导vanish或者explosion gradiant的过程就很简单了,我就不写了,请参考http://cs224d.stanford.edu/lecture_notes/LectureNotes4.pdf 中的公式(14)往后部分。
2. weight shared (tied) 时, the gradient of tied weight = sum of gradient of individual weights
举个例子你就明白了:假设有向前传播\(y=F[W_1f(W_2x)]\), 且weights \(W_1\) \(W_2\) tied, 现在要求gradient \(\frac{\partial y}{\partial W}\)
办法一:
先求gradient \(\frac{\partial F[]}{\partial W_2} = F'[]f() \)
再求gradient \(\frac{\partial F[]}{\partial W_1} = F'[] (W_2f'()x) \)
将上两式相加后得,\(F'[]f()+F'[] (W_2f'()x)=F'[](f()+W_2f'()x)\)
假设weights \(W_1\) \(W_2\) tied,则上式=\(F'[](f()+Wf'()x) = \frac{\partial y}{\partial W} \)
办法二:
现在我们换个办法,在假设weights \(W_1\) \(W_2\) tied的基础上,直接求gradient
\(\frac{\partial y}{\partial W} = F'[]( \frac{\partial Wf()}{\partial W} + W \frac{\partial f()}{\partial W} ) = F'[](f()+Wf'()x) \)
可见,两种方法的结果是一样的。所以,当权重共享时,关于权重的梯度=两个不同权重梯度的和。
3. LSTM & Gated Recurrent units 是如何避免vanish的?
To understand this, you will have to go through some math. The most accessible article wrt recurrent gradient problems IMHO is Pascanu's ICML2013 paper [1].
A summary: vanishing/exploding gradient comes from the repeated application of the recurrent weight matrix [2]. That the spectral radius of the recurrent weight matrix is bigger than 1 makes exploding gradients possible (it is a necessary condition), while a spectral radius smaller than 1 makes it vanish, which is a sufficient condition.
Now, if gradients vanish, that does not mean that all gradients vanish. Only some of them, gradient information local in time will still be present. That means, you might still have a non-zero gradient--but it will not contain long term information. That's because some gradient g + 0 is still g. (上文中公式1,因为是相加,所以有些为0,也不会引起全部为0)
If gradients explode, all of them do. That is because some gradient g + infinity is infinity.(上文中公式1,因为是相加,所以有些为无限大,会引起全部为无限大)
That is the reason why LSTM does not protect you from exploding gradients, since LSTM also uses a recurrent weight matrix(h(t) = o(t) ◦ tanh(c(t))?), not only internal state-to-state connections( c(t) = f (t) ◦ ˜c(t−1) +i(t) ◦ ˜c(t) h(t)). Successful LSTM applications typically use gradient clipping.
LSTM overcomes the vanishing gradient problem, though. That is because if you look at the derivative of the internal state at T to the internal state at T-1, there is no repeated weight application. The derivative actually is the value of the forget gate. And to avoid that this becomes zero, we need to initialise it properly in the beginning.
That makes it clear why the states can act as "a wormhole through time", because they can bridge long time lags and then (if the time is right) "re inject" it into the other parts of the net by opening the output gate.
[1] Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." arXiv preprint arXiv:1211.5063 (2012).
[2] It might "vanish" also due to saturating nonlinearities, but that is sth that can also happen in shallow nets and can be overcome with more careful weight initialisations.
ref: Recursive Deep Learning for Natural Language Processing and Computer Vision.pdf
CS224D-3-note bp.pdf
未完待续。。。
RNN(Recurrent Neural Network)的几个难点的更多相关文章
- Recurrent Neural Network系列2--利用Python,Theano实现RNN
作者:zhbzz2007 出处:http://www.cnblogs.com/zhbzz2007 欢迎转载,也请保留这段声明.谢谢! 本文翻译自 RECURRENT NEURAL NETWORKS T ...
- Recurrent Neural Network系列3--理解RNN的BPTT算法和梯度消失
作者:zhbzz2007 出处:http://www.cnblogs.com/zhbzz2007 欢迎转载,也请保留这段声明.谢谢! 这是RNN教程的第三部分. 在前面的教程中,我们从头实现了一个循环 ...
- 循环神经网络(Recurrent Neural Network,RNN)
为什么使用序列模型(sequence model)?标准的全连接神经网络(fully connected neural network)处理序列会有两个问题:1)全连接神经网络输入层和输出层长度固定, ...
- Recurrent neural network (RNN) - Pytorch版
import torch import torch.nn as nn import torchvision import torchvision.transforms as transforms # ...
- 4.5 RNN循环神经网络(recurrent neural network)
自己开发了一个股票智能分析软件,功能很强大,需要的点击下面的链接获取: https://www.cnblogs.com/bclshuai/p/11380657.html 1.1 RNN循环神经网络 ...
- Recurrent Neural Network系列1--RNN(循环神经网络)概述
作者:zhbzz2007 出处:http://www.cnblogs.com/zhbzz2007 欢迎转载,也请保留这段声明.谢谢! 本文翻译自 RECURRENT NEURAL NETWORKS T ...
- Recurrent Neural Network(循环神经网络)
Reference: Alex Graves的[Supervised Sequence Labelling with RecurrentNeural Networks] Alex是RNN最著名变种 ...
- 循环神经网络(RNN, Recurrent Neural Networks)介绍(转载)
循环神经网络(RNN, Recurrent Neural Networks)介绍 这篇文章很多内容是参考:http://www.wildml.com/2015/09/recurrent-neur ...
- Recurrent Neural Network系列4--利用Python,Theano实现GRU或LSTM
yi作者:zhbzz2007 出处:http://www.cnblogs.com/zhbzz2007 欢迎转载,也请保留这段声明.谢谢! 本文翻译自 RECURRENT NEURAL NETWORK ...
- Recurrent Neural Network[Content]
下面的RNN,LSTM,GRU模型图来自这里 简单的综述 1. RNN 图1.1 标准RNN模型的结构 2. BiRNN 3. LSTM 图3.1 LSTM模型的结构 4. Clockwork RNN ...
随机推荐
- Android进阶(十五)socket通信——聊天室
想做一个聊天室,花费了将近一天的时间,各种错误.讲解知识点之前,絮叨几句:动手能力还是很重要的,有时看似简单的一个问题,当你真正着手去解决的时候就有可能会遇到各种各样的问题,原因之一就是因为你的知识储 ...
- Java图形界面编程生成exe文件
1. 先将代码打成jar,然后使用exe4j转成exe ext4j下载 链接:http://pan.baidu.com/s/1kTCIZtX 密码:pvj1 打开EXE4J Advanced Opti ...
- android升级后错误:Unable to execute dex: java.nio.BufferOverflowException.Check
Android SDK Tools升级为22.3,Android SDK Platform-tools 升级为19后,编译工程出现错误: Unable to execute dex: java.nio ...
- com.android.dex.DexException: Multiple dex files define(jar包重复引用) 错误解决
前段时间开始转入Android studio,不料果真使用时候遇到些错误,在此记下! 出现这个错误往往是在libs目录下有个jar包,然后在gradle文件中又引用了,即: 共同引用了. 解决方法: ...
- Win8 HTML5与JS编程学习笔记(一)
微软的Visual Studio提供了多种构成win8应用的方式,其中最让我感到激动的是基于网页设计语言的开发模式,它提供了结合HTML5与Javascript来开发应用的方法,通过这种方法进行开发, ...
- EM实现
以下是实验设计 设计一个一维的数据,14个数据,7个成一组,一个高斯分布,整体数据隐含了2个高斯分布. 系统最初给出第一个数属于z0的概率0.9,最后一个数属于在z1的概率0.9,其余数据不可判定. ...
- 《java入门第一季》之面向对象(如何使用帮助文档)
1:打开帮助文档 2:点击显示,找到索引,看到输入框 3:知道你要找谁?以Scanner举例 4:在输入框里面输入Scanner,然后回车 5:看包 java.lang包下的类不需要导入包,其他的全部 ...
- SpriteBuilder中使用TrueType字体的一些障碍
在实践中,有一些小的陷阱和障碍可能阻止你使用一般的TrueType字体. 第一个,必须要有一个有效的字体文件.在Finder中双击该.ttf文件,应该会打开Font Book app,显示一个象形符号 ...
- 【一天一道LeetCode】#6 ZigZag Conversion
一天一道LeetCode系列 (一)题目 The string "PAYPALISHIRING" is written in a zigzag pattern on a given ...
- 运行Myeclipse时,如何删除IVM窗口
windows------>preference------>run/debug------->lauching--------->percpectives,改成never,n ...