Neural Networks

We will use the following diagram to denote a single neuron:

This "neuron" is a computational unit that takes as input x1,x2,x3 (and a +1 intercept term), and outputs , where is called the activation function. In these notes, we will choose to be the sigmoid function:

Thus, our single neuron corresponds exactly to the input-output mapping defined by logistic regression.

Although these notes will use the sigmoid function, it is worth noting that another common choice for f is the hyperbolic tangent, or tanh, function:

Here are plots of the sigmoid and tanh functions:

Finally, one identity that'll be useful later: If f(z) = 1 / (1 + exp( − z)) is the sigmoid function, then its derivative is given by f'(z) = f(z)(1 − f(z))

sigmoid 函数 或 tanh 函数都可用来完成非线性映射

Neural Network model

A neural network is put together by hooking together many of our simple "neurons," so that the output of a neuron can be the input of another. For example, here is a small neural network:

In this figure, we have used circles to also denote the inputs to the network. The circles labeled "+1" are called bias units, and correspond to the intercept term. The leftmost layer of the network is called the input layer, and the rightmost layer the output layer (which, in this example, has only one node). The middle layer of nodes is called the hidden layer, because its values are not observed in the training set. We also say that our example neural network has 3 input units (not counting the bias unit), 3 hidden units, and 1 output unit.

Our neural network has parameters (W,b) = (W(1),b(1),W(2),b(2)), where we write to denote the parameter (or weight) associated with the connection between unit j in layer l, and unit i in layerl + 1. (Note the order of the indices.) Also, is the bias associated with unit i in layer l + 1.

We will write to denote the activation (meaning output value) of unit i in layer l. For l = 1, we also use to denote the i-th input. Given a fixed setting of the parameters W,b, our neural network defines a hypothesis hW,b(x) that outputs a real number. Specifically, the computation that this neural network represents is given by:

每层都是线性组合 + 非线性映射

In the sequel, we also let denote the total weighted sum of inputs to unit i in layer l, including the bias term (e.g., ), so that .

Note that this easily lends itself to a more compact notation. Specifically, if we extend the activation function to apply to vectors in an element-wise fashion (i.e., f([z1,z2,z3]) = [f(z1),f(z2),f(z3)]), then we can write the equations above more compactly as:

We call this step forward propagation.

Backpropagation Algorithm

for a single training example (x,y), we define the cost function with respect to that single example to be:

This is a (one-half) squared-error cost function. Given a training set of m examples, we then define the overall cost function to be:

J(W,b;x,y) is the squared error cost with respect to a single example; J(W,b) is the overall cost function, which includes the weight decay term.

Our goal is to minimize J(W,b) as a function of W and b. To train our neural network, we will initialize each parameter and each to a small random value near zero (say according to a Normal(0,ε2) distribution for some small ε, say 0.01), and then apply an optimization algorithm such as batch gradient descent.Finally, note that it is important to initialize the parameters randomly, rather than to all 0's. If all the parameters start off at identical values, then all the hidden layer units will end up learning the same function of the input (more formally, will be the same for all values of i, so that for any input x). The random initialization serves the purpose of symmetry breaking.

One iteration of gradient descent updates the parameters W,b as follows:

The two lines above differ slightly because weight decay is applied to W but not b.

The intuition behind the backpropagation algorithm is as follows. Given a training example (x,y), we will first run a "forward pass" to compute all the activations throughout the network, including the output value of the hypothesis hW,b(x). Then, for each node i in layer l, we would like to compute an "error term" that measures how much that node was "responsible" for any errors in our output.

For an output node, we can directly measure the difference between the network's activation and the true target value, and use that to define (where layer nl is the output layer). For hidden units, we will compute based on a weighted average of the error terms of the nodes that uses as an input. In detail, here is the backpropagation algorithm:

  • 1,Perform a feedforward pass, computing the activations for layers L2, L3, and so on up to the output layer .

2,For each output unit i in layer nl (the output layer), set

For

For each node i in layer l, set

4,Compute the desired partial derivatives, which are given as:

We will use "" to denote the element-wise product operator (denoted ".*" in Matlab or Octave, and also called the Hadamard product), so that if , then . Similar to how we extended the definition of to apply element-wise to vectors, we also do the same for (so that ).

The algorithm can then be written:

  1. 1,Perform a feedforward pass, computing the activations for layers , , up to the output layer , using the equations defining the forward propagation steps

2,For the output layer (layer ), set

 

3,For

Set
 

4,Compute the desired partial derivatives:

 

Implementation note: In steps 2 and 3 above, we need to compute for each value of . Assuming is the sigmoid activation function, we would already have stored away from the forward pass through the network. Thus, using the expression that we worked out earlier for , we can compute this as .

Finally, we are ready to describe the full gradient descent algorithm. In the pseudo-code below, is a matrix (of the same dimension as ), and is a vector (of the same dimension as ). Note that in this notation, "" is a matrix, and in particular it isn't " times ." We implement one iteration of batch gradient descent as follows:

  1. 1,Set , (matrix/vector of zeros) for all .
  2. 2,For to ,
    1. Use backpropagation to compute and .
    2. Set .
    3. Set .
  3. 3,Update the parameters:

Sparse Autoencoder(一)的更多相关文章

  1. Deep Learning 1_深度学习UFLDL教程:Sparse Autoencoder练习(斯坦福大学深度学习教程)

    1前言 本人写技术博客的目的,其实是感觉好多东西,很长一段时间不动就会忘记了,为了加深学习记忆以及方便以后可能忘记后能很快回忆起自己曾经学过的东西. 首先,在网上找了一些资料,看见介绍说UFLDL很不 ...

  2. (六)6.5 Neurons Networks Implements of Sparse Autoencoder

    一大波matlab代码正在靠近.- -! sparse autoencoder的一个实例练习,这个例子所要实现的内容大概如下:从给定的很多张自然图片中截取出大小为8*8的小patches图片共1000 ...

  3. UFLDL实验报告2:Sparse Autoencoder

    Sparse Autoencoder稀疏自编码器实验报告 1.Sparse Autoencoder稀疏自编码器实验描述 自编码神经网络是一种无监督学习算法,它使用了反向传播算法,并让目标值等于输入值, ...

  4. 七、Sparse Autoencoder介绍

    目前为止,我们已经讨论了神经网络在有监督学习中的应用.在有监督学习中,训练样本是有类别标签的.现在假设我们只有一个没有带类别标签的训练样本集合  ,其中  .自编码神经网络是一种无监督学习算法,它使用 ...

  5. CS229 6.5 Neurons Networks Implements of Sparse Autoencoder

    sparse autoencoder的一个实例练习,这个例子所要实现的内容大概如下:从给定的很多张自然图片中截取出大小为8*8的小patches图片共10000张,现在需要用sparse autoen ...

  6. 【DeepLearning】Exercise:Sparse Autoencoder

    Exercise:Sparse Autoencoder 习题的链接:Exercise:Sparse Autoencoder 注意点: 1.训练样本像素值需要归一化. 因为输出层的激活函数是logist ...

  7. Sparse AutoEncoder简介

    1. AutoEncoder AutoEncoder是一种特殊的三层神经网络, 其输出等于输入:\(y^{(i)}=x^{(i)}\), 如下图所示: 亦即AutoEncoder想学到的函数为\(f_ ...

  8. Exercise:Sparse Autoencoder

    斯坦福deep learning教程中的自稀疏编码器的练习,主要是参考了   http://www.cnblogs.com/tornadomeet/archive/2013/03/20/2970724 ...

  9. DL二(稀疏自编码器 Sparse Autoencoder)

    稀疏自编码器 Sparse Autoencoder 一神经网络(Neural Networks) 1.1 基本术语 神经网络(neural networks) 激活函数(activation func ...

  10. sparse autoencoder

    1.autoencoder autoencoder的目标是通过学习函数,获得其隐藏层作为学习到的新特征. 从L1到L2的过程成为解构,从L2到L3的过程称为重构. 每一层的输出使用sigmoid方法, ...

随机推荐

  1. vcenter server appliance 5.5 管理中心 linux 版本的部署

    本文选自通过ovf模板部署: 需要下面两个文件即可: 打开vsphere client 登录到你安装了exsi5.5的物理机上面: 点击文件---> 部署ovf模板  ---->选择你的模 ...

  2. 去除windows编辑文本中的回车符

    情景描述: 最近,huskiesir的朋友遇到了一个很奇葩的问题.那就是他在windows上搭建了一个http服务,把A脚本放在了上面并用linux去下载和执行,但是在执行的时候出现了问题,在linu ...

  3. 以替换为主的疯狂填词、sub()介绍

    去年接到一个任务,一直给拖到了今天,再这么下去可不行,今天我就要让你们看看我的厉害 任务是这样的:创建一个程序,读入文本文件,并让用户在该文本出现ADJECTIVE .NOUN.ADVERB或VERB ...

  4. CMSIS-RTOS 时间管理之虚拟定时器Virtual Timers

    虚拟定时器Virtual Timers CMSIS-RTOS API里有几个向下计数的虚拟定时器,它们实现计数完成时用户的回调功能.每个定时器都可以配置成单次计数或重复计数模式,它们可以在定义定时器结 ...

  5. DBCP2配置详细说明(中文翻译)

    http://blog.csdn.net/kerafan/article/details/50382998 common-dbcp2数据库连接池参数说明 由于commons-dbcp所用的连接池出现版 ...

  6. LiquiBase注释

    LiquiBase的注释 <createTable tableName="Role_" remarks="角色表">             < ...

  7. 从零開始学android&lt;SlidingDrawer 隐式抽屉.三十三.&gt;

    SlidingDrawer是一种抽屉型的组件.当用户选择打开此抽屉之后,会得到一些能够使用的"程序集".这样当一个界面要摆放多个组件的时候,使用此组件就能够非常好的解决布局空间紧张 ...

  8. 翻翻git之---炫酷的自己定义翻滚View TagCloudView

    转载请注明出处:王亟亟的大牛之路 周一好,又到了每周最困的一天.近期都被啮齿类动物搞的累死,废话不多,今天上一个自己定义的ViewGroup实现一个3D球形集合. 效果图: 效果还不错,能够作为短小文 ...

  9. Android设计模式(十二)--抽象工厂模式

    问题: 抽象工厂模式,是一个,狠恶心的模式,那么这个模式在Android有没实用到过呢? 1.定义: 抽象工厂模式:为创建一组相关或者是相互依赖的对象提供一个接口,而不须要指定他们的详细类. 2.使用 ...

  10. 关于APP上架制作二维码相关

    1.安卓版本APP上架并生成二维码问题:安卓版本上架国内市场,这个情况比较复杂一些,比如百度,网址是以上传APP生成的一个编号来进行的,每次升级更新后都发生了变化,也就相当于每次升级后网址发生改变(比 ...