(转) Deep learning architecture diagrams
FastML
Machine learning made easy
Deep learning architecture diagrams
2016-09-30
Like a wild stream after a wet season in African savanna diverges into many smaller streams forming lakes and puddles, deep learning has diverged into a myriad of specialized architectures. Each architecture has a diagram. Here are some of them.
Neural networks are conceptually simple, and that’s their beauty. A bunch of homogenous, uniform units, arranged in layers, weighted connections between them, and that’s all. At least in theory. Practice turned out to be a bit different. Instead of feature engineering, we now have architecture engineering, as described by Stephen Merrity:
The romanticized description of deep learning usually promises that the days of hand crafted feature engineering are gone - that the models are advanced enough to work this out themselves. Like most advertising, this is simultaneously true and misleading.
Whilst deep learning has simplified feature engineering in many cases, it certainly hasn’t removed it. As feature engineering has decreased, the architectures of the machine learning models themselves have become increasingly more complex. Most of the time, these model architectures are as specific to a given task as feature engineering used to be.
To clarify, this is still an important step. Architecture engineering is more general than feature engineering and provides many new opportunities. Having said that, however, we shouldn’t be oblivious to the fact that where we are is still far from where we intended to be.
Not quite as bad as doings of architecture astronauts, but not too good either.
An example of architecture specific to a given task
LSTM diagrams
How to explain those architectures? Naturally, with a diagram. A diagram will make it all crystal clear.
Let’s first inspect the two most popular types of networks these days, CNN and LSTM. You’ve already seen a convnet diagram, so turning to the iconic LSTM:
It’s easy, just take a closer look:
As they say, in mathematics you don’t understand things, you just get used to them.
Fortunately, there are good explanations, for example Understanding LSTM Networks and Written Memories: Understanding, Deriving and Extending the LSTM.
LSTM still too complex? Let’s try a simplified version, GRU (Gated Recurrent Unit). Trivial, really.
Especially this one, called minimal GRU.
More diagrams
Various modifications of LSTM are now common. Here’s one, called deep bidirectional LSTM:
DB-LSTM, PDF
The rest are pretty self-explanatory, too. Let’s start with a combination of CNN and LSTM, since you have both under your belt now:
Convolutional Residual Memory Network, 1606.05262
Dynamic NTM, 1607.00036
Evolvable Neural Turing Machines, PDF
Recurrent Model Of Visual Attention, 1406.6247
Unsupervised Domain Adaptation By Backpropagation, 1409.7495
Deeply Recursive CNN For Image Super-Resolution, 1511.04491
This diagram of multilayer perceptron with synthetic gradients scores high on clarity:
MLP with synthetic gradients, 1608.05343
Every day brings more. Here’s a fresh one, again from Google:
Google’s Neural Machine Translation System, 1609.08144
And Now for Something Completely Different
Drawings from the Neural Network ZOO are pleasantly simple, but, unfortunately, serve mostly as eye candy. For example:
ESM, ESN and ELM
These look like not-fully-connected perceptrons, but are supposed to represent a Liquid State Machine, an Echo State Network, and an Extreme Learning Machine.
How does LSM differ from ESN? That’s easy, it has green neuron with triangles. But how does ESN differ from ELM? Both have blue neurons.
Seriously, while similar, ESN is a recursive network and ELM is not. And this kind of thing should probably be visible in an architecture diagram.
Posted by Zygmunt Z. 2016-09-30 basics, neural-networks
« Factorized convolutional neural networks, AKA separable convolutions
Comments
Recent Posts
- Deep learning architecture diagrams
- Factorized convolutional neural networks, AKA separable convolutions
- How to make those 3D data visualizations
- Adversarial validation, part two
- ^one weird trick for training char-^r^n^ns
- Adversarial validation, part one
- Coming out
Follow @fastml for notifications about new posts.
- Status updating...
Also check out @fastml_extra for things related to machine learning and data science in general.
GitHub
Most articles come with some code. We push it to Github.
Cubert
Visualize your data in interactive 3D, as described here.
Copyright © 2016 - Zygmunt Z. - Powered by Octopress
(转) Deep learning architecture diagrams的更多相关文章
- 15 cvpr An Improved Deep Learning Architecture for Person Re-Identification
http://www.umiacs.umd.edu/~ejaz/ * 也是同时学习feature和metric * 输入一对图片,输出是否是同一个人 * 包含了一个新的层: include a lay ...
- Deep Learning in a Nutshell: History and Training
Deep Learning in a Nutshell: History and Training This series of blog posts aims to provide an intui ...
- 深度学习材料:从感知机到深度网络A Deep Learning Tutorial: From Perceptrons to Deep Networks
In recent years, there’s been a resurgence in the field of Artificial Intelligence. It’s spread beyo ...
- 【Deep Learning】genCNN: A Convolutional Architecture for Word Sequence Prediction
作者:Mingxuan Wang.李航,刘群 单位:华为.中科院 时间:2015 发表于:acl 2015 文章下载:http://pan.baidu.com/s/1bnBBVuJ 主要内容: 用de ...
- Why GEMM is at the heart of deep learning
Why GEMM is at the heart of deep learning I spend most of my time worrying about how to make deep le ...
- 【深度学习Deep Learning】资料大全
最近在学深度学习相关的东西,在网上搜集到了一些不错的资料,现在汇总一下: Free Online Books by Yoshua Bengio, Ian Goodfellow and Aaron C ...
- (转) Awesome - Most Cited Deep Learning Papers
转自:https://github.com/terryum/awesome-deep-learning-papers Awesome - Most Cited Deep Learning Papers ...
- (转) Deep Learning Research Review Week 2: Reinforcement Learning
Deep Learning Research Review Week 2: Reinforcement Learning 转载自: https://adeshpande3.github.io/ad ...
- deep learning 的综述
从13年11月初开始接触DL,奈何boss忙or 各种问题,对DL理解没有CSDN大神 比如 zouxy09等 深刻,主要是自己觉得没啥进展,感觉荒废时日(丢脸啊,这么久....)开始开文,即为记录自 ...
随机推荐
- 【转】CentOS 6.5安装pyspider过程记录
原文地址:http://blog.sina.com.cn/s/blog_48c95a190102wczx.html 1.根据pyspider官方推荐的安装方法,使用pip命令直接安装pyspider ...
- PHP数组去重..............过滤字段
$test_data = M('hot'); //实例化数据表 $data = $test_data->Distinct(true)->field('descriprion')->o ...
- PHP中字符串的连接和换行
PHP中字符串的连接使用点,不是加号.换行\n需要用双引号括起来,不能用单引号. eg. <?php print_r("hello"."\n"); pri ...
- Server.UrlEncode()方法 空格转换成了+而非%20
在ASP.NET MVC 的Control类里提供了该方法.该方法可以很方便的对字符串进行url编码,但小猪今天却发现其将空格编码后变成了“+”而非JavaScript采用的encodeURIComp ...
- CentOS修改默认编码为UTF-8,使java程序字符集默认为UTF-8
java程序在本地接受php的utf8字符串好好的,到了服务器就行了. 解决,修改vi /etc/sysconfig/i18n,修改之后ssh断开,重连后KILL你的java. LANG=" ...
- .Net中的加密与解密
加密与解密概述 加密与解密属于数据安全的范畴.在消息传输时,通过对消息进行特殊编码(加密),建立一种安全的交流方式,使得只有发送者所期望的接收者能够理解(解密).这里我们定义一个场景:发送方,接收方, ...
- Hadoop 2.2.0学习笔记20131210
伪分布式单节点安装执行pi失败: [root@server- ~]# ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples ...
- 初学者SQL语句介绍
初学者SQL语句介绍 1.用 Select 子句检索记录 Select 子句是每一个检索数据的查询核心.它告诉数据库引擎返回什么字段. Select 子句的常见形式是: S ...
- leetcode 日记 3sumclosest java
思路一为遍历: public int thirdSolution(int[] nums, int target) { int result = nums[0] + nums[1] + nums[2]; ...
- (转)oracle 存储过程 带游标作为OUT参数输出
(转)oracle 存储过程 带游标作为OUT参数输出 存储过程返回OUT参数的游标 例子. 包中带过程 要自己定义一个type [cur_name] is ref cursor游标,返回的时候就直接 ...