[专题论文阅读]【分布式DNN训练系统】 FireCaffe
FireCaffe
Forrest N. Iandola FireCaffe: near-linear acceleration of deep neural network training on computer clusters 2016.1
Problem statements from data scientists
4 key pain points summarized by Jeff Dean from Google:
1. DNN researchers and users want results of experiments quickly.
2. There is a “patience threshold”: No one wants to wait more than a few days or a week for result.
3. This significantly affects scale of problems that can be tackled.
4. We sometimes optimize for experiment turnaround time, rather than absolute minimal system resources for performing the experiments
Problem analysis
The speed and scalability of distributed algorithm are almost always limited by the overhead of communicating between servers; DNN training is not an exception to this rule.
So the design focuses on the communication enhancement, including:
1. Upgrade to high throughput interconnects, i.e. use high throughput interconnections like IB etc.
2. Decrease the data transmission volume while training, which includes:
a) Balance carefully between data parallelism and model parallelism
b) Increase batch size to reduce communication quantity. And identify hyperparameters suitable for large batch size.
c) Communication data quantity balance among nodes to avoid single point dependency.
Key take-aways
Parallelism Scheme: Model parallelism or Data Parallelism
Model parallelism
Each worker gets a subset of the model parameters, and the workers communication by exchanging data gradients and exchanging activations . and data quantity is:
Data parallelism
Each worker gets a subset of the batch, and then the workers communicate by exchanging weight gradient updates , where and data quantity is:
Convolution layer and fully connection layer have different characteristics in data/weight ratio. So they can use different parallelism schemes.
So a basic conclusion is: convolution layers can be fitted into data parallelism, and fc layers can be fitted into model parallelism.
Further more, for more advanced CNNs like GoogLeNet and ResNet etc., we can directly use data parallelism, as this paper is using.
Gradient Aggregation Scheme: Parameter Server or Reduction Tree
One picture to show how parameter server and reduction tree work in data parallelism.
Parameter Server
Parameter communication time with regard to worker number in parameter server scheme.
The communication time scales linearly as we increase the number of workers. single parameter server becomes scalability bottleneck.
Microsoft Adam and Google DistBelief relief this issue by defining a poll of nodes taht colelctively behave as a parameter server. The bigger the parameter server hierarchy gets, the more it looks like a reduction tree.
Reduction Tree
The idea is same as allreduce in message passing model. Parameter communication time with regard to worker number in reduction tree scheme.
It scales logrithmatically as the number of workers.
Batch size selection
Larger batch size lead to less frequent communication and therefore enable more scalability in a distributed setting. But for larger batch size, we need identify a suitable hyperparameter setting to maintain the speed and accuracy produced in DNN training.
Hyperparameters includes:
1. Initial learning rate
2. learning rate update scheme
3. weight delay
4. momentum
Weight update rule used, here means iteration index:
Learning rate update rule:
On how to get hyperparameters according to batch size, I will write another article for this.
Results
Final results on GPU cluster w/ GoogleNet.
More thinking
1. 以上方案基本上是无损的,为了更进一步减少通信开销,大家开始尝试有损的方案,在训练速度和准确度之间进行折衷。典型的有:
1). Reduce parameter size using 16-bit floating-point - Google
2). Use 16-bit weights and 8-bit activations.
3). 1-bit gradients backpropagation - Microsoft
4). Discard gradients whose numerical values fall below a certain threshold - Amazon
5). Compress(e.g. using PCA) weights before transmitting
6). Network pruning/encoding/quantization - Intel, DeePhi
2. 使用新的底层技术来减少通信开销 - Matrix
1) RDMA rather than traditional TCP/IP?
[专题论文阅读]【分布式DNN训练系统】 FireCaffe的更多相关文章
- 暑假第二弹:基于docker的hadoop分布式集群系统的搭建和测试
早在四月份的时候,就已经开了这篇文章.当时是参加数据挖掘的比赛,在计科院大佬的建议下用TensorFlow搞深度学习,而且要在自己的hadoop分布式集群系统下搞. 当时可把我们牛逼坏了,在没有基础的 ...
- 分布式链路追踪系统Sleuth和ZipKin
1.微服务下的链路追踪讲解和重要性 简介:讲解什么是分布式链路追踪系统,及使用好处 进行日志埋点,各微服务追踪. 2.SpringCloud的链路追踪组件Sleuth 1.官方文档 http://cl ...
- 基于zipkin分布式链路追踪系统预研第一篇
本文为博主原创文章,未经博主允许不得转载. 分布式服务追踪系统起源于Google的论文“Dapper, a Large-Scale Distributed Systems Tracing Infras ...
- 高性能分布式内存队列系统beanstalkd(转)
beanstalkd一个高性能.轻量级的分布式内存队列系统,最初设计的目的是想通过后台异步执行耗时的任务来降低高容量Web应用系统的页面访问延迟,支持过有9.5 million用户的Facebook ...
- 转: 透过CAT,来看分布式实时监控系统的设计与实现
评注: 开源的分布式监控系统 转:http://www.infoq.com/cn/articles/distributed-real-time-monitoring-and-control-syste ...
- Cola:一个分布式爬虫框架 - 系统架构 - Python4cn(news, jobs)
Cola:一个分布式爬虫框架 - 系统架构 - Python4cn(news, jobs) Cola:一个分布式爬虫框架 发布时间:2013-06-17 14:58:27, 关注:+2034, 赞美: ...
- zipkin分布式链路追踪系统
基于zipkin分布式链路追踪系统预研第一篇 分布式服务追踪系统起源于Google的论文“Dapper, a Large-Scale Distributed Systems Tracing Inf ...
- 分布式日志收集系统Apache Flume的设计详细介绍
问题导读: 1.Flume传输的数据的基本单位是是什么? 2.Event是什么,流向是怎么样的? 3.Source:完成对日志数据的收集,分成什么打入Channel中? 4.Channel的作用是什么 ...
- Apache shiro集群实现 (七)分布式集群系统下---cache共享
Apache shiro集群实现 (一) shiro入门介绍 Apache shiro集群实现 (二) shiro 的INI配置 Apache shiro集群实现 (三)shiro身份认证(Shiro ...
随机推荐
- SonarQube的使用入门
SonarQube的安装.配置与使用 详情请参照原博客:http://www.cnblogs.com/qiaoyeye/p/5249786.html SonarQube是管理代码质量一个开放平台,可以 ...
- Hadoop总结篇之五---模块间是怎么驱动执行的
在MRv1中,各个模块间驱动运行的方式是函数调用的方式.这是同步的过程,上一模块调用下一模块函数后,等待其执行.效率不高. 在MRv2中做了改进,yarn基于事件驱动的并发模型.在详细介绍前,先看下图 ...
- SDWebImage总结
SDWebImage 支持异步的图片下载+缓存,提供了 UIImageView+WebCache的 category,方便使用. 优点:首先NSURLCache是缓存原始数据(raw data)到磁盘 ...
- 设置树莓派3 B+的静态IP
修改/etc/dhcpcd.conf 文件 sudo vim /etc/dhcpcd.conf interface eth0 static ip_address= static routers=192 ...
- delphi 串口 关于RS485总线通信协议开发注意事项
http://blog.csdn.net/shjhuang/article/details/9426739 关于RS485总线通信协议开发注意事项 1 前言 近段时间发现我们系统在进行设备 ...
- sql优化方法
1. SELECT子句中避免使用 “*” 当你想在SELECT子句中列出所有的COLUMN时,使用动态SQL列引用‘*’是一个方便的方法.不幸的是,这是一个非常低效的方法. 实际上,ORACLE在解析 ...
- memcpy和memmove
memcpy函数 函数原型 void *memcpy(void *dest, const void *src, size_t n); dest:目标地址 src: 起始地址 n: 字节数 头文件 st ...
- Tree:加载列表数据
Tree控件,需要提供一个树形的JSON数据,才能正常显示. 通常,开发者在后台可以这样做: 1)从数据库查询出一个列表数据 2)在后台,将列表数据转换为树形数据 3)通过JSON方式返回 在前台页面 ...
- List接口
1.List接口实现的方式有两种ArrayList 和 LinkedList ArrayList实现了可变的数组,允许保存所有元素包括null,缺点是删除的比较慢 LinkedList删除对象比较快, ...
- [转载]《民航科技》2012年4月专家论坛:程延松《关于中国民航SWIM框架及技术实现探讨》
专家介绍:程延松,现任成都民航空管发展有限公司总经理,理学博士,高级工程师,长期从事空管技术研究和产品研发工作,作为课题负责人,参与了国家863计划.国家科技支撑计划.国家空管委重点课题.民航局重点课 ...