(转) How to Train a GAN? Tips and tricks to make GANs work
How to Train a GAN? Tips and tricks to make GANs work
转自:https://github.com/soumith/ganhacks
While research in Generative Adversarial Networks (GANs) continues to improve the fundamental stability of these models, we use a bunch of tricks to train them and make them stable day to day.
Here are a summary of some of the tricks.
Here's a link to the authors of this document
If you find a trick that is particularly useful in practice, please open a Pull Request to add it to the document. If we find it to be reasonable and verified, we will merge it in.
1. Normalize the inputs
- normalize the images between -1 and 1
- Tanh as the last layer of the generator output
2: A modified loss function
In GAN papers, the loss function to optimize G is min (log 1-D)
, but in practice folks practically use max log D
- because the first formulation has vanishing gradients early on
- Goodfellow et. al (2014)
In practice, works well:
- Flip labels when training generator: real = fake, fake = real
3: Use a spherical Z
- Dont sample from a Uniform distribution
- Sample from a gaussian distribution
- When doing interpolations, do the interpolation via a great circle, rather than a straight line from point A to point B
- Tom White's Sampling Generative Networks has more details
4: BatchNorm
- Construct different mini-batches for real and fake, i.e. each mini-batch needs to contain only all real images or all generated images.
- when batchnorm is not an option use instance normalization (for each sample, subtract mean and divide by standard deviation).
5: Avoid Sparse Gradients: ReLU, MaxPool
- the stability of the GAN game suffers if you have sparse gradients
- LeakyReLU = good (in both G and D)
- For Downsampling, use: Average Pooling, Conv2d + stride
- For Upsampling, use: PixelShuffle, ConvTranspose2d + stride
- PixelShuffle: https://arxiv.org/abs/1609.05158
6: Use Soft and Noisy Labels
- Label Smoothing, i.e. if you have two target labels: Real=1 and Fake=0, then for each incoming sample, if it is real, then replace the label with a random number between 0.7 and 1.2, and if it is a fake sample, replace it with 0.0 and 0.3 (for example).
- Salimans et. al. 2016
- make the labels the noisy for the discriminator: occasionally flip the labels when training the discriminator
7: DCGAN / Hybrid Models
- Use DCGAN when you can. It works!
- if you cant use DCGANs and no model is stable, use a hybrid model : KL + GAN or VAE + GAN
8: Use stability tricks from RL
- Experience Replay
- Keep a replay buffer of past generations and occassionally show them
- Keep checkpoints from the past of G and D and occassionaly swap them out for a few iterations
- All stability tricks that work for deep deterministic policy gradients
- See Pfau & Vinyals (2016)
9: Use the ADAM Optimizer
- optim.Adam rules!
- See Radford et. al. 2015
- Use SGD for discriminator and ADAM for generator
10: Track failures early
- D loss goes to 0: failure mode
- check norms of gradients: if they are over 100 things are screwing up
- when things are working, D loss has low variance and goes down over time vs having huge variance and spiking
- if loss of generator steadily decreases, then it's fooling D with garbage (says martin)
11: Dont balance loss via statistics (unless you have a good reason to)
- Dont try to find a (number of G / number of D) schedule to uncollapse training
- It's hard and we've all tried it.
- If you do try it, have a principled approach to it, rather than intuition
For example
while lossD > A:
train D
while lossG > B:
train G
12: If you have labels, use them
- if you have labels available, training the discriminator to also classify the samples: auxillary GANs
13: Add noise to inputs, decay over time
- Add some artificial noise to inputs to D (Arjovsky et. al., Huszar, 2016)
- adding gaussian noise to every layer of generator (Zhao et. al. EBGAN)
- Improved GANs: OpenAI code also has it (commented out)
14: [notsure] Train discriminator more (sometimes)
- especially when you have noise
- hard to find a schedule of number of D iterations vs G iterations
15: [notsure] Batch Discrimination
- Mixed results
16: Discrete variables in Conditional GANs
- Use an Embedding layer
- Add as additional channels to images
- Keep embedding dimensionality low and upsample to match image channel size
Authors
- Soumith Chintala
- Emily Denton
- Martin Arjovsky
- Michael Mathieu
(转) How to Train a GAN? Tips and tricks to make GANs work的更多相关文章
- Matlab tips and tricks
matlab tips and tricks and ... page overview: I created this page as a vectorization helper but it g ...
- LoadRunner AJAX TruClient协议Tips and Tricks
LoadRunner AJAX TruClient协议Tips and Trickshttp://automationqa.com/forum.php?mod=viewthread&tid=2 ...
- Android Studio tips and tricks 翻译学习
Android Studio tips and tricks 翻译 这里是原文的链接. 正文: 如果你对Android Studio和IntelliJ不熟悉,本页提供了一些建议,让你可以从最常见的任务 ...
- Tips and Tricks for Debugging in chrome
Tips and Tricks for Debugging in chrome Pretty print On sources panel ,clicking on the {} on the bot ...
- [转]Tips——Chrome DevTools - 25 Tips and Tricks
Chrome DevTools - 25 Tips and Tricks 原文地址:https://www.keycdn.com/blog/chrome-devtools 如何打开? 1.从浏览器菜单 ...
- Nginx and PHP-FPM Configuration and Optimizing Tips and Tricks
原文链接:http://www.if-not-true-then-false.com/2011/nginx-and-php-fpm-configuration-and-optimizing-tips- ...
- 10 Essential TypeScript Tips And Tricks For Angular Devs
原文: https://www.sitepoint.com/10-essential-typescript-tips-tricks-angular/ ------------------------- ...
- WWDC笔记:2011 Session 125 UITableView Changes, Tips and Tricks
What’s New Automatic Dimensions - (CGFloat)tableView:(UITableView *)tableView heightForHeaderInSect ...
- C++ Tips and Tricks
整理了下在C++工程代码中遇到的技巧与建议. 0x00 巧用宏定义. 经常看见程序员用 enum 值,打印调试信息的时候又想打印数字对应的字符意思.见过有人写这样的代码 if(today == MON ...
随机推荐
- oracle通过修改控制文件scn推进数据库scn
数据库当前scn 代码如下 复制代码 idle> select checkpoint_change# from v$database; CHECKPOINT_CHANGE#----------- ...
- C语言程序设计第5堂作业
一.本次课学习主要内容及知识结构点: 二.实验内容:(60分) 1. 求奇数和.输入一批正整数(以零或负数为结束标志),求其中的奇数和.试编写相应程序. 2. 展开式求和.输入一个实数 x,计算并输出 ...
- jquery ajax详解
详细参数列表url:发送请求的连接地址type:请求方式 get:获取 post:发送 put和deletetimeout:设置请求超时时间async:默认true为异步请求,false同步请求锁住浏 ...
- Linux下swoole的安装配置
前几天搭建swoole环境,在安装php的swoole扩展时不知道什么原因,提示成功,但是使用的时候不能加载,最后决定重新安装php试试,顺便记录了php的安装过程 wget http://cn2.p ...
- C#方法参数传递机制
1:value(值传递).ref(引用传递).out(输出传递) ref和out使用效果上面是等效的,它们的区别在于:参数标记为ref,那么必须在调用函数之前初始化参数的值:参数标记为out,调用函数 ...
- React学习——ListView组件
(草稿) 先把代码放上来,再补充说明 <!DOCTYPE html> <html> <head> <title>React ListView</t ...
- 记账类APP竞品分析-挖财与随手记
注:本文更新中. 一.概览 1. 产品名称及版本 l 挖财11.2.0.0 免费版(2016/9/6发布) l 随手记10.2.8免费版(2016/8/22发布) 2. 设备信息 设备型号:i ...
- 导出excel表格。
导出也做了很多遍了,还是发现好记性不如烂笔头,还是记下来. public void exportLog(HttpServletRequest request,HttpServletResponse r ...
- 用eclipse碰到的一些错误,然后自己去网上找的解决办法
错误一: [Please check logcat output for more details.Launch canceled! 解决办法:在配置文件:AndroidManifest.xml加入如 ...
- Linux删除apt-get安装的软件
我们都知道安装软件最简单的方法是apt-get install,但是卸载就不常用了,如何卸载呢?1.sudo apt-get remove xxxxxsudo apt-get autoremove2. ...