Neural Architecture Search — Limitations and Extensions

2019-09-16 07:46:09

This blog is from: https://towardsdatascience.com/neural-architecture-search-limitations-and-extensions-8141bec7681f

For the past couple of years, researchers and companies have been trying to make deep learning more accessible to non-experts by providing access to pre-trained computer vision or machine translation models. Using a pre-trained model for another task is known as transfer learning, but it still requires sufficient expertise to fine-tune the model on another dataset. Fully automating this procedure allows even more users to benefit from the great progress that has been made in ML to date.

This is called AutoML, and it can cover many parts of predictive modelling such as architecture search and hyperparameter optimization. In this post, I focus on the former, as there has been a recent explosion of methods that search for the “best” architecture for a given dataset. The results presented are based on joint work with Jonathan Lorraine.

Importance of Neural Architecture Search

As a side note, keep in mind that a neural network with just a single hidden layer and non-linear activation function is able to represent any function possible, provided that there are sufficient neurons in that layer (UAT). However, such a simple architecture, though it is theoretically able to learn any function, does not represent the hierarchical processing that occurs in the human visual cortex. The architecture of a neural network gives it an inductive bias, and shallow, wide networks that do not use convolutions are significantly worse at image classification tasks than deep, convolutional networks. Thus, in order for neural networks to generalize and not overfit the training dataset, it’s important to find architectures with the right inductive bias (regardless if those architectures are inspired by the brain or not).

Neural Architecture Search (NAS) Overview

NAS was an inspiring work out of Google that lead to several follow up works such as ENASPNAS, and DARTS. It involves training a recurrent neural network (RNN) controller using reinforcement learning (RL) to automatically generate architectures. These architectures then have their weights trained and are evaluated on a validation set. Their performance on the validation set is the reward signal for the controller which then increases its probability of generating architectures that have done well, and decreases the probability of generating architectures that have done poorly. For non-technical readers, this essentially takes the process of a human manually tweaking a neural network and learning what works well, and automating it. The idea of automatically creating NN architectures was not coined by NAS, as other approaches using methods such as genetic algorithms existed long before, but NAS effectively used RL to efficiently search a space that is prohibitively large to search exhaustively. Below, the components of NAS are analyzed in a bit more depth, before I go on to discuss the limitations of the method as well as its more efficient successor ENAS, as well as an interesting failure mode. The next 2 subsections are best understood while comparing the text again the below figure showing how architectures are sampled and trained:

 

Diagram of how NAS Works

LSTM Controller

The controller generates architectures by making a series of choices for a pre-defined amount of time steps. For example, when generating a convolutional architecture, the controller begins by only creating architectures with 6 layers in them. For each layer, just 4 decisions are made by the controller: filter height, filter width, number of filters, and stride (so 24 time steps). Assuming that the first layer is numbered 0, then the decisions

Neural Architecture Search — Limitations and Extensions的更多相关文章

  1. 论文笔记:Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells

    Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells 2019-04- ...

  2. (转)Illustrated: Efficient Neural Architecture Search ---Guide on macro and micro search strategies in ENAS

    Illustrated: Efficient Neural Architecture Search --- Guide on macro and micro search strategies in  ...

  3. (转)The Evolved Transformer - Enhancing Transformer with Neural Architecture Search

    The Evolved Transformer - Enhancing Transformer with Neural Architecture Search 2019-03-26 19:14:33 ...

  4. 论文笔记:ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

    ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware 2019-03-19 16:13:18 Pape ...

  5. 论文笔记:Progressive Neural Architecture Search

    Progressive Neural Architecture Search 2019-03-18 20:28:13 Paper:http://openaccess.thecvf.com/conten ...

  6. 论文笔记:Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

    Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation2019-03-18 14:4 ...

  7. 论文笔记系列-Neural Architecture Search With Reinforcement Learning

    摘要 神经网络在多个领域都取得了不错的成绩,但是神经网络的合理设计却是比较困难的.在本篇论文中,作者使用 递归网络去省城神经网络的模型描述,并且使用 增强学习训练RNN,以使得生成得到的模型在验证集上 ...

  8. 小米造最强超分辨率算法 | Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search

    本篇是基于 NAS 的图像超分辨率的文章,知名学术性自媒体 Paperweekly 在该文公布后迅速跟进,发表分析称「属于目前很火的 AutoML / Neural Architecture Sear ...

  9. Research Guide for Neural Architecture Search

    Research Guide for Neural Architecture Search 2019-09-19 09:29:04 This blog is from: https://heartbe ...

随机推荐

  1. VirtualBox下设置 XP虚拟机桥接模式

    virtualBox下设置虚拟机桥接模式

  2. 继 首次使用DoNetCore EFCore DbFirst 更新数据实体

    //EFCore DB First 步骤 //第一步:Install-Package Microsoft.EntityFrameworkCore.SqlServer -version 2.1.1 // ...

  3. 【RAC】 RAC For W2K8R2 安装--创建ASM磁盘组(六)

    [RAC] RAC For W2K8R2 安装--创建ASM磁盘组(六) 一.1  BLOG文档结构图 一.2  前言部分 一.2.1  导读 各位技术爱好者,看完本文后,你可以掌握如下的技能,也可以 ...

  4. PHP Lumen Laravel 解决validate方法自定义message无效的问题

    /** * 由于 \Laravel\Lumen\Routing\ProvidesConvenienceMethods::validate 在验证不通过时, * 抛出 \Illuminate\Valid ...

  5. OpenResty(Nginx + Lua)常用操作

    平滑重启: /usr/servers/nginx/sbin/nginx  -s reload 重启前测试配置是否成功 /usr/servers/nginx/sbin/nginx  -t 平滑升级如果服 ...

  6. 网络空间安全基础篇(2)----wireshark

    wireshrak是一款通过本地端口,来抓取通过以太网获取的文件包,包括SMB HTTP FTP TELNET 等包 在网络安全比赛中最常用的就是HTTP协议,TELNET协议,FTP协议,SMB协议 ...

  7. 基于kubeamd初始化安装kubernetes集群

    环境:master,etcd 172.16.1.5node1 172.16.1.6node2 172.16.1.7前提:1.基于主机名通信,/etc/hosts 2.时间同步3.关闭firewalld ...

  8. NODE简易综合应用服务器搭建

    node搭建简易服务器 querystring和url模板学习地址 querystring&url 1. 目录结构 2. 代码结构 const http = require('http'); ...

  9. O(n) 取得数组中每个元素右边第一个比它大的元素

    题目: 给定一个整型数组,数组元素随机无序的,要求打印出所有元素右边第一个大于该元素的值. 如数组A=[6,8,9,2,3,5,6] 输出[8,9,-1,3,5,6,-1] 思路: 我们用栈来保存未找 ...

  10. 案例实战之如何写一个webpack plugin

    案例实战之如何写一个webpack plugin 1.写一个生成打包文件目录的file.md文件 // 生成一个目录项目目录的文件夹 class FileListPlugin { constructo ...