[Paper Reading] Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
论文链接:https://arxiv.org/pdf/1502.03044.pdf
代码链接:https://github.com/kelvinxu/arctic-captions & https://github.com/yunjey/show-attend-and-tell & https://github.com/jazzsaxmafia/show_attend_and_tell.tensorflow
主要贡献
在这篇文章中,作者将“注意力机制(Attention Mechanism)”引入了神经机器翻译(Neural Image Captioning)领域,提出了两种不同的注意力机制:‘Soft’ Deterministic Attention Mechanism & ‘Hard’ Stochastic Attention Mechanism。下图展示了"Show, Attend and Tell"模型的整体框架。
注意力机制的关键点在于,如何从图像的特征向量ai中计算得到上下文向量zt。对于每一个位置i,注意力机制能够产生一个权重eti。在Hard Attention机制中,权重αti所扮演的角色是图像区域向量ai在t时刻被选中作为解码器的信息的概率,有且只有一个区域会被选中,为此,引入变量st,i,当区域i被选中时为1,否则为0;在Soft Attention机制中,权重αti所扮演的角色是图像区域向量ai在t时刻输入解码器的信息中所占的比例。(参考Attention机制论文阅读——Soft和Hard Attention,Multimodal —— 看图说话(Image Caption)任务的论文笔记(二)引入attention机制)
实验细节
- 在文章中,作者提出使用在ImageNet数据集上预训练好、不进行微调的VGGNet提取图像特征,将block5_conv4(Conv2D)提取到的feature map(14×14×512)reshape为196×512(L×D,L=196,D=512,即196个图像区域,每个区域特征向量的维度是512)的图像区域向量ai。
To create the annotations ai used by our decoder, we used the Oxford VGGnet pretrained on ImageNet without finetuning.
In our experiments we use the 14×14×512 feature map of the fourth convolutional layer before max pooling. This means our decoder operates on the flattened 196×512 (i.e L × D) encoding.
- 在文章中,作者指出,解码器LSTM初始的细胞状态(init_c)与隐层状态(init_h)由从图像中提取到的特征向量及两个独立的多层感知机(Multi-Layer Perception, MLP)决定。
The initial memory state and hidden state of the LSTM are predicted by an average of the annotation vectors fed through two separate MLPs(init,c and init,h).
版权声明:本文为博主原创文章,欢迎转载,转载请注明作者及原文出处!
[Paper Reading] Show, Attend and Tell: Neural Image Caption Generation with Visual Attention的更多相关文章
- Paper Reading - Show, Attend and Tell: Neural Image Caption Generation with Visual Attention ( ICML 2015 )
Link of the Paper: https://arxiv.org/pdf/1502.03044.pdf Main Points: Encoder-Decoder Framework: Enco ...
- 论文笔记:Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention 2018-08-10 10:15:06 Pap ...
- 论文:Show, Attend and Tell: Neural Image Caption Generation with Visual Attention-阅读总结
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention-阅读总结 笔记不能简单的抄写文中的内容,得有自 ...
- Paper Reading - Show and Tell: A Neural Image Caption Generator ( CVPR 2015 )
Link of the Paper: https://arxiv.org/abs/1411.4555 Main Points: A generative model ( NIC, GoogLeNet ...
- [Paper Reading] Show and Tell: A Neural Image Caption Generator
论文链接:https://arxiv.org/pdf/1411.4555.pdf 代码链接:https://github.com/karpathy/neuraltalk & https://g ...
- [Paper Reading] Image Captioning using Deep Neural Architectures (arXiv: 1801.05568v1)
Main Contributions: A brief introduction about two different methods (retrieval based method and gen ...
- Paper Reading - CNN+CNN: Convolutional Decoders for Image Captioning
Link of the Paper: https://arxiv.org/abs/1805.09019 Innovations: The authors propose a CNN + CNN fra ...
- Paper Reading: Stereo DSO
开篇第一篇就写一个paper reading吧,用markdown+vim写东西切换中英文挺麻烦的,有些就偷懒都用英文写了. Stereo DSO: Large-Scale Direct Sparse ...
- Paper Reading - Mind’s Eye: A Recurrent Visual Representation for Image Caption Generation ( CVPR 2015 )
Link of the Paper: https://ieeexplore.ieee.org/document/7298856/ A Correlative Paper: Learning a Rec ...
随机推荐
- Nginx中ngx_stream_core_module和ngx_stream_proxy_module
ngx_stream_core_module模块该模块模拟基于tcp或udp的服务连接的反向代理理,即⼯工作于传输层的调度器器指令:17.1 streamSyntax: stream { ... }D ...
- Cookie操作、ASP.Net文件上传HttpPostedFile
概述 Cookie用来保存客户浏览器请求服务器页面的请求信息. 我们可以存放非敏感的用户信息,保存时间可以根据需要设置.如果没有设置Cookie失效日期,它的生命周期保存到关闭浏览器为止,Cookie ...
- [唐胡璐]Java操作Sql Server 2008数据库
下载Microsoft JDBC Driver for SQL Server 直接去官网下载即可: 下载解压文件,得到sqljdbc.jar和sqljdbc4.jar。如果你使用的是jre1.7版本, ...
- 第92题:反转链表II
一. 问题描述 反转从位置 m 到 n 的链表.请使用一趟扫描完成反转. 说明: 1 ≤ m ≤ n ≤ 链表长度. 示例: 输入: 1->2->3->4->5->NUL ...
- python 图像识别的小应用
前些天看见了几个有趣的python项目,在自己实际测试和理解后贴一下代码. https://www.shiyanlou.com/courses/589/labs/1964/document 算法主要逻 ...
- C# Transaction 事务处理
class //student [Serializable] public class Student { public string FirstName { get; set; } public s ...
- Codeforces Round #585 (Div. 2) B. The Number of Products(DP)
链接: https://codeforces.com/contest/1215/problem/B 题意: You are given a sequence a1,a2,-,an consisting ...
- IIS遇到过的问题
1. IIS的一个莫名错误Server Application Unavailable http://www.kesion.com/zzcd/asp/aspjq/474.html 新打开这个服务ASP ...
- .net SerialPort
虚拟串口并定时向虚拟串口定时发数据 http://scorpiomiracle.iteye.com/blog/653923 C#中如何使用SerialPort控件向单片机发送数据? http://zh ...
- 二进制学习——Blob,ArrayBuffer、File、FileReader和FormData的区别
前言: Blob.ArrayBuffer.File.fileReader.formData这些名词总是经常看到,知道一点又好像不知道,像是同一个东西好像又不是,总是模模糊糊,最近终于下决心要弄清楚. ...