目录 1. 动机详述 2. 相关工作 3. 转换器结构 3.1 注意力机制详解 3.1.1 放缩的点积注意力机制 3.1.2 多头注意力机制 3.2 全连接网络 3.3 编码位置信息 [这是一篇4000+引用的文章.博主虽然不做NLP,但还是很感兴趣.当然,博主对本文的理解和翻译非常生涩] 动机:注意力机制很重要,现在的SOTA的NLP模型将编码器和解码器通过注意力机制连接在一起. 贡献:在本文中,作者提出了转换器(transformer).该结构与循环和卷积网络无关,只与注意力机制有关.整体模…

Natural Language Processing Tasks and Selected References I've been working on several natural language processing tasks for a long time. One day, I felt like drawing a map of the NLP field where I earn a living. I'm sure I'm not the only person who…

https://jalammar.github.io/illustrated-transformer/ The Illustrated Transformer Discussions: Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments) Translations: Chinese (Simplified), Korean Watch: MIT’s Deep Learning S…

1. 早期C. Koch与S. Ullman的研究工作. 他们提出了非常有影响力的生物启发模型. C. Koch and S. Ullman . Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology, 4(4):219-227, 1985. C. Koch and T. Poggio. Predicting the Visual World: Silenc…

Papers Published in 2017 Convolutional Sequence to Sequence Learning - Jonas Gehring et al., CoRR 2017 - [ Paper Reading ] ★ Attention Is All You Need - Ashish Vaswani et al., NIPS 2017 - [ Paper Reading ] ★ Papers Published in 2014 Sequence to Seque…

Link of the Paper: https://arxiv.org/abs/1706.03762 Motivation: The inherently sequential nature of Recurrent Models precludes parallelization within training examples. Attention mechanisms have become an integral part of compelling sequence modeling…

Link of the Paper: https://arxiv.org/pdf/1502.03044.pdf Main Points: Encoder-Decoder Framework: Encoder uses a convolutional neural network to extract a set of feature vectors which the authors refer to as annotation vectors. The extractor produces L…

前言废话,作者说把代码公布在gitub上,但是迟迟没有公布,我发邮件询问代码情况,邮件也迟迟不回,表示很尴尬..虽然种种这些,但是工作还是好工作,这个没的黑,那我们今天就来详细的介绍这篇文章. 导论:不了解caption的童鞋可以去看下这两篇知乎专栏:     看图说话的AI小朋友--图像标注趣谈(上)     看图说话的AI小朋友--图像标注趣谈(下) 一:摘要     作者提出了一个新的attention模型,这个模型与以往的区别在于,不仅考虑了状态与预测单词之间的关系,同时也考虑了图像区域…

最近的图片caption真的越来越火了,CVPR ICCV ECCV AAAI很多顶级会议都有此类的文章,今天我来讲一篇发表在AAAI的文章,因为我看了大量的论文,最近感觉AAAI越来越水了.所以这篇文章相对还是比较简单的.很遗憾,我向作者要源码.作者也没理我,不开心.. Caption: 说简单点,就是给你一张图片,然后系统自动生成一句话或者若干句话去描述它.比如这样: Give a image: You will get : A beautiful girl stood in the cor…

目录 1. 相关工作 2. Residual Attention Network 2.1 Attention残差学习 2.2 自上而下和自下而上 2.3 正则化Attention 最近看了些关于attention的文章.Attention是比较好理解的人类视觉机制,但怎么用在计算机问题上并不简单. 实际上15年之前就已经有人将attention用于视觉任务,但为什么17年最简单的SENet取得了空前的成功?其中一个原因是,前人的工作大多考虑空间上的(spatial)注意力,而SENet另辟蹊径,…

论文链接:https://arxiv.org/pdf/1502.03044.pdf 代码链接:https://github.com/kelvinxu/arctic-captions & https://github.com/yunjey/show-attend-and-tell & https://github.com/jazzsaxmafia/show_attend_and_tell.tensorflow 主要贡献 在这篇文章中,作者将“注意力机制(Attention Mechanism…

Tips for writing a paper 1. Tips for Paper Writing 2.• Before you write a paper • When you are writing a paper • The expression 3.Before you start writing a paper • State your Contribution • Organize your paper structure, See Everything as a Facet on…

How to (seriously) read a scientific paper Adam Ruben’s tongue-in-cheek column about the common difficulties and frustrations of reading a scientific paper broadly resonated among Science Careers readers. Many of you have come to us asking for more (…

Author: Emmanuel Goossaert 翻译 This article is a short guide to implementing an algorithm from a scientific paper. I have implemented many complex algorithms from books and scientific publications, and this article sums up what I have learned while se…

Writing the first draft of your science paper — some dos and don’ts 如何起草一篇科学论文?经验丰富的Angel Borja教授告诉你哪些是必要的而哪些是应该避免的!这是继Angel Borja教授前两篇撰写论文技巧的文章大受欢迎后又一力作.如果您正在准备撰写论文并想在国际期刊上发表,那么这篇文章非常值得您参考并收藏!   Four steps to preparing your first draft Here is the p…

Attention and Augmented Recurrent Neural Networks CHRIS OLAHGoogle Brain SHAN CARTERGoogle Brain Sept. 8 2016 Citation: Olah & Carter, 2016 Recurrent neural networks are one of the staples of deep learning, allowing neural networks to work with seque…

Attention For Fine-Grained Categorization Google ICLR 2015 本文说是将Ba et al. 的基于RNN 的attention model 拓展为受限更少,或者说是非受限的视觉场景.这个工作和前者很大程度上的不同在于,用一个更加有效的视觉网络,并且在attention RNN之外进行视觉网络的预训练. 前人的工作在学习 visual attention model 时已经解决了一些计算机视觉问题,并且表明加上不同的attention mec…

This Chapter outlines the logical steps to writing a good research paper. To achieve supreme excellence or perfection in anything you do, you need more than just the knowledge. Like the Olympic athlete aiming for the gold medal, you must have a posit…

Paper about Event Detection. #@author: gr #@date: 2014-03-15 #@email: forgerui@gmail.com 看一些相关的论文. 1. <Efficient Visual Event Detection using Volumetric Features> ICCV 2005 扩展2D box 特征到3D时空特征. 构建一个实时的检测器基于容积特征. 采用传统的兴趣点方法检测事件. 2. <ARMA-HMM: A New…

背景 经典MLP不能充分利用结构化数据,本文提出的DIN可以(1)使用兴趣分布代表用户多样化的兴趣(不同用户对不同商品有兴趣)(2)与attention机制一样,根据ad局部激活用户兴趣相关的兴趣(用户有很多兴趣,最后导致购买的是小部分兴趣,attention机制就是保留并激活这部分兴趣).   评价指标 按照user聚合样本,累加每个user组的sum(shows*AUC)/sum(shows).paper说实验表明GAUC比AUC准确稳定.   DIN算法         左边是基础模型,也…

前言 Transfomer是一种encoder-decoder模型,在机器翻译领域主要就是通过encoder-decoder即seq2seq,将源语言(x1, x2 ... xn) 通过编码,再解码的方式映射成(y1, y2 ... ym), 之前的做法是用RNN进行encode-decoder,但是由于RNN在某一时间刻的输入是依赖于上一时间刻的输出,所以RNN不能并行处理,导致效率低效,而Transfomer就避开了RNN,因此encoder-decoder效率高. Transformer…

paper url: https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf year: 2014 abstract 这篇文章出发点是如何减少图像相关任务的计算量, 提出通过使用 attention based RNN 模型建立序列模型(recurrent attention model, RAM), 每次基于上下文和任务来适应性的选择输入的的 image patch, 而不是整张图片, 从而使得计算量…

Open-domain QA Overview The whole system is consisted with Document Retriever and Document Reader. The Document Retriever returns top five Wikipedia articles given any question, then the Document Reader will process these articles. Document Retriever…

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering 2019-04-25 21:43:11 Paper:https://arxiv.org/pdf/1904.04357.pdf Code: https://github.com/fanchenyou/HME-VideoQA 1. Background and Motivation:  用 Memory Network 做视觉问题…

paper: <Attention Augmented Convolutional Networks> https://arxiv.org/pdf/1904.09925.pdf 这篇文章是google brain的,应该有分量.上来就说:卷积神经网络有一个重要的弱点就是 它仅仅操作于于一个领域,对于没有考虑到全局信息有损失. (这就是全局和局部的辨证关系.) 注意力机制,以一种能够把握长距离作用的手段,在序列模型和生成模型里使用.这篇文章使用注意力机制到判别模型中来,作为替代卷积的手段.(非常…

目录 1. MTL的定义 2. MTL的机制 2.1. Representation Bias 2.2. Uncorrelated Tasks May Help? 3. MTL的用途 3.1. Using the Future to Predict the Present 3.2. Time Series Prediction 3.3. Using Extra Tasks to Focus Attention 3.4. Quantization Smoothing 3.5. Some Input…

Pedestrian Attributes Recognition Paper List  2018-12-22 22:08:55 [Note] you may also check the updated version of this blog from my github: https://github.com/wangxiao5791509/Pedestrian-Attribute-Recognition-Paper-List The survey paper of pedestrian…

Convolutional Image Captioning 2018-11-04 20:42:07 Paper: http://openaccess.thecvf.com/content_cvpr_2018/papers/Aneja_Convolutional_Image_Captioning_CVPR_2018_paper.pdf Code: https://github.com/aditya12agd5/convcap Related Papers: 1. Convolutional Se…

CBAM: Convolutional Block Attention Module 2018-09-14 21:52:42 Paper:http://openaccess.thecvf.com/content_ECCV_2018/papers/Sanghyun_Woo_Convolutional_Block_Attention_ECCV_2018_paper.pdf GitHub:https://github.com/luuuyi/CBAM.PyTorch 本文提出 channel atten…

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention 2018-08-10 10:15:06 Paper (ICML-2015):http://proceedings.mlr.press/v37/xuc15.pdf Theano (Offical Implementation): https://github.com/kelvinxu/arctic-captions TensorFlow: htt…