Deep Visual-Semantic Alignments for Generating Image Descriptions（深度视觉-语义对应对于生成图像描述）

https://cs.stanford.edu/people/karpathy/deepimagesent/

Abstract

We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.

我们展示了一个模型，它能生成图像和它们区域的自然语言描述。我们的方法杠杆平衡图像集与它们的句子描述，以学习语言和视觉数据之间内在模态的关系。我们的对齐模型是基于一种新的结合，图像区域的卷积神经网络，句子的双向递归神经网络，和通过多模态嵌入对齐两种模式的结构化目标。然后，我们描述了一种多模式递归神经网络架构，它是使用推断对齐方法来学习生成图像区域的新描述。我们证明我们的对齐模型在FLICKR8K、FLIKR30K和MSCCOO数据集的检索实验中产生最先进的结果。然后，我们表示，生成的描述显著地胜过无论是全图还是新的区域水平标注数据集的检索基线。

Code：链接 其他

1. Introduction简介

A quick glance at an image is sufficient for a human to point out and describe an immense amount of details about the visual scene [14]. However, this remarkable ability has proven to be an elusive task for our visual recognition models. The majority of previous work in visual recognition has focused on labeling images with a fixed set of visual categories and great progress has been achieved in these endeavors [45, 11]. However, while closed vocabularies of visual concepts constitute a convenient modeling assumption, they are vastly restrictive when compared to the enormous amount of rich descriptions that a human can compose.

对人类来说快速地看一眼图片并指出并描述视觉场景的详细细节是足够的。但是，这个杰出的能力已证明对视觉识别模型来说是一个难以捉摸的任务。

Some pioneering approaches that address the challenge of generating image descriptions have been developed [29,13]. However, these models often rely on hard-coded visual concepts and sentence templates, which imposes limits on their variety. Moreover, the focus of these works has been on reducing complex visual scenes into a single sentence, which we consider to be an unnecessary restriction.

In this work, we strive to take a step towards the goal of  generating dense descriptions of images (Figure 1). The primary challenge towards this goal is in the design of a model that is rich enough to simultaneously reason about contents of images and their representation in the domain of natural language. Additionally, the model should be free of assumptions about specific hard-coded templates, rules or categories and instead rely on learning from the training data. The second, practical challenge is that datasets of image captions are available in large quantities on the internet[21, 58, 37], but these descriptions multiplex mentions of several entities whose locations in the images are unknown.