[CVPR2017] Visual Translation Embedding Network for Visual Relation Detection 论文笔记

http://www.ee.columbia.edu/ln/dvmm/publications/17/zhang2017visual.pdf

Visual Translation Embedding Network for Visual Relation Detection Hanwang Zhang† , Zawlin Kyaw‡ , Shih-Fu Chang† , Tat-Seng Chua‡ †Columbia University, ‡National University of Singapore

p.p1 { margin: 0.0px 0.0px 0.0px 0.0px; font: 15.0px Arial; color: #323333 }
p.p2 { margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Arial; color: #323333 }
p.p3 { margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Arial; color: #323333; min-height: 15.0px }
li.li2 { margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Arial; color: #323333 }
span.s1 { }
ul.ul1 { list-style-type: disc }
ul.ul2 { list-style-type: circle }
ul.ul3 { list-style-type: square }

亮点

• 视觉关系预测问题的分析与化简：把一种视觉关系理解为在特征空间从主语到宾语的一种变换，很有效、很直白
• 实验设计的很棒，从多角度进行了分析对比：语言空间划分，多任务对物体检测的提升，零次学习等。

现有工作

• Mature visual detection [16, 35]
• Burgeoning visual captioning and question answering [2, 4]
• directly bridge the visual model (e.g., CNN) and the language model (e.g., RNN), but fall short in modeling and understanding the relationships between objects.
• poor generalization ability
• Visual Relation Detection: a visual relation as a subject-predicate-object triplet
• joint models, a relation triplet is considered as a unique class [3, 9, 33, 37].
• the long-tailed distribution is an inherent defect for scalability.
• separate model
• modeling the large visual variance of predicates is challenging.
• language priors to boost relation detection

主要思想

Translation Embedding 视觉关系预测的难点主要是：对于N个物体和R种谓语，有N^2R种关系，是一个组合爆炸问题。解决这个问题常用的办法是：

• 估计谓语，不估计关系，缺点：对于不同的主语、宾语，图像视觉差异巨大

受Translation Embedding (TransE) 启发，文章中将视觉关系看作在特征空间上从主语到宾语的一种映射，在低维空间上关系元组可看作向量变换，例如person+ride ≈ bike.

Knowledge Transfer in Relation 物体的识别和谓语的识别是互惠的。通过使用类别名、位置、视觉特征三种特征和端对端训练网络，使物体和谓语之前的隐含关系在网络中能够学习到。

算法

Visual Translation Embedding

p.p1 { margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #323333 }
span.s1 { }

 Loss function

p.p1 { margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #323333 }
p.p2 { margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #323333; min-height: 16.0px }
span.s1 { }

Feature Extraction Layer

classname + location + visual feature 不同的特征对不同的谓语（动词、介词、空间位置、对比）都有不一样的作用

Bilinear Interpolation

In order to achieve object-relation knowledge transfer, the relation error should be back-propagated to the object detection network and thus refines the objects. We replace the RoI pooling layer with bilinear interpolation [18]. It is a smooth function of two inputs:

p.p1 { margin: 0.0px 0.0px 0.0px 0.0px; font: 16.0px Arial; color: #323333 }
p.p2 { margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #323333 }
li.li2 { margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #323333 }
span.s1 { }
ul.ul1 { list-style-type: disc }

结果

Translation embeding: ＋18%

object detection ＋0.6% ～ 0.3%

State-of-art:

• Phrase Det.  +3% ~ 6%
• Relation Det. +1%
• Retrieval -1% ~ 2%
• Zero-shot phrase detection

• Phrase Det. －0.7% (without language prior)
• Relation Det. －1.4%
• Retrieval ＋0.2％

问题

• 两个物体之间可能有多种关系，比如person ride elephant，同时也存在person short elephant但是文章中的方法无法表示多样化的关系
• 没有使用语言先验知识，使用多模态信息可能会有所提升