Learning to Read Chest X-Rays:Recurrent Neural Cascade Model for Automated Image Annotation (CVPR 2016)

Goals:

-Learn to read chest x-rays from an existing dataset of images and text with minimal human effort

-To generate text description about disease in image as well as their context (with pre-defined grammar, thus not multiple-instance-learning)

Approach

-Text-mining based image labeling;train CNN for image, RNN for text

-Extensive regularization (e.g.,batch-normalization, data dropout) to deal with data bias(normal vs. diseased)

-Joint image/text context vector for more composite image labeling

The above picture is an illustration of how joint image/text context vector is obtained. RNN's state vector (h) is initialized with the CNN image embedding (CNN(I)), and it's unrolled over the annotation sequences with the words as input. Mean-pooling is applied over the state vectors in each word of the sequence, to obtain the joint image/text vector. All RNNs share the same parameters, which are trained in the first round.

MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (CVPR 2017)

MDNet can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize network attention.

TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References (MICCAI 2017)

Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation (NIPS 2018)

On the Automatic Generation of Medical Imaging Reports (ACL 2018)

Datasets: IU X-Ray , PEIR Gross

ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases (CVPR 2017) Xiaosong Wang

从标题就可以看到这篇论文和Medical  Image Report没啥关系, 为了便于继续学习后面的TieNet,还是将它放在这里。

TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays (CVPR 2018) Xiaosong Wang

Reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence ,due to

(1).shortage of large-scale machine-learnable medical image datasets

(2).lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training.

Contributions:

(1).proposed the Text-Image Embedding Network, which is a multi-purpose end-to-end trainable multi-task CNN-RNN framework

(2).show how raw report data, together with paired image, can be utilized to produce meaningful attention-based image and text representations using the proposed TieNet.

(3).outline how the developed text and image embeddings are able to boost the auto-annotation framework and achieve extremely high accuracy for chest x-ray labeling

(4).present a novel image classification framework which takes images as the sole input, but uses the paired text-image representations from training as a prior knowledge injection, in order to produce improved classification scores and preliminary report generations.

Datasets: ChestX-ray14, Hand-labeled, OpenI

The CNN component additionally includes a convolutional layer(transition layer) to manipulate the spatial grid size and feature dimension.

To obtain an interpretable global text and visual embedding for the purpose of classification, introduce two key enhancements in the form of the AETE and SW-GAP

AETE: Attention Encoded Text Embedding

SW-GAP: Saliecny Weighted Global Average Pooling

Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019)
Christy Y. Li, Xiaodan Liang**, Zhiting Hu, Eric Xing.

End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis (AAAI 2019)
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang**, Jianheng Tang, Liang Lin.

Medical Image Report论文合辑的更多相关文章

  1. Image Caption论文合辑2

    说明: 这个合辑里面的论文不全是Image Caption, 但大多和Image Caption相关, 同时还有一些Workshop论文. Guiding Long-Short Term Memory ...

  2. Image Captioning 经典论文合辑

    Image Caption: Automatically describing the content of an image domain:CV+NLP Category:(by myself, y ...

  3. Image Paragraph论文合辑

    A Hierarchical Approach for Generating Descriptive Image Paragraphs (CPVR 2017) Li Fei-Fei. 数据集地址: h ...

  4. 【Tips】史上最全H1B问题合辑——保持H1B身份终级篇

    [Tips]史上最全H1B问题合辑——保持H1B身份终级篇 2015-04-10留学小助手留学小助手 留学小助手 微信号 liuxue_xiaozhushou 功能介绍 提供最真实全面的留学干货,帮您 ...

  5. SSH三大框架合辑的搭建步骤

    v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VM ...

  6. 【OpenCV新手教程之十二】OpenCV边缘检測:Canny算子,Sobel算子,Laplace算子,Scharr滤波器合辑

    本系列文章由@浅墨_毛星云 出品,转载请注明出处. 文章链接:http://blog.csdn.net/poem_qianmo/article/details/25560901 作者:毛星云(浅墨) ...

  7. 【OpenCV新手教程之十八】OpenCV仿射变换 & SURF特征点描写叙述合辑

    本系列文章由@浅墨_毛星云 出品,转载请注明出处. 文章链接:http://blog.csdn.net/poem_qianmo/article/details/33320997 作者:毛星云(浅墨)  ...

  8. 【OpenCV新手教程之十七】OpenCV重映射 & SURF特征点检測合辑

    本系列文章由@浅墨_毛星云 出品.转载请注明出处. 文章链接:http://blog.csdn.net/poem_qianmo/article/details/30974513 作者:毛星云(浅墨)  ...

  9. [OpenCV入门教程之十二】OpenCV边缘检测:Canny算子,Sobel算子,Laplace算子,Scharr滤波器合辑

    http://blog.csdn.net/poem_qianmo/article/details/25560901 本系列文章由@浅墨_毛星云 出品,转载请注明出处. 文章链接:http://blog ...

随机推荐

  1. 【codeforces 755A】PolandBall and Hypothesis

    time limit per test2 seconds memory limit per test256 megabytes inputstandard input outputstandard o ...

  2. 5.3.3.1 deque其他使用方式

    在本节里提供了一些关于deque其他使用方式. 提供相似UNIX中的命令tail的功能,显示一个文件最后面一段文本: def tail(filename, n=10): '返回文件最后的n行文本' w ...

  3. ITFriend创业败局(二):初创公司应该怎样分配股权

    说到金钱,中国人有句口头禅,"谈钱多伤感情".这句话非常能代表,在熟人之间,中国人不喜欢在金钱上"斤斤计较". 但是,对于一起出来创业,尤其是没有经验的年轻人来 ...

  4. jsp页面遍历List<Map<String,Object>>

    多表联查会有此类结果出现, 查阅发现基本解决思路是双重遍历,获取map,entry.value等方法. 最终发现可以使用c:forEach单次遍历,map中的key值大写,即可得到object. Co ...

  5. 【erlang 网络编程学习】 分析cowboy acceptor实现

    http://www.tuicool.com/articles/vuymei 不知道为什么就看了cowboy代码,就继续看了下去了. 分析一下吧,主要写写cowboy 的acceptor pool 的 ...

  6. C# Tuple VS ValueTuple

    C# Tuple VS ValueTuple(元组类 VS 值元组) C# 7.0已经出来一段时间了,大家都知道新特性里面有个对元组的优化:ValueTuple.这里利用详尽的例子详解Tuple VS ...

  7. 【BZOJ 1023】[SHOI2008]cactus仙人掌图

    [题目链接]:http://www.lydsy.com/JudgeOnline/problem.php?id=1023 [题意] [题解] 如果不考虑有环的情况; 那么有一个经典的求树的直径的方法; ...

  8. 卷积与反卷积、步长(stride)与重叠(overlap)

    1. 卷积与反卷积 如上图演示了卷积核反卷积的过程,定义输入矩阵为 I(4×4),卷积核为 K(3×3),输出矩阵为 O(2×2): 卷积的过程为:Conv(I,W)=O 反卷积的过称为:Deconv ...

  9. Go语言并发

    Go语言并发机制初探   Go 语言相比Java等一个很大的优势就是可以方便地编写并发程序.Go 语言内置了 goroutine 机制,使用goroutine可以快速地开发并发程序, 更好的利用多核处 ...

  10. java 中的两个常用命令

    这两个命令,分别为,javac 和java 实际操作如下图: