|X

simNet: Stepwise Image-Topic Merging Network for

Generating Detailed and Comprehensive Image Captions

1

Fenglin Liu¹, Xuancheng Ren²*, Yuanxin Liu¹, Houfeng Wang² and Xu Sun²

¹ Beijing University of Posts and Telecommunications, Beijing, China

² Peking University, Beijing, China

* Equal Contributions

|X

CONTENTS

1 Introduction

2 Approach

3 Experiment

4 Analysis

|X

Introduction

|X

simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions

4

·Soft-Attention: Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015

·ATT-FCN : Image captioning with semantic attention. In CVPR 2016

Figure 1: Examples of using different attention mechanisms.

|X

Introduction: Soft-Attention

5

Image Image Features Caption

encode decodeSoft-Attention:

·Soft-Attention: Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015

omitting “dog” and “mouse”

|X

Introduction: ATT-FCN

6

Image Image Keywords Caption

encode decodeATT-FCN:

·ATT-FCN : Image captioning with semantic attention. In CVPR 2016

missing “open” and mislocating “dog”

|X

Introduction: SimNet

7

Image

Image Features

decode

Image Keywords

encode CaptionsimNet:

|X

Introduction: Main idea

8

The visual information captured by CNN

The topics extracted by a topic extractor

The merging gate then adaptively adjusts the weight between visual attention and topic attention

Figure 2: Illustration of the main idea.

|X

Contributions

9

We propose a novel approach that can effectively merge the information in the image and the topics.

The generated captions are both detailed and comprehensive.

The proposed approach outperforms previous works in terms of SPICE, which correlates the best with human judgments.

|X

Approach

|X 11

Overview

|X 12

Approach: Image Encoder

Image Encoder: ResNet152

He et al., 2016: Deep residual learning for image recognition. In CVPR 2016.

|X 13

Approach: Image Encoder

Feature map:

|X 14

Approach: Topic Extractor

Topic Extractor: Multiple Instance Learning

Zhang et al., 2006: Multiple instance boosting for object detection. In NIPS 2006.

Fang et al., 2015: From captions to visual concepts and back. In CVPR2015

|X 15

Approach: Input Attention

Input Attention:

Xu et al., 2015 : Show, attend and tell: Neural image caption generation with visual attention. In PMLR 2015

|X 16

Approach: Output Attention

Output Attention:

You et al., 2016 : Image captioning with semantic attention. In CVPR 2016

Lu et al ., 2017 : Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR 2017

|X 17

Approach: Visual Information

the visual information:

Output Attention:

|X 18

Approach: Previous Topic Attention

Topic Attention (Previous work):

Lacking the attentive visual informationwhen selecting topic!

You et al., 2016 : Image captioning with semantic attention. In CVPR 2016

|X 19

Approach: Our Topic Attention

Topic Attention (Our):

|X 20

Approach: Contextual Information

Topic Attention (Our):

the contextual information:

|X 21

Approach: Merging Gate

How to make full use of the visual information and the contextual information?

|X 22

Approach: Merging Gate

Visual information(e.g., “behind”, “red” is better)

Contextual information(e.g., “people”, “table” is better)

|X 23

Approach: Merging Gate

( Where σ is the sigmoid function )

|X 24

Approach: Merging Gate

The scoring function S is designed to evaluate the importance of the topic attention.

|X 25

Approach: Merging Gate

|X 26

Approach: Merging Gate

Share Weights

Share Weights

|X 27

Generating Words

the contextual information:

|X

Experiments

|X

Experiments

29

Microsoft COCO(MSCOCO) and Flickr30k

DatasetEvaluation Metrics

Sparrow bird on branch, withbeak inspecting leaves on branch.

A bird sitting on the branch of atree near leaves.

A bird that is sitting in a tree. a bird sitting on a branch of a

tree. a bird that is on a small branch

of a tree.

SPICE

CIDEr

BLEU

METEOR

ROUGE

Correlates the best with human Judgments !

|X

Experiments: Results (MSCOCO)

30

ComparableModels

|X

Experiments: Results (MSCOCO)

31

Competitive

|X

Analysis

|X

Analysis: The Contributions of The Sub-modules

33

Comprehensiveness Detailedness

|X

Analysis: Output Attention

34

The output attention is much more effective than the input attention

|X

Analysis: Visual Attention

35

A combination of the input attention and the output attention makes the results even better

|X

Analysis: Topic Attention

36

The topic attention is better at identifying objects but worse at identifying attributes.

|X

Analysis: Visual Attention + Topic Attention

37

Combing the visual attention and the topic attention directly results in a huge boost in performance

|X

Analysis: Full Model

38

Applying the merging gate is essential to the overall performance.

|X

Analysis: Visualization

39

The upper part shows the attention weights of each of 5 extracted topics. Deeper color means larger in value. The middle part shows the value of the merging gate. Determines the importance of the topic attention. The lower part shows the visualization of visual attention. The blue shade indicates the output attention.

The red shade indicates the input attention.

|X

Analysis: Visualization

40

Visual information “chair” is more important than contextual information “bed”

|X

Analysis: Examples

41

erroneoustopic “kitchen”

lacking “mouse”

missing “wooden” error count

|X

Conclusion

42

Stepwise image-topic merging network can adaptively combine the visual and the semantic attention to achieve substantial improvements.

The generated captions are both detailed and comprehensive

Our approach outperforms previous works in terms of SPICE on COCO and Flickr datasets.

|X 43

Thank you!If you have any questions about our paper, you can send a email to lfl@bupt.edu.cn

|X

Experiments: Results (Flickr30k)

44

|X

Analysis: Topic Extraction

45

The reason of using objects as topics is that they are easier to identify so that the generation suffers less from erroneous predictions.

It proves that for semantic attention, it is also important to limit the absolute number of incorrect visual words instead of merely the precision or the recall.

|X

Analysis: Merging Gate

46

|X

Analysis: Error Analysis

47

There are mainly three types of errors, i.e. distance (32, 26%), movement (22,18%), and object(60, 49%), with 9 (7%) other errors.

|X

Analysis: Error Analysis

48

There are mainly three types of errors, i.e. distance (32, 26%), movement (22,18%), and object(60, 49%), with 9 (7%) other errors.

|X

Experiments: Results (MSCOCO)

49

|X

Analysis: Topic Attention

50