Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1780–1790,Berlin, Germany, August 7-12, 2016. c©2016 Association for Computational Linguistics

Cross-Lingual Image Caption Generation

Takashi Miyazaki∗Yahoo Japan Corporation

Tokyo, Japantakmiyaz@yahoo-corp.jp

Nobuyuki Shimizu∗Yahoo Japan Corporation

Tokyo, Japannobushim@yahoo-corp.jp

Abstract

Automatically generating a natural lan-guage description of an image is a fun-damental problem in artificial intelligence.This task involves both computer visionand natural language processing and iscalled “image caption generation.” Re-search on image caption generation hastypically focused on taking in an imageand generating a caption in English as ex-isting image caption corpora are mostly inEnglish. The lack of corpora in languagesother than English is an issue, especiallyfor morphologically rich languages suchas Japanese. There is thus a need for cor-pora sufficiently large for image caption-ing in other languages. We have developeda Japanese version of the MS COCO cap-tion dataset and a generative model basedon a deep recurrent architecture that takesin an image and uses this Japanese ver-sion of the dataset to generate a captionin Japanese. As the Japanese portion ofthe corpus is small, our model was de-signed to transfer the knowledge represen-tation obtained from the English portioninto the Japanese portion. Experimentsshowed that the resulting bilingual compa-rable corpus has better performance than amonolingual corpus, indicating that imageunderstanding using a resource-rich lan-guage benefits a resource-poor language.

1 Introduction

Automatically generating image captions by de-scribing the content of an image using natural lan-guage sentences is a challenging task. It is es-pecially challenging for languages other than En-

∗∗ Both authors contributed equally to this work.

glish due to the sparsity of annotated resourcesin the target language. A promising solution tothis problem is to create a comparable corpus.To support the image caption generation task inJapanese, we have annotated images taken fromthe MS COCO caption dataset (Chen et al., 2015b)with Japanese captions. We call our corpus the“YJ Captions 26k Dataset.” While the size ofour dataset is comparatively large with 131,740captions, it greatly trails the 1,026,459 captionsin the MS COCO dataset. We were thus moti-vated to transfer the resources in English (sourcelanguage) to Japanese and thereby improve im-age caption generation in Japanese (target lan-guage). In natural language processing, a task in-volving transferring information across languagesis known as a cross-lingual natural language task,and well known tasks include cross-lingual senti-ment analysis (Chen et al., 2015a), cross-lingualnamed entity recognition (Zirikly and Hagiwara,2015), cross-lingual dependency parsing (Guo etal., 2015), and cross-lingual information retrieval(Funaki and Nakayama, 2015).

Existing work in the cross-lingual setting is usu-ally formulated as follows. First, to overcome thelanguage barrier, create a connection between thesource and target languages, generally by using adictionary or parallel corpus. Second, develop anappropriate knowledge transfer approach to lever-age the annotated data from the source languagefor use in training a model in the target language,usually supervised or semi-supervised. These twosteps typically amount to automatically generat-ing and expanding the pseudo-training data for thetarget language by exploiting the knowledge ob-tained from the source language.

We propose a very simple approach to cross-lingual image caption generation: exploit the En-glish corpus to improve the performance of imagecaption generation in another language. In this ap-

1780

proach, no resources besides the images found inthe corpus are used to connect the languages, andwe consider our dataset to be a comparable cor-pus. Paired texts in a comparable corpus describethe same topic, in this case an image, but unlike aparallel corpus, the texts are not exact translationsof each other. This unrestrictive setting enablesthe model to be used to create image caption re-sources in other languages. Moreover, this modelscales better than creating a parallel corpus withexact translations of the descriptions.

Our transfer model is very simple. We startwith a neural image caption model (Vinyals et al.,2015) and pretrain it using the English portion ofthe corpus. We then remove all of the trained neu-ral network layers except for one crucial layer, theone closest to the vision system. Next we attachan untrained Japanese generation model and trainit using the Japanese portion of the corpus. Thisresults in improved generation in Japanese com-pared to using only the Japanese portion of thecorpus. To the best of our knowledge, this is thefirst paper to address the problem of cross-lingualimage caption generation.

Our contribution is twofold. First, we have cre-ated and plan to release the first ever significantlylarge corpus for image caption generation for theJapanese language, forming a comparable corpuswith existing English datasets. Second, we havecreated a very simple model based on neural im-age caption generation for Japanese that can ex-ploit the English portion of the dataset. Again, weare the first to report results in cross-lingual im-age caption generation, and our surprisingly sim-ple method improves the evaluation metrics signif-icantly. This method is well suited as a baseline forfuture work on cross-lingual image caption gener-ation.

The paper is organized as follows. In the nextsection, we describe related work in image cap-tion generation and list the corpora currently avail-able for caption generation. Then in Section 3 wepresent the statistics for our corpus and explainhow we obtained them. We then explain our modelin Section 4 and present the results of our experi-mental evaluation in Section 5. We discuss the re-sults in Section 6, and conclude in Section 7 witha summary of the key points.

2 Related Work

Recent advances in computer vision research haveled to halving the error rate between 2012 and2014 at the Large Scale Visual Recognition Chal-lenge (Russakovsky et al., 2015), largely driven bythe adoption of deep neural networks (Krizhevskyet al., 2012; Simonyan and Zisserman, 2014; Don-ahue et al., 2014; Sharif Razavian et al., 2014).Similarly, we have seen increased adaptation ofdeep neural networks for natural language pro-cessing. In particular, sequence-to-sequence train-ing using recurrent neural networks has been suc-cessfully applied to machine translation (Cho etal., 2014; Bahdanau et al., 2015; Sutskever et al.,2014; Kalchbrenner and Blunsom, 2013).

These developments over the past few yearshave led to renewed interest in connecting visionand language. The encoder-decoder framework(Cho et al., 2014) inspired the development ofmany methods for generating image captions sincegenerating an image caption is analogous to trans-lating an image into a sentence.

Since 2014, many research groups have re-ported a significant improvement in image captiongeneration due to using a method that combinesa convolutional neural network with a recurrentneural network. Vinyals et al. used a convolu-tional neural network (CNN) with inception mod-ules for visual recognition and long short-termmemory (LSTM) for language modeling (Vinyalset al., 2015). Xu et al. introduced an attentionmechanism that aligns visual information and sen-tence generation for improving captions and un-derstanding of model behavior (Xu et al., 2015).The interested reader can obtain further informa-tion elsewhere (Bernardi et al., 2016).

These developments were made possible due toa number of available corpora. The following isa list of available corpora that align images withcrowd-sourced captions. A comprehensive list ofother kinds of corpora connecting vision and lan-guage, e.g., visual question answering, is availableelsewhere (Ferraro et al., 2015).

1. UIUC Pascal Dataset (Farhadi et al., 2010)includes 1,000 images with 5 sentences perimage; probably one of the first datasets.

2. Abstract Scenes Dataset (Clipart) (Zitnick etal., 2013) contains 10,020 images of childrenplaying outdoors associated with 60,396 de-scriptions.

1781

3. Flickr 30K Images (Young et al., 2014) ex-tends Flickr datasets (Rashtchian et al., 2010)and contains 31,783 images of people in-volved in everyday activities.

4. Microsoft COCO Dataset (MS COCO) (Linet al., 2014; Chen et al., 2015b) includesabout 328,000 images of complex everydayscenes with common objects in naturally oc-curring contexts. Each image is paired withfive captions.

5. Japanese UIUC Pascal Dataset (Funaki andNakayama, 2015) is a Japanese translation ofthe UIUC Pascal Dataset.

To the best of our knowledge, there are no largedatasets for image caption generation except forEnglish. With the release of the YJ Captions26k dataset, we aim to remedy this situation andthereby expand the research horizon by exploitingthe availability of bilingual image caption corpora.

3 Statistics for Data Set

In this section we describe the data statistics andhow we gathered data for the YJ Captions 26kdataset. For images, we used the Microsoft COCOdataset (Chen et al., 2015b). The images in thisdataset were gathered by searching for pairs of80 object categories and various scene types onFlickr. They thus tended to contain multiple ob-jects in their natural context. Objects in the scenewere labeled using per-instance segmentations.This dataset contains pictures of 91 basic objecttypes with 2.5 million labeled instances. To collectJapanese descriptions of the images, we used Ya-hoo! Crowdsourcing 1, a microtask crowdsourcingservice operated by Yahoo Japan Corporation.

Given 26,500 images taken from the train-ing part of the MS COCO dataset, we collected131,740 captions in total. The images had on av-erage 4.97 captions; the maximum number was 5and the minimum was 3. On average, each captionhad 23.23 Japanese characters. We plan to releasethe YJ Captions 26k dataset 2.

3.1 Crowdsourcing Procedure

Our captions were human generated using Yahoo!Crowdsourcing. As this crowdsourcing platformis operated in Japan, signing up for the service andparticipating require Japanese proficiency. Thus,

1http://crowdsourcing.yahoo.co.jp2http://research-lab.yahoo.co.jp/software/index.html

Figure 1: User Interface

we assumed that the participants were fluent inJapanese.

First, we posted a pilot task that asked the par-ticipants to describe an image. We then exam-ined the results and selected promising partici-pants (comprising a “white list”) for future task re-quests. That is, only the participants on the whitelist could see the next task. This selection pro-cess was repeated, and the final white list includedabout 600 participants. About 150 of them regu-larly participated in the actual image caption col-lection task. We modified the task request pageand user interface on the basis of our experiencewith the pilot task. In order to prevent their fa-tigue, the tasks were given in small batches sothat the participants were unable to work over longhours.

In our initial trials, we tried a direct translationof the instructions used in the MS-COCO Englishcaptions. This however did not produce Japanesecaptions comparable to those in English. This isbecause people describe what appears unfamiliarto them and do not describe things they take forgranted. Our examination of the results from thepilot tasks revealed that the participants generallythought that the pictures contained non-Japanesepeople and foreign places since the images origi-nated from Flickr and no scenery from Japan wasincluded in the image dataset. When Japanese

1782

crowds are shown pictures with scenery in the USor Europe in MS-COCO dataset, the scenes them-selves appear exotic and words such as ‘foreign’and ‘oversea’ would be everywhere in the descrip-tions. As such words are not common in the orig-inal dataset, and to make the corpus nicer comple-ment to the English dataset and to reduce the ef-fects of such cultural bias, we modified the instruc-tions: “2. Please give only factual statements”;“3. Please do not specify place names or nation-alities.” We also strengthened two sections in thetask request page and added more examples.

The interface is shown in Figure 1. The instruc-tions in the user interface can be translated intoEnglish as “Please explain the image using 16 ormore Japanese characters. Write a single sentenceas if you were writing an example sentence to beincluded in a textbook for learning Japanese. De-scribe all the important parts of the scene; do notdescribe unimportant details. Use correct punctu-ation. Write a single sentence, not multiple sen-tences or a phrase.”

Potential participants are shown task requestpages, and the participants select which crowd-sourcing task(s) to perform. The task request pagefor our task had the following instructions (En-glish translation):

1. Please explain an image using 16 or more Japanesecharacters. Please write a single sentence as if youwere writing an example sentence to be included in atextbook for learning Japanese.

(a) Do not use incorrect Japanese.(b) Use a polite style of speech (desu/masu style) as

well as correct punctuation.(c) Write a single complete sentence that ends with

a period. Do not write just a phrase or multiplesentences.

2. Please give only factual statements.(a) Do not write about things that might have hap-

pened or might happen in the future. Do not writeabout sounds.

(b) Do not speculate. Do not write about somethingabout which you feel uncertain.

(c) Do not state your feelings about the scene in thepicture. Do not use an overly poetic style.

(d) Do not use a demonstrative pronoun such as’this’ or ’here.’

3. Please do not specify place names or nationalities.(a) Please do not give proper names.

4. Please describe all the important parts of the scene; donot describe unimportant details.

Together with the instructions, we provided 15examples (1 good example; 14 bad examples).

Upon examining the collected data, manualchecks of first 100 images containing 500 captionsrevealed that 9 captions were clearly bad, and 12

captions had minor problems in descriptions. Inorder to further improve the quality of the corpus,we crowdsourced a new data-cleaning task. Weshowed each participant an image and five cap-tions that describe the image and asked to fix them.

The following is the instructions (English trans-lation) for the task request page for our data-cleaning task.

1. There are five sentences about a hyper-linked image,and several sentences require fixes in order to satisfythe conditions below. Please fix the sentences, andwhile doing so, tick a checkbox of the item (condition)being fixed.

2. The conditions that require fixes are:(a) Please fix typographical errors, omissions and

input-method-editor conversion misses.(b) Please remove or rephrase expressions such as

‘oversea’, ‘foreign’ and ‘foreigner.’(c) Please remove or rephrase expressions such as

‘image’, ‘picture’ and ‘photographed.’(d) Please fix the description if it does not match the

contents of the image.(e) Please remove or rephrase subjective expressions

and personal impressions.(f) If the statement is divided into several sentences,

please make it one sentence.(g) If the sentence is in a question form, please make

it a declarative sentence.(h) Please rewrite the entire sentence if meeting all

above conditions requires extensive modifica-tions.

(i) If there are less than 16 characters, please pro-vide additional descriptions so that the sentencewill be longer than 16 characters.

For each condition, we provided a pair of exam-ples (1 bad example and 1 fixed example).

To gather participants for the data-cleaning task,we crowdsourced a preliminary user qualificationtask that explained each condition requiring fixesin the first half, then quizzed the participants inthe second half. This time we obtained over 900qualified participants. We posted the data-cleaningtask to these qualified participants.

The interface is shown in Figure 2. The instruc-tions in the user interface are very similar to thetask request page, except that we have an addi-tional checkbox:

(j) All conditions are satisfied and no fixes were necessary.

We provided these checkboxes to be used as achecklist, so as to reduce failure by compensatingfor potential limits of participants’ memory and at-tention, and to ensure consistency and complete-ness in carrying out the data-cleaning task.

For this data-cleaning task, we had 26,500 im-ages totaling 132,500 captions checked by 267participants. The number of fixed captions are

1783

Figure 2: Data Cleaning Task User Interface

45,909. To our surprise, a relatively large por-tion of the captions were fixed by the participants.We suspect that in our data-cleaning task, the con-dition (e) was especially ambiguous for the par-ticipants, and they errored on the cautious side,fixing “a living room” to just “a room”, thinkingthat a room that looks like a living room may notbe a living room for the family who occupies thehouse, for example. Another example includes fix-ing “beautiful flowers” to just “flowers” becausebeauty is in the eye of the beholder and thoughtto be subjective. The percentage of the tickedcheckboxes is as follows: (a) 27.2%, (b) 5.0%, (c)12.3%, (d) 34.1%, (e) 28.4%, (f) 3.9%, (g) 0.3%,(h) 11.6%, (i) 18.5%, and (j) 24.0%. Note that acheckbox is ticked if there is at least one sentence

out of five that meets the condition. In machinelearning, this setting is called multiple-instancemultiple-label problem (Zhou et al., 2012). Wecannot directly infer how many captions corre-spond to a condition ticked by the participants.

After this data-cleaning task, we further re-moved a few more bad captions that came to ourattention. The resulting corpus finally contains131,740 captions as noted in the previous section.

4 Methodology

!"#$! !"#$!

%&'(

%)(

%*(

+,,-*.(

/01!

%)(

!"#$!

%*(

%)(

23!

23! *4*5623.!

*4*5623.!

&7!

7628*)(2'93:(423:;2:*7!

Figure 3: Model Overview

4.1 Model OverviewFigure 3 shows an overview of our model. Follow-ing the approach of Vinyals et al. (Vinyals et al.,2015), we used a discriminative model that max-imizes the probability of the correct descriptiongiven the image. Our model is formulated as

θ∗ = arg maxθ

∑(I,S)

N∑t=0

log p(St|I, S0, ..., St−1; θ), (1)

where the first summation is over pairs of an im-age I and its correct transcription S. For the sec-ond summation, the sum is over all words St in S,and N is the length of S. θ represents the modelparameters. Note that the second summation rep-resents the probability of the sentence with respectto the joint probability of its words.

We modeled p(St|I, S0, ..., St−1; θ) by using arecurrent neural network (RNN). To model the se-quences in the RNN, we let a fixed length hiddenstate or memory ht express the variable numberof words to be conditioned up to t − 1. The ht

1784

is updated after obtaining a new input xt using anon-linear function f , so that ht+1 = f(ht, xt).Since an LSTM network has state-of-the art per-formance in sequence modeling such as machinetranslation, we use one for f , which we explain inthe next section.

A combination of LSTM and CNN are used tomodel p(St|I, S0, ..., St−1; θ).

x−1 = WimCNN(I) (2)

xt = WeSt, t ∈ {0...N − 1} (3)

pt+1 = Softmax(WdLSTM(xt)),t ∈ {0...N − 1} (4)

where Wim is an image feature encoding matrix,We is a word embedding matrix, and Wd is a worddecoding matrix.

4.2 LSTM-based Language Model

An LSTM is an RNN that addresses the vanish-ing and exploding gradients problem and that han-dles longer dependencies well. An LSTM has amemory cell and various gates to control the in-put, the output, and the memory behaviors. Weuse an LSTM with input gate it, input modulationgate gt, output gate ot, and forgetting gate ft. Thenumber of hidden units ht is 256. At each timestep t, the LSTM state ct, ht is as follows:

it = σ(Wixxt + Wihht−1 + bi) (5)

ft = σ(Wfxxt + Wfhht−1 + bf ) (6)

ot = σ(Woxxt + Wohht−1 + bo) (7)

gt = ϕ(Wcxxt + Wchht−1 + bc) (8)

ct = ft ⊙ ct−1 + it ⊙ gt (9)

ht = ot ⊙ ϕ(ct), (10)

where σ(x) = (1 + e−x)−1 is a sigmoid function,ϕ(x) = (ex − e−x)/(ex + e−x) is a hyperbolictangent function, and ⊙ denotes the element-wiseproduct of two vectors. W and b are parameters tobe learned. From the values of the hidden units ht,the probability distribution of words is calculatedas

pt+1 = Softmax(Wdht). (11)

We use a simple greedy search to generate cap-tions as a sequence of words, and, at each timestep t, the predicted word is obtained using St =arg maxS pt.

4.3 Image Feature Extraction with DeepConvolutional Neural Network

The image recognition performance of deep con-volutional neural network models has rapidly ad-vanced in recent years, and they are now widelyused for various image recognition tasks. Weused a 16-layer VGGNet (Simonyan and Zisser-man, 2014), which was a top performer at the Im-ageNet Large Scale Visual Recognition Challengein 2014. A 16-layer VGGNet is composed of 13convolutional layers having small 3x3 filter ker-nels and 3 fully connected layers. An image fea-ture is extracted as a 4096-dimensional vector ofthe VGGNet’s fc7 layer, which is the second fullyconnected layer from the output layer. VGGNetwas pretrained using the ILSVRC2014 subset ofthe ImageNet dataset, and its weights were not up-dated through training.

4.4 Dataset Split

Because our caption dataset is annotated for only26,500 images of the MS COCO training set,we reorganized the dataset split for our experi-ments. Training and validation set images of theMS COCO dataset were mixed and split into fourblocks, and these blocks were assigned to training,validation, and testing as shown in Table 1. Allblocks were used for the English caption dataset.Blocks B, C, and D were used for the Japanesecaption dataset.

block no. of images split languageA 96,787 train EnB 22,500 train En, JaC 2,000 val En, JaD 2,000 test En, Ja

total 123,287

Table 1: Dataset Split

4.5 Training

The models were trained using minibatch stochas-tic gradient descent, and the gradients were com-puted by backpropagation through time. Parame-ter optimization was done using the RMSprop al-gorithm (Tieleman and Hinton, 2012) with an ini-tial learning rate of 0.001, a decay rate of 0.999,and ϵ of 1.0−8. Each image minibatch contained100 image features, and the corresponding cap-tion minibatch contained one sampled caption perimage. To evaluate the effectiveness of Japanese

1785

image caption generation, we used three learningschemes.

Monolingual learning This was the base-line method. The model had only one LSTMfor Japanese caption generation, and only theJapanese caption corpus was used for training.

Alternate learning In this scheme, a model hadtwo LSTMs, one for English and one for Japanese.The training batches for captions contained eitherEnglish or Japanese, and the batches were fedinto the model alternating between English andJapanese.

Transfer learning A model with one LSTMwas trained completely for the English dataset.The trained LSTM was then removed, and anotherLSTM was added for Japanese caption genera-tion. Wim was shared between the English andJapanese training.

These models were implemented using theChainer neural network framework (Tokui et al.,2015). We consulted NeuralTalk (Karpathy,2014), an open source implemenation of neuralnetwork based image caption generation system,for training parameters and dataset preprocessing.Training took about one day using NVIDIA TI-TAN X/Tesla M40 GPUs.

5 Evaluation

0 10000 20000 30000 40000 50000No. of iterations

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CIDEr

transfer

monolingual

alternate

Figure 4: Learning Curve Represented by CIDErScore

5.1 Evaluation Metrics

We used six standard metrics for evaluatingthe quality of the generated Japanese sentences:BLEU-1, BLEU-2, BLEU-3, BLEU-4 (Papineniet al., 2002), ROUGE-L (Lin, 2004), and CIDEr-D(Vedantam et al., 2014). We used the COCO cap-tion evaluation tool (Chen et al., 2015b) to com-

pute the metrics. BLEU (Papineni et al., 2002)was originally designed for automatic machinetranslation. By counting n-gram co-occurrences, itrates the quality of a translated sentence given sev-eral reference sentences. To apply BLEU, we con-sidered that generating image captions is the sameas translating images into sentences. ROUGE(Lin, 2004) is an evaluation metric designed byadapting BLEU to evaluate automatic text sum-marization algorithms. ROUGE is based on thelongest common subsequences instead of n-grams.CIDEr (Vedantam et al., 2014) is a metric devel-oped specifically for evaluating image captions.It measures consensus in image captions by per-forming a term-frequency inverse document fre-quency (TF-IDF) weighting for each n-gram. Weused a robust variant of CIDEr called CIDEr-D.For all evaluation metrics, higher scores are better.In addition to these metrics, MS COCO captionevaluation (Chen et al., 2015b) uses METEOR(Lavie, 2014), another metric for evaluating auto-matic machine translation. Although METEOR isa good metric, it uses an English thesaurus. It wasnot used in our study due to the lack of a thesaurusfor the Japanese language.

The CIDEr and METEOR metrics perform wellin terms of correlation with human judgment(Bernardi et al., 2016). Although BLEU is unableto sufficiently discriminate between judgments,we report the BLEU figures as well since their usein literature is widespread. In the next section, wefocus our analysis on CIDEr.

0 5000 10000 15000 20000 25000No. of images (Ja)

0.40

0.45

0.50

0.55

0.60

0.65

CIDEr

transfer

monolingual

Figure 5: CIDEr Score vs. Japanese Data Set Size

5.2 Results

Table 2 shows the evaluation metrics for varioussettings of cross-lingual transfer learning. All val-ues were calculated for Japanese captions gener-

1786

no. of images metricsEn Ja BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L CIDEr-D

monolingual 0 22,500 0.715 0.573 0.468 0.379 0.616 0.580alternate 119,287 22,500 0.709 0.565 0.460 0.370 0.611 0.568transfer 119,287 22,500 0.717 0.574 0.469 0.380 0.619 0.625

Table 2: Evaluation Metrics

ated for test set images. Our proposed model is la-beled “transfer.” As you can see, it outperformedthe other two models for every metric. In par-ticular, the CIDEr-D score was about 4% higherthan that for the monolingual baseline. The per-formance of a model trained using the Englishand Japanese corpora alternately is shown on theline label “alternate.” Surprisingly, this model hadlower performance than the baseline model.

In Figure 4, we plot the learning curves rep-resented by the CIDEr score for the Japanesecaptions generated for the validation set images.Transfer learning from English to Japanese con-verged faster than learning from the Japanesedataset or learning by training from both lan-guages alternately. Figure 5 shows the relation-ship between the CIDEr score and the Japanesedataset size (number of images). The modelspretrained using English captions (blue line) out-performed the ones trained using only Japanesecaptions for all training dataset sizes. As canbe seen by comparing the case of 4,000 im-ages with that of 20,000 images, the improvementdue to cross-lingual transfer was larger when theJapanese dataset was smaller. These results showthat pretraining the model with all available En-glish captions is roughly equivalent to training themodel with captions for 10,000 additional imagesin Japanese. This, in our case, nearly halves thecost of building the corpus.

Examples of machine-generated captions alongwith the crowd-written ground truth captions (En-glish translations) are shown in Figure 6.

6 Discussion

Despite our initial belief, training by alternatingEnglish and Japanese input batch data for learningboth languages did not work well for either lan-guage. As Japanese is a morphologically rich lan-guage and word ordering is subject-object-verb,it is one of most distant languages from English.We suspect that the alternating batch training inter-fered with learning the syntax of either language.

Moreover, when we tried character-based modelsfor both languages, the performance was signif-icantly lower. This was not surprising becauseone word in English is roughly two characters inJapanese, and presumably differences in the lan-guage unit should affect performance. Perhaps notsurprisingly, cross-lingual transfer was more ef-fective when the resources in the target languageare poor. Convergence was faster with the sameamount of data in the target language when pre-training in the source language was done ahead oftime. These two findings ease the burden of devel-oping a large corpus in a resource poor language.

7 Conclusion

We have created an image caption dataset for theJapanese language by collecting 131,740 captionsfor 26,500 images using the Yahoo! Crowdsourc-ing service in Japan. We showed that pretraining aneural image caption model with the English por-tion of the corpus improves the performance of aJapanese caption generation model subsequentlytrained using Japanese data. Pretraining the modelusing the English captions of 119,287 images wasroughly equivalent to training the model using thecaptions of 10,000 additional images in Japanese.This, in our case, nearly halves the cost of buildinga corpus. Since this performance gain is obtainedwithout modifying the original monolingual imagecaption generator, the proposed model can serve asa strong baseline for future research in this area.We hope that our dataset and proposed methodkick start studies on cross-lingual image captiongeneration and that many others follow our lead.

ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-

gio. 2015. Neural machine translation by jointlylearning to align and translate. In International Con-ference on Learning Representation (ICLR).

Raffaella Bernardi, Ruket Cakici, Desmond Elliott,Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis,Frank Keller, Adrian Muscat, and Barbara Plank.

1787

Figure 6: Image Caption Generation Examples

1788

2016. Automatic description generation from im-ages: A survey of models, datasets, and evaluationmeasures. arXiv preprint arXiv:1601.03896.

Qiang Chen, Wenjie Li, Yu Lei, Xule Liu, and Yanxi-ang He. 2015a. Learning to adapt credible knowl-edge in cross-lingual sentiment analysis. In Pro-ceedings of the 53rd Annual Meeting of the Associ-ation for Computational Linguistics and the 7th In-ternational Joint Conference on Natural LanguageProcessing (Volume 1: Long Papers), pages 419–429, Beijing, China, July. Association for Computa-tional Linguistics.

Xinlei Chen, Tsung-Yi Lin Hao Fang, Ramakr-ishna Vedantam, Saurabh Gupta, Piotr Dollr, andC. Lawrence Zitnick. 2015b. Microsoft coco cap-tions: Data collection and evaluation server. arXivpreprint arXiv:1504.00325.

Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder–decoderfor statistical machine translation. In Proceedings ofthe 2014 Conference on Empirical Methods in Nat-ural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar, October. Association for Com-putational Linguistics.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoff-man, Ning Zhang, Eric Tzeng, and Trevor Darrell.2014. Decaf: A deep convolutional activation fea-ture for generic visual recognition. In InternationalConference in Machine Learning (ICML).

Ali Farhadi, Mohsen Hejrati, Mohammad AminSadeghi, Peter Young, Cyrus Rashtchian, JuliaHockenmaier, and David Forsyth. 2010. Every pic-ture tells a story: Generating sentences from images.In Proceedings of the 11th European Conference onComputer Vision: Part IV, ECCV’10, pages 15–29,Berlin, Heidelberg. Springer-Verlag.

Francis Ferraro, Nasrin Mostafazadeh, Ting-HaoHuang, Lucy Vanderwende, Jacob Devlin, MichelGalley, and Margaret Mitchell. 2015. A survey ofcurrent datasets for vision and language research.In Proceedings of the 2015 Conference on Empiri-cal Methods in Natural Language Processing, pages207–213, Lisbon, Portugal, September. Associationfor Computational Linguistics.

Ruka Funaki and Hideki Nakayama. 2015. Image-mediated learning for zero-shot cross-lingual doc-ument retrieval. In Proceedings of the 2015 Con-ference on Empirical Methods in Natural Lan-guage Processing, pages 585–590, Lisbon, Portugal,September. Association for Computational Linguis-tics.

Jiang Guo, Wanxiang Che, David Yarowsky, HaifengWang, and Ting Liu. 2015. Cross-lingual depen-dency parsing based on distributed representations.In Proceedings of the 53rd Annual Meeting of the

Association for Computational Linguistics and the7th International Joint Conference on Natural Lan-guage Processing (Volume 1: Long Papers), pages1234–1244, Beijing, China, July. Association forComputational Linguistics.

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrentcontinuous translation models. In Proceedings ofthe 2013 Conference on Empirical Methods in Natu-ral Language Processing, pages 1700–1709, Seattle,Washington, USA, October. Association for Compu-tational Linguistics.

Andrej Karpathy. 2014. Neuraltalk. https://github.com/karpathy/neuraltalk.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hin-ton. 2012. Imagenet classification with deep con-volutional neural networks. In F. Pereira, C.J.C.Burges, L. Bottou, and K.Q. Weinberger, editors,Advances in Neural Information Processing Systems25, pages 1097–1105. Curran Associates, Inc.

Michael Denkowski Alon Lavie. 2014. Meteor univer-sal: Language specific translation evaluation for anytarget language. ACL 2014, page 376.

Tsung-Yi Lin, Michael Maire, Serge Belongie, JamesHays, Pietro Perona, Deva Ramanan, Piotr Dollar,and C. Lawrence Zitnick, 2014. Computer Vision– ECCV 2014: 13th European Conference, Zurich,Switzerland, September 6-12, 2014, Proceedings,Part V, chapter Microsoft COCO: Common Objectsin Context, pages 740–755. Springer InternationalPublishing, Cham.

Chin-Yew Lin. 2004. Rouge: A package for auto-matic evaluation of summaries. Text summarizationbranches out: Proceedings of the ACL-04 workshop,8.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automaticevaluation of machine translation. In Proceedingsof the 40th Annual Meeting on Association for Com-putational Linguistics, ACL ’02, pages 311–318,Stroudsburg, PA, USA. Association for Computa-tional Linguistics.

Cyrus Rashtchian, Peter Young, Micah Hodosh, andJulia Hockenmaier. 2010. Collecting image annota-tions using amazon’s mechanical turk. In Proceed-ings of the NAACL HLT 2010 Workshop on CreatingSpeech and Language Data with Amazon’s Mechan-ical Turk, CSLDAMT ’10, pages 139–147, Strouds-burg, PA, USA. Association for Computational Lin-guistics.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause,Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An-drej Karpathy, Aditya Khosla, Michael Bernstein,Alexander C. Berg, and Li Fei-Fei. 2015. Ima-geNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision (IJCV),115(3):211–252.

1789

Ali Sharif Razavian, Hossein Azizpour, Josephine Sul-livan, and Stefan Carlsson. 2014. Cnn features off-the-shelf: An astounding baseline for recognition.In The IEEE Conference on Computer Vision andPattern Recognition (CVPR) Workshops, June.

K. Simonyan and A. Zisserman. 2014. Very deep con-volutional networks for large-scale image recogni-tion. CoRR, abs/1409.1556.

Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014.Sequence to sequence learning with neural net-works. In Advances in neural information process-ing systems, pages 3104–3112.

T. Tieleman and G. Hinton. 2012. Lecture 6.5—RmsProp: Divide the gradient by a running averageof its recent magnitude. COURSERA: Neural Net-works for Machine Learning.

Seiya Tokui, Kenta Oono, Shohei Hido, and JustinClayton. 2015. Chainer: a next-generation opensource framework for deep learning. In Proceedingsof Workshop on Machine Learning Systems (Learn-ingSys) in The Twenty-ninth Annual Conference onNeural Information Processing Systems (NIPS).

Ramakrishna Vedantam, C Lawrence Zitnick, andDevi Parikh. 2014. Cider: Consensus-basedimage description evaluation. arXiv preprintarXiv:1411.5726.

Oriol Vinyals, Alexander Toshev, Samy Bengio, andDumitru Erhan. 2015. Show and tell: A neural im-age caption generator. In The IEEE Conference onComputer Vision and Pattern Recognition (CVPR),June.

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,Aaron Courville, Ruslan Salakhutdinov, RichardZemel, and Yoshua Bengio. 2015. Show, attend andtell: Neural image caption generation with visual at-tention. arXiv preprint arXiv:1502.03044.

Peter Young, Alice Lai, Micah Hodosh, and JuliaHockenmaier. 2014. From image descriptions tovisual denotations: New similarity metrics for se-mantic inference over event descriptions. Transac-tions of the Association for Computational Linguis-tics, 2:67–78.

Zhi-Hua Zhou, Min-Ling Zhang, Sheng-Jun Huang,and Yu-Feng Li. 2012. Multi-instance multi-labellearning. Artificial Intelligence, 176(1):2291–2320.

Ayah Zirikly and Masato Hagiwara. 2015. Cross-lingual transfer of named entity recognizers withoutparallel corpora. In Proceedings of the 53rd AnnualMeeting of the Association for Computational Lin-guistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 2: ShortPapers), pages 390–396, Beijing, China, July. Asso-ciation for Computational Linguistics.

C.L. Zitnick, D. Parikh, and L. Vanderwende. 2013.Learning the visual interpretation of sentences. InComputer Vision (ICCV), 2013 IEEE InternationalConference on, pages 1681–1688, Dec.

1790