CLEVR: A Diagnostic Dataset forCompositional Language and Elementary Visual Reasoning

Justin Johnson1,2∗

Li Fei-Fei1Bharath Hariharan2

C. Lawrence Zitnick2Laurens van der Maaten2

Ross Girshick2

1Stanford University 2Facebook AI Research

Abstract

When building artificial intelligence systems that canreason and answer questions about visual data, we needdiagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answer-ing can help, but have strong biases that models can exploitto correctly answer questions without reasoning. They alsoconflate multiple sources of error, making it hard to pinpointmodel weaknesses. We present a diagnostic dataset thattests a range of visual reasoning abilities. It contains mini-mal biases and has detailed annotations describing the kindof reasoning each question requires. We use this dataset toanalyze a variety of modern visual reasoning systems, pro-viding novel insights into their abilities and limitations.

1. Introduction

A long-standing goal of artificial intelligence researchis to develop systems that can reason and answer ques-tions about visual information. Recently, several datasetshave been introduced to study this problem [4, 10, 21, 26,32, 46, 49]. Each of these Visual Question Answering(VQA) datasets contains challenging natural language ques-tions about images. Correctly answering these questionsrequires perceptual abilities such as recognizing objects,attributes, and spatial relationships as well as higher-levelskills such as counting, performing logical inference, mak-ing comparisons, or leveraging commonsense world knowl-edge [31]. Numerous methods have attacked these prob-lems [2, 3, 9, 24, 44], but many show only marginal im-provements over strong baselines [4, 16, 48]. Unfortunately,our ability to understand the limitations of these methods isimpeded by the inherent complexity of the VQA task. Aremethods hampered by failures in recognition, poor reason-ing, lack of commonsense knowledge, or something else?

The difficulty of understanding a system’s competences

∗Work done during an internship at FAIR.

Q: Are there an equal number of large things and metal spheres?Q: What size is the cylinder that is left of the brown metal thing thatis left of the big sphere? Q: There is a sphere with the same size as themetal cube; is it made of the same material as the small red sphere?Q: How many objects are either small cylinders or metal things?

Figure 1. A sample image and questions from CLEVR. Questionstest aspects of visual reasoning such as attribute identification,counting, comparison, multiple attention, and logical operations.

is exemplified by Clever Hans, a 1900s era horse who ap-peared to be able to answer arithmetic questions. Care-ful observation revealed that Hans was correctly “answer-ing” questions by reacting to cues read off his human ob-servers [30]. Statistical learning systems, like those usedfor VQA, may develop similar “cheating” approaches tosuperficially “solve” tasks without learning the underlyingreasoning processes [35, 36]. For instance, a statisticallearner may correctly answer the question “What covers theground?” not because it understands the scene but becausebiased datasets often ask questions about the ground whenit is snow-covered [1, 47]. How can we determine whethera system is capable of sophisticated reasoning and not justexploiting biases of the world, similar to Clever Hans?

In this paper we propose a diagnostic dataset for study-ing the ability of VQA systems to perform visual reasoning.We refer to this dataset as the Compositional Language andElementary Visual Reasoning diagnostics dataset (CLEVR;pronounced as clever in homage to Hans). CLEVR contains100k rendered images and about one million automatically-generated questions, of which 853k are unique. It has chal-

1

arX

iv:1

612.

0689

0v1

[cs

.CV

] 2

0 D

ec 2

016

lenging images and questions that test visual reasoning abil-ities such as counting, comparing, logical reasoning, andstoring information in memory, as illustrated in Figure 1.

We designed CLEVR with the explicit goal of enablingdetailed analysis of visual reasoning. Our images depictsimple 3D shapes; this simplifies recognition and allows usto focus on reasoning skills. We ensure that the informa-tion in each image is complete and exclusive so that ex-ternal information sources, such as commonsense knowl-edge, cannot increase the chance of correctly answeringquestions. We minimize question-conditional bias via re-jection sampling within families of related questions, andavoid degenerate questions that are seemingly complex butcontain simple shortcuts to the correct answer. Finally, weuse structured ground-truth representations for both imagesand questions: images are annotated with ground-truth ob-ject positions and attributes, and questions are representedas functional programs that can be executed to answer thequestion (see Section 3). These representations facilitate in-depth analyses not possible with traditional VQA datasets.

These design choices also mean that while images inCLEVR may be visually simple, its questions are complexand require a range of reasoning skills. For instance, factor-ized representations may be required to generalize to unseencombinations of objects and attributes. Tasks such as count-ing or comparing may require short-term memory [15] orattending to specific objects [24, 44]. Questions that com-bine multiple subtasks in diverse ways may require compo-sitional systems [2, 3] to answer.

We use CLEVR to analyze a suite of VQA models anddiscover weaknesses that are not widely known. For ex-ample, we find that current state-of-the-art VQA modelsstruggle on tasks requiring short term memory, such as com-paring the attributes of objects, or compositional reasoning,such as recognizing novel attribute combinations. These ob-servations point to novel avenues for further research.

Finally, we stress that accuracy on CLEVR is not an endgoal in itself: a hand-crafted system with explicit knowl-edge of the CLEVR universe might work well, but will notgeneralize to real-world settings. Therefore CLEVR shouldbe used in conjunction with other VQA datasets in order tostudy the reasoning abilities of general VQA systems.

The CLEVR dataset, as well as code for generating newimages and questions, will be made publicly available.

2. Related WorkIn recent years, a range of benchmarks for visual under-

standing have been proposed, including datasets for imagecaptioning [7, 8, 23, 45], referring to objects [19], rela-tional graph prediction [21], and visual Turing tests [12, 27].CLEVR, our diagnostic dataset, is most closely related tobenchmarks for visual question answering [4, 10, 21, 26,32, 37, 46, 49], as it involves answering natural-language

questions about images. The two main differences betweenCLEVR and other VQA datasets are that: (1) CLEVR min-imizes biases of prior VQA datasets that can be used bylearning systems to answer questions correctly without vi-sual reasoning and (2) CLEVR’s synthetic nature and de-tailed annotations facilitate in-depth analyses of reasoningabilities that are impossible with existing datasets.

Prior work has attempted to mitigate biases in VQAdatasets in simple cases such as yes/no questions [12, 47],but it is difficult to apply such bias-reduction approachesto more complex questions without a high-quality se-mantic representation of both questions and answers. InCLEVR, this semantic representation is provided by thefunctional program underlying each image-question pair,and biases are largely eliminated via sampling. Winogradschemas [22] are another approach for controlling bias inquestion answering: these questions are carefully designedto be ambiguous based on syntax alone and require com-monsense knowledge. Unfortunately this approach doesnot scale gracefully: the first phase of the 2016 WinogradSchema Challenge consists of just 60 hand-designed ques-tions. CLEVR is also related to the bAbI question answer-ing tasks [38] in that it aims to diagnose a set of clearlydefined competences of a system, but CLEVR focuses onvisual reasoning whereas bAbI is purely textual.

We are also not the first to consider synthetic data forstudying (visual) reasoning. SHRDLU performed sim-ple, interactive visual reasoning with the goal of movingspecific objects in the visual scene [40]; this study wasone of the first to demonstrate the brittleness of manu-ally programmed semantic understanding. The pioneer-ing DAQUAR dataset [28] contains both synthetic andhuman-written questions, but they only generate 420 syn-thetic questions using eight text templates. VQA [4] con-tains 150,000 natural-language questions about abstractscenes [50], but these questions do not control for question-conditional bias and are not equipped with functional pro-gram representations. CLEVR is similar in spirit to theSHAPES dataset [3], but it is more complex and varied bothin terms of visual content and question variety and complex-ity: SHAPES contains 15,616 total questions with just 244unique questions while CLEVR contains nearly a millionquestions of which 853,554 are unique.

3. The CLEVR Diagnostic DatasetCLEVR provides a dataset that requires complex rea-

soning to solve and that can be used to conduct rich diag-nostics to better understand the visual reasoning capabili-ties of VQA systems. This requires tight control over thedataset, which we achieve by using synthetic images andautomatically generated questions. The images have asso-ciated ground-truth object locations and attributes, and thequestions have an associated machine-readable form. These

2

objectsobjectsobjects

yes/nonumberobjects

objects

objectsobject

valueobjects

numbernumber

CLEVRfunctioncatalog

Relate

EqualLess/More

yes/no

Equal yes/novaluevalue

ExistCount

AndOr

Filter<attr>

objectobjects Unique

Infrontvs.behind

Sizes,colors,shapes,andmaterials

Leftvs.right

Largegraymetalsphere

Largeredmetalcube

Smallbluemetalcylinder

Smallgreenmetalsphere

Largebrownrubbersphere

Largepurplerubbercylinder

Smallcyanrubbercube

Smallyellowrubbersphere

BehindInfront

Left Right

Filtercolor

Filtershape Unique Relate Filter

shape Unique Querycolor

yellow sphere

value

right cube

Whatcoloristhecubetotherightoftheyellowsphere?

object Query<attr> value

Filtercolor Unique Relate

green left

Filtersize Unique Relate

small infront

And Count

Howmanycylindersareinfrontofthesmallthingandontheleftsideofthegreenobject?

Samplechain-structuredquestion:

Sampletree-structuredquestion:

object Same<attr> objects

Filtershape

cylinder

Figure 2. A field guide to the CLEVR universe. Left: Shapes, attributes, and spatial relationships. Center: Examples of questions andtheir associated functional programs. Right: Catalog of basic functions used to build questions. See Section 3 for details.

ground-truth structures allow us to analyze models basedon, for example: question type, question topology (chainvs. tree), question length, and various forms of relationshipsbetween objects. Figure 2 gives a brief overview of the maincomponents of CLEVR, which we describe in detail below.

Objects and relationships. The CLEVR universe con-tains three object shapes (cube, sphere, and cylinder) thatcome in two absolute sizes (small and large), two materi-als (shiny “metal” and matte “rubber”), and eight colors.Objects are spatially related via four relationships: “left”,“right”, “behind”, and “in front”. The semantics of theseprepositions are complex and depend not only on relativeobject positions but also on camera viewpoint and context.We found that generating questions that invoke spatial rela-tionships with semantic accord was difficult. Instead werely on a simple and unambiguous definition: projectingthe camera viewpoint vector onto the ground plane definesthe “behind” vector, and one object is behind another ifits ground-plane position is further along the “behind” vec-tor. The other relationships are similarly defined. Figure 2(left) illustrates the objects, attributes, and spatial relation-ships in CLEVR. The CLEVR universe also includes onenon-spatial relationship type that we refer to as the same-attribute relation. Two objects are in this relationship ifthey have equal attribute values for a specified attribute.

Scene representation. Scenes are represented as collec-tions of objects annotated with shape, size, color, material,and position on the ground-plane. A scene can also be rep-resented by a scene graph [17, 21], where nodes are objectsannotated with attributes and edges connect spatially related

objects. A scene graph contains all ground-truth informa-tion for an image and could be used to replace the visioncomponent of a VQA system with perfect sight.

Image generation. CLEVR images are generated by ran-domly sampling a scene graph and rendering it usingBlender [6]. Every scene contains between three and tenobjects with random shapes, sizes, materials, colors, andpositions. When placing objects we ensure that no objectsintersect, that all objects are at least partially visible, andthat there are small horizontal and vertical margins betweenthe image-plane centers of each pair of objects; this helpsreduce ambiguity in spatial relationships. In each image thepositions of the lights and camera are randomly jittered.

Question representation. Each question in CLEVR is as-sociated with a functional program that can be executed onan image’s scene graph, yielding the answer to the question.Functional programs are built from simple basic functionsthat correspond to elementary operations of visual reason-ing such as querying object attributes, counting sets of ob-jects, or comparing values. As shown in Figure 2, complexquestions can be represented by compositions of these sim-ple building blocks. Full details about each basic functioncan be found in the supplementary material.

As we will see in Section 4, representing questions asfunctional programs enables rich analysis that would beimpossible with natural-language questions. A question’sfunctional program tells us exactly which reasoning abili-ties are required to solve it, allowing us to compare perfor-mance on questions requiring different types of reasoning.

3

We categorize questions by question type, defined by theoutermost function in the question’s program; for examplethe questions in Figure 2 have types query-color and exist.Figure 3 shows the number of questions of each type.

Question families. We must overcome several key chal-lenges to generate a VQA dataset using functional pro-grams. Functional building blocks can be used to constructan infinite number of possible functional programs, and wemust decide which program structures to consider. We alsoneed a method for converting functional programs to natu-ral language in a way that minimizes question-conditionalbias. We solve these problems using question families.

A question family contains a template for constructingfunctional programs and several text templates providingmultiple ways of expressing these programs in natural lan-guage. For example, the question “How many red thingsare there?” can be formed by instantiating the text tem-plate “How many <C> <M> things are there?”, bind-ing the parameters <C> and <M> (with types “color” and“material”) to the values red and nil. The functional pro-gram count(filter color(red, scene())) for this question canbe formed by instantiating the associated program templatecount(filter color(<C>, filter material(<M>, scene())))

with the same values, using the convention that functionstaking a nil input are removed after instantiation.

CLEVR contains a total of 90 question families, eachwith a single program template and an average of four texttemplates. Text templates were generated by manually writ-ing one or two templates per family and then crowdsourcingquestion rewrites. To further increase language diversity weuse a set of synonyms for each shape, color, and material.With up to 19 parameters per template, a small number offamilies can generate a huge number of unique questions;Figure 3 shows that of the nearly one million questions inCLEVR, more than 853k are unique. CLEVR can easily beextended by adding new question families.

Question generation. Generating a question for an im-age is conceptually simple: we choose a question family,select values for each of its template parameters, executethe resulting program on the image’s scene graph to find theanswer, and use one of the text templates from the questionfamily to generate the final natural-language question.

However, many combinations of values give rise to ques-tions which are either ill-posed or degenerate. The question“What color is the cube to the right of the sphere?” wouldbe ill-posed if there were many cubes right of the sphere, ordegenerate if there were only one cube in the scene since thereference to the sphere would then be unnecessary. Avoid-ing such ill-posed and degenerate questions is critical to en-sure the correctness and complexity of our questions.

A naıve solution is to randomly sample combinations ofvalues and reject those which lead to ill-posed or degenerate

Unique OverlapSplit Images Questions questions with trainTotal 100,000 999,968 853,554 -Train 70,000 699,989 608,607 -

Val 15,000 149,991 140,448 17,338Test 15,000 149,988 140,352 17,335

0 10 20 30 40Words per question

0%

10%

20%

30%

Que

stio

n fr

actio

n Length distributionDAQUARVQAV7WCLEVR

Exist13%

Count24%

Equal

2%

Less

3%

Greater

3%

Size9%Color

9%

Mat. 9%

Shape 9%

Size4%

Color4%

Mat.4%

Shape4%

Compare Integer

Query

Compare

Figure 3. Top: Statistics for CLEVR; the majority of questionsare unique and few questions from the val and test sets appearin the training set. Bottom left: Comparison of question lengthsfor different VQA datasets; CLEVR questions are generally muchlonger. Bottom right: Distribution of question types in CLEVR.

questions. However, the number of possible configurationsfor a question family is exponential in its number of param-eters, and most of them are undesirable. This makes brute-force search intractable for our complex question families.

Instead, we employ a depth-first search to find valid val-ues for instantiating question families. At each step ofthe search, we use ground-truth scene information to prunelarge swaths of the search space which are guaranteed toproduce undesirable questions; for example we need not en-tertain questions of the form “What color is the <S> to the<R> of the sphere” for scenes that do not contain spheres.

Finally, we use rejection sampling to produce an approx-imately uniform answer distribution for each question fam-ily; this helps minimize question-conditional bias since allquestions from the same family share linguistic structure.

4. VQA Systems on CLEVR4.1. Models

VQA models typically represent images with featuresfrom pretrained CNNs and use word embeddings or re-current networks to represent questions and/or answers.Models may train recurrent networks for answer genera-tion [10, 28, 41], multiclass classifiers over common an-swers [4, 24, 25, 32, 48, 49], or binary classifiers on image-question-answer triples [9, 16, 33]. Many methods incor-porate attention over the image [9, 33, 44, 49, 43] or ques-tion [24]. Some methods incorporate memory [42] or dy-namic network architectures [2, 3].

Experimenting with all methods is logistically challeng-ing, so we reproduced a representative subset of meth-ods: baselines that do not look at the image (Q-type mode,LSTM), a simple baseline (CNN+BoW) that performs nearstate-of-the-art [16, 48], and more sophisticated methods

4

020406080

100

Acc

urac

y

41.8 46.8 48.4 52.3 51.468.592.6

Foo

Overall

020406080

100

50.261.1 59.5 65.2 63.4 71.1

96.6

Foo

Exist

020406080

100

34.6 41.7 38.9 43.7 42.152.2

86.7

Foo

Count

020406080

100

5163

50 57 57 60

Equal

79

52

7354

72 7182

Less

87

5071

4969 68 74

More

91Compare Integer

020406080

100

Acc

urac

y

50 50 56 59 59

87

Size

97

12 1332 32 32

81

Color

95

49 51 58 58 57

88

Material

94

33 3347 48 48

85

Shape

94

Query Attribute

Figure 4. Accuracy per question type of the six VQA methods on the CLEVR dataset (higher is better). Figure best viewed in color.

using recurrent networks (CNN+LSTM), sophisticated fea-ture pooling (CNN+LSTM+MCB), and spatial attention(CNN+LSTM+SA).1 These are described in detail below.

Q-type mode: Similar to the “per Q-type prior” methodin [4], this baseline predicts the most frequent training-setanswer for each question’s type.

LSTM: Similar to “LSTM Q” in [4], the question is pro-cessed with learned word embeddings followed by a word-level LSTM [15]. The final LSTM hidden state is passed toa multi-layer perceptron (MLP) that predicts a distributionover answers. This method uses no image information so itcan only model question-conditional bias.

CNN+BoW: Following [48], the question is encoded byaveraging word vectors for each word in the question andthe image is encoded using features from a convolutionalnetwork (CNN). The question and image features are con-catenated and passed to a MLP which predicts a distributionover answers. We use word vectors trained on the Google-News corpus [29]; these are not fine-tuned during training.

CNN+LSTM: As above, images and questions are en-coded using CNN features and final LSTM hidden states,respectively. These features are concatenated and passed toan MLP that predicts an answer distribution.

CNN+LSTM+MCB: Images and questions are encodedas above, but instead of concatenation, their features arepooled using compact multimodal pooling (MCB) [9, 11].

CNN+LSTM+SA: Again, the question and image areencoded using a CNN and LSTM, respectively. Follow-ing [44], these representations are combined using one ormore rounds of soft spatial attention and the final answerdistribution is predicted with an MLP.

Human: We used Mechanical Turk to collect human re-sponses for 5500 random questions from the test set, takinga majority vote among three workers for each question.

Implementation details. Our CNNs are ResNet-101models pretrained on ImageNet [14] that are not finetuned;images are resized to 224×224 prior to feature extraction.

1We performed initial experiments with dynamic module networks [2]but its parsing heuristics did not generalize to the complex questions inCLEVR so it did not work out-of-the-box; see supplementary material.

CNN+LSTM+SA extracts features from the last layer of theconv4 stage, giving 14× 14× 1024-dimensional features.All other methods extract features from the final averagepooling layer, giving 2048-dimensional features. LSTMsuse one or two layers with 512 or 1024 units per layer.MLPs use ReLU functions and dropout [34]; they have oneor two hidden layers with between 1024 and 8192 units perlayer. All models are trained using Adam [20].

Experimental protocol. CLEVR is split into train, vali-dation, and test sets (see Figure 3). We tuned hyperparam-eters (learning rate, dropout, word vector size, number andsize of LSTM and MLP layers) independently per modelbased on the validation error. All experiments were de-signed on the validation set; after finalizing the design weran each model once on the test set. All experimental find-ings generalized from the validation set to the test set.

4.2. Analysis by Question Type

We can use the program representation of questions toanalyze model performance on different forms of reason-ing. We first evaluate performance on each question type,defined as the outermost function in the program. Figure 4shows results and detailed findings are discussed below.

Querying attributes: Query questions ask about an at-tribute of a particular object (e.g. “What color is the thingright of the red sphere?”). The CLEVR world has twosizes, eight colors, two materials, and three shapes. Onquestions asking about these different attributes, Q-typemode and LSTM obtain accuracies close to 50%, 12.5%,50%, and 33.3% respectively, showing that the datasethas minimal question-conditional bias for these questions.CNN+LSTM+SA substantially outperforms all other mod-els on these questions; its attention mechanism may help itfocus on the target object and identify its attributes.

Comparing attributes: Attribute comparison questionsask whether two objects have the same value for some at-tribute (e.g. “Is the cube the same size as the sphere?”). Theonly valid answers are “yes” and “no”. Q-Type mode andLSTM achieve accuracies close to 50%, confirming there isno dataset bias for these questions. Unlike attribute-query

5

0255075

100

Acc

urac

y

36 36 37 3750 46 49 50 50 50

9378

Query

What color is thecube to the rightof the sphere?

What color is thecube that is the samesize as the sphere?

0255075

100

Acc

urac

y

40 3450 40 42 40 48 44 50 42

55 46

Count

0255075

100

Acc

urac

y

53 4966 58 58 60 65 65 60 64 72 69

Exist

Figure 5. Accuracy on questions with a single spatial relationshipvs. a single same-attribute relationship. For query and count ques-tions, models generally perform worse on questions with same-attribute relationships. Results on exist questions are mixed.

questions, attribute-comparison questions require a limitedform of memory: models must identify the attributes of twoobjects and keep them in memory to compare them. Inter-estingly, none of the models are able to do so: all modelshave an accuracy of approximately 50%. This is also truefor the CNN+LSTM+SA model, suggesting that its atten-tion mechanism is not capable of attending to two objects atonce to compare them. This illustrates how CLEVR can re-veal limitations of models and motivate follow-up research,e.g., augmenting attention models with explicit memory.

Existence: Existence questions ask whether a certaintype of object is present (e.g., “Are there any cubes tothe right of the red thing?”). The 50% accuracy of Q-Type mode shows that both answers are a priori equallylikely, but the LSTM result of 60% does suggest a question-conditional bias. There may be correlations between ques-tion length and answer: questions with more filtering oper-ations (e.g., “large red cube” vs. “red cube”) may be morelikely to have “no” as the answer. Such biases may bepresent even with uniform answer distributions per questionfamily, since questions from the same family may have dif-ferent numbers of filtering functions. CNN+LSTM(+SA)outperforms LSTM, but its performance is still quite low.

Counting: Counting questions ask for the number of ob-jects fulfilling some conditions (e.g. “How many red cubesare there?”); valid answers range from zero to ten. Im-ages have three and ten objects and counting questions re-fer to subsets of objects, so ensuring a uniform answer dis-tribution is very challenging; our rejection sampler there-fore pushes towards a uniform distribution for these ques-tions rather than enforcing it as a hard constraint. Thisresults in a question-conditional bias, reflected in the 35%and 42% accuracies achieved by Q-type mode and LSTM.CNN+LSTM(+MCB) performs on par with LSTM, sug-gesting that CNN features contain little information relevantto counting. CNN+LSTM+SA performs slightly better, butat 52% its absolute performance is low.

Integer comparison: Integer comparison questions askwhich of two object sets is larger (e.g. “Are there fewercubes than red things?”); this requires counting, memory,

0255075

100

Acc

urac

y

36 37 36 37 48 51 47 49 47 48

9274

Query

0255075

100

Acc

urac

y

41 37 49 50 44 41 48 45 45 45 55 48

Count

How many cubes areto the right of thesphere that is to theleft of the red thing?

How many cubesare both to the rightof the sphere andleft of the red thing?

Figure 6. Accuracy on questions with two spatial relationships,broken down by question topology: chain-structured questions vs.tree-structured questions joined with a logical AND operator.

and comparing integer quantities. The answer distributionis unbiased (see Q-Type mode) but a set’s size may corre-late with the length of its description, explaining the gapbetween LSTM and Q-type mode. CNN+BoW performsno better than chance: BoW mixes the words describingeach set, making it impossible for the learner to discrimi-nate between them. CNN+LSTM+SA outperforms LSTMon “less” and “more” questions, but no model outperformsLSTM on “equal” questions. Most models perform betteron “less” than “more” due to asymmetric question families.

4.3. Analysis by Relationship Type

CLEVR questions contain two types of relationships:spatial and same-attribute (see Section 3). We can com-pare the relative difficulty of these two types by comparingmodel performance on questions with a single spatial rela-tionship and questions with a single same-attribute relation-ship; results are shown in Figure 5. On query-attribute andcounting questions we see that same-attribute questions aregenerally more difficult; the gap between CNN+LSTM+SAon spatial and same-relate query questions is particularlylarge (93% vs. 78%). Same-attribute relationships may re-quire a model to keep attributes of one object “in memory”for comparison, suggesting again that models augmentedwith explicit memory may perform better on these ques-tions.

4.4. Analysis by Question Topology

We next evaluate model performance on differentquestion topologies: chain-structured questions vs. tree-structured questions with two branches joined by a logicalAND (see Figure 2). In Figure 6, we compare performanceon chain-structured questions with two spatial relationshipsvs. tree-structured questions with one relationship alongeach branch. On query questions, CNN+LSTM+SA showsa large gap between chain and tree questions (92% vs. 74%);on count questions, CNN+LSTM+SA slightly outperformsLSTM on chain questions (55% vs. 49%) but no methodoutperforms LSTM on tree questions. Tree questions maybe more difficult since they require models to perform twosubtasks in parallel before fusing their results.

6

Question: There is a large objectthat is on the left side of the largeblue cylinder in front of the rubbercylinder on the right side of the pur-ple shiny thing; what is its shape?Effective Question: What shape isa large object left of a cylinder?

4 6 8 10 12 14 16 18 20Actual question size

20

40

60

80

100

Acc

urac

y

Query Attribute

2 4 6 8 10 12 14 16Effective question size

20

40

60

80

100

Acc

urac

y

Query Attribute

Figure 7. Top: Many questions can be answered correctly with-out correctly solving all subtasks. For a given question and scenewe can prune functions from the question’s program to generatean effective question which is shorter but gives the same answer.Bottom: Accuracy on query questions vs. actual and effectivequestion size. Accuracy decreases with effective question size butnot with actual size. Shaded area shows a 95% confidence interval.

4.5. Effect of Question Size

Intuitively, longer questions should be harder since theyinvolve more reasoning steps. We define a question’s size tobe the number of functions in its program, and in Figure 7(bottom left) we show accuracy on query-attribute questionsas a function of question size.2 Surprisingly accuracy ap-pears unrelated to question size.

However, many questions can be correctly answeredeven when some subtasks are not solved correctly. For ex-ample, the question in Figure 7 (top) can be answered cor-rectly without identifying the correct large blue cylinder, be-cause all large objects left of a cylinder are cylinders.

To quantify this effect, we define the effective question ofan image-question pair: we prune functions from the ques-tion’s program to find the smallest program that, when ex-ecuted on the scene graph for the question’s image, givesthe same answer as the original question.3 A question’s ef-fective size is the size of its effective question. Questionswhose effective size is smaller than their actual size neednot be degenerate. The question in Figure 7 is not degener-ate because the entire question is needed to resolve its ob-ject references (there are two blue cylinders and two rubbercylinders), but it has a small effective size since it can becorrectly answered without resolving those references.

In Figure 7 (bottom), we show accuracy on query ques-tions as a function of effective question size. The error rateof all models increases with effective question size, suggest-ing that models struggle with long reasoning chains.

2We exclude questions with same-attribute relations since their maxsize is 10, introducing unwanted correlations between size and difficulty.Excluded questions show the same trends (see supplementary material).

3Pruned questions may be ill-posed (Section 3) so they are executedwith modified semantics; see supplementary material for details.

Question: There is a purple cubethat is in front of the yellow metalsphere; what material is it?Absolute question: There is apurple cube in the front half ofthe image; what material is it?

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

Query

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

Count

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

Exist

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

0 1 2 3Number of "relate"

25

50

75

100

Acc

urac

y

Figure 8. Top: Some questions can be correctly answered usingabsolute definitions for spatial relationships; for example in thisimage there is only one purple cube in the bottom half of the im-age. Bottom: Accuracy of each model on chain-structured ques-tions as a function of the number of spatial relationships in thequestion, separated by question type. Top row shows all chain-structured questions; bottom row excludes questions that can becorrectly answered using absolute spatial reasoning.

4.6. Spatial Reasoning

We expect that questions with more spatial relationshipsshould be more challenging since they require longer chainsof reasoning. The top set of plots in Figure 8 shows ac-curacy on chain-structured questions with different num-bers of relationships.4 Across all three question types,CNN+LSTM+SA shows a significant drop in accuracy forquestions with one or more spatial relationship; other mod-els are largely unaffected by spatial relationships.

Spatial relationships force models to reason about ob-jects’ relative positions. However, as shown in Figure 8,some questions can be answered using absolute spatial rea-soning. In this question the purple cube can be found bysimply looking in the bottom half of the image; reasoningabout its position relative to the metal sphere is unnecessary.

Questions only requiring absolute spatial reasoning canbe identified by modifying the semantics of spatial relation-ship functions in their programs: instead of returning setsof objects related to the input object, they ignore their in-put object and return the set of objects in the half of theimage corresponding to the relationship. A question onlyrequires absolute spatial reasoning if executing its programwith these modified semantics does not change its answer.

The bottommost plots of Figure 8 show accuracy onchain-structured questions with different number of re-lationships, excluding questions that can be answered

4We restrict to chain-structured questions to avoid unwanted correla-tions between question topology and number of relationships.

7

0255075

100

13.0 12.4 20.27.4

LSTM

32.1 34.545.9

17.1

CNN+LSTM

28.1 31.5 32.722.2

CNN+LSTM+MCB

83.8 83.5 85.2

51.1

CNN+LSTM+SA

Query color

0255075

100

49.5 50.7 50.7 49.8

LSTM

64.7 64.5 61.1 61.2

CNN+LSTM

63.9 65.0 59.5 60.0

CNN+LSTM+MCB

91.7 90.4 90.080.8

CNN+LSTM+SA

Query material

A → A (sphere)A → B (sphere)

A → A (cube / cylinder)

A → B (cube / cylinder)

Figure 9. In Condition A all cubes are gray, blue, brown, or yel-low and all cylinders are red, green, purple, or cyan; in ConditionB color palettes are swapped. We train models in Condition A andtest in both conditions to assess their generalization performance.We show accuracy on “query color” and “query material” ques-tions, separating questions by shape of the object being queried.

with absolute spatial reasoning. On query questions,CNN+LSTM+SA performs significantly worse when abso-lute spatial reasoning is excluded; on count questions nomodel outperforms LSTM, and on exist questions no modeloutperforms Q-type mode. These results suggest that mod-els have not learned the semantics of spatial relationships.

4.7. Compositional Generalization

Practical VQA systems should perform well on imagesand questions that contain novel combinations of attributesnot seen during training. To do so models might need tolearn disentangled representations for attributes, for exam-ple learning separate representations for color and shape in-stead of memorizing all possible color/shape combinations.

We can use CLEVR to test the ability of VQA models toperform such compositional generalization. We synthesizetwo new versions of CLEVR: in Condition A all cubes aregray, blue, brown, or yellow and all cylinders are red, green,purple, or cyan; in Condition B these shapes swap colorpalettes. Both conditions contain spheres of all eight colors.

We retrain models on Condition A and compare theirperformance when testing on Condition A (A → A) andtesting on Condition B (A→ B). In Figure 9 we show accu-racy on query-color and query-material questions, separat-ing questions asking about spheres (which are the same inA and B) and cubes/cylinders (which change from A to B).

Between A→A and A→B, all models perform about thesame when asked about the color of spheres, but performmuch worse when asked about the color of cubes or cylin-ders; CNN+LSTM+SA drops from 85% to 51%. Modelsseem to learn strong biases about the colors of objects andcannot overcome these biases when conditions change.

When asked about the material of cubes and cylinders,CNN+LSTM+SA shows a smaller gap between A→A and

A→B (90% vs 81%); other models show no gap. Hav-ing seen metal cubes and red metal objects during training,models can understand the material of red metal cubes.

5. Discussion and Future WorkThis paper has introduced CLEVR, a dataset designed

to aid in diagnostic evaluation of visual question answering(VQA) systems by minimizing dataset bias and providingrich ground-truth representations for both images and ques-tions. Our experiments demonstrate that CLEVR facilitatesin-depth analysis not possible with other VQA datasets: ourquestion representations allow us to slice the dataset alongdifferent axes (question type, relationship type, questiontopology, etc.), and comparing performance along these dif-ferent axes allows us to better understand the reasoning ca-pabilities of VQA systems. Our analysis has revealed sev-eral key shortcomings of current VQA systems:

• Short-term memory: All systems we tested per-formed poorly in situations requiring short-term mem-ory, including attribute comparison and integer equal-ity questions (Section 4.2), same-attribute relation-ships (Section 4.3), and tree-structured questions (Sec-tion 4.4). Attribute comparison questions are of par-ticular interest, since models can successfully identityattributes of objects but struggle to compare attributes.• Long reasoning chains: Systems struggle to answer

questions requiring long chains of nontrivial reason-ing, including questions with large effective sizes (Sec-tion 4.5) and count and existence questions with manyspatial relationships (Section 4.6).• Spatial Relationships: Models fail to learn the true

semantics of spatial relationships, instead relying onabsolute image position (Section 4.6).• Disentangled Representations: By training and test-

ing models on different data distributions (Section 4.7)we argue that models do not learn representations thatproperly disentangle object attributes; they seem tolearn strong biases from the training data and cannotovercome these biases when conditions change.

Our study also shows cases where current VQA systemsare successful. In particular, spatial attention [44] allowsmodels to focus on objects and identify their attributes evenon questions requiring multiple steps of reasoning.

These observations present clear avenues for future workon VQA. We plan to use CLEVR to study models with ex-plicit short-term memory, facilitating comparisons betweenvalues [13, 18, 39, 42]; explore approaches that encouragelearning disentangled representations [5]; and investigatemethods that compile custom network architectures for dif-ferent patterns of reasoning [2, 3]. We hope that diagnos-tic datasets like CLEVR will help guide future research inVQA and enable rapid progress on this important task.

8

Supplementary Material

A. Basic FunctionsAs described in Section 3 and shown in Figure 2, each

question in CLEVR is associated with a functional programbuilt from a set of basic functions. In this section we detailthe semantics of these basic functional building blocks.

Data Types. Our basic functional building blocks operateon values of the following types:

• Object: A single object in the scene.• ObjectSet: A set of zero or more objects in the

scene.• Integer: An integer between 0 and 10 (inclusive).• Boolean: Either yes or no.• Value types:

– Size: One of large or small.– Color: One of gray, red, blue, green,brown, purple, cyan, or yellow.

– Shape: One of cube, sphere, or cylinder.– Material: One of rubber or metal.

• Relation: One of left, right, in front, orbehind.

Basic Functions. The functional program representationsof questions are built from the following set of basic build-ing blocks. Each of these functions takes the image’s scenegraph as an additional implicit input.

• scene (∅ → ObjectSet)Returns the set of all objects in the scene.• unique (ObjectSet→ Object)

If the input is a singleton set, then return it as a stan-dalone Object; otherwise raise an exception and flagthe question as ill-posed (See Section 3).• relate (Object× Relation→ ObjectSet)

Return all objects in the scene that have the speci-fied spatial relation to the input object. For exampleif the input object is a red cube and the input relationis left, then return the set of all objects in the scenethat are left of the red cube.• count (ObjectSet→ Integer)

Returns the size of the input set.• exist (ObjectSet→ Boolean)

Returns yes if the input set is nonempty and no if itis empty.

• Filtering functions: These functions filter the inputobjects by some attribute, returning the subset of in-put objects that match the input attribute. For examplecalling filter sizewith the first input smallwillreturn the set of all small objects in the second input.

– filter size (ObjectSet× Size→ ObjectSet)

– filter color (ObjectSet× Color→ ObjectSet)

– filter material (ObjectSet×Material→ ObjectSet)

– filter shape (ObjectSet× Shape→ ObjectSet)

• Query functions: These functions return the speci-fied attribute of the input object; for example callingquery color on a red object returns red.

– query size (Object→ Size)

– query color (Object→ Color)

– query material (Object→ Material)

– query shape (Object→ Shape)

• Logical operators:

– AND (ObjectSet× ObjectSet→ ObjectSet)Returns the intersection of the two input sets.

– OR (ObjectSet× ObjectSet→ ObjectSet)Returns the union of the two input sets.

• Same-attribute relations: These functions return theset of objects that have the same attribute value as theinput object, not including the input object. For exam-ple calling same shape on a cube returns the set ofall cubes in the scene, excluding the query cube.

– same size (Object→ ObjectSet)

– same color (Object→ ObjectSet)

– same material (Object→ ObjectSet)

– same shape (Object→ ObjectSet)

• Integer comparison: Checks whether the two inte-ger inputs are equal, or whether the first is less thanor greater than the second, returning either yes or no.

– equal integer (Integer× Integer→ Boolean)

– less than (Integer× Integer→ Boolean)

– greater than (Integer× Integer→ Boolean)

• Attribute comparison: These functions return yes iftheir inputs are equal and no if they are not equal.

– equal size (Size× Size→ Boolean)

– equal material (Material× Material→ Boolean)

– equal color (Color× Color→ Boolean)

– equal shape (Shape× Shape→ Boolean)

B. Effective Question SizeIn Section 4.5 we note that some questions can be cor-

rectly answered without correctly resolving all intermediateobject references, and define a question’s effective questionto quantitatively measure this effect.

For any question we can compute its effective questionby pruning functions from the question’s program; the ef-fective question is the smallest such pruned program that,when executed on the scene graph for the question’s image,gives the same answer as the original question.

9

Figure 10. For the above image and the question “What color is thecube behind the cylinder?”, the effective question is “What coloris the cube?” (see text for details).

Some pruned questions may be ill-posed, meaning thatsome object references do not refer to a unique object. Forexample, consider the question “What color is the cube be-hind the cylinder?”; its associated program is

query color(unique(filter shape(cube,

relate(behind, unique(filter shape(cylinder, scene()))))))

Imagine executing this program on the scene shown inFigure 10. The innermost filter shape gives a set containingthe cylinder, the relate returns a set containing just the largecube in the back, the outer filter shape does nothing, andthe query color returns brown.

This question is not ill-posed (Section 3) because the ref-erence to “the cube” cannot be resolved without the rest ofthe question; however this question’s effective size is lessthan its actual size because the question can be correctlyanswered without resolving this object reference correctly.

To compute the effective question, we attempt to prunefunctions from this program. Starting from the innermostfunction and working out, whenever we find a functionwhose input type is Object or ObjectSet, we constructa pruned question by replacing that function’s input with ascene function and executing it. The smallest such prunedprogram that gives the same answer as the original programis the effective question.

Pruned questions may be ill-posed, so we execute themwith modified semantics. The output type of the uniquefunction is changed from Object to ObjectSet, andit simply returns its input set. All functions taking anObject input are modified to take an ObjectSet inputinstead by mapping the original function over its input setand flattening the resulting set; thus the relate functions re-turn the set of objects in the scene that have the specifiedrelationship with any of the input objects, and the queryfunctions return sets of values rather than single values.

Therefore for this example question we consider the fol-

6 8 10Actual question size

20

40

60

80

100

Acc

urac

y

Query Attribute

2 4 6 8 10Effective question size

20

40

60

80

100

Acc

urac

y

Query Attribute

Figure 11. Accuracy on query questions vs. actual and effectivequestion size, restricting to questions with a same-attribute rela-tionship. Figure 7 shows the same plots for questions without asame-attribute relationship. For both groups of questions we seethat accuracy decreases as effective question size increases.

lowing sequence of pruned programs. First we prune theinner filter shape function:

query color(unique(filter shape(cube,

relate(behind, unique(scene())))))

The relate function now returns the set of objects whichare behind some object, so it returns the large cube and thecylinder (since it is behind the small cube). The filter shapefunction removes the cylinder, and the query color returnsa singleton set containing brown.

Next we prune the inner unique function:

query color(unique(filter shape(cube,

relate(behind, scene()))))

Since unique computes the identity for pruned questions,execution is the same as above.

Next we prune the relate function:

query color(unique(filter shape(cube, scene())))

Now the filter shape returns the set of both cubes, but sinceboth are brown the query color still returns a singleton setcontaining brown.

Next we prune the filter shape function:

query color(unique(scene()))

Now the query color receives the set of all three input ob-jects, so it returns a set containing brown and gray, whichis different from the original question.

The effective question is therefore:

query color(unique(filter shape(cube, scene())))

B.1. Accuracy vs Question Size

Figure 7 of the main paper shows model accuracy onquery-attribute questions as a function of actual and effec-tive question size, excluding questions with same-attributerelationships. Questions with same-attribute relationshipshave a maximum question size of 10 but questions with-out same-attribute relationships have a maximum size of 20;combining these questions thus leads to unwanted correla-

10

tions between question size and difficulty.In Figure 11 we show model accuracy vs. actual and ef-

fective question size for questions with same-attribute rela-tionships. Similar to Figure 7, we see that model accuracyeither remains constant or increases as actual question sizeincreases, but all models show a clear decrease in accuracyas effective question size increases.

C. Dynamic Module NetworksModule networks [2, 3] are a novel approach to visual

question answering where a set of differentiable modulesare used to assemble a custom network architecture to an-swer each question. Each module is responsible for per-forming a specific function such as finding a particular typeof object, describing the current object of attention, or per-forming a logical and operation to merge attention masks.This approach seems like a natural fit for the rich, compo-sitional questions in CLEVR; unfortunately we found thatparsing heuristics tuned for the VQA dataset did not gener-alize to the longer, more complex questions in CLEVR.

Dynamic module networks [2] generate network archi-tectures by performing a dependency parse of the question,using a set of heuristics to compute a set of layout frag-ments, combining these fragments to create candidate lay-outs, and ranking the candidate layouts using an MLP.

For some questions, the heuristics are unable to produceany layout fragments; in this case, the system uses a simpledefault network architecture as a fallback for answering thatquestion. On a random sample of 10,000 questions from theVQA dataset [4], we found that dynamic module networksresorted to default architecture for 7.8% of questions; ona random sample of 10,000 questions from CLEVR, thedefault network architecture was used for 28.9% of ques-tions. This suggests that the same parsing heuristics usedfor VQA do not apply to the questions in CLEVR; thereforethe method of [2] did not work out-of-the box on CLEVR.

D. Example images and questionsThe remaining pages show randomly selected images

and questions from CLEVR. Each question is annotatedwith its answer, question type, and size. Recall from Sec-tion 3 that a question’s type is the outermost function in thequestion’s functional program, and a question’s size is thenumber of functions in its program.

11

Q: There is a rubbercube in front of the bigcylinder in front of thebig brown matte thing;what is its size?A: smallQ-type: query sizeSize: 14

Q: What color is theobject that is on theleft side of the smallrubber thing?A: grayQ-type: query colorSize: 7

Q: There is anothercube that is made ofthe same material asthe small brown block;what is its size?A: largeQ-type: query sizeSize: 9

Q: What number ofother matte objects arethe same shape as thesmall rubber object?A: 1Q-type: countSize: 7

Q: Are there fewermetallic objects thatare on the left side ofthe large cube thancylinders to the left ofthe cyan shiny block?A: yesQ-type: less thanSize: 16

Q: There is a greenrubber thing that is leftof the rubber thing thatis right of the rubbercylinder behind thegray shiny block; whatis its size?A: largeQ-type: query sizeSize: 17

Q: What is the size ofthe matte thing that ison the left side of thelarge cyan object andin front of the smallblue thing?A: largeQ-type: query sizeSize: 14

Q: The green mattething that is behind thelarge green thing thatis to the right of thebig cyan metal objectis what shape?A: cylinderQ-type: query shapeSize: 14

Q: Is the number oftiny metal objects leftof the purple metalliccylinder the same asthe number of smallmetal objects to theleft of the small metalsphere?A: noQ-type: equal integerSize: 19

Q: Do the large shinyobject and the thing tothe left of the big grayobject have the sameshape?A: noQ-type: equal shapeSize: 13

Q: There is a smallthing that is the samecolor as the tiny shinyball; what is it madeof?A: rubberQ-type:query materialSize: 9

Q: Is there anythingelse that is the sameshape as the tinyyellow matte thing?A: noQ-type: existSize: 7

Q: What color is thesmall shiny cube?A: brownQ-type: query colorSize: 6

Q: There is a tinyshiny sphere left of thecylinder in front of thelarge cylinder; whatcolor is it?A: brownQ-type: query colorSize: 13

Q: Is there a tinything that has the samematerial as the browncylinder?A: yesQ-type: existSize: 7

Q: What is thematerial of the tinyobject to the right ofthe brown shiny ballbehind the tiny shinycylinder?A: metalQ-type:query materialSize: 14

Q: What number ofcylinders are purplemetal objects or purplematte things?A: 0Q-type: countSize: 9

Q: Does the largepurple shiny objecthave the same shapeas the tiny object thatis behind the mattething?A: noQ-type: equal shapeSize: 14

Q: There is a objectthat is both on theleft side of the brownmetal block and infront of the largepurple shiny ball; howbig is it?A: smallQ-type: query sizeSize: 16

Q: There is a bigpurple shiny object;what shape is it?A: sphereQ-type: query shapeSize: 6

Q: There is a largebrown block in frontof the tiny rubbercylinder that is behindthe cyan block; arethere any big cyanmetallic cubes that areto the left of it?A: yesQ-type: existSize: 20

Q: There is a bigshiny object; are thereany blue shiny cubesbehind it?A: noQ-type: existSize: 9

Q: What number ofcylinders have thesame color as themetal ball?A: 0Q-type: countSize: 7

Q: The cyan block thatis the same material asthe tiny gray thing iswhat size?A: largeQ-type: query sizeSize: 9

12

Q: How big is the grayrubber object that isbehind the big shinything behind the bigmetallic thing that ison the left side of thepurple ball?A: smallQ-type: query sizeSize: 17

Q: There is a purpleball that is the samesize as the red cylin-der; what material isit?A: metalQ-type:query materialSize: 9

Q: Is there anothergreen rubber cube thathas the same size asthe green matte cube?A: noQ-type: existSize: 10

Q: Is the large mattething the same shapeas the big red object?A: yesQ-type: equal shapeSize: 11

Q: There is a tinyrubber thing that isthe same color as themetal cylinder; whatshape is it?A: cylinderQ-type: query shapeSize: 9

Q: What is the shapeof the tiny green thingthat is made of thesame material as thelarge cylinder?A: cylinderQ-type: query shapeSize: 9

Q: Do the bluemetallic object and thegreen metal thing havethe same shape?A: noQ-type: equal shapeSize: 11

Q: The big matte thingis what color?A: purpleQ-type: query colorSize: 5

Q: There is a smallball that is made ofthe same material asthe large block; whatcolor is it?A: blueQ-type: query colorSize: 9

Q: Is the size of thered rubber sphere thesame as the purplemetal thing?A: yesQ-type: equal sizeSize: 12

Q: What is the ma-terial of the purplecylinder?A: metalQ-type:query materialSize: 5

Q: There is a blue ballthat is the same size asthe brown thing; whatmaterial is it?A: metalQ-type:query materialSize: 8

Q: How many smallspheres are the samecolor as the big rubbercube?A: 0Q-type: countSize: 9

Q: Is the tiny ballmade of the samematerial as the largepurple ball?A: yesQ-type:equal materialSize: 12

Q: There is a largecube that is rightof the red sphere;what number of largeyellow things are onthe right side of it?A: 0Q-type: countSize: 12

Q: Do the mattecylinder and thepurple shiny objecthave the same size?A: noQ-type: equal sizeSize: 11

Q: How many smallgreen things arebehind the greenrubber sphere in frontof the blue thing thatis in front of the largepurple metal cylinder?A: 1Q-type: countSize: 18

Q: The cylinder thatis the same size as theblue metallic sphere iswhat color?A: purpleQ-type: query colorSize: 9

Q: What is the size ofthe green metal objectright of the large ballthat is on the left sideof the big blue metalsphere?A: smallQ-type: query sizeSize: 15

Q: There is a metallicball that is the samecolor as the tinycylinder; what size isit?A: largeQ-type: query sizeSize: 9

Q: What is the colorof the matte thing thatis left of the thingbehind the tiny greenthing behind the tinyshiny sphere?A: greenQ-type: query colorSize: 15

Q: There is a thing infront of the tiny block;is its color the sameas the matte object infront of the tiny brownrubber thing?A: yesQ-type: equal colorSize: 17

Q: The green thingbehind the small greenthing on the right sideof the brown matteobject is what shape?A: cubeQ-type: query shapeSize: 12

Q: Is there a greenmatte cube that has thesame size as the shinything?A: yesQ-type: existSize: 8

13

Q: Are there morebrown shiny objectsbehind the largerubber cylinder thangray blocks?A: yesQ-type: greater thanSize: 14

Q: What color isthe matte object tothe right of the largeblock?A: blueQ-type: query colorSize: 8

Q: Do the blue cubeand the cylinder havethe same size?A: yesQ-type: equal sizeSize: 10

Q: The ball has whatsize?A: smallQ-type: query sizeSize: 4

Q: Are there anyother things that arethe same shape as thelarge green thing?A: noQ-type: existSize: 6

Q: The matte thingthat is both in frontof the purple cubeand to the left of theblue rubber cylinder iswhat color?A: greenQ-type: query colorSize: 15

Q: Are there fewersmall rubber cylindersin front of the greenball than purplecylinders?A: noQ-type: less thanSize: 14

Q: What number ofother objects are thereof the same shapeas the large rubberobject?A: 0Q-type: countSize: 6

Q: What number ofcubes are the samecolor as the rubberball?A: 1Q-type: countSize: 7

Q: Is the big cyanmetal thing the sameshape as the brownthing?A: noQ-type: equal shapeSize: 11

Q: What shape isthe tiny metal thingbehind the large blockin front of the tinybrown block?A: cubeQ-type: query shapeSize: 14

Q: Is the size of thecyan cube the sameas the metal cylinderthat is behind the cyancylinder?A: noQ-type: equal sizeSize: 15

Q: There is a tinybrown rubber thing;is its shape the sameas the thing that isin front of the smallmatte cylinder?A: noQ-type: equal shapeSize: 15

Q: What number ofyellow rubber thingsare the same size asthe gray thing?A: 0Q-type: countSize: 7

Q: What number oftiny brown rubberobjects are behind therubber object that ison the right side of thecylinder on the rightside of the big cyancylinder?A: 0Q-type: countSize: 16

Q: Are there an equalnumber of brownmatte objects that areright of the tiny brownobject and gray rubbercylinders that are tothe left of the largecyan shiny cylinder?A: yesQ-type: equal integerSize: 20

Q: Are there anyrubber things thathave the same size asthe yellow metalliccylinder?A: noQ-type: existSize: 8

Q: What is the size ofthe brown sphere?A: smallQ-type: query sizeSize: 5

Q: What number ofyellow metal objectsare the same size asthe metallic cube?A: 1Q-type: countSize: 8

Q: Is the number ofpurple matte cylindersbehind the large purplething less than thenumber of tiny rubberobjects in front of thebig cylinder?A: yesQ-type: less thanSize: 18

Q: What is the shapeof the shiny thing thatis behind the smallblue rubber object andto the right of the tinybrown thing?A: cubeQ-type: query shapeSize: 15

Q: What is the shapeof the blue rubberobject in front ofthe brown object tothe right of the bigmetallic block?A: cylinderQ-type: query shapeSize: 13

Q: Does the large redobject have the sameshape as the large bluething?A: noQ-type: equal shapeSize: 11

Q: There is a largecylinder that is thesame color as the tinycylinder; what is itmade of?A: rubberQ-type:query materialSize: 9

14

Q: There is a redshiny thing right of thepurple metal sphere;what is its shape?A: cylinderQ-type: query shapeSize: 10

Q: What number ofpurple shiny things arethere?A: 1Q-type: countSize: 4

Q: Are the red objectand the cyan cubemade of the samematerial?A: yesQ-type:equal materialSize: 10

Q: Are there moremetallic objects thatare right of the largered shiny cylinder thangray matte objects?A: noQ-type: greater thanSize: 14

Q: Does the small ballhave the same coloras the small cylinderin front of the bigsphere?A: noQ-type: equal colorSize: 15

Q: How many otherthings are the samesize as the greensphere?A: 0Q-type: countSize: 6

Q: How many blocksare yellow rubberthings or large greenshiny things?A: 0Q-type: countSize: 10

Q: There is a shinyobject in front of theyellow cylinder; is itthe same shape as theyellow thing?A: noQ-type: equal shapeSize: 13

Q: There is a objectthat is behind the biggreen metal cylinder;what is its material?A: rubberQ-type:query materialSize: 9

Q: How big is the graything?A: largeQ-type: query sizeSize: 4

Q: Do the greenobject behind the tinygreen matte cylinderand the small purpleobject have the samematerial?A: yesQ-type:equal materialSize: 16

Q: What number ofrubber blocks arethere?A: 1Q-type: countSize: 4

Q: How big is theyellow thing behindthe brown shiny thing?A: smallQ-type: query sizeSize: 8

Q: Is the size of thebrown metal ball thesame as the yellowthing that is behind theyellow rubber sphere?A: noQ-type: equal sizeSize: 16

Q: Are there fewersmall yellow thingsto the left of the largeyellow matte ball thanlarge brown objects?A: yesQ-type: less thanSize: 15

Q: What material isthe other thing that isthe same shape as thebrown thing?A: rubberQ-type:query materialSize: 6

Q: How many thingsare either green thingsor matte cubes behindthe green ball?A: 2Q-type: countSize: 11

Q: There is a cyanobject that is to theright of the cyanrubber sphere; is itssize the same as thegray rubber cylinder?A: noQ-type: equal sizeSize: 16

Q: There is a sphereto the right of thelarge yellow ball; whatmaterial is it?A: metalQ-type:query materialSize: 9

Q: Are there thesame number of tinycylinders that arebehind the cyan metalobject and purpleblocks right of thegray cube?A: noQ-type: equal integerSize: 17

Q: What color is therubber ball in frontof the metal cube tothe left of the mattecube left of the bluemetallic sphere?A: grayQ-type: query colorSize: 18

Q: What shape is thecyan shiny thing thatis the same size as thered matte object?A: cubeQ-type: query shapeSize: 9

Q: There is a greencylinder on the rightside of the big grayball; does it have thesame size as the ballthat is behind themetal ball?A: noQ-type: equal sizeSize: 19

Q: What is the size ofthe green block behindthe big gray mattesphere?A: largeQ-type: query sizeSize: 11

15

Acknowledgments We thank Deepak Pathak, PiotrDollar, Ranjay Krishna, Animesh Garg, and Danfei Xu forhelpful comments and discussion.

References[1] A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior

of visual question answering models. In EMNLP, 2016. 1[2] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learn-

ing to compose neural networks for question answering. InNAACL, 2016. 1, 2, 4, 5, 8, 11

[3] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neuralmodule networks. In CVPR, 2016. 1, 2, 4, 8, 11

[4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Zit-nick, and D. Parikh. VQA: Visual question answering. InICCV, 2015. 1, 2, 4, 5, 11

[5] Y. Bengio, A. Courville, and P. Vincent. Representa-tion learning: A review and new perspectives. TPAMI,35(8):1798–1828, 2014. 8

[6] Blender Online Community. Blender - a 3D modelling andrendering package. Blender Foundation, Blender Institute,Amsterdam, 2016. 3

[7] J. Chen, P. Kuznetsova, D. Warren, and Y. Choi. Deja image-captions: A corpus of expressive image descriptions in repe-tition. In NAACL, 2015. 2

[8] A. Farhadi, M. Hejrati, A. Sadeghi, P. Young, C. Rashtchian,J. Hockenmaier, and D. Forsyth. Every picture tells a story:Generating sentences for images. In ECCV, 2010. 2

[9] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell,and M. Rohrbach. Multimodal compact bilinear poolingfor visual question answering and visual grounding. InarXiv:1606.01847, 2016. 1, 4, 5

[10] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu.Are you talking to a machine? Dataset and methods for mul-tilingual image question answering. In NIPS, 2015. 1, 2,4

[11] Y. Gao, O. Beijbom, N. Zhang, and T. Darrell. Compactbilinear pooling. In CVPR, 2016. 5

[12] D. Geman, S. Geman, N. Hallonquist, and L. Younes. VisualTuring test for computer vision systems. Proceedings of theNational Academy of Sciences, 112(12):3618–3623, 2015. 2

[13] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka,A. Grabska-Barwinska, S. Colmenarejo, E. Grefenstette,T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols,G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blun-som, K. . Kavukcuoglu, and D. Hassabis. Hybrid computingusing a neural network with dynamic external memory. Na-ture, 2016. 8

[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In CVPR, 2016. 5

[15] S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997. 2, 5

[16] A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visualquestion answering baselines. In ECCV, 2016. 1, 4

[17] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma,M. S. Bernstein, and L. Fei-Fei. Image retrieval using scenegraphs. In CVPR, 2015. 3

[18] A. Joulin and T. Mikolov. Inferring algorithmic patterns withstack-augmented recurrent nets. In NIPS, 2015. 8

[19] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg.Referitgame: Referring to objects in photographs of naturalscenes. In EMNLP, 2014. 2

[20] D. Kingma and J. Ba. Adam: A method for stochastic opti-mization. In ICLR, 2015. 5

[21] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz,S. Chen, Y. Kalantidis, L. Jia-Li, D. Shamma, M. Bernstein,and L. Fei-Fei. Visual genome: Connecting language andvision using crowdsourced dense image annotations. IJCV,2016. 1, 2, 3

[22] H. J. Levesque, E. Davis, and L. Morgenstern. The Wino-grad schema challenge. In AAAI Spring Symposium: Logi-cal Formalizations of Commonsense Reasoning, volume 46,page 47, 2011. 2

[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra-manan, P. Dollar, and C. Zitnick. Microsoft COCO: Com-mon objects in context. In ECCV, 2014. 2

[24] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchicalquestion-image co-attention for visual question answering.In NIPS, 2016. 1, 2, 4

[25] L. Ma, Z. Lu, and H. Li. Learning to answer questions fromimage using convolutional neural network. In AAAI, 2016. 4

[26] M. Malinowski and M. Fritz. A multi-world approach toquestion answering about real-world scenes based on uncer-tain input. In NIPS, 2014. 1, 2

[27] M. Malinowski and M. Fritz. Towards a visual Turing chal-lenge. In NIPS 2014 Workshop on Learning Semantics, 2014.2

[28] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neu-rons: A neural-based approach to answering questions aboutimages. In ICCV, 2015. 2, 4

[29] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficientestimation of word representations in vector space. In arXiv1301.3781, 2013. 5

[30] O. Pfungst. Clever Hans (The horse of Mr. von Osten): Acontribution to experimental animal and human psychology.Henry Holt, New York, 1911. 1

[31] A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh.Question relevance in vqa: Identifying non-visual and false-premise questions. In EMNLP, 2016. 1

[32] M. Ren, R. Kiros, and R. Zemel. Exploring models and datafor image question answering. In NIPS, 2015. 1, 2, 4

[33] K. Shih, S. Singh, and D. Hoiem. Where to look: Focusregions for visual question answering. In CVPR, 2016. 4

[34] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, andR. Salakhutdinov. Dropout: a simple way to prevent neuralnetworks from overfitting. JMLR, 15(1):1929–1958, 2014. 5

[35] B. Sturm. A simple method to determine if a music infor-mation retrieval system is a horse. IEEE Transactions onMultimedia, 16(6):1636–1644, 2014. 1

[36] B. Sturm. Horse taxonomy and taxidermy. HORSE2016,2016. 1

[37] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urta-sun, and S. Fidler. Movieqa: Understanding stories in moviesthrough question-answering. In CVPR, 2016. 2

16

[38] J. Weston, A. Bordes, S. Chopra, A. Rush, B. vanMerrienboer, A. Joulin, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks.In ICLR, 2016. 2

[39] J. Weston, S. Chopra, and A. Bordes. Memory networks. InICLR, 2015. 8

[40] T. Winograd. Understanding Natural Language. AcademicPress, 1972. 2

[41] Q. Wu, C. Shen, A. van den Hengel, P. Wang, and A. Dick.Image captioning and visual question answering based onattributes and their related external knowledge. In arXiv1603.02814, 2016. 4

[42] C. Xiong, S. Merity, and R. Socher. Dynamic memory net-works for visual and textual question answering. ICML,2016. 4, 8

[43] H. Xu and K. Saenko. Ask, attend, and answer: Exploringquestion-guided spatial attention for visual question answer-ing. In ECCV, 2016. 4

[44] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stackedattention networks for image question answering. In CVPR,2016. 1, 2, 4, 5, 8

[45] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From im-age descriptions to visual denotations: New similarity met-rics for semantic inference over event descriptions. In TACL,pages 67–78, 2014. 2

[46] L. Yu, E. Park, A. Berg, and T. Berg. Visual madlibs: Fillin the blank image generation and question answering. InICCV, 2015. 1, 2

[47] P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, andD. Parikh. Yin and yang: Balancing and answering binaryvisual questions. In CVPR, 2016. 1, 2

[48] B. Zhou, Y. Tian, S. Sukhbataar, A. Szlam, and R. Fer-gus. Simple baseline for visual question answering. InarXiv:1512.02167, 2015. 1, 4, 5

[49] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w:Grounded question answering in images. In CVPR, 2016. 1,2, 4

[50] C. Zitnick and D. Parikh. Bringing semantics into focus us-ing visual abstraction. In CVPR, 2013. 2

17