Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4134–4148November 7–11, 2021. c©2021 Association for Computational Linguistics

4134

Improving Unsupervised Question Answering viaSummarization-Informed Question Generation

Chenyang Lyu† Lifeng Shang‡ Yvette Graham¶ Jennifer Foster†Xin Jiang‡ Qun Liu‡

† School of Computing, Dublin City University, Dublin, Ireland‡ Huawei Noah’s Ark Lab, Hong Kong, China

¶ School of Computer Science and Statistics, Trinity College Dublin, Dublin, Irelandchenyang.lyu2@mail.dcu.ie, ygraham@tcd.ie, jennifer.foster@dcu.ie

{shang.lifeng, jiang.xin, qun.liu}@huawei.com

Abstract

Question Generation (QG) is the task of gen-erating a plausible question for a given <pas-sage, answer> pair. Template-based QGuses linguistically-informed heuristics to trans-form declarative sentences into interrogatives,whereas supervised QG uses existing QuestionAnswering (QA) datasets to train a system togenerate a question given a passage and an an-swer. A disadvantage of the heuristic approachis that the generated questions are heavily tiedto their declarative counterparts. A disadvan-tage of the supervised approach is that theyare heavily tied to the domain/language of theQA dataset used as training data. In orderto overcome these shortcomings, we proposean unsupervised QG method which uses ques-tions generated heuristically from summariesas a source of training data for a QG sys-tem. We make use of freely available newssummary data, transforming declarative sum-mary sentences into appropriate questions us-ing heuristics informed by dependency pars-ing, named entity recognition and semanticrole labeling. The resulting questions are thencombined with the original news articles totrain an end-to-end neural QG model. We ex-trinsically evaluate our approach using unsu-pervised QA: our QG model is used to gen-erate synthetic QA pairs for training a QAmodel. Experimental results show that, trainedwith only 20k English Wikipedia-based syn-thetic QA pairs, the QA model substantiallyoutperforms previous unsupervised models onthree in-domain datasets (SQuAD1.1, NaturalQuestions, TriviaQA) and three out-of-domaindatasets (NewsQA, BioASQ, DuoRC), demon-strating the transferability of the approach.

1 Introduction

The aim of Question Generation (QG) is the pro-duction of meaningful questions given a set of inputpassages and corresponding answers, a task withmany applications including dialogue systems as

Figure 1: Example questions generated via heuristicsinformed by semantic role labeling of summary sen-tences using different candidate answer spans

well as education (Graesser et al., 2005). Addition-ally, QG can be applied to Question Answering(QA) for the purpose of data augmentation (Puriet al., 2020) where labeled <passage, answer, ques-tion> triples are combined with synthetic <passage,answer, question> triples produced by a QG systemto train a QA system, and unsupervised QA (Lewiset al., 2019), in which only the QG system outputis used to train the QA system.

Early work on QG focused on template or rule-based approaches, employing syntactic knowledgeto manipulate constituents in declarative sentencesto form interrogatives (Heilman and Smith, 2009,2010). Although template-based methods are ca-pable of generating linguistically correct questions,the resulting questions often lack variety and incurhigh lexical overlap with corresponding declarativesentences. For example, the question generatedfrom the sentence Stephen Hawking announced theparty in the morning, with Stephen Hawking as thecandidate answer span, could be Who announcedthe party in the morning?, with a high level of lexi-cal overlap between the generated question and thedeclarative sentence. This is undesirable in a QAsystem (Hong et al., 2020) since the strong lexicalclues in the question would make it a poor test ofreal comprehension.

4135

Neural seq2seq models (Sutskever et al., 2014)have come to dominate QG (Du et al., 2017),and are commonly trained with <passage, an-swer, question> triples taken from human-createdQA datasets (Dzendzik et al., 2021) and this lim-its applications to the domain and language ofdatasets. Furthermore, the process of construct-ing such datasets involves a significant investmentof time and resources.

We subsequently propose a new unsupervisedapproach that frames QG as a summarization-questioning process. By employing freely availablesummary data, we firstly apply dependency pars-ing, named entity recognition and semantic rolelabeling to summaries, before applying a set ofheuristics that generate questions based on parsedsummaries. An end-to-end neural generation sys-tem is then trained employing the original newsarticles as input and the heuristically generatedquestions as target output.

An example is shown in Figure 1. The summaryis used as a bridge between the questions and pas-sages. Because the questions are generated fromthe summaries and not from the original passages,they have less of a lexical overlap with the passages.Crucially, however, they remain semantically closeto the passages since the summaries by definitioncontain the most important information containedin the passages. A second advantage of this QGapproach is that it does not rely on the existenceof a QA dataset, and it is arguably easier to obtainsummary data in a given language than equivalentQA data since summary data is created for manypurposes (e.g. news, review and thesis summaries)whereas many QA datasets are created specificallyfor training a QA system.

In order to explore the effectiveness of ourmethod, we carry out extensive experiments. Weprovide an extrinsic evaluation, and train an En-glish QG model using news summary data. Weemploy our QG model to generate synthetic QAdata to train a QA model in an unsupervised set-ting and test the approach with six English QAdatasets: SQuAD1.1, Natural Questions, TriviaQA,NewsQA, BioASQ and DuoRC (Rajpurkar et al.,2016; Kwiatkowski et al., 2019; Joshi et al., 2017;Trischler et al., 2017; Tsatsaronis et al., 2015; Sahaet al., 2018). Experiment results show that ourapproach substantially improves over previous un-supervised QA models even when trained on sub-stantially fewer synthetic QA examples.

Our contributions can be summarized as follows:

1. We propose a novel unsupervised QG ap-proach that employs summary data and syn-tactic/semantic analysis, which to our bestknowledge is the first work connecting textsummarization and question generation in thisway;

2. We employ our QG model to generate syn-thetic QA data achieving state-of-the-art per-formance even at low volumes of synthetictraining data.

2 Related Work

Question Generation Traditional approaches toQG mostly employ linguistic templates and rulesto transform declarative sentences into interroga-tives (Heilman and Smith, 2009). Recently, Dholeand Manning (2020) showed that, with the help ofadvanced neural syntactic parsers, template-basedmethods are capable of generating high-qualityquestions from texts.

Neural seq2seq generation models have addition-ally been widely employed in QG, with QG datausually borrowed from existing QA datasets (Duet al., 2017; Sun et al., 2018; Ma et al., 2020).Furthermore, reinforcement learning has been em-ployed by Zhang and Bansal (2019); Chen et al.(2019); Xie et al. (2020) to directly optimize dis-crete evaluation metrics such as BLEU (Papineniet al., 2002). Lewis et al. (2020) and Song et al.(2019) show that a large-scale pre-trained modelcan achieve state-of-the-art performance for super-vised QG (Dong et al., 2019; Narayan et al., 2020).

Question Generation Evaluation BLEU (Pap-ineni et al., 2002), ROUGE (Lin, 2004) and Me-teor (Banerjee and Lavie, 2005) metrics are com-monly borrowed from text generation tasks to eval-uate QG. Even with respect to original text gener-ation tasks, however, the use of such metrics hasbeen questioned (Callison-Burch et al., 2006; Re-iter, 2018). Such metrics are particularly prob-lematic for QG evaluation since multiple plausi-ble questions exist for a given passage and answer.Consequently, there has been a shift in focus toevaluating QG using an extrinsic evaluation thatgenerates synthetic QA pairs for the purpose ofevaluating their effectiveness as a data augmenta-tion or unsupervised QA approach (Alberti et al.,2019; Puri et al., 2020; Shakeri et al., 2020).

4136

Figure 2: An overview of our approach where Answer and Question are generated based on Summary by theQuestion Generation Heuristics, the Answer is combined with the Article to form the input to the Encoder, theQuestion is employed as the ground-truth label for the outputs of the Decoder.

Unsupervised QA In unsupervised QA, the QAmodel is trained using synthetic data based on a QGmodel instead of an existing QA dataset. Insteadof resorting to existing QA datasets, unsupervisedQG methods have been employed, such as Unsu-pervised Neural Machine Translation (Lewis et al.,2019). Fabbri et al. (2020) and Li et al. (2020)propose template/rule-based methods for generat-ing questions and employ retrieved paragraphs andcited passages as source passages to alleviate theproblems of lexical similarities between passagesand questions. Alberti et al. (2019); Puri et al.(2020); Shakeri et al. (2020) additionally employexisting QA datasets to train a QG model. Al-though related, this work falls outside the scope ofunsupervised QA.

3 Methodology

Diverging from supervised neural question gener-ation models trained on existing QA datasets, theapproach we propose employs synthetic QG data,that we create from summary data using a numberof heuristics, to train a QG model. We providean overview of the proposed method is shown inFigure 2. We then employ the trained QG model togenerate synthetic QA data that is further employedto train an unsupervised QA model.

3.1 Question GenerationIn order to avoid generating trivial questions thatare highly similar to corresponding declarative

statements, we employ summary data as a bridgeconnecting the generated question and the originalarticle.1 The process we employ involves, firstlyDependency Parsing (DP) of summary sentences,followed by Named-Entity Recognition (NER) andfinally Semantic Role Labeling (SRL). DP is firstlyemployed as a means of identifying the main verb(root verb), in addition to other constituents suchas auxiliaries. NER is then responsible for taggingall entities in the summary sentence to facilitatediscovery of the most appropriate question wordsto generate. The pivotal component of linguisticanalysis is then SRL, employed to obtain all seman-tic frames for the summary sentence. Each frameconsists of a verb followed by a set of argumentswhich correspond to phrases in the sentence. Anargument could comprise, for example, an Agent(who initiates the action described by the verb),a Patient (who undergoes the action), and a setof modifier arguments such as a temporal ARG-TMP or locative argument ARG-LOC. Questionsare then generated from the arguments accordingto argument type and NER tags, which means thatwh-words can be determined jointly.

Returning to the example in Figure 1: given theSRL analysis [U2’s lead singer Bono ARG-0] has[had VERB] [emergency spinal surgery ARG-1][after suffering an injury while preparing for tour

1Data we employ in experiments is news summary dataoriginally from BBC News (Narayan et al., 2018) and thenews articles are typically a few hundred words in length.

4137

dates ARG-TMP]., the three questions shown inFigure 1 can be generated based on these threearguments.

The pseudocode for our algorithm to generatequestions is shown in Algorithm 1. We first ob-

Algorithm 1: Question Generation Heuris-ticsS = summarysrl_frames = SRL(S)ners = NER(S)dps = DP (S)examples = []for frame in srl_frames do

root_verb = dpsrootverb = frameverbif root_verb equal to verb then

for arg in frame dowh∗ =identify_wh_word(arg, ners)

base_verb, auxs =decomp_verb(arg, dps, root_verb)

Qarg =wh_move(S,wh∗, base_verb, auxs)

Qarg = post_edit(Qarg)examples.append(context,Qarg, arg)

endend

end

tain all dependency edges and labels (dps), NERtags (ners) and SRL frames (srl_frames) of asummary sentence. We then iterate through all ar-guments in the frame of the root_verb (the verbwhose dependency label is root) and identify ap-propriate wh-words (wh∗) for each argument usingthe function identify_wh_word according to itsargument type and the NER tags of entities in theargument. We follow Dhole and Manning (2020)to use the standard wh-words in English associatedwith appropriate argument types and NER tags.We then decompose the current main verb to itsbase form (base_verb) and appropriate auxiliarywords (auxs) in the decomp_verb function, beforefinally inserting the wh-words and the auxiliaryverbs in appropriate positions using the wh_move.As can be seen from Figure 1, a single summarysentence generates multiple questions when its SRLframe has multiple arguments.

3.2 Training a Question Generation ModelThe summarization data we employ consists of<passage-summary> pairs. Questions are gener-ated from the summaries using the heuristics de-scribed in Section 3.1, so that we have <passage-summary> pairs and <summary-question-answer>triples, which we then combine to form <passage-

answer-question> triples to train a QG model. Wetrain an end-to-end seq2seq model rather than de-ploying a pipeline in which the summary is firstgenerated followed by the question to eliminate therisk of error accumulation in the generation process.By using this QG data to train a neural generationmodel, we expect the model to learn a combinationof summarization and question generation. In otherwords, such knowledge can be implicitly injectedinto the neural generation model via our QG data.

To train the question generation model, we con-catenate each passage and answer to form a se-quence: passage <SEP> answer <SEP>, where<SEP> is a special token used to separate the pas-sage and answer. This sequence is the input andthe question is the target output (objective). In ourexperiments, we use BART (Lewis et al., 2020) forgeneration, which is optimized by the followingnegative log likelihood loss function:

L = −N∑i=1

logP (qi|C,A) (1)

where qi is the i-th token in the question, and Cand A are context and answer, respectively.

4 Experiments

We test our idea of using summaries in questiongeneration by applying the questions generated byour QG system in unsupervised QA. We describethe details of our experiment setup, followed by ourunsupervised QA results on six English benchmarkextractive QA datasets.

4.1 Experiment Setup4.1.1 Question GenerationDatasets We test the proposed method usingnews summary data from XSUM (Narayan et al.,2018), crawled from BBC news website.2 XSUMcontains 226,711 <passage-summary> pairs, witheach summary containing a single sentence.

QG Details We employ AllenNLP3 (Gardneret al., 2017) to obtain dependency trees, namedentities and semantic role labels for summary sen-tences, before further employing this knowledge togenerate questions from summaries following thealgorithm described in Section 3.1. We remove anygenerated <passage-answer-question> triples thatmeet one or more of the following three conditions:

2www.bbc.com3https://demo.allennlp.org/

4138

1. Articles longer than 480 tokens (exceedingthe maximum BART input length);

2. Articles in which fewer than 55% of tokens inthe answer span are not additionally present inthe passage (to ensure sufficient lexical over-lap between the answer and passage);

3. Questions shorter than 5 tokens (very shortquestions are likely to have removed too muchinformation)

For the dataset in question, this process resultedin a total of 14,830 <passage-answer-question>triples.

For training the QG model, we employ imple-mentations of BART (Lewis et al., 2020) from Hug-gingface (Wolf et al., 2019). The QG model weemploy is BART-base. We train the QG model onthe QG data for 3 epochs with a learning rate of3×10−5, using the AdamW optimizer (Loshchilovand Hutter, 2019).

4.1.2 Unsupervised QADatasets We carry out experiments on six extrac-tive QA datasets, namely, SQuAD1.1 (Rajpurkaret al., 2016), NewsQA (Trischler et al., 2017), Nat-ural Questions (Kwiatkowski et al., 2019), Triv-iaQA (Joshi et al., 2017), BioASQ (Tsatsaroniset al., 2015) and DuoRC (Saha et al., 2018). Weemploy the official data of SQuAD1.1, NewsQAand TriviaQA and for Natural Questions, BioASQand DuoRC, we employ the pre-processed datareleased by MRQA (Fisch et al., 2019).

Unsupervised QA Training Details To gener-ate synthetic QA training data, we make useof Wikidumps 4 by firstly removing all HTMLtags and reference links, then extracting para-graphs that are longer than 500 characters, re-sulting in 60k paragraphs sampled from all para-graphs of Wikidumps. We employ the NER toolk-its of Spacy5 (Honnibal et al., 2020) and Al-lenNLP6 (Gardner et al., 2017) to extract entitymentions in the paragraphs. We then remove para-graph, answer pairs that meet one or more of thefollowing three conditions: 1) paragraphs with lessthan 20 words and more than 480 words; 2) para-graphs with no extracted answer, or where the ex-tracted answer is not in the paragraph due to text

4https://dumps.wikimedia.org/5https://spacy.io/6https://demo.allennlp.org/named-entity-

recognition/named-entity-recognition

SQuAD1.1

Models EM F-1

SUPERVISED MODELS

Match-LSTM 64.1 73.9BiDAF 66.7 77.3BERT-base 81.2 88.5BERT-large 84.2 91.1

UNSUPERVISED MODELS

Lewis et al. (2019) 44.2 54.7Li et al. (2020) 62.5 72.6Our Method 65.6 74.5

Table 1: In-domain experimental results of supervisedand unsupervised methods on SQuAD1.1. The highestscores of unsupervised methods are in bold.

tokenization; 3) answers consisting of a single pro-noun.

Paragraphs and answers are concatenated toform sequences of the form passage <SEP> an-swer <SEP>, before being fed into the trainedBART-QG model to obtain corresponding ques-tions. This results in 20k synthetic QA pairs, whichare then employed to train an unsupervised QAmodel.

The QA model we employ is BERT-large-whole-word-masking (which we henceforth refer to asBERT-large for ease of reference). Documentlength and stride length are 364 and 128 respec-tively, the learning rate is set to 1 × 10−5. Eval-uation metrics for unsupervised QA are ExactMatch (EM) and F-1 score.

4.2 ResultsWe use the 20k generated synthetic QA pairs totrain a BERT QA model and first validate its perfor-mance on the development sets of three benchmarkQA datasets based on Wikipedia – SQuAD1.1, Nat-ural Questions and TriviaQA. The results of ourmethod are shown in Tables 1 and 2. The unsuper-vised baselines we compare with are as follows:

1. Lewis et al. (2019) employ unsupervised neu-ral machine translation (Artetxe et al., 2018)to train a QG model; 4M synthetic QA exam-ples were generated to train a QA model;

2. Li et al. (2020) employ dependency trees togenerate questions and employed cited docu-ments as passages.

For comparison, we also show the results of somesupervised models fine-tuned on the correspond-

4139

NQ TriviaQA

Models EM F-1 EM F-1

SUPERVISED MODELS

BERT-base 66.1 78.5 65.1 71.2BERT-large 69.7 81.3 67.9 74.8

UNSUPERVISED MODELS

Lewis et al. (2019) 27.5 35.1 19.1 23.8Li et al. (2020) 31.3 48.8 27.4 38.4Our Method 46.0 53.5 36.7 43.0

Table 2: In-domain experimental results: Natural Ques-tions and TriviaQA.

ing training sets: Match-LSTM (Wang and Jiang),BiDAF (Seo et al., 2016), BERT-base and BERT-large (Devlin et al., 2019).

SQuAD1.1 results are shown in Table 1. Theresults of all baseline models are taken directlyfrom published work. As can be seen from re-sults in Table 1, our proposed method outper-forms all unsupervised baselines, and even exceedsthe performance of one supervised model, Match-LSTM (Wang and Jiang).

Results for Natural Questions and TriviaQA areshown in Table 2. The results of all baseline modelswere produced using the released synthetic QAdata to finetune a BERT-large model. Our methodoutperforms previous state-of-the-art unsupervisedmethods by a substantial margin, obtaining relativeimprovements over the best unsupervised baselinemodel of 47% with respect to EM, 10% F-1 onNatural Questions, and by 34% EM and 12% F-1on TriviaQA.

In summary, our method achieves the best perfor-mance (both in terms of EM and F-1) out of threeunsupervised models on all three tested datasets.Furthermore, this high performance is possiblewith as few as 20k training examples. Compared toprevious work, this is approximately less than 10%of the training data employed (Li et al., 2020).

Transferability of Our Generated Synthetic QAData We also validate our method’s efficacy onthree out-of-domain QA datasets: NewsQA createdfrom news articles, BioASQ created from biomedi-cal articles, and DuoRC created from movie plots,for the purpose of evaluating the transferability ofthe Wikipedia-based synthetic data. Results in Ta-ble 3 show that our proposed method additionallyoutperforms the unsupervised baseline models onthe out-of-domain datasets, achieving F1 improve-ments over previous state-of-the-art methods by

NewsQA BioASQ DuoRC

EM F-1 EM F-1 EM F-1

Lewis et al. (2019) 19.6 28.5 18.9 27.0 26.0 32.6Li et al. (2020) 33.6 46.3 30.3 38.7 32.7 41.1Our Method 37.5 50.1 32.0 43.2 38.8 46.5

Table 3: Out-of-domain experimental results of unsu-pervised methods on NewsQA, BioASQ and DuoRC.The results of two baseline models on NewsQA aretaken from Li et al. (2020) and their results on BioASQand DuoRC are from fine-tuning a BERT-large modelon their synthetic data.

3.8, 4.5 and 5.4 points respectively. It is worthnoting that our data adapts very well to DuoRC,created from movie plots where the narrative styleis expected to require more complex reasoning.Experiment results additionally indicate that ourgenerated synthetic data transfers well to domainsdistinct from that of the original summary data.

5 Analysis

5.1 Effect of Answer Extraction

In the unsupervised QA experiments, we extractedanswers from Wikipedia passages before feedingthem into our QG model to obtain questions. These<passage, answer, question> triples constitute thesynthetic data employed to train the QA model. Ad-ditionally, we wish to consider what might happenif we instead employ passages and answers taken di-rectly from the QA training data? Doing this wouldmean that the QA system is no longer consideredunsupervised but we carry out this experiment inorder to provide insight into the degree to whichthere may be room for improvement in terms of ourNER-based automatic answer extraction method(described in Section 4.1.2). For example, therecould well be a gap between the NER-extractedanswers and human-extracted answers, and in thiscase, the NER could extract answers, for example,that are not entirely worth asking about or indeedmiss answers that are highly likely to be askedabout. Results of the two additional settings areshown in Table 5 – answer extraction has quite alarge effect on the quality of generated syntheticQA data. When we employ the answers from thetraining set, the performance of the QA model isimproved by 5 F-1 points for SQuAD1.1, and over10 F-1 points for Natural Questions and TriviaQA.

4140

Questions Answer Comments

who is the frontman of swedish rock band mhiam ? Mattis Malinen 3

which sultan has been in bosnia for more than a year ? Sultan Mehmed II 3

what is a major economic driver for the state of ohio ? Ohio’s geographic location 3

in what time was the first parish council elected ? March 1972 3

what do the chattanooga area will host in 2017 ? the Ironman Triathlon 3grammar errorwhat have sold five cars in the uk this year ? Surrey Motors missing informationwhen did the first military college in the us open ? 2009 factual errorwhat has been described as a " giant fish " ? Darwin mismatch

Table 4: Examples of generated questions with corresponding answers.3represents correct examples.

SQuAD1.1 NewsQA NQ TriviaQA

Models EM F-1 EM F-1 EM F-1 EM F-1

Our Method (NER-extracted answers)† 65.6 74.5 37.5 50.1 46.0 53.5 36.7 43.0Our Method (Human-extracted answers) ‡ 68.0 79.5 40.5 59.3 57.3 66.7 54.2 61.1

Table 5: Comparison between synthetic data generated based on Wikipedia and synthetic data generated basedon corresponding training set. †are results of QA model finetuned on synthetic data generated based on NER-extracted answers, ‡are results of QA model finetuned on synthetic data based on the answers in the training set ofSQuAD1.1, NewsQA, NQ and TriviaQA.

5.2 Effect of Different HeuristicsWe additionally investigate the effect of a rangeof alternate heuristics employed in the process ofconstructing the QG training data described in Sec-tion 3.1. Recall that the QG data is employed totrain a question generator which is then employedto generate synthetic QA data for unsupervised QA.

The heuristics are defined as follows:

• Naive-QG only employs summary sen-tences as passages (instead of the original arti-cles) and generates trivial questions in whichonly the answer spans are replaced with theappropriate question words. For example,for the sentence Stephen Hawking announcedthe party in the morning, with the party asthe answer span, the question generated byNaive-QG would be Stephen Hawking an-nounced what in the morning? We employthe summary sentences as input and questionsas target output to form the QG training data.

• Summary-QGmakes use of the original newsarticles of the summaries as passages ratherthan summary sentences to avoid high lexicaloverlap between the passage and question.

Summary-QG can work with the followingheuristics:

– Main Verb: we only generate ques-tions based on the SRL frame of the main

Heuristics EM F-1

Naive-QG 31.1 43.3Summary-QG 50.9 59.4

+Main Verb 53.8 63.6+Wh-Movement 59.5 67.7+Decomp-Verb 64.1 73.9+NER-Wh 65.4 74.8

Table 6: Experiment results of the effects to unsuper-vised QA performance on SQuAD1.1 of using differentheuristics in constructing QG data.

verb (root verb) in the dependency treeof the summary sentences, rather thanusing verbs in subordinate clauses;

– Wh-Movement: we move the questionwords to the beginning of the sentence.For example, in the sentence StephenHawking announced what in the morn-ing? we move what to the beginningto obtain what Stephen Hawking an-nounced in the morning?;

– Decomp-Verb: the main verb is de-composed to its base form and auxil-iaries;

– NER-Wh: we employ the NER tags toget more precise question words for ananswer. For example, for the answer

4141

span NBA player Michael Jordan, thequestion words would be which NBAplayer instead of who or what.

We employ the QG data generated by theseheuristics to train QG models, which leads to sixBART-QG models. We then employ these six mod-els to further generate synthetic QA data basedon the same Wikipedia data and compare theirperformances on the SQuAD1.1 dev set. The re-sults in Table 6 show that using articles as pas-sages to avoid lexical overlap with their summary-generated questions greatly improves QA perfor-mance. Summary-QG outperforms Naive-QGby roughly 20 EM points and 16 F-1 points. Theresults for the other heuristics show that theycontinuously improve the performance, especiallyWh-Movement and Decomp-Verb which makethe questions in the QG data more similar to thequestions in the QA dataset.

20

30

40

EM

EM on NQ

20

40

60

EM on SQuAD1.1

0 20 40 60 80Size of Synthetic Data (K)

30

40

50

F-1

Scor

e

F1 on NQ

0 20 40 60 80Size of Synthetic Data (K)

20

40

60

F1 on SQuAD1.1

Figure 3: Experimental results on NQ and SQuAD1.1of using different amount of synthetic data.

5.3 Effect of the Size of Synthetic QA Data

We investigate the effects of varying the quantityof synthetic QA data. Results in Figure 3 show thatour synthetic data allows the QA model to achievecompetitive performance even with fewer than 20kexamples, which suggests that our synthetic datacontains sufficient QA knowledge to enable modelsto correctly answer a question with less syntheticdata compared to previous unsupervised methods.The data-efficiency of our approach increases thefeasibility of training a QA system for a target do-main where there is no labeled QA data available.

0

20

40

60

80

100

EM

EM on NQ

BERT-LLi et al.Ours

0

20

40

60

80

100EM on SQuAD1.1

BERT-LLi et al.Ours

1 5 10 50 100

300

50010

0020

0030

0050

0010

00030

00050

00080

000

Size of Labeled Data

0

20

40

60

80

100

F-1

Scor

e

F1 on NQ

BERT-LLi et al.Ours

1 5 10 50 100

300

50010

0020

0030

0050

0010

00030

00050

00080

000

Size of Labeled Data

0

20

40

60

80

100F1 on SQuAD1.1

BERT-LLi et al.Ours

Figure 4: Experimental results of our method with com-parison of Li et al. (2020) and BERT-large using differ-ent amount of labeled QA examples in the training setof NQ and SQuAD1.1.

5.4 Few-shot Learning

We conduct experiments in a few-shot learningsetting, in which we employ a limited number oflabeled QA examples from the training set. Wetake the model trained with our synthetic QA data,the model trained with the synthetic QA data of Liet al. (2020) and a vanilla BERT model, with allQA models employing BERT-large (Devlin et al.,2019). We train these models using progressivelyincreasing amounts of labeled QA samples fromNatural Questions (NQ) and SQuAD1.1 and as-sess their performance on corresponding dev sets.Results are shown in Figure 4 where with only asmall amount of labeled data (less than 5,000 ex-amples), our method outperforms Li et al. (2020)and BERT-large, clearly demonstrating the efficacyof our approach in a few-shot learning setting.

5.5 QG Error Analysis

Despite substantial improvements over baselines,our proposed approach inevitably still incurs errorand we therefore take a closer look at the questionsgenerated by our QG model. We manually exam-ine 50 randomly selected questions, 31 (62%) ofwhich were deemed high quality questions. The re-maining 19 contain various errors with some ques-tions containing more than one error, includingmismatched wh-word and answer (12%), missinginformation needed to locate the answer (8%), fac-tual errors (10%) and grammatical errors (8) (16%)Typical examples are shown in Table 4.

4142

6 Conclusion

We propose an unsupervised question generationmethod which uses summarization data to 1) mini-mize the lexical overlap between passage and ques-tion and 2) provide a QA-dataset-independent wayof generating questions. Our unsupervised QA ex-trinsic evaluation on SQuAD1.1, NQ and TriviaQAusing synthetic QA data generated by our methodshows that our method substantially outperformsprevious methods for generating synthetic QA forunsupervised QA. Furthermore, our synthetic QAdata transfers well to the out-of-domain datasets.Future work includes refining our question genera-tion heuristics and applying our approach to otherlanguages.

Acknowledgements

This work was funded by Science Foundation Ire-land through the SFI Centre for Research Train-ing in Machine Learning (18/CRT/6183). We alsothank the reviewers for their insightful and helpfulcomments.

ReferencesChris Alberti, Daniel Andor, Emily Pitler, Jacob De-

vlin, and Michael Collins. 2019. Synthetic QA cor-pora generation with roundtrip consistency. In Pro-ceedings of the 57th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 6168–6173, Florence, Italy. Association for Computa-tional Linguistics.

Mikel Artetxe, Gorka Labaka, Eneko Agirre, andKyunghyun Cho. 2018. Unsupervised neural ma-chine translation. In International Conference onLearning Representations.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR:An automatic metric for MT evaluation with im-proved correlation with human judgments. In Pro-ceedings of the ACL Workshop on Intrinsic and Ex-trinsic Evaluation Measures for Machine Transla-tion and/or Summarization, pages 65–72, Ann Ar-bor, Michigan. Association for Computational Lin-guistics.

Chris Callison-Burch, Miles Osborne, and PhilippKoehn. 2006. Re-evaluating the role of Bleu in ma-chine translation research. In 11th Conference ofthe European Chapter of the Association for Com-putational Linguistics, Trento, Italy. Association forComputational Linguistics.

Yu Chen, Lingfei Wu, and Mohammed J Zaki. 2019.Reinforcement learning based graph-to-sequencemodel for natural question generation. In Interna-tional Conference on Learning Representations.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Kaustubh Dhole and Christopher D. Manning. 2020.Syn-QG: Syntactic and shallow semantic rules forquestion generation. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 752–765, Online. Association forComputational Linguistics.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi-aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou,and Hsiao-Wuen Hon. 2019. Unified languagemodel pre-training for natural language understand-ing and generation.

Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn-ing to ask: Neural question generation for readingcomprehension. In Proceedings of the 55th AnnualMeeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), pages 1342–1352,Vancouver, Canada. Association for ComputationalLinguistics.

Daria Dzendzik, Carl Vogel, and Jennifer Foster. 2021.English machine reading comprehension datasets: Asurvey. In Proceedings of the 2021 Conference onEmpirical Methods in Natural Language Processing(EMNLP). Association for Computational Linguis-tics.

Alexander Fabbri, Patrick Ng, Zhiguo Wang, RameshNallapati, and Bing Xiang. 2020. Template-basedquestion generation from retrieved sentences for im-proved unsupervised question answering. In Pro-ceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 4508–4513, Online. Association for Computational Lin-guistics.

Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eu-nsol Choi, and Danqi Chen. 2019. MRQA 2019shared task: Evaluating generalization in readingcomprehension. In Proceedings of 2nd MachineReading for Reading Comprehension (MRQA) Work-shop at EMNLP.

Matt Gardner, Joel Grus, Mark Neumann, OyvindTafjord, Pradeep Dasigi, Nelson F. Liu, MatthewPeters, Michael Schmitz, and Luke S. Zettlemoyer.2017. Allennlp: A deep semantic natural languageprocessing platform.

A. C. Graesser, P. Chipman, B. C. Haynes, and A. Ol-ney. 2005. Autotutor: an intelligent tutoring systemwith mixed-initiative dialogue. IEEE Transactionson Education, 48(4):612–618.

4143

Michael Heilman and Noah A Smith. 2009. Questiongeneration via overgenerating transformations andranking. Technical report, Carnegie-Mellon Univer-sity.

Michael Heilman and Noah A. Smith. 2010. Goodquestion! statistical ranking for question genera-tion. In Human Language Technologies: The 2010Annual Conference of the North American Chap-ter of the Association for Computational Linguistics,pages 609–617, Los Angeles, California. Associa-tion for Computational Linguistics.

Giwon Hong, Junmo Kang, Doyeon Lim, and Sung-Hyon Myaeng. 2020. Handling anomalies of syn-thetic questions in unsupervised question answer-ing. In Proceedings of the 28th International Con-ference on Computational Linguistics, pages 3441–3448, Barcelona, Spain (Online). International Com-mittee on Computational Linguistics.

Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy:Industrial-strength Natural Language Processing inPython.

Mandar Joshi, Eunsol Choi, Daniel Weld, and LukeZettlemoyer. 2017. TriviaQA: A large scale dis-tantly supervised challenge dataset for reading com-prehension. In Proceedings of the 55th Annual Meet-ing of the Association for Computational Linguistics(Volume 1: Long Papers), pages 1601–1611, Van-couver, Canada. Association for Computational Lin-guistics.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Al-berti, Danielle Epstein, Illia Polosukhin, Jacob De-vlin, Kenton Lee, Kristina Toutanova, Llion Jones,Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai,Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.Natural questions: A benchmark for question an-swering research. Transactions of the Associationfor Computational Linguistics, 7:452–466.

Mike Lewis, Yinhan Liu, Naman Goyal, Mar-jan Ghazvininejad, Abdelrahman Mohamed, OmerLevy, Veselin Stoyanov, and Luke Zettlemoyer.2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation,and comprehension. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7871–7880, Online. Associationfor Computational Linguistics.

Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel.2019. Unsupervised question answering by clozetranslation. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguistics,pages 4896–4910, Florence, Italy. Association forComputational Linguistics.

Zhongli Li, Wenhui Wang, Li Dong, Furu Wei, andKe Xu. 2020. Harvesting and refining question-answer pairs for unsupervised QA. In Proceedings

of the 58th Annual Meeting of the Association forComputational Linguistics, pages 6719–6728, On-line. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for auto-matic evaluation of summaries. In Text Summariza-tion Branches Out, pages 74–81, Barcelona, Spain.Association for Computational Linguistics.

Ilya Loshchilov and Frank Hutter. 2019. Decoupledweight decay regularization. In International Con-ference on Learning Representations.

Xiyao Ma, Qile Zhu, Yanlin Zhou, and Xiaolin Li.2020. Improving question generation with sentence-level semantic matching and answer position infer-ring. In Proceedings of the AAAI Conference on Ar-tificial Intelligence, volume 34, pages 8464–8471.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata.2018. Don’t give me the details, just the summary!topic-aware convolutional neural networks for ex-treme summarization. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, pages 1797–1807, Brussels, Bel-gium. Association for Computational Linguistics.

Shashi Narayan, Gonçalo Simoes, Ji Ma, HannahCraighead, and Ryan Mcdonald. 2020. Qurious:Question generation pretraining for text generation.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proceedings ofthe 40th Annual Meeting of the Association for Com-putational Linguistics, pages 311–318, Philadelphia,Pennsylvania, USA. Association for ComputationalLinguistics.

Raul Puri, Ryan Spring, Mohammad Shoeybi, MostofaPatwary, and Bryan Catanzaro. 2020. Training ques-tion answering models from synthetic data. InProceedings of the 2020 Conference on EmpiricalMethods in Natural Language Processing (EMNLP),pages 5811–5826, Online. Association for Computa-tional Linguistics.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. SQuAD: 100,000+ questions formachine comprehension of text. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 2383–2392, Austin,Texas. Association for Computational Linguistics.

Ehud Reiter. 2018. A Structured Review of the Validityof BLEU. Computational Linguistics, 44(3):393–401.

Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, andKarthik Sankaranarayanan. 2018. DuoRC: Towardscomplex language understanding with paraphrasedreading comprehension. In Proceedings of the 56thAnnual Meeting of the Association for Computa-tional Linguistics (Volume 1: Long Papers), pages1683–1693, Melbourne, Australia. Association forComputational Linguistics.

4144

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, andHannaneh Hajishirzi. 2016. Bidirectional attentionflow for machine comprehension. arXiv preprintarXiv:1611.01603.

Siamak Shakeri, Cicero Nogueira dos Santos, HenghuiZhu, Patrick Ng, Feng Nan, Zhiguo Wang, RameshNallapati, and Bing Xiang. 2020. End-to-end syn-thetic data generation for domain adaptation of ques-tion answering systems. In Proceedings of the 2020Conference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 5445–5460, On-line. Association for Computational Linguistics.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to se-quence pre-training for language generation. In In-ternational Conference on Machine Learning, pages5926–5936. PMLR.

Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, YanjunMa, and Shi Wang. 2018. Answer-focused andposition-aware neural question generation. In Pro-ceedings of the 2018 Conference on Empirical Meth-ods in Natural Language Processing, pages 3930–3939, Brussels, Belgium. Association for Computa-tional Linguistics.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.Sequence to sequence learning with neural networks.In Advances in Neural Information Processing Sys-tems, volume 27. Curran Associates, Inc.

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har-ris, Alessandro Sordoni, Philip Bachman, and Ka-heer Suleman. 2017. NewsQA: A machine compre-hension dataset. In Proceedings of the 2nd Work-shop on Representation Learning for NLP, pages191–200, Vancouver, Canada. Association for Com-putational Linguistics.

George Tsatsaronis, Georgios Balikas, ProdromosMalakasiotis, Ioannis Partalas, Matthias Zschunke,Michael R Alvers, Dirk Weissenborn, AnastasiaKrithara, Sergios Petridis, Dimitris Polychronopou-los, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question an-swering competition. BMC bioinformatics, 16(1):1–28.

Shuohang Wang and Jing Jiang. Machine comprehen-sion using match-lstm and answer pointer.(2017). InICLR 2017: International Conference on LearningRepresentations, Toulon, France, April 24-26: Pro-ceedings, pages 1–15.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s trans-formers: State-of-the-art natural language process-ing. ArXiv, abs/1910.03771.

Yuxi Xie, Liangming Pan, Dongzhe Wang, Min-YenKan, and Yansong Feng. 2020. Exploring question-specific rewards for generating deep questions. In

Proceedings of the 28th International Conferenceon Computational Linguistics, pages 2534–2546,Barcelona, Spain (Online). International Committeeon Computational Linguistics.

Shiyue Zhang and Mohit Bansal. 2019. Address-ing semantic drift in question generation for semi-supervised question answering. In Proceedings ofthe 2019 Conference on Empirical Methods in Nat-ural Language Processing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 2495–2509, Hong Kong,China. Association for Computational Linguistics.

4145

A Appendix

A.1 Effects of Different Beam Size

2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0Beam Size in Decoding

62.0

62.5

63.0

63.5

64.0

64.5

65.0

65.5

EM

71.5

72.0

72.5

73.0

73.5

74.0

74.5

F-1

Scor

e

EMF1 Score

Figure 5: Experimental results of the effects of usingdifferent beam-size in decoding process when generat-ing synthetic questions.

We also study the effects of different beam sizein generating synthetic questions to the perfor-mance of downstream QA task. Experiments areconducted on SQuAD1.1 dev set using BERT-large,questions in the synthetic QA data are generatedwith different beam size using the same BART-QGmodel. The experimental results in Figure 5 showthat the beam size is an important factor affectingthe performance of unsupervised QA, the largestmargin between the highest score (beam-15) andthe lowest score (beam-1) in Figure 5 is close to 4points on EM and F-1 score.

A.2 Question Type Distribution

What When Where Who Why How0.0

0.1

0.2

0.3

0.4

0.5

0.6

Question Type DistributionQG dataSQuAD1.1 training setOur synthetic QA data

Figure 6: Question type distribution

We show the distribution of question types ofQG data described in Section 4.1.1, training set

of SQuAD1.1 and our synthetic QA data in Sec-tion 4.1.2 in Figure 6, question types are defined asWhat, When, Where, Who, Why, How. The QG datahas more what, when, where questions, indicatingthe existence of more SRL arguments associatedwith such question types in the summary sentences.

A.3 Generated QA ExamplesSome Wikipedia-based <passage, answer, ques-tion> examples generated by our BART-QG modelare shown in Table 7, Table 8 and Table 9.

4146

Passage Answer QuestionAt a professional level, most matchesproduce only a few goals. For exam-ple, the 2005–06 season of the EnglishPremier League produced an average of2.48 goals per match. The Laws of theGame do not specify any player posi-tions other than goalkeeper, but a num-ber of specialised roles have evolved.

the 2005–06 season when did the english footballteam produce an average of 2.49goals per match , according to thelaws of the game ?

The Hebrew Book Week is held eachJune and features book fairs, publicreadings, and appearances by Israeli au-thors around the country. During theweek, Israel’s top literary award, theSapir Prize, is presented.

The Hebrew Book Week what is held every june to cele-brate the publication of books inhebrew ?

On December 12, 2016, Senate Ma-jority Leader Republican Mitch Mc-Connell expressed confidence in U.S. in-telligence. McConnell added that inves-tigation of Russia’s actions should bebipartisan and held by the Senate Intel-ligence Committee. The next day, Sen-ate Intelligence Committee ChairmanRichard Burr (R-NC) and Vice Chair-man Mark Warner (D-VA) announcedthe scope of the committee’s .

Republican Mitch Mc-Connell

which republican has called for aspecial committee to investigaterussia ’s alleged meddling in the2016 presidential election ?

Meanwhile, the Soho Mint struck coinsfor the East India Company, SierraLeone and Russia, while producinghigh-quality planchets, or blank coins,to be struck by national mints elsewhere.The firm sent over 20 million blanksto Philadelphia, to be struck into centsand half-cents by the United States Mint—Mint Director Elias Boudinot foundthem to be "perfect and beautifully pol-ished".

Elias Boudinot who has been working for a com-pany that made coins for the usmint ?

Table 7: Some generated QA examples.

4147

Passage Answer QuestionIn March 2008 as part of the annualbudget, the government introduced sev-eral laws to amend the Immigration andRefugee Protection Act. The changeswould have helped to streamline immi-grant application back-up, to speed upapplication for skilled workers and torapidly reject other ones that are judgednot admissible by immigration officers.Immigrant applications had risen to ahigh of 500,000, creating a delay of upto six months for an application to beprocessed.

March 2008 when did the uk introduce newimmigration laws ?

The other group members as far backas 1996 had noticed Paddy Clancy’sunusual mood swings. In the springof 1998 the cause was finally detected;Paddy had a brain tumor as well as lungcancer. His wife waited to tell himabout the lung cancer, so as not to dis-courage him when he had a brain opera-tion.

the spring of 1998 in what time was paddy diag-nosed with lung cancer ?

In 1365 officials were created to super-vise the fish market in the town, whilstillegal fishing and oyster cultivation wastargeted by the bailiffs in an edict from1382, which prohibited the forestallingof fish by blocking the river, the dredg-ing of oysters out of season and the ob-structing of the river. Colchester arti-sans included clockmakers, who main-tained clocks in church towers acrossnorth Essex and Suffolk.

north Essex where were hundreds of clocksmade by local artisans ?

Badge numbers for Sheriffs andDeputies consist of a prefix number,which represents the county number,followed by a one to three digit num-ber, which represents the Sheriff’s orDeputy’s number within that specificoffice. The Sheriff’s badge number ineach county is always #1. So the Sherifffrom Bremer County would have an IDnumber of 9-1 (9 is the county numberfor Bremer County and 1 is the numberfor the Sheriff).

The Sheriff’s badge num-ber

what is the number used to iden-tify the sheriff in each county ?

Table 8: Some generated QA examples.

4148

Passage Answer QuestionAppian wrote that Calpurnius Piso wassent as a commander to Hispania be-cause there were revolts. The followingyear Servius Galba was sent without sol-diers because the Romans were busywith Cimbrian War and a slave rebel-lion in Sicily (the [Third Servile War],104-100 BC). In the former war the Ger-manic tribes of the Cimbri and the Teu-tones migrated around Europe and in-vaded territories of allies of Rome, par-ticularly in southern France, and routedthe Romans in several battles until theirfinal defeat.

Calpurnius Piso who was sent to the south of italyto fight for the roman empire ?

The parish churches of Sempringham,Birthorpe, Billingborough, and Kirkbywere already appropriated. Yet in 1247,Pope Innocent IV granted to the mas-ter the right to appropriate the churchof Horbling, because there were 200women in the priory who often lackedthe necessaries of life. The legal ex-penses of the order at the papal curiaperhaps accounted for their poverty.

200 there were how many women inthe priory of horbling in the 12thcentury ?

"Jerry West is the reason I came to theLakers", O’Neal later said. They usedtheir 24th pick in the draft to selectDerek Fisher. During the 1996–97 sea-son, the team traded Cedric Ceballos toPhoenix for Robert Horry. O’Neal ledthe team to a 56–26 record, their besteffort since 1990–91, despite missing31 games due to a knee injury. O’Nealaveraged 26.2 ppg and 12.5 rpg andfinished third in the league in blockedshots (2.88 bpg) in 51 games.

the 1996–97 season when do the phoenix suns beginwith a trade to the los angelesclippers ?

Finnish popular music also includes var-ious kinds of dance music; tango, astyle of Argentine music, is also pop-ular. One of the most productive com-posers of popular music was ToivoKärki, and the most famous singer OlaviVirta (1915–1972). Among the lyricists,Sauvo Puhtila (1928–2014), Reino He-lismaa (died 1965) and Veikko "Vexi"Salmi are a few of the most notable writ-ers. The composer and bandleader JimiTenor is well known for his brand ofretro-funk music.

Reino Helismaa who has been hailed as one offinland ’s most important writers?

Table 9: Some generated QA examples.