![Page 1: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/1.jpg)
Question Generation with Minimal RecursionSemantics
Xuchen Yao
European Masters in Language and Communication Technologies
Handleiding gebruik logo’s, versie 2.0, augustus 2007
Handleidinglogobestandenversie 2.0, augustus 2007
Supervisors: Prof. Hans Uszkoreit and Dr. Yi Zhang, Saarland UniversityCo-supervisor: Dr. Gosse Bouma, University of Groningen
28 July 2010, Master Colloquium
![Page 2: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/2.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 3: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/3.jpg)
Introduction Background System Architecture Evaluation Conclusion
Question Generation (QG)
The task of generating reasonable questions from a text.Deep QG: why, why not, what-if, what-if-not, howShallow QG:.who, what, when, where, which, how many/much,yes/noJackson was born on August 29, 1958 in Gary, Indiana.
• Who was born on August 29 , 1958 in Gary , Indiana?• Which artist was born on August 29 , 1958 in Gary , Indiana?• Where was Jackson born?• When was Jackson born?• Was Jackson born on August 29 , 1958 in Gary , Indiana?
![Page 4: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/4.jpg)
Introduction Background System Architecture Evaluation Conclusion
Question Generation (QG)
The task of generating reasonable questions from a text.Deep QG: why, why not, what-if, what-if-not, howShallow QG:.who, what, when, where, which, how many/much,yes/noJackson was born on August 29, 1958 in Gary, Indiana.
• Who was born on August 29 , 1958 in Gary , Indiana?• Which artist was born on August 29 , 1958 in Gary , Indiana?• Where was Jackson born?• When was Jackson born?• Was Jackson born on August 29 , 1958 in Gary , Indiana?
![Page 5: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/5.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 6: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/6.jpg)
Introduction Background System Architecture Evaluation Conclusion
Usage
• Intelligent tutoring systems• QG can ask learners questions based on learning materials in
order to check their accomplishment or help them focus on thekeystones in study.
• QG can also help tutors to prepare questions intended forlearners or prepare for questions possibly from learners.
• Closed-domain question answering (QA) systems• Some closed-domain QA systems use pre-defined (sometimes
hand-written) question-answer pairs to provide QA services.• By employing a QG approach such systems could expand to
other domains with a small effort.
![Page 7: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/7.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 8: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/8.jpg)
Introduction Background System Architecture Evaluation Conclusion
Approaches
• Template-based• What did <character> <verb>?
• Syntax-based• John plays football. (S V O)• John plays what? (S V WHNP)• John does play what? (S Aux-V V WHNP)• Does John play what? (Aux-V S V WHNP)• What does John play? (WHNP Aux-V S V)
• Semantics-based• play(John, football)• play(who, football)• play(John, what) || play(John, what sport)
![Page 9: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/9.jpg)
Introduction Background System Architecture Evaluation Conclusion
Approaches
• Template-based• What did <character> <verb>?
• Syntax-based• John plays football. (S V O)• John plays what? (S V WHNP)• John does play what? (S Aux-V V WHNP)• Does John play what? (Aux-V S V WHNP)• What does John play? (WHNP Aux-V S V)
• Semantics-based• play(John, football)• play(who, football)• play(John, what) || play(John, what sport)
![Page 10: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/10.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 11: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/11.jpg)
Introduction Background System Architecture Evaluation Conclusion
DELPH-IN (MRS/ERG/PET/LKB)Deep Linguistic Processing with HPSG: http://www.delph-in.net/
INDEX: e2RELS: <[ PROPER_Q_REL<0:4> LBL: h3 ARG0: x6 RSTR: h5 BODY: h4 ][ _like_v_1_rel<5:10> LBL: h8 ARG0: e2 [ e SF: PROP TENSE: PRES ] ARG1: x6 ARG2: x9 [ PROPER_Q_REL<11:17> LBL: h10 ARG0: x9 RSTR: h12 BODY: h11 ]>HCONS: < h5 qeq h7 h12 qeq h13 >
[ NAMED_REL<0:4> LBL: h7 ARG0: x6 (PERS: 3 NUM: SG) CARG: "John" ]
[ NAMED_REL<11:17> LBL: h13 ARG0: x9 (PERS: 3 NUM: SG) CARG: "Mary" ]
John likes Mary.like(John, Mary)
Parsingwith PET
Generationwith LKB
John likes Mary.
Minimal Recursion Semantics
English Resource Grammar
![Page 12: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/12.jpg)
Introduction Background System Architecture Evaluation Conclusion
Dependency MRSlike(John, Mary)
_like_v_1
named("John")
proper_q
rstr/h
arg1/neq
named("Mary")
proper_q
rstr/h
arg2/neq
Figure: DMRS for “John likes Mary.”
![Page 13: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/13.jpg)
Introduction Background System Architecture Evaluation Conclusion
Initial Idealike(John,Mary)->like(who,Mary)
_like_v_1
named("John")
proper_q
rstr/h
arg1/neq
named("Mary")
proper_q
rstr/h
arg2/neq_like_v_1
person
which_q
rstr/h
arg1/neq
named("Mary")
proper_q
rstr/h
arg2/neq
Figure: “John likes Mary” → “Who likes Mary?”
![Page 14: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/14.jpg)
Introduction Background System Architecture Evaluation Conclusion
Details(THEORY)MRS: Minimal Recursion Semanticsa meta-level language for describing semantic structures in someunderlying object language.
(GRAMMAR)ERG: English Resource Grammara general-purpose broad-coverage grammar implementation underthe HPSG framework.
(TOOL)LKB: Linguistic Knowledge Buildera grammar development environment for grammars in typedfeature structures and unification-based formalisms.(TOOL)PET: a platform for experimentation with efficientHPSG processing techniquesa two-stage parsing model with HPSG rules and PCFG models,balancing between precise linguistic interpretation and robustprobabilistic coverage.
![Page 15: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/15.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 16: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/16.jpg)
Introduction Background System Architecture Evaluation Conclusion
MrsQGhttp://code.google.com/p/mrsqg/
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8Output to
console/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
![Page 17: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/17.jpg)
Introduction Background System Architecture Evaluation Conclusion
Term Extraction
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1
Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8
Output toconsole/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4●Stanford Named Entity Recognizer●a regular expression NE tagger●an Ontology NE tagger
Jackson was born on August 29, 1958 in Gary, Indiana.
who which day
where
which location
NEperson NElocationNEdate
when
![Page 18: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/18.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 19: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/19.jpg)
Introduction Background System Architecture Evaluation Conclusion
MRS Transformation
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1
Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8
Output toconsole/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
![Page 20: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/20.jpg)
Introduction Background System Architecture Evaluation Conclusion
WHO
_like_v_1
named("John")
proper_q
rstr/h
arg1/neq
named("Mary")
proper_q
rstr/h
arg2/neq_like_v_1
person
which_q
rstr/h
arg1/neq
named("Mary")
proper_q
rstr/h
arg2/neq
Figure: “John likes Mary” → “Who likes Mary?”
![Page 21: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/21.jpg)
Introduction Background System Architecture Evaluation Conclusion
WHERE_sing_v_1
named("Mary")
proper_q
rstr/h
arg1/neq
_on_p
named("Broadway")
proper_q
rstr/h
arg2/neq
arg1/eq_sing_v_1
named("Mary")
proper_q
rstr/h
arg1/neq
loc_nonsp
place_n
which_q
rstr/h
arg2/neq
arg1/eq
Figure: “Mary sings on Broadway.” → “Where does Mary sing?”
![Page 22: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/22.jpg)
Introduction Background System Architecture Evaluation Conclusion
WHEN_sing_v_1
named("Mary")
proper_q
rstr/h
arg1/neq
at_p_temp
numbered_hour("10")
def_implict_q
rstr/h
arg2/neq
arg1/eq_sing_v_1
named("Mary")
proper_q
rstr/h
arg1/neq
loc_nonsp
time
which_q
rstr/h
arg2/neq
arg1/eq
Figure: “Mary sings at 10.” → “When does Mary sing?”
![Page 23: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/23.jpg)
Introduction Background System Architecture Evaluation Conclusion
WHY_fight_v_1
named("John")
proper_q
rstr/h
arg1/neq
for_p
named("Mary")
proper_q
rstr/h
arg2/neq
arg1/eq
_fight_v_1
named("John")
proper_q
rstr/h
arg1/neq
for_p
reason_q
which_q
rstr/h
arg2/neq
arg1/eq
Figure: “John fights for Mary.” → “Why does John fight?”
![Page 24: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/24.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 25: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/25.jpg)
Introduction Background System Architecture Evaluation Conclusion
Why?Question Transformation does not Work Well without Sentence Simplification
Input Sentence:ASC takes a character as input, and returns theinteger giving the ASCII code of the input character.Desired question:(a) What does ASC take as input?(b) What does ASC return?
Actual questions that could have been generated from mrstransformation:(c) What does ASC take as input and returns theinteger giving the ASCII code of the input character?(d) ASC takes a character as input and returns whatgiving the ASCII code of the input character?
![Page 26: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/26.jpg)
Introduction Background System Architecture Evaluation Conclusion
Why?Question Transformation does not Work Well without Sentence Simplification
Input Sentence:ASC takes a character as input, and returns theinteger giving the ASCII code of the input character.Desired question:(a) What does ASC take as input?(b) What does ASC return?
Actual questions that could have been generated from mrstransformation:(c) What does ASC take as input and returns theinteger giving the ASCII code of the input character?(d) ASC takes a character as input and returns whatgiving the ASCII code of the input character?
![Page 27: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/27.jpg)
Introduction Background System Architecture Evaluation Conclusion
MRS DecompositionComplex Sentences -> Simple Sentences
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1
Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8
Output toconsole/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
![Page 28: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/28.jpg)
Introduction Background System Architecture Evaluation Conclusion
Coordination Decomposer“John likes cats very much but hates dogs a lot.”
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
very+much a 1
arg1/eql-hndl/heq
l-index/neq
named(”John”)
proper q
rstr/h
hate v 1
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eqr-hndl/heq
r-index/neq
arg1/neq arg1/neq
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
named(”John”)
proper q
rstr/h
arg1/neq
very+much a 1
arg1/eq
hate v 1
named(”John”)
proper q
rstr/h
arg1/neq
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eq
1
![Page 29: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/29.jpg)
Introduction Background System Architecture Evaluation Conclusion
Coordination Decomposerleft: “John likes cats very much.“ right: “John hates dogs a lot.”
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
very+much a 1
arg1/eql-hndl/heq
l-index/neq
named(”John”)
proper q
rstr/h
hate v 1
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eqr-hndl/heq
r-index/neq
arg1/neq arg1/neq
but c
like v 1
cat n 1
udef q
rstr/h
arg2/neq
named(”John”)
proper q
rstr/h
arg1/neq
very+much a 1
arg1/eq
hate v 1
named(”John”)
proper q
rstr/h
arg1/neq
dog n 1
udef q
rstr/h
arg2/neq
a+lot a 1
arg1/eq
1
![Page 30: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/30.jpg)
Introduction Background System Architecture Evaluation Conclusion
Subclause Decomposeridentifies the verb, extracts its arguments and reconstructs MRS
be v id
named(”Bart”)
proper q
rstr/h
arg1/neq
cat n 1
the q
rstr/h
chase v 1
dog n 1
the q
rstr/h
arg2/neqarg1/eqarg2/neq
be v id
named(”Bart”)
proper q
rstr/h
arg1/neq
cat n 1
the q
rstr/h
arg2/neq
be v id
cat n 1
the q
rstr/h
chase v 1
dog n 1
the q
rstr/h
arg2/neqarg1/neqarg2/neq
(a): Bart is the cat that chases the dog.
(b): Bart is the cat. (c): The cat chases the dog.
decompose(({ chase v 1},{})decompose(({ be v id},{},keepEQ = 0)
1
![Page 31: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/31.jpg)
Introduction Background System Architecture Evaluation Conclusion
Connected Dependency MRS GraphConnected DMRS GraphA Connected DMRS Graph is a tuple G = (N,E , L, Spre , Spost) of:a set N, whose elements are called nodes;a set E of connected pairs of vertices, called edges;a function L that returns the associated label for edges in E ;a set Spre of pre-slash labels and a set Spost of post-slash labels.Specifically,N is a set of all Elementary Predications (eps) defined in a grammar;Spre contains all pre-slash labels, namely {arg*, rstr, l-index, r-index, l-hndl, r-hndl, null};Spost contains all post-slash labels, namely {eq, neq, h, heq, null};L is defined as: L(x , y) = [pre/post, . . .]. For every node x , y ∈ N,L returns a list of pairs pre/post that pre ∈ Spre , post ∈ Spost . Ifpre 6= null, then the edge between (x , y) is directed: x is the governer,y is the dependant; otherwise the edge between x and y is not directed.If post = null, then y = null, x has no dependant by a pre relation.
![Page 32: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/32.jpg)
Introduction Background System Architecture Evaluation Conclusion
Generic Decomposing Algorithmfunction decompose(rEPS, eEPS, relaxEQ = 1, keepEQ = 1)
parameters:rEPS: a set of eps for which we want to find related eps.eEPS: a set of exception eps.relaxEQ: a boolean value of whether to relax the post-slash value from eqto neq for verbs and prepositions (optional, default:1).keepEQ: a boolean value of whether to keep verbs and prepositions with apost-slash eq value(optional, default:1).returns:a set of eps that are related to rEPS;; assuming concurrent modification of a set is permitted in a for loopaEPS ← the set of all eps in the dmrs graphretEPS ← ∅ ;; initialize an empty setfor tEP ∈ rEPS and tEP /∈ eEPS do
for ep ∈ aEPS and ep /∈ eEPS and ep /∈ rEPS do
pre/post← L(tEP, ep) ;; ep is the dependant of tEPif pre 6= null then ;; ep exists
if relaxEQ and post = eq and (tEP is a verb ep or (tEP is a prepo-sition ep and pre = arg2)) then
assign ep a new label and change its qeq relation accordinglyend ifretEPS.add(ep)aEPS.remove(ep)
end if
pre/post← L(ep, tEP ) ;; ep is the governor of tEPif pre 6= null then ;; ep exists
if keepEQ = 0 and ep is a (verb ep or preposition ep) and post = eqand ep has no empty arg* then
continue ;; continue the loop without going further belowend ifretEPS.add(ep)aEPS.remove(ep)
end if
end forend for
if retEPS 6= ∅ thenreturn rEPS ∪ decompose(retEPS, eEPS, relaxEQ = 0) ;; the unionof two
elsereturn rEPS
end if
1
![Page 33: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/33.jpg)
Introduction Background System Architecture Evaluation Conclusion
English Sentence Structureand corresponding decomposers
English Sentence Structure
Complex
dependent clause +independent clause
Subordinate Clause
Causal | Non-causalRelative Clause
Compound
coordinationof sentences
Simple
independent &simple clause
Coordinationof phrases
Apposition Others
CoordinationSubclauseSubordinateWhy Apposition
Decomposer Pool
DecomposedSentence
![Page 34: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/34.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 35: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/35.jpg)
Introduction Background System Architecture Evaluation Conclusion
The ProblemWill the wedding be held next Monday?
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1
Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8
Output toconsole/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
Unranked Realizations from LKB for“Will the wedding be held next Monday?”
Next Monday the wedding will be held?Next Monday will the wedding be held?Next Monday, the wedding will be held?Next Monday, will the wedding be held?The wedding will be held next Monday?Will the wedding be held next Monday?
![Page 36: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/36.jpg)
Introduction Background System Architecture Evaluation Conclusion
Question RankingWill the wedding be held next Monday?
MaxEnt Model for Declaratives
4.31 The wedding will be held next Monday?1.63 Will the wedding be held next Monday?1.35 Next Monday the wedding will be held?1.14 Will the wedding be held next Monday?0.77 Next Monday, the wedding will be held?0.51 Next Monday will the wedding be held?0.29 Next Monday, will the wedding be held?
Language Model for Interrogatives
1.97 Next Monday will the wedding be held?1.97 Will the wedding be held next Monday?1.97 Will the wedding be held next Monday?1.38 Next Monday, will the wedding be held?1.01 The wedding will be held next Monday?0.95 Next Monday the wedding will be held?0.75 Next Monday, the wedding will be held?
Combined Scores
1.78 Will the wedding be held next Monday?1.64 The wedding will be held next Monday?1.44 Will the wedding be held next Monday?1.11 Next Monday the wedding will be held?0.81 Next Monday will the wedding be held?0.76 Next Monday, the wedding will be held?0.48 Next Monday, will the wedding be held?
![Page 37: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/37.jpg)
Introduction Background System Architecture Evaluation Conclusion
Review of System Architecture
MRSXML
Plain text
Termextraction
FSCconstruction
Parsingwith PET
2
3
1Generationwith LKB
Outputselection
5
6
7
MRSTransformationMRS Decomposition
8Output to
console/XML
FSCXML
Apposition Decomposer
Coordination Decomposer
Subclause Decomposer
Subordinate Decomposer
Why Decomposer
MRSXML
4
![Page 38: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/38.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 39: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/39.jpg)
Introduction Background System Architecture Evaluation Conclusion
QGSTEC2010The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010
Task B: QG from Sentences.Participants are given one complete sentence from which theirsystem must generate questions.1. Relevance. Questions should be relevant to the input
sentence.2. Question type. Questions should be of the specified target
question type.3. Syntactic correctness and fluency. The syntactic
correctness is rated to ensure systems can generate sensibleoutput.
4. Ambiguity. The question should make sense when askedmore or less out of the blue.
5. Variety. Pairs of questions in answer to a single input areevaluated on how different they are from each other.
![Page 40: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/40.jpg)
Introduction Background System Architecture Evaluation Conclusion
QGSTEC2010The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010
Task B: QG from Sentences.Participants are given one complete sentence from which theirsystem must generate questions.1. Relevance. Questions should be relevant to the input
sentence.2. Question type. Questions should be of the specified target
question type.3. Syntactic correctness and fluency. The syntactic
correctness is rated to ensure systems can generate sensibleoutput.
4. Ambiguity. The question should make sense when askedmore or less out of the blue.
5. Variety. Pairs of questions in answer to a single input areevaluated on how different they are from each other.
![Page 41: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/41.jpg)
Introduction Background System Architecture Evaluation Conclusion
Test Set
• 360 questions were required to be generated from 90 sentences• 8 question types: yes/no, which, what, when, how many,where, why and who.
Wikipedia OpenLearn YahooAnswers Allsentence count 27 28 35 90average length 22.11 20.43 15.97 19.20question count 120 120 120 360
![Page 42: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/42.jpg)
Introduction Background System Architecture Evaluation Conclusion
Participants
• Lethbridge, syntax-based, University of Lethbridge, Canada• MrsQG, semantics-based, Saarland University, Germany• JUQGG, rule-based, Jadavpur University, India.• WLV, syntax-based, University of Wolverhampton, UK
![Page 43: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/43.jpg)
Introduction Background System Architecture Evaluation Conclusion
Evaluation Grades
Relevance Question Type
Correctness Ambiguity Variety0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Results per criterion without penalty on missing questions
WLVMrsQGJUQGGLethbridgeWorst
![Page 44: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/44.jpg)
Introduction Background System Architecture Evaluation Conclusion
Generation Coverage
coverage
Page 1
MrsQG WLV JUQGG Lethbridge
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Coverages on input and output(generating 360 questions from 90 sentences)
sentencesquestions
![Page 45: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/45.jpg)
Introduction Background System Architecture Evaluation Conclusion
Evaluation Gradeswith penalty on missing questions
Relevance Question Type
Correctness Ambiguity Variety0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Results per criterion with penalty on missing questions
MrsQGWLVJUQGGLethbridgeWorst
![Page 46: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/46.jpg)
Introduction Background System Architecture Evaluation Conclusion
Evaluation Grades per Question Type
Relevance Question Type Correctness Ambiguity Variety0
0.5
1
1.5
2
2.5
3
3.5
4
Performance of MrsQG per Question Type
yes/no(28)which(42)what(116)when(36)how many(44)where(28)why(30)who(30)worst(354)
![Page 47: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/47.jpg)
Introduction Background System Architecture Evaluation Conclusion
OutlineIntroduction
DefinitionUsageTemplate/Syntax/Semantics-based Approaches
BackgroundMRS/ERG/PET/LKB
System ArchitectureOverviewMRS Transformation for Simple SentencesMRS Decomposition for Complex SentencesQuestion Reranking
EvaluationQGSTEC 2010
![Page 48: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/48.jpg)
Introduction Background System Architecture Evaluation Conclusion
Back to our one-line abstractJohn plays football –(1)–> play(John, football) –(2)–> play(who, which sports) –(3)–>
Who plays which sports?
Natural LanguageText
Natural LanguageQuestions
Symbol Representation
for Text
Symbol Representationfor Questions
NLU NLG
Question Generation
Simplification
Ranking
Transformation
![Page 49: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/49.jpg)
Introduction Background System Architecture Evaluation Conclusion
Conclusion
• semantics-based (easy in theory, difficult in practice)• multi-linguality• cross-domain
• deep grammar (worry less, wait more)• generation <-> grammaticality• heavy machinery
![Page 50: QuestionGenerationwithMinimalRecursion Semanticsxuchen/paper/MrsQG_Slides_Colloquium.pdfIntroduction Background System Architecture EvaluationConclusion CoordinationDecomposer “John](https://reader033.vdocuments.us/reader033/viewer/2022060512/5f2b3fe4e854b6756e7425fb/html5/thumbnails/50.jpg)
Introduction Background System Architecture Evaluation Conclusion
Demo?
and never forgetFourier Transform:
Xk =N−1∑n=0
xne−i2πk nN