grammatical processing with lfg and xle ron kaplan arda symposium, august 2004 advanced question...
Post on 20-Dec-2015
214 Views
Preview:
TRANSCRIPT
Grammatical processing withGrammatical processing with LFG and XLE LFG and XLE
Ron KaplanRon Kaplan
ARDA Symposium, August 2004ARDA Symposium, August 2004
Advanced QUestionAnswering for INTelligence
F-structure Conceptual semantics
KR
XLE/LFG Parsing KR MappingTarget
KRRText
Text
Sources
Question
Assertions
Query
Text to user
ComposedF-StructureTemplates
AnswersExplanationsSubqueries
XLE/LFGGeneration
MMatch M
Layered Architecture for Question AnsweringLayered Architecture for Question Answering
F-structure Conceptual semantics
KR
XLE/LFG Parsing KR MappingTarget
KRRText
Text
Sources
Question
Assertions
Query
Text to user
ComposedF-StructureTemplates
AnswersExplanationsSubqueries
XLE/LFGGeneration
Layered Architecture for Question AnsweringLayered Architecture for Question Answering
Layered Architecture for Question AnsweringLayered Architecture for Question Answering
F-structure Conceptual semantics
KR
XLE/LFG Parsing KR MappingTarget
KRRText
Sources
Question
Assertions
Query
TheoriesLexical Functional GrammarAmbiguity managementGlue Semantics
ResourcesEnglish GrammarGlue lexiconKR mapping
InfrastructureXLEMaxEnt models Linear deductionTerm rewriting
Deep analysis matters…Deep analysis matters… if you care about the answerif you care about the answer
Example:
A delegation led by Vice President Philips, head of the chemical division, flew to Chicago a week after the incident.
Question: Who flew to Chicago?
Candidate answers:
division closest nounhead next closestV.P. Philips next
shallow but wrong
delegation furthest away butSubject of flew
deep and right
“grammatical function”
F-structure: localizes argumentsF-structure: localizes argumentsWas John pleased? “John was easy to please” Yes “John was eager to please” Unknown
PRED please(SUBJ, OBJ)SUBJ someoneOBJ John
PRED easy(SUBJ, COMP)SUBJ John
COMP
PRED please(SUBJ, OBJ)SUBJ JohnOBJ someone
PRED eager(SUBJ, COMP)SUBJ John
COMP
“lexical dependency”
TopicsTopics
Basic LFG architecture Ambiguity management in XLE Pargram project: Large scale grammars Robustness Stochastic disambiguation [Shallow markup] [Semantic interpretation]
Focus on the language end, not knowledge
“Tony decided to go.”
The Language Mapping: LFG & XLEThe Language Mapping: LFG & XLE
Functionalstructures
Parse
LFGGrammar
NamedEntities
Generate
XLE
English, German, etc.
Sentence
Know
ledge
XLE: Efficient ambiguitymanagement
TokensMorphology
StochasticModel
Why deep analysis is difficultWhy deep analysis is difficult Languages are hard to describe
– Meaning depends on complex properties of words and sequences
– Different languages rely on different properties
– Errors and disfluencies
Languages are hard to compute– Expensive to recognize complex patterns– Sentences are ambiguous– Ambiguities multiply: explosion in time and space
EnglishGroup, order
JapaneseGroup, mark
The small children are chasing the dog.
P
gaSbj
S
NP
NAdj
NP
tiisaismall
kodomotatichildren
N
inudog
V
oikaketeiruare chasing
oObj
P
Different patterns code same meaningDifferent patterns code same meaning
S
NP
NAdjDet
V’ NP
the small
Aux
children
Det
the
N
dogare
V
chasing
EnglishGroup, order
JapaneseGroup, mark
The small children are chasing the dog.
P
gaSbj
S
NP
NAdj
NP
tiisaismall
kodomotatichildren
N
inudog
V
oikaketeiruare chasing
oObj
P
Different patterns code same meaningDifferent patterns code same meaning
S
NP
NAdjDet
V’ NP
the small
Aux
children
Det
the
N
dogare
V
chasing
WarlpiriMark only
S
NP
N
NP
witajarrarlusmall-Sbj
malikidog-Obj
N
kurdujarrarluchildren-Sbj
V
wajilipinyichase
Aux
kapalaPresent
NP
A chase(small(children), dog)
Pred ‘chase<Subj, Obj>’
Subj
Obj
Tense Present
PredMod
childrensmall
Pred dog
LFG theory: minor adjustments on universal theme
LFG architectureLFG architecture
C(onstituent)-structures and F(unctional)-structures
NP
John
VP
NP
Mary
V
likes
S
Formal encoding of grammatical relations
Formal encoding of order and grouping
ModularityNearly-decomposable
SUBJPRED ‘John’NUM SG
TENSE PRESENT
PRED ‘Mary’NUM SG
OBJ
PRED ‘like<SUBJ,OBJ>’
related by a piecewise correspondence
LFG LFG grammargrammar
Rules
S NP VP( SUBJ)= =
Lexical entries
N John
( PRED)=‘John’
( NUM)=SG
V likes
( PRED)=‘like<SUBJ, OBJ>’
( SUBJ NUM)=SG
(↑ SUBJ PERS)=3
Context-free rules define valid c-structures (trees). Annotations on rules give constraints that corresponding f-
structures must satisfy. Satisfiability of constraints determines grammaticality. F-structure is solution for constraints (if satisfied).
VP V (NP)= ( OBJ)=
NP (Det) N = =
Rules as well-formedness conditionsRules as well-formedness conditions
S NP( SUBJ)=
VP=
S
NP VPSUBJ [ ]
A tree containing S over NP - VP is OK if F-unit corresponding to NP node is SUBJ of f-unit corresponding to S node The same f-unit corresponds to both S and VP nodes.
If * denotes a particular daughter: : f-structure of mother (M(*)) : f-structure of daughter (*)
vv
v be the f-structure of the VP
v
ss
s
s be the f-structure of the NP
ff
Let f be the (unknown) f-structure of the S
f
(f SUBJ NUM)=PL and (f SUBJ NUM)=SG
=> SG=PL => FALSE
NP( SUBJ)=
walks( SUBJ NUM)=SG
S VP =
they( NUM)=PL
Inconsistent equations = UngrammaticalInconsistent equations = Ungrammatical
What’s wrong with “They walks” ?
(f SUBJ) = s and (s NUM)=PL => (f SUBJ NUM)=PL
Then (substituting equals for equals):
f = v and (v SUBJ NUM)=SG => (f SUBJ NUM)=SG
If a valid inference chain yields FALSE, the premises are unsatisfiable, no f-structure.
S
NP VP
walksthey
English and JapaneseEnglish and Japanese
S V =
NP*( ( GF))=
Japanese: Any number of NP’s before Verb Particle on each defines its grammatical function
ga: ( GF)=SUBJ
o: ( GF)=OBJ
English: One NP before verb, one after: Subject and Object
S V =
NP( SUBJ)=
NP( OBJ)=
Warlpiri: Discontinuous constituentsWarlpiri: Discontinuous constituents
witajarrarlusmall-Sbj
malikidog-Obj
kurdujarrarluchildren-Sbj
wajilipinyichase
kapalaPresent
S
NP
N
NP
N
VAux NP
A
PRED ‘chase<Subj, Obj>’
SUBJ
OBJ
TENSE Present
PREDMOD
childrensmall
PRED dog
Like Japanese: Any number of NP’s Particle on each defines its grammatical function
Unlike Japanese, head Noun is optional in NP
NP A* ( MOD)
N =
rlu: ( GF)=SUBJki: ( GF)=OBJ
S … NP* …( ( GF))=
English: Discontinuity in questionsEnglish: Discontinuity in questionsWho did Mary see?Who did Bill think Mary saw?Who did Bill think saw Mary?
Who is understood as subject/object of distant verb.Uncertainty: which function of which verb?
PRED see<SUBJ,OBJ>TENSE pastSUBJ MaryOBJ
Mary saw
S’
NP S
Aux NP V S
NP V
Who
did Bill think
Q WhoTENSE pastPRED think<SUBJ, COMP>
COMP
S’ → NP S(↑ Q)=↓ ↑=↓
(↑ COMP* SUBJ|OBJ)=↓
OBJCOMP OBJCOMP SUBJ
Summary: Lexical Functional GrammarSummary: Lexical Functional Grammar
Modular: c-structure/f-structure in correspondence Mathematically simple, computationally transparent
– Combination of Context-free grammar, Quantifier-free equality theory– Closed under composition with regular relations: finite-state morphology
Grammatical functions are universal primitives– Subject and Object expressed differently in different languages
English: Subject is first NP Japanese: Subject has ga
– But: Subject and Object behave similarly in all languages Active to Passive: Object becomes Subject English: move words Japanese: move ga
Adopted by world-wide community of linguists– Large literature: papers, (text)books, conferences; reference theory– (Relatively) easy to describe all languages– Linguists contribute to practical computation
Stable: Only minor changes in 25 years
Kaplan and Bresnan, 1982
Efficient computation withEfficient computation with LFG grammars: LFG grammars: Ambiguity Management in XLE Ambiguity Management in XLE
Computation challenge: Pervasive ambiguityComputation challenge: Pervasive ambiguity
walks Noun or Verb? untieable knot (untie)able or un(tieable)? bank river or financial?
walks Noun or Verb? untieable knot (untie)able or un(tieable)? bank river or financial?
The duck is ready to eat. Cooked or hungry?
The duck is ready to eat. Cooked or hungry?
Every proposer wants an award. The same award or each their own?
Every proposer wants an award. The same award or each their own?
I like Jan. |Jan|.| or |Jan.|.| (sentence end or abbreviation) I like Jan. |Jan|.| or |Jan.|.| (sentence end or abbreviation)
The sheet broke the beam. Atoms or photons?
The sheet broke the beam. Atoms or photons?
KnowledgeSemanticsSyntaxMorphologyTokenization
Coverage vs. AmbiguityCoverage vs. Ambiguity
I fell in the park.
+I know the girl in the park.
I see the girl in the park.
Ambiguity can be explosiveAmbiguity can be explosiveIf alternatives multiply within or across components…
Tokenize
Morphology
Syntax
Sem
antics
Know
ledge
Computational consequences of ambiguityComputational consequences of ambiguity
Serious problem for computational systems– Broad coverage, hand written grammars frequently produce
thousands of analyses, sometimes millions– Machine learned grammars easily produce hundreds of
thousands of analyses if allowed to parse to completion
Three approaches to ambiguity management:– Prune: block unlikely analysis paths early– Procrastinate: do not expand alternative analysis paths until
something else requires them» Also known as underspecification
– Manage: compact representation and computation of all possible analyses
Pruning Pruning Premature Disambiguation⇒ Premature Disambiguation⇒ Conventional approach: Use heuristics to kill as soon as possible
Tokenize
Morphology
Syntax
Sem
antics
Know
ledge
XX
X
Statistics
X
Oops: Strong constraints may reject the so-far-best (= only) option
Fast computation, wrong result
X
Procrastination: Passing the BuckProcrastination: Passing the Buck
Chunk parsing as an example:– Collect noun groups, verb groups, PP groups– Leave it to later processing to put these together– Some combinations are nonsense
Later processing must either:– Call (another) parser to check constraints– Have its own model of constraints (= grammar)– Solve constraints that chunker includes with output
Computational Complexity of LFG Computational Complexity of LFG
LFG is simple combination of two simple theories– Context-free grammars for trees– Quantifier free theory of equality for f-structures
Both theories are easy to compute– Cubic CFG Parsing– Linear equation solving
Combination is difficult: Parsing problem is NP Complete– Exponential/intractible in the worst case
(but computable, unlike some other linguistic theories
– Can we avoid the worst case?
Some syntactic dependenciesSome syntactic dependencies
Local dependencies: These dogs *This dogs (agreement)
Nested dependencies: The dogs [in the park] bark (agreement)
Cross-serial dependencies:
Jan Piet Marie zag helpen zwemmen (predicate/argument
map)
See(Jan, help(Piet, swim(Marie)))
Long distance dependencies:
The girl who John says that Bob believes … likes Henry left.
Left(girl) Says(John, believes(Bob, (…likes(girl, Henry))))
ExponentialIntractable!
Expressiveness vs. complexityExpressiveness vs. complexity
But languages have mostly local and nested dependencies... so (mostly) cubic performance should be possible.
Linear
Cubic
The Chomsky Hierarchy
n n is length of sentenceis length of sentence
Type DependencyComputational
Complexity
Regular Local O(n)
Context-free Nested O(n3)
Context-sensitive
Cross-serial and
Long DistanceO(2n)
NP Complete ProblemsNP Complete Problems Problems that can be solved by a Nondeterministic Turing Machine
in Polynomial time General characterization: Generate and test
– Lots of candidate solutions that need to be verified for correctness– Every candidate is easy to confirm or disconfirm
n elementsn elements
22nn candidates candidates
NonNondeterministic TM has an oracledeterministic TM has an oraclethat provides only the right candidatesthat provides only the right candidatesto test, doesn’t search.to test, doesn’t search.
Deterministic TM doesn’t have oracle,Deterministic TM doesn’t have oracle,must test all (exponentially many) candidates. must test all (exponentially many) candidates.
Polynomial search problemsPolynomial search problems Subparts of a candidate are independent of other parts: outcome is
not influenced by other parts (context free) The same independent subparts appear in many candidates We can (easily) determine that this is the case Consequence: test subparts independent of context, share results
Why is LFG parsing NP Complete?Why is LFG parsing NP Complete?
Classic generate-and-test search problem
Exponentially many tree-candidates– CFG chart parser quickly produces packed representation
of all trees– CFG can be exponentially ambiguous– Each tree must be tested for f-structure satisfiability
Exponentially many exponential problems
Boolean combinations of per-tree constraintsEnglish base verbs: Not 3rd singular
( SUBJ NUM)SG ( SUBJ PERS)3
Disjunction!
XLE Ambiguity Management: The intuitionXLE Ambiguity Management: The intuition
The sheep saw the fish.How many sheep?
How many fish?
Packed representation is a “free choice” system– Encodes all dependencies without loss of information– Common items represented, computed once– Key to practical efficiency
The sheep-sg saw the fish-sg.The sheep-pl saw the fish-sg.The sheep-sg saw the fish-pl.The sheep-pl saw the fish-pl.
Options multiplied out
In principle, a verb might require agreement of Subject and Object: Have to check it out.
The sheep saw the fishsgpl
sgpl
Options packed
But English doesn’t do that: Subparts are independent
… but it’s wrongDoesn’t encode all dependencies, choices are not free.
Dependent choicesDependent choices
Das Mädchen-nom sah die Katze-nomDas Mädchen-nom sah die Katze-accDas Mädchen-acc sah die Katze-nomDas Mädchen-acc sah die Katze-acc
Das Mädchen sah die Katzenomacc
nomacc
The girl saw the cat
Again, packing avoids duplication
badThe girl saw the catThe cat saw the girl bad
Who do you want to succeed? I want to succeed John want intrans, succeed trans I want John to succeed want trans, succeed intrans
Solution: Label dependent choicesSolution: Label dependent choices
Das Mädchen-nom sah die Katze-nomDas Mädchen-nom sah die Katze-accDas Mädchen-acc sah die Katze-nomDas Mädchen-acc sah die Katze-acc
badThe girl saw the catThe cat saw the girl bad
• Label each choice with distinct Boolean variables p, q, etc.• Record acceptable combinations as a Boolean expression • Each analysis corresponds to a satisfying truth-value assignment
(free choice from the true lines of ’s truth table)
Das Mädchen sah die Katze p:nom
p:acc
q:nom
q:acc
(pq)
(pq) =
Boolean SatisfiabilityBoolean Satisfiability
Produces simple conjunctions of literal propositions (“facts”--equations)
Easy checks for satisfiability
If ad FALSE, replace any conjunction with a and d by FALSE.
Blow-up of disjunctive structure before fact processing
Individual facts are replicated (and re-processed): Exponential
(a b) x (c d)
(a x c) (a x d) (b x c) (b x d)
Can solve Boolean formulas by multiplying out: Disjunctive Normal Form
Alternative: “Contexted” normal formAlternative: “Contexted” normal form
Produce a flat conjunction of contexted facts
(a b) x (c d) pa pb x qc qd
context fact
– Each fact is labeled with its position in the disjunctive structure– Boolean hierarchy discarded
(a b) x (c d) pa pb x qc qd
a b dc
xp qp q
xp a p b q c q d
Produce a flat conjunction of contexted facts
No blow-up, no duplicates– Each fact appears and can be processed once– Claims:
» Checks for satisfiability still easy» Facts can be processed first, disjunctions deferred
Alternative: “Contexted” normal formAlternative: “Contexted” normal form
Conversion to logically equivalent contexted form
Lemma: iff p p (p a new Boolean variable)
Proof:(If) If is true, let p be true, in which case p p is true. (Only if) If p is true, then is true, in which case is true.
A sound and complete methodA sound and complete method
Ambiguity-enabled inference (by trivial logic):
If is a rule of inference, then so is [C1 ] [C2 ] [(C1C2) ]
Maxwell & Kaplan, 1987, 1991
E.g. Substition of equals for equals: x=y x/y is a rule of inference
Therefore: (C1 x=y) (C2 ) (C1 C2 x/y)
Valid for any theory
Test for satisfiabilityTest for satisfiability
Perform all fact-inferences, conjoining contexts If infer FALSE, add context to nogoods Solve conjunction of nogoods
– Boolean satisfiability: exponential in nogood context-Booleans– Independent facts: no FALSE, no nogoods
Implicitly notices independence/context-freeness
E.g. R SG=PL ⇒ R → FALSE.
R is called a “nogood” context.
Suppose R FALSE is deduced from a contexted formula . Then is satisfiable only if R.
Example 1Example 1
“They walk”– No disjunction, all facts are in the default “True” context– No change to inference
T(f SUBJ NUM)=SG T(f SUBJ NUM)=SG T SG=SG
reduces to: (f SUBJ NUM)=SG (f SUBJ NUM)=SG SG=SG
“They walks”– No disjunction, all facts still in the default “True” context– No change to inference:
T(f SUBJ NUM)=PL T(f SUBJ NUM)=SG TPL=SG T→FALSE
Satisfiable iff ¬T, so unsatisfiable
Examples 2Examples 2
“The sheep walks”– Disjunction of NUM feature from sheep
(f SUBJ NUM)=SG (f SUBJ NUM)=PL
– Contexted facts: p(f SUBJ NUM)=SG p(f SUBJ NUM)=PL
(f SUBJ NUM)=SG (from walks)
– Inferences: p(f SUBJ NUM)=SG (f SUBJ NUM)=SG p SG=SG
p(f SUBJ NUM)=PL (f SUBJ NUM)=SG p PL=SG p FALSE
p FALSE is true iff p is false iff p is True. Conclusion: Sentence is grammatical in context p: Only 1 sheep
Contexts and packing: Index by facts Contexts and packing: Index by facts
The sheep saw the fish.
Contexted unification concatenation, when choices don’t interact.
NUM p SGp PL
SUBJ
NUM q SGq PL
OBJ
NUM p SGp PL
SUBJ
NUM q SGq PL
OBJ
Compare: DNF unificationCompare: DNF unification
The sheep saw the fish.
DNF cross-product of alternatives: Exponential
[ SUBJ [NUM SG]]
[SUBJ [NUM PL ]]
[ OBJ [NUM SG]]
[ OBJ [NUM PL ]]
SUBJ [ NUM SG]OBJ [ NUM SG]
SUBJ [ NUM SG]OBJ [ NUM PL]
SUBJ [ NUM PL]OBJ [ NUM SG]
SUBJ [ NUM PL]OBJ [ NUM PL]
The XLE wagerThe XLE wager (for real sentences of real languages)(for real sentences of real languages)
Alternatives from distant choice-sets can be freely chosen without affecting satisfiability– FALSE is unlikely to appear
Contexted method optimizes for independence– No FALSE no nogoods nothing to solve.
Bet: Worst case 2n reduces to k2m where m << n
Ambiguity-enabled inference:Ambiguity-enabled inference:Choice-logic common to all modulesChoice-logic common to all modules
If is a rule of inference,then so is C1 C2 (C1C2)
1. Substitution of equals for equals (e.g. for LFG syntax) x=y x/y
Therefore: C1x=y C2 (C1C2) x/y
Ambiguity-enabled components propagate choices, can defer choosing, enumerating
2. Reasoning Cause(x,y) Prevent(y,z) Prevent(x,z)
Therefore: C1Cause(x,y) C2Prevent(y,z) (C1C2)Prevent(x,z)
3. Log-linear disambiguation Prop1(x) Prop2(x) Count(Featuren)
Therefore: C1 Prop1(x) C2 Prop2(x) (C1C2) Count(Featuren)
Summary: Contexted constraint satisfactionSummary: Contexted constraint satisfaction
Packed– facts not duplicated– facts not hidden in Boolean structure
Efficient– deductions not duplicated– fast fact processing (e.g. equality) can prune slow disjunctive processing– optimized for independence
General and simple– applies to any deductive system, uniform across modules – not limited to special-case disjunctions– mathematically trivial
Compositional free-choice system– enumeration of (exponentially many?) valid solutions deferred across
module boundaries– enables backtrack-free, linear-time, on-demand enumeration– enables packed refinement by cross-module constraints: new nogoods
The remaining exponentialThe remaining exponential
Contexted constraint satisfaction (typically) avoids the Boolean explosion in solving f-structure constraints for single trees
How can we suppress tree enumeration? (and still determine satisfiability)
Ordering strategy: Easy things firstOrdering strategy: Easy things first
Do all c-structure before any f-structure processing– Chart is a free choice representation, guarantees valid trees
Only produce/solve f-structure constraints for constituents in complete, well-formed trees
[NB: Interleaved, bottom-up pruning is a bad idea] Bets on inconsistency, not independence
Asking the right questionAsking the right question
How can we make it faster?– More efficient unifier: undoable operations, better
indexing, clever data structures, compiling.– Reordering for more effective pruning.
Why not cubic?– Intuitively, the problem isn’t that hard.– GPSG: Natural language is nearly context free.– Surely for context-free equivalent grammars!
No f-structure filtering, no nogoods...No f-structure filtering, no nogoods... but still explosive but still explosive
LFG grammar for a context-free language:
a( A)=+
S S S( L)=
S( R)=
S Sa aa
S
S
S
S
Sa
S
S
S
Chart:Packed trees
S
S S
S
a aa
S
SS
a
F-structuresenumerate trees
R [A +]
L [A +]
L [A +]
R [A +]RL
Disjunctive lazy copyDisjunctive lazy copy
• Pack functional information from alternative local subtrees.
• Unpack/copy to higher consumers only on demand.
S
Automatically takes advantage of context freeness, without grammar analysis or compilation
SS
SS
S S
L
R
p f1q f2r f3
p f6q f5r f4
( L)= on Sdoesn’t access
internal features
41
63
2 5
The XLE wagerThe XLE wager
Most feature dependencies are restricted to local subtrees – mother/daughter/sister interactions– maybe a grandmother now and then– very rarely span an unbounded distance
Optimize for local case– bounded computation per subtree gives cubic curve– graceful degradation with non-local interactions
… but still correct
Packing Equalities in F-structurePacking Equalities in F-structure
NP(SUBJ)=
NP(NUM)=sg
NPV
NP(SUBJ)=
Adj NP=
S
VP
V
visiting relatives
visiting relatives (NUM)=pl
is(SUBJ NUM)=sg
Adj
boring
A1 A2
T (SUBJ NUM)=sg
A1 (SUBJ NUM)=sg
A2 (SUBJ NUM)=pl
T & A1 sg=sg
T & A2 sg=pl nogood(A2)
a:1 a:2
XLE Performance: HomeCentre CorpusXLE Performance: HomeCentre Corpus
0
2
4
6
8
10
12
14
0 20 40 60
Words
Time (secs)
About 1100 English sentences
0
500
1000
1500
2000
2500
3000
3500
0 20 40 60
Words
Local Subtrees
Time is ~linear in subtrees: Time is ~linear in subtrees: Nearly cubic Nearly cubic
0
2
4
6
8
10
12
14
0 1000 2000 3000 4000
Local Subtrees
Time (secs)
2.1 ms/subtree
R2=.79
French HomeCentreFrench HomeCentre
0
5
10
15
20
25
30
0 2000 4000 6000 8000
Local Subtrees
Time (secs)
3.3 ms/subtree
R2=.80
German HomeCentre German HomeCentre
0
5
10
15
20
25
30
35
0 1000 2000 3000 4000
Local Subtrees
Time (secs)
3.8 ms/subtree
R2=.44
Generation with LFG/XLEGeneration with LFG/XLE
Parse: string → c-structure → f-structure Generate: f-structure → c-structure → string Same grammar: shared development, maintenance Formal criterion: s ∈ Gen(Parse(s)) Practical criterion: don’t generate everything
– Parsing robustness → undesired strings, needless ambiguity– Use optimality marks to restrict generation grammar– Restricted (un)tokenizing transducer: don’t allow arbitrary white
space, etc.
Mathematics and ComputationMathematics and Computation
Formal properties Gen(f) is a (possibly infinite) set
– Equality is idempotent: x=y x=y ⇔ x=y∧– Longer strings with redundant equations map to same f-structure
What kind of set?Context-free language (Kaplan & Wedekind, 2000)
ComputationComputationXLE/LFG generation: Convert LFG grammar to CFG only for strings that map to f
– NP complete, ambiguity managed (as usual)– All strings in CFL are grammatical w.r.t. LFG grammar– Composition with regular relations is crucial
CFG is a packed, free-choice representation of all strings– Can use ordinary CF generation algorithms to enumerate strings– Can defer enumeration, give CFG for client to enumerate– Can apply other context-free technology
» Choose shortest string» Reduce to finite set of unpumped strings (Context free Pumping Lemma)» Choose most probable (for fluency, not grammaticality)
Generating from incomplete f-structuresGenerating from incomplete f-structures
Grammatical features can’t be read from– Back-end question-answering logic– F-structure translated from other language
Generating from a bounded underspecification of a complete f-structure is still context-free– Example: a skeleton of predicates– Proof: CFL’s are closed under union, bounded extensions produce
finite alternatives
Generation from arbitrary underspecification is undecidable– Reduces to undecidable emptiness problem (= Hilbert’s 10th)
(Dymetman, van Noord, Wedekind, Roach)
Question: What is the graph partitioning problem?– Generated Queries: “The graph partitioning problem is *”– Answer (Google): The graph partitioning problem is defined as
dividing a graph into disjoint subsets of nodes …
Question: When were the Rolling Stones formed?– Generated Queries: “The Rolling Stones were formed *”
“* formed the Rolling Stones *” – Answer (Google): Mick Jagger, Keith Richards, Brian Jones, Bill
Wyman,and Charlie Watts formed the Rolling Stones in
1962.
A (light-weight?) approach to QAA (light-weight?) approach to QA
ParseAsk GenerateQuestion F-structure Queries Search
Analyze the question, anticipate and search for possible answer phrases
Pipeline for Answer AnticipationPipeline for Answer Anticipation
Convert
GeneratorParserQuestionAnswerPhrases
Englishgrammar
Englishgrammar
Search(Google...)
Question f-structures Answer f-structures
Grammar engineering:Grammar engineering: The Parallel Grammar ProjectThe Parallel Grammar Project
Pargram projectPargram project Large-scale LFG grammars for several languages
– English, German, Japanese, French, Norwegian– Coming along: Korean, Urdu, Chinese, Arabic, Welsh, Malagasy, Danish– Intuition + Corpus: Cover real uses of language--newspapers, documents, etc.
Parallelism: test LFG universality claims– Common c- to f-structure mapping conventions
(unless typologically motivated variation)
– Similar underlying f-structures Permits shared disambiguation properties, Glue interpretation premises
– Practical: all grammars run on XLE software International consortium of world-class linguists
– PARC, Stuttgart, Fuji Xerox, Konstanz, Bergen, Copenhagen, Oxford, Dublin City University, PIEAS…
– Full week meetings, twice a year– Contributions to linguistics and comp-ling: books and papers – Each group is self-funded, self-managed
Pargram goalsPargram goals Practical
– Create grammatical resources for NL applications» translation, question answering, information retrieval, ...
– Develop discipline of grammar engineering» what tools, techniques, conventions make it easy to develop
and maintain broad-coverage grammars?» how long does it take?» how much does it cost?
Theoretical– Refine and guide LFG theory through broad coverage
of multiple languages– Refine and guide XLE algorithms and implementation
Parallel f-structures (where possible)Parallel f-structures (where possible)
……but different c-structuresbut different c-structures
Pargram grammarsPargram grammars
German
English*
French
Japanese (Korean)
#Rules
251
388
180
56
#States
3,239
13,655
3,422
368
#Disjuncts
13,294
55,725
16,938
2,012
* English allows for shallow markup: labeled bracketing, named-entities
Why Norwegian and Japanese?Why Norwegian and Japanese?
Engineering assessment: given mature system, parallel grammar specs. How hard is it?
Norwegian: best case– Well-trained LFG linguists– Users of previous Parc software– Closely related to existing Pargram languages
Japanese: worst case– One computer scientist, one traditional Japanese linguist--no LFG
experience– Typologically different language– Character sets, typographical conventions
Conclusion: not that hardFor both languages: good coverage, accuracy in ~2 person years
Engineering resultsEngineering results
Grammars and Lexicons Grammar writer’s cookbook (Butt et al., 1999)
New practical formal devices– Complex categories for efficiency NP[nom] vs. NP: ( CASE) = NOM
– Optimality marks for robustness
enlarge grammar without being overrun by peculiar analyses
– Lexical priority: merging different lexicons
Integration of off-the-shelf morphologyFrom Inxight, based on earlier PARC research, and Kyoto
Accuracy and coverageAccuracy and coverage WSJ F scores for English Pargram grammar
– Produces dependencies, not labeled trees
– Stochastic model trained on sections 2-22
– Tested on dependencies for 700 sentences in section 23
– Robustness: some output for every input
Full
74.7%
Fragments
25.3%
Best 88.5 76.7
Most probable
82.5 69
Random 78.4 67.7
(Named Entities seem to bump these by ~3%)
Riezler et al., 2002
““Meridian will pay a premium of $30.5 million to Meridian will pay a premium of $30.5 million to assume $2 billion in deposits.” assume $2 billion in deposits.”
mood(pay~0, indicative),tense(pay~0, fut),adjunct(pay~0, assume~7),obj(pay~0, premium~3),stmt_type(pay~0, declarative),subj(pay~0, Meridian~5),det_type(premium~3, indef),adjunct(premium~3, of~23),num(premium~3, sg),pers(premium~3, 3),adjunct(million~4, 30.5~28),number_type(million~4, cardinal),num(Meridian~5, sg),pers(Meridian~5, 3),obj(assume~7, $~9),stmt_type(assume~7, purpose),
subj(assume~7, pro~8),number($~9, billion~17),adjunct($~9, in~11),num($~9, pl),pers($~9, 3),adjunct_type(in~11, nominal),obj(in~11, deposit~12),num(deposit~12, pl),pers(deposit~12, 3),adjunct(billion~17, 2~19),number_type(billion~17, cardinal),number_type(2~19, cardinal),obj(of~23, $~24),number($~24, million~4),num($~24, pl),pers($~24, 3),number_type(30.5~28, cardinal))
Accuracy and coverageAccuracy and coverage Japanese Pargram grammar
– ~97% coverage on large corpora» 10,000 newspaper sentences (EDR)» 460 copier manual sentences» 9,637 customer-relations sentences
– F-scores against 200 hand-annotated sentences from newspaper corpus:
» Best: 87%» Average: 80%
Recall: Grammar constructed with ~2 person-years of effort (compare: Effort to create an annotated training corpus)
Robustness:Robustness: Some output for every input Some output for every input
Sources of BrittlenessSources of Brittleness
Vocabulary problems– Gaps in coverage, neologisms, terminology– Incorrect entries, missing frames…
Missing constructions– No theoretical guidance (or interest)
(e.g. dates, company names)
– Core constructions overlooked» Intuition and corpus both limited
Ungrammatical input– Real world text is not perfect– Sometimes it’s horrendous
Strict performance limits (XLE parameters)
Real world inputReal world input
Other weak blue-chip issues included Chevron, which went down 2 to 64 7/8 in Big Board composite trading of 1.3 million shares; Goodyear Tire & Rubber, off 1 1/2 to 46 3/4, and American Express, down 3/4 to 37 1/4. (WSJ, section 13)
``The croaker's done gone from the hook” (WSJ, section 13)
(SOLUTION 27000 20) Without tag P-248 the W7F3 fuse is located in the rear of the machine by the charge power supply (PL3 C14 item 15. (Copier repair tip)
LFG entries from Finite-State MorphologiesLFG entries from Finite-State Morphologies
Broad-coverage inflectional transducers falls → fall +Noun +Pl
fall +Verb +Pres +3sg
Mary → Mary +Prop +Giv +Fem +Sg
vienne → venir +SubjP +SG {+P1|+P3} +Verb
For listed words, transducer provides– canonical stem form– inflectional information
On-the-fly LFG entries On-the-fly LFG entries “-unknown” head-word matches unrecognized stems Grammar writer defines -unknown and affixes
-unknown N (↑ PRED)=‘%stem’ (↑ NTYPE)=common;
V (↑ PRED)=‘%stem<SUBJ,OBJ>’.
+Noun N-AFX (↑ PERS)=3.
+Pl N-AFX (↑ NUM)=pl.
+Pres V-AFX (↑ TENSE)=present
+3sg V-AFX (↑ SUBJ PERS)=3 (↑ SUBJ NUM)=sg
Pieces assembled by sublexical rules: NOUN → N N-AFX*.
VERB → V V-AFX*.
(transitive)
Guessing for unlisted wordsGuessing for unlisted words
Use FST guesser for general patterns– Capitalized words can be proper nouns
» Saakashvili → Saakashvili +Noun +Proper +Guessed
– ed words can be past tense verbs or adjectives» fumped → fump +Verb +Past +Guessed
fumped +Adj +Deverbal +Guessed
Languages with richer morphology allow better guessers
Subcategorization and Argument Mapping?Subcategorization and Argument Mapping?
Transitive, intransitive, inchoative…– Not related to inflection– Can’t be inferred from shallow data
Fill in gaps from external sources– Machine readable dictionaries– Other resources: VerbNet, WordNet, FrameNet, Cyc– Not always easy, not always reliable
» Current research
Grammatical failuresGrammatical failures
Fall-back approach First try to get a complete analysis
– Prefer standard rules, but– Allow for anticipated errors
E.g. subject/verb disagree, but interpretation is obvious
– Optimality-theory marks to prefer standard analyses
If fail, enlarge grammar, try again– Build up fragments that get complete sub-parses
(c-structure and f-structure)– Allow tokens that can’t be chunked– Link chunks and tokens in a single f-structure
Fall-back grammar for fragmentsFall-back grammar for fragments
Grammar writer specifies REPARSECAT– Alternative c-structure root if no complete parse– Allows for fragments and linking
Grammar writer specifies possible chunks– Categories (e.g. S, NP, VP but not N, V)– Looser expansions
Optimality theoryGrammar writer specifies marks to
» Prefer standard rules over anticipated errors» Prefer parse with fewest chunks» Disprefer using tokens over chunks
ExampleExample
“The the dog appears.”
Analyzed as– “token” the– sentence “the dog appears”
C-structureC-structure
F-structureF-structure
• Many chunks have useful analyses• XLE/LFG degrades to shallow parsing in worst case
Robustness summaryRobustness summary
External resources for incomplete lexical entries – Morphologies, guessers, taggers– Current work: Verbnet, Wordnet, Framenet, Cyc– Order by reliability
Fall back techniques for missing constructions– Disprefered rules– Fragment grammar
Current WSJ evaluation:– 100% coverage, ~85% full parses– F-score (esp. recall) declines for fragment parses
Brief demo Brief demo
Stochastic disambiguation:Stochastic disambiguation: When you have to choose When you have to choose
Finding the most probable parse Finding the most probable parse XLE produces many candidates
– All valid (with respect to grammar and OT marks)– Not all equally likely– Some applications are ambiguity enabled (defer selection)– … But some require a single best guess
Grammar writers have only coarse preference intuitions– Many implicit properties of words and structures with unclear
significance Appeal to probability model to choose best parse
– Assume: previous experience is a good guide for future decisions– Collect corpus of training sentences– Build probability model that optimizes for previous good results– Apply model to choose best analysis of new sentences
IssuesIssues
What kind of probability model? What kind of training data? Efficiency of training, disambiguation? Benefit vs. random choice of parse?
– Random is awful for treebank grammars– Hard LFG constraints restrict to plausible candidates
Probability modelProbability model Conventional models: stochastic branching process
– Hidden Markov models– Probabilistic Context-Free grammars
Sequence of decisions, each independent of previous decisions, each choice having a certain probability– HMM: Choose from outgoing arcs at a given state– PCFG: Choose from alternative expansions of a given category
Probability of an analysis = product of choice probabilities Efficient algorithms
– Training: forward/backward, inside/outside– Disambiguation: Viterbi
Abney 1997 and others: Not appropriate for LFG, HPSG…– Choices are not independent: Information from different CFG branches
interacts through f-structure– Relative-frequency estimator is inconsistent
Exponential models are appropriateExponential models are appropriate (aka Log-linear models) (aka Log-linear models)
Assign probabilities to representations, not to choices in a derivation
No independence assumption Arithmetic combined with human insight
– Human:» Define properties of representations that may be relevant
» Based on any computable configuration of f-structure features, trees
– Arithmetic:» Train to figure out the weight of each property
Stochastic Disambiguation in XLEStochastic Disambiguation in XLE All parses All parses Most probable Most probable
Discriminative ranking– Conditional log-linear model on c/f-structure pairs
Probability of parse x for string s, wheref is a vector of feature values for x is a vector of feature weightsZ is normalizer for all parses of s
– Discriminative estimation of from partially labeled data(Riezler et al. ACL’02)
– Combined l1-regularization and feature-selection» Avoid over-fitting, choose best features
(Riezler & Vasserman, EMNLP’04)
€
pλ (x | s) =eλ • f (x )
Z
Coarse training data for XLECoarse training data for XLE“Correct” parses are consistent with weak annotation
Considering/VBG (NP the naggings of a culture imperative), (NP-SBJ I) promptly signed/VBD up.
Sufficient for disambiguation, not for grammar induction
(S (S-ADV (NP-SBJ (-NONE- *-1)) (VP (VBG Considering) (NP (NP (DT the) (NNS naggings)) (PP (IN of) (NP (DT a) (NN culture) (NN imperative)))))) (, ,) (NP-SBJ-1 (PRP I)) (VP (ADVP-MNR (RB promptly)) (VBD signed) (PRT (RB up))) (. .))
Compare with full PTB annotation:
Classes of propertiesClasses of properties C-structure nodes and subtrees
– indicating certain attachment preferences Recursively embedded phrases
– indicating high vs. low attachment F-structure attributes
– presence of grammatical functions Atomic attribute-value pairs in f-structure
– particular feature values Left/right/ branching behavior of c-structures (Non)parallelism of coordinations in c- and f-structures Lexical elements
– tuples of head words, argument words, grammatical relations
~60,000 candidate properties, ~1000 selected
Some properties and weightsSome properties and weights0.937481 cs_embedded VPv[pass] 1-0.126697cs_embedded VPv[perf] 3-0.0204844 cs_embedded VPv[perf] 2-0.0265543 cs_right_branch-0.986274cs_conj_nonpar 5-0.536944cs_conj_nonpar 4-0.0561876 cs_conj_nonpar 30.373382 cs_label ADVPint-1.20711 cs_label ADVPvp-0.57614 cs_label AP[attr]-0.139274cs_adjacent_label DATEP PP-1.25583 cs_adjacent_label MEASUREP PPnp-0.35766 cs_adjacent_label NPadj PP-0.00651106 fs_attrs 1 OBL-COMPAR0.454177 fs_attrs 1 OBL-PART-0.180969fs_attrs 1 ADJUNCT0.285577 fs_attr_val DET-FORM the0.508962 fs_attr_val DET-FORM this0.285577 fs_attr_val DET-TYPE def0.217335 fs_attr_val DET-TYPE demon0.278342 lex_subcat achieve OBJ,SUBJ,VTYPE SUBJ,OBL-AG,PASSIVE=+0.00735123 lex_subcat acknowledge COMP-EX,SUBJ,VTYPE
EfficiencyEfficiency Properties counts
– Associated with AND/OR tree of XLE contexts (a1, b2)» Detectors may add new nodes to tree: conjoined contexts
– Shared among many parses Training
– Dynamic programming algorithm applied to AND/OR tree» Avoids unpacking of individual parses (Miyao and Tsujii HLT’02)
» Similar to inside-outside algorithm of PCFG
– Fast algorithm for choosing best properties– Can train only on sentences with relatively low-ambiguity
» Shorter, perhaps easier to annotate
– 5 hours to train over WSJ (given file of parses) Disambiguation
– Viterbi algorithm applied to Boolean tree– 5% of parse time to disambiguate– 30% gain in F-score from random-parse baseline
Integrating Shallow Mark up:Integrating Shallow Mark up:Part of speech tagsPart of speech tagsNamed entitiesNamed entitiesSyntactic bracketsSyntactic brackets
Shallow mark-up of input stringsShallow mark-up of input strings Part-of-speech tags (tagger?)
I/PRP saw/VBD her/PRP duck/VB. I/PRP saw/VBD her/PRP$ duck/NN.
Named entities (named-entity recognizer)
<person>General Mills</person> bought it. <company>General Mills</company> bought it Syntactic brackets (chunk parser?)
[NP-S I] saw [NP-O the girl with the telescope]. [NP-S I] saw [NP-O the girl] with the telescope.
HypothesisHypothesis Shallow mark-up
– Reduces ambiguity– Increases speed– Without decreasing accuracy– (Helps development)
Issues– Markup errors may eliminate correct analyses– Markup process may be slow– Markup may interfere with existing robustness mechanisms
(optimality, fragments, guessers)– Backoff may restore robustness but decrease speed in 2-
pass system
Implementation in XLEImplementation in XLEInput string
Marked up string
Tokenizer (FST)(plus POS, NE converter)
Morphology (FST)(plus POS filter)
LFG grammar(plus bracket metarule,
NE sublexical rule)
f-strc-strf-str
Input string
Tokenizer (FST)
Morphology (FST)
LFG grammar
c-str
Integration with minimal changes to existing system/grammar
Experimental Results: PARC 700Experimental Results: PARC 700
Full/All % Full
parses
Optimalsol’ns
Best
F-sc
Time
%
Unmarked 76 482/1753 82/79 65/100
Named ent 78 263/1477 86/84 60/91
POS tag 62 248/1916 76/72 40/48
Lab brk 65 158/ 774 85/79 19/31
Comparison: Shallow vs. Deep parsingComparison: Shallow vs. Deep parsingHLT, 2004HLT, 2004
Popular myth– Shallow statistical parsers are fast, robust… and useful
– Deep grammar-based parsers are slow and brittle
Is this true?Comparison on predicate-argument relations, not phrase-trees
– Needed for meaning-sensitive applications (= usefulness)
(translation, question answering…but maybe not IR)
– Collins (1999) parser: state-of-the-art, marks arguments
(for fair test, wrote special code to make relations explicit--not so easy)
– LFG/XLE with morphology, named-entities, disambiguation
– Measured time, accuracy against PARC 700 Gold Standard
– Results:» Collins is a bit times faster than LFG/XLE
» LFG/XLE makes somewhat fewer errors, provides more useful detail
XLE SystemXLE System Parser/generator for LFG grammars: multilingual Composition with finite-state transductions Careful ambiguity-management implementation
– Preserves context-free locality in equational disjunctions– Exports ambiguity-enabling interfaces
Efficient implementation of clause-conjunction (C1C2)\
Log-linear disambiguation– Appropriate for LFG representations– Ambiguity-enabled theory and implementation
Robustness: shallow in the worst-case Scales to broad-coverage grammars, long sentences Semantic interface: Glue
LFG/XLE: Current issuesLFG/XLE: Current issues Induction of LFG grammars from treebanks
– Basic work in ParGram: Dublin City University– Principles of generalization, for human extension, combination with
manual grammar DCU + PARC
Large grammars for more language typologies– E.g. verb initial: Welsh, Malagasy, Arabic
Reduce performance variance; why not linear?– Competence vs. performance: limit center embedding?– Investigate speed/accuracy trade-off
Embedding in applications: XLE as a black box– Question answering(!), Translation, Sentence condensation …– Develop, combine with other ambiguity-enabled modules
Reasoning, transfer-rewriting…
Matching for Question AnsweringMatching for Question Answering
F-structure
ParserParserQuestion
Englishgrammar
Englishgrammar
F-structure
AnswerSources
Semantics
Semantics
Overlap detector
Glue SemanticsGlue Semantics
Logical & Collocational SemanticsLogical & Collocational Semantics
Logical Semantics– Map sentences to logical representations of meaning– Enables inference & reasoning
Collocational semantics – Represent word meanings as feature vectors– Typically obtained by statistical corpus analysis– Good for indexing, classification, language modeling, word
sense disambiguation– Currently does not enable inference
Complementary, not conflicting, approaches
Example Semantic RepresentationExample Semantic Representation
F-structure gives basic predicate-argument structure, but lacks:
– Standard logical machinery (variables, connectives, etc)
– Implicit arguments (events, causes)
– Contextual dependencies (the wire = part25)
Mapping from f-structure to logical form is systematic, but non-trivial
The wire broke
PRED
SUBJ
TENSE
break<SUBJ>
PRED wireSPEC defNUM sg
past
Syntax (f-structure)
w. wire(w) & w=part25 & t. interval(t) & t<now & e. break_event(e) & occurs_during(e,t) & object_of_change(e,w) & c. cause_of_change(e,c)
Semantics (logical form)
Glue Semantics Glue Semantics Dalrymple, Lamping & Saraswat 1993 Dalrymple, Lamping & Saraswat 1993 and subsequentlyand subsequently
Syntax-semantics mapping as linear logic inference
Two logics in semantics:– Meaning Logic (target semantic representation) any suitable semantic representation– Glue Logic (deductively assembles target meaning) fragment of linear logic
Syntactic analysis produces lexical glue premises
Semantic interpretation uses deduction to assemble final meaning from these premises
Linear LogicLinear Logic Influential development in theoretical computer
science (Girard 87) Premises are resources consumed in inference
(Traditional logic: premises are non-resourced)
• Linguistic processing typically resource sensitiveWords/meanings used exactly once
Traditional LinearA, AB |= B A, A -o B |= BA, AB |= A&B A, A -o B |= AB A re-used A consumed
A, B |= B A, B |= B A discarded Cannot discard A
/
/
Glue Interpretation (Outline)Glue Interpretation (Outline) Parsing sentence instantiates lexical entries to produce
lexical glue premises Example lexical premise (verb “saw” in “John saw Fred”):
see : g -o (h -o f)Meaning Term Glue Formula2-place predicate g, h, f: constituents in parse “consume meanings of g and h to produce meaning of f”
• Glue derivation |= M : f
• Consume all lexical premises ,
• to produce meaning, M, for entire sentence, f
Glue Interpretation Glue Interpretation Getting the premisesGetting the premises
PRED
SUBJ
OBJ
see
PRED John
PRED Fred
f: g:
h:
S
NP VP
V NP
John saw Fred
Syntactic Analysis:
Lexicon: John NP john: Fred NP fred: saw V see: SUBJ -o (OBJ -o )
Instantiated premises: john: g fred: h see: g -o (h -o f)
Glue InterpretationGlue InterpretationDeduction with premisesDeduction with premises
Premises john: g fred: h see: g -o (h -o f)
Linear Logic Derivation g -o (h -o f) g
h -o f h f Using linear modus ponens
Derivation with Meaning Terms see: g -o (h -o f) john: g
see(john) : h -o f fred : h
see(john)(fred) : f
Linear modus ponens = function application
Fun: Arg:
Fun(Arg):
Modus Ponens = Function ApplicationModus Ponens = Function ApplicationThe Curry-Howard IsomorphismThe Curry-Howard Isomorphism
Curry Howard Isomorphism: Pairs LL inference rules with operations on meaning terms
g -o f g
f
Propositional linear logic inference constructs meanings LL inference completely independent of meaning language
(Modularity of meaning representation)
Semantic AmbiguitySemantic AmbiguityMultiple derivations from single set of premisesMultiple derivations from single set of premises
PRED criminal
MODSalleged
from London
f:
Alleged criminal from London Premises
criminal: f
alleged: f -o f
from-London: f -o f
Two distinct derivations:
1. from-London(alleged(criminal))
2. alleged(from-London(criminal))
Semantic Ambiguity & ModifiersSemantic Ambiguity & Modifiers
Multiple derivations from single premise set– Arises through different ways of permuting modifiers
around an skeleton Modifiers given formal representation in glue as
-o logical identities– E.g. an adjective is a noun -o noun modifier
Modifiers prevalent in natural language, and lead to combinatorial explosion– Given N f -o f modifiers, N! ways of permuting
them around f skeleton
Ambiguity management in semanticsAmbiguity management in semantics
Efficient theorem provers that manage combinatorial explosion of modifiers– Packing of N! analyses
» Represent all N! analyses in polynomial space» Compute representation in polynomial time» Free choice: Read off any given analysis in linear time
– Packing through structure re-use» N! analyses through combinations of N sub-analyses» Compute each sub-analysis once, and re-use
Parse
Generate
Select
Transfer
Interpret
Glue SemanticsLFG Syntax
FS Morphology
EnglishFrench
JapaneseGerman
AlgorithmsProgramsData structures
Mathematics
MultidimensionalArchitecture
Parc Linguistic EnvironmentParc Linguistic Environment
Translation
Condensation
Applications
Dialog
Question Answering
Email Routing
Knowledge tracking
Email ResponseTheorySoftwareTableware
UrduNorwegian
Models, parametersAm
big
uit
yM
anag
emen
t
ScaleModularityRobustness
top related