and now for something completely differenttbergkir/11711fa17/11711 flt f17.pdf · 2017-10-04 ·...
TRANSCRIPT
Andnowforsomethingcompletelydifferent
AlgorithmsforNLP(11-711)Fall2017
FormalLanguageTheoryInonelecture
RobertFrederking
NowforSomethingCompletelyDifferent
• Wewilllookatgrammarsfroma“mathematical”pointofview
• ButDiscreteMath(logic)– Norealnumbers– Symbolicdiscretestructures,proofs
• Thisisthesourceofmanycommonalgorithms/models
• Interestedincomplexity/powerofdifferentformalmodelsofcomputation– Relatedtoasymptoticcomplexitytheory
Twomainclassesofmodels
• Automata–Machines,likeFinite-StateAutomata
• Grammars– Rulesets,likewehavebeenusingtoparse
• Wewilllookateachclassofmodel,goingfromsimplertomorecomplex/powerful
• Wecanformallyprovecomplexity-classrelationsbetweentheseformalmodels
Simplestlevel:FSA/Regularsets
Finite-StateAutomata(FSAs)
• Simplestformalautomata• We’veseenthesewithnumbersonthemasHMMs,etc.
(fromWikipedia)
Formaldefinitionofautomata
• Afinitesetofstates,Q• Afinitealphabetofinputsymbols,Σ• Aninitial(start)state,Q0∈Q• Asetoffinalstates,Fi∈Q• Atransitionfunction,δ:QxΣ →Q
• ThisrigorouslydefinestheFSAsweusuallyjustdrawascirclesandarrows
RegularGrammars
• Left-linearorright-lineargrammars• Left-lineartemplate:
A→ Bx orA→x• Right-lineartemplate:
A→ xB orA→x
• Example:S→aA |bB |ε ,A→aS ,B→bbS
FormalDefinitionofaGrammar
• Vocabularyofterminalsymbols,Σ (e.g.,a)• Setofnonterminalsymbols,N(e.g.,A)• Specialstartsymbol,S∈ N• Productionrules,suchasA→aB• Restrictionsontherulesdeterminewhatkindofgrammaryouhave
• AformalgrammarGdefinesaformallanguage,L(G),thesetofstringsitgenerates
RegularExpressions
• Forregulargrammars,there’sasimplerwaytowriteexpressions:regularexpressions:
Terminalsymbols(r+s)(r•s)r*ε
• Forexample:(aa+bbb)*
Amazingfact#1:FSAsareequivalenttoRGs
• Proof:twoconstructiveproofs:– 1:givenanarbitraryFSA,constructthecorrespondingRegularGrammar(andprovethatitwillonlyproducethestringstheFSAwould)
– 2:givenanarbitraryRegularGrammar,constructthecorrespondingFSA(andprovethatitwillonlyproducethestringsthegrammarwould)
DFSAs,NDFSAs
• DeterministicorNon-deterministic– Isδ functionambiguousornot?
– ForFSAs,weaklyequivalent
Intersecting,etc.,FSAs
• WecaninvestigatewhathappensafterperformingdifferentoperationsonFSAs:– Union– Intersection– Concatenation– Negation– otheroperations:determinizing andminimizingFSAs
Provingalanguageisnot regular
• So,whatkindsoflanguagesarenot regular?
• Informally,aFSAcanonlyremember afinitenumberofspecific things.Soalanguagerequiringanunboundedmemorywon’tberegular.
Provingalanguageisnot regular
• So,whatkindsoflanguagesarenot regular?
• Informally,aFSAcanonlyremember afinitenumberofspecific things.Soalanguagerequiringanunboundedmemorywon’tberegular.
• Howaboutanbn? “equalcountofa’s and b’s”
PumpingLemma:argument:
• ConsideramachinewithNstates• NowconsideraninputoflengthN;sincewestartedinQ0,wewillnowbeinthe(N+1)ststatevisited
• Theremust bealoop:wehadtovisitatleast1statetwice;letxbethestringuptotheloop,ythepartintheloop,andzaftertheloop
• SoitmustbeokaytoalsohaveMcopiesofyforanyM(including0copies)
PumpingLemma:formally:
• IfLisaninfiniteregularlanguage,thentherearestringsx,y,andzsuchthaty≠ε andxynz∈ L,foralln≥0.
• xyzbeinginthelanguagerequiresalso:• xz,xyyz,xyyyz,xyyyyz,…, xyyyyyyyyyyz,…
PumpingLemma:figure:
q0 qNqx z
y
ExampleproofthataLisnotregular
• Whataboutanbn?abaabbaaabbbaaaabbbbaaaaabbbbb…
• Wheredoyoudrawthexynz lines?
ExampleproofthataLisnotregular
• Whataboutanbn?Wheredoyoudrawthelines?• Threecases:– y isonlya’s:thenxynz willhavetoomanya’s– y isonlyb’s:thenxynz willhavetoomanyb’s– y isamix:thentherewillbeintersperseda’sandb’s
• Soanbn cannotberegular,sinceitcannotbepumped
Nextlevel:PDA/CFG
Push-DownAutomata(PDAs)
• Let’saddsomeunboundedmemory,butinalimitedfashion
• So,addastack:
• Allowsyoutohandlesomenon-regularlanguages,butnoteverything
Context-FreeGrammars
• Ruletemplate:A→γ whereγ isanysequenceofterminals/non-terminals
• Example:S→aSb|ε• WeusethesealotinNLP– Expressiveenough,nottoocomplextoparse.• Weoftenaddhackstoallownon-CFinformationflow.
– Itjustreallyfeelsliketherightlevelofanalysis.• (Moreonthislater.)
AmazingFact#2:PDAsandCFGsareequivalent
• SamekindofproofasforFSAsandRGs,butmorecomplicated
• Aretherenon-CFlanguages?Howaboutanbncn?
Highestlevel:TMs/Unrestrictedgrammars
TuringMachines
• Justletthemachinemoveandwriteonthetape:
• Thissimplechangeproducesgeneral-purposecomputer:Church-TuringHypothesis
TMmadeofLEGOs
UnrestrictedGrammars
• α→β,whereeachcanbeanysequence(αnotempty)
• Thus,thereiscontext intherules:aAb →aabbAb →bbb
• Nosurpriseatthispoint:equivalenttoTMs
Evenmoreamazingfact:Chomskyhierarchy
• Provablethateachofthesefourclassesisapropersubsetofthenextone:
Type0:TMType1:CSGType2:CFGType3:RE
01
* 2 3
Linear-BoundedAutomata/Context-SensitiveGrammars
• TMthatusesspacelinearintheinput• αAβ→αγβ(γ notempty)
• Wemostlyignorethese;theygetnorespect• Correspondtoeachother• Limitedcomparedtofull-blownTM– Butcomplexitycanalreadybeundecidable
ChomskyHierarchy:proofs
• Formofhierarchyproofs:– Foreachclass,youcanprovetherearelanguagesnotintheclass,similartoPumpingLemmaproof
– Youcaneasilyprovethatthelargerclassreallydoescontainalltheonesinthesmallerclass
Intersecting,etc.,Ls
• WecanagaininvestigatewhathappenswithLsinthesevariousclassesunderdifferentoperationsonLs:– Union– Intersection– Concatenation– Negation– otheroperations
Chomskyhierarchy:table
MildlyContext-SensitiveGrammars
• WereallylikeCFGs,butaretheyinfactexpressiveenoughtocaptureallhumangrammar?
• Manyapproachesstartwitha“CFbackbone”,andaddregisters,equations,etc.,thatarenot CF.
• Severalnon-hackextensions(CCG,TAG,etc.)turnouttobeweaklyequivalent!– “Mildlycontextsensitive”– SoCSFsgetevenlessrespect…– AndsomuchfortheChomskyHierarchybeingsuchabigdeal
Tryingtoprovehumanlanguagesarenot CF
• Certainlytrueofsemantics.ButNLsyntax?• Cross-serialdependenciesseemlikeagoodtarget:–Mary,Jane,andJimlikered,green,andblue,respectively.
– Butisthissyntactic?• Surprisinglyhardtoprove
SwissGermandialect!dative-NPaccusative-NPdative-taking-VPaccusative-taking-VP
•JansaitdasmeremHanseshuushalfedaastriiche•JansaysthatweHansthehousehelpedpaint•“JansaysthatwehelpedHanspaintthehouse”•Jansaitdasmerd’chindemHanseshuushaendwelelaahalfeaastriiche•JansaysthatwethechildrenHansthehousehavewantedtolethelppaint•“JansaysthatwehavewantedtoletthechildrenhelpHanspaintthehouse”
(Alittlelike“Thecatthedogthemousescaredchasedlikestunafish”)
IsSwissGermanContext-Free?
Shieber’scomplexargument…
L1=Jansaitdasmer(d’chind)*(emHans)*eshuushaendwele(laa)*(halfe)*aastriiche
L2=SwissGerman
L1∩L2=Jansaitdasmer(d’chind)n(emHans)meshuushaendwele(laa)n(halfe)maastriiche
Whydowecare?(1)
• Mathisfun?• Complexity:– IfyoucanuseaRE,don’tuseaCFG.– BecarefulwithanythingfancierthanaCFG.
• Safety:hardertowritecorrectsystemsonaTuringMachine.
• Beingabletouseaweakerformalismmayhaveexplanatorypower?
Whydowecare?(2)
• Probablyasourceforfuturenewalgorithms• Probablynot howhumansactuallyprocessNL• MightnotmatterasmuchforNLPnowthatweknowaboutrealnumbers?– Butwedon’twantyourfriendsmakingfunofyou
MoreExamples
•Thecatlikestunafish•Thecatthedogchasedlikestunafish•Thecatthedogthemousescaredchasedlikestunafish•Thecatthedogthemousetheelephantsquashedscaredchasedlikestunafish•Thecatthedogthemousetheelephantthefleabitsquashedscaredchasedlikestunafish•Thecatthedogthemousetheelephantthefleathevirusinfectedbitsquashedscaredchasedlikestunafish