posterior regularization for structured latent variable models li zhonghua i2r smt reading group
TRANSCRIPT
![Page 1: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/1.jpg)
Posterior Regularization for Structured Latent Variable Models
Li ZhonghuaI2R SMT Reading Group
![Page 2: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/2.jpg)
Outline
• Motivation and Introduction• Posterior Regularization• Application• Implementation• Some Related Frameworks
![Page 3: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/3.jpg)
Motivation and Introduction
Prior Knowledge
We posses a wealth of prior knowledge about most NLP tasks.
![Page 4: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/4.jpg)
Motivation and Introduction --Prior Knowledge
![Page 5: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/5.jpg)
Motivation and Introduction --Prior Knowledge
![Page 6: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/6.jpg)
Motivation and Introduction
Leveraging Prior Knowledge Possible approaches and their limitations
![Page 7: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/7.jpg)
Motivation and Introduction --Limited Approach
Bayesian Approach : Encode prior knowledge with a prior on parameters
Limitation: Our prior knowledge is not about parameters!Parameters are difficult to interpret; hard to get desired effect.
![Page 8: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/8.jpg)
Augmenting Model : Encode prior knowledge with additional variables and dependencies.
Motivation and Introduction --Limited Approach
limitation: may make exact inference intractable
![Page 9: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/9.jpg)
Posterior Regularization
• A declarative language for specifying prior knowledge
-- Constraint Features & Expectations
• Methods for learning with knowledge in this language
-- EM style learning algorithm
![Page 10: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/10.jpg)
Posterior Regularization
![Page 11: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/11.jpg)
Posterior Regularization
Original Objective :
![Page 12: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/12.jpg)
Posterior RegularizationEM style learning algorithm
![Page 13: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/13.jpg)
Posterior Regularization
Computing the Posterior Regularizer
![Page 14: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/14.jpg)
Application
Statistical Word AlignmentsIBM Model 1 and HMM
![Page 15: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/15.jpg)
Application
One feature for each source word m, that counts how many times it is aligned to a target word in the alignment y.
![Page 16: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/16.jpg)
Application
Define feature for each target-source position pair i,j . The feature takes the value zero in expectation if a word pair i ,j is aligned with equal probability in both directions.
![Page 17: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/17.jpg)
Application
Learning Tractable Word Alignment Models with Complex Constraints CL10
![Page 18: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/18.jpg)
Application
• Six language pairs• both types of constraints improve over the
HMM in terms of both precision and recall• improve over the HMM by 10% to 15%• S-HMM performs slightly better than B-HMM• S-HMM performs better than B-HMM in 10
out of 12 cases• improve over IBM M4 9 times out of 12
![Page 19: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/19.jpg)
Application
![Page 20: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/20.jpg)
Implementation
• http://code.google.com/p/pr-toolkit/
![Page 21: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/21.jpg)
Some Related Frameworks
![Page 22: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/22.jpg)
Some Related Frameworks
![Page 23: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/23.jpg)
Some Related Frameworks
![Page 24: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/24.jpg)
Some Related Frameworks
![Page 25: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/25.jpg)
Some Related Frameworks
![Page 26: Posterior Regularization for Structured Latent Variable Models Li Zhonghua I2R SMT Reading Group](https://reader036.vdocuments.us/reader036/viewer/2022062314/56649f005503460f94c161dc/html5/thumbnails/26.jpg)
more info: http://sideinfo.wikkii.com many of my slides get from there
Thanks!