trustworthy, useful languages for probabilistic modeling and...
TRANSCRIPT
![Page 1: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/1.jpg)
Trustworthy, Useful Languages forTrustworthy, Useful Languages forTrustworthy, Useful Languages for
Probabilistic Modeling and InferenceProbabilistic Modeling and InferenceProbabilistic Modeling and Inference
Neil Toronto
Dissertation Defense
Brigham Young University
2014/06/11
![Page 2: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/2.jpg)
Master’s Research: Super-ResolutionMaster’s Research: Super-ResolutionMaster’s Research: Super-Resolution
Toronto et al. Super-Resolution via Recapture and Bayesian EffectModeling. CVPR 2009 1111111111111111
![Page 3: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/3.jpg)
Master’s Research: Super-ResolutionMaster’s Research: Super-ResolutionMaster’s Research: Super-Resolution
• Model and query: Half a page of beautiful math
2222222222222222
![Page 4: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/4.jpg)
Master’s Research: Super-ResolutionMaster’s Research: Super-ResolutionMaster’s Research: Super-Resolution
• Query implementation: 600 lines of Python
2222222222222222
![Page 5: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/5.jpg)
Main Results: Super-ResolutionMain Results: Super-ResolutionMain Results: Super-Resolution
• Competitor and BEI on 4x super-resolution:
Resolution Synthesis
3333333333333333
![Page 6: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/6.jpg)
Main Results: Super-ResolutionMain Results: Super-ResolutionMain Results: Super-Resolution
• Competitor and BEI on 4x super-resolution:
Resolution Synthesis Bayesian Edge Inference
3333333333333333
![Page 7: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/7.jpg)
Main Results: Super-ResolutionMain Results: Super-ResolutionMain Results: Super-Resolution
• Competitor and BEI on 4x super-resolution:
Resolution Synthesis Bayesian Edge Inference
• Beat state-of-the-art on “objective” measures
3333333333333333
![Page 8: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/8.jpg)
Main Results: Super-ResolutionMain Results: Super-ResolutionMain Results: Super-Resolution
• Competitor and BEI on 4x super-resolution:
Resolution Synthesis Bayesian Edge Inference
• Beat state-of-the-art on “objective” measures
• Was capable of other reconstruction tasks with few changes 3333333333333333
![Page 9: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/9.jpg)
Only Mostly SatisfyingOnly Mostly SatisfyingOnly Mostly Satisfying
Problem 1: Still not sure the program is right
4444444444444444
![Page 10: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/10.jpg)
Only Mostly SatisfyingOnly Mostly SatisfyingOnly Mostly Satisfying
Problem 1: Still not sure the program is right
Problem 2: smooth edges instead of discontinuous
4444444444444444
![Page 11: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/11.jpg)
Only Mostly SatisfyingOnly Mostly SatisfyingOnly Mostly Satisfying
Problem 1: Still not sure the program is right
Problem 2: smooth edges instead of discontinuous
“To approximate blurring with a spatially varying point-spreadfunction (PSF), we assign each facet a Gaussian PSF andconvolve each analytically before combining outputs.”
4444444444444444
![Page 12: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/12.jpg)
Only Mostly SatisfyingOnly Mostly SatisfyingOnly Mostly Satisfying
Problem 1: Still not sure the program is right
Problem 2: smooth edges instead of discontinuous
“To approximate blurring with a spatially varying point-spreadfunction (PSF), we assign each facet a Gaussian PSF andconvolve each analytically before combining outputs.”
i.e. “We can’t model it correctly so here’s a hack.” 4444444444444444
![Page 13: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/13.jpg)
Solution Idea: Probabilistic LanguageSolution Idea: Probabilistic LanguageSolution Idea: Probabilistic Language
5555555555555555
![Page 14: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/14.jpg)
Solution Idea: Probabilistic LanguageSolution Idea: Probabilistic LanguageSolution Idea: Probabilistic Language
• Also somehow let me model correctly 5555555555555555
![Page 15: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/15.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
6666666666666666
![Page 16: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/16.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice
6666666666666666
![Page 17: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/17.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice
Mimic human translation
6666666666666666
![Page 18: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/18.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice
Mimic human translation
Can’t tell error from feature
6666666666666666
![Page 19: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/19.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice
Mimic human translation
Can’t tell error from feature
Limited: usually no recursion orloops; conditions
6666666666666666
![Page 20: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/20.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice Designed for functionalprogrammers or FP theorists
Mimic human translation
Can’t tell error from feature
Limited: usually no recursion orloops; conditions
6666666666666666
![Page 21: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/21.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice Designed for functionalprogrammers or FP theorists
Mimic human translation May not be implemented
Can’t tell error from feature
Limited: usually no recursion orloops; conditions
6666666666666666
![Page 22: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/22.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice Designed for functionalprogrammers or FP theorists
Mimic human translation May not be implemented
Can’t tell error from feature Behavior is well-defined
Limited: usually no recursion orloops; conditions
6666666666666666
![Page 23: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/23.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice Designed for functionalprogrammers or FP theorists
Mimic human translation May not be implemented
Can’t tell error from feature Behavior is well-defined
Limited: usually no recursion orloops; conditions
Limited: usually finitedistributions, no conditioning
6666666666666666
![Page 24: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/24.jpg)
Prior WorkPrior WorkPrior Work
Defined by animplementation
Defined by a semantics (i.e. mathematically)
Designed for Bayesian practice Designed for functionalprogrammers or FP theorists
Mimic human translation May not be implemented
Can’t tell error from feature Behavior is well-defined
Limited: usually no recursion orloops; conditions
Limited: usually finitedistributions, no conditioning
Best of all worlds: define language using functional programmingtheory, make it for Bayesians, and remove limitations 6666666666666666
![Page 25: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/25.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
7777777777777777
![Page 26: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/26.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
• Useful: let you think abstractly and handle details for you
7777777777777777
![Page 27: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/27.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
• Useful: let you think abstractly and handle details for you
• Trustworthy: defined mathematically
7777777777777777
![Page 28: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/28.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
• Useful: let you think abstractly and handle details for you
• Trustworthy: defined mathematically
• Functional programming theory has the tools to defineprogramming languages mathematically
7777777777777777
![Page 29: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/29.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
• Useful: let you think abstractly and handle details for you
• Trustworthy: defined mathematically
• Functional programming theory has the tools to defineprogramming languages mathematically
• Measure-theoretic probability is the most complete account ofprobability; should allow shedding common limitations 7777777777777777
![Page 30: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/30.jpg)
Simple Example ProcessSimple Example ProcessSimple Example Process
• Example process: Normal-Normal
8888888888888888
![Page 31: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/31.jpg)
Simple Example ProcessSimple Example ProcessSimple Example Process
• Example process: Normal-Normal
• Intuition: Sample , then sample using
8888888888888888
![Page 32: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/32.jpg)
Simple Example ProcessSimple Example ProcessSimple Example Process
• Example process: Normal-Normal
• Intuition: Sample , then sample using
• Density model :
8888888888888888
![Page 33: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/33.jpg)
Simple Example ProcessSimple Example ProcessSimple Example Process
• Example process: Normal-Normal
• Intuition: Sample , then sample using
• Compute query by integrating:
8888888888888888
![Page 34: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/34.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 35: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/35.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 36: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/36.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 37: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/37.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 38: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/38.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 39: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/39.jpg)
Conditional QueriesConditional QueriesConditional Queries
• Compute query using Bayes’ law:
9999999999999999
![Page 40: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/40.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
10101010101010101010101010101010
![Page 41: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/41.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
10101010101010101010101010101010
![Page 42: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/42.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
Distributions given non-axial, zero-probability conditions
10101010101010101010101010101010
![Page 43: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/43.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
Distributions given non-axial, zero-probability conditions
Discontinuous change of variable (e.g. a thermometer)
10101010101010101010101010101010
![Page 44: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/44.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
Distributions given non-axial, zero-probability conditions
Discontinuous change of variable (e.g. a thermometer)
Distributions of variable-dimension random variables
10101010101010101010101010101010
![Page 45: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/45.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
Distributions given non-axial, zero-probability conditions
Discontinuous change of variable (e.g. a thermometer)
Distributions of variable-dimension random variables
Nontrivial distributions on infinite products
10101010101010101010101010101010
![Page 46: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/46.jpg)
So What Can’t Densities Model?So What Can’t Densities Model?So What Can’t Densities Model?
• Tons of useful things that are easy to write down
Distributions given non-axial, zero-probability conditions
Discontinuous change of variable (e.g. a thermometer)
Distributions of variable-dimension random variables
Nontrivial distributions on infinite products
• Tricks to get around limitations aren’t general enough 10101010101010101010101010101010
![Page 47: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/47.jpg)
Measure-Theoretic ProbabilityMeasure-Theoretic ProbabilityMeasure-Theoretic Probability
• Main ideas:
Don’t assign probability-like quantities to values, assignprobabilities to sets — the probability query is king
11111111111111111111111111111111
![Page 48: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/48.jpg)
Measure-Theoretic ProbabilityMeasure-Theoretic ProbabilityMeasure-Theoretic Probability
• Main ideas:
Don’t assign probability-like quantities to values, assignprobabilities to sets — the probability query is king
Confine assumed randomness to one place by makingrandom variables deterministic functions that observe arandom source
11111111111111111111111111111111
![Page 49: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/49.jpg)
Measure-Theoretic ProbabilityMeasure-Theoretic ProbabilityMeasure-Theoretic Probability
• Main ideas:
Don’t assign probability-like quantities to values, assignprobabilities to sets — the probability query is king
Confine assumed randomness to one place by makingrandom variables deterministic functions that observe arandom source
• Measure-theoretic model of example process:
11111111111111111111111111111111
![Page 50: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/50.jpg)
Measure-Theoretic ProbabilityMeasure-Theoretic ProbabilityMeasure-Theoretic Probability
• Main ideas:
Don’t assign probability-like quantities to values, assignprobabilities to sets — the probability query is king
Confine assumed randomness to one place by makingrandom variables deterministic functions that observe arandom source
• Measure-theoretic model of example process:
11111111111111111111111111111111
![Page 51: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/51.jpg)
Measure-Theoretic QueriesMeasure-Theoretic QueriesMeasure-Theoretic Queries
• Specific query:
12121212121212121212121212121212
![Page 52: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/52.jpg)
Measure-Theoretic QueriesMeasure-Theoretic QueriesMeasure-Theoretic Queries
• Specific query:
• Generalized:
12121212121212121212121212121212
![Page 53: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/53.jpg)
Measure-Theoretic QueriesMeasure-Theoretic QueriesMeasure-Theoretic Queries
• Specific query:
• Generalized:
• Conditional query: if then
12121212121212121212121212121212
![Page 54: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/54.jpg)
Measure-Theoretic QueriesMeasure-Theoretic QueriesMeasure-Theoretic Queries
• Specific query:
• Generalized:
• Conditional query: if then
Can we avoid densities when ?
12121212121212121212121212121212
![Page 55: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/55.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 56: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/56.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 57: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/57.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 58: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/58.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 59: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/59.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 60: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/60.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 61: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/61.jpg)
Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)Zero-Probability Conditions (Axial)
13131313131313131313131313131313
![Page 62: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/62.jpg)
Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)
14141414141414141414141414141414
![Page 63: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/63.jpg)
Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)
14141414141414141414141414141414
![Page 64: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/64.jpg)
Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)
14141414141414141414141414141414
![Page 65: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/65.jpg)
Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)
14141414141414141414141414141414
![Page 66: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/66.jpg)
Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)Zero-Probability Conditions (Circular)
14141414141414141414141414141414
![Page 67: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/67.jpg)
Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)
• Integration is hard!
15151515151515151515151515151515
![Page 68: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/68.jpg)
Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)
• Integration is hard!
• But random variables and are an abstraction boundaryhiding and , so we can choose convenient ones
15151515151515151515151515151515
![Page 69: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/69.jpg)
Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)
• Integration is hard!
• But random variables and are an abstraction boundaryhiding and , so we can choose convenient ones
A uniform random source model:
15151515151515151515151515151515
![Page 70: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/70.jpg)
Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)
• Integration is hard!
• But random variables and are an abstraction boundaryhiding and , so we can choose convenient ones
A uniform random source model:
where is the Normal CDF
15151515151515151515151515151515
![Page 71: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/71.jpg)
Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)Contribution: Don’t Integrate, Compute Backwards (1)
• Integration is hard!
• But random variables and are an abstraction boundaryhiding and , so we can choose convenient ones
A uniform random source model:
where is the Normal CDF
• Stretches instead of integrates 15151515151515151515151515151515
![Page 72: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/72.jpg)
Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)
• Generalized query:
16161616161616161616161616161616
![Page 73: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/73.jpg)
Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)
• Generalized query:
i.e. output distributions are defined by preimages
16161616161616161616161616161616
![Page 74: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/74.jpg)
Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)
• Generalized query:
i.e. output distributions are defined by preimages
• For a uniform random source model,
Compute probabilities by computing preimage areas
16161616161616161616161616161616
![Page 75: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/75.jpg)
Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)
• Generalized query:
i.e. output distributions are defined by preimages
• For a uniform random source model,
Compute probabilities by computing preimage areas
Compute conditional probabilities as quotients of preimageareas
16161616161616161616161616161616
![Page 76: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/76.jpg)
Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)Contribution: Don’t Integrate, Compute Backwards (2)
• Generalized query:
i.e. output distributions are defined by preimages
• For a uniform random source model,
Compute probabilities by computing preimage areas
Compute conditional probabilities as quotients of preimageareas
• Is this really more feasible than integrating?
16161616161616161616161616161616
![Page 77: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/77.jpg)
Queries Using PreimagesQueries Using PreimagesQueries Using Preimages
17171717171717171717171717171717
![Page 78: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/78.jpg)
Queries Using PreimagesQueries Using PreimagesQueries Using Preimages
Uniform Random Source Original Model
17171717171717171717171717171717
![Page 79: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/79.jpg)
Queries Using PreimagesQueries Using PreimagesQueries Using Preimages
Uniform Random Source Original Model
17171717171717171717171717171717
![Page 80: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/80.jpg)
Queries Using PreimagesQueries Using PreimagesQueries Using Preimages
Uniform Random Source Original Model
17171717171717171717171717171717
![Page 81: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/81.jpg)
Queries Using PreimagesQueries Using PreimagesQueries Using Preimages
Uniform Random Source Original Model
17171717171717171717171717171717
![Page 82: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/82.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
18181818181818181818181818181818
![Page 83: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/83.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
Efficient way to compute preimage sets
18181818181818181818181818181818
![Page 84: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/84.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
Efficient way to compute preimage sets
Efficient representation of arbitrary sets
18181818181818181818181818181818
![Page 85: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/85.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
Efficient way to compute preimage sets
Efficient representation of arbitrary sets
Efficient way to compute areas of preimage sets
18181818181818181818181818181818
![Page 86: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/86.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
Efficient way to compute preimage sets
Efficient representation of arbitrary sets
Efficient way to compute areas of preimage sets
Proof of correctness w.r.t. standard interpretation
18181818181818181818181818181818
![Page 87: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/87.jpg)
Crazy Idea is Feasible If...Crazy Idea is Feasible If...Crazy Idea is Feasible If...
• Seems like we need:
Standard interpretation of programs as pure functions from arandom source
Efficient way to compute preimage sets
Efficient representation of arbitrary sets
Efficient way to compute areas of preimage sets
Proof of correctness w.r.t. standard interpretation
• Completely infeasible! But...
18181818181818181818181818181818
![Page 88: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/88.jpg)
What About Approximating?What About Approximating?What About Approximating?
Conservative approximation with rectangles:
19191919191919191919191919191919
![Page 89: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/89.jpg)
What About Approximating?What About Approximating?What About Approximating?
Conservative approximation with rectangles:
19191919191919191919191919191919
![Page 90: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/90.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 91: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/91.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 92: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/92.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 93: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/93.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 94: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/94.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 95: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/95.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 96: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/96.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 97: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/97.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 98: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/98.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 99: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/99.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 100: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/100.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 101: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/101.jpg)
What About Approximating?What About Approximating?What About Approximating?
Restricting preimages to rectangular subdomains:
20202020202020202020202020202020
![Page 102: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/102.jpg)
What About Approximating?What About Approximating?What About Approximating?
Sampling: exponential to quadratic (e.g. days to minutes)
21212121212121212121212121212121
![Page 103: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/103.jpg)
What About Approximating?What About Approximating?What About Approximating?
Sampling: exponential to quadratic (e.g. days to minutes)
21212121212121212121212121212121
![Page 104: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/104.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute preimage sets
• Efficient representation of arbitrary sets
• Efficient way to compute volumes of preimage sets
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 105: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/105.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute approximate preimage subsets
• Efficient representation of arbitrary sets
• Efficient way to compute volumes of preimage sets
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 106: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/106.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute approximate preimage subsets
• Efficient representation of approximating sets
• Efficient way to compute volumes of preimage sets
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 107: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/107.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute approximate preimage subsets
• Efficient representation of approximating sets
• Efficient way to sample uniformly in preimage sets
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 108: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/108.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute approximate preimage subsets
• Efficient representation of approximating sets
• Efficient way to sample uniformly in preimage sets
Efficient domain partition sampling
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 109: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/109.jpg)
Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...Crazy Idea is Actually Feasible If...
• Standard interpretation of programs as pure functions from arandom source
• Efficient way to compute approximate preimage subsets
• Efficient representation of approximating sets
• Efficient way to sample uniformly in preimage sets
Efficient domain partition sampling
Efficient way to determine whether a domain sample isactually in the preimage (just use standard interpretation)
• Proof of correctness w.r.t. standard interpretation
22222222222222222222222222222222
![Page 110: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/110.jpg)
Standard InterpretationStandard InterpretationStandard Interpretation
• Grammar:
23232323232323232323232323232323
![Page 111: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/111.jpg)
Standard InterpretationStandard InterpretationStandard Interpretation
• Grammar:
• Semantic function
23232323232323232323232323232323
![Page 112: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/112.jpg)
Standard InterpretationStandard InterpretationStandard Interpretation
• Grammar:
• Semantic function
• Math has no general recursion, so (i.e. interpretation ofprogram ) is a λ-calculus term
23232323232323232323232323232323
![Page 113: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/113.jpg)
Standard InterpretationStandard InterpretationStandard Interpretation
• Grammar:
• Semantic function
• Math has no general recursion, so (i.e. interpretation ofprogram ) is a λ-calculus term
• Easy implementation in any language with lambdas
23232323232323232323232323232323
![Page 114: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/114.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
24242424242424242424242424242424
![Page 115: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/115.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
24242424242424242424242424242424
![Page 116: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/116.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
• Example: meaning of
24242424242424242424242424242424
![Page 117: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/117.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
• Example: meaning of
24242424242424242424242424242424
![Page 118: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/118.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
• Example: meaning of
• Nonexample:
24242424242424242424242424242424
![Page 119: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/119.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
• Example: meaning of
• Nonexample:
24242424242424242424242424242424
![Page 120: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/120.jpg)
Compositional SemanticsCompositional SemanticsCompositional Semantics
• Compositional: every term’s meaning depends only on itsimmediate subterms’ meanings
• Advantage: proofs about all programs by structural induction
• Example: meaning of
• Nonexample:
• Can preimages be computed compositionally? 24242424242424242424242424242424
![Page 121: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/121.jpg)
Pair PreimagesPair PreimagesPair Preimages
25252525252525252525252525252525
![Page 122: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/122.jpg)
Pair PreimagesPair PreimagesPair Preimages
25252525252525252525252525252525
![Page 123: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/123.jpg)
Pair PreimagesPair PreimagesPair Preimages
:
25252525252525252525252525252525
![Page 124: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/124.jpg)
Pair PreimagesPair PreimagesPair Preimages
and :
25252525252525252525252525252525
![Page 125: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/125.jpg)
Pair PreimagesPair PreimagesPair Preimages
:
25252525252525252525252525252525
![Page 126: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/126.jpg)
Nonstandard Interpretation: Computing PreimagesNonstandard Interpretation: Computing PreimagesNonstandard Interpretation: Computing Preimages
• Preimage computation:
26262626262626262626262626262626
![Page 127: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/127.jpg)
Nonstandard Interpretation: Computing PreimagesNonstandard Interpretation: Computing PreimagesNonstandard Interpretation: Computing Preimages
• Preimage computation:
26262626262626262626262626262626
![Page 128: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/128.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
27272727272727272727272727272727
![Page 129: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/129.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
Theorem (correctness under pairing). If
computes preimages under
27272727272727272727272727272727
![Page 130: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/130.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
Theorem (correctness under pairing). If
computes preimages under
computes preimages under
27272727272727272727272727272727
![Page 131: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/131.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
Theorem (correctness under pairing). If
computes preimages under
computes preimages under
then computes preimages under .
27272727272727272727272727272727
![Page 132: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/132.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
Theorem (correctness under pairing). If
computes preimages under
computes preimages under
then computes preimages under .
Proof sketch. Preimages distribute over cartesian products.
27272727272727272727272727272727
![Page 133: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/133.jpg)
Nonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under PairingNonstandard Interpretation: Preimages Under Pairing
• Pairing types:
Theorem (correctness under pairing). If
computes preimages under
computes preimages under
then computes preimages under .
Proof sketch. Preimages distribute over cartesian products.
• Similar theorems for every kind of term 27272727272727272727272727272727
![Page 134: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/134.jpg)
Nonstandard Interpretation: CorrectnessNonstandard Interpretation: CorrectnessNonstandard Interpretation: Correctness
Theorem. For all programs , computes preimages under.
Proof. By structural induction on program terms.
28282828282828282828282828282828
![Page 135: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/135.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
29292929292929292929292929292929
![Page 136: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/136.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
29292929292929292929292929292929
![Page 137: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/137.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
• Q. Where do I get a computer that runs them?
29292929292929292929292929292929
![Page 138: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/138.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
• Q. Where do I get a computer that runs them?
A. Nowhere, but we’ll approximate them soon.
29292929292929292929292929292929
![Page 139: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/139.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
• Q. Where do I get a computer that runs them?
A. Nowhere, but we’ll approximate them soon.
• Q. Why interpret programs as uncomputable functions, then?
29292929292929292929292929292929
![Page 140: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/140.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
• Q. Where do I get a computer that runs them?
A. Nowhere, but we’ll approximate them soon.
• Q. Why interpret programs as uncomputable functions, then?
A. So we know exactly what to approximate.
29292929292929292929292929292929
![Page 141: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/141.jpg)
Wait a MinuteWait a MinuteWait a Minute
• Q. Don’t the interpretations of do uncountable things?
A. Yes. Yes, they do.
• Q. Where do I get a computer that runs them?
A. Nowhere, but we’ll approximate them soon.
• Q. Why interpret programs as uncomputable functions, then?
A. So we know exactly what to approximate.
• Q. Where did you get a λ-calculus that could operate on arbitrary,possibly infinite sets, anyway?
A. Well...
29292929292929292929292929292929
![Page 142: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/142.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
30303030303030303030303030303030
![Page 143: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/143.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
+
Infinite sets and operations
30303030303030303030303030303030
![Page 144: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/144.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
+
Infinite sets and operations
=
λZFC
30303030303030303030303030303030
![Page 145: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/145.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
+
Infinite sets and operations
=
λZFC
• Contemporary math, but with lambdas and general recursion; orfunctional programming, but with infinite sets
30303030303030303030303030303030
![Page 146: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/146.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
+
Infinite sets and operations
=
λZFC
• Contemporary math, but with lambdas and general recursion; orfunctional programming, but with infinite sets
• Can express uncountably infinite operations, can’t solve its ownhalting problem
30303030303030303030303030303030
![Page 147: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/147.jpg)
Lambda-ZFCLambda-ZFCLambda-ZFC
λ calculus
+
Infinite sets and operations
=
λZFC
• Contemporary math, but with lambdas and general recursion; orfunctional programming, but with infinite sets
• Can express uncountably infinite operations, can’t solve its ownhalting problem
• Can use contemporary mathematical theorems directly
30303030303030303030303030303030
![Page 148: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/148.jpg)
Rectangular ApproximationRectangular ApproximationRectangular Approximation
• A rectangle is
An interval or union of intervals
for rectangles and
31313131313131313131313131313131
![Page 149: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/149.jpg)
Rectangular ApproximationRectangular ApproximationRectangular Approximation
• A rectangle is
An interval or union of intervals
for rectangles and
• Easy representation; easy intersection and join (union-like)operation, empty test, other operations
31313131313131313131313131313131
![Page 150: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/150.jpg)
Rectangular ApproximationRectangular ApproximationRectangular Approximation
• A rectangle is
An interval or union of intervals
for rectangles and
• Easy representation; easy intersection and join (union-like)operation, empty test, other operations
• Recall:
31313131313131313131313131313131
![Page 151: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/151.jpg)
Rectangular ApproximationRectangular ApproximationRectangular Approximation
• A rectangle is
An interval or union of intervals
for rectangles and
• Easy representation; easy intersection and join (union-like)operation, empty test, other operations
• Recall:
• Define:
31313131313131313131313131313131
![Page 152: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/152.jpg)
Rectangular ApproximationRectangular ApproximationRectangular Approximation
• A rectangle is
An interval or union of intervals
for rectangles and
• Easy representation; easy intersection and join (union-like)operation, empty test, other operations
• Recall:
• Define:
• Derive 31313131313131313131313131313131
![Page 153: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/153.jpg)
In Theory...In Theory...In Theory...
Theorem (sound). computes overapproximations of thepreimages computed by .
• Consequence: Sampling within preimages doesn’t leave anythingout
32323232323232323232323232323232
![Page 154: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/154.jpg)
In Theory...In Theory...In Theory...
Theorem (sound). computes overapproximations of thepreimages computed by .
• Consequence: Sampling within preimages doesn’t leave anythingout
Theorem (monotone). is monotone.
• Consequence: Partitioning the domain never increasesapproximate preimages
32323232323232323232323232323232
![Page 155: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/155.jpg)
In Theory...In Theory...In Theory...
Theorem (sound). computes overapproximations of thepreimages computed by .
• Consequence: Sampling within preimages doesn’t leave anythingout
Theorem (monotone). is monotone.
• Consequence: Partitioning the domain never increasesapproximate preimages
Theorem (decreasing). never returns preimages larger thanthe given subdomain.
• Consequence: Refining preimage partitions never explodes32323232323232323232323232323232
![Page 156: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/156.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 157: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/157.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 158: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/158.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 159: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/159.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 160: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/160.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 161: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/161.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 162: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/162.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 163: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/163.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 164: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/164.jpg)
In Practice...In Practice...In Practice...
Theorems prove this always works:
33333333333333333333333333333333
![Page 165: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/165.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
34343434343434343434343434343434
![Page 166: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/166.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
First, refine using preimage computation:
34343434343434343434343434343434
![Page 167: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/167.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
Second, randomly choose from arbitrarily fine partition:
34343434343434343434343434343434
![Page 168: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/168.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
Third, refine again:
34343434343434343434343434343434
![Page 169: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/169.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
Fourth, sample uniformly:
34343434343434343434343434343434
![Page 170: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/170.jpg)
Importance SamplingImportance SamplingImportance Sampling
• Alternative to arbitrarily low-rate rejection sampling:
Do process “in the limit”; i.e. choose :
34343434343434343434343434343434
![Page 171: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/171.jpg)
What About Recursion?What About Recursion?What About Recursion?
• General recursion, programs that halt with probability 1; e.g.
(define/drbayes (geometric p) (if (bernoulli p)
0(+ 1 (geometric p))))
35353535353535353535353535353535
![Page 172: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/172.jpg)
What About Recursion?What About Recursion?What About Recursion?
• General recursion, programs that halt with probability 1; e.g.
(define/drbayes (geometric p) (if (bernoulli p)
0(+ 1 (geometric p))))
• Consider programs as being fully inlined (thus infinite):
(if (bernoulli p)0(+ 1 (if (bernoulli p)
0(+ 1 (if (bernoulli p)
0(+ 1 ...))))))
35353535353535353535353535353535
![Page 173: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/173.jpg)
What About Recursion?What About Recursion?What About Recursion?
• General recursion, programs that halt with probability 1; e.g.
(define/drbayes (geometric p) (if (bernoulli p)
0(+ 1 (geometric p))))
• Consider programs as being fully inlined (thus infinite):
(if (bernoulli p)0(+ 1 (if (bernoulli p)
0(+ 1 (if (bernoulli p)
0(+ 1 ...))))))
• Random domain needs to be big enough and the right shape35353535353535353535353535353535
![Page 174: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/174.jpg)
Program Domain ValuesProgram Domain ValuesProgram Domain Values
• Values are infinite binary trees:
36363636363636363636363636363636
![Page 175: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/175.jpg)
Program Domain ValuesProgram Domain ValuesProgram Domain Values
• Values are infinite binary trees:
• Every expression in a program is assigned a node
36363636363636363636363636363636
![Page 176: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/176.jpg)
Program Domain ValuesProgram Domain ValuesProgram Domain Values
• Values are infinite binary trees:
• Every expression in a program is assigned a node
• Implemented using lazy trees of random values
36363636363636363636363636363636
![Page 177: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/177.jpg)
Program Domain ValuesProgram Domain ValuesProgram Domain Values
• Values are infinite binary trees:
• Every expression in a program is assigned a node
• Implemented using lazy trees of random values
• No probability density for domain, but there is a measure 36363636363636363636363636363636
![Page 178: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/178.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
• Normal-Normal process:
37373737373737373737373737373737
![Page 179: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/179.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
• Normal-Normal process:
• Objective: Find the distribution of
37373737373737373737373737373737
![Page 180: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/180.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
• Normal-Normal process:
• Objective: Find the distribution of
• Implementation:
(define/drbayes e (let* ([x (normal 0 1)]
[y (normal x 1)]) (list x y (sqrt (+ (sqr x) (sqr y))))))
37373737373737373737373737373737
![Page 181: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/181.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
• Normal-Normal process:
• Objective: Find the distribution of
• Implementation:
(define/drbayes e (let* ([x (normal 0 1)]
[y (normal x 1)]) (list x y (sqrt (+ (sqr x) (sqr y))))))
• Goal: Sample in the preimage of
(set-list reals reals (interval (- 1 ε) (+ 1 ε)))
37373737373737373737373737373737
![Page 182: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/182.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
For ε = 0.01:
Preimage rectangles
38383838383838383838383838383838
![Page 183: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/183.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
For ε = 0.01:
Preimage samples
38383838383838383838383838383838
![Page 184: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/184.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
For ε = 0.01:
Preimage samples Output samples
38383838383838383838383838383838
![Page 185: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/185.jpg)
Demo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular ConditionDemo: Normal-Normal With Circular Condition
For ε = 0.01:
Preimage samples Output samples
• Works fine with much smaller ε38383838383838383838383838383838
![Page 186: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/186.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
• Normal-Normal thermometer process:
39393939393939393939393939393939
![Page 187: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/187.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
• Normal-Normal thermometer process:
• Objective: Find the distribution of
39393939393939393939393939393939
![Page 188: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/188.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
• Normal-Normal thermometer process:
• Objective: Find the distribution of
• Implementation:
(define/drbayes e (let* ([x (normal 90 10)]
[y (normal x 1)]) (list x (if (> y 100) 100 y))))
39393939393939393939393939393939
![Page 189: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/189.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
• Normal-Normal thermometer process:
• Objective: Find the distribution of
• Implementation:
(define/drbayes e (let* ([x (normal 90 10)]
[y (normal x 1)]) (list x (if (> y 100) 100 y))))
• Goal: Sample in the preimage of
(set-list reals (interval 100 100)) 39393939393939393939393939393939
![Page 190: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/190.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
Preimage rectangles
40404040404040404040404040404040
![Page 191: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/191.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
Preimage samples
40404040404040404040404040404040
![Page 192: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/192.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
Preimage samples Density of
40404040404040404040404040404040
![Page 193: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/193.jpg)
Demo: ThermometerDemo: ThermometerDemo: Thermometer
Preimage samples Density of
Calculated from samples: mean 105.1, stddev 4.6
40404040404040404040404040404040
![Page 194: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/194.jpg)
Demo: Stochastic Ray TracingDemo: Stochastic Ray TracingDemo: Stochastic Ray Tracing
• Idea: Model light transmission and reflection, condition on pathsthat pass through aperture
41414141414141414141414141414141
![Page 195: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/195.jpg)
Demo: Stochastic Ray TracingDemo: Stochastic Ray TracingDemo: Stochastic Ray Tracing
• Idea: Model light transmission and reflection, condition on pathsthat pass through aperture
41414141414141414141414141414141
![Page 196: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/196.jpg)
Demo: Stochastic Ray TracingDemo: Stochastic Ray TracingDemo: Stochastic Ray Tracing
• Part of the implementation (totals ~50 lines):(define/drbayes (ray-plane-intersect p0 v n d) (let ([denom (- (vec-dot v n))])
(if (positive? denom)(let ([t (/ (+ d (vec-dot p0 n)) denom)]) (if (positive? t)
(collision t (vec+ p0 (vec-scale v t)) n)#f))
#f)))
42424242424242424242424242424242
![Page 197: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/197.jpg)
Demo: Stochastic Ray TracingDemo: Stochastic Ray TracingDemo: Stochastic Ray Tracing
• Part of the implementation (totals ~50 lines):(define/drbayes (ray-plane-intersect p0 v n d) (let ([denom (- (vec-dot v n))])
(if (positive? denom)(let ([t (/ (+ d (vec-dot p0 n)) denom)]) (if (positive? t)
(collision t (vec+ p0 (vec-scale v t)) n)#f))
#f)))
• Constrained light path outputs:
Paths Through Aperture Projected and Accumulated
42424242424242424242424242424242
![Page 198: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/198.jpg)
Other Inference TasksOther Inference TasksOther Inference Tasks
• Typical
Hierarchical models
Bayesian regression
Model selection
43434343434343434343434343434343
![Page 199: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/199.jpg)
Other Inference TasksOther Inference TasksOther Inference Tasks
• Typical
Hierarchical models
Bayesian regression
Model selection
• Atypical
Programs that halt with probability < 1, or never halt
Probabilistic program verification (sample in preimage of errorcondition)
43434343434343434343434343434343
![Page 200: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/200.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
44444444444444444444444444444444
![Page 201: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/201.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
True.
44444444444444444444444444444444
![Page 202: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/202.jpg)
Thesis StatementThesis StatementThesis Statement
Functional programming theory and measure-theoreticprobability provide a solid foundation
for trustworthy, useful languages for constructiveprobabilistic modeling and inference.
True.
• Was it falsifiable?
44444444444444444444444444444444
![Page 203: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/203.jpg)
MeasurabilityMeasurabilityMeasurability
• Only measurable sets can have probabilities
45454545454545454545454545454545
![Page 204: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/204.jpg)
MeasurabilityMeasurabilityMeasurability
• Only measurable sets can have probabilities
• Computing preimages under must preserve measurability—wesay itself is measurable
45454545454545454545454545454545
![Page 205: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/205.jpg)
MeasurabilityMeasurabilityMeasurability
• Only measurable sets can have probabilities
• Computing preimages under must preserve measurability—wesay itself is measurable
Theorem (measurability). For all programs , is measurable,regardless of errors or nontermination, if language primitives aremeasurable.
45454545454545454545454545454545
![Page 206: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/206.jpg)
MeasurabilityMeasurabilityMeasurability
• Only measurable sets can have probabilities
• Computing preimages under must preserve measurability—wesay itself is measurable
Theorem (measurability). For all programs , is measurable,regardless of errors or nontermination, if language primitives aremeasurable.
• Primitives include uncomputable operations like limits
45454545454545454545454545454545
![Page 207: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/207.jpg)
MeasurabilityMeasurabilityMeasurability
• Only measurable sets can have probabilities
• Computing preimages under must preserve measurability—wesay itself is measurable
Theorem (measurability). For all programs , is measurable,regardless of errors or nontermination, if language primitives aremeasurable.
• Primitives include uncomputable operations like limits
• Applies to all probabilistic programming languages45454545454545454545454545454545
![Page 208: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/208.jpg)
What I DidWhat I DidWhat I Did
45454545454545454545454545454545
![Page 209: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/209.jpg)
What I DidWhat I DidWhat I Did
The core calculus for this:
45454545454545454545454545454545
![Page 210: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/210.jpg)
Future WorkFuture WorkFuture Work
• Expressiveness
Lambdas and macros
Exceptions, parameters (or continuations and marks)
46464646464646464646464646464646
![Page 211: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/211.jpg)
Future WorkFuture WorkFuture Work
• Expressiveness
Lambdas and macros
Exceptions, parameters (or continuations and marks)
• Optimization
Direct implementation is in depth; cut to
Incremental computation
Adaptive sampling algorithms
Static analysis
46464646464646464646464646464646
![Page 212: Trustworthy, Useful Languages for Probabilistic Modeling and …ntoronto/papers/toronto-2014diss... · 2015-04-22 · Thesis Statement Functional programming theory and measure-theoretic](https://reader034.vdocuments.us/reader034/viewer/2022042216/5ebe397699121e0ec7250663/html5/thumbnails/212.jpg)
Future WorkFuture WorkFuture Work
• Expressiveness
Lambdas and macros
Exceptions, parameters (or continuations and marks)
• Optimization
Direct implementation is in depth; cut to
Incremental computation
Adaptive sampling algorithms
Static analysis
• Branching out: investigate preimage computation connection withtype systems and predicate transformer semantics 46464646464646464646464646464646