cs b553 : a lgorithms for optimization and learning
DESCRIPTION
CS b553 : A lgorithms for Optimization and Learning. Variable Elimination. Last Time. Variable elimination on polytrees Top down inference Linear in size of network Variable elimination in general No guarantees… NP hard in worst case… but when?. Variable Elimination in General Networks. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/1.jpg)
CS B553: ALGORITHMS FOR OPTIMIZATION AND LEARNINGVariable Elimination
![Page 2: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/2.jpg)
LAST TIME Variable elimination on polytrees
Top down inference Linear in size of network
Variable elimination in general No guarantees… NP hard in worst case… but when?
![Page 3: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/3.jpg)
VARIABLE ELIMINATION IN GENERAL NETWORKS
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 4: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/4.jpg)
VARIABLE ELIMINATION IN GENERAL NETWORKS
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 5: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/5.jpg)
JOINT DISTRIBUTION P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
![Page 6: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/6.jpg)
GOING THROUGH VE P(X) = P(C)P(D|C)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
1(D)=SCP(C)P(D|C)
![Page 7: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/7.jpg)
GOING THROUGH VE SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
1(D)=SCP(C)P(D|C)
![Page 8: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/8.jpg)
GOING THROUGH VE SCP(X) = 1(D)P(I)P(G|I,D)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
2(G,I)=SD1(D)P(G|I,D)
![Page 9: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/9.jpg)
GOING THROUGH VE SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
2(G,I)=SD1(D)P(G|I,D)
![Page 10: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/10.jpg)
GOING THROUGH VE SC,DP(X) = 2(G,I)P(I)P(S|I)P(L|G) P(J|
L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
3(G,S)=SI2(G,I)P(I)P(S|I)
![Page 11: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/11.jpg)
GOING THROUGH VE SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
3(G,S)=SI2(G,I)P(I)P(S|I)
![Page 12: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/12.jpg)
GOING THROUGH VE SC,D,IP(X) = 3(G,S)P(L|G)P(J|L,S)P(H|G,J) Apply elimination ordering C,D,I,H,G,S,L
4(G,J)=SHP(H|G,J)
What values does this factor store?
![Page 13: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/13.jpg)
GOING THROUGH VE SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) Apply elimination ordering C,D,I,H,G,S,L
4(G,J)=SHP(H|G,J)
![Page 14: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/14.jpg)
GOING THROUGH VE SC,D,I,HP(X) = 3(G,S)P(L|G)P(J|L,S)4(G,J) Apply elimination ordering C,D,I,H,G,S,L
5(S,L,J)=SG 3(G,S)P(L|G)4(G,J)
![Page 15: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/15.jpg)
GOING THROUGH VE SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L
5(S,L,J)=SG 3(G,S)P(L|G)4(G,J)
![Page 16: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/16.jpg)
GOING THROUGH VE SC,D,I,H,GP(X) = 5(S,L,J)P(J|L,S) Apply elimination ordering C,D,I,H,G,S,L
6(L,J)=SS 5(S,L,J)P(J|L,S)
![Page 17: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/17.jpg)
GOING THROUGH VE SC,D,I,H,G,SP(X) = 6(L,J) Apply elimination ordering C,D,I,H,G,S,L
6(L,J)=SS 5(S,L,J)
![Page 18: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/18.jpg)
GOING THROUGH VE SC,D,I,H,G,SP(X) = 6(L,J) Apply elimination ordering C,D,I,H,G,S,L
7(J)=SL 6(S,L)
![Page 19: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/19.jpg)
GOING THROUGH VE SC,D,I,H,G,S,LP(X) = 7(J) Apply elimination ordering C,D,I,H,G,S,L
7(J)=SL 6(L,J)
![Page 20: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/20.jpg)
COMPARING ORDERINGS Consider G,I,S,L,H,C,D
![Page 21: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/21.jpg)
UNDERSTANDING VE: FROM BNS TO UNDIRECTED GRAPHS Consider each factor as a variable i Draw an edge between any variables
appearing in the same factor
![Page 22: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/22.jpg)
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
P(D|C)
P(C)
P(I)
P(G|I,D)
P(H|G,J)
P(J|S,L)
P(S|I)
P(L|G)
![Page 23: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/23.jpg)
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
P(D|C)
P(C)
P(I)
P(G|I,D)
P(H|G,J)
P(J|S,L)
P(S|I)
P(L|G)
![Page 24: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/24.jpg)
BUILDING THE UNDIRECTED GRAPH
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 25: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/25.jpg)
VARIABLE ELIMINATION
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 26: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/26.jpg)
VARIABLE ELIMINATION
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 27: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/27.jpg)
VARIABLE ELIMINATION
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 28: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/28.jpg)
VARIABLE ELIMINATION
Intelligence
Happy
Grade SAT
Letter
Job
![Page 29: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/29.jpg)
VARIABLE ELIMINATION
Intelligence
Happy
Grade SAT
Letter
Job
![Page 30: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/30.jpg)
VARIABLE ELIMINATION
Happy
Grade SAT
Letter
Job
New fill edge
![Page 31: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/31.jpg)
VARIABLE ELIMINATION
Happy
Grade SAT
Letter
Job
![Page 32: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/32.jpg)
VARIABLE ELIMINATION
Grade SAT
Letter
Job
![Page 33: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/33.jpg)
VARIABLE ELIMINATION
Grade SAT
Letter
Job
![Page 34: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/34.jpg)
VARIABLE ELIMINATION
SAT
Letter
Job
![Page 35: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/35.jpg)
INDUCED GRAPH FROM A VE ORDERING
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
![Page 36: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/36.jpg)
INDUCED GRAPH FROM A VE ORDERING
Coherence
Difficulty Intelligence
Happy
Grade SAT
Letter
Job
Theorem:
• The scope of every intermediate factor in VE is a clique in the induced graph
• Every maximal clique in the induced graph is the scope of an intermediate factor
![Page 37: CS b553 : A lgorithms for Optimization and Learning](https://reader035.vdocuments.us/reader035/viewer/2022081517/56816134550346895dd08ef5/html5/thumbnails/37.jpg)
DETERMINING OPTIMAL ORDERINGS Again, NP hard! Good heuristics in practice:
Min-neighbors, min-fill, etc Search among elimination orderings while
counting size of introduced factors Greedy search often works well