consistent and efficient reconstruction of latent tree models
DESCRIPTION
Stochastic Systems Group. Consistent and Efficient Reconstruction of Latent Tree Models. Myung Jin Choi Joint work with Vincent Tan, Anima Anandkumar, and Alan S. Willsky Laboratory for Information and Decision Systems Massachusetts Institute of Technology September 29, 2010. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/1.jpg)
Consistent and Efficient Reconstructionof Latent Tree Models
Myung Jin Choi
Joint work with Vincent Tan, Anima Anandkumar, and Alan S. Willsky
Laboratory for Information and Decision Systems Massachusetts Institute of Technology
September 29, 2010
Stochastic Systems Group
![Page 2: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/2.jpg)
Latent Tree Graphical Models
Dell
CVS Disney
Microsoft
Apple
![Page 3: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/3.jpg)
Latent Tree Graphical Models
Dell
CVS Disney
Microsoft
Apple
Computer
Computer Equipment
Market
![Page 4: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/4.jpg)
![Page 5: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/5.jpg)
Outline
– Reconstruction of a latent tree
– Algorithm 1: Recursive Grouping
– Algorithm 2: CLGrouping
– Experimental results
![Page 6: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/6.jpg)
Reconstruction of a Latent Tree
Reconstruct a latent tree using samples of the observed nodes.
• Gaussian model:
each node – a scalar
Gaussian variable
• Discrete model:
each node – a discrete
variable with K states
![Page 7: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/7.jpg)
Minimal Latent Trees (Pearl, 1988)
Conditions for Minimal Latent Trees• Each hidden node should have at least three neighbors.• Any two variables are neither perfectly dependent nor
independent.
![Page 8: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/8.jpg)
Desired Properties for Algorithms
1. Consistent for minimal latent trees
Correct recovery given exact distributions.
2. Computationally efficient
3. Low sample complexity
4. Good empirical performance
![Page 9: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/9.jpg)
Desired Properties for Algorithms
1. Consistent for minimal latent trees
Correct recovery given exact distributions.
2. Computationally efficient
3. Low sample complexity
4. Good empirical performance
![Page 10: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/10.jpg)
Related Work
• EM-based approaches– ZhangKocka04, HarmelingWilliams10, ElidanFriedman05– No consistency guarantees– Computationally expensive
• Phylogenetic trees− Neighbor-joining (NJ) method
(SaitouNei87)
![Page 11: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/11.jpg)
Information Distance
• Gaussian distributions
![Page 12: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/12.jpg)
Information Distance
• Gaussian distributions
• Discrete distributions
ex)
Joint probability matrix Marginal probability matrix
![Page 13: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/13.jpg)
Information Distance
• Gaussian distributions
• Discrete distributions
• Algorithms use information distances of observed variables.• Assume first that the exact information distances are given.
Joint probability matrix Marginal probability matrix
![Page 14: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/14.jpg)
Additivity of Information Distances on Trees
![Page 15: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/15.jpg)
Testing Node Relationships
Node j – a leaf node
Node i – parent of j
for all k ≠ i, j .
Can identify
(parent, leaf child) pair
![Page 16: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/16.jpg)
Testing Node Relationships
Node i and j – leaf nodes
and share the same parent
(sibling nodes)
for all k ≠ i, j .
Can identify leaf-sibling pairs.
![Page 17: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/17.jpg)
Recursive Grouping
Step 1. Compute for all observed nodes (i, j,
k).
![Page 18: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/18.jpg)
Recursive Grouping
Step 2. Identify (parent, leaf child) or (leaf siblings) pairs.
![Page 19: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/19.jpg)
Recursive Grouping
Step 3. Introduce a hidden parent node for each sibling
group without a parent.
![Page 20: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/20.jpg)
Recursive Grouping
Step 4. Compute the information distance for new hidden nodes.
e.g.)
![Page 21: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/21.jpg)
Recursive Grouping
Step 5. Remove the identified child nodes and repeat
Steps 2-4.
![Page 22: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/22.jpg)
Recursive Grouping
Step 5. Remove the identified child nodes and repeat
Steps 2-4.
![Page 23: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/23.jpg)
Recursive Grouping
Step 5. Remove the identified child nodes and repeat
Steps 2-4.
![Page 24: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/24.jpg)
Recursive Grouping
• Identifies a group of family nodes at each step.
• Introduces hidden nodes recursively.
• Correctly recovers all minimal latent trees.
• Computational complexity O(diam(T) m3).
• Worst case O(m4)
![Page 25: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/25.jpg)
CLGrouping Algorithm
![Page 26: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/26.jpg)
Chow-Liu Tree
Minimum spanning tree of V
using D as edge weights
V = set of observed nodes
D = information distances
![Page 27: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/27.jpg)
Chow-Liu Tree
Minimum spanning tree of V
using D as edge weights
V = set of observed nodes
D = information distances
• Computational complexity O(m2 log m)
• For Gaussian models, MST(V; D) = Chow-Liu tree
(minimizes KL-divergence to the distribution given by D).
![Page 28: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/28.jpg)
Surrogate Node
V = set of observed nodes
Surrogate node of i
![Page 29: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/29.jpg)
Property of the Chow-Liu Tree
![Page 30: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/30.jpg)
CLGrouping Algorithm
Step 1. Using information distances of observed nodes,
construct the Chow-Liu tree, MST(V; D). Identify the
set of internal nodes {3, 5}.
![Page 31: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/31.jpg)
CLGrouping Algorithm
Step 2. Select an internal node and its neighbors, and
apply the recursive-grouping (RG) algorithm.
![Page 32: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/32.jpg)
CLGrouping Algorithm
Step 3. Replace the output of RG with the sub-tree
spanning the neighborhood.
![Page 33: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/33.jpg)
CLGrouping Algorithm
Repeat Steps 2-3 until all internal nodes are operated on.
![Page 34: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/34.jpg)
CLGrouping
• Step 1: Constructs the Chow-Liu tree, MST(V; D).
• Step 2: For each internal node and its neighbors,
applies latent-tree-learning subroutines (RG or NJ).• Correctly recovers all minimal latent trees.
• Computational complexity
O(m2 log m + (#internal nodes) (maximum degree)3).
O(m2 log m)
![Page 35: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/35.jpg)
Sample-based Algorithms
• Compute the ML estimates of information distances.
• Relaxed constraints for testing node relationships.
• Consistent.
• More details in the paper
![Page 36: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/36.jpg)
Experimental Results
• Simulations using Synthetic Datasets
- Compares RG, NJ, CLRG, and CLNJ.
- Robinson-Foulds Metric and KL-divergence.
![Page 37: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/37.jpg)
![Page 38: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/38.jpg)
![Page 39: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/39.jpg)
![Page 40: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/40.jpg)
Performance Comparisons
• For a double star, RG is clearly the best.
• NJ is poor in recovering HMM.
• CLGrouping performs well in all three structures.
• Average running time for CLGrouping < 1 second.
![Page 41: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/41.jpg)
Monthly Stock Returns
• Monthly returns of 84 companies in S&P 100.
• Samples from 1990 to 2007.
• Latent tree learned using CLNJ.
![Page 42: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/42.jpg)
![Page 43: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/43.jpg)
![Page 44: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/44.jpg)
![Page 45: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/45.jpg)
![Page 46: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/46.jpg)
20 Newsgroups with 100 Words
• 16,242 binary samples of 100 words
• Latent tree learned using regCLRG.
![Page 47: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/47.jpg)
![Page 48: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/48.jpg)
![Page 49: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/49.jpg)
![Page 50: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/50.jpg)
![Page 51: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/51.jpg)
Contributions
• Recursive-grouping- Identifies families and introduces hidden nodes recursively.
• CLGrouping- First learns the Chow-Liu tree - Then applies latent-tree-learning subroutines locally.
![Page 52: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/52.jpg)
Contributions
• Recursive-grouping• CLGrouping
• Consistent.• CLGrouping - superior experimental results in both accuracy and
computational efficiency.
• Longer version of the paper and MATLAB implementation available at the project webpage.
http://people.csail.mit.edu/myungjin/latentTree.html
![Page 53: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/53.jpg)
Testing Node Relationships
for all k ≠ i, j .
constant
![Page 54: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/54.jpg)
Edge Contractions
Contract edge (h1, x1)
Contract edge (h1, x1) and then (h2, x1)
![Page 55: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/55.jpg)
CLGrouping and Edge Contractions
Chow-Liu tree = Obtained by
contracting each hidden node and
its surrogate node.
CLGrouping reverses edge contraction operations.
(a) (b) (c) (d)
![Page 56: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/56.jpg)
Desired Properties for Algorithms
![Page 57: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/57.jpg)
Desired Properties for Algorithms
1. Consistency Correct recovery given exact
distributions.
![Page 58: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/58.jpg)
Desired Properties for Algorithms
1. Consistency Correct recovery given exact
distributions.
2. Computational Efficiency
![Page 59: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/59.jpg)
Desired Properties for Algorithms
1. Consistency Correct recovery given exact
distributions.
2. Computational Efficiency
3. Sample Complexity
![Page 60: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/60.jpg)
CLGrouping Algorithm
Step 3. Replace the output of RG with the sub-tree
spanning the neighborhood.
![Page 61: Consistent and Efficient Reconstruction of Latent Tree Models](https://reader036.vdocuments.us/reader036/viewer/2022062519/56815286550346895dc0ac4c/html5/thumbnails/61.jpg)
CLGrouping Algorithm
Step 3. Replace the output of RG with the sub-tree
spanning the neighborhood.