a quick introduction to the chow liu algorithm
DESCRIPTION
A very simple and quick introduction to the Chow-Liu algorithm.TRANSCRIPT
What is the Chow-Liu Algorithm? Reported by Chow and Liu (1968).
Also known as maximum weight spanning tree (MWST) algorithm or Kruskal’s algorithm.
Running time complexity is O(n2log(n)), where n is number of variables.
The probability distribution associated to the tree constructed by the Chow-Liu algorithm is the one that is closest to the probability distribution associated to the data as measured by the Kullback-Leibler divergence (Pearl and Dechter 1989).
One of many uses include constructing graphical relationships between variables.
Assumptions No missing data Variables are discrete Data is independently and identically distributed
2 Copyright 2009 by Jee Vang
Pseudo-code of Chow-Liu Algorithm Denote the following
A(Xi,Xj) : a measure of association between two variables, Xi and Xj
U = {X1, X2, …, Xn} : a set of n variables D : a database of cases of the variables in U T : a tree whose nodes have a 1-to-1 correspondence
to the variables in U Inputs to Chow-Liu algorithm
A, U, D Output of Chow-Liu algorithm
T
3 Copyright 2009 by Jee Vang
Pseudo-code of Chow-Liu Algorithm, Procedure
Begin i <- 1 //a counter, set to one L = {L1, L2, …, L((n+1)/2)-n} <- all pairwise
associations //L is a list of all pairwise associations in descending order; no self-pairing
P <- Li //P is a pointer to the first pair of variables in L construct T //create a disconnected tree with nodes
corresponding to variables in U Do until n-1 edges have been added
If a path does not exists between the pair of variables in P, add an undirected arc between them
P <- L(i+1) //set P to the next pair of variables in L
End4 Copyright 2009 by Jee Vang
Measure of Association Mutual information was originally used as the
measure of association (Chow and Liu 1968) However, any measure of association
satisfying the following property may be used Give three variables, Xi, Xj, and Xk, such that Xi
and Xk are conditionally independent given Xj, I(Xi,Xj,Xk), any measure of association, A(Xi,Xj), such that, min(A(Xi,Xj), A(Xj,Xk)) > A(Xi,Xk), can be used for the Chow-Liu algorithm (Acid and de Campos 1994).
5 Copyright 2009 by Jee Vang
Example—Inputs A(Xi,Xj) : mutual information between two
variables, Xi and Xj
U = {X1, X2, X3, X4, X5} : 5 variables Domains of X1 through X5 are {true, false}
D : database of cases of X1 through X5X1 X2 X3 X4 X5
true false false false false
false true true false true
false true false false true
… … … … …
6 Copyright 2009 by Jee Vang
Example—Procedure, Pairwise Associations Compute pairwise associations
L1,2 = 0.55 L1,3 = 0.34 L1,4 = 0.11 L1,5 = 0.59 L2,3 = 0.68 L2,4 = 0.03 L2,5 = 0.25 L3,4 = 0.01 L3,5 = 0.22 L4,5 = 0.10
Sort pairwise associations descendingly into list L L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4, L3,4}
7 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction, Empty Tree L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4} Construct disconnected tree, T, with 5 nodes
corresponding to variables in U
X2 X3
X5
X4
X1
8 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4}
Add arc between X2 and X3 in T, X2—X3
X3 X2
X5
X4
X1
9 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4}
Add arc between X1 and X5 in T, X1—X5
X3 X2
X5
X4
X1
10 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4}
Add arc between X1 and X2 in T, X1—X2
X3 X2
X5
X4
X1
11 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4} Skip adding arc between X1 and X3 //path exists
Skip adding arc between X2 and X5 //path exists
Skip adding arc between X3 and X5 //path exists
X3 X2
X5
X4
X1
12 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4}
Add arc between X1 and X4 in T, X1—X4
X3 X2
X5
X4
X1
13 Copyright 2009 by Jee Vang
Example—Procedure, Tree Construction (cont…) L = {L2,3, L1,5, L1,2, L1,3, L2,5, L3,5 ,L1,4 ,L4,5, L2,4,
L3,4} Final tree constructed
X3 X2
X5
X4
X1
14 Copyright 2009 by Jee Vang
References Chow, C.K., and Liu, C.N. Approximating discrete
probability distributions with dependence trees. IEEE Transactions on Information Theory, 14(3):462-467, 1968.
Pearl, J., and Dechter, R. Learning Structure from Data: A Survey. UCLA Cognitive Systems Laboratory, Technical Report CSD-910048 (R-132), June 1989.
Acid, S., and de Campos, L.M. Approximation of causal networks by polytrees. Proceedings of Information Processing and Management of Uncertainty in Knowledge-Based Systems, 1994.
15 Copyright 2009 by Jee Vang