an o(log(n))-approximation for decision trees brent heeringa [email protected] (joint work with...

An O(log(n))-Approximation for Decision Trees

Brent Heeringa

[email protected](joint work with Micah Adler)

11 March 2005

Question:• I am thinking of a Williams College CS faculty

member. Which one?

• Rule: Ask YES/NO questions from a finite set Q

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and a

TIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.




Q1: Is the professor female?

Q1: Is the professor female? YES




QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Q1YES NO


Q2: Does the professor drive an old Volvo?







Q1YES NO


Q2: Does the professor drive an old Volvo? NO







Q1YES NO

Q2YES

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

NO


are needed to see this picture.




Q3: Ph.D. from CMU?







Q1YES NO

Q2YES


NO






Q3: Ph.D. from CMU? YES





Q1YES NO

Q2YES


NO




Q3QuickTime™ and a


NOYES


Professor Barbara Lerner

Modeling the Professor Game

• Q = Set of m YES/NO questions• X = Set of possible professors = {Bailey, Bruce, …, Wyman}

= {10…1, 11…1, …, 01…00}

• Goal: Minimize average number of questions– Solution: Build a binary tree– Each professor is a leaf

m m m

profs are bits strings where bit k answers question k

Decision Tree Problem (DT)• Input: A set X=(x1,…,xn) of m-bit binary

strings (called items)

• Solution: A binary tree with n leaves– Each internal node is a bit k

• partitions items into two groups

– Each item is a leaf

• Cost: Total Sum of Leaf Depths

• Optimal Solution: DT with minimum cost

k0 1

Example:11110 10111 11010 01101

30 1

11010

01101

0

1

1

2

0

10111

1

11110

1

2

3 3

Cost: 1 + 2 + 3 + 3 = 9

Example:11110 10111 11010 01101

30 1

11010

01101

0

1

1

2

0

10111

1

11110

1

2

3 3

4

3

2 +

9

Example:11110 10111 11010 01101

50 1

11010 01101

0 11

1011111110

2

Cost: 2 + 2 + 2 + 2 = 8

OPTIMAL!0 13

2 2 2

Example:11110 10111 11010 01101

50 1

11010 01101

0 11

1011111110

2

0 13

2 2 2

4 + 2 + 2 = 8

Decision Trees• Decision Trees (DT) model many natural tasks in

– 20 Questions– Medical Diagnosis– Compiler optimizations

• DT is NP-Complete– No known polynomial-time algorithm (intractable)– As hard as Traveling Salesperson, 3SAT, etc.

• How do we deal with intractability?

Approximation AlgorithmsOptimization Problems (minimization)

Goal: minimize some costExamples: Traveling Salesperson, Max-3SAT

-approximation:

C = cost given by approximation algorithmCopt = cost of optimal solution

log(n)-approximation

• X be an instance of an optimization problemoptimal solution cost Copt

Approximation gives a solution w cost at most:

In DT, on input of size n, we know

So log of both sides is:

Outline

• Problem Introduction

• A Greedy Approximation Algorithm for DT

• An Analysis of the Greedy Algorithm– O(log n)-approximation

• Clearing up the historical confusion

A Greedy DT Algorithm

01101 10001 11101 11110 10111 11010

?

IDEA: Always choose bit which most evenly partitions items

0 1


01101 10001 11101 11110 10111 11010

4


0 1

01101 10001 11101 11110 10111 11010


4


0 1

11101

11110 10111 11010

1

01101 2

10001

01101 10001 11101 11110 10111 11010


4


0 1

1110111010

1

01101 2

10001

01101 10001 11101 11110 10111 11010

10111

2

11110

3

A Greedy DT AlgorithmIDEA: Always choose bit which

most evenly partitions items

GREEDY-DT(X)If X=Ø

Return NILElse

Let k be the bit most evenly separating XLet T be a tree nodeT[left] GREEDY-DT({X | X(k)=0})T[right] GREEDY-DT({X | X(k)=1})Return T

Optimal vs. Greedy

a

b c

h

e b

c df g

a

Optimal Tree T* Greedy Tree T

Cost(T)=26Cost(T*)=25

d eh

f g

Outline





Analysis Outline• Accounting Scheme

– Each pair of items {xi, xj} is separated exactly once in any decision tree

• Analyze the cost of greedy tree with respect to the structure of the optimal tree

Theorem: The greedy algorithm has cost at most a factor of O(log n) greater than the optimal tree

Accounting Method

• Divide size of each interior node Sij equally among the pairs of items {xi,xj} split at Sij

xi

Sij

Sij-

Node Sij separates xi from xj

Sij+

xj

|Sij+| ≥ |Sij

-|

Greedy Tree T

# pairs:

size of Sij:

Pair cost (cij):

Accounting Method

pairs of items

Claim:

Proof:

1.

2.

3.

xi

Sij

Sij-

Node Sij separates xi from xj

Sij+

xj

|Sij+| ≥ |Sij

-|

Greedy Tree T

Accounting MethodAccounting Method

01101 10001 11101 11110 10111

40 1

01101 10001 11101 11110 10111

x1 x2 x3 x4 x5

Sij-

Sij+

Cost(x1, x4) = c14 = 2/2 = 1

Sij

Sij =

Accounting Method

01101 10001 11101 11110 10111

40 1

01101 10001 11101 11110 10111

x1 x2 x3 x4 x5

Sij-

Sij+

Cost(x1, x4) = c14 = 2/2 = 1

Cost(x3, x4) = c34 = 2/2 = 1

Sij

Sij =

cij = 1

= 2 x 3 = 6 pairs of items

6 x cij = 6 x 1 = 6 ≥ |Sij| = 5

Bound holds:

Accounting Method

40 1

01101 10001 11101 11110 10111

Sij-

Sij+

Sij

cij = 2/3

= 3 x 3 = 9 pairs of items

9 x cij = 9 x (2/3) = 6 = |Sij| = 6

Bound holds:

Accounting Method

40 1

01101 10001 11101 11110 10111 00111

Sij-

Sij+

Sij

Accounting Method

Bound holds for all nodes Sij in T

Cost C of greedy tree T is the sum of each node size

Accounting Method

Bound holds for all nodes Sij in T

Any ordering of cij is acceptable.

Sum of all cij

Cost C of greedy tree T is the sum of each node size

Any ordering of cij is acceptable.

Sum of all cij

Reorder cij according to T*

Reorder cij according to T*

xi

Sij

Sij-Sij

+

xj

TT*

xi

Zij

Zij-Zij

+

xj

OPTIMAL

GREEDY

h

e b

c df g

a


a

b c d eh

f g

h

e b

c df g

a


a

b c d eh

f g

f,d f,e g,d g,e h,d h,e

cij:

h

e b

c df g

a


a

b c d eh

f g


2/4cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1 2/4cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1 2/4 2/1cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1 2/4 2/1cij: + + + + + = 7.5

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1 2/4 2/1+ + + + + = 7.5

7.5 ≥ | | = 5

cij:

h

e b

c df g

a


a

b c d eh

f g


2/4 2/1 2/4 2/1 2/4 2/1+ + + + + = 7.5

log(n)| | ≥ 7.5 ≥ | | = 5

cij:

h

e b

c df g

a


a

b c d eh

f g

Lemma: For any node Zij in T*


• Proof Intuition:– cij help capture trade-offs between a good move locally and a good

move globally– For any {xi,xj} the split at Sij is better than the split at Zij for the

pairs in common to both nodes– Enough wiggle room to show that a greedy split can’t lose too

much ground to an optimal split– For every step taken by the optimal algorithm, the greedy

algorithm takes at most log(n) steps

Reorder Cij according to T*

T*

xi

Zij

Zij-Zij

+

xj


OPTIMAL


T*

xi

Zij

Zij-Zij

+

xj

C* = Cost of optimal tree T*


OPTIMAL


T*

xi

Zij

Zij-Zij

+

xj


OPTIMAL

Outline





The ConDT Problem:• Input: A set X=(x1,…,xn) of m-bit binary strings (called items)

– Each item xi has a label TRUE or FALSE

• Solution: A binary tree– Each internal node is a bit k– Each leaf is a label– The tree correctly labels each item

• (consistent decision tree)

• Cost: Total number of leaves• Optimal Solution: Consistent decision tree with minimum

number of leaves




F F F FF F FTT

Each professor has a TRUE / FALSE label

Label answers some “hidden” question

Approximate “hidden” question with YES/NO questions




F F F FF F FTT





F F F FF F FTT





YES NO

FALSE

Q1




F F F FF F FTT


Q2: Ph.D. in mathematics?




YES NO

FALSE

Q1




F F F FF F FTT


Q2: Ph.D. in mathematics?




YES NO

FALSE

Q1

Q2QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

NOYES

FALSE TRUE




F F F FF F FTT

Hidden Question:

Does the professor have a beard?




YES NO

FALSE

Q1

Q2QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

NOYES

FALSE TRUE

Clarification

• NP-Complete• No polytime log(n)-approximation

– modulo unlikely complexity results

• Many consider DT and ConDT equivalent• log(n)-approximation proves otherwise

– DT and ConDT are fundamentally different problems

• log(n)-approximation is also first non-trivial upperbound on the aprox. ratio for DT

Mind the Gap!

• Lower bound on approximation for DT:– No PTAS– No -approximation for some constant

• Close the gap between upper bound and lower bound– No log(n)-approximation for some

an o(log(n))-approximation for decision trees brent heeringa [email protected] (joint work with...

Documents

cost of optimal solution

professor female

professor barbara lerner

greedy dt algorithm

finite set q slide

greedy approximation

approximation algorithm

leaf cost