Modern Methods in Decision Making (2016)

Assignment 2 - Solutions

Problem 4 in Problem Sheet 6. VC Theory.

(i) This is a simple consequence of the VC inequality, once you notice the relation R(y

n

)�inf

y2F R(y) 2 sup

y2F | ˆRn

(y)�R(y)| derived on page 16 of the lecture notes.

(ii) First, note that Jensen’s inequality ensures that (EZ)

2 E(Z2). Thus

E(Z2) =

Z 1

0

P(Z

2> s)ds =

Z

u

0

P(Z

2> s)ds+

Z 1

u

P(Z

2> s)ds

u+ c

Z 1

u

e

�2nsds

= u+

c

2n

e

�2nu.

The minimum is obtained for u = log c/(2n).

(iii) The relation established in (ii) is valid for all t > 0. Put t = ✏/16, for some ✏ > 0. Thenthe condition becomes

P(Z > t) = P(Z > ✏/16) = P(16Z > ✏) c e

�2nt2= c e

�n✏

2/128

.

This relation is satisfied for the random variable 16Z = |R(y

n

) � inf

y2F R(y)|, withc = 8S(F , n). Applying the lemma derived in (ii) yields

ER(y

n

)� inf

y2FR(y) = 16EZ 16

r

log(8eS(F , n))

2n

,

as required.

Problem 4 Part I in Problem Sheet 8. AdaBoost.See hand-written notes at the end

Problem 5 in Problem Sheet 9. Convex Optimization.

(i) The Lagrangian is

L(p,�, ⌫) =

X

i

p

i

log p

i

+ �

t

(Ap� b) + ⌫(1

t

p� 1) ,

and the Lagrange dual function is

g(�, ⌫) = inf

p

L(p,�, ⌫) = �b

t

�� ⌫ + inf

p

(

X

i

p

i

log p

i

+ (A

t

�+ 1

t

⌫)

t

p

)

.

Put (p) =P

i

p

i

log p

i

. The Lagrange dual function can be rewritten

g(�, ⌫) = �b

t

�� ⌫ �

⇤(�A

t

�� 1

t

⌫) ,

1

Modern Methods in Decision Making (2016)

where

⇤(y) = sup

x

(y

t

x� (x)) .

Note that in the special case (x) = x log x, we obtain ⇤(y) = e

y�1. Thus

g(�, ⌫) = �b

t

�� ⌫ �n

X

i=1

exp(�a

t

i

�� ⌫ � 1) ,

where a

i

denotes the i-th column of A. The dual problem is

maximize � b

t

�� ⌫ � e

�⌫�1n

X

i=1

e

�a

ti�

subject to A↵ + b� � = 0

� ⌫ 0 ,

The expression of the dual problem can be further simplified. For a fixed �,

@g(�, ⌫)

@⌫

= �1 + e

�⌫�1n

X

i=1

e

�a

ti�

= 0 ,

yields

⌫ = log

n

X

e

�(ati�+1)o

.

The function to maximise in the dual problem simplifies to

g(�) = �b

t

�� log

n

X

i=1

e

�a

ti�

!

.

(ii) Optimal gap is zero if there exists a p � 0 such that Ap � b and 1

t

p = 1.

2