learning and fourier analysis grigory yaroslavtsev cis 625: computational learning theory

Learning and Fourier Analysis

Grigory Yaroslavtsevhttp://grigory.us

CIS 625: Computational Learning Theory

Fourier Analysis and Learning

• Powerful tool for PAC-style learning under uniform distribution over

• Sometimes requires queries of the form • Works for learning many classes of functions, e.g:– Monotone, DNF, decision trees, low-degree polynomials – Small circuits, halfspaces, k-linear, juntas (depend on small

# of variables)– Submodular functions (analog of convex/concave)– …

• Can be extended to product distributions over , i.e. where means that draws are independent

Boolean Functions

• Book: “Analysis of Boolean Functions”, Ryan O’Donnellhttp://analysisofbooleanfunctions.org• Notation switch:

• Example:

–

Fourier Expansion

• Example:

• For let character

• Thm: Every function can be uniquely represented as a multilinear polynomial

• Fourier coefficient of on

Orthonormal Basis: Proof

• Thm: Characters form an orthonormal basis in

• Since we have:– = 0 if – = 1 if

• This proves that are an orthonormal basis

Functions = Vectors, Inner Product

• Functions as vectors form a vector space:

• Inner product on functions = “correlation”:

where

Fourier Coefficients

• Recall linearity of dot products:

• Lemma:

Parsevel’s Theorem

• Parseval’s Thm: For any

If then • Example:

Plancharel’s Theorem

• Plancharel’s Thm: For any

• Proof:

Basic Fourier Analysis

• Mean – For we have

• Variance

(Parseval’s Thm)

Convolution

• Def.: For their convolution :

• Properties:1. 2. 3. For all

Convolution: Proof of Property 3

• Property 3: For all • Proof:

(def. convolution)

Approximate Linearity

• Def: is linear if– for all – such that for all

• Def: is approximately linear if1. for most 2. such that for most

• Q: Does 1. imply 2.?

Property Testing [Goldreich, Goldwasser, Ron; Rubinfeld, Sudan]

NO

YES

Randomized Algorithm

Accept with probability

Reject with probability

⇒

⇒⇒

YES

NO

Property Tester

-close

Accept with probability

Reject with probability

⇒

⇒Don’t care

-close : fraction has to be changed to become YES

• = class of linear functions

• -close:

Linearity Testing

⇒

Linear

Non-linear

Linearity Tester

-close

Accept with probability

Reject with probability

⇒

⇒Don’t care

Linearity Testing [Blum, Luby, Rubinfeld]

• BLR Test (given access to ):– Choose and independently– Accept if

• Thm: If BLR Test accepts with prob. then is -close to being linear

• Proof: – Apply notation switch – BLR accepts if

• if • if

Linearity Testing: Analysis

(def of convolution) (Plancharel’s thm)

Linearity Testing: Analysis Continued

• Recall • :

Local Correction• Learning linear functions takes queries• Lem: If is -close to linear function then for every

one can compute w.p. as:– Pick – Output

• Proof: By union bound:

Then

Thanks!