advanced machine learning - nyu courantmohri/amls/aml_time_series.pdfadvanced machine learning -...

MEHRYAR MOHRI MOHRI@ COURANT INSTITUTE & GOOGLE RESEARCH..

GOOGLE RESEARCH..VITALY KUZNETSOV KUZNETSOV@

Advanced Machine LearningTime Series Prediction

pageAdvanced Machine Learning - Mohri@

MotivationTime series prediction:

• stock values.

• economic variables.

• weather: e.g., local and global temperature.

• sensors: Internet-of-Things.

• earthquakes.

• energy demand.

• signal processing.

• sales forecasting.

Google Trends

ChallengesStandard Supervised Learning:

• IID assumption.

• Same distribution for training and test data.

• Distributions fixed over time (stationarity).

none of these assumptions holds for time series!

OutlineIntroduction to time series analysis.

Learning theory for forecasting non-stationary time series.

Algorithms for forecasting non-stationary time series.

Time series prediction and on-line learning.

Introduction to Time Series Analysis

Classical FrameworkPostulate a particular form of a parametric model that is assumed to generate data.

Use given sample to estimate unknown parameters of the model.

Use estimated model to make predictions.

Autoregressive (AR) ModelsDefinition: AR( ) model is a linear generative model based on the th order Markov assumption:where

• s are zero mean uncorrelated random variables with variance .

• are autoregressive coefficients.

• is observed stochastic process.

8t, Yt =pX

aiYt�i + ✏t

a1, . . . , ap

Moving Averages (MA)Definition: MA( ) model is a linear generative model for the noise term based on the th order Markov assumption:where

• are moving average coefficients.

8t, Yt = ✏t +qX

bj✏t�j

b1, . . . , bq

ARMA modelDefinition: ARMA( , ) model is a generative linear model that combines AR( ) and MA( ) models:

(Whittle, 1951; Box & Jenkins, 1971)

8t, Yt =pX

aiYt�i + ✏t +qX

bj✏t�j

StationarityDefinition: a sequence of random variables is stationary if its distribution is invariant to shifting in time.

Z = {Zt}+1�1

(Zt, . . . , Zt+m) (Zt+k, . . . , Zt+m+k)

same distribution

Weak StationarityDefinition: a sequence of random variables is weakly stationary if its first and second moments are invariant to shifting in time, that is,

• is independent of .

• for some function .

Z = {Zt}+1�1

tE[Zt]

E[ZtZt�j ] = f(j)

Lag OperatorLag operator is defined by

ARMA model in terms of the lag operator:

Characteristic polynomialcan be used to study properties of this stochastic process.

L LYt = Yt�1.

P (z) = 1�pX

Weak Stationarity of ARMATheorem: an ARMA( , ) process is weakly stationary if the roots of the characteristic polynomial are outside the unit circle.

ProofIf roots of the characteristic polynomial are outside the unit circle then: where for all and are constants.

| i| > 1 i = 1, . . . , p c, c0

P (z) = 1�pX

aizi = c( 1 � z) · · · ( p � z)

= c0(1� �11 z) · · · (1� �1

ProofTherefore, the ARMA( , ) process admits MA( ) representation:where is well-defined since

1� �1

⇣� �1

i L⌘k

| �1i | < 1.

p q 1�

✓1� �1

◆�1

· · ·✓1� �1

◆�1✓1 +

◆✏t

ProofTherefore, it suffices to show thatis weakly stationary.

The mean is constant

Covariance function only depends on the lag :

Yt =1X

�j✏t�j

lE[YtYt�l]

E[YtYt�l] =1X

�k�jE[✏t�j✏t�l�k] =1X

�j�j+l

E[Yt] =1X

�jE[✏t�j ] = 0.

ARIMANon-stationary processes can be modeled using processes whose characteristic polynomial has unit roots.

Characteristic polynomial with unit roots can be factored:where has no unit roots.

Definition: ARIMA( , , ) model is an ARMA( , ) modelfor :

P (z) = R(z)(1� z)D

(1� L)DYt

p pq qD

! 1� L

ARIMANon-stationary processes can be modeled using processes whose characteristic polynomial has unit roots.

Characteristic polynomial with unit roots can be factored:where has no unit roots.

Definition: ARIMA( , , ) model is an ARMA( , ) modelfor :

Other ExtensionsFurther variants:

• models with seasonal components (SARIMA).

• models with side information (ARIMAX).

• models with long-memory (ARFIMA).

• multi-variate time series models (VAR).

• models with time-varying coefficients.

• other non-linear models.

Modeling VarianceDefinition: the generalized autoregressive conditional heteroscedasticity GARCH( , ) model is an ARMA( , ) model for the variance of the noise term :where

• s are zero mean Gaussian random variables with variance conditioned on .

• is the mean parameter.

�t ✏t

8t, �2t+1 = ! +

p�1X

↵i�2t�i +

q�1X

�j✏2t�j

✏t�t {Yt�1, Yt�2, . . .}

p pq q

(Engle, 1982; Bollerslev, 1986)

GARCH ProcessDefinition: the generalized autoregressive conditional heteroscedasticity (GARCH( , )) model is an ARMA( , ) model for the variance of the noise term :where

• s are zero mean Gaussian random variables with variance conditioned on .

• is the mean parameter.

State-Space ModelsContinuous state space version of Hidden Markov Models:where

• is an -dimensional state vector.

• is an observed stochastic process.

• and are model parameters.

• and are noise terms.

Xt+1 = BXt +Ut,

Yt = AXt + ✏t

Ut ✏t

State-Space Models

EstimationDifferent methods for estimating model parameters:

• Maximum likelihood estimation:

• Requires further parametric assumptions on the noise distribution (e.g. Gaussian).

• Method of moments (Yule-Walker estimator).

• Conditional and unconditional least square estimation.

• Restricted to certain models.

Invertibility of ARMADefinition: an ARMA( , ) process is invertible if the roots of the polynomial are outside the unit circle.

Q(z) = 1 +qX

Learning guaranteeTheorem: assume ARMA( , ) is weakly stationary and invertible. Let denote the least square estimate of and assume that is known. Then, converges in probability to zero.

Similar results hold for other estimators and other models.

baTkbaT � ak

qYt ⇠

a = (a1, . . . , ap)

NotesMany other generative models exist.

Learning guarantees are asymptotic.

Model needs to be correctly specified.

Non-stationarity needs to be modeled explicitly.

Theory

Time Series ForecastingTraining data: finite sample realization of some stochastic process,

Loss function: , where is a hypothesis set of functions mapping from to .

Problem: find with small path-dependent expected loss,

L : H ⇥ Z ! [0, 1]

(X1, Y1), . . . , (XT , YT ) 2 Z = X ⇥ Y.

L(h,ZT1 ) = E

⇥L(h, ZT+1)|ZT

Standard AssumptionsStationarity:

Mixing:

(Zt, . . . , Zt+m) (Zt+k, . . . , Zt+m+k)

same distribution

n n+ k

dependence between events decaying with k.

𝜷-MixingDefinition: a sequence of random variables is -mixing if

Z = {Zt}+1�1

n n+ k

dependence between events decaying with k.

�(k) = supn

EB2�n

A2�1n+k

��P[A | B]� P[A]��! 0.

Learning TheoryStationary and -mixing process: generalization bounds.

• PAC-learning preserved in that setting (Vidyasagar, 1997).

• VC-dimension bounds for binary classification (Yu, 1994).

• covering number bounds for regression (Meir, 2000).

• Rademacher complexity bounds for general loss functions (MM and Rostamizadeh, 2000).

• PAC-Bayesian bounds (Alquier et al., 2014).

Learning TheoryStationarity and mixing: algorithm-dependent bounds.

• AdaBoost (Lozano et al., 1997).

• general stability bounds (MM and Rostamizadeh, 2010).

• regularized ERM (Steinwart and Christmann, 2009).

• stable on-line algorithms (Agarwal and Duchi, 2013).

ProblemStationarity and mixing assumptions:

• often do not hold (think trend or periodic signals).

• not testable.

• estimating mixing parameters can be hard, even if general functional form known.

• hypothesis set and loss function ignored.

QuestionsIs learning with general (non-stationary, non-mixing) stochastic processes possible?

Can we design algorithms with theoretical guarantees?

need a new tool for the analysis.

Key Quantity - Fixed h

1 Tt T + 1

key difference

Key average quantity:��1

hL(h,ZT

1 )� L(h,Zt�11 )

i��.

L(h,Zt�11 ) L(h,ZT

DiscrepancyDefinition:

• captures hypothesis set and loss function.

• can be estimated from data, under mild assumptions.

• in IID case or for weakly stationary processes with linear hypotheses and squared loss (K and MM, 2014).

� = 0

� = suph2H

��L(h,ZT1 )�

L(h,Zt�11 )

��.

Weighted DiscrepancyDefinition: extension to weights .

• strictly extends discrepancy definition in drifting (MM and

Muñoz Medina, 2012) or domain adaptation (Mansour, MM,

Rostamizadeh 2009; Cortes and MM 2011, 2014); or for binary loss (Devroye et al., 1996; Ben-David et al., 2007).

• admits upper bounds in terms of relative entropy, or in terms of -mixing coefficients of asymptotic stationarity for an asymptotically stationary process.

(q1, . . . , qT ) = q

�(q) = suph2H

��L(h,ZT1 )�

qt L(h,Zt�11 )

��.

EstimationDecomposition: .

1 T�s T T + 1

�(q) suph2H

t=T�s+1

L(h,Zt�11 )�

qt L(h,Zt�11 )

+ suph2H

✓L(h,ZT

1 )�1

t=T�s+1

L(h,Zt�11 )

�(q) �0(q) +�s

Learning GuaranteeTheorem: for any , with probability at least , for all and ,

� > 0 1� �

where G = {z 7! L(h, z) : h 2 H}.

↵ > 0

L(h,ZT1 )

qtL(h, Zt) +�(q) + 2↵+ kqk2

r2 log

E[N1(↵,G, z)]�

Bound with Emp. DiscrepancyCorollary: for any , with probability at least , for all and ,

� > 0 1� �

h 2 H ↵ > 0

8>>>><

b�(q) = sup

t=T�s+1

L(h, Zt)�TX

qtL(h, Zt)

us unif. dist. over [T � s, T ]

G = {z 7! L(h, z) : h 2 H}.

L(h,ZT1 )

qtL(h, Zt) +b�(q) +�s + 4↵

hkqk2 + kq� usk2

⇥N1(↵,G, z)

Weighted Sequential α-CoverDefinition: let be a -valued full binary tree of depth . Then, a set of trees is an -norm -weighted -cover of a function class on if

��

" v1�g(z1)v3�g(z3)v6�g(z6)v12�g(z12)

#��1

(Rakhlin et al., 2010; K and MM, 2015)

g(z1), v1

g(z2), v2 g(z3), v3

g(z4), v4 g(z5), v5 g(z6), v6 g(z7), v7

g(z12), v12

+1�1

+1+1�1 �1

8g 2 G, 8� 2 {±1}T , 9v 2 V :TX

|vt(�)� g(zt(�))| ↵

Sequential Covering NumbersDefinitions:

• sequential covering number:

• expected sequential covering number:

N1(↵,G, z) = min{|V| : V l1-norm q-weighted ↵-cover of G}.

Z1, Z01 ⇠ D1

Z3, Z03 ⇠ D2(·|Z 0

1)Z2, Z02 ⇠ D2(·|Z1)

Z5, Z05

⇠ D3(·|Z1, Z02)

Z4, Z04

⇠ D3(·|Z1, Z2) ZT : distribution based on Zts.

Ez⇠ZT

[N1(↵,G, z)].

ProofKey quantities:

Chernoff technique: for any ,

�(q) = suph2H

��L(h,ZT1 )�

qt L(h,Zt�11 )

��.

�(ZT1 ) = sup

⇣L(h, ZT )�

qtL(h, Zt)⌘

P⇥�(ZT

1 ��(q) > ✏)⇤

qt⇥L(h,Zt�1

1 )� L(h, Zt)⇤> ✏

#(sub-add. of sup)

= P"exp

⇣t suph2H

qt⇥L(h,Zt�1

1 )� L(h, Zt)⇤⌘

> et✏#

(t > 0)

e�t✏ E"exp

⇣t suph2H

qt⇥L(h,Zt�1

1 )� L(h, Zt)⇤⌘

#. (Markov’s ineq.)

SymmetrizationKey tool: decoupled tangent sequence associated to .

• and i.i.d. given .

Z0T1 ZT

Zt Z 0t Zt�1

P⇥�(ZT

1 ��(q) > ✏)⇤

e�t✏ E"exp

⇣t suph2H

qt⇥L(h,Zt�1

1 )� L(h, Zt)⇤⌘

= e�t✏ E"exp

⇣t suph2H

qt⇥E[L(h, Z 0

t)|Zt�11 ]� L(h, Zt)

⇤⌘#

(tangent seq.)

= e�t✏ E"exp

⇣t suph2H

qt⇥L(h, Z 0

t)� L(h, Zt)⇤ ��ZT

i⌘#(lin. of expectation)

e�t✏ E"exp

⇣t suph2H

qt⇥L(h, Z 0

t)� L(h, Zt)⇤⌘

#. (Jensen’s ineq.)

Symmetrization

P⇥�(ZT

1 ��(q) > ✏)⇤

e�t✏ E"exp

⇣t suph2H

qt⇥L(h, Z 0

t)� L(h, Zt)⇤⌘

= e�t✏ E(z,z0)

⇣t suph2H

qt�t

⇥L(h, z0t(�))� L(h, zt(�))

⇤⌘#

(tangent seq. prop.)

= e�t✏ E(z,z0)

⇣t suph2H

qt�tL(h, z0t(�)) + t sup

qt�tL(h, zt(�))⌘#

(sub-add. of sup)

e�t✏ E(z,z0)

⇣2t sup

qt�tL(h, z0t(�))

⇣2t sup

(convexity. of exp)

= e�t✏ EzE�

⇣2t sup

Covering Number

P⇥�(ZT

1 ��(q) > ✏)⇤

e�t✏ EzE�

⇣2t sup

e�t✏ EzE�

⇣2thmaxv2V

qt�tvt(�) + ↵i⌘#

(↵-covering)

e�t(✏�2↵) Ez

v2VE�

qt�tvt(�)⌘##

(monotonicity of exp)

e�t(✏�2↵) Ez

v2Vexp

⇣ t2kqk2

⌘##(Hoe↵ding’s ineq.)

hN1(↵,G, z)

� t(✏� 2↵) +

t2kqk2

Algorithms

ReviewTheorem: for any , with probability at least , for all and ,

This bound can be extended to hold uniformly over at the price of the additional term: .

Data-dependent learning guarantee.

� > 0 1� �

eO(kq� uk1p

log2 log2(1� kq� uk)�1)

↵ > 0

L(h,ZT1 )

qtL(h, Zt) +b�(q) +�s + 4↵

hkqk2 + kq� usk2

⇥N1(↵,G, z)

Discrepancy-Risk MinimizationKey Idea: directly optimize the upper bound on generalization over and .

This problem can be solved efficiently for some and .

Kernel-Based RegressionSquared loss function:

Hypothesis set: for PDS kernel ,

Complexity term can be bounded by .

x 7! w · �K(x) : kwkH ⇤o

O⇣(log

3/2 T )⇤ sup

K(x, x)kqk2⌘

L(y, y0) = (y � y0)2

Instantaneous DiscrepancyEmpirical discrepancy can be further upper bounded in terms of instantaneous discrepancies:where and .

b�(q) TX

qtdt +Mkq� uk1

M = supy,y0

L(y, y0)

dt = suph2H

t=T�s+1

L(h, Zt)� L(h, Zt)

ProofBy sub-additivity of supremum

b�(q) = suph2H

t=T�s+1

L(h, Zt)�TX

qtL(h, Zt)

= suph2H

t=T�s+1

L(h, Zt)� L(h, Zt)

T� qt

t=T�s+1

L(h, Zt)

qt suph2H

t=T�s+1

L(h, Zt)� qtL(h, Zt)

!+Mku� qk1.

Computing DiscrepanciesInstantaneous discrepancy for kernel-based hypothesis with squared loss: .

Difference of convex (DC) functions.

Global optimum via DC-programming: (Tao and Ahn, 1998).

dt = supkw0k⇤

us(w0 · �K(xs)� ys)

2 � (w0 · �K(xt)� yt)2

Discrepancy-Based ForecastingTheorem: for any , with probability at least , for all kernel-based hypothesis and all .

Corresponding optimization problem: .

� > 0 1� �

h 2 H 0 < kq� uk1 1

minq2[0,1]T ,w

qt(w · K(xt)� yt)2 + �1

qtdt + �2kwkH + �3kq� uk1

L(h,ZT

1 ) TX

L(h, Zt

b�(q) +�

eO⇣log

3/2 T sup

K(x, x)⇤+ kq� uk1⌘

� > 0 1� �

h 2 H 0 < kq� uk1 1

minq2[0,1]T ,w

qt(w · K(xt)� yt)2 + �1

L(h,ZT

1 ) TX

L(h, Zt

b�(q) +�

eO⇣log

3/2 T sup

K(x, x)⇤+ kq� uk1⌘

� > 0 1� �

h 2 H 0 < kq� uk1 1

minq2[0,1]T ,w

qt(w · K(xt)� yt)2 + �1

L(h,ZT

1 ) TX

L(h, Zt

b�(q) +�

eO⇣log

3/2 T sup

K(x, x)⇤+ kq� uk1⌘

� > 0 1� �

h 2 H 0 < kq� uk1 1

minq2[0,1]T ,w

qt(w · K(xt)� yt)2 + �1

L(h,ZT

1 ) TX

L(h, Zt

b�(q) +�

eO⇣log

3/2 T sup

K(x, x)⇤+ kq� uk1⌘

� > 0 1� �

h 2 H 0 < kq� uk1 1

minq2[0,1]T ,w

qt(w · K(xt)� yt)2 + �1

L(h,ZT

1 ) TX

L(h, Zt

b�(q) +�

eO⇣log

3/2 T sup

K(x, x)⇤+ kq� uk1⌘

Convex ProblemChange of variable: .

Upper bound: .

• where .

• convex optimization problem.

rt = 1/qt

D = {r : rt � 1}

|r�1t � 1/T | T�1|rt � T |

minr2D,w

(w · K(xt)� yt)2 + �1dt

rt+ �2kwkH + �3

|rt � T |)

Two-Stage AlgorithmMinimize empirical discrepancy over (convex optimization).

Solve (weighted) kernel-ridge regression problem: where is the solution to discrepancy minimization problem.

⇤t (w · K(xt)� yt)

2 + �kwkH

b�(q)

Preliminary ExperimentsArtificial data sets:

True vs. Empirical Discrepancies

Discrepancies Weights

Running MSE

Real-world DataCommodity prices, exchange rates, temperatures & climate.

Time Series Prediction & On-line Learning

Two Learning ScenariosStochastic scenario:

• distributional assumption.

• performance measure: expected loss.

• guarantees: generalization bounds.

On-line scenario:

• no distributional assumption.

• performance measure: regret.

• guarantees: regret bounds.

• active research area: (Cesa-Bianchi and Lugosi, 2006; Anava et al. 2013, 2015, 2016; Bousquet and Warmuth, 2002; Herbster and Warmuth, 1998, 2001; Koolen et al., 2015).

On-Line Learning SetupAdversarial setting with hypothesis/action set .

For to do

• player receives .

• player selects .

• adversary selects .

• player incurs loss .

Objective: minimize (external) regret

ht 2 H

Tt = 1

xt 2 X

yt 2 Y

L(ht(xt), yt)

RegT =TX

L(ht(xt), yt)� minh2H⇤

L(h(xt), yt).

Example: Exp. Weights (EW)Expert set , .

EW({E1, . . . ,EN

})1 for i 1 to N do

2 w1,i 13 for t 1 to T do

4 Receive(xt

i=1 wt,iEiPNi=1 wt,i

6 Receive(yt

)7 Incur-Loss(L(h

))8 for i 1 to N do

t+1,i w

�⌘L(Ei(xt),yt). (parameter ⌘ > 0)

10 return h

H⇤ = {E1, . . . ,EN} H = conv(H⇤)

EW GuaranteeTheorem: assume that is convex in its first argument and takes values in . Then, for any and any sequence , the regret of EW at time satisfies

[0, 1] ⌘>0y1, . . . , yT 2 Y T

RegT logN

p8 logN/T ,

RegT p

(T/2) logN.

✓rlogN

EW - ProofPotential:

Upper bound:

�t = log

PNi=1 wt,i.

� �t�1 = log

i=1 wt�1,i e�⌘L(Ei(xt),yt)

i=1 wt�1,i

= log�

Ewt�1

[e�⌘L(Ei(xt),yt)]�

wt�1

✓�⌘

⇣L(E

)� Ewt�1

)]⌘� ⌘ E

wt�1

◆�◆

�⌘ Ewt�1

)] +⌘

8(Hoe↵ding’s ineq.)

�⌘L

wt�1

)], yt

8(convexity of first arg. of L)

= �⌘L(ht

) +⌘

EW - ProofUpper bound: summing up the inequalities yields

Lower bound:

Comparison:

�T � �0 �⌘

L(ht(xt), yt) +⌘

� �0 = log

�⌘

PTt=1 L(Ei(xt),yt) � logN

� log

�⌘

PTt=1 L(Ei(xt),yt) � logN

= �⌘

)� logN.

L(ht(xt), yt)�Nmin

L(Ei(xt), yt) logN

QuestionsCan we exploit both batch and on-line to

• design flexible algorithms for time series prediction with stochastic guarantees?

• tackle notoriously difficult time series problems e.g., model selection, learning ensembles?

Model SelectionProblem: given time series models, how should we use sample to select a single best model?

• in i.i.d. case, cross-validation can be shown to be close to the structural risk minimization solution.

• but, how do we select a validation set for general stochastic processes?

• use most recent data?

• use the most distant data?

• use various splits?

• models may have been pre-trained on .

Learning EnsemblesProblem: given a hypothesis set and a sample , find accurate convex combination with and .

• in most general case, hypotheses may have been pre-trained on .

t=1 qtht h 2 HA

q 2 �

on-line-to-batch conversion for general non-stationary non-mixing processes.

On-Line-to-Batch (OTB)Input: sequence of hypotheses returned after rounds by an on-line algorithm minimizing general regret

h = (h1, . . . , hT )

RegT =TX

L(ht, Zt)� infh⇤2H⇤

L(h⇤, Zt).

On-Line-to-Batch (OTB)Problem: use to derive a hypothesis with small path-dependent expected loss,

• i.i.d. problem is standard: (Littlestone, 1989), (Cesa-Bianchi et al.,

2004).

• but, how do we design solutions for the general time-series scenario?

h = (h1, . . . , hT ) h 2 H

LT+1(h,ZT1 ) = E

⇥L(h, ZT+1)|ZT

QuestionsIs OTB with general (non-stationary, non-mixing) stochastic processes possible?

Can we design algorithms with theoretical guarantees?

need a new tool for the analysis.

Relevant Quantity

Average difference:

1 T T + 1

key difference

LT+1(ht,ZT1 )

Lt(ht,Zt�11 )

hLT+1(ht,Z

T1 )� Lt(ht,Z

t�11 )

On-line DiscrepancyDefinition:

• : sequences that can return.

• : arbitrary weight vector.

• natural measure of non-stationarity or dependency.

• captures hypothesis set and loss function.

• can be efficiently estimated under mild assumptions.

• generalization of definition of (Kuznetsov and MM, 2015) .

HA Aq = (q1, . . . , qT )

disc(q) = suph2HA

��

qthLT+1(ht,Z

T1 )� Lt(ht,Z

t�11 )

i��.

Discrepancy EstimationBatch discrepancy estimation method.

Alternative method:

• assume that the loss is -Lipschitz.

• assume that there exists an accurate hypothesis :

⌘ = infh⇤

EhL(ZT+1, h

⇤(XT+1))|ZT1

i⌧ 1.

Discrepancy EstimationLemma: fix sequence in . Then, for any , with probability at least , the following holds for all :

ZT1 Z � > 0

1� � ↵ > 0

ddiscH(q) = suph2H,h2HA

��

qthL�ht(XT+1), h(XT+1)

�� L

�ht, Zt

�i��.

disc(q) ddiscHT (q) + µ⌘ + 2↵+Mkqk2

r2 log

E[N1(↵,G, z)]�

Proof Sketch

disc(q) = suph2HA

��

qthLT+1(ht,Z

T1 )� Lt(ht,Z

t�11 )

i��

suph2HA

��

LT+1(ht,Z

T1 )� E

hL(ht(XT+1), h

⇤(XT+1))��ZT

i��

+ suph2HA

��

qthEhL(ht(XT+1), h

⇤(XT+1))��ZT

i�� Lt(ht,Z

t�11 )

i��

µ suph2HA

qt EhL(h⇤(XT+1), YT+1)

��ZT1

+ suph2HA

��

qthEhL(ht(XT+1), h

⇤(XT+1))��ZT

i�� Lt(ht,Z

t�11 )

i��

= µ suph2HA

EhL(h⇤(XT+1), YT+1)

��ZT1

+ suph2HA

��

qthEhL(ht(XT+1), h

⇤(XT+1))��ZT

i�� Lt(ht,Z

t�11 )

i��.

Learning GuaranteeLemma: let be a convex loss bounded by and a hypothesis sequence adapted to . Fix . Then, for any , the following holds with probability at least for the hypothesis :

� > 0 1� �

t=1 qtht

ZT1 q 2 �

LT+1(h,ZT1 )

qtL(ht, Zt) + disc(q) +Mkqk2

r2 log

ProofBy definition of the on-line discrepancy,

is a martingale difference, thus by Azuma’s inequality, whp,

By convexity of the loss:

At = qthLt(ht, Z

t�11 )� L(ht, Zt)

qtLt(ht, Zt�11 )

qtL(ht, Zt) + kqk2q

1� .

qthLT+1(ht,Z

T1 )� Lt(ht,Z

t�11 )

i disc(q).

LT+1(h,ZT1 )

qtLT+1(ht,ZT1 ).

Learning GuaranteeTheorem: let be a convex loss bounded by and a set of hypothesis sequences adapted to . Fix . Then, for any , the following holds with probability at least for the hypothesis :

L M H⇤

� > 0 1� �

t=1 qtht

ZT1 q 2 �

LT+1(h,ZT1 )

h⇤2H

LT+1(h⇤,ZT

1 ) + 2disc(q) +RegT

+Mkq� uk1 + 2Mkqk2

r2 log

ConclusionTime series forecasting:

• key learning problem in many important tasks.

• very challenging: theory, algorithms, applications.

• new and general data-dependent learning guarantees for non-mixing non-stationary processes.

• algorithms with guarantees.

Time series prediction and on-line learning:

• proof for flexible solutions derived via OTB.

• application to model selection.

• application to learning ensembles.

ReferencesT. M. Adams and A. B. Nobel. Uniform convergence of Vapnik-Chervonenkis classes under ergodic sampling. The Annals of Probability, 38(4):1345–1367, 2010.

A. Agrawal, J. Duchi. The generalization ability of online algorithms for dependent data. Information Theory, IEEE Transactions on, 59(1):573–587, 2013.

P. Alquier, X. Li, O. Wintenberger. Prediction of time series by statistical learning: general losses and fast rates. Dependence Modelling, 1:65–93, 2014.

Oren Anava, Elad Hazan, Shie Mannor, and Ohad Shamir. Online learning for time series prediction. COLT, 2013.

ReferencesO. Anava, E. Hazan, and A. Zeevi. Online time series prediction with missing data. ICML, 2015.

O. Anava and S. Mannor. Online Learning for heteroscedastic sequences. ICML, 2016.

P. L. Bartlett. Learning with a slowly changing distribution. COLT, 1992.

R. D. Barve and P. M. Long. On the complexity of learning from drifting distributions. Information and Computation, 138(2):101–123, 1997.

ReferencesP. Berti and P. Rigo. A Glivenko-Cantelli theorem for exchangeable random variables. Statistics & Probability Letters, 32(4):385 – 391, 1997.

S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In NIPS. 2007.

T. Bollerslev. Generalized autoregressive conditional heteroskedasticity. J Econometrics, 1986.

G. E. P. Box, G. Jenkins. (1990) . Time Series Analysis, Forecasting and Control.

ReferencesO. Bousquet and M. K. Warmuth. Tracking a small set of experts by mixing past posteriors. COLT, 2001.

N. Cesa-Bianchi, A. Conconi, and C. Gentile. On the generalization ability of on-line learning algorithms. IEEE Trans. on Inf. Theory , 50(9), 2004.

N. Cesa-Bianchi and C. Gentile. Tracking the best hyperplane with a simple budget perceptron. COLT, 2006.

C. Cortes and M. Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519, 2014.

ReferencesV. H. De la Pena and E. Gine. (1999) Decoupling: from dependence to independence: randomly stopped processes, U-statistics and processes, martingales and beyond. Probability and its applications. Springer, NY.

P. Doukhan. (1994) Mixing: properties and examples. Lecture notes in statistics. Springer-Verlag, New York.

R. Engle. Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica, 50(4):987–1007, 1982.

ReferencesD. P. Helmbold and P. M. Long. Tracking drifting concepts by minimizing disagreements. Machine Learning, 14(1): 27-46, 1994.

M. Herbster and M. K. Warmuth. Tracking the best expert. Machine Learning, 32(2), 1998.

M. Herbster and M. K. Warmuth. Tracking the best linear predictor. JMLR, 2001.

D. Hsu, A. Kontorovich, and C. Szepesvári. Mixing time estimation in reversible Markov chains from a single sample path. NIPS, 2015.

ReferencesW. M. Koolen, A. Malek, P. L. Bartlett, and Y. Abbasi. Minimax time series prediction. NIPS, 2015.

V. Kuznetsov, M. Mohri. Generalization bounds for time series prediction with non-stationary processes. In ALT, 2014.

V. Kuznetsov, M. Mohri. Learning theory and algorithms for forecasting non-stationary time series. In NIPS, 2015.

V. Kuznetsov, M. Mohri. Time series prediction and on-line learning. In COLT, 2016.

ReferencesA. C. Lozano, S. R. Kulkarni, and R. E. Schapire. Convergence and consistency of regularized boosting algorithms with stationary β-mixing observations. In NIPS, pages 819–826, 2006.

Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT. 2009.

R. Meir. Nonparametric time series prediction through adaptive model selection. Machine Learning, pages 5–34, 2000.

ReferencesD. Modha, E. Masry. Memory-universal prediction of stationary random processes. Information Theory, IEEE Transactions on, 44(1):117–133, Jan 1998.

M. Mohri, A. Munoz Medina. New analysis and algorithm for learning with drifting distributions. In ALT, 2012.

M. Mohri, A. Rostamizadeh. Rademacher complexity bounds for non-i.i.d. processes. In NIPS, 2009.

M. Mohri, A. Rostamizadeh. Stability bounds for stationary φ-mixing and β-mixing processes. Journal of Machine Learning Research, 11:789–814, 2010.

ReferencesV. Pestov. Predictive PAC learnability: A paradigm for learning from exchangeable input data. GRC, 2010.

A. Rakhlin, K. Sridharan, A. Tewari. Online learning: Random averages, combinatorial parameters, and learnability. In NIPS, 2010.

L. Ralaivola, M. Szafranski, G. Stempfel. Chromatic PAC-Bayes bounds for non-iid data: Applications to ranking and stationary beta-mixing processes. JMLR 11:1927–1956, 2010.

C. Shalizi and A. Kontorovitch. Predictive PAC-learning and process decompositions. NIPS, 2013.

ReferencesA. Rakhlin, K. Sridharan, A. Tewari. Online learning: Random averages, combinatorial parameters, and learnability. NIPS,2010.

I. Steinwart, A. Christmann. Fast learning from non-i.i.d. observations. NIPS, 2009.

M. Vidyasagar. (1997). A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems. Springer-Verlag New York, Inc.

V. Vovk. Competing with stationary prediction strategies. COLT 2007.

ReferencesB. Yu. Rates of convergence for empirical processes of stationary mixing sequences. The Annals of Probability, 22(1):94–116, 1994.

advanced machine learning - nyu courantmohri/amls/aml_time_series.pdfadvanced machine learning -...

Documents

an adaptive mls surface for reconstruction with...

beyond amls: domain decomposition with...

· amls" affective characteristics field triage at-risk...

amls users manual

advanced machine learning - nyu...

better information in the legal services market · better...

enhancing eigenvector approximations of huge gyroscopic...

automated mud logging system ecotek...

will we ever cure breast...

advanced machine learning - nyu...

an implementation and evaluation of the amls method for

2018 gaems providers & educators conference - call for...

amls 2 edition

livestock demographic data group:...

1 distributing relevant information to a remotely-located...

amls - ksu

ahmad abdelrahim al dahiyat · 2019. 11. 3. · ahmad...

disability in the library setting anna ercoli schnitzer....

iterative projection methods for sparse … · iterative...

amls: amls advanced grupo emergentes 24 medical ......amls...