Desheng Liu, Maggi Kelly and Peng GongDept. of Environmental Science, Policy &

ManagementUniversity of California, Berkeley

May 18, 2005

Classifying Multi-temporal TM Classifying Multi-temporal TM Imagery Using Markov Random Imagery Using Markov Random

Fields and Support Vector Fields and Support Vector MachinesMachines

OutlineOutlineOutlineOutline

I. Introduction1. Two aspects of Multi-temporal

Imagery 2. Classification Models

II. Methods1. Support Vector Machines2. Markov Random Fields3. Spatio-temporal Classification

III.Results IV.Conclusions

Two Aspects of Multi-temporal Two Aspects of Multi-temporal ImageryImagery

Spatial Dependence– Pixels are not I.I.D.– Spatial

Autocorrelation

Temporal Correlation– Land Use– Phenology– Disturbance

Introduction

XX

YY

TT

Classification ModelsClassification Models

Non-contextual Model

Contextual Models– Spatial

– Temporal

– Spatio-temporal

Introduction

*

( ), ( ( ))( ) { ( ( ) ( ) )}

Spatialc i Cd c N ic i Arg Max P c i i

*

( ), ( ( ))( ) { ( ( ) ( ) )}

Tempc i Cd c N ic i Arg Max P c i i

*

( ), ( ( )) ( ( ))( ) { ( ( ) ( ) , )}

Spatial Tempc i Cd c N i c N ic i Arg Max P c i i

*

( )( ) { ( ( ) ( ))}

c i Cdc i Arg Max P c i i : ; : c class label d spectral value

( ) : ; ( ) : Spatial Temporal

N i spatial neighbors of pixel i N i temporal neighbors of pixel i

Generative Spatio-temporal Generative Spatio-temporal ModelsModels

Estimation of conditional probability– Maximum Likelihood Classifier (MLC)– Support Vector Machines (SVM)

Modeling spatio-temporal context – Markov Random Fields (MRF)

Introduction

, ( ( )) ( ( )) ( ( )) ( ( ))( ( ) ( ) , ) ( ( ) ( )) ( ( ) , ) spatial temp spatial temp

d c N i c N i d c N i c N iP c i i P c i i P c i

SVM: A Graphic View SVM: A Graphic View (1)(1)

Linear Cases: find the optimal linear separating boundary with (a) maximum margin ρ (b) best trade-off between maximum margin ρ and minimum classification errors ξ ρ

Methods

ρ

ξj

ξi

(a) (b)

SVM: A Graphic View SVM: A Graphic View (2)(2)

Non-Linear Cases: find the optimal linear separating boundary in a transformed higher dimensional feature space

Φ(x)

Methods

Methods

SVM: A Mathematic SVM: A Mathematic View (1)View (1)

Training samples: Decision function: Discriminant function

– Linear cases:

– Nonlinear cases:

Probability output:

1

( , ) , 1,1

n

di i i i

ix y x R y

( ( ))y sign f x

( ) ,

i i ii SV

f x y x x b

( ) ( ), ( )

,

i i i

i SV

i i ii SV

f x y x x b

y K x x b

( )

1( 1 )

1 f x

p y xe

Binary Cases:Binary Cases:

Methods

SVM: A Mathematic SVM: A Mathematic View (2)View (2)

Combination of binary SVM– “One-versus-one”– “One-versus-all”

Probability output– Pairwise coupling of binary probability

outputs– Soft-max function

Multi-category Cases:Multi-category Cases:

( )

( )

1

( )

i i i

i i i

f x

K f x

i

ep y i x

e

Markov Random FieldsMarkov Random Fields

Markov Random FieldsMarkov Random Fields (MRF) ---(MRF) ---

Probabilistic image models which define the inter-Probabilistic image models which define the inter-pixel contextual information in terms of the pixel contextual information in terms of the conditional priorconditional prior probability of a pixel given its probability of a pixel given its neighboring pixelsneighboring pixels

Methods

1, if ( ) ( )( ( ), ( )) ; ( ( ) ( )) : ( ) ( )

0, if ( ) ( )

, : / Spatial Temp

c i c jI c i c j P c i c k transition probability of c k c i

c i c j

parameters to control the influnces of spatial temporal neighbors

( ( ))1( ( ) ( ( )) ( ( ))),

spatial temp

U c iP c i c N i c N i eZ

( ) ( )

( ( )) ( ( ), ( )) ( ( ) ( ))

spatial temp

Spatial Tempj N i k N i

U c i I c i c j P c i c k

: ; : Z normalizing constant U energy function

Time 1

Time 2

S1 S2 S3

S4 S5

S6 S7 S8

S1 S2 S3

S4 S5

S6 S7 S8

P

T1 T2 T3

T4 T5 T6

T7 T8 T9

T1 T2 T3

T4 T5 T6

T7 T8 T9

Bayes’ Decision RuleBayes’ Decision Rule:: Maximum a posterior (MAP)Maximum a posterior (MAP)

MAP-MRFMAP-MRF: : the joint formulation of MAP and MRFthe joint formulation of MAP and MRF

MAP-MRF MAP-MRF

Methods

( ( ) ( ) , ( ( )) ( ( ))) ( ( ) ( )) ( ( ) ( ( )) ( ( )))

1

, ,

spatial temp spatial temp

spectral context

P c i d i c N i c N i P d i i P c i c N i c N i

U Ue

Z

c

( ) { ( ( ) ( ) , ( ( )) ( ( )))}( )

, spatial temp

c i Arg Max P c i d i c N i c N ic i C

ln( ( ( ) ( ))), ( ( )) spectral contextwhere U P d i i U U c ic

( ) { }( )

spectral contextc i Arg Min U U

c i CMAP-MRFMAP-MRF

Spatio-temporal Spatio-temporal ClassificationClassification

Methods

( ) { ( ( ) ( ) , ( ( )) ( ( )))}( )

{ ( ( ) ( )) ( ( ) ( ( )) ( ( )))}( )

,

,

spatial temp

spatial temp

c i Arg Max P c i d i c N i c N ic i C

Arg Max P d i i P c i c N i c N ic i C

c

Conditional Probability Conditional Prior

Support Vector Machines Markov Random Fields

MAP-MRF

( )1

( )2

( 1)( ) ( ) ( ) ( 1)

11 1 1 2

( 1)( ) ( ) ( ) ( 1)

22 2 2 1

1

2

1 ( )

2 ( )

ˆ( ( ) ( )) ( ( ) ( ( )), ( ( ))

ˆ( ( ) ( )) ( ( ) ( ( )), ( ( ))

ˆ ( ) { }

ˆ ( ) { }

spatial temp

spatial temp

t

t

tt t t t

tt t t t

c i C

c i C

P d i c i P c i c N i c N i

P d i c i P c i c N i c N i

c i Arg Max

c i Arg Max

Iterative Conditional Mode (ICM)Iterative Conditional Mode (ICM)

iteratively estimate the class label of each pixel given the iteratively estimate the class label of each pixel given the estimates of all its neighbors estimates of all its neighbors

Implementation Implementation AlgorithmAlgorithm

Methods

TM Imagery of June 11, 1997

Results

Data and Study SiteData and Study SiteSan Bernardino National Forest,

CA

Results

Data and Study SiteData and Study SiteSan Bernardino National Forest,

CATM Imagery of June

10, 2002

Classification FlowClassification Flow

Results

TM Image

Convergence?

SVM

Conditional probability

Classification(intermediate)

MAP-MRF

SVM

Conditional probability

Classification(intermediate)

Convergence?

1997 2002

Yes Yes

No No

Initialization Initialization

MAP-MRF

TM Image

Classification(Final)

Classification(Final)

Fire Perimeter

Training Sample Test Sample Class Name 1997 2002 1997 2002

Bare Land 1069 1363 356 454 Conifer 3162 2352 1054 784 Conifer Open 1355 2618 452 873 Hardwood 3443 2231 1148 744 Hardwood Open 1016 992 339 331 Herbaceous 503 417 168 139 Shrub 4153 2920 1384 973 Residential 1352 1119 451 373 Water 512 512 171 171

Results

Training/Test SamplesTraining/Test Samples

Results

Classification Accuracies of TM Classification Accuracies of TM 19971997

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

Non-Context Spatial-Only Temporal-Only Spatio-Temporal

MLC

SVM

Results

Classification Accuracies of TM Classification Accuracies of TM 20022002

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

85.00%

90.00%

95.00%

Non-Context Spatial-Only Temporal-Only Spatio-Temporal

MLC

SVM

Results

Convergence RateConvergence Rate

1 2 3 4 5 6 7 8 9 100

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Iteration

Ch

ang

e R

ate

(%)

MLC 97

MLC 02

SVM 97

SVM 02

Results 19971997Original Image MLC MLC-Spatio-Temp

SVM SVM-Spatio-TempBare Land

Conifer

Conifer Open

Hardwood

Hardwood Open

Herbaceous

Shrub

Residential

Water

Results 20022002Original Image MLC MLC-Spatial-Temp

SVM SVM-Spatial-TempBare Land

Conifer

Conifer Open

Hardwood

Hardwood Open

Herbaceous

Shrub

Residential

Water

ConclusionsConclusions

SVM are much better in the processing of spectral data than MLC for the initialization of the iterative algorithm.

MRF are efficient probabilistic models for the analysis of spatial / temporal contextual information.

The combination of SVM and MRF unifies the strengths of two algorithms and leads to an improved integration of the spectral, spatial and temporal components of multi-temporal remote sensing imagery.

AcknowledgementsAcknowledgements

USDA Forest Service

NASA Earth System Science Graduate Student Fellowship

Thank you!Thank you!