secogis 20081 managing sensor data of urban traffic m. joliveau 1, f.de vuyst 1, g. jomier 2, c.m....

CADDYCADDY SeCoGIS 2008 1

Managing Sensor Data of Urban Traffic

M. Joliveau1, F.De Vuyst1, G. Jomier2,

C.M. Bauzer Medeiros3

ACI Masses de Données CADDY (2003-2007)

(1) MAS, Ecole Centrale de Paris

(2) LAMSADE, Université Paris-Dauphine

(3) IC, UNICAMP

CADDYCADDY SeCoGIS 2008 2

Goals Urban road traffic

analysis congestions Query the past

behavior Foresee the future

behavior Show understandable

résults

(Google Maps)

CADDYCADDY SeCoGIS 2008 3

Outline

Received Data Exploratory studies Deeper Analysis Work to do Concluding remarks

(Google Maps)

CADDYCADDY SeCoGIS 2008 4

Data about the system to be studied

- Graph with hundreds of sensors

- Flow rate, occupancy rate, 3’

- States: fluid (0) / congestion (1)

- Annotations

From INRETS

CADDYCADDY SeCoGIS 2008 5

Mass of Data

Sensor number (I)

Day number (J)

Number of measures in a day (K)

High rate of missing dataBad quality of dataSize order of the volume O(109) as I, J, K : O(103)

CADDYCADDY SeCoGIS 2008 6

Exploratory study

Temporal view

Space-time view : dynamic vizualization of the sensor

state map

Flow rate Occupancy Rate

Hours 0->24h

CADDYCADDY SeCoGIS 2008 7

Traffic States: fluid/congestion

It appears : 2 states are not enough to characterize the dynamic behavior of the system Urban Traffic 

Spatio-temporal patterns

CADDYCADDY SeCoGIS 2008 8

Space-Time Vizualization flow rate

x

Time

y

CADDYCADDY SeCoGIS 2008 9

Analysis of temporal series

Extract of one week for a sensor among 400 Regularity of the human activity generating traffic

CADDYCADDY SeCoGIS 2008 10

Schema of Data Base for Analysis

sensor-id

Sensors

day-id

Days

hour-id

Hours

annotation-id

Annotation

weather-id

Weather

sensor-idday-idhour-idannotation-idweather-id

Traffic

flow rateoccupancy ratetraffic state

CADDYCADDY SeCoGIS 2008 11

Symbolic representation of sets of temporal series

Symbol = label associated to a class reduction of size and intelligibility Class identification of typical behavior, detection of atypical behaviors

Episod partitionSymbol AlphabetSymbolic Representation

CADDYCADDY SeCoGIS 2008 12

Plan

Received Data Exploratory studies Deeper Analysis

STPCA Continuous Traffic

State Variable Concluding remarks

CADDYCADDY SeCoGIS 2008 13

STPCA Spatio-Temporal Principal Component Analysis

Goal : data representation in a reduced number of spatial dimensions => sensors temporal dimensions => daily instants Result :Data projection simultaneously on the first spatial and temporal

eigenmodes

1st experiment : Flow rate (Monday to Friday) for a family of reliable sensors

CADDYCADDY SeCoGIS 2008 14

Spatial Reduction

Xd (complete) matrix of daily realizations

element xi,t ,i sensor , t instant , d day

T number of instants by dayN number of daysI number of sensors

Y assembles horizontally N matrices Xd : Y = col (X1, X2,...... ,XN)

CADDYCADDY SeCoGIS 2008 15

Sensors Number

Number of Measure Instants

Daily Data

Matrix Y for spatial reduction

CADDYCADDY SeCoGIS 2008 16

Spatial Reduction

Y assembles horizontally N matrices Xd : Y = col (X1, X2,... ,XN)Each line is a temporal serie for 1 sensor

Singular value decomposition of Y Spatial correlation matrix: MS = YYT

Eigenvalues l1 >= l2 >= ... lKM

Eigenvectors (Fk) for k = 1…KM

CADDYCADDY SeCoGIS 2008 17

Spatial Reduction

Spatial correlation matrix Ms = YYT

Eigenvalues: λ1 ≥ λ2 ≥... λKM

Eigenvectors: Ψk for k = 1…KM

P matrix of the K first eigenvectors Ψk

P = col (Ψ1, Ψ2, ... ΨK) for K<< KM

CADDYCADDY SeCoGIS 2008 18

Spatial Reduction

Estimate X’d of each realization Xd :X’d = P PT Xd K reduced spatial orderReduced order matrix : Xr = PT Xcontains latent (hidden) variables of Xsize : K * T (T instants)

If T is large, the dimension of the reduced order representation is too large

CADDYCADDY SeCoGIS 2008 19

Temporal Reduction

Z assembles vertically N day realizations Xd :

Z = row (X1, X2,... ,XN)

one colon corresponds to one instant t

one line corresponds to one sensor i for one day d

the data of one day d are grouped

I* N lines

CADDYCADDY SeCoGIS 2008 20

Sensors Number

Number of Measure Instants

Daily Data

Matrix Z for temporal reduction

CADDYCADDY SeCoGIS 2008 21

Temporal Reduction

Z assembles vertically N day realizations Xd :

Z = row (X1, X2,... ,XN)

Singular value decomposition of Z Temporal correlation matrix Mt = ZTZ

Eigenvalues μ1 ≥ μ2 ≥ ... μLM

Eigenvectors (Φl) for l = 1, 2…LM

Q matrix of the L first eigenvectors Φl

Q = col (Φ1, Φ2, ...ΦL) for L << LM

CADDYCADDY SeCoGIS 2008 22

Temporal Reduction

Estimate X’ for each realization X:X’ = X Q QT

Reduced order matrix: Xr = XQcontains the latent variables of Xsize : I *L

If I (space : number of sensors) is large the dimension of the reduced order representation is too high

CADDYCADDY SeCoGIS 2008 23

Results of temporal component analysis

temps temps

temps temps

tempstemps

Mode 1

Mode 3

Mode 5

Mode 2

Mode 4

Mode 6

temps temps

temps temps

tempstemps

Mode 1

Mode 3

Mode 5

Mode 2

Mode 4

Mode 6

The 6 first temporal modes(ACP-t)-order :colon- definethe matrix Q (Jr=6)

CADDYCADDY SeCoGIS 2008 24

Reduction: projection on the first temporal mode

Flow rates, 1 work day, 6 sensors - Observed flow rate- Projection on the1rst temporal mode

tempstemps

tempstemps

temps temps

Cap

teur

1C

apte

ur 3

Cap

teur

5

Cap

teur

2C

apte

ur 4

Cap

teur

6

Time Time

Time Time

Time Time

Sens

or 1

Sens

or 2

Sens

or 3

Sens

or 4

Sens

or 5

Sens

or 6

CADDYCADDY SeCoGIS 2008 25

Spatio-Temporal Reduction

Combines spatial and temporal analysis

new estimate of each realization X

X’ = PPTXQQT

Reduced order matrix:

Xr =PTXQ

contains the latent variables of X

size K*L

CADDYCADDY SeCoGIS 2008 26

Cumulative Energy

Spatial correlation matrix

Eigenvalue Index

Eigenvalue Index

Cum

ulat

ive

Ene

rgy

Cum

ulat

ive

Ene

rgy

Temporal correlation matrix

CADDYCADDY SeCoGIS 2008 27

Sensor 1 Sensor 2 Sensor 3 Sensor 4

Sensor 5 Sensor 6 Sensor 7 Sensor 8

Sensor 9 Sensor 10 Sensor 11 Sensor 12

Sensor 13 Sensor 14 Sensor 15 Sensor 16

Work days K=3, L=3

CADDYCADDY SeCoGIS 2008 28

Mean Direct Error

Standard Deviation

Reduced-order Matrix

Size

Mean Direct Error

Standard Deviation

Reduced-order Matrix

Size

Mean Direct Error

Standard Deviation

Reduced-order Matrix

Size

CADDYCADDY SeCoGIS 2008 29

g

Sensor 1 Sensor 2 Sensor 3 Sensor 4

Sensor 5

Sensor 9

Sensor 13

Sensor 6

Sensor 10

Sensor 14

Sensor 7

Sensor 11

Sensor 15

Sensor 8

Sensor 12

Sensor 16

Chrismas Day K=3, L=3

CADDYCADDY SeCoGIS 2008 30

Error Distribution FunctionSensors

Num

ber

of S

enso

rs

CADDYCADDY SeCoGIS 2008 31

Error Distribution Function Days

Nu

mb

er

of

da

ys

CADDYCADDY SeCoGIS 2008 32

Plan

Received Data Exploratory studies Deeper Analysis

STPCA Continuous Traffic State

Variable Concluding remarks

CADDYCADDY SeCoGIS 2008 33

Generation of 7 new traffic states using analysis in phase space

Saturé

Fluide

Grandecirculation

Occ

upan

cy R

ate

Flow Rate

CADDYCADDY SeCoGIS 2008 34

Continuous traffic state variable

Occupancy rate

Throughput

CADDYCADDY SeCoGIS 2008 35

Sensor 1

Time (hour)

Time (hour)

Time (hour)

Flo

w R

ate

(n

b v

eh

icle

s)O

ccu

pa

ncy

Ra

te (

%)

Circ

ula

tion

Sta

te (

%)

CADDYCADDY SeCoGIS 2008 36

State Name Symb. State Num. Symbol E value at t Deriv. Sign in t

Calm

Negative

Very high level circ.

Saturation level 1

High level circul.

Saturation level 2

Saturation level 3

Positive

Back to Calm

New circulation states

CADDYCADDY SeCoGIS 2008 37

Dynamic Visualization of the Traffic State

Fluid

Congestion

Animation :spatio-temporal patterns appear

CADDYCADDY SeCoGIS 2008 38

Other results

Missing Data STPCA for state variables Spatio-temporal patterns

See Marc Joliveau ‘s PhD Thesis

CADDYCADDY SeCoGIS 2008 39

Work to be done

Enrich the datawarehouse with summaries, GIS, results of STPCA…

Symbolic spatio-temporal analysis Adaptation to evolution Visualization, user interaction Refinement on types of days, episodes Datawarehouse : queries

CADDYCADDY SeCoGIS 2008 40

Concluding Remarks Reduction : from data masses to intelligible and manipulable

elements

Generic Approach For spatio-temporal analysis of flow systems, described by data coming from a network of static

georeferenced sensors with diffuse sources and wells

CADDYCADDY SeCoGIS 2008 41

Future Prospects

Data coming from embarked sensors

Go farther in spatio-temporal reduction