artificial PErception under Adverse CONditions:

The Case of the Visibility Range

LCPC in cooperation with INRETS, France

Nicolas Hautière

Young Researchers Seminar 2007Brno, Czech Republic, 27 to 30 May 2007

Overview

• ADAS and adverse visibility conditions• Visibility range under daytime fog

Modelling Measurement methods Experimental validation

• Under way applications • Discussion and future works

PErception under Adverse CONditions

Nicolas Hautière, LCPC

ADAS and Adverse Visibility Conditions

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Sensor aspects• To improve the operation range of exteroceptive sensors• To qualify / adapt / stop the other driving assistances

Human aspects• To switch or to adapt the operation of vehicle lights (AFS)• To adapt the speed according to the weather conditions (ISA)

To detect the visibility conditions allows:

Improve the safety !

[Hautière and Aubert, 2005a] Hautière, N. and Aubert, D. (2005). Onboard evaluation of the atmospheric visibility for driving assistance systems, Recherche Transports Sécurité, 87:89-108.

Let’s talk about daytime fog

Daylight

Visibility Range under Daytime Fog

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Scattering

Atmospheric veil

• Koschmieder’s Law:

• Let express the contrast of an object against the sky:

Contrast attenuation

kkVmet

3)05.0log(

1

kd

f

f eCL

LLC

0

fLeeLL kdkd )1(0

“the greatest distance at which a black object of suitable dimensions can be recognized by aday against

the horizon sky” (CIE, 1987)

• For a black object (C0=1) and a visibility contrast threshold of 5%:

Direct transmission

Exploitation of the Atmospheric Veil

PErception under Adverse CONditions

Nicolas Hautière

Extraction of a region of interestFitting of a measurement bandwidth

VVmetmet = 50m = 50m

Estimation of the meteorological visibility distance

Measurement and derivation of intensity curveExtraction of the inflection point

B&W camera

zx f

d

SX

YZ

C

y

v

u vh

H

MRoad plane

Imageplane

hi vvk

dv

Ld

20

2

2

hvv

k

f eLLLL

100

Assuming a flat road:

hh

vvifvv

d

)(2

3

himet vv

V

vh horizon line, camera parameters

Method: Instanciation of Koschmieder’s Law

[Hautière et al., 2006a] Hautière, N., Tarel, J.-P, Lavenant, J. and Aubert, D. (2006). Automatic Fog Detection and Measurement of the visibility Distance through use of an Onboard Camera. Machine Vision Applications Journal, 17(1):8-20

Exploitation of Contrast Attenuation

PErception under Adverse CONditions

Nicolas Hautière, LCPC

• Estimation of the so-called mobilized visibility distance

• Range map obtained by “v-disparity” stereovision approach

• Visible objects are those having a local contrast above 5%

Method: computation of the range to the most distant visible object

f(x1)

f(x)

Cx,x1(s)s

x x1

F(s)

)(),(

1]255,0[

0

1

))(,())((

1max)(

sFxxssxxC

sFcardsC

)(,max

)(,

)(,max

)(min)(

1

1, 1 xfs

xfs

xfs

xfssC xx

Evaluated contrast on F(s) is equal to 2C(s0)

where

Local contrast measurement based on a binarisation method:

[Hautière et al., 2006b] Hautière, N., Labayrade, R. and Aubert, D. (2005). Real-Time Disparity Contrast Combination for Onboard Estimation of the Visibility Distance . IEEE Transactions on Intelligent Transportation Systems, 7(2):201-212.

Video samples

PErception under Adverse CONditions

Nicolas Hautière, LCPC

0

50

100

150

200

250

300

0 50 100 150

Time

Vis

ibili

ty d

ista

nce

[m]

0

50

100

150

200

250

300

0 200 400 600 800 1000Time

Vis

ibili

ty d

ista

nce

[m

]

Daytime fog Twilight fog

Experimental Validation

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Development of a validation site

• Objectives : To estimate Vmet thanks to

the targets:

To obtain a ground truth, To compare it with the in-

vehicle methods

)()(

)()(log

1

11

22

12 dLdL

dLdL

ddk

NB

NB

Triangle based pattern Constant solid angle 5 fixed targets:d=65m 1mx1md=98m 1.5mx1.5md=131m 2mx2md=162m 2.5mx2.5md=195m 3mx3m 1 mobile targe: 0.5mx0.5m

kVmet

3

Experimental Validation

• Content

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Sunny weather Vmet= 5000 m Light rain Vmet= 3400 m Haze Vmet= 2130 m

Snow fall Vmet= 1000 m Fog Vmet= 255 m Thick fog Vmet= 61 m

Sample images of the validation site

Experimental Validation

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Meteorological visibility estimation Mobilized visibility distance estimation

Quantitative results

[Hautière et al., 2006c] Hautière, N., Aubert, D., Dumont, E. and Tarel, J.-P. (2008). Experimental Validation of Dedicated Methods to In-Vehicle Estimation of Atmospheric Visibility. IEEE Transactions on Instrumentation and Measurement, 57(10), 2218-2225.

• Principle: reversal of Koschmieder’s law

• Assuming a flat world, we have:

Under Way Applications

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Improved Road Departure Prevention

)1(0kd

fkd eLLeL

i

h

hi

h

hi

vv

hiif

vv

vv

fvv

vv

dv

dLvvLL

eLLeL

2

122

0

Enhancement of road markings extraction under adverse visibility conditions

[Hautière and Aubert, 2005b] Hautière, N. and Aubert, D. (2005). Contrast Restoration of Foggy Images through use of an Onboard Camera, IEEE Conference on Intelligent Transportation Systems (ITSC’05), Vienna, Austria

d1=28m

d2=62m

• Contrast enhancement of the road scene

• Iterative contrast restoration

Under Way Applications

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Improved obstacle detection

22 )()(,min

hhh vvuuvvd

[Hautière et al., 2007] Hautière, N., Tarel, J.-P., Aubert, D. (2007). Towards Fog-Free In-Vehicle Vision Systems through Contrast Restoration. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’07), Minneapolis, USA

• Objective: to compute an adequate speed according to the weather conditions based solely on a digital map and a camera

• One of the requirements: a mapping function between the driver and the sensor visions:

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Under Way ApplicationsISA and “risk mitigation” or “safety margin”

[Hautière and Aubert, 2006d] Hautière, N. and Aubert, D. (2006). Visible Edges Thresholding: a HVS based Approach, International Conference on Pattern Recognition (ICPR’06), Hong-Kong, China

)0,0(),(

*

)(

)(

jiijij

R

CTFbA

dfCTFBA

Discussion, Current and Future works

• We have presented two methods, issued from the ARCOS French project, to estimate the visibility range and some applications,

• Methods are being extended in the REACT project by the Mines de Paris to develop probe vehicles,

• Currently, we are adapting the methods to the use of fixed CCTV cameras in the SAFESPOT IP,

• In the future, we would like to apply our scientific processes to other adverse visibility conditions, like:

PErception under Adverse CONditions

Nicolas Hautière, LCPC

Rain Glare Nocturnal Fog

This is the heart of the FP7 PEACON proposal lead by LCPC/INRETS !

Acknowledgments

PErception under Adverse CONditions

Nicolas Hautière, LCPC

• I would like to acknowledge the contributions of my colleagues Didier Aubert, Jean-Philippe Tarel, Raphaël Labayrade, Benoit Lusetti, Eric Dumont from LCPC and INRETS, of Michel Jourlin from the University of Saint-Etienne, and of Clément Boussard from the Mines Paris.