recognition of traffic lights in live video streams on mobile devices

Recognition of Traffic Lights in Live Video Streams on Mobile

Devices

Jan RotersXiaoyi JiangKai Rothaus

2011 IEEE Transactions on CSVT

OutlineIntroductionProblemsSystem Architecture

IdentificationClassificationVideo AnalysisTime-Based Verification

Experiment ResultsEvaluationsConclusion

IntroductionPeople with visual disabilities are limited in

mobility.Orientate pedestrians with zebra crossings at

intersectionsPortable PC with a digital camera and a pair of

auricular stereoPresent a system for mobile devices to help

sightless people cross roads.

ProblemsProgram usage

Real world conditionsCamera resolutionDifferent appearances

ProblemsThe scale of traffic lightsMany traffic lightsOccludedIlluminationRotation

Pedestrian Lights in Germany

1) Installation2) Shape3) Color arrangement4) Circuitry5) Background

Mobile Device & DatabasesNokia N95330MHZ ARM processor18Mb RAM320240

2 publicly available databaseGround truth segmentation was made manually

System Architecture

1.

2.

3.

4.

1. Localization

Red and Green Color Filter(1/3)

1. Analyze the data

Red and Green Color Filter(2/3)

2. Design the filter rules (ex : red traffic light)

The Gaussian distribution of the red cluster is defined by its mean color = (0.48,0.06,0.07) and has three eigenvectors

A color c = (r, g, b) is a red traffic light color when

Red and Green Color Filter(3/3)

3. Optimize parameters different parameter settings for each color Use 300 images to train Measure the quality of each setting by TP, FP, FN

Recall = , Precision =

Size/Circuitry FilterAssume the traffic light is 4 to 24 meters awayFixed camera focal length and possible aspect

ratios

1. Filter out regions that are too small or too large2. Vertical neighbor should not have different color

Background Color FilterInspect the region under a red light candidate or

above a green light candidateIf there are no dark pixels within search region,

refuse this candidate

Search region

Search region

Validation of LocalizationValidate the localization results with 201 images

Optimal Validationrecall precision recall precision

Red 76% 89.5% 71.8% 87%

green 85% 98.5% 83.3% 92.6%

Error = 33.7%

2. ClassificationTLC is the broadestTLC has the smallest distance to the top of imageNo other traffic light has similar height with TLC

Performance of Classification

Red GreenRecall 86.3% 86.3%

Precision 97.4% 98.1%

3. Video Analysis(1/2)Temporary OcclusionFalsified ColorsContradictory SceneRepeating Results

3. Video Analysis(2/2)Find the motion vector between two frames

Use KLT tracker to track feature pointsOnly search in a small area around crucial traffic light

candidate (30 pixels in each direction)Correlate the features by using SAD

Search region

Crucial traffic light

Candidate region

Feature point

𝑡𝑖 −1 𝑡𝑖

4. Time-Based VerificationReduce the false positive detections by comparing

2 kinds of resultsUse state queue with 4 scenarios

1) Identification and video analysis are both successful and the locations match with each other.

2) Identification and video analysis are successful but the locations are different.

3) Video analysis succeeds but identification fails.4) Video analysis fails but identification succeeds.

Experiment Results and Compute at least 5 frames per secondAt least consecutive correct detection with the

same color

Experiment Results

EvaluationsReliability

Prevent false positive green light detection

EvaluationsInteractivity

Temporal analysis reduce the interactivityThe feedback is normally given within 2 seconds

ConclusionThe system can be helpful on driver assistance

systemsLimited computational power on mobile devicesThe verification ideas can be improved