driver’s view and vehicle surround estimation using omnidirectional video stream abstract our...
TRANSCRIPT
Driver’s View and Vehicle Surround Estimation using Omnidirectional Video Stream
Abstract
Our research is focused on the development of novel machine vision based telematic systems, which provide non-intrusive probing of the state of the driver and driving conditions. In this paper we present a system which allows simultaneous capture of the driver's head pose, driving view, and surroundings of the vehicle. The integrated machine vision system utilizes a video stream of full 360 degree panoramic field of view. The processing modules include perspective transformation, feature extraction, head detection, head pose estimation, driving view synthesis, and motion segmentation. The paper presents a multi-state statistical decision models with Kalman filtering based tracking for head pose detection and face orientation estimation. The basic feasibility and robustness of the approach is demonstrated with a series of systematic experimental studies.
Keywords: Driver head tracking, face orientation estimation, driver’s view generation, surround vehicle detection.
Kohsia S. Huang, Mohan M. Trivedi, and Tarak [email protected], [email protected], [email protected]
Computer Vision & Robotics Research (CVRR) LaboratoryUniversity of California at San Diego
La Jolla, CA 92093-0434
Research Objective
Accurate and real-time estimation of driver’s face orientation, driver’s view, as well as vehicle surround for a driver assistance system.
Perspective Transformation on
Driver’s Seat
Ellipse Search Window
To Face Orientation Estimation
Sub-sample and Grayscale
Edge Detection
Constrained Ellipse Detection (RHT) Face/Non-face
Classification (DFFS)
Equalization
Head Candidate Extraction
Predict Head Location in Next
Frame
Update Kalman Filter for Head
Tracking
Computation of Driver’s Viewing Direction
Direction of driver
Direction of car
0 degree360 degree 180 degree
0 degree of camera
Driver Head Detection and Tracking
Head Detection & Tracking
Head Tilting Compensation
Projection into Feature
Subspace
1 2 N
1 MGaussian Likelihood Functions
State Sequence
Head Detection & Tracking
Head Tilting Compensation
View-Based Face Orientation
Likelihood Fns.Pan/tilt angles to camera
Omnicamera
Viewing DirectionKalman
Filter
Orientation
ML
Estimation of Head Pose and Face OrientationScheme
1
Scheme 2(Future)
Face Orientation Estimation
Driver’s View
Generation
Head Detection &
Tracking
Results
Average Performance
DFFS Bound False Positive2500 9%2000 7%
Clip FramesError before KF Error after KF
NoteMean Std Mean Std
#1 200 -1° 8° -1° 7°
#2 75 -19° 27° 18° 24° Uneven illum.
#3 70 1° 7° 0° 8°
#4 30 16° 28° -15° 16° Face occlusion
#5 15 0° 19° 4° 7°
#6 15 -3° 8° -2° 3°
Head Detection before KF(DFFS Bound = 2500)
Setup 1(Side View)
Setup 2(Front View)
Rough RHT, 1 Epoch 32% 50%Rough RHT, 2 Epochs 52% 61%Extensive RHT, 10 Epochs 71% 79%RHT+Feedback, 10→1 Epoch 64% 73%RHT+Feedback, 10→2 Epochs 67% 87%Head Detection after KF: 100 %
Head Detection
Face Orientation
Head Detection &
Tracking
Face Orientation Estimation
Driver’s View
Generation
Current frame of the image, with estimated image motion in the area of interest.
Points used for estimation of ego-motion. Gray: inliers, White: outliers, Black: unused.
Normalized frame difference in the area of interest.
Output after post-processing and clustering.
CAN Bus Calibration
DelayMotion Transform
Parameters
Spatial/Temporal Gradients
Post Processing & Clustering
Obstacle Positions
Motion Parameter Correction
Omni-Video Stream
Flat-Plane Transform
Ego-Motion Compensation
Inverse Flat-Plane Tx.
Normalized Frame Difference
H
gx, gy, gt
x- , P- x, P
Surround Vehicle Detection
Surround Vehicle Detection
Bayesian Correction toMotion Parameters
• Approximate motion parameters obtained from calibration, CAN bus.• Planar motion compensation equation:
• Optical flow constraint satisfied under favorable conditions:
• Image motion is expressed parametrically in terms of motion parameters for a number of image points as:
• Correction performed by update similar to iterated extended Kalman filter:
0 tyyxx gugug
tppyppx gyygxxg
HHHH
zxh
x
vxhz
),'()'()(
...
)(
33321211
)ˆˆ()(ˆˆ 11
1
xxPxhzRHPxx
PHRHP11T
111T
iiiiiii
iii
TT zyxHzyx '''
Driver’s View Generation
Summary
• Simultaneous driver head detection, driver face pose estimation, driving view generation, and surround vehicle monitoring in real-time using a single omni-video stream.
• Suitable for novel televiewing interfaces, driver assistance systems, and driver distraction studies.