geo-spatial aerial processing for scene understanding and object tracking

Geo-Spatial Aerial Processing for Scene Understanding and Object Tracking

Jiangjian Xiao, Hui Cheng, Feng Han, Harpreet Sawhney

Problem Given Aerial Video Understand the Scene

Find buildings Trees Roads Cars

Use understanding Object Detection Tracking

Cool Idea Trees and buildings are in

3D

Related Work CVPR 2006

Hui Cheng, Darren Butler and Chumki Basu

ViTex: Video To Tex and Its Applications in Aerial Video Survellance.

Related Work CVPR2008

Jake Porway, Kristy Wang, Benjamin Yao, Song Chun ZhuA Hierarchical and Contextual Model for Aerial Image Understanding

System Overview

Input Frames

Geo-reference image

Initial camera location

Geo-registration Pose estimation

Depth estimation

Non-ground object detection

Planar + depth extension for structure detection

Road Detection

GIS

Scene segmentation output

Stage 1

Stage 2

Stage1

Input Frames

Geo-reference image

Initial camera location

Geo-registration Pose estimation

Depth estimation

Stage 1

GeoRegistration

Input Frames

Geo-reference image

Geo-registration

Meta Data

GPS

Aircraft Parameters

Camera Parameters

GeoRegistration

GPS

Aircraft Parameters

Camera Parameters

Frame To Frame transformations

Bundle Adjustment

SIFT matching

Stage1

Input Frames

Geo-reference image

Initial camera location

Geo-registration Pose estimation

Depth estimation

Stage 1

Adjusting camera position Metadata Gives camera position

Along with many other parameters Metadata has error

In all parameters Georegistration overcomes error

Returns a 3x3 homography matrix Want to figure out the exact camera position

Adjusting camera position

Ground Point

Image Point

Project Ground point to image

Adjusting camera position

Alternatively the point can be projected using homography obtained from georegistration

Get rid of translation parameters

Adjusting camera position

Extract rotation and calibration parameters using SVD

smooth

and Using Kalman filter

Use refined

and to estimate translation parameters

Stage1

Input Frames

Geo-reference image

Initial camera location

Geo-registration Pose estimation

Depth estimation

Stage 1

Depth Estimation Use graphcuts to estimate depth

A difficult task due to poor image quality, and unconstrained motion

Solution Fuse depthmaps

Project several depthmaps unto the DOQ Take their average Smooth out the average map

Depth is quantized along Z direction

Depth Estimation

Stage 2

Non-ground object detection

Planar + depth extension for structure detection

Road Detection

GIS

Scene segmentation output

Stage 2

Detect Non-Ground Regions

Threshold Depth Map

Stage 2

Non-ground object detection

Planar + depth extension for structure detection

Road Detection

GIS

Scene segmentation output

Stage 2

Detect Roofs

Threshold Depth Map Fit Plane

Remove Trees

“Roof” Refinement Fit a plane to the detected “roofs”. We have a set of x,y,z points Want to fit

“Roof” refinement

Z

Y

Z

z

u

v

Depth Along u

Must be invariant

Building Detection

Extend Roof To GroundGives Building height

Tree Detector Classify each pixel as tree non-tree 9D Gaussian Mixture

Color, Depth, Texture Supervised offline training

Stage 2

Non-ground object detection

Planar + depth extension for structure detection

Road Detection

GIS

Scene segmentation output

Stage 2

GIS constrained Road Detection

Road Information Provided by GIS

Want to determine

Precise road center

Road Width

Training Sample Patches along roads Align patches along road direction Extract Features

Color Gradient

Feature Vector = histogram of color and gradients

Model: Gaussian Mixture model Offline Training

DetectionAlign the Road

Extract patches

Feed patches into MOG model

Response of the modelGives Road

center

Gradient Histogram

Peaks Give Road bounds

Road Detection

Object Detection Stabilization Optical flow warping Depth warping

Tracking with/without depth

without depth

with depth

Tracking with/without depth

without depth

with depth

Quantitative Results

False acceptance count

False rejection count

False identity switches

Ground truth object count

Multiple object racking accuracy

Quantitative Results

MOTA improvement: 0.740 to 0.851 (15% improvement)

FAR improvement: 0.190 to 0.072 (62% improvement)

More Results

More Results

More Results