multiple target detection in video using quadratic multi...
TRANSCRIPT
![Page 1: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/1.jpg)
1
Multiple target detection in video using quadratic multi-frame
correlation filtering
Ryan KerekesOak Ridge National Laboratory
B. V. K. Vijaya KumarCarnegie Mellon University
March 17, 2008
![Page 2: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/2.jpg)
2
Outline
PART I• Correlation filtering overview• Database
PART II• Multi-frame correlation theory• Multi-frame simulation results
![Page 3: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/3.jpg)
3
CF
Why use correlation filters?
• Shift invariance: useful when we don’t know:– Target locations in the scene– Number of targets in the scene
• Distortion tolerance: CF’s can be trained to tolerate some distortion and reject false targets
• Graceful degradation: noise, occlusion
False patterns
Scaled versionsTraining pattern
![Page 4: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/4.jpg)
4
Correlation peak metric
• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR)
![Page 5: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/5.jpg)
5
Correlation peak metric
• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR) metric– invariance to overall image brightness
PSR = 7.46
![Page 6: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/6.jpg)
6
FFT
Correlation filter usage
• Typically computed in the frequency domain– 2 FFTs– complexity reduction
• PSR computed at every point in frequency domain– 4 more FFTs
IFFT
input
FT of filter
correlation array
FT of PSR window
4FFTs
PSR array
*
![Page 7: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/7.jpg)
7
Linear filter vs. quadratic filter
• Consider thresholding the output– results in a discriminant at each shift
pixel 2
pixe
l 3
pixel 2
pixe
l 3
threshold
class 1class 2
example: elliptical boundary
threshold
linear filter quadratic filter
![Page 8: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/8.jpg)
8
Turntable database
• Captured rotational imagery of three targets• Depression angles used: 17 , 19 , 21 , 23• Green background used to aid in segmentation
17°
211cmtarget
camera
turntable
green background
![Page 9: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/9.jpg)
9
Filter training
• Each filter trained on scaled and out-of-plane-rotated images of true- and false-class targets
0° 3° 30°
…
…
…
…
Azimuth
… …Scale (PHTW)
True
False (see
n)
Missile
Leopard
False (unsee
n)Abrams
16
20
![Page 10: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/10.jpg)
10
Filter comparisons
• Showed that RQQCF filters outperform others– each filter bank trained to divide up 360 azimuth range– different number of filters/bank, equal computation
filter design
LEGEND
EOTSDF Eigen-filter
UOTSDF Unconstrained OTSDF
EMACH Extended MACH
CPCF/UPCF
Polynomial CFs
RQQCF/SSQSDF
Quadratic filters
![Page 11: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/11.jpg)
11
PART IIMulti-frame correlation filtering
MFCF
![Page 12: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/12.jpg)
12
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 1: SINGLE-FRAME
![Page 13: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/13.jpg)
13
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)
Relate information
![Page 14: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/14.jpg)
14
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)
Relate information
Information loss due to thresholding
step
Data association
problem due to false alarms
![Page 15: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/15.jpg)
15
CF’s in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 3: RELATION-THEN-DETECTION (our work)
Relate information
![Page 16: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/16.jpg)
16
Efficient output combination
• We want to combine outputs while preserving information– Avoid preliminary thresholding step – Avoid assumptions on number of targets– Avoid a large computational increase
• We have developed a probabilistic approach– We can derive defensible theory– We can confidently utilize all available information
( )Pr | 1, 2,target frame frame …
![Page 17: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/17.jpg)
17
Typical detection schematic
![Page 18: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/18.jpg)
18
Probability mapping
• We map each correlation value to probability of target presence using all available information
PSR array probability array
mapping
high likelihood of target (PSR ≈ 10)
reasonable likelihood of target (PSR ≈ 4)
![Page 19: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/19.jpg)
19
PSR score distributions
• Must assume distributions for target scores and non-target scores for each filter– (we used Gaussians)
Example from a FLIR sequencetarget non-target (clutter)
![Page 20: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/20.jpg)
20
Target motion model
• Must assume transition probability function between adjacent frames– can take velocity into account
• Function of displacement (position-independent)– (we used centered, rotationally symmetric functions)
Gaussian exponential uniform
![Page 21: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/21.jpg)
21
Posterior probability array
• We can derive a pixel-wise mapping function for the posterior probability array
“enhanced output”
“knowledge” from Frames 1
and 2
map
Frame 3 prior probs.
Frame 3 output
Frame 3 posterior probs.
information from Frame 3
![Page 22: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/22.jpg)
22
i-th frame posterior probability array
i-th frame prior probability array
score pdf of filter l on target k
Mapping function
• Various properties of the mapping function
Uses prior arrays from all filters
Uses PSR scores from all filters
Already normalized “filter communication”
sub-class (filter) index
frame index
![Page 23: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/23.jpg)
23
Mapping function
• Example of a mapping in a two-class problemPSR value target probability
![Page 24: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/24.jpg)
24
Prior probability array
• We can derive the prior probability array for the next frame: based on convolution
Frame 3 posterior probs.
“predictor” for Frame 4
motion model priorformula
“knowledge” from Frames 1,
2, and 3 Frame 4 prior probs.
![Page 25: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/25.jpg)
25
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
Prior array formula
• Various properties of the prior array formula
probability of target emergence
ε(x)
non-zero probability
region
zero probability region
![Page 26: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/26.jpg)
26
Prior array formula
• Various properties of the prior array formula
probability of target disappearance
δ(x)
non-zero probability
region
zero probability region
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
![Page 27: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/27.jpg)
27
Prior array formula
• Various properties of the prior array formula
only one convolution per filter
* i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
![Page 28: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/28.jpg)
28
Prior array formula
• Various properties of the prior array formulauses posterior arrays from all
filters
subclass transition probability
“filter communication”
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
![Page 29: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/29.jpg)
29
Multi-frame schematic
mapFrame i
Frame ioutput
(to next iteration)
THRESHOLD
Frame i priors
Frame iposterior
probabilitiesMotion model
Frame i+1 priorspriorformula
detections
![Page 30: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/30.jpg)
30
Multi-frame schematic
mapFrame i
Frame ioutput
(to next iteration)
THRESHOLD
Frame i priors
Frame iposterior
probabilitiesMotion model
Frame i+1 priorsbig
formula
detections
Table lookup
2 FFTs
![Page 31: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/31.jpg)
31
Synthetic sequence demo
Sequence
Prior probability
arrayOriginal output
Posterior probability
array
log-likelihood values
PSR values
![Page 32: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/32.jpg)
32
Synthetic sequence demo
• MFCF can handle multiple targets– Example: sequence with 3 true-class targets, 3 false-class
MFCF detections
single-frame
MFCF
![Page 33: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/33.jpg)
33
Synthetic sequence demo
• False alarms reduced in target and non-target areas@ 90% detection rate
target paths
single-frame MFCF
![Page 34: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/34.jpg)
34
FLIR sequence results• Two FLIR sequences tested • Two types of noise each• MFCF offers greater performance improvement in
white noise than in compression noise
H.264 compression noise (65:1)
AWGN, SNR=20dB
Sequence L2117
MFCF
![Page 35: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/35.jpg)
35
Noise vs. clutter in MFCF
• MFCF favors filters with better clutter rejection– Example: sequence Synthetic3, noisy (SNR = 20dB)– Varied number of eigenvalues (Ne) retained in filter design
single-frame multi-frameworse than
single-frame
![Page 36: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/36.jpg)
36
Summary of main findings
MFCF algorithm:• MFCF is best-suited for handling temporally
uncorrelated noise• Clutter rejection should be handled by the filter(s)• Filter communication can help reject false targets• MFCF degrades gracefully under parameter
changes
![Page 37: Multiple target detection in video using quadratic multi ...users.ece.cmu.edu/~kumar/SPIE08_Kerekes_pres.pdfMultiple target detection in video using quadratic multi-frame correlation](https://reader034.vdocuments.us/reader034/viewer/2022051917/6009877776a1b0151f24d817/html5/thumbnails/37.jpg)
37
Future work
• Multi-view correlation filtering (multiple cameras)
scene
View 1 View 2