an investigation into segmenting traffic images using various types of graph cuts

Jonathan Dinger

1

Traffic footage example

2

Important step in video analysis Background subtraction is often used

3

Uses pixelwise computations Performance could be better Better segmentation = better traffic

detection

4

Use interaction between neighboring pixels Keep objects segmented together Better segmentations

5

G = (V,E) Two vertices in V

called source and sink ◦ s and t, respectively

Remaining vertices called M

Vertices in M connected to both s and t (T-links)

Vertices in M connected to neighboring vertices (N-links)

6

T-links are uni-directional N-links are bi-directional Each edge has a weight

◦ also known as a capacity Each pixel has its own

vertex

7

Path◦ List of vertices connected by edges

s-t cut◦ Removal of edges such that all vertices have a

path to either the source or the sink, but not both Flow

◦ Each edge has a capacity◦ That much flow can be pushed through each edge◦ Flow through a graph is the cumulative amount of

flow going from the source to the sink

8

Minimum cut/maximum flow◦ A cut with the smallest cost (weight)◦ Maximum flow that can be pushed from source to

sink (capacity)◦ By max-flow min-cut theorem,

these are equal Graph cuts

◦ Minimum cut on a graph

9

Method to find maximum flow Augmenting path

◦ Path where flow can be increased in all edges between vertices in path

Run search1.Find augmenting path from source to sink2.Add more flow to that path3.Loop back to 1.

10

Step 1. Step 2.

Step 3. Step 4.

11

Step 5. Step 6.

Step 7. Step 8.

12

Two separate implementations◦ Our implementation◦ Kolmogorov’s implementation

http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software/maxflow-v3.01.src.tar.gz

13

Downsampled images Cut performed over smaller area Then upsample, and perform cut over band Faster than graph cuts Less detailed than graph cuts

14

15

Downsampling images loses information Use Laplacian pyramid to store lost data

Computes differencebetween image and imagegained by downsamplingand upsampling again

Add back some of the lost detail◦ Resegment in areas where detail was lost

16

17

BGC faster, less accurate GC slightly slower, more accurate

Graph cut Banded graph cut

18

3 or more labels α-β swap

◦ Loop through label pairs◦ Run graph cut on current pair of labels

If current label of a pixel is not one of the pair, do not use pixel in graph cut

◦ Graph cuts will swap some pixels with label α to label β and vice versa

19

Graph cut on each image Background image computed per pixel

◦ N is the number of images, xi is the grayscale value of the current pixel, and μ is the average grayscale value over all image frames

N

iixN 1

1

20

Find variance of each pixel

N, xi, and μ are as above σ2 is the variance Threshold the variance If variance is below the threshold, do not

include pixel in graph cut◦ Assume non-varying pixels are background pixels◦ Avoid divide-by-zero errors in weights

N

iixN 1

22 )(1

21

Exponentials

x is the grayscale value of the current pixel◦ μ and σ2 are as above

β is a constant that forces the two functions to be equal at α standard deviations

22

Absolute differences

x and μ are as above L is the maximum possible distance

between x and μ, so for grayscale images

K is a shift constant that forces f3 and f4 to be equal at α standard deviations

23

Simple Grayscale Differences

x is the grayscale value of the pixel. Cm and Cn are two grayscale values used as a basis for segmentation

24

Distance

◦ Euclidean distance between two neighboring pixels with coordinates (x1, y1) and (x2, y2)

Smoothing (similarity)

◦ x and y are the grayscale values of two neighboring pixels.

◦ is the maximum possible difference between the pixel values

◦ γ is a modifier that defines the amount of smoothing that takes place

25

Image Graph cut Banded graph cut

Augmented BGC Our graph cut Multi-way cut

26

Computing times for cut results in milliseconds (ms)

27

Image Background Variance

Exponential cut Absolute differencecut

28

ExponentialImage Absolute difference

29

ExponentialImage Absolute difference

30

ExponentialImage Background subtraction

31

ExponentialImage Absolute difference

32

ExponentialImage Absolute difference

33

Absolute differenceImage Background subtraction

34

35

Image Graph cut BackgroundSubtraction

Segmentation performance without smoothing comparable to background subtraction◦ Background subtraction is faster, easier

Smoothing model ◦ Segments larger pieces of vehicles into one

section◦ Vehicle segmentations more “solid”

Absolute difference T-link weights combined with smoothing N-link weights give best results

36

Use multi-way cuts to add shadow segmentation

Extend to RGB

37