Adaptive Segmentation Based on a Learned Quality

MetricI. Frosio1, E. Ratner2

1 NVIDIA, USA, 2 Lyrical Labs, USA

2

Motivation: good / bad segmentation

SLIC (Achanta, 2012)

3

Motivation: good / bad segmentation

GRAPH-CUT (Felzenszwalb, 2004)

4

Motivation: good / bad segmentation

ADAPTIVE GRAPH-CUT (our)

5

Motivation: good / bad segmentation

> >

? ? ?SLIC (Achanta, 2012) GRAPH-CUT (Felzenszwalb, 2004) ADAPTIVE GRAPH-CUT (our)

6

Motivation: good / bad segmentation

Achanta, 2012 (SLIC); Kaufhold, 2004: segmentation algorithms aggregate sets of perceptually similar pixels in an image.

Felzenszwalb, 2004 (graph-cut): a segmentation algorithm should capture perceptually important groupings or regions, which often reflect global aspects of the image.

7

Motivation: segmentation & video compression

Segment motion estimationFrame segmentation

Encoding True block and sub-block motion vectors

8

Aim #1: use the human factor(aka segmentation quality metric)

9

Aim #2: automatic parameter tuning

10

Road map

1) Pick a segmentation algorithm…

2) … Learn a quality metric including the human factor (application needs) …

3) … And put them together (autotuning).

11

Graph:

Nodes:

Edges:

Weights:

vi

vj

w(vi, vj)=0

w(vi, vj)>0

Graph-cut

w(vi, vj)>>0

Vvi

Evv ji ,

0, ji vvw

EVG ,

12

Internal difference:

Graph-cut

Cm

ijCvvm wCIntmji ,max

13

Difference between components:

Graph-cut

Cm

ijCvCvnm wCCDifnjmi ,min,

Cn

14

Boundary predicate:

Graph-cut

Ck

Cn

nn

mmnm C

kCInt

C

kCIntCCDif ,min,

10 15 12

15

Graph-cut

C1

C2

Boundary predicate:

nn

mmnm C

kCInt

C

kCIntCCDif ,min,

15 8 11

16

Graph-cut

C1

C2

Boundary predicate:

Observation scale ~ k

nn

mmnm C

kCInt

C

kCIntCCDif ,min,

17

Graph-cutK

= 3

K =

100

K =

10,

000

18

Road map

1) Pick a segmentation algorithm…

2) … Learn a quality metric including the human factor…

3) … And put them together (autotuning).

19

(Weighted) symmetric uncertainty

seg

BGRimg

BGR

BGRBGRBGR SS

segimgIU

,,,,

,,,,,,

,2

segB

segG

segR

segBB

segGG

segRR

w SSS

SUSUSUU

4 bits------------------ = 33%7 bits + 5 bits

Entropy based average

20

k vs. Uw vs. quality

160 x 120 image block

21

k vs. Uw vs. quality

Training

160 x 120 blocks

320x240 rgb images

K = [1, …, 10,000]

visual inspection & classification

22

k vs. Uw vs. quality

Training

160 x 120 blocks

640x480 rgb images

K = [1, …, 10,000]

visual inspection & classification

23

Learning the metric

WEUS N

iiWE

iWiN

iiUS

iWi

m

bUkm

m

bUkmbmE

1,2

,

1,2

,

1

log

1

log,

Uw = m log(k) + b 1, iWE

24

Road map

1) Pick a segmentation algorithm…

2) … Learn a quality metric including the human factor…

3) … And put them together (autotuning).

25

Automatic k selection

26

Automatic k selection

27

Automatic k selection

28

Automatic k selection

29

Automatic k selection

30

… and adaptivity

k = k(x,y)

31

Road map

32

Results - Quality

Adaptive graph-cut (ours)

Graph-cut (Felzensswalb, 2004) *

SLIC (Achanta, 2012) *

* Same number of segments forced for each algorithm

33

Results

34

Results

SLIC

Graph-cut

Adaptive graph-cut

35

Results

36

Results

SLIC Graph-cut Adaptive graph-cut

37

Results: inter-class contrast(the higher the better)

Sum of the contrasts among segments weighted by their areas (Chabrier, 2004)

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

Inter class contrast

0

0.04

0.08

0.12

0.16

0.2

320x240

averagemedian

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

Inter class contrast

00.020.040.060.080.10.120.140.160.18

640x480

averagemedian

38

Results: intra-class uniformity(the lower the better)

Sum of the normalized standard deviation for each region (Chabrier, 2004)

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

Intra class uniformity

0

2

4

6

8

10

12

14

320x240

averagemedian

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

Intra class uniformity

051015202530354045

640x480

averagemedian

39

Results: contrast - uniformity ratio(the higher the better)

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

1000 * Inter / Intra

0

5

10

15

20

25

30

35

320x240

averagemedian

Ad

ap

tive

gra

ph

-cu

t

Gra

ph

-cu

t

SL

IC

1000 * Inter / Intra

0

2

4

6

8

10

12

14

640x480

averagemedian

40

Discussion

LEARNED segmentation quality metric including the HUMAN FACTOR

Iterative method to AUTOMATICALLY and ADAPTIVELY compute the optimal scale parameter

41

A more general approach(edge thresholding segmentation in YUV)

42

A more general approach(edge thresholding segmentation in YUV)

Openboradcast encoding (x264)

Lyricallabs encoding (adaptive

segmentation)

Show

43

A more general approach(edge thresholding segmentation in YUV)

Openboradcast encoding (x264) Lyricallabs encoding (adaptive segmentation)

Show

44

Open issues & improvements

Resolution dependency (160x120 blocks)

Learning: the Berkeley Segmentation Dataset

Avoid iterations (see I. Frosio, SPIE EI 2015)

45

Questions

? ? ?