top-down & bottom-up segmentation presented by: joseph djugash is this a building or a horse? do...

Top-Down & Bottom-Up Segmentation

Presented By:

Joseph Djugash

Is this a Building or a

Horse?

Do these edges and contours

represent anything?

What’s wrong with Segmentation from Image Statistics?

Is this an object boundary?

Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Not all Image Statistics are Helpful!

Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

How can Class Information help?

Where is the object boundary?

Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

The Class can help resolve ambiguities!

Slides from Eitan Sharon, ”Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Motivation

Bottom-Up segmentation Capture image properties Segmentation based on similarities between

image regions

How can we capture prior knowledge of a specific object (class)? Answer: Top-Down Segmentation

Class-Specific, Top-Down Segmentation

Eran Borenstein and Shimon Ullman

MethodInput Fragments

Matching Cover

Method Outline

Fragment Extraction Figure Ground Label Reliability Value

Fragment Matching Individual Correspondences Consistency Reliability

Segmentation Optimal Cover

Fragment Extraction

Want to find fragments that: Generalize well Are specific to the class Add information that other fragments haven’t already given

us Fragment Size varies from 1/50 to 1/7 of object size

Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

Fragment Extraction

Figure-ground label Manual labeling Learned from relative motion or grey level

variability Reliability Value – Class Specific

Hit rate: A fixed level of false alarms is achieved by

the criterion: Select the k best fragments according to the Hit

rate

Strength of Response –Maximal normalized correlation of a fragment i with each image I in C and NC

x y

x y

vyuxI

vyuxIyxTvuC

),(

),(),(),(

2

Method Outline

Fragment Extraction Figure Ground Label Reliability Value

Fragment Matching Individual Correspondences Consistency Reliability

Segmentation Optimal Cover

Fragment Matching – Individual Correspondences Measuring Similarity

Region Correlation Normalized Correlation Restrict to pixels with the “figure” label

Edge Information Derived from the boundary of the

figure-ground label

The Similarity Measure:

Fragment Matching – Special Requirements The Difference

Entire Template

Figure Part Only Figure & Edge Similarity

Input Template

Fragment Matching – Consistency To acquire a good global cover of the shape,

each local match needs to satisfy the consistency measure.

Consistency Measure:

Fragment Matching – Reliability Using more “reliable” (anchor) fragments is likely

to increase the chance of finding the optimal cover

A fragment’s reliability is evaluated by the likelihood ratio between the hit/detection rate and the false alarm rate

Reliable fragments used first to guide the covering process

Fragment Matching – Reliability

Easy to identify and more commonly seen fragments used first

Problem Areas –Do not exactly follow image discontinuities

Method Outline

Fragment Extraction Figure Ground Label Reliability Value

Fragment Matching Individual Correspondences Consistency Reliability

Segmentation Optimal Cover

Segmentation – The Cover Algorithm The best cover should maximize individual

match quality, consistency and reliability Thus the cover score is written:

Penalizes for inconsistent overlapping fragments

Rewards for match quality and reliability

Constant that determines the magnitude of the penalty for insufficient consistency

Zero for non-overlapping pairs

Segmentation – The Cover Algorithm Initialize with a sub-window that has the maximal

concentration of reliable fragments Similarity of all the reliable fragments is examined at 5

scales at all possible locations Iterative Algorithm:

Select a small number (M=15) of good candidate fragments

Add to cover a subset of the M fragments that maximally improve the score

Remove existing fragments inconsistent with new cover (fragments with cumulative negative score)

Guaranteed to converge to a local max – score is bounded and increases each iteration

Results I

Results I (cont.)

Results I (cont.)

Learning to Segment

Eran Borenstein and Shimon Ullman

Method – OldInput Fragments

Matching Cover

Method – Updated

Learning Figure-Ground Segmentation – Degree of Cover Start with over-segmented fragments –

each fragment now contains many regions Degree of Cover (ri)

Calculated by counting the average number of fragments (from C) overlapping the region Ri

The fragment selection method extracts most fragments from the figure region Higher ri higher likelihood to be “figure”

Lower ri lower likelihood to be “background”

Learning Figure-Ground Segmentation – Degree of Cover

By thresholding the degree of cover, ri, we can choose the figure part to be:

Most likely figure region

Learning Figure-Ground Segmentation – Border Consistency A fragment often contains multiple edges Determine the boundary that optimally

separates figure from the background Fragment hit (Hj={1,n}) – image patches

where fragment Fi is detected Border Consistency:

Learning Figure-Ground Segmentation – Border Consistency

This approach emphasizes consistent edges (border and interior edges) while diffusing noise edges (background features).

Learning Figure-Ground Segmentation Combining degree of cover and boarder

consistency we get the figure part (P)

Maximized when P contains the most of the consistent edges

Maximized when the boundary between the figure and ground are supported by the consistent edges

Fragments detected in an image applies its figure-ground “vote” for all the pixels it covers

Li(x,y) = +1 – vote for figure label

Li(x,y) = –1 – vote for background label

i w(i) Li(x,y) – total votes for pixel (x,y)

Reliability of fragment i

Improving Figure-Ground Labeling

Fragments that are not consistent with the cover (S) is removed and a new cover (S') is generated

Further Refinements: Modify the degree of cover to be the average number

of times its pixels cover figure parts With a more accurate degree of cover, individual

pixels can be substituted for the sub-regions This new degree of cover can them produce an

improved cover This iterative approach converges within 3 iterations

Results II

Results II (cont.)

Results II (cont.)

Results II (cont.)

Bottom-Up Segmentation“Segmentation and Boundary Detection Using Multiscale Intensity Measurements”

Eitan Sharon, Achi Brandt, and Ronen Basri

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs

Detect segments that optimize a NCut measure Hierarchical Structure

Recursively coarsen a graph reflecting similarities between intensities of neighboring points

Aggregates of pixels of increasing size are gradually collected to form segments

Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,

and boundary integrity

Normalized-Cut Measure2( ) ( )ij i j

i j

E S w u u

Si

Siui 0

1

( ) ij i jN S w u u

( )( )

( )

E SS

N S

Minimize:

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs

Detect segments that optimize a NCut measure Hierarchical Structure

Recursively coarsen a graph reflecting similarities between intensities of neighboring points

Aggregates of pixels of increasing size are gradually collected to form segments

Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,

and boundary integrity

Bottom-Up Segmentation

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Segmentation by Weighted Aggregation Normalized-cuts measure in graphs

Detect segments that optimize a NCut measure Hierarchical Structure

Recursively coarsen a graph reflecting similarities between intensities of neighboring points

Aggregates of pixels of increasing size are gradually collected to form segments

Modify the graph to reflect the coarse scale measurements based on computed properties of the aggregates Use multiscale measures of intensity, texture, shape,

and boundary integrity

Full Texture – Lion Cub

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Full Texture – Polar Bear

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Full Texture - Zebra

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Benefits of the Hierarchical Structure

Able to detect regions that differ by fine as well as coarse properties

Accurate detection of individual object boundaries

Able to detect regions separated by weak, yet consistent edges By combining intensity difference with

measures of boundary integrity across neighboring aggregates

Combining Top-Down and Bottom-Up Segmentation

Eran Borenstein, Eitan Sharon and Shimon Ullman

Another step towards the middle

Bottom-Up

Top-Down

Some Definitions & Constraints

Measure of saliency h(i), hi є [0,1)

A configuration vector s contains labels si (1/-1) of all the segments (Si) in the tree

The label si can be different from its parent’s label s i

–

Cost function for a given s

Top-down term Bottom-up termDefines the weighted edge between Si & Si

–

Classification Costs

The terminal segments of the tree determine the final classification

The top-down term is defined as:

The saliency of a segment should restrict its label (based on its parent’s label)

The bottom-up term is defined as:

Minimizing the Costs – Information Exchange in a Tree Bottom-up message:

Top-down message:

Min-cost Label:

Cost of si = –1and s = x

Message from si = –1Cost of si = +1

and s = xMessage

from si = +1

Computed at each node – minimal of the values is the selected label of node s in s

Minimal Cost if the region was classified as background

Minimal Cost if the region was classified as figure

Confidence Map Evaluating the confidence of a region:

Causes of Uncertainty of Classification Bottom-up uncertainty – regions where there is no

salient bottom-up segment matching the top-down classification

Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions)

The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

Results III

Results III (cont.)

Results III (cont.)

Results III (cont.)

Results III (cont.)

Results III (cont.)

Questions?

– Appendix – Why Fragments?

Image fragments make good features especially when training data is limited

Image fragments contain more information than wavelets allows for simpler classifiers

Information theory framework for feature selection

Vs.

Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.

– Appendix – Intermediate complexity

Slides from David Bradley, ” Object Recognition with Informative Features and Linear Classification”.