Prof. YingLi TianNov. 7, 2018

Department of Electrical EngineeringThe City College of New York

The City University of New York (CUNY)

Lecture 8: Image Segmentation

1

I2200: Digital Image processing

Thanks to G&W website, Dr. Shahram Ebadollahi for slide materials

Outline Introduction Thresholding Edge Segmentation and Linking

Hough Transform Region-based Approach Using Morphology for Segmentation

Watershed Algorithm

Image Analysis and Segmentation Low-level image processing: inputs and

outputs are all images Mid-/High-level image processing: inputs

are images; outputs are information or attributes of the images

Image analysis: extract useful information from the result of low-level image processing

Image segmentation: divides an image into meaningful parts or objects 3

Image Segmentation Image segmentation -- decompose image into

“meaningful” regions or objects Discontinuity: the abrupt change in graylevels -- point,

line, edge detection… Similarity: predefined criteria of similarity --

thresholding, region growing, region splitting and merging…

Approaches: Thresholding edge-based vs. region-based global vs. local Feature-based – texture, motion, color

4

Image Segmentation

5

Detection of discontinuities Points, Lines, Edges

6

Point Detection

7

MATLAB:w = [-1 -1 -1; -1 8 -1; -1 -1 -1];

g = abs(imfilter(double(f), w) >= T;

Point Detection Example

8

Line Detection

9

Line Detection Example

10

Edge Detection Common method to detect discontinuities

because isolated points or one-pixel wide lines are rare

Edge: the borders of regions with different graylevels

Digital edge model:

11

Ramp Edge More clear edge: width of the ramp is narrow;

fuzzy edge: width of the ramp is wider All pixels in the ramp are edge points First derivatives:

12

Ramp Edge

13

Second derivatives:

Ramp Edge

14

Second derivatives:

Sign of second derivative determines if an edge point is on the bright or dark side There are positive and negative values on a ramp

edge Zero-crossing: intersection of the 0-value line and

the line connecting the positive and negative points position of edge

15

Image Derivatives

Image Derivatives 1st order derivative produces “thick” edges 2nd order derivative has stronger response

to fine edges 2nd order derivative produces double edge

response at ramp and step transitions in intensity

2nd order derivative’s sign can be used to find out if going from dark to light or vice versa

16

Edge Detection Edge point: 2nd order derivative is greater

than a threshold Edge: set of connected edge points (need to

predefine “connection”) Edge segment: short edges

17

Edge Detection Implementation

18

Edge Detection Implementation

19

2nd order derivative: Laplacian or LoG operator

Edge Detection

20

After thresholding

Averaged prior to edge detection

Edge detection: Marr-Hildreth method

21

Edge detection operator should be “tunable” to detect edges at different “scales”

Edge detection: Marr-Hildreth method

22

In practice- sample Gaussian function [nxn samples]- convolve with image f(x,y): image smoothing- convolve result with Laplacian mask- find zero-crossings of g(x,y) How?

Low zero-crossingthreshold

High zero-crossingthreshold

g(x, y)

Edge detection: Canny method Goal: Satisfy the following 3 criteria:

Detection: should not miss important edges Localization: distance between the actual and located

position of the edge should be minimal One response: only one response (edge) to a single

actual edge Algorithm

Step 1: Smooth input image with a Gaussian filter Step 2: Compute the gradient magnitude and angle

images

23

Canny Edge detection – Step 3: nonmaxima suppression

24

Canny Edge detection – Step 3: nonmaxima suppression

25

Canny Edge detection – Step 4: Reducing false edge points (hysteresis thresholding)

26

Canny Edge detection

27

Thresholding Goal: setting threshold T in histogram to

segment the image Global Thresholding: T is constant

everywhere in the image Variable Thresholding: T varies

Local Thresholding: T changes based on the neighborhood properties

Adaptive Thresholding: T changes based on the coordinates

28

Thresholding

29

Thresholding Multimodal

T related to f(x, y) only: global thresholding

T related to f(x, y) and p(x, y) only: local thresholding

T related to f(x, y), p(x, y), and (x, y): dynamic (adaptive) thresholding

30

Thresholding - pitfalls

31

Effect of Illumination

Effect of Noise

Global Thresholding

32

1. T=T02. Segment using T3. Get average gray-level for region G1 (f > T) and region G2 (f <= T)4. T_new = average of average of gray-levels5. Repeat until convergence

Global Thresholding

33

Global Thresholding

34

Optimal Threshold

35

Pose problem as: minimizing error of assigning pixels in image to two or more groups

Optimal Threshold

36

Optimal Threshold – Otsu’s method Otsu defines the measure of “goodness” of

a threshold based on how well it can separate two (or more) classes (regions)

Goal: Measure of goodness

37

Optimal Threshold – Otsu’s method

38

Thresholding in MATLAB using Otsu’s method fordetermining the threshold:

Adaptive Thresholding – Image Partitioning

39

Adaptive Thresholding – Moving Average

40http://en.wikipedia.org/wiki/Thresholding_(image_processing)

Hough Method for Line Detection

41

Hough Method: Normal line representation

42

Hough Method: Matlab

43

44

Edge-based segmentation: Problems Spurious edges due to noise and low

quality image. Difficult to identify spurious edges.

Dependent on local neighborhood information

No notion of higher order organization of the image

Gaps and discontinuities in the linked edges

45

Region-based segmentation

46

Region Growing Goal: group pixels or subregions into larger

regions based on predefined criteria for growth

Method:1. Start with a set of “seed” points2. Grow regions by appending to each seed

those neighboring pixels that have predefined properties similar to the seed (e.g., graylevel, texture…)

47

Region Growing

48

Use points (3, 2) and (3, 4) are seeds; property: grayleveldifference < 3:

Region splitting and merging Goal: Partition image into disjoint

subregions and then perform splitting and merging

Method: subdivide R until subregion P(Ri) = TRUE

1. Splitting: if P(R) = FALSE, divide R into quadrants (Ri, i = 1, 2, …, 4); if P(Ri) = FALSE, divide Ri into subquadrants, and so on Quadtree of quadregions

49MATLAB quadtree decomposition: qtdecomp( )

Region splitting and merging

50

Morphological Watersheds “Topographic” points:

Points belong to regional minimum Points at which drop of water, if placed at the

location of any of those points, would fall with certainty to a single minimum (catchment basin or watershed)

Points at which water would be equally likely to fall to more than one such minimum (divide lines or watershed lines)

51

Morphological Watersheds

52

Goal: Find watershed lines

Build dams to prevent water flow from one catchment basin to other

Boundaries between flooded catchment basins

53

Segmentation using Watershed

54

55

Moving Object Detection Main goal: detecting moving objects from a

video sequence of a fixed camera Background – static scene Foreground – moving objects Approach: detect the moving objects as the

difference between the current frame and the image of the scene background.

Current imagefrom C. Stauffer and W. Grimson

Background image Foreground pixels

56

Challenges How to automatically obtain the background

image? Foreground processing The background image must adapt to:

Illumination changes (gradual and sudden) Distracting motions (camera shake, swaying

tree, moving elevators, ocean waves…) Scene changes (parked car)

Others: Shadows Black/blue screen Bad weathers Foreground fragment

57

Basic BGS – basic idea (1) Pixels belongs to foreground if | Current

Frame – BG Image| > Threshold BG image can be just the previous frame or

the average image of a number of frames Works only in particular conditions of

objects’ speed and frame rate Very sensitive to the threshold

58

Basic BGS – example (2)

Current Frame

Difference

High threshold

Lowthreshold

59

Basic BGS -- average (3) Median/average: use the average or the

median of the previous n frames Quick but memory consuming

Running average Alpha is adapt rate (0.05).

)()1(1 iii FBB

60

Basic BGS – selectivity (4)

BGS Examples

61

62

Background Subtraction and Foreground Analysis

Moving Object Detection with Lighting Changes

63

Improved BGS Results

64

Recent Deep Learning-based Image Segmentation

65

Image Segmentation (PSPnet):https://www.youtube.com/watch?v=HYghTzmbv6Q&t=27s

https://www.youtube.com/watch?v=rB1BmBOkKTw&t=23s

Announcement Reading

G&W 8.1-8.2 (exclude 8.2.2) McKay book chapter 1, 5

http://www.inference.phy.cam.ac.uk/mackay/itila/

Final project is finalized. The projects and report template can be found from course website.

Peter presents HW3 today.

66

Final Project -- Due: 12/11/2018 Each Project,

final report (doc file)

presentation (ppt file) 12/11/2018 source code

Final Report Introduction (Problem statement, Motivation) Approach Work Performed (Experiments, software programming, etc.) Results Discussion Reference (cited papers or website) A table listing tasks performed in the project and the percentage

contributions by each team member of individual tasks, as well as overall project. All team members must sign this table.

67

Final Project -- Due: 12/11/2018 Steps to do final project,

Understand the problem to be solved Literature review to understand state-of-the-art Decide which methods you want to try Method implementation Method testing and evaluation Analyze results Write final report Prepare presentation slides

Total 15 minutes (12 mins talk + 3 mins questions)

68

Final Project -- Due: 12/11/2018 Project Presentation on 12/12/2018,

Project 1: Peter Vertenten Project 2: Jinglun Feng Project 3: Rezwon Bhuiyan Project 4: Dingdong Li Project 5: Md Rashiduzzaman and SM Asraful

Alam Project 6: Subash Shrestha Project 7: Sandeep kumar and Azita darvishi Project 8: Tahiya Tazreen and Md. Minhaz Zaman

Lasker

69

Final Project -- Due: 12/11/2018 Project Presentation on 12/19/2018,

Project 9: Muhammad Bhatti Project 10: Ching Chan and Tamanna Tanjil Project 11: Munib Ahsan Project 12: Mollie Murray Project 13: Keshav Teeluck Project 14: Michael Ousseinov Project 15: Ashfaq A Khan

70

HW3 Presentation

71