imagine camp, developing image processing app for windows phone platform

Imagine Camp:Image Processing Camp

followed by 2 weeks HackathonConducted by: Rahat Yasir Anindo

Imagine Camp Overview

• Image• Image Processing• Applications of image processing• Image processing algorithms (10-12)• Matlab live demo• Image manipulation in Matlab• Windows Phone Demo• Brain storming for Hackathon• Idea selection (one out of three)• Live presentation on selected idea• Suggestion for converting your idea into an app

Time Schedule:10:30-1:00 & 2:00-3:00 = Image processing

discussion & demo3:00-3:30 = Brain storming for Hackathon idea3:30-4:00 = Idea selection4:00-5:00 = presentation on idea & suggestion for

development.

A Picture is Worth 100 Words

A Picture is Worth 10,000 Words

A Picture is Worth a Million Words

Lighting

Scene

Camera

Computer

Scene Interpretation

Components of a Computer Vision System

The goal of computer vision• To extract “meaning” from pixels

What we see What a computer sees

The goal of computer vision• To extract “meaning” from pixels

Source: “80 million tiny images” by Torralba et al.

Humans are remarkably good at this…

What kind of information can be extracted from an image?

• Metric 3D information• Semantic information

10

What is a digital image?

11

Processing of images which are Digital in nature by a Digital computer

Common image formats include:– 1 sample per point (B&W or Gray scale)– 3 samples per point (Red, Green, and Blue)– 4 samples per point (Red, Green, Blue, and “Alpha”,

Opacity)

12

Why we need image processing?It is motivated by three major application

Improvement of pictorial information for human perception

Image processing for autonomous machine application

Efficient storage and transmission

Matrix Representation of an Image

13

1,11,10,1

1,10,10,1

1,01,00,0

)1,1()1,1()0,1(

)1,1()1,1()0,1(

)1,0()1,0()0,0(

),(

:image NAn

NMMM

N

N

aaa

aaa

aaa

A

NMFMfMf

Nfff

Nfff

yxf

M

Visual image formation-Digital Version

14

3D object

Digital Camera

Digitizer 2D Digital ImageCCD / CMOS

sensor

Lens

Sampling and Quantization, 2D Case

15

Continuous Sampled

f(0,0) f(0,1)

f(N-1,M-1)

16

DIGITAL IMAGES: Digital images are 2D arrays (matrices) of numbers:

Hybrid Images

• A. Oliva, A. Torralba, P.G. Schyns, “Hybrid Images,” SIGGRAPH 2006

Why do we get different, distance-dependent interpretations of hybrid images?

?

Slide: Hoiem

Why study computer vision?• Vision is useful• Vision is interesting• Vision is difficult

• Half of primate cerebral cortex is devoted to visual processing• Achieving human-level visual perception is probably “AI-complete”

Why is computer vision difficult?

Challenges: viewpoint variation

Michelangelo 1475-1564

Challenges: illumination

image credit: J. Koenderink

Challenges: scale

Challenges: deformation

Xu, Beihong 1943

Challenges: occlusion

Magritte, 1957

Challenges: background clutter

Challenges: Motion

Challenges: object intra-class variation

Challenges: local ambiguity

Challenges: local ambiguity

Challenges: local ambiguity

Challenges or opportunities?• Images are confusing, but they also reveal the structure of

the world through numerous cues• Our job is to interpret the cues!

Image source: J. Koenderink

Biometrics

Fingerprint scanners on many new laptops, other devices

Face recognition systems now beginning to appear more widelyhttp://www.sensiblevision.com/

Source: S. Seitz

Biometrics

How the Afghan Girl was Identified by Her Iris Patterns

Source: S. Seitz

Vision-based interaction: Xbox Kinect

. Digitizing an image ( convert an continuous image to a digital one)

. Enhancing an image ( Let an image better suit for a specific application)

. Restoring an image ( Recover a damaged image)

. Compressing an image ( Store it with less bytes )

. Segmenting an image ( Partition objects in an image from background )

. Recognizing an image ( Tell what the objects are in an image )

36

What can digital image processing do?

When an image damaged, we can recover it

37

Cracked parts

torn

Restoring an image

04/18/2023 38

Original image 257kb Compressed image 147kb

Redundant Info.

Compressing an image

Take car license plate recognition as an example.

04/18/2023 39

Recognizing an image

EXAMPLE OF DIP•One of the most common uses of DIP techniques: improve quality, remove noise etc

EXAMPLES:

(e) Poorly exposed x-ray image

(f) The result from contrast and edge enhancement

(g) Image blurred by motion

(h) The result of de-blurring

Object Recognition (CVPR 2006)

Examples: The Hubble Telescope•Launched in 1990 the Hubble telescope can take images of very distant objects•However, an incorrect mirror made many of Hubble’s images useless•Image processing techniques were used to fix this....

Finding the outline and shape of image objects, e.g. character recognition.

FACE DETECTION:

Face detection

FACE TRACKING

1)Biological Research: e.g. DNA typing and matching; automatic counting and classification of cell structures in bone and tissue.

2) Defence and Intelligence: e.g. Reconnaissance photo-interpretation of objects in satellite images; target acquisition and missile guidance.

3) Document Processing: e.g. Scanning, archiving and transmission (fax); automatic detection and recognition of printed text (postal sorting office, tax return processing, banking cheques).

4) Law Enforcement Forensics: e.g. Photo-ID kits, criminal photo-search, automatic fingerprint matching, DNA matching and fibre analysis

.

APPLICATIONS OF (DIP):

5) Photography: e.g. altering colours, zooming; adding and subtracting objects to a scene;

6) Remote Sensing: e.g.

Land cover analysis (water, roads, cities and cultivation),

vegetation features (water content and temperature) and crop yield analysis;

3-D terrain rendering from satellite or aircraft data (road and dam planning); fire and smoke detection.

7) Space exploration and Astronomy: satellite navigation and altitude control using star positions.

8) Video and Film Special Effects: Animation,and special effects (Star Wars).

APPLICATIONS OF (DIP):

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing:Image Aquisition

IMAGE ACQUISITION:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing:

Image Enhancement

Image Enhancement:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image restoration:

Digital Image restoration:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image processing, morphological processing:

Digital Image Morphological Processing:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital ImageProcessing,segmentation:

Digital Image Processing: Segmentation:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing object recognition:

Digital Image ProcessingObject Recognition:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing:Representation & Description:

Key Stages in Digital Image Processing: Representation & Description:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing,Image Compression:

IMAGE COMPRESSION:

Image Restoration

Morphological Processing

Segmentation

Object Recognition

Representation & Description

Image Compression

Colour Image Processing

Problem Domain

Image Acquisition

Image Enhancement

Key Stages in Digital Image Processing,Colour Image Processing:

Colour Image Processing:

Image Sub-sampling-Gray

68

Image Re-sampling

69

70

Effect of Sampling and Quantization

250 x 210 samples 256 gray levels

125 x 105 samples

50 x 42 samples

25 x 21 samples

8 gray levels

4 gray levels

Binary image

16 gray levels

Windows Phone Camera Basics• We can build a camera application that can capture still photos.

• We can query device camera capabilities (where there are multiple cameras on the device (front and/or rear), where it has flash, etc.).

• We can get raw camera frames and process them ourselves (encode/decode).

• We can programmatically set flash and focus when taking a photo.

• We can change the resolution of the photo.

• We can continue using the hardware button for the camera shutter to capture a photo and for auto-focus when taking shots outside our application.

• A camera need not be present (for example, front facing camera was not a part of the original devices). Your application would need to handle that condition gracefully.

• To allow our application to have access to the camera capability, you need to specify ID_CAP_ISV_CAMERA in our application manifest file.

• To allow our application to have access to a front facing camera (if one exists), we need to specify ID_HW_FRONTCAMERA in our application manifest file.

Code to Capture images on Windows Phone

using Microsoft.Phone.Tasks;using Microsoft.Phone;private CameraCaptureTask ccTask;public MainPage() { InitializeComponent(); ccTask = new CameraCaptureTask();}private void btnCamera_Click(object sender, RoutedEventArgs e) { ccTask.Show(); }

Code to Capture images on Windows Phone

using Microsoft.Devices;using Microsoft.Xna.Framework.Media;PhotoCamera myCamera;MediaLibrary mediaLibrary;public MainPage() { InitializeComponent(); mediaLibrary = new MediaLibrary();

}private void buttonStartCamera_Click(object sender, RoutedEventArgs e) { myCamera = new Microsoft.Devices.PhotoCamera(CameraType.Primary); myCamera.CaptureCompleted += new EventHandler<CameraOperationCompletedEventArgs>(camera_CaptureCompleted); myCamera.CaptureImageAvailable += new EventHandler<Microsoft.Devices.ContentReadyEventArgs>(camera_CaptureImageAvailable); viewfinderBrush.SetSource(myCamera); }

Code to Preview images on Windows Phone

private void ccTaskCompleted(object sender, PhotoResult pr) { byte[] imgLocal; if (pr.ChosenPhoto != null) { imgLocal = new byte[(int)pr.ChosenPhoto.Length]; pr.ChosenPhoto.Read(imgLocal, 0, imgLocal.Length); pr.ChosenPhoto.Seek(0, System.IO.SeekOrigin.Begin); var bitmapImage = PictureDecoder.DecodeJpeg(pr.ChosenPhoto); this.imgCaptured.Source = bitmapImage; } }

Windows phone camera API’s

Windows phone camera API’s

77

Image manipulation Demo on Matlab (Matrix representation )

RGB to GREY Scale Conversion

RGB to GREY Scale Conversion

80

RGB 2 Grey Matlab & Windows Phone Demo

81

Effect of averaging:

(1) Signal power unchanged

(2) Noise power reduced by 1/M

Conditions:

(1) Signal and noise uncorrelated

(2) Observations are independent

(3) Noise in different observations has the same distribution (iid noise model, independent identically distributed random variables)

Effect of Image Averaging

www.cvcrbd.org

82

Moving Window Transform: Example

originaloriginal 3x3 average3x3 average

Another example Another example

www.cvcrbd.org

83

Moving Window Transform: Example

originaloriginal 3x3 average3x3 average

www.cvcrbd.org

84

Moving Window Transform: Example

originaloriginal 3x3 average3x3 average

www.cvcrbd.org

85

Image Averaging, Example

Averaging 2 images Original image

Averaging 8 images Averaging 16 images

Averaging 32 images Averaging 128 images

www.cvcrbd.org

86

Convolution Examples: Original Images

www.cvcrbd.org

87

Convolution Examples: 33 Blur

111111111

9

1

www.cvcrbd.org

88

Convolution Examples: 55 Blur

1111111111111111111111111

25

1

www.cvcrbd.org

89

Convolution Examples: 99 Blur

111111111111111111111111111111111111111111111111111111111111111111111111111111111

811

www.cvcrbd.org

90

Convolution Examples: 1717 Blur

1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

289

1

www.cvcrbd.org

91

Convolution Examples: 99 Blur using Gaussian mask

0.01124 0.01164 0.01193 0.01211 0.01217 0.01211 0.01193 0.01164 0.011240.01164 0.01205 0.01236 0.01254 0.01261 0.01254 0.01236 0.01205 0.011640.01193 0.01236 0.01267 0.01286 0.01292 0.01286 0.01267 0.01236 0.011930.01211 0.01254 0.01286 0.01305 0.01312 0.01305 0.01286 0.01254 0.012110.01217 0.01261 0.01292 0.01312 0.01319 0.01312 0.01292 0.01261 0.012170.01211 0.01254 0.01286 0.01305 0.01312 0.01305 0.01286 0.01254 0.012110.01193 0.01236 0.01267 0.01286 0.01292 0.01286 0.01267 0.01236 0.011930.01164 0.01205 0.01236 0.01254 0.01261 0.01254 0.01236 0.01205 0.011640.01124 0.01164 0.01193 0.01211 0.01217 0.01211 0.01193 0.01164 0.011242

22

222

1),(

yx

eyxh

www.cvcrbd.org

92

Average filter, Example 1

93

Average filter, Example 2

Beauty Mirror App of IOS

94

Average Filtering Demo (Matlab & Windows Phone)

95

MEDIAN FILTER

96

Effective for salt and pepper noise

- remove noise spikes

987654321

10025,20,20,20,20,20,15,10

100,25,20,20,15,20,20,20,10

Median = 20

ranking

Median Filtering-more later

www.cvcrbd.org

97

Impulse Noise Removal, Example 1

Original image Corrupted image

5 X 5 averaging 5 X 5 medianfiltering

www.cvcrbd.org

98

Impulse Noise Removal, Example 2

Original image 3 X 3 averaging 3 X 3 medianfiltering

www.cvcrbd.org

99

NOICE REDUCTION DEMO (Matlab + Windows Phone)

Effect of Reducing Gray Levels

100

256 128

64 32

16 8

4 2

101

Set OperationsTwo Sets Union Intersection

Complement Difference

102

Set Operations, Example

Complement(Negative)

Union of Constant Imageand Complement Image

The Original Image

103

Logic Operations on Binary Images

BA (A) AND (B)

AND

A B (A) OR (B)

OR

A B (A) XOR (B)

XOR

A B [NOT (A)] AND (B)

NOT-AND

A NOT (A)

NOT

104

Windows phone demo of set operation

105

rk

rk

Histogram

n

nrp kk MNnn

kk 10 Lk

Image Pixel Intensity Histogram

1 1 3 3 7

1 1 3 3 5

1 0 3 3 2

0 0 2 2 2

0 0 2 2 2

0 4321 5 76r

nk

0

2

4

6

r 0 1 2 3 4 5 6 7

nk 5 5 7 6 0 1 0 1

106

Image Histograms, Examples

107

Uneven Pixel Intensity DistributionsThe Histogram of a Grayscale Image

Let I be a 1-band (grayscale) image.

I(r,c) is an 8-bit integer between 0 and 255.

Histogram, hI, of I: – a 256-element array, hI – hI (g), for g = 1, 2, 3, …,

256, is an integer – hI (g) = number of pixels in

I that have value g-1.

Image Histogram

108

The Histogram of a Grayscale

Image

16-level (4-bit) image

black marks pixels w

ith intensity g

low

er R

HC

: nu

mbe

r of

pixe

ls w

ith

inte

nsit

y g

109

The Histogram of a Grayscale Image

Plot of histogram: number of pixels with intensity g

Black marks pixels with intensity g

110

The Histogram of a Grayscale Image

Plot of histogram: number of pixels with intensity g

Black marks pixels with intensity g

111

Image Histogram (Matlab & Windows Phone Demo)

112

. graylevel with

in pixels of

number the

g

I

ghI 1

Luminosity

The Histogram of a Grayscale Image

113

If I is a 3-band image (truecolor, 24-bit) then I(r,c,b) is an integer between 0 and 255. Either I has 3 histograms:

– hR(g+1) = # of pixels in I(:,:,1) with intensity value g– hG(g+1) = # of pixels in I(:,:,2) with intensity value g– hB(g+1) = # of pixels in I(:,:,3) with intensity value g

or 1 vector-valued histogram, h(g,1,b) where– h(g+1,1,1) = # of pixels in I with red intensity value

g– h(g+1,1,2) = # of pixels in I with green intensity

value g– h(g+1,1,3) = # of pixels in I with blue intensity value

g

The Histogram of a Color Image

114

There is one histo-gram per color bandR, G, & B. Luminosity histogram is from 1 band = (R+G+B)/3

There is one histo-gram per color bandR, G, & B. Luminosity histogram is from 1 band = (R+G+B)/3

RIhGIh

BIh

LIh

Luminosity

The Histogram of a Color Image

115

The Histogram of a Color Image

Histogram Processing• Histogram processing re-scales an image intensities so

that the enhanced image histogram follows some desired form.

• The modification can take on many forms:• histogram equalization or• histogram shaping

• e.g. exponential or hyperbolic histogram

116

117

Point Processes: Histogram Equalization

after

1255, gPcrJ I

before

Luminosity

118

Histogram EQ- LUT

The CDF (cummulative distribution) is the LUT for remapping.

The CDF (cummulative distribution) is the LUT for remapping.CDF

LUT

HEQ

119

Histogram EQ

HEQ HEQ

120

10

Image CDFImage CDF Target CDFTarget CDF

LUTLUT

Look Up Table for Histogram Matching

121

Remap an Image:

To Have Two of its Color pdfs Match the Third

original G & B R B & R G R & G B

122

Example: Histogram Matching

original target remapped

123

255,

256,

255

,,,

gcrI

gcrIgcrIcrJ

k

kkk if

if

,

Point Processes: Increase Brightness

0 127 255

0

127

255

g

transform mapping index. band the is and 3210 ,,kg

124

Point Processes: Decrease Brightness

0 127 255

0

127

255

transform mapping

255-g

index. band the is and 3210 ,,kg

crI

gcrI

gcrIcrJ

k

k

kk ,

0,

,,

0,

if

if

,

125

Image Brightness Matlab Demo

126

Image Sharpening Matlab Demo

127

Point Processes: Increase Contrast

.255,

,255,0

,0,

,255

,,

,0

,

crT

crT

crT

crTcrJ

k

k

k

kk

if

if

if

0.1127127,),( acrIacrT kk where, Let

0 127 255

0

127

255

transform mapping 321 ,,k

128

Point Processes: Decrease Contrast

.3,2,10.10

127127,),(

ka

crIacrT kk

and where

,

0 127 255

0

127

255

transform mapping

129

Point Processes: Contrast Stretch

.

,,

Then,

.,max , ,min

,,max , ,min Let

JII

IJJ

JJ

II

mmM

mcrImMcrJ

crJMcrJm

crIMcrIm

0 127 255

0

127

255

transform mappingmI MI

mJ

MJ

130

Information Loss from Contrast Adjustment

origorig

a = 0.5

a = 2.0

lo-clo-c

lo-cRestored

lo-cRestored

hi-chi-c

hi-cRestored

hi-cRestored

131

Information Loss from Contrast Adjustment

origorig

origorig

origorig

lo-clo-c hi-chi-c

lo-clo-c

hi-chi-c

restrest

restrest

lo-clo-c

hi-chi-c

diff diff

diffdiff

abbreviations: original low-contrast high-contrast restored difference

abbreviations: original low-contrast high-contrast restored difference

difference between original and restored low-contrast

difference between original and restored low-contrast

difference between original and restored high-contrast

difference between original and restored high-contrast

132

0 127 255

0

127

255

transform mapping

0.1

255,

255,/1

for

crIcrJ

Point Processes: Increased Gamma

133

Point Processes: Decreased Gamma

0 127 255

0

127

255

transform mappingm M

0.1

255,

255,/1

for

crIcrJ

134

Image Subtraction for Background Removal

g(x,y) = f(x,y) – h(x,y)

www.cvcrbd.org

135

Edge Detection Operator

136

Discrete Laplacian

www.cvcrbd.org

137

Highboost Filtering, Example 3

(a) Image of whole body bone scene

(b) Laplacian of (a)

+

(d) Sobel gredient of (a)

=(c) = (a)+(b)

Sharpened image

(g) = (a)+(f) Sharpened

image

(h) power-law transform of

(g)

www.cvcrbd.org

138

Commonly Used Derivative Masks

www.cvcrbd.org

139

Convolution Examples: Original Images

www.cvcrbd.org

140

Convolution Examples: Vertical Difference

010

020

010

www.cvcrbd.org

141

Convolution Examples: Horizontal Difference

000

121

000

www.cvcrbd.org

142

010141010

Convolution Examples: H + V Diff.

www.cvcrbd.org

143

Edge Detection (Sobel Operator) Matlab Demo

144

Edge Enhancement, Example 1

Original image MagnitudeGradientPrewitt Mask

Original image +Edges with

magnitude > 25

Edges magnitude > 25

www.cvcrbd.org

145

Hybrid Contrast Enhancement Method, Example

Original Image

Histogram Equalization

Highboot Filtering

HighbootFiltering

+Histogram

Equalization

www.cvcrbd.org

148

Medical Image Analysis Demo

149

Brain Storming for Hackathon

150

Idea Selection (One Idea our of Three)

151

Presentation on Hackathon App Ideas

152

Suggestion for converting the idea into an image processing app

Q / A ?

thank you