1

Image ProcessingPattern Recognition

Computer VisionXiaojun Qi

Utah State University

Image Processing (IP)• Manipulate and analyze digital images

(pictorial information) by computer.

• Applications: The applications applied to almost every area of human activities– Biological Research, Defense/Intelligence, Document

Processing, Factory Automation, Law Enforcement, Medical Diagnostic Imaging, Photography, Astronomy, Image Database Retrieval and etc.

1. Biological Research• Automatic analysis of a biological example

(specimen analysis)• Bone, tissue, and cell analysis (counting and

classification)• Analysis, classification, and matching of DNA

material

2. Defense/Intelligence• Automatic interpretation of

earth satellite imagery• Recognize and track targets in

real time• Security and surveillance

3. Document Processing• Scanning, archiving, and

transmission of documents• Automatic detection and

recognition of printed characters

4. Factory Automation• Visual inspection and assembly• Industrial Inspection

5. Law Enforcement• Fingerprint feature extraction, classification,

and identification• DNA Matching

6. Medical Diagnostic Imaging• Digital Angiography• Skin Cancer Detection• Computed Tomography • Brain Tumor• Mammography (Breast

Cancer)

7. Photography• Add/Subtract objects

to and from a scene• Special effects

(Morphing, Warping)

2

8. Astronomy• Separating stars

from galaxies• Galaxy classification

9. Image Database Retrieval• Shape Retrieval• Color Retrieval• Texture Retrieval• Content-based Image

Retrieval Image query by example: Query Image (left), and two most similar images produced by an image database system

Pattern Recognition• Classify what inside of the image

• Applications:– Speech Recognition/Speaker Identification– Fingerprint/Face Identification– Signature Verification– Character Recognition– Biomedical: DNA Sequence Identification– Remote Sensing– Meteorology– Industrial Inspection– Robot Vision

Linear Classifier

Computer Vision

• Focus on view analysis using techniques from IP, PR and artificial intelligence (AI). It is the area of AI concerned with modeling and replicating human vision using computer software and hardware.

• Applications:– Robotics– Traffic Monitoring– Face Identification– 3D Modeling in Medical Imaging

Current Research-- Content-based Image Retrieval

and Annotation System• The driving forces

– Internet– Storage devices– Computing power

• Two approaches– Text-based approach– Content-based approach

3

• Input keywords descriptionsText-Based Approach Text-Based Approach

Index images using keywords• Advantages: (Google, Lycos, etc.)

– Easy to implement– Fast retrieval– Web image search (surrounding text)

• Disadvantages:– Manual annotation is not always available– Manual annotation is impossible for a large DB– Manual annotation is not accurate– A picture is worth a thousand words– Surrounding text may not describe the image

How to describe this image? Content-Based ApproachIndex images using low-level features

Content-Based Approach Index images using images

• Advantages– Visual features, such as color, texture, and

shape information, of images are extracted automatically

– Similarities of images are based on the distances between features

Query Formation

Visual Content

Description

Feature Vectors

Similarity Comparison

Image Database

Visual Content

Description

Feature Database

Relevance Feedback

Indexing & Retrieval

Retrieval results

user

output

Diagram for content-based retrieval system

4

A Data Flow Diagram CBIR is a highly interdisciplinary research area

CBIR Applications

• Commerce (fashion, catalogue, … …)• Biomedicine (X-ray, CT, ……)• Crime prevention (security filtering, … …)• Cultural (art galleries, museums, … …)• Military (radar, aerial, … …)• Entertainment (personal album, … …)

Open Problems• Nature of digital images: arrays of numbers• Descriptions of images: high-level concepts.

– Sunset, mountain, dogs, … …• Semantic gap

– Discrepancy between low-level features and high-level concepts

– High feature similarity may not always correspond to semantic similarity

– Different users at different time may give different interpretations for the same image.

Image Categorization-- High-Level Concepts

• What is image categorization– To label images into one or several predefined

categories (e.g., Dinosaur, Elephant, Horse, Bus, Building, etc.)

– To map low-level visual features to high level semantics.

• Challenges faced by automatic image categorization– Various imaging condition.– Complex and hard-to-describe objects.– Highly textured background.– Occlusions.

Common Techniques for Categorization

• General used techniques– Statistics– Support Vector Machines (SVMs)– Neural Networks– Multiple-Instance Learning (MIL)

Our Approach: Expand SVMs to multi-Category SVMs

5

Our Categorization Approach-- Feature Extraction

• Only global features are used to avoid the problems of inaccurate image segmentation

• Features include global color histogram and edge histogram

• HSV color space is used for the color histogram, which is one of the MPEG-7 color descriptors.

• MPEG-7 also defines the edge histogram descriptor (EHD), which captures the edge distribution in 16 non-overlapping sub-images.

• Based on the original EHD, we construct global EHD (gEHD) and semi-global EHD (sEHD).

• gEHD represent the edge distribution of the whole image.

• sEHD can be constructed as follows:

R3

C4C3C1 C2

R4

R2

1 2

3

4

5

R1

Our Categorization Approach-- Feature Extraction

Our Categorization Approach-- Multi-Category SVMs

• Radial Basis Function kernel is used• 3-fold cross-validation and grid-search

algorithm are used to decide the parameters C and .

• Pairwise coupling approach is used to handle the multiple category case.

• The output of the SVMs is also mapped to the probability so we can assign confidence to each labeled keywords.

γ

Our Categorization Results

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

0 1 2 3 4 5 6 7 8 9 10Category ID

Ave

rage

Acc

urac

yProp.1Prop.2ALIPHistSVM

Our Categorization Results

Horse: 96%Food: 1%

Building: 92%Beach: 3%

Beach: 47%Mountain: 40%

Vehicle: 31%Building: 25%

• Our system can classify images by a set of confidence values for each automatically labeled keywords.

Our Retrieval Results-- using both global and regional

features

(7) 0.8982 0.889922 0.8869 0.8856

0.8855 0.8844 0.8832 0.8826 0.8805 0.8786

(A) 10 matches out of 11, 18 matches out of 20

6

Our Retrieval Results

(B) 10 matches out of 11, 17 matches out of 20

(5) 0.8844 0.882922 0.8821 0.8810

0.8805 0.8783 0.878322 0.8750 0.8726 0.8702

Our Retrieval Results

(C) 9 matches out of 11, 14 matches out of 20

(3) 0.9531 0.916322 0.9088 0.9079

0.9069 0.9065 0.902922 0.9009 0.8957 0.8954

Our Retrieval Results

(D) 10 matches out of 11, 19 matches out of 20

(6) 0.9453 0.912622 0.9100 0.9093

0.9019 0.8970 0.895222 0.8944 0.8918 0.8904

Our Retrieval Results

0.0

0.2

0.4

0.6

0.8

1.0

0 1 2 3 4 5 6 7 8 9 10 11

Category ID

Aver

age

Prec

isio

nProp.NFECRHisC

Average retrieval precision for 20 returned images

Our Retrieval Results

0.250.300.350.400.450.500.550.600.650.700.75

20 30 40 50 60 70 80 90 100

Number of Returned Images

Ave

rage

Pre

cisi

on

Prop.UFM

NFECRHisC

Average retrieval precision for different number of returned images

Image Semantics

• Image semantics may be related to objects in the image

• Semantically similar images may contain semantically similar objects

• Can a computer program learn semantic concepts about images based on objects?

7

Our Image Segmentation Approach

Sample Segmentation Results

Original Image 2 Regions 3 Regions 4 Regions

5 Regions 6 Regions 7 Regions

Original Image 2 Regions 3 Regions 4 Regions

Learning

• Semantically similar images may contain semantically similar objects.– Find similar objects (feature vectors) among

positive images– At the same time, they should be as distinct

from all objects in “negative” images as possible

• Conceptual feature vector:– Multiple-instance Learning (MIL) using diverse

density Learn which region represent the semantic meaning!

Example – Data Mining

• Three conceptual feature vectors– Water, Rock, Trees.

• Rule description of a semantic concept– If one of the regions is similar to water AND

one of the regions is similar to rock, then it is a waterfall image, OR

– If one of the regions is similar to water AND one of the regions is similar to trees, then it is a waterfall image.

What is What?

Current Research-- Shape Representation and Matching

8

Shape Representation

• Shape representation methods:– Region based– Boundary based

• Shape descriptors:– Fourier descriptors– Moments– Chain codes– Etc.

Our Shape Representation and Matching Approach

• Shape indexing: – global signature construction– local signature construction

• Shape retrieval:– calculate similarity score using global

signature– calculate similarity score using local signature– Use a fuzzy method to combine the scores.– Retrieval results are those with higher scores

Our Shape Retrieval Results Current Research-- Face Detection

• Face detection: To determine whether or not there are any faces in the arbitrary still images with cluttered background and to return the image location and extent of each face if present

• Significance:– The most important first step of face identification

series. – It is the preprocessing of face recognition, face

tracking, etc.

Our Approach1. Apply color quantization and segmentation to

preprocess the original image.2. Apply a skin model to find possible skin regions.3. Apply morphological processing to remove noise.4. Merge skin regions if needed.5. Apply some constraints to eliminate non-faces.6. Apply wavelet packet to extract features.7. Apply the neural networks to classify face and

non-face.8. Solve overlapping areas.

Our Results

9

Current Research-- Digital Watermarking and steganography

• Watermarks: Secret messages used for protecting copyrights of digital multimedia data (images, audio, and video)– Content and/or authentication– For detecting unauthorized copies of images

• Characteristics: Imperceptible, security, robustness, and blindness.

• Common Techniques Used: Spatial Domain Approach, Frequency Domain Approach, and Hybrid Approach.

Watermark

Our Watermarking Results-- Wavelet-based Approach

Our Watermarking Results-- Content-based Approach

Stegnography• Steganography: A way of hiding a classified

message.

• Cover image + classified message = StegoObject Transmit over an insecure communication channel (Internet)

• The designated recipient will retrieve the classified message from the stego object, while others do not know the existence of the classified message in the “innocent” looking stego object.

10

Our Stegnography Approach-- Preliminary Test

• Study the OutGuess approach

• Study the JPEG images

• Study the characteristics of the DCT coefficients

• Study several attacks– Histogram analysis– statistics– OutGuess attacks

2χ

Current Research-- Vision-based Navigation (Road Detection)

Our Approach-- Preliminary Test

• Apply Principal Component Analysis (PCA) to find the remote scene.

• Apply Bayes statistics to learn the road features.

• Apply deformable templates to get rid of the shadows.

• Apply the curve functions to approximate the road.

Research Interests• Speech Recognition• Intrusion Detection (One Student)• Gene Sequence Analysis (One Student)• Multi-media Data Mining• Time/Spatial Data Mining• Visualization• Microarray Analysis• Network simulation (One Student)