gray image coloring using texture similarity measures

1

Gray Image Coloring Using Texture Similarity Measures

by

E. Noura Abd El-Moez E. Noura Abd El-Moez SemarySemary

Thesis Submitted in accordance with the requirements of Thesis Submitted in accordance with the requirements of The University of Monofiya for the degree ofThe University of Monofiya for the degree of

Master of Computers and InformationMaster of Computers and Information( Information Technology )( Information Technology )

Gray Image Coloring Using Texture Similarity Measures

Presented by

E. Noura Abd El-Moez E. Noura Abd El-Moez SemarySemary

For Master degree in Computers and InformationIT department, Faculty of Computers and information,

Menofia University

Supervised by:Prof.

Mohiy .M.Hadhoud

Prof. Nabil .A.Ism

ail

Dr.Waiel .S.Al-Kilani

Thesis summary on:

الرمادية الصور تلوينتشابه معايير بإستخدام

األنسجة

من :مقدم

المعز. عبد نورا المعز. م عبد نورا مسمري سمري السباعي السباعي

في الماجستير درجة على والمعلومات للحصول الحاسبات المعلومات تكنولوجيا المعلومات - - قسم و الحاسبات كلية

المنوفية جامعة

شوقي. وائل دالكيالني

. عبد. نبيل د أالواحد

إسماعيل

. محمد. محي د أهدهود

إشراف :تحت

بعنوان رسالة :ملخص

4

Outlines

Introduction Automatic coloring in the literature TRICS ‘Texture Recognition based Image

Coloring System’ Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

5

Introduction Gray image principles

Gray values

0 . .5

0 . .1

00 . .

15

0 . .2

00 . .

25

0 .2

55

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

6

Introduction Gray image principles

21.2 kb246256170 ×120 px

60.0 kb15,99816,777,216170 ×120 px

Total size (byte)

Actually No. used colors

Possible No. colors

Image Size (pixel)

Image

7

Introduction Coloring Problem

There are two definitions to describe the gray value as an equation of the three basic components of RGB color model (red, green, blue):

1: Intensity (most common used)Gray = (Red + Green + Blue) /3

2: Luminance (NTSC standard for luminance)Gray = (0.299 × Red) + (0.587 × Green) + (0.114 × Blue)

RGB Color R, G, B values Gray value Gray Color

100 ,150 ,87 128

147, 87, 149 128

149, 147, 87 128THERE IS NO MATHEMATICAL FORMULA TO CONVERT FROM GRAY TO RGB COLOR.

8

Introduction Coloring Problem

HSL Color wheel Grayed Color Wheel

Similar Gray ValuesSimilar Gray Values

9

Introduction Coloring Types

1 . Hand coloring Adobe Photoshop

and Paintshop Pro Layers Changing Hue

BlackMagic, photo colorization software, version 2.8, 2003

10

Introduction Coloring Types

2 . Semi automatic coloring Pseudocoloring is a

common example for semi automatic coloring technique

11

Introduction Coloring Types

3 . Automatic coloring i. Transformational coloring

ii. Matched image coloring

iii. User selected coloring

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

12

Automatic coloring in the literature

1. Transformational Coloring

A transformation function Tk is applied on the intensity value of each pixel Ig(i,j) resulting in the chromatic value Ick(i,j) for channel k

)],([),( jiITjiI gkkc

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

13

Automatic coloring in the literature

1. Transformational Coloring

Al-Gindy et al * system. × Results have unnatural look

* A. N. Al-Gindy, H. Al Ahmad, R. A. Abd Alhameed, M. S. Abou Naaj and P. S. Excell ’Frequency Domain Technique For Colouring Gray Level Images’ 2004 found in www.abhath.org/html/modules/pnAbhath/download.php?fid=32

14

Automatic coloring in the literature

2. Matched image coloring

The most similar pixel color is transferred to the corresponding gray one by the color transfer technique proposed by E.Reinhard*;

* Reinhard, E. Ashikhmin, M., Gooch B. And Shirley, P., Color Transfer between Images, IEEE Computer Graphics and Applications, September/October 2001, 34-40

ll αα ββ

15

Automatic coloring in the literature

2. Matched image coloring

* T. Welsh, M. Ashikhmin, K. Mueller. “Transferring color to greyscale images.” In Proceedings of the 29th Annual Conference on Computer Graphics and interactive Techniques, pp 277–280, 2002

** Y. Tai, J. Jia, C. Tang ‘Local Color Transfer via Probabilistic Segmentation by Expectation- maximization‘, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Volume 1, pp. 747-754, 2005

“Global matching procedure” of T. Welsh et al* “Local color transfer” of Y. Tai et al**.

× All these algorithms fail, when different colored regions have similar intensities

16

Automatic coloring in the literature

2. Matched image coloring

Welsh et al proposed also another technique to improve the coloring results when the matching results are not satisfying. It was achieved by asking users to identify

and associate small rectangles, called “swatches” in both the source and destination images to indicate how

certain key colors should be transferred

17

Automatic coloring in the literature

3. User selection coloring

User selection coloring gives high quality colors× User dependent color quality× Time-consuming× Colorization must be fully recomputed for any slight change in the

initial marked pixels

* A. Levin, D. Lischinski, Y. Weiss. “Colorization using optimization.” ACM Transactions on Graphics, Volume 23, Issue 3, pp.689–694, 2004

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

18

TRICS System

Research Objectives

To simulate the human vision in coloring process

To be fully automatic coloring system To spend so little execution time as possible

as a basic requirement for video coloring.

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

19

TRICS System

Structure Gray image

Segmentation

Classification

Database

Samples features

Classes HuesClass labels

Coloring

Colored image

Segmented image, Clusters

Features extraction(Joint, wavelets, laws,…)

Segmentation

(Mean Shift, K-Mean, FCM,..)

1 2

Features extraction(Co-occurrence, Tamura, Wavelets energies)

Classification(K-NN classifier,..)

43

Convert image to HSV channels

Convert to RGB

Set Hue, Saturation, and Brightness

5 6 7

A

B

C

20

TRICS System

Structure 1. Segmentation Stage

Feature extraction: (Pixel based )1. pixel position

2. pixel intensity

3. texture features wavelets coefficients Laws kernels coefficients.

21

TRICS System

Structure 1. Segmentation Stage

1. Wavelets coefficients× Quarter the image size.

Up sampling Upper level construction

22

TRICS System

Structure 1. Segmentation Stage

Up sampling

Upper level construction

23

TRICS System

Structure 1. Segmentation Stage

2. Laws Kernels :

Level L5 = [ 1 4 6 4 1]Edge E5 = [ -1 –2 0 2 1]Spot S5 = [ -1 0 2 0 –1]Wave W5= [ -1 2 0 –2 1]Ripple R5 = [ 1 –4 6 –4 1]

L5S5’ = -1 -4 -6 -4 -1

0 0 0 0 0

2 8 12 8 2

0 0 0 0 0

-1 -4 -6 -4 -1

24

TRICS System

Structure 1. Segmentation Stage

Segmentation technique:Mean Shift *K-mean (Fast k-mean) **

Adaptive Fast k-mean

* D. Comaniciu and P. Meer. ‘Mean shift: A robust approach toward feature space analysis.’ PAMI, 24(5):603–619, May 2002

** C.Elkan, ‘Using the triangle inequality to accelerate k-Means.’ In Proc. of ICML 2003. pp 147--153

32

TRICS System

Structure 1. Segmentation Stage

1. Mean Shift :× So slow× Many parameters

(170×256)- hs=16,hr=16,m=500- Time: 0 34 15- classes : 9

- hs=8,hr=8,hw=4,m=500- Time: 0 39 54- classes : 7

34

TRICS System

Structure 1. Segmentation Stage

2. Fast K-mean : with spatial features

× structured segmentation Increase no. clusters

k=3 k=9

35

TRICS System

Structure 1. Segmentation Stage

2. Fast K-mean : without spatial features

× scattered regions of same cluster disjoint region separation

before after

1 1 2

1

3

1 2 3

4

5

36

TRICS System

Structure 1. Segmentation Stage

2. Fast K-mean :× Small regions (noise) Small regions elimination

Original Before After

37

TRICS System

Structure 1. Segmentation Stage

3. Adaptive Fast K-mean : Clusters number generation Minimum region size estimation

Fully automatic segmentation techniqueFully automatic segmentation technique

38

• CEC “Combined Estimation Criterion”*:•If the VRC index, for k clusters, is smaller than 98 of the VRC index, for k-1 clusters, the CEC is not satisfied.

•If the VRC index, for k clusters, is larger than 102 of the VRC index for k-1 clusters, or if k=1, the CEC is satisfied.

•If the VRC index, for clusters, is smaller than 102 but larger than 98 of the VRC index for clusters, the CEC is satisfied only if TSS for k-1 clusters is smaller than 70 of TSS for k clusters.

TRICS System

Structure 1. Segmentation Stage

a. Clusters number generation :

WCSSTSSBCSS

MxTSS

mxWCSS

f

i

n

jiij

f

i

n

jijij

1 1

2

1 1

2

)(

)(

WCSSk

BCSSknVRC

)1(

)(

* D.Charalampidis, T.Kasparis, ‘Wavelet-Based Rotational Invariant Roughness Features for Texture Classification and Segmentation’. IEEE Transactions on Image Processing.Vol.11.No.8 August 2002

39

TRICS System

Structure 1. Segmentation Stage

Fast k-mean

Calculate CEC

CEC Satisfied?

k=k+1

Yes

No

Start k=1

Stop…Clusters number =k

Gray Image

Segmented Image

40

TRICS System

Structure 1. Segmentation Stage

b. Minimum region size estimation :

• Split the disjoint regions.

• Count all regions size.

• Sort regions size and calculate the step between them.

• Select the regions size of step more than the largest image dimension.

• Consider the minimum region size.

41

TRICS System

Structure 1. Segmentation Stage

59 sec

59 sec

EM-time

31 sec

31 sec

E-time

Laws

Waveletssize = 170×256 M=500, EM = 324

10 min

36 sec

3 sec

42 min

12 sec

T- time

F-time

11 sec

19 sec

A-time

7

8

Regions

4

4

ClustersFeaturesImage

Original gray wavelets Laws

A-time : adaptive fast k-mean time, F-time: fixed k fast k-mean time, T-time: traditional k-mean , E-time: elimination time , EM-time : Elimination time with estimating minimum size region

42

TRICS System

Structure Database set1

The training set consists of 32 classes of Brodatz texture database

Each image has a size of 256 × 256. Each image was mirrored horizontally and vertically to produce a 512 ×512 image.

The image is split into 16 images of size 128 ×128.

256 × 256 512 × 512 16 × 128 × 128

43

TRICS System

Structure Database set2

The training set consists of 9 classes ‘cloud, sky, sea, sand, tree, grass, stone, water, and wood’

Each class has number of samples from 12 to 25 samples.

These samples are taken from real natural images as random 64x64 rectangles.

44

TRICS System

Structure Database

Database record: Sample

Class (level1,level2)

58 Features (6 Moment statistics, 4 Co-ocurance measures, 3 Tamura, and “ 15 wavelets mean, 15 wavelets variance , 15 wavelets energy” for five levels wavelets decomposition)

Hue

45

TRICS System

Structure 2. Classification Stage

a. Feature extraction (Region based) Rectangular region:

1. Maximum rectangle

2. 64 x 64 rectangle× Arbitrary shape Padding rectangle*

* Ying Liu, Xiaofang Zhou, Wei-Ying Ma, ‘Extracting Texture Features

from Arbitrary-shaped Regions for Image Retrieval ‘. 2004 IEEE

International Conference on Multimedia and Expo., Taipei, Jun. 2004

46

TRICS System

Structure 2. Classification Stage

a. Feature extraction (Region based) Region based features:

GLCM * measures (Energy, Entropy, Inertia, Homogeneity ) Tamura * (Coarseness, Contrast , Directionality’) Wavelets coefficients for 5 levels

Mean and variance ** Energy ***

* P.Howarth, S.Ruger,: Evaluation of texture features for content-based image retrieval. In: proceedings of the International Conference on Image and Video Retrieval, Springer-Verlag (2004) 326–324

** O. Commowick – C. Lenglet – C. Louchet, ‘Wavelet-Based Texture Classification and Retrieval’ 2003 found in http://www.tsi.enst.fr/tsi/enseignement/ressources/mti/classif-textures/

*** Eka Aulia, ‘Hierarchical Indexing For Region Based Image Retrieval’, Master thesis of Science in Industrial Engineering, Louisiana State University and Agricultural and Mechanical College, May 2005

49

b. Classification technique KNN classifier with (k=1,k=5,k=10,k=20) Distance Metric is L2 “Euclidean distance”

k=5 gives accuracy up to 94% using “N-fold “ (the collection of (sub) images is divided into N disjoint sets, of which N-1 serve as training data in turn and the Nth set is used for testing)

TRICS System

Structure 2. Classification Stage

21

2)()(),(

i

iJiIJIE

50

TRICS System

Structure 2. Classification Stage

51

TRICS System

Structure 3. Coloring Stage

Color model conversion HSV/HSB color model

HSI/HLS color model

Change in SaturationHue = 0, Luminance=0.5

Change in BrightnessHue = 0, Saturation = 1

Change in HueSat=1, Luminance=1

Change in SaturationHue = 0, Luminance=0.5

Change in LuminanceHue = 0, Saturation = 1

Change in HueSat=1, Luminance=1

52

TRICS System

Structure 3. Coloring Stage

HSV & HSL Channels

53

TRICS System

Structure 3. Coloring Stage

Setting Channels valuesBrightness

The gray image itself

Hue One hue value for each texture

Saturation HSV: 1- brightness HSI: 0.5(1-lightness)

54

TRICS System

Structure 3. Coloring Stage

55

TRICS System

Structure 3. Coloring Stage

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

56

Results and Conclusion Results

Perfect Results Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

57

Results and Conclusion Results

Perfect Results

58

Results and Conclusion Results

PANN Database:

59

Results and Conclusion Results

Misclassified results

2 classification levels: •if the KNN results in 5 classes “grass, sea, water, grass, sea”

•The traditional solution is the class of the grass.

•The 2 levels classification solution is sea.

•(Sea and water), (trees and grass), (sky and clouds) and (wood

and stone) are considered as one class in level one.

60

Results and Conclusion Comparisons

HSV/HSB

HIS/HLS

61

Results and Conclusion Comparisons Local color transfer

Global Image Matching

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

62

Results and Conclusion Conclusion We proposed a new computer coloring technique

that simulates the human vision in this area.

The proposed coloring system is contributed for coloring gray natural scenes.

The execution time of TRICS is minimized using Fast k-mean segmentation technique and the results are enhanced by splitting the disjoint regions and by eliminating small regions.

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

63

Results and Conclusion Conclusion

Clusters number generation algorithm and the minimum region size estimation algorithm increase the professionalism of the system but also increases the time of the execution. And by using both of them TRICS becomes a fully unsupervised intelligent recognition based coloring system.

HSV coloring model is very suitable for our system and the coloring results have good natural look.

64

Results and Conclusion Conclusion

We consider our proposed system structure as an abstract structure for building any more intelligent coloring systems for any other types of images

Our proposed system results perform the other coloring systems.

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

65

Future workGray image

Segmentation

Classification

Database

Samples features

Classes HuesClass labels

Coloring

Colored image

Segmented image, Clusters

Features extraction(Joint, wavelets, laws,…)

Segmentation

(Mean Shift, K-Mean, FCM,..)

Features extraction(Co-occurrence, Tamura, Wavelets energies)

Classification(K-NN classifier,..)

Convert image to HSV channels

Convert to RGB

Set Hue, Saturation, and Brightness

A

B

C

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

Outlines

Introduction Automatic

coloring in the literature

TRICS ‘Texture Recognition based Image Coloring System’

Results Conclusion Future work

66

Future workGray image

Segmentation

Classification

Database

Samples features

Classes HuesClass labels

Coloring

Colored image

Segmented image, Clusters

Features extraction(Joint, wavelets, laws,…)

Segmentation

(Mean Shift, K-Mean, FCM,..)

Features extraction(Co-occurrence, Tamura, Wavelets energies)

Classification(K-NN classifier,..)

Convert image to HSV channels

Convert to RGB

Set Hue, Saturation, and Brightness

A

B

C

Intelligent System for Classifying the image

SOFMAdaptive Learning

68

List Of Publications

Noura A.Semary, Mohiy M. Hadhoud, W. S. El-Kilani, and Nabil A. Ismail, “Texture Recognition Based Gray Image Coloring”, The 24th National Radio Science Conference (NRSC2007), pp. C22, March 13-15, 2007, Faculty of Engineering, Ain-Shams Univ., Egypt.

69

Thanks…

و الحمد لله الذي بفضله تتم الصالحات