UNIVERSITY OF OULU P .O. Box 8000 F I -90014 UNIVERSITY OF OULU FINLAND

A C T A U N I V E R S I T A T I S O U L U E N S I S

Professor Esa Hohtola

University Lecturer Santeri Palviainen

Postdoctoral research fellow Sanna Taskila

Professor Olli Vuolteenaho

University Lecturer Veli-Matti Ulvinen

Director Sinikka Eskelinen

Professor Jari Juga

University Lecturer Anu Soikkeli

Professor Olli Vuolteenaho

Publications Editor Kirsti Nurkkala

ISBN 978-952-62-0876-3 (Paperback)ISBN 978-952-62-0877-0 (PDF)ISSN 0355-3213 (Print)ISSN 1796-2226 (Online)

U N I V E R S I TAT I S O U L U E N S I SACTAC

TECHNICA

U N I V E R S I TAT I S O U L U E N S I SACTAC

TECHNICA

OULU 2015

C 538

Matteo Pedone

ALGEBRAIC METHODS FOR CONSTRUCTING BLUR-INVARIANT OPERATORS AND THEIR APPLICATIONS

UNIVERSITY OF OULU GRADUATE SCHOOL;UNIVERSITY OF OULU,FACULTY OF INFORMATION TECHNOLOGY AND ELECTRICAL ENGINEERING,DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING;INFOTECH OULU

C 538

ACTA

Matteo Pedone

C538etukansi.kesken.fm Page 1 Thursday, June 18, 2015 11:10 AM

A C T A U N I V E R S I T A T I S O U L U E N S I SC Te c h n i c a 5 3 8

MATTEO PEDONE

ALGEBRAIC METHODS FOR CONSTRUCTING BLUR-INVARIANT OPERATORS AND THEIR APPLICATIONS

Academic dissertation to be presented with the assentof the Doctoral Training Committee of Technologyand Natural Sciences of the University of Oulu forpublic defence in the OP auditorium (L10), Linnanmaa,on 19 August 2015, at 12 noon

UNIVERSITY OF OULU, OULU 2015

Copyright © 2015Acta Univ. Oul. C 538, 2015

Supervised byProfessor Janne Heikkilä

Reviewed byProfessor Alessandro FoiProfessor Hans Burkhardt

ISBN 978-952-62-0876-3 (Paperback)ISBN 978-952-62-0877-0 (PDF)

ISSN 0355-3213 (Printed)ISSN 1796-2226 (Online)

Cover DesignRaimo Ahonen

JUVENES PRINTTAMPERE 2015

OpponentProfessor Karen Egiazarian

Pedone, Matteo, Algebraic methods for constructing blur-invariant operators andtheir applications. University of Oulu Graduate School; University of Oulu, Faculty of Information Technologyand Electrical Engineering, Department of Computer Science and Engineering; Infotech OuluActa Univ. Oul. C 538, 2015University of Oulu, P.O. Box 8000, FI-90014 University of Oulu, Finland

Abstract

Image acquisition devices are always subject to physical limitations that often manifest asdistortions in the appearance of the captured image. The most common types of distortions can bedivided into two categories: geometric and radiometric distortions. Examples of the latter ones are:changes in brightness, contrast, or illumination, sensor noise and blur. Since image blur can havemany different causes, it is usually not convenient and also computationally expensive to developad hoc algorithms to correct each specific type of blur. Instead, it is often possible to extract a blur-invariant representation of the image, and utilize such information to make algorithms that areinsensitive to blur.

The work presented here mainly focuses on developing techniques for the extraction and theapplication of blur-invariant operators. This thesis contains several contributions. First, wepropose a generalized framework based on group theory to constructively generate complete blur-invariants. We construct novel operators that are invariant to a large family of blurs occurring inreal scenarios: namely, those blurs that can be modeled by a convolution with a point-spreadfunction having rotational symmetry, or combined rotational and axial symmetry.

A second important contribution is represented by the utilization of such operators to developan algorithm for blur-invariant translational image registration. This algorithm is experimentallydemonstrated to be more robust than other state-of-the-art registration techniques. The blur-invariant registration algorithm is then used as pre-processing steps to several restoration methodsbased on image fusion, like depth-of-field extension, and multi-channel blind deconvolution.

All the described techniques are then re-interpreted as a particular instance of Wienerdeconvolution filtering. Thus, the third main contribution is the generalization of the blur-invariants and the registration techniques to color images, by using respectively a representationof color images based on quaternions, and the quaternion Wiener filter. This leads to thedevelopment of a blur-and-noise-robust registration algorithm for color images. We observeexperimentally a significant increase in performance in both color texture recognition, and inblurred color image registration.

Keywords: blur invariant, color image, deconvolution, dihedral symmetry, group theory,image registration, noise, orbit, quaternion, rotational symmetry, Wiener filter

Pedone, Matteo, Algebralliset menetelmät sumentumiselle invarianttien operaatto-reiden muodostamiseen ja niiden sovellukset. Oulun yliopiston tutkijakoulu; Oulun yliopisto, Tieto- ja sähkötekniikan tiedekunta,Tietotekniikan osasto; Infotech OuluActa Univ. Oul. C 538, 2015Oulun yliopisto, PL 8000, 90014 Oulun yliopisto

Tiivistelmä

Kuvauslaitteet ovat aina fyysisten olosuhteiden rajoittamia, mikä usein ilmenee tallennetunkuvan ilmiasun vääristyminä. Yleisimmät vääristymätyypit voidaan jakaa kahteen kategoriaan:geometrisiin ja radiometrisiin distortioihin. Jälkimmäisestä esimerkkejä ovat kirkkauden, kont-rastin ja valon laadun muutokset sekä sensorin kohina ja kuvan sumeus. Koska kuvan sumeusvoi johtua monista tekijöistä, yleensä ei ole tarkoitukseen sopivaa eikä laskennallisesti kannatta-vaa kehittää ad hoc algoritmeja erityyppisten sumeuksien korjaamiseen. Sitä vastoin on mahdol-lista erottaa kuvasta sumeuden invariantin edustuma ja käyttää tätä tietoa sumeudelle epäherkki-en algoritmien tuottamiseen.

Tässä väitöskirjassa keskitytään esittämään, millaisia eri tekniikoita voidaan käyttää sumeu-den invarianttien operaattoreiden muodostamiseen ja sovellusten kehittämiseen. Tämä opinnäy-te sisältää useammanlaista tieteellistä vaikuttavuutta. Ensiksi, väitöskirjassa esitellään ryhmäteo-riaan perustuva yleinen viitekehys, jolla voidaan generoida sumeuden invariantteja. Konstruoim-me uudentyyppisiä operaattoreita, jotka ovat monenlaiselle kuvaustilanteessa ilmenevällesumeudelle invariantteja. Kyseessä ovat ne rotationaalisesti (ja/tai aksiaalisesti) symmetrisensumeuden lajit, jotka voidaan mallintaa pistelähteen hajaantumisen funktion (PSF) konvoluutiol-la.

Toinen tämän väitöskirjan tärkeä tutkimuksellinen anti on esitettyjen sumeuden invarianttienoperaattoreiden hyödyntäminen algoritmin kehittelyssä, joka on käytössä translatorisen kuvanrekisteröinnissä. Tällainen algoritmi on tässä tutkimuksessa osoitettu kokeellisesti johtavia kuvi-en rekisteröintitekniikoita robustimmaksi. Sumeuden invariantin rekisteröinnin algoritmia onkäytetty esiprosessointina tässä tutkimuksessa useissa kuvien restaurointimenetelmissä, jotkaperustuvat kuvan fuusioon, kuten syväterävyysaluelaajennus ja monikanavainen dekonvoluutio.

Kaikki kuvatut tekniikat ovat lopulta uudelleen tulkittu erityistapauksena Wienerin dekonvo-luution suodattimesta. Näin ollen tutkimuksen kolmas saavutus on sumeuden invarianttien jarekisteröintiteknikoiden yleistäminen värikuviin käyttämällä värikuvien kvaternion edustumaasekä Wienerin kvaternion suodatinta. Havaitsemme kokeellisesti merkittävän parannuksen sekäväritekstuurin tunnistuksessa että sumean kuvan rekisteröinnissä.

Asiasanat: dekonvoluutio, diedriryhmä, kohina, kuvien rekisteröinti, kvaternio, rata,rotaatiosymmetria, ryhmäteoria, sumentumiselle invariantti, värikuva, Wienerinsuodatin

Preface

This work was carried out in the Computer Vision and Research group of the Universityof Oulu. I want to express my gratitude to Professor Janne Heikkilä for supervising mywork and for introducing me to the field of signal processing and pattern recognition.With his patience and open-mindedness, he has always encouraged me to follow anddevelop my own ideas. I also thank Professor Matti Pietikäinen for giving me theopportunity to work in his research group.

I am grateful to Professor Jan Flusser for the long and fruitful collaboration that heestablished during and after my first research visit in UTIA, and whose expertise had avery significant impact on my research.

I am also grateful to Professor Emilio Musso and Professor Francesco Vaccarinofor offering their vast knowledge in differential geometry and invariant theory duringmy visit in the Polytechnic University of Turin, to Professor Leo Dorst and ProfessorEckhard Hitzer for constantly providing their invaluable help in the field of Cliffordalgebras, and to Professor Eduardo Bayro-Corrochano for offering me the possibility tovisit his research group in the CINVESTAV of Guadalajara.

I shall also thank Professor Alessandro Foi and Professor Hans Burkhardt for theireffort in reviewing my thesis, and Dr. Pertti Väyrynen for the language revision. Theiradvice contributed to improve the quality of this manuscript.

The financial support of Infotech Oulu Graduate School, Nokia Foundation, TaunoTönning Foundation and Oulu University Foundation is gratefully acknowledged aswell.

I would like to thank my parents for their constant and unconditional support duringall the stages of my education.

Finally, my warmest thanks go to Eeva whose love, patience and support, made thiswork possible to complete.

Oulu, June 2015Matteo Pedone

7

8

Abbreviations

PSF Point-spread functionFT Fourier TransformQFT Quaternion Fourier TransformWSS Wide-sense stationaryPSD Power spectral densityMBD Multi-channel blind deconvolutionSV-MBD Spatially-variant multi-channel blind deconvolutionLPQ Local phase quantizationLBP Local binary patternQWF Quaternion Wiener filteriff if and only ifZ Integers numbersR Real numbersC Complex numbersH Quaternionsz∗ Complex conjugate of z

q Quaternion conjugate of q

|q| Absolute value of a quaternion q

f : R2→ R Function from R2 to RLp(R2) Lp space of functions f : R2→ RLp(R2;H) Lp space of functions f : R2→HL2

E (R2) Space of square integrable real-valued 2-dimensional continuous stochas-

tic processesL2

E (R2;H) Space of square integrable quaternion-valued 2-dimensional continuous

stochastic processesΨ f Power spectral density of a stochastic process f

‖·‖2 Squared norm for the space L2

‖·‖2E Squared norm for the space L2

E

F ( f ) Fourier transform operator applied to a function f

F f(u) Fourier transform of f evaluated at uF(u) Alternative notation for F f(u)

9

supp( f ) Support of the function f

ϕ† Adjoint of the linear transformation ϕ

x ·y Dot-product between vectors x and ydet(ϕ) Determinant of a matrix ϕ

Ox Orbit of an element x under a group actionΓ Invariant operatorN Degree of rotational symmetryRk Linear operator that rotates points in the plane by 2πk/N radiansSa Linear operator that reflects points in the plane across the axis vector aR f Abbreviation for ( f R−k)(u) for some fixed k and u and a function f

S f Abbreviation for ( f Sa)(u)

10

List of original publications

I Pedone M & Heikkilä J (2012) Local phase quantization descriptors for blur robust andillumination invariant recognition of color textures. Proc. 21st International Conference onPattern Recognition (ICPR 2012): 2476-2479.

II Pedone M, Flusser J & Heikkilä J (2013) Blur invariant translational image registration forN-fold symmetric blurs. IEEE Transactions on Image Processing 22(9): 3676-3689.

III Pedone M, Flusser J & Heikkilä J (2015) Registration of Images with N-fold Dihedral Blur.IEEE Transactions on Image Processing 24(3): 1036-1045.

IV Pedone M, Bayro-Corrochano E, Flusser J & Heikkilä J (2015) Quaternion WienerDeconvolution for Noise Robust Color Image Registration. IEEE Signal Processing Letters,22(9), 1278-1282.

The writing of Papers I, II, III, IV was mainly carried out by the author, who wasalso in charge of the experiments in these papers. The ideas were processed togetherwith the co-authors.

11

12

Contents

AbstractTiivistelmäPreface 7Abbreviations 9List of original publications 11Contents 131 Introduction 15

1.1 Background and motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

1.2 The contribution of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.3 Summary of original papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2 Blur invariant image analysis 212.1 Image blur model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.2 The symmetries of the PSFs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

2.3 Basic concepts of blur-invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.4 Calculation of blur invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4.1 Invariants based on integral transforms with polynomialfunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.4.2 Invariants in spectral domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5 Registration in presence of blur . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

2.6 Analysis of color images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3 Algebraic framework for blur invariance 333.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.2 Definitions and theorems related to Fourier transforms . . . . . . . . . . . . . . . . . . . 33

3.3 The image space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.4 Symmetric functions and convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.5 Basics of group theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

3.6 The convolution group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.7 Group actions and invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.8 Construction of invariants with respect to group actions . . . . . . . . . . . . . . . . . . 43

3.9 Constructive calculation of blur-invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

13

4 Blur-invariant image registration 474.1 Translation as the action of convolution groups . . . . . . . . . . . . . . . . . . . . . . . . . .474.2 Image registration as an operation between group elements . . . . . . . . . . . . . . . 484.3 N-fold blur-invariant registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.4 Estimation of translational shift: the geometry of the N-fold peaks . . . . . . . . 52

5 Blur-robust registration of noisy images by Wiener filter 555.1 Brief review of Wiener filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.2 Noisy image registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.3 Obtaining blur and noise robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6 Blur-invariance and registration of color images 616.1 Quaternion representation of color images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.2 The quaternion Wiener filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.3 Registration of noisy color images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.4 Blur-robust registration of noisy color images . . . . . . . . . . . . . . . . . . . . . . . . . . . 666.5 Color blur-invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

7 Applications 677.1 Multi-channel blind deconvolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.2 Extended depth-of-field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .707.3 Blurred texture recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727.4 Blurred videos stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

8 Conclusion 75References 77Original publications 81

14

1 Introduction

1.1 Background and motivation

In the last few decades, in parallel with the technological progress, we have witnessed aremarkably rapid diffusion of different kinds of hardware devices capable of acquiring,manipulating and analyzing digital images. Such devices are constantly seen in oureveryday lives in a vast range of different contexts: cameras integrated in mobile phones,security cameras, medical diagnosis, microscopic and astronomical imaging, and soforth. The process of acquisition of a digital image is almost inevitably disturbed byexternal factors, or simply by physical limitations, that degrade the ideal quality of theimage (Campisi & Egiazarian 2007). Many of these degradations have an inherentlyradiometric nature (Flusser et al. 2009), and some of them, like blur (Figure 1) andnoise, occur so commonly that virtually any owner of an ordinary consumer camera isaccustomed to them. There are many potential factors that can contribute in causingradiometric degradations; in the case of image blur these are, for instance, lens aberration,diffraction, limited depth-of-field, motion of the scene, camera shake, wrong focus,atmospheric turbulence, just to mention some (Flusser et al. 1995). In some cases, theamount of blur varies across the image, while in other cases it can be approximatelyconsidered uniform (Figure 1, right). Whatever the cause or type of blur, the extractionof useful information from a blurred image always presents many difficulties, as mostalgorithms for image analysis rely on the assumption that the scene is not degraded, asif it were captured under ideal conditions. The causes of noise in digital images areinstead to be found mostly in the physics of the camera circuitry, and in particular in thecamera sensor. The main sources of image noise are: voltage leaks of the photosensor,fluctuations of the photon flux, and differences in pixel sensitivities (Janesick 2001).The standard approach to deal with noisy images is to attempt to recover the latentnon-noisy image by employing a denoising algorithm. There is a vast literature onimage denoising and extensive overviews on the topic can be found in (Katkovnik et al.

2010, Milanfar 2011).There are two main strategies to address the problems caused by image blur:

deblurring, and the invariant approach (Ojansivu 2009). Deblur (or alternatively,deconvolution) essentially consists in estimating the ideal sharp version of the imagefrom an observed blurred image. The roots of the techniques of signal deconvolution

15

can be traced back to the work of Norbert Wiener in the early 50’s (see, Wiener (1949)),while concrete implementations of image deblurring started to appear in the 60’s(Helstrom 1967). Deconvolution of images is an ill-posed problem which still todaypresents a lot of technical challenges (Campisi & Egiazarian 2007). In fact, even currentstate-of-the-art deblurring algorithms typically introduce evident visual artifacts in therecovered image (Ojansivu 2009). The artifacts found in deblurred images, besidesgiving an unrealistic feel, can also be harmful in applications related to automaticimage recognition. The instability of the results of deblurring algorithms makes theunsupervised use of deconvolution techniques unsuitable for general-purpose imagingapplications. Another issue associated with deblurring is its typically high computationalcost. In applications where one needs to process several, or even hundreds of images ina short time frame, deblurring may not necessarily be considered a practical solution.

The alternative approach to image deblurring is based on the philosophy of avoidingdeconvolution, and thus any attempt to recover the ideal image, limiting oneself toextract information that is already invariant to blur from a given image. These ideaswere pioneered in the field of computer vision already in the early 60’s by Hu (1962);nonetheless, Hu’s work was mainly focused in the derivation of invariants for geometric

distortion (e.g. rotation, translations, etc.), while invariant operators to blurring wereexplicitly introduced by Flusser et al. (1995) in the late 90’s. It must be said, however,that all the ideas behind the invariant approach have very deep roots in mathematics, inparticular in the discipline of classical invariant theory, whose origins date back to the19th century (see, Olver (1999) for an historical introduction). The main advantageof using blur-invariant techniques rather than performing image deblur is that thecalculation of blur-invariant quantities is more stable and computationally fast comparedto the nature of the operations involved in sophisticated deconvolution algorithms. Forthis reason, blur-invariants fit very well in many applications where one is only interestedin recovering some descriptors that would provide an approximate representation of theimage that would be insensitive to blur. On the other hand, blur-invariants should not beconsidered as a competing approach with image deblurring techniques; in fact, manyrestoration techniques, like multi-channel blind deconvolution (Sroubek & Flusser 2005)or extended depth-of-field (Redondo et al. 2009), operate according to the paradigm ofimage fusion, in which a stack of images representing the same scene and taken withdifferent camera settings are combined to produce a single enhanced image. Imagefusion techniques typically assume that the images in the stack are perfectly aligned,which is an assumption that is generally difficult to fulfill in practice, thus image

16

registration algorithms that are insensitive to blur become a necessity. This is a commonscenario where blur-invariant operators and blur-invariant registration algorithms can beemployed in order to assist a fusion algorithm, like a multi-channel deblurring method.

From the above observations, it emerges that blur-invariant registration is a fun-damental pre-processing step that often precedes an image fusion algorithm. As theexisting algorithms for registration of ordinary images are unsuited for handling blurredimages, due to their sensitivity to increasing amounts of blur, there has been recently anincrease of interest in designing specific algorithms to register blurred images.

This thesis introduces novel techniques and algorithms for the construction ofblur-invariant operators, and for registration and recognition of blurred images.

Fig 1. (Left) A photo taken with high aperture settings and short exposure time: the imagehas a sharp appearance and the details are visible. (Right) The same subject photographedusing a low aperture and a longer exposure time: the effect of atmospheric turbulencecaused the image to appear blurred.

1.2 The contribution of this thesis

The physical limitations of today’s optical acquisition devices have motivated thecomputer vision community to develop strategies to overcome such limitations, andto extract as much useful information as possible from degraded images (Campisi &Egiazarian 2007). Since radiometric degradations like blur, illumination changes, etc.can be mathematically modeled as transformations of the pixel data, the mathematicalconcept of invariance w.r.t. families of transformations has been frequently exploited in

17

order to perform operations of recognition and registration of images, while reducing theeffects of such degradations on the performance of the algorithms (Flusser & Suk 1998).

This thesis focuses mainly on the development of techniques for generation of blur-invariant operators, and on their application to fundamental problems in computer vision,like blurred image registration, recognition, and restoration. Many different techniqueshave been proposed in the literature to obtain global blur-invariant representations of animage. Such invariant representations are essentially mathematical formulas that take animage as input, and return an output which has the property of being the same as theoutput obtained when a blurred version of the same image is fed into the formula. It iscustomary in the literature to present such formulas to the reader without any motivation,and then simply prove that the property of blur-invariance is fulfilled. In the first partof this thesis we will demonstrate that, in many cases, such kind of guesswork canbe entirely eliminated, as long as certain geometric properties of the blur PSFs areknown. Some fundamental concepts borrowed from a branch of abstract algebra calledgroup theory are utilized in order to show that blur-invariant operators can be obtainedconstructively, from the sole knowledge of the rotational and axial symmetries of thePSFs.

Successively, the blur-invariant operators are used to formulate a global registrationalgorithm for blurred images. This contribution is inspired by, and extends, the work ofOjansivu & Heikkilä (2007a) who proposed a blur-invariant phase correlation algorithmfor registering pairs of blurred images. Ojansivu’s algorithm works under the assumptionthat the blur PSFs are centrally-symmetric. The approach proposed in this thesis canhandle a much larger family of blur kernels: namely, the ones having rotational andaxial symmetries of arbitrary order, and it is experimentally proven to be more robustthan both the traditional blur-invariant phase correlation, as well as other popularlandmark-based registration techniques, when the images are heavily blurred. Examplesof successful applications of our registration algorithm are presented, in particular,multichannel blind deconvolution, and depth-of-field extension, which both requireblurred image registration as an important pre-processing step.

As an additional contribution of this thesis, a noise-robust version of the proposedregistration algorithm is presented. Noise-robustness is obtained by reformulatingall the proposed registration techniques in terms of Wiener deconvolution filtering.Following the same line of thought, this thesis also describes a generalization of all theproposed techniques to color images. Color images are represented as quaternion-valuedfunctions, and once the ordinary Wiener filter is proven to be the fundamental tool on

18

which the blur-invariant registration algorithm is based, the quaternion Wiener filter isthen utilized to formulate a blur and noise robust registration algorithm operating oncolor images.

1.3 Summary of original papers

This manuscript is based on four publications. The author of this thesis is also the mainauthor of all these four papers.

Paper I proposes a generalization of the local phase quantization (LPQ) descriptorsto color images, and demonstrates its successful application to classify textures thatare both corrupted by blur and illumination changes. The generalization of the LPQdescriptors makes use of a representation of color information based on quaternions.The concept of local phase is thus replaced with the concept of local phase of the 2Dshort-time quaternion Fourier transform (QFT). The quaternion LPQ (qLPQ) is shownto possess both the properties of blur-robustness and robustness w.r.t. illuminationchanges. We observed that its performance in blurred color texture recognition surpassesthe one obtained with other popular descriptors in the computer vision literature.

Paper II introduces a method for translational registration of blurred images. Theapproach presented in the paper is based on the concept of invariance w.r.t. uniform blurwith point-spread functions (PSF) manifesting rotational symmetry. Blur PSFs withrotational symmetries are typically observed in optical systems that are equipped with ashutter made of a certain number of blades arranged circularly around the lens. Theidea proposed in this paper is the one of obtaining a blur-invariant representation of theimages, and then perform a “phase correlation” between these two representations, inorder to estimate the shift between the images. This method is shown to be considerablymore robust than other well-known types of blur-robust image registration algorithms,and it is successfully implemented as a pre-processing step for multichannel blinddeconvolution.

Paper III completes the theory described in Paper II by developing a registrationalgorithm that is robust to blur w.r.t. to PSFs having both rotational and axial symmetry(dihedral symmetry). Dihedral phase correlation requires two parameters to be known a

priori, i.e. the number of blades that compose the shutter, and the orientation of thesymmetry axis of the shape of the aperture. This paper contains also algorithms for theestimation of these parameters. Furthermore, it is shown how blur-invariant registrationcan be used in conjunction with other image fusion techniques, like depth-of-field

19

extension.Paper IV discusses the reinterpretation of the well-known registration algorithm of

phase correlation in terms of Wiener filtering, and it presents a natural extension of it tothe case of color images. Inspired by the quaternion representation of color images, thealgorithm of registration is entirely based on the use of the quaternion Wiener filter(QWF). The QWF allows us to formulate a noise-robust version of phase correlation forcolor images. This approach based on the QWF makes it straightforward to formulatea generalized version of blur-invariant-and-noise-robust phase correlation for colorimages; however, blur-invariant registration of color images is an aspect that is discussedonly in this thesis, and not in the original paper.

20

2 Blur invariant image analysis

In practical scenarios optical devices frequently capture images whose quality is visiblyimpaired by the effect of radiometric degradations, like illumination changes, noise, andblur. Blur can be caused by a multitude of different causes, which in turn determinedifferent types of blurs, like motion, camera shake, out-of-focus blur and so forth. Insome cases, the effect of blurring varies locally in different regions of the image, whilein other cases blur is approximately the same across the whole image. In this thesis wemostly consider uniform blur, since this type of blur can be more easily modeled and itis very often observed in shots taken with ordinary consumer cameras, as in out-of-focusscenes, and less frequently in shots where the camera (or the scene) moves quickly alonga linear axis. The technical difficulties associated with ordinary deconvolution methods(e.g. high computational cost, and introduction of visual artifacts) are a strong motivationto develop algorithms that are insensitive to blur, but at the same time avoiding anydirect attempt at recovering the latent sharp image. Moreover, there exist situations inwhich some sort of blur-invariant operation must precede the operation of deblurring,for instance in multi-channel blind deconvolution, where a sharp image is estimatedfrom a set of differently blurred images that are assumed to be perfectly aligned. In thisspecific situation, it is clear that some form of image registration that is insensitive toblur must be performed before even attempting to obtain a deblurred image.

In this section we present the most popular approaches to obtain blur invariantoperators, and we discuss existing methods for registration of blurred images, which isan important application that is used in conjunction of several restoration algorithmsbased on image fusion.

2.1 Image blur model

An observed blurred image f captured with an acquisition device can be mathematicallydescribed in the following way (Banham & Katsaggelos 1997): suppose f0 is an ideal“sharp” image, understood as a two-dimensional function f0 : R2→R, then f is assumedto obey the following model:

f (x) = ( f0 ∗h)(x)+n(x) =¨

R2f (y)h(x−y)d2y+n(x) (1)

21

where h is a two-dimensional convolution kernel, and n is additive noise. The convolutionkernel h mostly depends on the properties of the optical system, and it represents thepoint-spread function (PSF), which can be intuitively understood as the impulse responseof the imaging system. It must be emphasized that Equation (1) is an adequate model toformalize the effect of uniform blur in an image. If we neglect the effect of noise n

then, (1) essentially reduces to an operation of linear filtering. There exist more generalmathematical models to describe spatially varying blur (Banham & Katsaggelos 1997);however, the generality of such models invariably introduces a wide range of technicaldifficulties that would make it very hard to extend a theory formulated for uniform blurto the case of spatially varying blur. For this reason, we do not consider this very generaltype of blur in this thesis.

2.2 The symmetries of the PSFs

When we described the mathematical model of image blur in the previous section, wedid not elaborate our discussion on the nature of the PSF. It turns out that many blurPSFs have important geometrical properties. To understand many of these properties, itis customary to introduce the pupil function (or aperture function) which is defined as(Goodman et al. 1968):

P(x,y) =

1 if(x,y) is inside theaperture

0 otherwise

and which can be interpreted as a function that is non-zero in correspondence of thepoints where light passes through the aperture hole and zero elsewhere. It can bedemonstrated that, assuming an ideal imaging system free of aberrations, and followingthe principles of geometrical optics , the intensity of the PSF produced by a defocusedlens is proportional to the scaled aperture function P , where the scaling factor dependson the distance of the object from the lens, and on the distance between the sensor planeand the lens (see, Lee (1990) and Chapters 5-6 of Goodman et al. (1968) for details).Since the observed PSF is directly related to the shape of the aperture described by P, itis possible to deduce that, whenever the aperture function manifests any symmetry (e.g.rotational symmetry and/or axial symmetry), the out-of-focus blur PSF will have exactlythe same type of symmetry. Very often, the mechanical design of the diaphragm shutterof ordinary cameras is such that the shutter blades are arranged symmetrically. Thiscontributes to the typical appearance of out-of-focus scenes, where in correspondence of

22

Fig 2. An image taken with a camera having a shutter with four blades. The shape of the PSFcan be clearly discerned in the out-of-focus background. The PSF has approximately 4-foldrotational symmetry.

brighter spots, it is possible to discern very clearly the blur PSF with its symmetries (seeFigure 2). This kind of symmetry is called N-fold symmetry, since the PSF appears thesame as its rotated versions by 2π

N k radians (k = 1, . . . ,N). Other types of symmetricPSFs, like circularly symmetric PSFs, can arise from different causes; one example isthe blur caused by a long exposure of a scene taken under the effect of atmosphericturbulence (Banham & Katsaggelos 1997). There are two main reasons that motivate usto study the symmetry properties of PSFs:

1. Symmetric apertures (and symmetric PSFs) are observed very often in imagessuffering out-of-focus blur.

2. It is relatively easy to construct blur-invariant operators for symmetric PSFs

We anticipate that the derivation of invariants w.r.t. to blur, necessitates the identificationof a family of PSFs having the property of closure with respect to the operation ofconvolution. Many families of symmetric PSFs possess this property: i.e. convolvingtwo symmetric functions of the same kind yields another function with the same type ofsymmetry. However, before embarking on a detailed description of the construction ofblur invariants, we give a brief overview of already existing methods.

23

2.3 Basic concepts of blur-invariance

Although we will postpone a more rigorous treatment of the concept of invariance inChapter 3, it is useful at this stage to introduce some basic notions on blur-invariants.For the sake of discussion, let us consider the blur model in Equation (1) and neglectthe effect of noise. Before finding a blur-invariant operator, one must first specify afamily of two-dimensional blur kernels (e.g., PSFs) that is closed under the operationof convolution, in other words, one shall identify a set of functions for which theconvolution of two functions in the set yields another function in the same set. It is infact important to realize that blur-invariance is always understood to be defined withrespect to some closed set of functions, and such a set is usually called the null space

of blur kernels associated with an invariant operator (Flusser & Suk 1998). With thisrestriction in mind, we can implicitly define a blur-invariant for some family Ωblur ofblur kernels closed w.r.t. convolution, as a general operator Γ having the property:

∀h ∈Ωblur, Γ( f ∗h) = Γ( f ) (2)

where, f is an image. It is also crucial to introduce the concept of completeness of aninvariant. An invariant Γ w.r.t. convolutions with the elements of Ωblur is said to becomplete when the following proposition holds for any two arbitrary images f , g:

Γ( f ) = Γ(g) ⇐⇒ ∃h ∈Ωblur : g = f ∗h (3)

Loosely speaking, the above condition states that if we are given two different functionsf ,g /∈ Ωblur such that one is not the “blurred” version of the other, then their blur-invariant representations can always be discriminated, i.e. Γ( f ) 6= Γ(g). The importanceof completeness can be clearly illustrated by the extreme example of a trivial invariant Γ

defined as: Γ( f ) = 0 for any arbitrary function f . Despite being (also) a blur-invariant(with null space equal to the set of all possible images), Γ has no discrimination power,as it discards all the information of f . It is in fact true that, if one finds an operator Γ′

that is a complete invariant for a subset Ω′blur ⊂Ωblur, then Γ′ will surely have higherdiscrimination power than Γ, since it can discriminate also the elements of Ωblur \Ω′blur.In other words, a blur-invariant Γ′ with higher discrimination power will have a smallernull space (i.e. invariance w.r.t. a smaller family of blur kernels), but on the otherhand, the quantity Γ′( f ) will preserve more information of f than Γ( f ) will do. Thisis essentially a trade-off between blur-invariance (measured in terms of the “size” ofthe null space) and the loss of information between f and Γ( f ). This constitutes the

24

main theoretical motivation for finding invariants with given null spaces of blur kernels.However, trading-off blur-invariance with discriminability is also important (and actually,required) in many practical scenarios. Perhaps the easiest practical example to illustratethis fact is given in Chapter 2.5 of (Flusser et al. 2009), where the author demonstratethat if one is interested in recognizing blurred symmetric objects, then using an operatorthat is invariant to blur with centrally-symmetric kernels will make it impossible todiscriminate, for instance, an hexagon from a square: they are both centrally-symmetric,and thus contained in the null space. Instead, by employing an operator that is invariantto blur with circular kernels (which is a subset of centrally-symmetric kernels), thediscrimination between hexagons and squares becomes possible. Surprisingly, betweenthe introduction of the first blur-invariants for centrally symmetric, linear motion,and circular blur, and the introduction of the new families of rotational and dihedralblur-invariants, there is a time span of about twelve years during which, little progresshas been made in this aspect.

In Chapter 3 of this thesis we will show that once the problem of finding blur-invariants is formulated in an abstract algebraic setting, obtaining blur-invariants formost of the important null spaces of blur kernels becomes a mechanical exercise thathardly presents any technical difficulty. However, in order to understand what are theaspects of this research field into which the computer vision community has put itsefforts, we shall first describe the techniques by which blur-invariant operators arederived.

2.4 Calculation of blur invariants

Given a set of PSFs Ωblur, and considering the definition of convolution in Equation(1), it is, at least in principle, far from straightforward to find an operator Γ that wouldsatisfy the invariance property expressed in (2). The mathematical tool that is most oftenutilized to overcome this difficulty is the concept of integral transform (Debnath &Bhatta 2014). In general, an integral transform T of a two-dimensional function f canbe expressed as a mapping between different sets of functions in the following way:

T f(u) =¨

Df (x)Φ(x,u)d2x (4)

where D and Φ(·,u) are respectively the domain of integration and the kernel functions ofthe transform T . Different sets of kernel functions lead to different integral transforms,each having its own algebraic properties. Among the algebraic properties, one is

25

typically interested in investigating how the operation of convolution is transformed bysome integral transform operator T , or in other words, in finding the explicit form ofthe function

T f ∗h(u) (5)

which of course depends on the choice of the kernel functions Φ(x,u). In fact, it is oftenthe case that the operation of convolution between two functions becomes easier tomanipulate in the integral transform domain. Perhaps the most remarkable and well-known example of such a simplification is seen with the Fourier transform (obtainedby setting Φ(x,u) = e−ix·u and D = R2) which turns the operation of convolutionin the space domain into a pointwise multiplication in the frequency domain, i.e.F f ∗h(u) = F f(u)F h(u). In the next sections we will distinguish the mostpopular techniques for finding blur-invariants according to the integral transform domainon which the invariants are defined.

2.4.1 Invariants based on integral transforms with polynomialfunctions

Chronologically, the first blur-invariants to be explicitly derived and discussed in depthare found in the early work of Flusser et al. (1995, 1996a,b) and Flusser & Suk (1997,1998). In these papers, the authors use an integral transform based on geometricmoments (or central moments), which is equivalent to setting Φ(x,y, p,q) = xpyq in (4).By employing the notation T f(p,q) = m( f )

p,q they note that the convolution of twofunctions f and h becomes:

m( f∗h)p,q =

p

∑k=0

q

∑l=0

(pk

)(ql

)m( f )

k,l m(h)p−k,q−l

and they demonstrate that from the above property it is relatively easy to find invariantsfor linear motion blur (Flusser et al. 1996b), and complete invariants for “circular”blur (Flusser & Zitová 2004) and centrally symmetric blur (Flusser & Suk 1998). Inthe case of linear motion blur-invariants, the authors present a set of invariants w.r.t.horizontal motion blur and combine them with rotation-invariants in order to obtain a setof invariants w.r.t. general linear motion blur. Since the set of general linear motion blurkernels is not closed w.r.t. convolution, these invariants cannot be complete, so theauthors test experimentally their discriminative power by identifying a blurred versionof a photo between a given pair of reference images. In (Flusser et al. 1995, 1996a), the

26

authors derive blur-invariants for PSFs h having symmetries across the main axes and themain diagonal: h(x,y) = h(y,x) = h(−x,y), in other words, 4-fold rotational symmetries.On the other hand, they do not prove the completeness of their invariants, so it is not clearwhether such operators are complete 4-fold blur-invariants, or if they are just invariant tocentrally symmetric blur. Flusser & Suk (1998) present a complete set of invariants w.r.t.centrally symmetric PSFs, i.e. h(x,y) = h(−x,−y). Such invariants are expressed interms of the central moments µ

( f )p,q =

˜R2 f (x,y)

(x−m( f )

1,0

)q(y−m( f )

0,1

)qdxdy and they

have the form:

C( f )(p,q)=

0 ; if p+q is even

µ( f )p,q − 1

µ( f )0,0

∑pk=0 ∑

ql=0

(pk

)(ql

)µ( f )k,l C( f )(p− k, q− l) ; if p+q is odd

where p+ q is the order of the invariant. Despite the theoretical completeness ofthese invariants, due to the sensitivity to noise of higher order invariants, the authorsutilize only invariants of lower order (e.g. p+q≤ 7); however, they demonstrate theireffectiveness in practical applications like satellite image registration based on templatematching.

The work of Flusser & Suk (1998) apparently stimulated many researchers in thescientific community to rederive geometric and radiometric invariants with respect to thesame types of degradations considered in their paper (e.g. centrally symmetric blur,Euclidean transformations, and so forth) using different kinds of integral transforms.The motivation for this effort apparently came from the interest in addressing certainissues related to the numerical stability and the correlatedness of the moment invariantswhich is due to the non-orthogonality of the polynomial kernel functions (Kautsky &Flusser 2011). The integral transforms that were usually chosen in these cases werebased on kernel functions that are orthogonal polynomials, like Legendre polynomials(Wee & Paramesran 2007, Zhang et al. 2010) or Zernike polynomials (Ji & Zhu 2009,Chen et al. 2010, Zhu et al. 2010, Shao et al. 2014). In most cases, this task turned outto involve extremely tedious and lengthy mathematical calculations, which, perhaps notsurprisingly, led to the publication of several articles containing serious mistakes andmisunderstandings; the kinds of questionable statements found in these paper are diverse,and they range from presenting “invariant” operators that are not invariant, to claimingto have derived “new” invariants for specific types of blur, while in fact, the invariantswere not complete. Many of these mistakes were successively pointed out by Kautsky &Flusser (2011); in their paper the authors also demonstrate the important result that itis not necessary to rederive “from scratch” the already known invariants separately

27

for each polynomial basis, as there exist simple mathematical relationships that allowone to easily obtain invariants in any polynomial basis from the sole knowledge of themoment-based invariants.

More recently, Makaremi & Ahmadi (2012) derived moment-based blur-invariantsfor centrally symmetric blur in wavelet-domain. Their invariants are complete, and thus,contain the same information and have the same discriminative power of the invariantsin (Flusser & Suk 1998). They however observe increased robustness against noise inimage recognition and registration.

From some claims expressed by many authors (e.g. Flusser & Suk (1998), Makaremi& Ahmadi (2012), Zhang et al. (2010), Ojansivu (2009)), there appear (or at least, thereappeared) to be a widespread belief that centrally symmetric (i.e. 2-fold symmetric)PSFs are enough to deal with out-of-focus blur. While it is true that circular PSFs arealso centrally symmetric, such claims can be immediately disproved by the argumentsof Section 2.2, in fact, it is sufficient to consider the blur produced by an ordinarycamera with a shutter having an odd number of blades. Clearly, triangular, pentagonal,heptagonal, and in general (2N+ 1)-gonal PSFs do not exhibit central symmetry,therefore, all the operators mentioned above are actually not blur-invariants for thesefamilies of PSFs.

It was not until 2009 that Flusser et al. (2009) published a section in their bookdevoted to the derivation of two new sets of operators that are complete invariants withrespect to N-fold rotational blur, and Gaussian blur.

It must be noted however, that a set of complete moment-based invariants for circularblur (i.e. blur PSFs with ∞-fold symmetry) was already published in (Flusser & Zitová2004).

Later, the theory of N-fold blur-invariants was expanded by elucidating theirrelationship with Fourier-based invariants (Flusser et al. 2014), and by combiningrotational and axial blur-invariance, thus, obtaining a complete set of invariants for blurhaving dihedral symmetry (Boldyš & Flusser 2013).

In the next section, we describe similar work that, chronologically speaking, wascarried out more or less in parallel with the progress made by using geometric moments,but which makes use of integral transforms that yield a frequency domain representationof the images.

28

2.4.2 Invariants in spectral domain

As explained in Section 2.4, the Fourier transform possesses the remarkably convenientproperty of turning a convolution in the space domain into a multiplication betweenFourier spectra (Bracewell 1965). It is probably for this reason that early formulations ofcentrally symmetric blur-invariants date back to the early 80’s, when for instance Hayeset al. (1980) suggested that the tangent of the Fourier phase spectrum is invariant to blur.

A more detailed explanation of the blur-invariants based on the Fourier phasetangent was first given by Flusser & Suk (1998), who also proved mathematically theirequivalence with the corresponding moment-based invariants proposed in the samepaper. In their proof they provide an explicit formula to derive the moment-basedinvariants from the power series expansion of the Fourier-based invariants. The authors,however, advocate the use of moments justifying this position with the observation thatthe tangent function is very sensitive to noise, especially near the points of discontinuity(Flusser & Suk 1998).

Successively, Ojansivu & Heikkilä (2007b) realized that the tangent functionis unnecessary, and that the issues related to the sensitivity of noise can be easilycircumvented by observing that the square of the Fourier phase spectrum of an image f

is a complete blur-invariant to centrally symmetric blur:

P f(u) =(

F f(u)|F f(u)|

)2

(6)

They extract the phase angle ∠P f(u) and propose the angular distance as a metricto quantify the similarity of two images, thus avoiding the discontinuity problem.Furthermore, they propose similar invariants based on higher order spectra (Fourierbispectrum) to obtain also invariance to translation and affine transformations (Ojansivu& Heikkilä 2008b). They demonstrate experimentally that their features are robust tonoise and outperform the moment-based approach in object recognition.

Metari & Deschênes (2008) employed the Mellin transform to derive novel invariantsfor both geometric transformations and blur. The null-space of their set of blur-invariants is represented by the kernels having axial symmetry across the diagonal, i.e.h(x,y) = h(y,x). According to the authors, their invariants require less computationsand outperform moment-based invariants in image recognition. Furthermore, theirwork exposed the underlying simplicity by which one could obtain both geometricand radiometric invariants by simply calculating ratios of Mellin transforms; thesimplification of the formulas of blur-invariants seems to emerge from a property of the

29

Mellin transform that is totally analogous to the Fourier convolution theorem:

M f ∗h(s,v) = M f(s,v)M h(s,v)

wheres,v ∈ C. These invariants have the following form:

k f (s,v) =M f(s,v)M f(v,s)

Despite these facts, their work apparently did not receive enough attention by thecomputer vision community.

In Papers I-II of this thesis, we demonstrate that the technique of calculating ratios ofFourier transforms can be adapted to derive easy formulas for complete invariants in thefrequency domain for all blurs with PSFs having N-fold rotational and axial symmetry,with arbitrary N.

2.5 Registration in presence of blur

One of the most important applications in which blur-invariant features find their use isthe registration of differently blurred images. The necessity of registering blurred imagesarises frequently as a sub-problem of some restoration algorithms based on image fusion(Sroubek et al. 2005, Sroubek & Flusser 2005, Šorel 2007). Currently, the standardmethod for aligning images undergoing geometric and/or radiometric degradation is thatof using a SIFT-based registration algorithm (Lowe 2004); nonetheless, SIFT features,strictly speaking, only enjoy a certain degree of robustness to low amounts of blur, butin general, they are not invariant to blur.

Flusser et al. (2014) tackled this problem by proposing an approach based onmatching automatically detected control points between the reference image and thesensed image; the matching is obtained by comparing the distance between the moment-based N-fold invariants extracted from the local neighborhood of the control points. Thematching pairs are then used to estimate the parameters of rotation and translation.

Ojansivu & Heikkilä (2007a) were the first to introduce a global registration methodthat is mathematically invariant to blur with any kind of centrally symmetric kernel.Their approach is inspired by the well-known algorithm of phase-correlation (Kuglin1975, De Castro & Morandi 1987) and, in its essence, consists in “phase-correlating”the blur-invariant quantities in (6). More formally, given two images f and g that areassumed to be blurred by two arbitrary centrally symmetric kernels, they calculate the

30

following quantity:

S(u) =(

Pg(u)P f (u)∗

|Pg(u)| |P f (u)|

)2

(7)

where P f (u) =P f(u) according to (6), and then they prove that the inverse Fouriertransform of S yields a peak in correspondence of a location that is twice the true shiftvector between the ideal non-blurred versions of the images f ,g.

In Papers I-II the theory of Ojansivu’s blur-invariant phase-correlation is generalizedin order to obtain a global registration algorithm specifically designed to achieveblur-invariance with respect to blur having respectively N-fold rotational symmetriesand N-fold dihedral symmetries. The higher number of symmetries allows one to findmore invariants, and thus to extract more correlation peaks using formulas similar to (7).The additional correlation peaks contribute to an increase of performance in registration.

2.6 Analysis of color images

There has been a recent increase of interest in extending known techniques in imageanalysis and image recognition to the domain of color images. The most commonapproaches to deal with color images are: 1) to keep only the luminance channel anddiscard the color information, 2) to perform the given operation separately for eachcolor channel, 3) to treat a color (or multi-channel) image as a vector-valued functionon the plane. An approach that is intimately related to the third category, and whichis slowly gaining popularity, is that of representing an image as a two-dimensionalquaternion-valued function f : R2→H. The algebra of quaternions can be considered ageneralization of the algebra of complex numbers, and it was introduced by Hamilton(1866). Quaternions are quantities having one real scalar component and three imaginarycomponents. In image processing, the scalar component is usually set to zero, and animage f is represented as:

f (x) = f1(x)i+ f2(x)j+ f3(x)k

where, x ∈ R2, i, j,k are the imaginary units, and fi : R2→ R (i = 1,2,3) are the imagefunctions for each color channel. Some pioneering work in formulating a theory forquaternion-valued images can be found in (Ell 1992, Sangwine 1996). These ideas gavea strong impulse in the late 90’s and in the first years of the new millennium, whenpapers generalizing common signal processing techniques to quaternionic signals startedto appear; few examples are: the formulation of a quaternion Fourier transform (QFT)

31

for color images (Sangwine 1997, Ell & Sangwine 2007), quaternion edge detectors(Sangwine 1998), “color-sensitive” smoothing filters (Evans et al. 2000), generalizationsof autocorrelation and cross-correlation of quaternionic signals (Sangwine & Ell 1999).

This approach is currently being followed also by the researchers in the area ofgeometric invariants. Loosely speaking, the idea is that of generalizing the integraltransforms used for the invariants for grayscale images by replacing the kernel functionsΦ in (4) with quaternion-valued generalized kernel functions, and then finding invariantsby re-adapting the old derivations for scalar (grayscale) functions to quaternionic (color)images; some examples can be found in (Chen et al. 2010, Karakasis et al. 2014, Shaoet al. 2014, Xiang-yang et al. 2015).

Quaternionic representations of images have been used also by Moxey et al. (2003)for color image registration. The authors started from the definition of QFT, andgeneralized the concept of cross-correlation for quaternionic frequency spectra, whichenabled them to generalize the well-known registration method of phase-correlation(Kuglin 1975, De Castro & Morandi 1987). Interestingly, they also proved that thequaternionic quantity in correspondence of the magnitude peak of the correlationspectrum encodes the rotation that brings the colors of one image to globally “match”the colors of the other image.

In Paper III, we give an alternative generalization of phase-correlation for colorimages. We prove that ordinary phase-correlation can be interpreted as an operation ofdeconvolution by Wiener filtering, and after deriving a quaternionic extension of theWiener filter, we utilize it to obtain a noise-robust global registration algorithm for colorimages. Such a contribution clearly paves the way for generalizing our algorithms ofN−fold rotational and dihedral blur-invariant phase-correlation (Papers I-II) to colorimages. Paper I presents an extension to color images of some popular blur-robustdescriptors (LPQ) introduced by Ojansivu & Heikkilä (2008a) which is widely used inimage recognition. Such descriptors are based on the quantization of the Fourier phaseof the low frequency spectra of local neighborhoods. The generalization of this idea isachieved by quantizing two phase angles of the quaternion Fourier spectra. Robustnessto contrast and illumination changes is also obtained by discarding the (0,0)-componentof the QFT.

32

3 Algebraic framework for blur invariance

In this chapter we introduce an algebraic framework that unifies and generalizes themain theoretical results contained in Papers II-III. The material presented in the nextsections should be considered as a main contribution of this thesis, as it shows that thetechniques used for finding blur invariants for images are essentially special cases of amore general algebraic construction.

3.1 Motivation

In order to give a short motivation for this chapter, it is useful to recall (see Section 2.4.2)that given a real-valued image f : R2→ R uniformly blurred by a centrally symmetrickernel, and its Fourier transform F , the following quantity,

F(u)F(−u)

=

(F(u)|F(u)|

)2

(8)

is invariant to centrally symmetric blur. In Papers II-III we illustrated how to generalize(8) in order to obtain novel blur-invariants with respect to different families of symmetrickernels. Although such operators have a simple and compact form, and it is indeedeasy to prove that they are actually invariants, there are still aspects that often remainobscure. In fact, most (if not all) papers in the literature dealing with blur-invariantsfollow the typical structure of: 1) presenting some candidate formula for a blur-invariantoperator, 2) mathematically proving that such formula constitutes indeed a blur-invariantfor some family of kernels. As a matter of fact, the reader is frequently left with thestrong impression that such formulas somehow come “out of the blue”, and they are theresult of a good amount of guesswork, as very little insight is provided into why theseformulas actually work. In the following sections we will show that given a certainfamily of convolution kernels, it is possible (and relatively easy) to constructively obtaincomplete blur-invariant operators, without almost any need of skillful guesswork.

3.2 Definitions and theorems related to Fourier transforms

We report here some basic definitions and theorems that we will frequently needthroughout this chapter.

33

Definition 1. The two-dimensional Fourier transform (FT) of a function f ∈ L2(R2) isdefined as:

F(u) = F f(u) =ˆR2

f (x)e−iu·xd2x (9)

where · denotes the dot-product between two vectors.

Definition 2. A function f ∈ L2(R2) is said to be band-limited (with unit bandwidth)when |u|> 1 =⇒ F(u) = 0.

Definition 3. The two-dimensional convolution between two functions f and h isdefined as:

( f ∗h)(x) =ˆR2

f (y)h(x−y)d2y (10)

Theorem 4. (Convolution) Given two functions f ,h∈ L2(R2), the FT of the convolutionf ∗h is given by:

F f ∗h(u) = F(u)H(u) (11)

Theorem 5. (Linear transformation) Given a linear and invertible transformationϕ : R2→ R2, and a function f ∈ L2(R2), the FT of the transformed function f ϕ−1 isgiven by:

F(

f ϕ−1)= |det(ϕ)|F ϕ

† (12)

where ϕ† is the adjoint transformation of ϕ . Note that, if we express ϕ in matrix form,then, ϕ† would simply correspond to a matrix transposition, and of course, det(ϕ) is thedeterminant of the corresponding transformation matrix.

Corollary 6. (Orthogonal Transformation) Given an orthogonal transformation (i.e. aroto-reflection) φ : R2→ R2 and a function f ∈ L2(R2), the FT of the roto-reflectedfunction f φ−1 is given by:

F(

f φ−1)= F φ

−1 (13)

which intuitively means that rotating (and/or reflecting) a function in the space domain isequivalent to rotating (and/or reflecting) its Fourier spectrum by the same amount. Thisresult immediately follows from Theorem 5, since for any orthogonal transformation φ

we have that: det(φ) =±1 and ρ† = ρ−1.

34

3.3 The image space

Definition 7. Suppose U =

u ∈ R2 : |u| ≤ 1

is the closed unit ball centered at theorigin of R2. We define a grayscale image as a function f : R2→ R with the followingproperties:

1. f ∈ L2(R2)

2. f is band-limited and supp(F) =U.

We denote by Ω the space of functions satisfying the above requirements, and letΩF be the space of the Fourier transformed functions in Ω. Condition 1 and band-limitedness ensure that the Fourier transform F = F ( f ) exists and is bounded, andthat f = F−1 (F ( f )).1 Condition 2 is a stronger version of band-limitedness which isneeded in order to guarantee also the existence of F−1

( 1F

). In fact, imposing such a

condition is equivalent to state that F is never zero in its domain U . The importance ofthe last requirement will become clear later, when we will introduce convolution groups.

For now, it is sufficient to intuitively understand that, if f ∈Ω is an image, and h is aconvolution kernel (filter) in Ω, then, there always exists an inverse filter h−1 ∈Ω suchthat ( f ∗h)∗h−1 = f .

3.4 Symmetric functions and convolution

Now that we have defined our space of interest, we shall discuss the properties of someimportant subsets of Ω that are useful to model optical blur. A uniformly blurred imageis typically modeled as a convolution of an image function f with another function h

that represents the point spread function (PSF) of the optical system. We have seen thatdue to the geometry of the camera shutters, many PSFs manifest some kind of symmetry,e.g. rotational symmetry, axial symmetry, or both (see Figure 3). It is useful to studythe properties of these functions, since it will later become clear that it is relativelyeasy to find blur-invariants with respect to these families of PSFs. We now give precisedefinitions of the different types of symmetric blur kernels.

Definition 8. Given N∈Z≥2 and an orthogonal transformation R :R2→R2 representinga 2π/N rotation around the origin, we say that a function h ∈Ω has rotational symmetry

1More rigorously, the Fourier transform operator here is to be interpreted as a Plancherel transform operator,while the equality sign should be interpreted as convergence in `2-norm (see pages 185-186 of Rudin (1986)for more details).

35

Fig 3. Examples of point-spread functions having different types of symmetries. (Top-left)Circular Symmetry. (Top-right, bottom-left) 6-fold rotational symmetry. (Bottom-right) 4-foldrotational symmetry. All the PSFs in the figure are also dihedrally symmetric, i.e. they haveaxial symmetries as well. The symmetry axes were superimposed on one image. (Adaptedfrom Papers II-III, c©IEEE).

36

of order N (or alternatively N-fold rotational symmetry) if it satisfies the followingrelationship:

hR = h (14)

One can easily verify that the above equation is valid even when we replace R withRk = R . . .R, where the exponentiation notation stands for composing the functionwith itself k times, with the convention that R1 = R, and R0 = id (the identity function).The rotations Rk, where k ∈ Z, can be obviously expressed in matrix form as:

Rk =

(cos(2πk/N) sin(2πk/N)

−sin(2πk/N) cos(2πk/N)

)(15)

The above definition includes two special cases that are worth mentioning:

Definition 9. A function h∈Ω is said to be centrally symmetric if it has 2-fold rotationalsymmetry.

The other interesting case occurs when we allow the limiting case N→∞ to representcircular symmetry. In this case, we should replace Rk in Equation (15), which is definedonly for k ∈ Z, with the rotation matrix Rθ , where θ ∈ R, defined as follows:

Rθ =

(cosθ sinθ

−sinθ cosθ

)(16)

Definition 10. Given any rotation Rθ with corresponding matrix representation given inEquation (16), a function h ∈Ω has circular symmetry (or alternatively ∞-fold rotational

symmetry) if it satisfies the following relationship:

∀θ ∈ R, hRθ = h

Another type of symmetry which many PSFs manifest in practical scenarios is theso-called dihedral symmetry, which is essentially obtained from a combination of axialsymmetry and rotational symmetry.

Definition 11. Given N∈Z≥1, a 2π/N rotation R : R2→R2, and a reflection Sa : R2→R2 across an axis a ∈ R2, we say that a function h ∈Ω has dihedral symmetry of orderN (or alternatively N-fold dihedral symmetry) if it satisfies the following relationships:

hR = hSa = h (17)

37

Supposing that the vector a is oriented at α radians counterclockwise from thex−axis, we can express the reflection Sa in matrix form as follows:

Sα =

(cos2α sin2α

sin2α −cos2α

)(18)

It is useful to make several remarks regarding Definition 11. First of all, it is evident thatany function having dihedral symmetry with order N≥ 2 has also N-fold rotationalsymmetry. Furthermore, any circularly symmetric function has also dihedral symmetry,and in such a case any arbitrary vector a ∈ R2 would be a symmetry axis of h. Anotherimportant observation is that, due to the rotational symmetry of h, we can re-write thesecond identity of (17) as h

(Rk Sa

)= h. It is possible to prove that Rk Sa = SR−k/2a

which implies that h has exactly N symmetry axes. More precisely, if a is one arbitrarysymmetry axis of h, then all the N symmetry axes ak are given by:

ak = Rπk/Na k ∈ 0, . . . ,N−1

The reader should notice that there exist also dihedral symmetries of order 1. By imposingN = 1 in Definition 11 the rotation operator R reduces to the identity transformation, andEquation (17) simply becomes hSa = h, which means that h has only axial symmetry,and no rotational symmetry.

Definition 12. A function h ∈ Ω has axial symmetry if it has dihedral symmetry oforder 1.

There exists an important relationship between symmetric functions in the space-domain and their counterparts in the Fourier-domain, which is formalized in thefollowing statement.

Proposition 13. A function f ∈Ω has N−fold rotational symmetry (or dihedral sym-

metry with symmetry axis a) if and only if its Fourier transform F ∈ΩF has N−fold

rotational symmetry (or dihedral symmetry with the same symmetry axis a).

Proof: From the rotational symmetry of f we have f = f Rk for any k ∈ Z.Applying the FT operator to both sides yields F = F Rk in virtue of Corollary 6. Thesame argument holds for dihedral symmetries. The proof of the converse statement isanalogous.

Now that we have given formal definitions of symmetric functions, we call Ωsym ⊂Ω

the subset of functions in Ω that have either dihedral or rotational symmetry, and we

38

shall investigate the relationships between the elements of Ωsym with respect to theoperation of convolution.

Definition 14. Let us denote by Ωrot(N) and Ωdih(N), respectively the subsets of functionshaving rotational and dihedral symmetry of order N with some fixed symmetry axisa ∈ R2.

The following important statement holds:

Proposition 15. Given two functions h and h′ both in Ωrot(N) (or in Ωdih(N)), the

convolution h∗h′ is also in Ωrot(N) (or in Ωdih(N)).

Proof: Consider the FT of a rotated version of the convolution; from Corollary 6and the convolution theorem we have: F

(h∗h′)Rk

(u) = H(Rku)H ′(Rku). But

from Proposition 13 we know that H(Rku)H ′(Rku) = H(u)H ′(u), and taking theinverse FT yields h∗h′ = (h∗h′)Rk.

We have essentially proved that any subset of N−fold symmetric functions is closed

with respect to the operation of convolution. The importance of this fact will becomeclear in the next section, where we will see that the property of closure is necessary toform algebraic structures called groups. We will need convolution groups in order toformulate the general framework to constructively generate blur-invariants from the soleknowledge of the properties of the group of PSFs of interest.

3.5 Basics of group theory

Before attempting to formulate the framework for blur-invariance, we must introducethe reader to some basic concepts of group theory (more details on the subject can befound in (Rose 1978)).

Definition 16. A group (G,) is a set G with a binary operation : G×G→ G suchthat the following conditions hold:

1. (closure) G is closed under the group operation, i.e. ∀g,h ∈ G gh ∈ G

2. (associativity) The group operation is associative, i.e. ∀g,h, f ∈ G f (gh) =

( f g)h

3. (identity element) G has an identity element, i.e. ∃1G ∈ G such that g1G = g for allg ∈ G

4. (inverse element) All the elements of G have an inverse, i.e. ∀g ∈ G there exists anelement g−1 ∈ G such that gg−1 = 1G

39

We shall remark that the operation of the group does not need to be commutative,however whenever this happens, we can say that:

Definition 17. A group (G,) is commutative (or alternatively, abelian) iff gh = hg

for all the elements g,h ∈ G.

Definition 18. A nonempty subset H ⊆G of a group G is called subgroup of G (denotedby H 6 G) iff it is a group with respect to the operation of G.

An example of group is given by the set of 2×2 real invertible matrices togetherwith the operation of matrix multiplication, and it is often denoted by GL(2) . The readercan easily verify that the four requirements in Definition 16 are satisfied. Analogously,the group O(2) of 2×2 real orthogonal matrices is a subgroup of GL(2).

An important concept that formalizes the idea of “equivalence” between two groupsis expressed by the notion of group isomorphism.

Definition 19. Two groups (G,) and (G′,?) are isomorphic iff there exists an invertiblemapping γ : G→ G′ such that ∀g,h ∈ G:

γ(gh) = γ(g)? γ(h) (19)

The mapping γ is called isomorphism between G and G′. When γ is not bijective, but itsatisfies (19) it is simply called morphism.

Definition 20. An isomorphism γ : G→ G between a group (G,) and itself, is calledautomorphism of G.

3.6 The convolution group

We now use the concepts introduced in the previous section to prove the followingimportant statement:

Proposition 21. The image space (Ω,∗) together with the operation of convolution is a

group.

This is easily verified after applying the Fourier transform to the elements of Ω, andby observing that the Fourier transform F : (Ω,∗)→ (ΩF , ·) is invertible and it mapsthe operation of convolution into pointwise multiplication. Pointwise multiplication isobviously associative, and clearly the multiplication of two functions in ΩF is still in

40

ΩF . The reader can then easily verify that ΩF has an identity element given by thepillbox function:

Θ(u) =

1 |u| ≤ 1

0 |u|> 1(20)

and that the inverse of a function F(u) is simply given by the function 1F(u) , which

always exists because F(u) is bounded and never equal to zero in U . Hence, (ΩF , ·)is a group, and the convolution theorem tells us that the inverse Fourier mappingF−1 : (ΩF , ·)→ (Ω,∗) is a group isomorphism, as it satisfies Equation (19). Weconclude that (Ω,∗) is also a group, and it is interesting to observe that its identityelement is given by Λ = F−1 (Θ), that is:

Λ(x) = (2π)−1 J1(|x|)|x|

(21)

where J1 is the Bessel function of the first kind of order 1 (see Figure 4). We shallobserve as well that the convolution group Ω has subgroups. There are in fact manysubsets of Ω that are closed under convolution, and we now know from Proposition15 that all the subsets of symmetric kernels of any order, i.e. Ωrot(N) and Ωdih(N) areconvolution subgroups of Ω. In order to derive blur-invariant operators we must studyhow these subgroups “act” on the space of images, which is elucidated in the nextsection.

3.7 Group actions and invariance

The main goal of this section is to formalize the concept of invariance within the domainof group theory, and thus provide the necessary tools to derive a general constructiveprocedure to obtain invariance with respect to blur. The underlying idea is to representthe “blur” applied on a sharp image as the result of the “action” of some convolutiongroup on that image. One can intuitively visualize the result of such action by takingone image f ∈Ω and considering all the possible “blurred” version of f , where in thiscontext the blur is generated by convolving f with all the possible PSFs containedin some subgroup H 6 Ω. This operation generates a set of “blurred” images that iscalled “orbit of f ”. It is a remarkable fact of fundamental importance that group actionsgenerate orbits that partition the space Ω (Rose 1978, Olver 1999). This actually meansthat the problem of finding an invariant operator can be now seen as the problem offinding a mapping that would associate each element of Ω to its own orbit.

41

Fig 4. (Left) The identity element Λ of the convolution group Ω defined in Section 3.3: con-volving an image in Ω with Λ leaves the image unchanged. (Right) The corresponding iden-tity element in the isomorphic convolution group ΩF : multiplying the Fourier transform ofan image in Ω by Θ leaves the image unchanged.

Definition 22. Given a group (G,) and a set X , an action of G on X is a mapping

• : G×X → X such that:

1. (identity) 1G • x = x for all elements x ∈ X

2. (associativity) g• (g′ • x) = (gg′)• x for all g,g′ ∈ G and x ∈ X

To give a practical example, by first letting X = Ω, and (G,) = (Ωrot(2),∗), anddefining the mapping •= ∗ to be the convolution between elements of Ωrot(2) and Ω,the reader can easily convince himself/herself that such a mapping is an action. Ofcourse, in this example we have simply defined the action to be the group operationitself. Another less obvious example is given by the action of the group of invertiblelinear transformations on the image space, that is: • : GL(2)×Ω→Ω. In this case thegroup operation corresponding to in Definition 22 would be the matrix multiplication,while the action • could be defined in the following way: M• f = f (M−1x). Such anaction would essentially produce affinely transformed versions of the images.

Definition 23. Given an action • : G×X → X and some element x ∈ X , the set G• x =

g• x | g ∈ G is called orbit of x with respect to the action •.We will sometimes use the notation Ox to denote the set G• x whenever it is clear

from the context which group acts on x.

42

Definition 24. Given an action • : G×X → X , an invariant operator (or more simply,an invariant) on X with respect to the action • is a mapping Γ : X → Y such that for allx ∈ X and for all g ∈ G, the following identity holds:

Γ(g• x) = Γ(x) (22)

The reader might notice that we can trivially obtain an invariant Γ for any action byletting Γ(x) = 0 for all x ∈ X . Of course the trivial invariant is of little or no practicalutility because such a mapping does not allow one to discriminate the orbits in X . Thismotivates one to seek for complete invariants.

Definition 25. An operator Γ : X → Y is a complete invariant with respect to a givenaction on X if the following relationship holds:

∀x∈X, y ∈ Ox ⇐⇒ Γ(x) = Γ(y) (23)

3.8 Construction of invariants with respect to group actions

We now start to explain how we can exploit the algebraic concepts to formulate a generalprocedure to obtain invariants. We should stress that the following reasoning is restrictedto the case of a subgroup acting on the parent group through the group operation. Inprinciple, it would be possible to generalize the following ideas to the more generalscenario of groups acting on some set whose elements are not the elements of the groupitself, however this would require an increased amount of machinery, more concepts tobe introduced, and it might obscure the simplicity of the method. Since the convolutionsubgroups of real PSFs are always subgroups of Ω, we impose the above restriction tosimplify the discussion.

Suppose we are given a group (G,) with an operation , and let H 6 G be asubgroup. We will now write the product gh of two group elements simply as gh, forthe sake of brevity. Note that the mapping : H×G→ G is clearly an action (compareProperties 1-2 of the definition of action with Properties 2-3 in the definition of group).If we want to find an invariant with respect to the action of the subgroup H on G, thenwe must find an operator Γ such that Γ(hx) = Γ(x) for all x ∈ G and h ∈ H. It is notalways easy to directly find a Γ with such property, so we try to achieve the same goal byimposing that Γ is defined as the “product” of two automorphisms of the group G, i.e.:

Γ(x) = γ(x)−1β (x) (24)

43

and we shall now ask ourselves what properties should γ and β have in order to obtaininvariance w.r.t. to the action of H. If we let H act on x we have that for some h ∈ H:

Γ(hx) = γ(hx)−1β (hx)

= (γ(h)γ(x))−1β (h)β (x) (bydefinitionof isomorphism)

= γ(x)−1γ(h)−1β (h)β (x) (inverseelement)

(25)

If we impose the last term in (25) to be equal to Γ(x) we obtain the equation γ(h) = β (h),and by arbitrarily letting β be the identity mapping we have:

∀h ∈ H, γ(h) = h (26)

Equation (26) tells us that it in order for Γ to be an invariant it is sufficient to setβ = id and let γ be an automorphism of G that fixes H. Hence, the problem of findingan operator Γ that satisfies (22) is cast into the usually simpler problem of findingan automorphism γ that fixes H. The above argument was essentially a proof of thefollowing theorem:

Theorem 26. Given a subgroup H of G, and an automorphism γ of G that fixes H, the

operator Γ : G→ G having the form:

Γ(x) = γ(x)−1x (27)

is an invariant with respect to the action of the subgroup H on G.

Of course, the above definition also includes the case of the trivial invariant, i.e.when γ = id, it fixes all the elements of G besides those in H, thus, we shall ask whetherthe invariants having the form of (27) are complete.

Theorem 27. Any non-trivial invariant Γ with respect to the action of a subgroup H on

G that is constructed according Equation (27), and such that γ fixes only the elements in

H is complete.

Proof: We have to prove that Equation (23) holds for any non-trivial Γ definedaccording to Equation (27). The =⇒ statement is already proven by the fact that Γ is aninvariant. It remains to prove that for any two elements x,y ∈ G the identity Γ(x) = Γ(y)

implies that y is in the orbit of x. This can be seen in the following steps:

γ(y)−1y = γ(x)−1x (by definition of Γ)

yx−1 = γ(y)γ(x−1) (isomorphism of inverse element)

yx−1 = γ(yx−1) (definition of isomorphism)

(28)

44

Fig 5. Pictorial representation of the convolution subgroup Ωrot(5) acting on a image of Ω

(marked with a red box) and the corresponding elements of its orbit. The image on thetop-left corner is Λ, the identity element of Ω (Note: the convolution kernels in Ωrot(5) weremagnified for the sake of visibility).

By hypothesis, any element of G that is fixed by γ must be in H, thus from the lastidentity in (28), we know that the element h = yx−1 is in H, which in turn implies thaty = hx, so we conclude that y ∈ Ox, and the invariant Γ is complete.

3.9 Constructive calculation of blur-invariants

At this point, we are already able to constructively derive blur-invariants for any kind offamily of symmetric kernels. Simply suppose for instance that G = Ω, and H = Ωrot(N)

for some integer N≥ 1. The group operation is the convolution between two functionsin Ω (Figure 5). We want to find a complete invariant Γ : Ω→Ω with respect to theaction ∗ : Ωrot(N)×Ω→Ω given by the convolution of the rotationally symmetric PSFswith the other functions in Ω (Figure 5). From Theorems 26-27 we know that we havejust to find some automorphism γ that fixes only the elements of Ωrot(N). In fact, we donot even need to look for it, as the automorphisms that fix the elements of Ωrot(N) are bydefinition all the transformations of the rotation group of order N. Therefore, for any

45

image f ∈Ω, we simply define:

γk( f ) = f R−k (29)

for some k ∈ 1, . . . ,N−1. A complete invariant with respect to rotationally symmetricblur is then simply obtained by:

Γk( f ) =(

f R−k)−1∗ f (30)

where the −1 denotes the convolution inverse. Clearly, the convolution inverse isbetter represented in the Fourier domain, where convolution becomes a pointwisemultiplication, and rotations (and orthogonal transformations) are preserved from thespace domain, so we can easily obtain the blur-invariants in ΩF by first applying thegroup isomorphism F : Ω→ΩF and defining:

Γk(F) =F

F R−k (31)

which are the well-known invariants discussed in detail in Paper II. A totally analogousreasoning can be applied to derive the dihedral blur-invariants presented in Paper III.

It is quite remarkable that from the sole knowledge of the degree of symmetry of thePSFs it was possible to obtain effortlessly, and in a constructive manner, the invariantsin Equations (30)-(31), without resorting to complicated algebraic manipulations ordifferent kinds of integral transforms. In the next chapter we will use the knowledgepresented so far to show that also blur-invariant image registration can be formulated interms of this group-theoretical framework.

46

4 Blur-invariant image registration

In the previous section we have demonstrated how to obtain blur-invariant quantities in aconstructive manner, and from the sole knowledge of the geometric properties of theblur PSFs. Although such invariants could be directly used in applications involvingrecognition of blurred images, our goal is to utilize them to accurately register blurredimages. The interest in blur-invariant image registration is justified by the fact thathaving a set of correctly aligned blurred images, is a necessary requirement for severalrestoration techniques based on the fusion of multiple images, like multichannel blinddeconvolution or depth-of-field extension. In this section we reformulate the problemof blurred image registration within the context of group theory, and show how theconcepts introduced in the previous section can be used to easily re-derive the mainresults presented in Papers II-III.

4.1 Translation as the action of convolution groups

Let us suppose that f ∈ Ω is a grayscale image and let us consider another image g

defined as the translated version of f , i.e.:

g(x) = f (x− s) (32)

for some shift vector s ∈ R2. It is well-known that, from the definition of delta function,the above equation can be rewritten as:

g(x) = ( f ∗δs)(x) (33)

where δs(x) = δ (x− s) is a delta function centered at some s ∈ R2. Since the functionsin Ω are band-limited by definition, we can obviously replace the shifted delta functionδs in (33) with Λs(x) = Λ(x− s), the shifted identity element of the convolution groupΩ, i.e.:

g = f ∗Λs (34)

Although this substitution might appear irrelevant, we have now expressed the imageg as the convolution of two elements of the group (Ω,∗), as both f ,Λs ∈ Ω, whileδs /∈Ω. The reader should also note that the subset S =

Λs ∈Ω |s ∈ R2

⊂Ω of all

the translated versions of Λ forms a subgroup of (Ω,∗), and that the subgroup S is

47

isomorphic to the group of 2D translations. In fact, the translated image g in Equation(34) is an element of the orbit of f under the action of S on G, that is: g ∈ S ∗ f . Ofcourse, in the above discussion, we have assumed that f ∈ΩrS; in other words, theimage itself cannot be an element of S.

In the next section we reformulate the problem of image registration in terms of asimple operation between group elements.

4.2 Image registration as an operation between group elements

The discussion found in the previous section should have made it clear that Equation(34) not only tells us that the image g ∈ Ω is defined as the translated version of f ,but also that g is given by the product of two elements of the group (Ω,∗). In imageregistration, the two images g and f are known, and one wishes to recover the shift sbetween them. Since the subgroup S is isomorphic to the group of translation, there is aone-to-one correspondence between the vectors s ∈ R2 and the functions Λs ∈ S. Thisimplies that recovering the shift s is clearly equivalent to recovering Λs in Equation(34). But since the right term of (34) is nothing more than a product between two groupelements, we should be able to obtain Λs by multiplying g by the inverse of f :

Λs = f−1 ∗g (35)

We now define the function Θs(u) = Θ(u)e−iu·s as the pillbox function (identity elementof the group (ΩF , ·)) multiplied by the complex sinusoid e−iu·s. If we consider theFourier transform of both sides of (35) we obtain:

Θs =

GF

=GF∗

|F |2=

GF∗

|GF∗|(36)

where we used the superscript ∗ to denote complex conjugation. The reader mightrecognize that the rightmost term of (36) is the well-known formula for calculating thephase correlation in the frequency domain between the two images, thus Equation (35)is essentially equivalent to performing a phase correlation in the frequency domain andthen taking the inverse FT, which yields Λs. The actual shift s is simply extracted fromΛs by finding the location of the global maximum of Λs.

Before moving forward, it is also beneficial to emphasize that the group theoreticalinterpretation of the problem of image registration has immediately led us to Equations(35)-(36). These equations respectively involve the terms f−1 and F−1, which clearlysuggests that there is an intimate relationship between phase correlation and inverse

48

filtering. We will come back to this point in Chapter 5. We first continue our discussionby presenting a group theoretical approach to blur-invariant phase correlation.

4.3 N-fold blur-invariant registration

In Section 3.9 we have learned how to constructively derive blur-invariants with a grouptheoretical approach. We can now combine that knowledge with the observations madein Section 4.2 in order to derive a general method to perform blur-invariant imageregistration. Suppose that two observed blurred images f , g obey the following model:

f (x) = ( f0 ∗h1)(x)g(x) = ( f0 ∗h2)(x− s)

(37)

where h1,h2 ∈Ωrot(N) are two unknown PSFs having N-fold rotational symmetry, andf0 ∈Ω is a sharp unknown image. Clearly, we can write f0 ∗h2 = ( f0 ∗h1)∗ (h−1

1 ∗h2),and by defining a term h as:

h = h−11 ∗h2 (38)

we observe that the above model is equivalent to:

g(x) = ( f ∗h)(x− s) (39)

for some h ∈Ωrot(N). By re-expressing the shift between f and g as a convolution with ashifted delta function, we can rewrite (39) as:

g = h∗ f ∗Λs (40)

This equation is very similar to (34), with the only difference that now the right term isconvolved with the PSF h which is unknown. However, it is important to observe thatthe rightmost term of (40) is simply a product of elements of the group Ω, and since theelements Λs have no symmetry (i.e. S∩Ωrot(N) = Λ), we know that finding a completeinvariant Γ w.r.t. to the action of Ωrot(N) will ensure that when applying Γ to g in (40),the shift function Λs is preserved, while the effect of h is canceled. We shall explain thisin more detail. According to Equation (30), the invariants to N-fold rotational blur aregiven by:

Γk(g) = γk(g)−1 ∗g =(

gR−k)−1∗g (41)

where the superscript −1 denotes the inverse w.r.t. to convolution, and γk is an automor-phism of Ω that fixes the elements of the subgroup Ωrot(N), i.e.: γk(h) = hR−k = h

49

for all h ∈ Ωrot(N), where k ∈ 1, . . . ,N−1. Before proceeding we should list twoimportant properties of the functions Λs whose proofs are left to the reader:

Λx ∗Λy = Λx+y (42)

Λx R−1 = ΛRx (43)

where x,y ∈ R2 and R is a 2π/N rotation operator with corresponding 2×2 rotationmatrix R. Now, since Γk is a morphism2, applying Γk to both sides of (40) yields:

Γk(g) = Γk(h∗ f )∗Γk(Λs) (morphisms preserve group operation)

= Γk( f )∗Γk(Λs) (invariance property of Γ)

= Γk( f )∗ γk(Λs)−1 ∗Λs (definition of Γ)

= Γk( f )∗(Λs R−k

)−1 ∗Λs (definition of γ)

= Γk( f )∗Λs−Rks (Eqs. (42)-(43))

(44)

Since both Γ(g) and Γ( f ) are known, we can proceed as done in Section 4.2 bymultiplying both sides of (44) by Γk( f )−1 and recover Λs−Rks and its peak locationpk = s−Rks:

Λpk = Γk( f )−1 ∗Γk(g) (45)

which corresponds in the frequency domain to:

Θpk =

Γk(g)Γk( f )

=Γk(g)Γk( f )∗

|Γk( f )|2=

Γk(g)Γk( f )∗

|Γk(g)Γk( f )|(46)

in evident analogy with (44). Equation (46) can be interpreted as a “phase correlation”between the blur-invariants Γk of f and g (see Figure 6). By replacing Γk with theexplicit formula in (30) (or alternatively (31) if we are considering the group (ΩF , ·))one can easily realize that we have remarkably reproduced the results of Papers II-III bymechanically applying the general operators Γk to Equation (40).

Of course, all the above arguments are trivially generalized to the case of dihedralsymmetry, h ∈Ωdih(N). In fact there are N additional automorphisms that fix Ωdih(N),and they are given by the reflections across the symmetry axes. The N additionalinvariants are given by:

Γ′k(g) = γ

′k(g)

−1 ∗g =(

gSa R−k)−1∗g (47)

2The fact that Γk is a morphism can be proven from (27), and from the observation that (Ω,∗) is a commutativegroup.

50

Fig 6. Two differently blurred images to be registered: (Top-left) The image f . (Top-right)The image g which is a blurred and translated version of f (the 5-fold blur PSF h is alsoshown). (Bottom-left) The correlation spectra Λp1 , . . . ,Λp5 obtained by (45) superimposed forthe sake of illustration according to the formula Λp1 + . . .+Λp5 ; the green arrow representsthe true shift s. (Bottom-right) The result of applying ordinary phase correlation accordingto (35). (Adapted from Paper II, c©IEEE).

51

where k ∈ 0, . . . ,N−1, and the symmetry operator Sa is defined as in Definition 11.We postpone the discussion on how to determine the true shift s from the quantitiespk = s−Rks to the next section.

4.4 Estimation of translational shift: the geometry of the N-foldpeaks

From Equations (45)-(46) we learned that, in general, N−1 peaks can be extracted bylocating the global maxima of Λpk , i.e.:

pk = argmaxx

Λpk(x)

= argmax

x

F−1 Θpk(x)

(48)

and we saw from (44) that such rotational peaks are located at:

pk = s−Rks (49)

where k ∈ 1, . . . ,N−1. It is evident that one single peak is theoretically sufficient torecover the shift vector s, in fact, (49) can be rewritten as:(

I−Rk)

s = pk (50)

where I is the 2×2 identity matrix. The matrix(I−Rk

)is always invertible, so s is

obtained by left-multiplying both terms of (50) by(I−Rk

)−1. This also confirms thatcalculating all the N−1 complete invariants Γk is redundant, as they contain the sameinformation; this fact is thoroughly discussed in Paper II. However, one could takeadvantage of such redundancy by observing that the locations of the peaks pk in (49)follow an interesting geometric pattern. In fact, the term −Rks in (49) denotes thevector s rotated by 2π

N k+π radians, therefore, the peaks pk are located at equally spacedpositions on a circle centered at s and having radius s (thus, a circle passing through theorigin); this is illustrated by the blue dots in Figure 7. With this knowledge, the problemof recovering s from such a configuration of peaks can be seen as the problem of fittinga circle to N−1 points on the plane, with the constraint that the circle must pass throughthe origin. The center of such a circle represents the shift s between the two images. Thedetails of a simple and robust method to perform the circle-fit are described in Paper II.

When the PSF h has dihedral symmetry, there are N additional peaks qk (see PaperIII) that we call dihedral peaks, and that are located at:

qk = s−SaRks (51)

52

where k ∈ 0, . . . ,N−1. Since−Rks = pk +s, we have that qk = s−Sa (pk + s), whichclearly means that the peaks q1, . . . ,qN−1 are located on the same circle, at positionsthat correspond to the positions of p1, . . . ,pN−1 reflected across the line parallel to thesymmetry axis a and passing through s (see Figure 7, left). By the same reasoning, thepeak q0 = s−Sas is the reflection of the point at the origin (we call it p0) across the lineparallel to the symmetry axis a and passing through s. Another important geometricalrelationship (see Paper III for details) that emerges from (51) is that:

qk = 2ProjRk/2a⊥(s) (52)

which essentially determines that the dihedral peaks q0, . . . ,qN−1 are exactly the doubleof the orthogonal projections of s onto the N−1 lines passing through the origin andthe points Rk/2a⊥ (where a⊥ is some vector perpendicular to the symmetry axis a), asillustrated in Figure 7. This consequently means that, differently from the rotationalpeaks pk, each of the dihedral peaks qk is constrained to move along straight lines. Sucha constraint is useful when extracting the global maxima qk from the correlation spectra.Clearly, when h has dihedral symmetries, recovering s is equivalent to fit a circle passingthrough the origin to 2N−1 points, i.e. p1, . . . ,pN−1,q0, . . . ,qN−1.

Before concluding this chapter, we shall mention that, interestingly, if h has onlyaxial symmetry, i.e. h ∈Ωdih(1), then according to (51)-(52), the only peak that can berecovered is q0 = s−Sas = 2Proja⊥(s). Obviously, the projection operator cannot beinverted, and therefore s cannot be recovered from q0. This is also easily verified byobserving that the matrix (I−Sa) is always singular. It is also worth noticing that theblur-invariant registration methods described in this chapter require the parameters N(number of folds of the PSF) and a (symmetry axis of the dihedral PSF) to be known.Some techniques for estimating these two parameters are described in Paper III.

53

Fig 7. (Top) The geometry of the correlation peaks: the true shift between two blurred im-ages f , g is depicted by the black arrow. The images were blurred respectively with 3-folddihedral PSFs h1, h2 having the symmetry axis a oriented in the direction of the green dashedline. The rotational peaks are depicted in blue, and the dihedral peaks in red. The regularpolygons inscribed in the circle visually emphasize that rotational and dihedral peaks arealways equally spaced. The red dotted lines represent the lines along which the dihedralpeaks can move. (Bottom) When the shift vector between the two images changes, thedihedral peaks still lie on the same lines.

54

5 Blur-robust registration of noisy images byWiener filter

In the previous chapter we described how to register two blurred images under theassumption that the observed images f and g follow the model expressed by Equations(37)-(39). That model clearly neglected the presence of noise in the images. We shall nowinclude noise in our model and describe an efficient way to register images containingboth N-fold blur and additive noise. In order to do that, we will reformulate the problemof translational image registration as a particular application of the deconvolution oftwo-dimensional signals.

5.1 Brief review of Wiener filter

Given an observed image defined as:

g = f ∗h+n

where h is a known convolution kernel and n is random noise, one seeks to find an esti-mate f for the unknown image f . This is clearly a problem of deconvolution/denoisingof signals. In this context, both f and h are assumed to be deterministic, while n is awide-sense stationary (WSS) random processes that is real valued, square integrable,and whose power spectral density (PSD) Ψn is known (see Papoulis & Pillai (2002) forformal definitions). For a WSS random process g, square-integrability is expressed bythe relationship E

´R2 |g(x)|2d2x

=´R2 E

|g(x)|2

d2x < ∞, where E is the expected

value operator. The power spectral density for a WSS random process g is given byΨg(u) = E

|G(u)|2

, where G is the Fourier transform of g. The FT of g is defined

as E´

R2 g(x)e−iu·xd2x=´R2 E g(x)e−iu·xd2x. The technique of Wiener filtering

consists in finding the estimate f = g∗w obtained by convolving the observed image g

with some filter w, such that the mean squared error between f and f is minimized. Thisminimization problem is more easily expressed in Fourier domain as:

W (u) = argminW (u)

E|G(u)W (u)−F(u)|2

55

for all u ∈ R2. The solution can be proven to have the following closed-form:

W (u) =H(u)∗

|H(u)|2 + Ψn(u)|F(u)|2

and by adopting the more succinct notation in which the argument (u) is omitted, wecan write the Wiener filtered estimate F as:

F =GH∗

|H|2 + Ψn|F |2

(53)

The estimate f in space-domain is obtained by taking the inverse FT of F .

5.2 Noisy image registration

Before introducing noisy images in our discussion, we shall redefine our space of imagesto accommodate random signals.

Definition 28. In complete analogy with Definition 7, we define ΩE as the space oftwo-dimensional continuous wide-sense stationary (WSS) random processes that arereal valued, square integrable, and whose power spectral densities have support in theunit ball U , i.e.:

ΩE =

g ∈ L2E (R

2) |supp(Ψg) =U

(54)

The definition of the space ΩE allows us to model a noisy, translated image as:

g = f ∗Λs +n (55)

where f ,Λs ∈Ω⊂ΩE are deterministic and g,n ∈ΩE are random processes. The noisen is assumed to have zero mean (i.e. E n(x)= 0) and known power spectral densityΨn. In the frequency domain, Equation (55) becomes:

G = FΘs +N (56)

After observing that |Θs|2 = 1 and assuming f is known, we can estimate the term Λs

using the Wiener filtering technique, i.e.:

Λs = F−1(

GF∗

|F |2 +Ψn

)(57)

56

This is approach is further described (and generalized to the case of color images)in Paper IV. Observe that when the noise n is white and Gaussian, we can writeΨn(u) = σ2, where σ2 is the noise variance, therefore (57) becomes:

Λs = F−1(

GF∗

|F |2 +σ2

)(58)

Note also that, when no noise is present we have Ψn = 0, therefore the term inside thebrackets on the right side of (57) becomes G

F , which is clearly an operation of inversefiltering, and which happens to be the same quantity expressed in in Equation (36). Thissuggests that the phase correlation algorithm is essentially an inverse filtering neglectingthe noise term (Figure 8). Once we realized that the problem of noisy image registrationcan be expressed in terms of Wiener filtering, it is legitimate to ask if it is possibleto incorporate blur in Equation (55) and recover the registration shift using the sameapproach as described above. We now demonstrate that this is indeed possible. Thefollowing discussion is a contribution of this thesis that extends the theory presented inPaper IV.

5.3 Obtaining blur and noise robustness

In this section, for the sake of clarity, we will occasionally simplify the notation adoptedso far for representing rotated functions in the following way: for some function f ,a fixed index k ∈ 1, . . . ,N−1, and u ∈ R2, the quantity ( f R−k)(u) will be simplywritten as Rk f .

Let us have two images f ∈Ω and g ∈ΩE such that:

g = f ∗h∗Λs +n

= f ∗hs +n(59)

where h ∈Ωrot(N) is a deterministic and unknown blur kernel with N-fold symmetry,hs = h∗Λs is the shifted version of h, and n is zero-mean noise having known PSD Ψn.Without loss of generality we will assume n to be white noise with variance σ2. In theFourier domain (59) can be written as:

G = FHΘs +N

FHs +N

The shift function Λs (or equivalently its Fourier transform Θs) can be estimated byapplying a Wiener filter followed by an inverse filter. In fact, we can first obtain an

57

Fig 8. (Top-left) Original image. (Top-right) Translated version of the original image withadditive white Gaussian noise. (Bottom-left) Peak generated by ordinary phase correla-tion. (Bottom-right) Peak generated by using the Wiener filter registration of Equation (58).(Adapted from Paper IV, c©IEEE).

58

estimate for Hs = HΘs by denoising and deconvolving G using a Wiener filter:

Hs =GF∗

|F |2 + Ψn|H|2

(60)

At this point the reader should observe that by calculating the quantity Γk(Hs) using(31) one should obtain an estimate for Θpk , which is the FT of the peak function Λpk

seen in Equation (45), i.e.:

Θpk = Γk(Hs) =Hs

RkHs(61)

which is essentially an inverse filtering preceded by a Wiener filtering. After estimatingthe functions Θpk for k ∈ 0, . . . ,N−1, the estimate for the shift s can be obtained inthe same way as described in Chapter 4 and in Papers II-III. To understand the nature ofthis sequence of operations, it is sufficient to plug (60) into (61), and one obtains:

Θpk = Γk

(Hs)=

GF∗

(RkG)(RkF)∗

|RkF |2 + σ2

|H|2

|F |2 + σ2

|H|2

which can be rewritten as:

Θpk = Γk(G)Γk(F)∗

|RkF |2 + σ2

|H|2

|F |2 + σ2

|H|2

(62)

= Γk(G)Γk(F)∗B (63)

Interestingly, the rightmost term of (62) represents the cross-correlation in the frequencydomain between the blur-invariant operators of G and F filtered with B. When nonoise is assumed to be present, then σ2 = 0 and B = |Γk(F)|−2, hence (62) reducesto the well known formula of phase correlation between the N-fold blur-invariantsof G and F , i.e. Θpk = Γk(G)

ΓK(F) . On the other hand, when σ2 becomes large, we have

that limσ2→∞B = 1, and Θpk → Γk(G)Γk(F)∗, which is simply the cross-correlation

between the two invariants.From (62) one can immediately see that the estimator Θpk requires the knowledge

of |H|2, the power spectrum of the blur kernel h, which is unknown. In Paper III, astraightforward method for estimating |H|2 is described in detail, and in particular, it isshown that the PSD of h can be estimated by calculating:

|H|2 = PN |G|2

PN |F |2(64)

59

where PN is the projection operator (Flusser et al. 2014), which extracts the N-foldsymmetric part of a function, and it is defined as the average of the N rotated versions ofthe function, i.e.:

PNF =1N

N

∑i=1

R iF

From the observations made in this chapter, we have proved that the phase-correlationregistration algorithm proposed by Kuglin (1975), De Castro & Morandi (1987) isessentially an operation of inverse filtering that recovers the delta function expressingthe shift between the two images, according to the model (55), and neglecting noise. Inthis chapter we generalized this idea by incorporating noise and blur in the model, andwe used Wiener deconvolution to recover the shifted delta function.

In the next chapter we further extend this generalization to color images, representedas quaternion-valued functions. It is interesting to anticipate that in the case ofquaternionic images, the original formulation of phase-correlation expressed in Equation(36) does not have a unique generalization. In fact, Moxey et al. (2003) replaced thenumerator of (36) with a quaternionic generalization of the cross-correlation. Thisapproach leads to a formula that is different from the one for the inverse filtering ofquaternionic signals, in contrast to the case of scalar signals. Quaternion inverse filteringand quaternion Wiener filtering have not been considered so far as a tool for registeringcolor images, which will be the main topic of Chapter 6.

60

6 Blur-invariance and registration of colorimages

The techniques described so far are primarily aimed at registering grayscale imagesdegraded by blur and/or noise. For this reason, even when the user has a pair of RGBcolor images at his/her disposal, he/she would typically retain only the luminanceinformation by calculating an average of the three color channels, and then apply one ofthe registration techniques presented in the previous chapters. Since most acquisitiondevices are nowadays capable of capturing color images, it is legitimate to ask howcolor information may contribute to increasing the registration performance betweendegraded images. Certainly, one could treat an RGB color image as three separategrayscale images for each color channel, and perform a channel-by-channel imageregistration. Nevertheless, this naive approach would inevitably produce the estimates ofthree different shifts, and it would require a criterion to combine these three estimatesinto a single one. Since the registration techniques proposed so far are based on theuse of Wiener filter, the most straightforward way to adapt them to the case of colorimages, is to generalize the Wiener filter itself to handle “vector-valued” signals. In theimage processing literature, it has been frequently proposed to encode a color tripletas three components of a quaternion. By following this strategy we can deduce thatblur-invariant color image registration could be efficiently done using a quaternion

Wiener filter (QWF).

6.1 Quaternion representation of color images

A quaternion q∈H is defined as q = q0+q1i+q2j+q3k, where the quantities q0, . . . ,q3

are real numbers, and i, j,k are the imaginary units that follow the relationship i2 =j2 = k2 = ijk =−1, where juxtaposition stands for multiplication. Such a relationshipbetween the imaginary unit allows one to define the multiplication of two arbitraryquaternions. Analogously to complex numbers, the conjugate q of a quaternion isdefined as: q = q0− q1i− q2j− q3k and its norm |q| is: |q| =

√q0 2 + . . .+q3 2. As

discussed in (Sangwine 1996), a convenient way to represent a color image is the one ofdefining f as a two dimensional function whose values are quaternions, so that:

61

f (x) = `(x)+ c1(x)i+ c2(x)j (65)

where `,c1,c2 are scalar-valued functions that typically encode the luminance and thetwo chrominance coordinates respectively. By using basic manipulations of quaternionalgebra, one can split the right side of (65) into a pair of two complex-valued functions:

f (x) = f+(x)+ f−(x)i (66)

where f+(x) = `(x) and f−(x) = c1(x)+ c2(x)k. The choice for the symbols + and −as subscripts is merely a notational convention adopted by Hestenes & Sobczyk (1987)that originates from the concept of even-grade and odd-grade part of a quaternion, whenrepresented within the Clifford algebra C `0,2(R) (Jost 2009). The main advantagebrought by the representation of color images in Equation (66) is that one can easilydefine from (66) the quaternion Fourier transform (QFT) of f as two ordinary complexFTs:

F(u) = F+(u)+F−(u)i (67)

Here, F+ is the complex Fourier transform of the luminance channel f+, and F− is thecomplex FT of f−, which is the chrominance of f interpreted as a complex-valuedsignal in which the real and imaginary parts represent respectively the channels c1,c2.Before introducing a formulation of Wiener filter for quaternion signals, it is worthmentioning that, given two signals f ,g such that f is quaternion-valued and g is real, theconvolution theorem takes the usual form:

f ∗g F7−→ FG

where the operator F denotes the QFT (Hitzer & Mawardi 2008).

6.2 The quaternion Wiener filter

The space of images ΩE can be extended in order to contain quaternion-valued signals.This is simply accomplished by replacing in Definition 28 the space L2

E (R2) with the

space L2E (R

2;H) of quaternion-valued square-integrable 2D random processes. Let usconsider a blurred and noisy color image g defined as follows:

g = f ∗h+n (68)

where:

62

– f ∈ΩE is defined according to (65)-(66), and such that f`, f1, f2 are uncorrelated.– h ∈Ω is deterministic, real-valued, with known power spectrum.– n ∈ΩE is noise defined as n = n`+n1i+n2j, having known PSD Ψn, and such that

n`,n1,n2 are uncorrelated.

Under these assumptions, it is possible to prove (see Paper IV for details) that anestimate for the QFT h is given by:

H(u) =G(u)E F(u)

E |F(u)|2+ Ψn(u)|H(u)|2

(69)

If we replace the expectation operators in (69) with estimates extracted directly from theobserved signal f , then, one has:

H(u) =G(u)F(u)

|F(u)|2 + Ψn(u)|H(u)|2

(70)

It is evident that, in analogy with the ordinary Wiener filter, when no noise is present,(70) reduces to an inverse filtering:

H(u) = G(u)F(u)−1

The only differences between (70) and the corresponding formula for real signals arefound in the numerator. In fact, the quantity G(u)F(u) now involves a quaternionmultiplication, and a quaternion conjugation. Using the quaternion split of Equation(66), the right term of (69) can be expanded into:

H =G+F+ ∗+G−F− ∗+(G−F+−G+F−) i

|F+|2 + |F−|2 + Ψn|H|2

(71)

where, for the sake of brevity, we omitted the function argument u in each term. Sincethe estimated function h should be real-valued, it follows that the odd-grade part of H

should be zero. Imposing such constraint yields:

H =G+F+ ∗+G−F− ∗

|F+|2 + |F−|2 + Ψn|H|2

(72)

It can be proven that Equation (72) is equivalent to the formulation of multi-channelWiener filter described in (Campisi & Egiazarian 2007). If the noise n is white withchannel variances σ2

` ,σ21 ,σ

22 , then:

Ψn(u) = σ2` +σ

21 +σ

22 = σ

2

63

It is important to remark that, since the QWF assumes the channels of f to be uncorre-lated, the color space used to represent the color triples of each pixel should be chosensensibly. In Paper IV a color space transformation T : R3→R3 satisfying the followingproperties is suggested:

– T decorrelates the RGB color channels.– T preserves the additivity of noise (i.e., noise must be additive before and after the

application of T ).– T preserves convolution, i.e.: T ( f ∗h) = T ( f )∗T (h).

As remarked in Paper IV, the above requirements can be fulfilled by applying a rotation ofthe RGB color space. The rotation matrix can be obtained either by PCA, or by imposingthe luminance axis to have specific RGB coordinates, like [0.299, 0.587, .0.114], inanalogy with the luminance axis of the YUV color space.

6.3 Registration of noisy color images

In this section we generalize the noise-robust registration method described in Section5.2 to the case of color images. Since the registration formula expressed in (57) is aWiener filter, its generalization is easily achieved using the QWF introduced in theprevious sections of this chapter. The details of this approach are discussed in Paper IV.Given two color images f ,g ∈ΩE such that:

g = f ∗Λs +n

where f ∈ΩE is a color images with uncorrelated color channels, n is noise with knownPSD Ψn and such that the noise channels are uncorrelated with each other. The functionΛs is of course scalar valued, and it represents the shift between f and g; its powerspectrum is clearly |Θs|2 = 1. Assuming, without loss of generality, white noise, suchthat Ψn(u) = σ2, we can use the QWF of Equation (70) to estimate the FT of Λs:

Θs =GF

|F |2 +σ2

Using the expanded form expressed in (72), the above quantity becomes:

Θs =G+F+ ∗+G−F− ∗

|F+|2 + |F−|2 +σ2 (73)

The inverse QFT of Θs directly yields an estimate of the shift between f and g (Figure9).

64

Fig 9. (Top-left) Original image. (Top-right) Translated version of the original image withadditive white Gaussian noise. (Bottom-left) Peak generated by ordinary phase correlationof the luminance channels. (Bottom-right) Peak generated by using the QWF registration,i.e. by taking the inverse FT of (73).

65

6.4 Blur-robust registration of noisy color images

We have so far discussed how to utilize the QWF in order to register color noisyimages. In order to perform blur and noise-robust color image registration, it issufficient to follow the same approach described in Section 5.3. Let us consider twocolor images f ,g ∈ΩE fulfilling the assumptions listed in Section 6.2, and such that:g = f ∗h∗Λs +n = f ∗hs +n, where h is a real-valued N-fold symmetric PSF. Since hs

is real-valued, the QFT of G is G = FHs +N. In analogy with (60), an estimate forHs can be obtained from the color images f ,g using (73). Since the estimate Hs isreal-valued, one can estimate (the FT of) the peak function Θpk by calculating Γk(Hs)

using Equation (61). Also in this case, this sequence of operations essentially consists inapplying a (quaternion) Wiener filter to reduce noise, followed by an inverse filter toeliminate the effect of N-fold blur.

6.5 Color blur-invariants

Before concluding this chapter, we briefly discuss how the blur-invariant operatorscan be extended to the case of color images. This is useful in applications like blurredimage recognition. The blur-invariant operator Γ in Equations (30)-(31) turns out to berelatively straightforward to generalize to the case of color images. In fact, let us supposethat f ∈Ω is a color image, and that h ∈Ωrot(N) is a real-valued N-fold symmetric PSF.If we consider the blurred color image g = f ∗h, we have from the convolution theoremof the QFT that G = FH. By defining the operator Γk as:

Γk(G) = G(GR−k)−1 = G(RkG)−1

we have:

Γk(FH) = FH(RkF H

)−1

= FH H−1(RkF)−1

= F(RkF)−1

= Γk(F)

which proves that Γk is an invariant w.r.t. blur with N-fold symmetric real-valuedPSFs. The reader may have noticed that, since quaternion multiplication is in generalnon-commutative, we had to use the right-multiplication of the term (RkG)−1 in thedefinition of Γk.

66

7 Applications

The methods presented in the previous chapter are almost entirely based on the conceptof invariance w.r.t. image blur with point-spread functions having certain kind ofrotational symmetries. This fact alone, may already suggest what the applications of thetechniques described in this thesis are. In fact, PSFs that manifest rotational symmetriesmostly arises from out-of-focus blur generated by acquisition devices equipped with ashutter composed of several blades. We shall deduce that important applications are to befound in those scenarios where the user is interested in registering or recognizing imagesdegraded by out-of-focus blur and noise. Some of these applications are discussed morein detail in this chapter.

7.1 Multi-channel blind deconvolution

An important problem in computer vision is that of recovering a “sharp” image fromone (or multiple) blurred photo(s). Many different approaches have been proposedto address this problem (see Campisi & Egiazarian (2007) for an overview) andmost of them are targeted at deblurring single images degraded by specific typesof blur, e.g. motion-blur, out-of-focus blur, spatially-variant blur, and so forth. Asuccessful and well-known approach to image deblurring is the one of multi-channelblind deconvolution (MBD) proposed by Sroubek & Flusser (2005). Their method doesnot impose specific constraints on the geometry of the PSFs, but on the other hand, itrelies on the assumptions that several images of the same subject are at the disposal ofthe user, and that:

– each image is blurred with different PSFs– all the images differ from each other only by the amount/type of blur.

The second assumption means that there should be no geometric transformation thatrelates one image to another image, therefore each image should ideally depict exactlythe same static scene blurred in different ways. This assumption is difficult to fulfill inpractice, and typically image registration must be used to align all the images beforeapplying multichannel deconvolution. This is a typical scenario where blurred imageregistration is most useful. Figure 10 shows a successful example of multichannel deblurapplied to a pair of images with different blurs, taken with a hand-held camera. Due

67

to camera shake the two blurred images had first to be registered using the algorithmof N-fold dihedral “phase correlation” described in Paper III. More experiments aredocumented in Papers III-IV.

68

Fig 10. (Top) Two images taken with different focus settings using a hand-held camera.(Middle) Spectral patterns obtained using the 6−fold dihedral phase correlation method ex-plained in Chapter 4 (the red parts represent the dihedral peaks, while the green parts therotational peaks). (Bottom) Result of multichannel blind deconvolution of Sroubek & Flusser(2005). (Adapted from Paper III, c©IEEE).

69

7.2 Extended depth-of-field

In many circumstances, it happens that the acquisition device has a relatively narrowdepth-of-field (DoF), and as a consequence, only a small portion of the capturedscene appears in focus. This often happens in microscopy (Wu et al. 2010) or macrophotography (Savazzi 2010). A popular technique to extend the DoF is that of takingseveral shots using different focus settings, and then fuse the images in such a waythat the resulting image appears all in focus (Sroubek et al. 2005). Similarly to MBD,also DoF-extension techniques are based on the assumption that the input images areperfectly aligned and only the focus settings of the camera change between the shots.Whenever this assumption is difficult to fulfill, one has to resort to image registration toalign the images. However, the same portion of the scene is likely to appear in focus inone image, and out of focus in the other images, so that a registration algorithm that isrobust to blur must be used. Although the out-of-focus blur is non-uniform in imageswith narrow DoF, it is demonstrated in Paper III that the registration techniques describedin this thesis can be successfully utilized as a pre-processing step to DoF-extensionalgorithms (Figure 11).

70

Fig 11. (Top) Two frames from a video sequence taken with automatic focus mode; the PSFof a dihedral shape (N = 4) is clearly visible in the second frame due to the bright spotsin the foreground which are out of focus. (Middle) Spectral patterns obtained using the4−fold dihedral phase correlation method explained in Chapter 4 (the red parts representthe dihedral peaks, while the green parts the rotational peaks). (Bottom) Result of the focusstacking algorithm by Sroubek et al. (2005) after having registered the images. (Adaptedfrom Paper III, c©IEEE).

71

7.3 Blurred texture recognition

Blur invariant operators find a large use in computer vision applications involvingrecognition and classification of objects, textures, and faces (Ahonen et al. 2008). Insuch applications, one typically aims at extracting certain information from the images(or from local portions of the image) that allows one to construct descriptors which canbe used to recognize the presence of certain kinds of objects in an image (e.g. faces, cars,and so forth) or to classify a segmented object or a texture into a certain category. Inimage recognition/classification, the concept of invariant is constantly utilized in orderto achieve different goals, e.g. build feature descriptors that are invariant to geometricdistortions (e.g. rotations, scaling, affine transformations) (Rahtu 2007), and/or toradiometric distortions such as brightness, contrast, illumination changes. Recently,also blur is considered to be in the list of radiometric changes that are more frequentlytaken into account in problems of image recognition, and as a consequence there hasbeen a growing interest in developing blur-invariant local descriptors. This interestmotivated the development of local phase quantization (LPQ) descriptors (Ojansivu &Heikkilä 2008a). Although it is reasonable to suspect that introducing blur-invariancemay decrease the discriminative power of the descriptors, it has been shown that LPQdescriptors can perform very successfully in blurred texture recognition, and also inblurred face recognition (Ahonen et al. 2008).

The work contained in Paper I is inspired by the success of the LPQ descriptors, andit describes an extension to color images of the LPQ descriptor called quaternion LPQ(qLPQ) that is robust to both blur and illumination changes in the scene (Figure 12).Obtaining robustness w.r.t. illumination changes is crucial: as it has been observedin (Mäenpää & Pietikäinen 2004), illumination changes may potentially decrease theperformance of color texture classification to the point that one may obtain betterperformance by discarding color information and just using the luminance channel.The qLPQ is shown to outperform ordinary LPQ and LBP descriptors in color texturerecognition, also in challenging scenarios where the textures images are captured underdifferent illuminations.

72

Fig 12. (Left) Illustration of some properties of the quaternion Fourier spectrum used inthe qLPQ descriptor (see Paper I for details). A color image f suffering a reddish colorcast due to wrong white balance settings. (Middle) The resulting image obtained by settingto zero the imaginary component of the QFT F f(0,0): the operation has the effect ofnormalizing f with respect to its average chrominance. (Right) Image reconstructed fromthe QFT spectrum of the middle image, where the two phase angles were quantized usingonly 2-bits per angle: although the quantization introduces some noise in the reconstruction,the details of the original image are clearly visible.

7.4 Blurred videos stabilization

An obvious application of blur-invariant image registration is that of blurred videostabilization. A typical scenario where one needs to register frames of a video degradedby out-of-focus blur occurs frequently in time-lapse microscopy (Wu et al. 2010). Intime-lapse microscopy the optical device has a very narrow depth of field, and it isusually anchored to a mechanical support. Due to the high degree of magnification,even small vibrations may generate considerable translational displacements betweensuccessive frames in the time-lapse video. Furthermore, because of the narrow depth offield, such vibrations can easily induce a large amount of out-of-focus blur in somecaptured frames (Figure 13). Blur-invariant image registration can be used to stabilizesuch time-lapse microscopy videos.

73

Fig 13. (Top) Two consecutive frames extracted from a time-lapse video acquired usingboth phase-contrast microscopy and fluorescence microscopy. The slight movement of themechanical support of the microscope caused blur and translation in one frame. (Bottom)The motion trajectory estimated by using N−fold phase-correlation between consecutiveframes.

74

8 Conclusion

This thesis presented several contributions in the theory of blur-invariants. First, analgebraic framework to constructively obtain blur-invariants is described. The blur-invariant operators that are derived from the framework are complete, and they can betrivially constructed from the knowledge of the type of symmetry of the point-spreadfunctions. Group theory, the branch of abstract algebra used to study symmetries, wasutilized in order to provide a general solution to the problem of finding blur-invariants,without resorting explicitly to the algebraic properties of any specific integral transform.The framework was shown to produce invariants w.r.t. PSFs having any arbitrary typeof symmetry, in contrast with most of the operators found in the literature, which aremostly invariant w.r.t. centrally-symmetric or Gaussian blur. Since the most commontypes of symmetries observable in point-spread functions of real optical systems arerotational and axial, this thesis and the original papers focused mainly on developingblur-invariant techniques w.r.t. to these kinds of symmetries.

In the second part of the thesis, we focused our attention on the problem of blur-invariant image registration. The problem of blur-invariant image registration is firstreformulated within the group theoretical framework, and afterward it is reinterpretedwithin a signal processing context, i.e. as a particular application of a Wiener deconvolu-tion filter. The reinterpretation of blur-invariant image registration in terms Wienerdeconvolution opened the way to the development of a more general type of registration,in which images that are degraded by both blur and high amounts of additive noise canbe registered efficiently.

In the last part of this thesis, all the techniques introduced in the previous chapterswere generalized to the case of color images. To achieve this goal, color images wererepresented as quaternionic signals, and consequently quaternion blur-invariants werederived, and the quaternion Wiener filter was used to perform blur and noise-robustregistration of color images.

The original papers included in this thesis contain experiments that demonstrate thatthe techniques described in this thesis can be successfully used for registration andrecognition of highly degraded images. In particular, in Papers II-IV, it is shown that thetranslational shift between images corrupted by a large amount of blur and/or noise canbe more robustly recovered by using the proposed techniques, rather than using the more

75

popular landmark-based registration. This is a crucial contribution whenever blurredimage registration is intended to be used as a pre-processing step for image fusionalgorithms like multichannel blind deconvolution, depth-of-field extension, etc. Sincethe use of blur-invariant operators has many useful applications in image recognition,we also proposed a color extension of the popular blur-robust descriptors based on localphase quantization (LPQ).

The work presented in this manuscript suggests future research directions. As it wasmentioned in Section 2.4, a theoretical limitation of our framework for constructingblur-invariants arose from the fact that the operation of blurring was modeled as theaction of a subgroup of Ω (i.e. the symmetric PSFs) acting on the whole group ofimages Ω through the group operation itself: convolution, in this case. In order toconstructively obtain invariants with respect to a group T of geometric transformations,one would need to extend the group theoretical framework described in Chapter 3to the case in which the degradations are modeled as an action T ×Ω→Ω. In sucha case, the group of transformations T (for instance, 2×2 orthogonal matrices withdeterminant 1) is not anymore a subgroup of (Ω,∗) and it does not act on Ω through thegroup operation of convolution. An extension of our framework that would not requirespecific constraints on the nature of the group action, is likely to enable the possibility ofobtaining constructively invariants for geometric transformation, as well as combinedinvariants for both geometric and radiometric degradations.

76

References

Ahonen T, Rahtu E, Ojansivu V & Heikkilä J (2008) Recognition of blurred faces using localphase quantization. Proc. 19th International Conference on Pattern Recognition, (ICPR2008)., IEEE, 1–4.

Banham MR & Katsaggelos AK (1997) Digital image restoration. IEEE Signal ProcessingMagazine 14(2): 24–41.

Boldyš J & Flusser J (2013) Invariants to symmetrical convolution with application to dihedralkernel symmetry. Proc. International Conference on Image Analysis and Processing (ICIAP2013), Springer, 369–378.

Bracewell R (1965) The Fourier Transform and Its Applications. McGraw-Hill, New York.Campisi P & Egiazarian K (2007) Blind image deconvolution: theory and applications. CRC

press.Chen B, Shu H, Zhang H, Chen G & Luo L (2010) Color image analysis by quaternion Zernike

moments. Proc. 20th International Conference on Pattern Recognition, ICPR 2010), IEEE,625–628.

De Castro E & Morandi C (1987) Registration of translated and rotated images using finite Fouriertransforms. IEEE Transactions on pattern analysis and machine intelligence 9(5): 700–703.

Debnath L & Bhatta D (2014) Integral transforms and their applications. CRC press.Ell TA (1992) Hypercomplex spectral transformations. Ph.D. thesis.Ell TA & Sangwine SJ (2007) Hypercomplex Fourier transforms of color images. IEEE

Transactions on Image Processing 16(1): 22–35.Evans CJ, Sangwine SJ & Ell TA (2000) Hypercomplex color-sensitive smoothing filters. Proc.

International Conference on Image Processing (ICIP 2000), IEEE, 1: 541–544.Flusser J & Suk T (1997) Classification of degraded signals by the method of invariants. Signal

processing 60(2): 243–249.Flusser J & Suk T (1998) Degraded image analysis: an invariant approach. IEEE Transactions on

Pattern Analysis and Machine Intelligence 20(6): 590–603.Flusser J, Suk T, Boldys J & Zitová B (2014) Projection operators and moment invariants to image

blurring. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(4): 786–802.Flusser J, Suk T & Saic S (1995) Image features invariant with respect to blur. Pattern Recognition

28(11): 1723–1732.Flusser J, Suk T & Saic S (1996a) Recognition of blurred images by the method of moments.

IEEE Transactions on Image Processing 5(3): 533–538.Flusser J, Suk T & Saic S (1996b) Recognition of images degraded by linear motion blur without

restoration. Springer.Flusser J, Suk T & Zitová B (2009) Moments and Moment Invariants in Pattern Recognition.

Wiley Online Library.Flusser J & Zitová B (2004) Invariants to convolution with circularly symmetric PSF. Proc. 17th

International Conference on Pattern Recognition, ICPR 2004, 11–14.Goodman JW et al. (1968) Introduction to Fourier optics, volume 2. McGraw-hill New York.Hamilton WR (1866) Elements of quaternions. Longmans, Green, & Company.Hayes MH, Lim JS & Oppenheim AV (1980) Signal reconstruction from phase or magnitude.

IEEE Transactions on Acoustics, Speech and Signal Processing 28(6): 672–680.

77

Helstrom CW (1967) Image restoration by the method of least squares. JOSA 57(3): 297–303.Hestenes D & Sobczyk G (1987) Clifford algebra to geometric calculus: a unified language for

mathematics and physics, volume 5. Springer Science & Business Media.Hitzer EM & Mawardi B (2008) Clifford Fourier transform on multivector fields and uncertainty

principles for dimensions n=2 (mod 4) and n=3 (mod 4). Advances in Applied CliffordAlgebras 18(3-4): 715–736.

Hu MK (1962) Visual pattern recognition by moment invariants. IRE Transactions on InformationTheory 8(2): 179–187.

Janesick JR (2001) Scientific charge-coupled devices, volume 117. SPIE press Bellingham.Ji H & Zhu H (2009) Degraded image analysis using Zernike moment invariants. Proc. IEEE

International Conference on Acoustics, Speech and Signal Processing, 2009. ICASSP 2009.,IEEE, 1941–1944.

Jost J (2009) Geometry and physics. Springer Science & Business Media.Karakasis EG, Papakostas GA, Koulouriotis DE & Tourassis VD (2014) A unified methodology

for computing accurate quaternion color moments and moment invariants. IEEE Transactionson Image Processing 23(2): 596–611.

Katkovnik V, Foi A, Egiazarian K & Astola J (2010) From local kernel to nonlocal multiple-modelimage denoising. International journal of computer vision 86(1): 1–32.

Kautsky J & Flusser J (2011) Blur invariants constructed from arbitrary moments. IEEETransactions on Image Processing 20(12): 3606–3611.

Kuglin C (1975) The phase correlation image alignment method. Proc. Proc. Int. Conf. Cyberneticsand Society, Sept. 1975, 163–165.

Lee HC (1990) Review of image-blur models in a photographic system using the principles ofoptics. Optical Engineering 29(5): 405–421.

Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journalof computer vision 60(2): 91–110.

Mäenpää T & Pietikäinen M (2004) Classification with color and texture: jointly or separately?Pattern recognition 37(8): 1629–1640.

Makaremi I & Ahmadi M (2012) Wavelet-domain blur invariants for image analysis. IEEETransactions on Image Processing 21(3): 996–1006.

Metari S & Deschênes F (2008) New classes of radiometric and combined radiometric-geometricinvariant descriptors. IEEE Transactions on Image Processing 17(6): 991–1006.

Milanfar P (2011) A tour of modern image filtering. IEEE Signal Processing Magazine 2.Moxey CE, Sangwine SJ & Ell TA (2003) Hypercomplex correlation techniques for vector images.

IEEE Transactions on Signal Processing 51(7): 1941–1953.Ojansivu V (2009) Blur invariant pattern recognition and registration in the Fourier domain,

volume 339. Acta Universitatis Ouluensis Technica, C 339.Ojansivu V & Heikkilä J (2007a) Image registration using blur-invariant phase correlation. IEEE

Signal Processing Letters 14(7): 449–452.Ojansivu V & Heikkilä J (2007b) Object recognition using frequency domain blur invariant

features. In: Image Analysis, 243–252. Springer.Ojansivu V & Heikkilä J (2008a) Blur insensitive texture classification using local phase

quantization. In: Image and signal processing, 236–243. Springer.Ojansivu V & Heikkilä J (2008b) A method for blur and affine invariant object recognition using

phase-only bispectrum. In: Image Analysis and Recognition, 527–536. Springer.Olver PJ (1999) Classical invariant theory, volume 44. Cambridge University Press.

78

Papoulis A & Pillai SU (2002) Probability, random variables, and stochastic processes. TataMcGraw-Hill Education.

Rahtu E (2007) A multiscale framework for affine invariant pattern recognition and registration.Acta Universitatis Ouluensis Technica, C 282.

Redondo R, Šroubek F, Fischer S & Cristóbal G (2009) Multifocus image fusion using thelog-gabor transform and a multisize windows technique. Information Fusion 10(2): 163–171.

Rose JS (1978) A course on group theory. Courier Corporation.Rudin W (1986) Real and complex analysis (3rd ed.). New York: McGraw-Hill Inc.Sangwine SJ (1996) Fourier transforms of colour images using quaternion or hypercomplex,

numbers. Electronics letters 32(21): 1979–1980.Sangwine SJ (1997) The discrete quaternion Fourier transform. Proc. 6th International Conference

on Image Processing and its Applications (ICIAP 1997), IET.Sangwine SJ (1998) Colour image edge detector based on quaternion convolution. Electronics

Letters 34(10): 969–971.Sangwine SJ & Ell TA (1999) Hypercomplex auto-and cross-correlation of color images. Proc.

International Conference on Image Processing (ICIP 1999), IEEE, 4: 319–322.Savazzi E (2010) Digital photography for science. Lulu.com.Shao Z, Shu H, Wu J, Chen B & Coatrieux JL (2014) Quaternion Bessel-Fourier moments and

their invariant descriptors for object reconstruction and recognition. Pattern Recognition47(2): 603–611.

Šorel M (2007) Multichannel blind restoration of images with space-variant degradations. Ph.D.thesis, PhD Thesis, Charles University, Prague.

Sroubek F & Flusser J (2005) Multichannel blind deconvolution of spatially misaligned images.IEEE Transactions on Image Processing 14(7): 874–883.

Sroubek F, Gabarda S, Redondo R, Fischer S & Cristobal G (2005) Multifocus fusion withoriented windows. Proc. Microtechnologies for the New Millennium 2005, InternationalSociety for Optics and Photonics, 264–273.

Wee CY & Paramesran R (2007) Derivation of blur-invariant features using orthogonal Legendremoments. IET Computer Vision 1(2): 66–77.

Wiener N (1949) Extrapolation, interpolation, and smoothing of stationary time series, withengineering applications.

Wu Q, Merchant F & Castleman K (2010) Microscope image processing. Academic press.Xiang-yang W, Wei-yi L, Hong-ying Y, Pan-pan N & Yong-wei L (2015) Invariant quaternion

radial harmonic Fourier moments for color image retrieval. Optics & Spectroscopy LaserTechnology 66: 78–88.

Zhang H, Shu H, Han GN, Coatrieux G, Luo L & Coatrieux JL (2010) Blurred image recognitionby Legendre moment invariants. IEEE Transactions on Image Processing 19(3): 596–611.

Zhu H, Liu M, Ji H & Li Y (2010) Combined invariants to blur and rotation using Zernike momentdescriptors. Pattern Analysis and Applications 13(3): 309–319.

79

80

Original publications

I Pedone M & Heikkilä J (2012) Local phase quantization descriptors for blur robust andillumination invariant recognition of color textures. Proc. 21st International Conference onPattern Recognition (ICPR 2012): 2476-2479.

II Pedone M, Flusser J & Heikkilä J (2013) Blur invariant translational image registration forN-fold symmetric blurs. IEEE Transactions on Image Processing 22(9): 3676-3689.

III Pedone M, Flusser J & Heikkilä J (2015) Registration of Images with N-fold Dihedral Blur.IEEE Transactions on Image Processing 24(3): 1036-1045.

IV Pedone M, Bayro-Corrochano E, Flusser J & Heikkilä J (2015) Quaternion Wiener Decon-volution for Noise Robust Color Image Registration. IEEE Signal Processing Letters, 22(9),1278-1282.

Paper I is reprinted with permission from ICPR 2012. Papers II, III, and IV are reprinted withpermission from IEEE.

Original publications are not included in the electronic version of the dissertation.

81

82

A C T A U N I V E R S I T A T I S O U L U E N S I S

Book orders:Granum: Virtual book storehttp://granum.uta.fi/granum/

S E R I E S C T E C H N I C A

521. Putaala, Jussi (2015) Reliability and prognostic monitoring methods of electronicsinterconnections in advanced SMD applications

522. Pirilä, Minna (2015) Adsorption and photocatalysis in water treatment : active,abundant and inexpensive materials and methods

523. Alves, Hirley (2015) On the performance analysis of full-duplex networks

524. Siirtola, Pekka (2015) Recognizing human activities based on wearable inertialmeasurements : methods and applications

525. Lu, Pen-Shun (2015) Decoding and lossy forwarding based multiple accessrelaying

526. Suopajärvi, Terhi (2015) Functionalized nanocelluloses in wastewater treatmentapplications

527. Pekuri, Aki (2015) The role of business models in construction businessmanagement

528. Mantere, Matti (2015) Network security monitoring and anomaly detection inindustrial control system networks

529. Piri, Esa (2015) Improving heterogeneous wireless networking with cross-layerinformation services

530. Leppänen, Kimmo (2015) Sample preparation method and synchronizedthermography to characterize uniformity of conductive thin films

531. Pouke, Matti (2015) Augmented virtuality : transforming real human activity intovirtual environments

532. Leinonen, Mikko (2015) Finite element method and equivalent circuit baseddesign of piezoelectric actuators and energy harvester dynamics

533. Leppäjärvi, Tiina (2015) Pervaporation of alcohol/water mixtures using ultra-thinzeolite membranes : membrane performance and modeling

534. Lin, Jhih-Fong (2015) Multi-dimensional carbonaceous composites for electrodeapplications

535. Goncalves, Jorge (2015) Situated crowdsourcing : feasibility, performance andbehaviours

536. Herrera Castro, Daniel (2015) From images to point clouds : practicalconsiderations for three-dimensional computer vision

C538etukansi.kesken.fm Page 2 Thursday, June 18, 2015 11:10 AM

UNIVERSITY OF OULU P .O. Box 8000 F I -90014 UNIVERSITY OF OULU FINLAND

A C T A U N I V E R S I T A T I S O U L U E N S I S

Professor Esa Hohtola

University Lecturer Santeri Palviainen

Postdoctoral research fellow Sanna Taskila

Professor Olli Vuolteenaho

University Lecturer Veli-Matti Ulvinen

Director Sinikka Eskelinen

Professor Jari Juga

University Lecturer Anu Soikkeli

Professor Olli Vuolteenaho

Publications Editor Kirsti Nurkkala

ISBN 978-952-62-0876-3 (Paperback)ISBN 978-952-62-0877-0 (PDF)ISSN 0355-3213 (Print)ISSN 1796-2226 (Online)

U N I V E R S I TAT I S O U L U E N S I SACTAC

TECHNICA

U N I V E R S I TAT I S O U L U E N S I SACTAC

TECHNICA

OULU 2015

C 538

Matteo Pedone

ALGEBRAIC METHODS FOR CONSTRUCTING BLUR-INVARIANT OPERATORS AND THEIR APPLICATIONS

UNIVERSITY OF OULU GRADUATE SCHOOL;UNIVERSITY OF OULU,FACULTY OF INFORMATION TECHNOLOGY AND ELECTRICAL ENGINEERING,DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING;INFOTECH OULU

C 538

ACTA

Matteo Pedone

C538etukansi.kesken.fm Page 1 Thursday, June 18, 2015 11:10 AM