Colour image filters based on hypercomplex convolution

S.J.Sangwine and T.A.EII

Abstract: There are very few examples of true vector filters known for colour images. The authors introduce a new class of filter based on convolution with hypercomplex masks, and present three colour edge detecting filters inspired by the Prewitt, Sobel and Kirsch filters. These filters, when applied to a colour image, produce an almost greyscale image with colour edges where the original image had a sharp change of colour. They rely on a three-space rotation about the grey line of RGB space, implemented with the rotation operator R[ ]R-’ where R is a quaternion and R-’ is its inverse. Pixels of similar colour corresponding to opposing positions in the filter masks cancel to give a grey or monochromatic result, while pixels of different colour are differenced in a vector sense to give coloured edge pixels. The linearity of the new filters is discussed and the paper concludes with a discussion of the impulse response of a hypercomplex filter.

1 Introduction

The history of colour image processing extends back to the 1970s at least, yet in over 30 years there have been very few developments of image filters which are not simply greyscale filters applied to the separate components of a colour image. A notable exception is the class of non-linear vector filters of which the vector median filter is the most well known [ 1, 21.

It is striking that until recently, no elegant generalisation of linearfilters to colour had been achieved. More impor- tantly, no simple mathematical framework had been suggested which might be applicable to linear filtering of colour images. In this paper we rectify this position by suggesting a very general approach (convolution with hypercomplex coefficients) and demonstrate its validity by presenting new filters based on rotations in colour space. In 1998, one of the authors published the first example of a colour image filter based on convolution with hypercomplex or quaternion masks, representing the colour image pixels using imaginary hypercomplex numbers [3]. One filter was presented in that paper, inspired by the classic Prewitt edge detector. In this paper we discuss the linearity of hypercomplex filters, and present coefficients for generalising two other types of classic greyscale edge detectors: the Sobel and the Kirsch filters. All three of these filters rely on three-space rotations (where the classic filters have real negation) and chromatic cancellation of the rotated pixel values. The filters in their classic sense may be applied to colour images, but since the coefficients are scalar, this is equiva-

0 IEE, 2000 IEE Proceedings online no. 2000021 1

DUI: 10.1049/ip-vis:20000211 Paper first received 6th April and in revised form 2nd December 1999 S.J. Sangwine is with the School of Computer Science, Cybemetics and Electronic Engineering, The University of Reading, Whitehights, Reading RG6 6AY, UK T.A. Ell is at 5620 Oak View Court, Savage, Minnesota, USA

IEE Pvoc.-Vis. linage Signal Process.. Vol. 147, No. 2, April 2000

lent to separate filtering of each colour component. Natural images often have luminance edges at the same image positions as chrominance edges, and the effect of the new filters and the classic filters is very similar in terms of the positions of the edges generated, although very different in terms of the edge values.

2 Properties of quaternions

The material in this Section could be said to be ‘classic’ mathematical knowledge, but because it is not likely to be well known or readily accessible to those working in image processing, we include it here to make the rest of the paper intelligible. In this paper we view the quaternions as simple hypercomplex numbers with one real part and three imaginary parts, or (equivalently) a vector imaginary part, and use notations consistent with the established notations for complex numbers as employed by electrical engineers. A minor exception is our use of an overbar to represent a conjugate.

The quaternions were discovered by Hamilton in Dublin in 1843 141 and were extensively studied in the ensuing 60 or more years. They led to the development of vector methods which proved of tremendous utility, especially in electromagnetics, but eventually became eclipsed by the vector ideas of scalar and crossproducts.

In the 20th century, quaternions have been employed by aerospace engineers, where their convenient representation of rotations has proved useful in such matters as stabilisa- tion and attitude control.

2. I Basic definitions A quaternion may be represented in hypercomplex form as

q = a + bi + cj + dk (1)

where a, b, c and d are real and i, j and k are operators obeying the following multiplication rules:

(2) i2 = j2 = k2 = qk .‘ = -1

89

and 3 Three-space rotations

i j = k j i = - k

From these rules it is clear that multiplication is not commutative. This is the only significant way in which the quaternions depart from the usual rules of algebra. Addition and multiplication are defined exactly analo- gously to complex addition and multiplication, but it must be remembered that the operators may not be re- ordered arbitrarily. Implementation of quaternion arith- metic in hardware or software is straightforward. Sadly, there are no programming languages which have built-in quaternion operations and thus implementation of hyper- complex filters requires a custom-written quaternion arith- metic package, e.g. quaternion packages exist for the Mathematica and Maple programs.

A quaternion has a real part, (a in eqn. 1) and an imaginary part. The latter has three components and is thus a vector quantity, often denoted by V(q)= bi+cj+dk. For this reason the real part is sometimes referred to as the scalar part of the quaternion S(q) and the whole quaternion may be represented by the sum of its scalar and vector parts as q = S(q) + V(q). (We are using a more modern notation here than the pioneers of quater- nions used. In 19th century texts, and even as late as 1946 [5], the reader will encounter the notations Vq and Sq.) A quaternion with a zero real or scalar part is called a pure quaternion.

The modulus and conjugate of a quaternion follow the definitions for complex numbers

141 = Ja2 + b2 + c2 + d2 (4)

q = a - bi - cj - dk = S(q) - V(q) (5)

A quaternion with unit modulus is called a unit quaternion. The inverse of a quaternion q-' such that q- 'q =qq-' = 1 is given by

2.2 Polar form Euler's formula for the complex exponential generalises to hypercomplex form

cos @ + p sin @ (7) =

where p is a unit pure quaternion. Any quaternion may be represented in polar form as

4 = 141epa' (8)

p and @ are referred to as the eigenaxis and eigenangle of the quaternion, respectively. p identifies the direction in three-space of the vector part and may be regarded as a true generalisation of the complex operator i, since p2 = - 1. @ is analogous to the argument of a complex number, but is unique only in the range [0, n] because a value greater than n can be reduced to this range by negating or reversing the eigenaxis. We can visualise the eigenaxis as the imaginary axis of an Argand diagram, the real axis of which is aligned with the scalar axis of the quaternion four- space. The eigenaxis is perpendicular toithe real axis, but need not be aligned with any of the three imaginary axes.

90

A three-space rotation about an axis p through an angle a is represented by the quaternion operator

(9)

where R=exp(pa/2) and the square brackets indicate a space for the quaternion quantity to be operated upon. Reversing the ordering of the left and right components of this operator (R and R ) reverses the sense of the rotation. This operator is employed in the filters presented in this paper. According to Coxeter [SI, who gives a proof of its action, it was first published by Cayley in 1845 [6].

4 Quaternion representation of colour image pixels

A colour image has pixels with three components. In this paper we consider only RGB images, which can be represented in quaternion form as pure quaternions f(x, y ) = r(x, y)i +g(x, y ) j + b(x, y)k. The choice of the imaginary part to represent the pixel values is fairly obvious, and is supported by the coincidence between the three-space imaginary part of the quaternion and the three- space of the RGB colour values. When we represent colour images in this way, a three-space rotation as given by eqn. 9 will always map a pixel value in three-space into another pixel value in three-space, but not necessarily within the bounds of the RGB colour space.

The representation of colour image pixels by pure quaternions has been conceived independently at least twice, by one of the authors of this paper [7], and by two researchers working on colour image compression algo- rithms [8, 91. The idea is obviously not limited to RGB images, and could equally well be applied to images in any of the luminance/chrominance formats such as YUV, YCC, CIELUV or CIELAB.

5 Chrominance-cancellation edge detectors

The filters presented in this paper rely on chrominance cancellation and are inspired by three classic greyscale edge detectors, attributed to Prewitt, Sobel and Kirsch, which are standard material in image processing texts [lo]. Only one rotation axis has been employed so far (the grey line in RGB space), but there are obviously other possibi- lities for cancellation of chrominance-like characteristics in colour images. Given a suitable offset colour space (with mid-grey at the origin), axes such as red-cyan or green- magenta or blue-yellow could be chosen, as well as red, green or blue. The particular filters presented in this paper could be implemented by first transforming the pixel values into a luminance/chrominance representation and then taking a chrominance difference. Convolution with hypercomplex coefficients achieves the same result in only one step and, because it works directly in the original RGB colour space, avoids the rounding errors that occur when converting from one representation to another. Further- more, by using convolution, we admit the possibility of combining two or more linear filtering operations into one convolution: In this paper we are concerned primarily with the existence and properties of hypercomplex filters rather than their utility. Further work is in progress to discover new variants of the filters presented here as well as other types of hypercomplex filter.

IEE Proc.-Vis. Image Signal Process., Vol. 147, No. 2, April 2000

5.1 Prewitt-type filter The first and simplest filter of the three presented in this paper has been published previously in a slightly different form [3]. The classic greyscale Prewitt filter is defined by the real-valued convolution mask (for a 3 x 3 neighbour- hood, and horizontal edges)

1 1

1 0 0

L-1 -1 - 1 1

A different mask is needed for vertical edges and the results of both can be combined to yield an edge strength image which may be thresholded to give a binary edge image. A similar process of combination may be employed with the colour generalisations described in this paper, but as will be seen below, a chromatic threshold is needed,

The new filters employ a rotation in colour space about an axis Y =g = b, the so-called ‘greyline’ of RGB space on which lie all achromatic pixel values. Representing this axis in unit quaternion form, as in Section 4, we have p=( i+j+k) / J3 . Any rotation about this axis maps a colour pixel value into another colour pixel value with the same luminance but different hue. Thus we define a rotation operator consisting of a pair of hypercomplex conjugate masks thus (for detection of horizontal edges)

which is fortunately trivial to compute. Fig. 1 Original colour image Zena’

where R=exp(pn/2) and S,= 116 is a scale factor to account for the addition of six pixel values. The rotation operator R[ ] R rotates a pixel value through z radians in a plane normal to the luminance axis or grey line.

A more efficient implementation can be realised by noting that the quaternion values employed in the rotation operator need not be unit quaternions if the following form of rotation operator is employed:

P[ ]P-‘ (12) where /PI # 1. This is easily seen to be a rotation equiva- lent to R[ 3R where R = P/lPl This possibility allows the coefficients of the left and right masks to be chosen so that some of the quaternion components are unity, and this could be exploited in an implementation to reduce the number of nontrivial multiplications needed.

We now explain in detail the application of the pair of masks given in eqn. 11. The classic Prewitt filter is implemented by convolving the mask with the image to be filtered. The definition of this operation is covered in any image processing textbook. For completeness we give here a mathematical definition using circular convolution (for eventual compatibility with filtering implemented in the Fourier transform domain)

Y Y

f ( ( n - x) mod N , (m - y ) mod M)hR(x, y ) (13)

where g(a m) is the filtered image andf(x, y ) is the original image, both of dimension N x M; hL(x, y ) and h,(x, y ) are the left and right masks, respectively, both of dimension

The effect of the filter defined by eqn. 1 1 on the colour image shown in Fig. 1 is shown in Fig. 2, which is a colour

(2x+ 1) x (2Y+ 1).

IEE Pmc.-Vis. Image Signal Process., Vol. 147, No. 2, April 2000

Fig. 2 Filtered colour image showing horizontal chromatic ‘edges’

image, despite superficially appearing to be a greyscale image. Note, for example, the top edge of the hat, where there is a red edge, and the lower edge of the white of the eye. Where the original colour image had gradually vary- ing areas of colour, the filtered image has been reduced to grey (achromatic) pixels. Where the original image had a colour edge (that is a sudden change of colour), the filtered image exhibits colour and the ‘edge’ pixels have a vector ‘strength’. Fig. 3 shows a test image consisting of 64 coloured squares (including some black and white squares). When filtered by the filter defined by eqn. 11, the result is as in Fig. 4.

The operation of the filter may be understood easily in terms of rotations in colour space. Where the mask overlies an image region of gradually varying colour, the pixel values along the top and bottom rows of the mask will be similar. Since one row is multiplied by unity (and therefore remains unchanged) and the other row is rotated through n about the grey line, the pixel values in the top row and the pixel values in the bottom row are positioned opposite each other at equal distances from the grey line after rotation. When added (using quaternion or vector addition) and scaled, the resultant pixel value lies on or near to the grey line, in a plane normal to the grey line positioned between the planes containing the original pixel values. In contrast, when the mask overlies a horizontal colour edge, the result after rotation, vector addition and scaling does

91

Fig. 3 Colour test image

Fig. 4 Filtered colour test image

not lie near to the grey line and exhibits colour. When the mask overlies a horizontal luminance edge where the pixels above and below the edge have similar or identical hues, the chrominance components of the pixels will almost cancel. The cancellation is less perfect where there is a large difference in luminance.

Fig. 5 colour purity threshold of 15

92

Thresholded result of the Sobel-typeJilter of eqn. 16 using a

A simple chromatic threshold can be applied to the edge-detected image to isolate the colour edges from the achromatic pixels. Saturation could be used, but a more suitable threshold is that given by the colour purity func- tion [l 11, which has the merit of being an integer quantity

5.2 Sobel-type filter The classic Sobel filter has the mask values

1 2

[ o 0 :] (15) -1 -2 -1

The following mask pair is a colour generalisation of this filter

R 2/2R R

where R is as for the Prewitt-type filter, and the scale factor

The result of applying the Sobel-type filter to the Lena image of Fig. 1 is very similar to the result of applying the Prewitt-type filter. When thresholded using a colour purity threshold of 15, the result is as shown in Fig. 5. This is similar to the result which can be obtained with a classic (scalar) Sobel filter applied to a colour image, because luminance and chrominance edges tend to be colocated in the image.

Ss = 1/8.

5.3 Kirsch-type filter The classic Kirsch filter has the mask values

[I' : '1 -3 -3 -3

and seven other similar masks, each sensitive to a hori- zontal or vertical edge or edges inclined at 45" to the horizontal [lo]. The colour generalisation is easily seen by analogy with the Sobel-type filter

s ~ [ f i R 0 1/51 [ 1 [ & E 0 z/J]

(18)

&R 1/5 z/J f i E 1/5 z/J

f i R &R f i R A R &R &R

where R is as for the Prewitt-type filter, and the scale factor Ss = 1/30.

6 Discussion

6. I Linearity In this Section we discuss the linearity of the new filters and show that they are truly vector filters. We explain that the filters cannot be implemented by scalar filtering (that is convolution with scalar or real coefficients) on the sepa- rated colour components, but that they may be implemen- ted by vector filtering on the separated colour components or indeed on any other decomposition of the image to be

IEE Proc.-Vis. Image Signal Process., Vol. 147, No. 2, April 2000

filtered. Thatis superposition applies, since, if p 1 = RqlR and p2 = Rq2R, then

pi +p2 = RqIZ + RqJ = R(ql + q2)R

An obvious decomposition of a colour image is that into separated colour components such as the red, green and blue components of an RGB image. In what follows, we must make clear that by a separated component of a colour image, we mean a colour image represented by an array of pure quatemion pixels in which all the pixel values lie on one of the axes of the colour space, not a greyscale (scalar) image in which the pixel component (red say) has been utilised as a greyscale intensity. Thus our decomposition of an RGB image may be represented as

f ( x , y> = r(x, Y) i + g(x , y ) j + b(x, y)k =A(% +&(XI +fh(x, Y> (19)

where

f , ( x , y ) = r(x, y)i + oj + Ok

f&X, y ) = oi + g(x, y>j + Ok (20)

fb (x , y ) = oi + oj + b(x, y)k

Clearly, using the results of the previous paragraph it is possible to implement the filters presented in this paper by separate application to the colour components. However, this separate application requires hypercomplex convolu- tion or some vector equivalent of the hypercomplex rota- tion operator. When a filter based on rotation operators R[ ]R or P[ ]P-I is applied to a component of a colour image, such as a separated red component image as given in eqn. 20, the result is not a purely red image because the rotation (in the case of filters presented in this paper) is about a line which is not in any plane containing the pixel values. The pixel values of a red component image lie on the red axis of RGB space, but after rotation through 71

radians they will lie on a line in the cyan direction of colour space. Thus, after filtering, the image f,(.x, y ) will have nonzero green and blue components.

We conclude that the new types of filter based on hypercomplex convolution with mask pairs containing coefficients of the type R[ ]z are linear vectorjlters.

6.2 Impulse response Linear filters in the conventional (scalar) sense are char- acterised by their impulse response or by their frequency response, the two being related by the Fourier transform. Fourier transforms of colour images have been defined by the authors in several publications [ 12, 131 but the imple- mentation of a filter with a pair of hypercomplex masks has not yet been achieved in the Fourier transform domain.

We may extend the argument given in Section 6.1 to any decomposition into hypercomplex component images, including a decomposition into impulse images (that is images with only one nonzero pixel value), a large number of which would need to be added to compose the original

image. A hypercomplex filter cannot be characterised by a single impulse response. A simple argument to demon- strate this is that the result of convolving a pair of left and right quatemion masks with a single impulse image cannot have enough information to characterise the filter, since the result of the convolution has only one quaternion value per position of the m2sk which is insufficient to distinguish, for example R[ ]R from R[ ]R, which are two very differ- ent filters. A careful study of this problem has revealed that three mutually perpendicular impulses are required to determine the mask coefficients.

7 Conclusions

The new linear vector filters presented in this paper can be applied to colour or other multispectral images and are the first published examples of filters based on convolution with quaternion or hypercomplex mask coefficients. They have thus established a new subject area in colour image processing. Much further work remains to be done as a result of the discovery of these filters, especially on the theoretical understanding of such linear vector filters and their frequency responses.

8 References

1 ASTOLA, J., HAAVISTO, F!, and NEUVO, Y.: ‘Vector median filters’, Proc. IEEE, 1990, 78, (4), pp. 678-689

2 PLATANIOTIS, K.N., and VENETSANOPOULOS, A.N.: ‘Vector filterine’. In SANGWINE. S.J.. and HORNE. R.E.N. (Eds.): ‘The colour-image processing handbook’, (Chapman and Hall, London, 1998), chapter 10, pp. 188-209 SANGWINE, S.J.: ‘Colour image edge detector based on quatemion convolution’, Electron. Lett., 1998, 34, (lo), pp. 969-971 HAMILTON, W.R.: ‘Elements of quaternions’ (Longmans, Green and

3

4

~ - - > ~.. ~

colour image p 1998), chapter 10, pp. 188-209 SANGWINE, S.J.: ‘Colour image edge detector based on quatemion convolution’, Electron. Lett., 1998, 34, (lo), pp. 969-971 HAMILTON, W.R.: ‘Elements of quaternions’ (Longmans, Green and

3

4 Co., London, 1866) COXETER, H.S.M.: ‘Quatemions and reflections’, Am. Math. Mon., 5 1946. 53. DD. 136-146 ~- ~i ~~

6

7

CAYLEY,,’A.: ‘On certain results relating to quaternions’, Philos. Mug., 1845, 26, (3), pp. 142 SANGWINE, S.J.: ‘Fourier transforms of colour images using quater- nion, or hypercomplex, numbers’, Elecrron. Lett., 1996, 32, (21), pp. 1979-1 9x0 -_ - -

8 PEI, S.C., and CHENG, C.M.: ‘A novel block truncation coding of color images by using quatemion-moment preserving principle’. Proceedings of ISCAS 96: 1996 IEEE Intemational Symposium on Circuits and Systems, 12-15 May, Atlanta, GA, USA, 2, pp. 681-687 PEI, S.C., and CHENG, C.M.: ‘A novel block truncation coding of color images using a quatemion-moment-preserving principle’, IEEE Trans. Commun.. 1997. 45. (5). DD. 583-595

9

10 CASTLEMAN, ’K.R.; ‘D&ital image processing’ (Prentice-Hall, Engle-

11 PRITCHARD, A.J., HORNE, R.E.N., and SANGWINE, S.J.: ‘Achiev- wood Cliffs, NJ, 1996)

ing brightness-insensitive measurements of colour saturation for use in colour object recognition’. Proceedings 5th Int. Conf. on Image Proces- sing and its Applications, Edinburgh, UK, (IEE) 3-6 July 1995, pp. 791- 795

12 SANGWINE, S.J., and ELL, T.A.: ‘The discrete Fourier transform of a colour image’, in BLACKLEDGE, J.M., and TURNER, M.J. (Eds.): ‘Image processing 11: Mathematical methods, algorithms and applications’, ‘Proceedings of Second IMA Conference on Image Processing, De Montfort University, Leicester, UK, 22-25 September 1998 (Honvood Publishing for Institute of Mathematics and its Applica- tions, 2000) pp. 430441

13 SANGWINE, S.J., and ELL, T.A.: ‘Hypercomplex auto- and cross- correlation of color images’. Proceedings of IEEE Int. Conf. on Image Processing (ICIP99), 24-28 October 1999, Kobe, Japan, pp. 319-322 (missing one page), Full 5pp on CD-ROM proceedings, ISBN 0-7803- 5470-2

IEE Proc.-Vis. Image Signal Process., Vol. 147, No. 2, April 2000 93