Evolutionary algorithm for dense pixel matching in presence of distortions!A.C. dos-Santos-Paulino, J.-C. Nebel, F. Flórez-Revuelta!

Find the correspondence to all the pixels in one image in a different image The second image is a “distortion” of the first image

Dense pixel matching

Analogy between alignment of proteins and matching pixels between scanlines Protein sequences can mutate through substitution, insertion and deletion of characters The global alignment of two protein sequences is made following the “Needleman-Wunsch” algorithm, wh ich is based on dynamic programming

A line in the second image can be seen as a mutated version of the first image The approach was robust to camera rotation and translation, but also to local and global distortions As each scaline is processed independent l y there was no consistency between neighbouring scanlines

Bio-inspired algorithm Thevenon,  J.  et  al.:  Dense  pixel  matching  between  unrec;ed  and  distorted  images  using  dynamic  programming,  Intl.  Conf.  on  Computer  Vision  Theory  and  Applica;ons,  pp.  216-­‐224,  2012

Approach similar to Cellular GA However, fitness function varies between individuals As neighbouring scanlines have similar distortions, each individual wil l optimise a similar fitness function

Recombination takes place between close neighbours in order to propagate good solutions The final result for the whole population represents the distortion between the images

Evolutionary proposal

Initialise  the  population  with  a  number  of  individuals  equal  to  the  number  of  lines  in  the  image    repeat    

 Select  an  individual  I  at  random    

 Select  a  couple  of  parents  I1  and  I2  in  the  neighbourhood  of  I    

 Recombine  I1  and  I2  generating  Inew    

 Mutate  Inew    

 if  fitness(Inew)<fitness(I)  then      Substitute  I  by  Inew  

 until  an  ending  condition  is  fulfilled  

Algorithm

Each individual represents the path that a scanline in the image follows in the distorted image Three different kind of symbols:

•  Match •  Stay •  Gap

Chromosomes can have different lengths

Individuals

Fitness function

Each individual is initialised to a trajectory beginning from the left of the distorted image Two types of initialisation:

•  at random, •  locally optimised: which direction from ME, MNE, MSE adds a lower

penalty?

The path is complete when: 1.  The path reaches one of the borders (right, top or bottom) 2.  It provides correspondence for all the pixels in the scanline

Repairing process: stay or gap symbols are included at random positions

Initialisation

A new individual I is evolved from a pair of individuals in its neighbourhood Parents could be selected in different ways. For instance:

•  according to their proximity to I:

•  according to their fitness:

Individual I can be selected as one of the parents

I

Selection of parents

Parents usually have different lengths Usual single- or two-point crossover is not possible as parent could not be aligned Crossover point(s) are columns in the image Offspring is generated by swapping the paths from each parent that reach/depart from that/those column(s) Offspring does not need to be repaired

Crossover

Different mutations: •  gene alteration by selecting at random an alternative direction, •  gene deletion, •  gene replication over a contiguous interval of the path, •  simultaneous deletion of a pair gap/stay in the scanline, as they

correspond to inverse operations; and •  local optimisation by substituting an interval of the path by a locally

optimised path (as in the initialisation).

Repairing process could be needed: •  Adding gaps or stays as needed (if too short), or •  cropping the path (if too long)

Mutation

The size of the population is equal to the number of lines in the images Penalty g=181 Stopping process = 10000 generations without changes in any individual All the mutation types are equally considered

Experimentation

Original image Distorted image

Distortion Final result Reconstructed image

Image with global distortion

Original image Ground truth Result

Detail Reconstructed image

Image with local distortions

100% 50% 0%

Effect of the initialisation

Higher neighbourhood is better if the distortion is global to the image, as the good solutions propagate globally.

Neighbourhood should be selected according to the size of the distortions. In any case, directed initialisation of individuals is better.

Effect of the neighbourhood

Effect of the selection method

No significant diferences Evolution is more affected by the size of the neighbourhood

Stereo matching: If both views are aligned only horizontal directions are needed

Other problems we are testing

Discusion

Medical images: In ultrasound videos there are motions between consecutive images If the motion is considered as a distortion, the same process can be followed and the video can be stabilised However, in this problem the scanlines are not straight Can we use the same approach to track objects or persons?

Other problems we are testing

Discusion

Medical images: In ultrasound videos there are motions between consecutive images If the motion is considered as a distortion, the same process can be followed and the video can be stabilised However, in this problem the scanlines are not straight Can we use the same approach to track objects or persons?

Other problems we are testing

Discusion

It is not a Cellular GA It is not a Multi-Objective EA It is not a Co-EA Any idea? Application to other problems where the population could replicate the topology of the input space and where fitness are not the same but similar Not restricted to linear topology

Any idea about the structure of the population?

Discusion

Results are good. However, sometimes the evolution gets stack in local minima Good solutions are propagated throughout the population New mutations, topologies and fitness functions need to be explored Can we have different parallel evolutions where individuals represent paths in different directions? Can we recombine them? Promising for other problems where a population can replicate the structure of the problem space Instead of working with 2D spaces (images), apply the method to a 3D space (video) Application to different granularities:

ê sub-pixel é matching of lattices of points

Conclusions

Evolutionary algorithm for dense pixel matching in presence of distortions!A.C. dos-Santos-Paulino, J.-C. Nebel, F. Flórez-Revuelta!!F.Florez@kingston.ac.uk