thesis, image registration methods

Image Registration Methods for Reconstructing a Gene Expression

Atlas of Early Zebrafish Embryogenesis

Evangelia BalanouMASTER THESIS

EUROPEAN POSTGRADUATE PROGRAM ON BIOMEDICAL ENGINEERINGUNIVERSITY OF PATRAS – NATIONAL TECHNICAL UNIVERSITY OF ATHENS

DEPARTMENT OF ELECTRONIC ENGINEERING TECHNICAL SCHOOL OF TELECOMMUNICATIONS ENGINEERING

TECHNICAL UNIVERSITY OF MADRID

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling

Design & Implementation

Concept

Overview

Registration Pipeline

Atlas Construction Pipeline

Tools

Implementation

Results & Evaluation

Comparison of Registration Methods

Atlas

Conclusions & Future Work

• Introduction– Motivation– Problem– Goal

• Image Registration– Components

• Design and Implementation– Concept– Overview– Registration Pipeline– Atlas Construction Pipeline– Tools– Implementation

• Results and Evaluation– Comparison of Registration Methods– Atlas Construction

• Conclusions and Future Work

Outline

IntroductionMotivation

Problem to be solved

Motivation

Early development of a zebrafish embryo

• Study the genes that regulate embryonic development (developmental biology)

• Study embryonic development of vertebrates:– Vertebrate developmental disorders– Human hereditary disease

• Vertebrate model: zebrafish– Rapidly developing transparent embryos– Small size (4-5 cm length)– Short generation time

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Quantitative spatio-temporal data at cellular level about gene expression required

Provided by Fluorescence In Situ Hybridization techniques and Laser Scanning Microscopy

Problem

z

x

y

Second gene expression pattern

One gene expression pattern

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Qualitative spatio-temporal data at cellular level about gene expression required

Provided by Fluorescence In Situ Hybridization techniques and Laser Scanning Microscopy

Problem

• However, not more than five gene expression patterns simultaneously revealed on the same embryo!

Image processing methods to integrate different expression patterns (from different embryos) into a 3-D gene expression atlas

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


GoalDesign and implement an image processing framework able to register different datasets with different gene expression patterns to a common template at a given developmental stage

Template One datasetTemplate + registered image

“Registration is the process of determining a geometrical transformation that aligns points in one view of an object with corresponding points in another view of that object or another object.”

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Image RegistrationFundamental task in image processingVarious techniques (data, application)

Image Registration• Intensity-based :

Calculates the transformation using voxel values alone

• Input: 2 images – fixed, moving Output: geometrical transformation

• Optimization problem

• Decomposed into a set of basic elements (defining different methods)

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage

TransformationParameters

Registration

Similarity measure

Interpolation

Initial Parameters

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Transformation

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage


Registration

Similarity measure

Interpolation

Initial Parameters

• Defines the type of parameters whose values align the two images (search space)

• Spatial mapping of points from the fixed image space to points in the moving image space (inverse mapping)

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Interpolation

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage


Similarity measure

Interpolation

Initial Parameters

• Evaluate moving image intensities at the mapped, non-grid positions

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Similarity Measure

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage


Similarity measure

Interpolation

Initial Parameters

• A measure of “how well” fixed and transformed moving match each other• Provides a quantitative criterion to be optimized over the search space

(similarity measure function, S(T) )• The desired optimum may be one of the local ones

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Correlation Coefficient

Similarity Measures

• Mutual Information

Intensities in two images linearly related As written, function to be maximized

Intensities in two images statistically related As written, function to be maximized

Fixe

d Im

age

inte

nsity

(I2)

Moving Image intensity (I1)

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Fixe

d Im

age

inte

nsity

(I2)

Moving Image intensity (I1)

Optimization

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage


Similarity measure

Interpolation

Initial Parameters

• Most complex component• Starting from an initial set of parameters, iteratively searches the optimal

solution of the similarity measure function over the parameter space defined by the transformation

• Stops when stopping criterion is met

Transformation Parameters

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Optimization Algorithms• Gradient Descent

• Differential Evolution

Derivative of similarity measure function (S) wrt to each transformation parameterAttracted by local extrema

Stochastic, population-basedGlobal optimization technique – slow in computation

n

nn pSpp

1

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Initialization Mutation Recombination Selection

Cost FunctionOutline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Local optimization

Start

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Local optimization

End

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Global optimization

Start

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Global optimization

End

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas

Conclusions & Future Work Capture range of correct optimum (initial parameter range or initialization)

Resampling

Optimization

Transformation

Resampling

Fixedimage

Movingimage

Registeredimage


Similarity measure

Interpolation

Initial Parameters

• Once a stopping criterion is met or iteration number has reached, the last transformation parameters are used to produce the registered image

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Design & ImplementationImplemented framework’s concept

Different steps it is composed of

Concept

Template embryoPartial views

Nuclei channel

Reference gene channel(goosecoid)

Another gene channel

*All images are 3D and grayscale *Colourmap just for visualization

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Goal: Design and implement an image processing framework able to register different datasets with different gene expression patterns to a common template at a given developmental stage

ConceptTemplate embryoPartial views

Registration

Gene expression atlas

Nuclei channel

Reference gene channel

Nuclei channel

Reference gene channel

Another gene channel

Partial view of another embryo

*All images are 3D and grayscale *Reference gene (position): goosecoid (gsc)*Colourmap just for visualization

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Overview

Moving image

Whole embryo view, nuclei channel

Whole embryo view, gsc channel

Partial embryo view, nuclei channel

registration


Registered image

Partial embryo view, gsc channel

Fixed image

addition

addition

initialization

Fixed image

Initialized Moving image

preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre

Partial embryo view, third channel

preprocessing transformation Third channel mapped

Registration pipeline

Atlas construction pipeline

initialization

Gravity centres

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas



Moving image




registration


Registered image


Fixed image

addition

addition

initialization

Fixed image


preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre





initialization

Gravity centres

Purpose: Determine the transformation parameters that bring into spatial alignment the template and one partial view

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Preprocessing & Addition Step

Moving image




registration


Registered image


Fixed image

addition

addition

initialization

Fixed image


preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre


Purpose: Remove noise, blur, downsample, thresholdCombine information from nuclei and gsc channels into a single image

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Preprocessing & Addition Step

• Preprocessing depends on images (noise, size)• Weighted Addition

additionpreprocessing

preprocessingOriginal gsc channel

Original nuclei channel Combined

image

2550

Original channels Preprocessed channels Combined image

addition

preprocessing

preprocessing

Resolution: 512 x 512 x 465Voxel size: 1.517 x 1.517 x 1,509μm

Resolution: 128 x 128 x 116Voxel size: 6.068 x 6.068 x 6.036μm

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Initialization Step

Moving image




registration


Registered image


Fixed image

addition

addition

initialization

Fixed image


preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre


Purpose: Initial positioning of moving to fixed image’s space (no initial parameters in registration)

If NOT sufficient overlapping, registration fails

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Based on nature of data (nuclei and gsc channel)

• For both views one gravity centre from each channel

• The resulting four points define a spatial transformation that is applied on the moving image

Initialization StepPreprocessed partial embryo view, nuclei channel (binary)

Preprocessed partial embryo view, gsc channel


Rotation centre

Moving image

Fixed image

initialization

Preprocessed whole embryo view, nuclei channel (binary)

Preprocessed whole embryo view, gsc channel

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Initialization Step

x

y

z

gscfixed

nfixed

nmoving

gscmoving

Translated nmoving

Rotation axis

Rotation angle

translation

vF

vM

nfixed

gscfixed

nmoving

gscmoving

Fixed (template view)

Moving (partial view)

*Blue/Orange-nuclei Green/Yellow-gsc expression pattern

Translated nmoving

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Initialization Step

Before initialization

After Initialization

Fixed (template) + Initialized Moving (partial) Partial view before and after initialization

Initialized Moving

Fixed Image


Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Registration Step

Moving image




registration


Registered image


Fixed image

addition

addition

initialization

Fixed image


preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre


Purpose: Find the transformation parameters that register the initialized moving image to the fixed

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Registration StepOutline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Global, Rigid 3D Transformation

Resampling

Fixed image

Initialized Movingimage

Registeredimage


Implemented Registration step

Trilinear Interpolation

Initial Parameters (rotation centre)

Mutual Information

Correlation Coefficient

Gradient Descent

Differential Evolution

or

or

• Global, rigid transformation-> Assumption: embryos similar in size and shape

-> 3 rotations + 3 translation = 6 transformation parameters

• 2 similarity measures and 2 optimization algorithms

Registration Step

Initialized Moving

Fixed Image

Fixed (template) + Initialized Moving (partial)

Fixed Image

Registered Image

Fixed (template) + Registered Image


Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Atlas Construction PipelineMoving image




registration


Registered image


Fixed image

addition

addition

initialization

Fixed image


preprocessing

preprocessing

preprocessing

preprocessing

Rotation centre





initialization

Gravity centres

Purpose: Transformation of the third channel of the partial viewOnly transformation step is implemented as a new program

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas



Registration Pipeline -> Transformation Parameters

Partial view White-nucleiRed-gsc expression patternGreen-snail expression pattern

TemplateOrange-nucleiYellow-gsc expression pattern

Atlas Construction Pipeline -> Apply Transformation Parameters

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Development• Insight Segmentation and Registration Toolkit

– Available at www.itk.org• CMake

– Available at www.cmake.org • Microsoft Visual Studio 2008

Tools

Visualization• Amide– Available at http://amide.sourceforge.net/

• Amira– Commercial product

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


http://www.itk.org/

http://www.cmake.org/

http://amide.sourceforge.net/

Implementation

• User’s manual provided• Run from command line configuring parameters

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Developmental stage: Shield (6 hpf)

• Framework tested with six datasets (six embryos)

– One template, one whole embryo view

– Partial views of five different embryos

Data

Dorsal

Animal

Ventral

Vegetal

* Images provided by: DEPSN , France

nuclei channel

gsc channel

co-stained gene expression pattern e.g. snail

Template embryoPartial view

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Questions

1. Does the implemented framework succeed in registering our data?

2. What is the combination of similarity measure and optimization algorithm that results in a successful registration?

In other words…

What is the most appropriate registration method for our application?

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Preprocessing & Addition

• Framework works with 2 datasets each time• Preprocessing: smoothed, downsampled, nuclei channel turned to binary• Addition: nuclei and gsc channels combined into a single image• 5 partial -> 5 iterations (6 images in total – 1 fixed, 5 moving)

addition

preprocessing

preprocessing

preprocessing

preprocessing

addition

Original channels Preprocessed channels Combined imageTe

mpl

ate

One

par

tial V

iew

Fixed image

One moving image

Slice Volume rendering

Slice Volume rendering

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Initialization

Initialization looks promising…

Template + Initialized partial 1 Template + Initialized partial 2 Template + Initialized partial 3

Template + Initialized partial 4 Template + Initialized partial 5


Initialized Partial 1

Template ImageInitialized Partial 2

Template Image Initialized Partial 3

Template Image

Initialized Partial4

Template Image

Initialized Partial 5

Template Image

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Method Evaluation• Four different methods implemented

• Evaluation only by visual inspection of the results

– Optimization algorithms not comparable unless running with optimized parameters

– Lack of golden standard

– Point-to-point correspondence does not exist (different embryos)

Similarity measures Optimization algorithmsCorrelation Coefficient Gradient Descent

Mutual Information Differential Evolution

Outline

IntroductionMotivation

Problem

Goal

ImageRegistrationTransformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

OverviewRegistration Pipeline


Tools

Implementation


Comparison of Registration MethodsAtlas


Method EvaluationOptimization algorithm

Similarity measure Gradient descent Differential evolution

Correlation Coefficient

Mutual Information

*After 100 iterations

• Monomodal case, intensities are linearly related (C.C. ideal)• Global optimization algorithm is still computing (D.E. not suitable)• Initialization sufficient (Gradient Descent is “myopic”)

Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Transformation parameters taken from the registration method with the most coherent performance and successful results

– Correlation Coefficient optimized by the Gradient Descent algorithm

• Initialization parameters (initialization) and transformation parameters (registration) applied on the third channel of three datasets

Atlas Construction

Original view

After mappingOriginal view

After mapping

Original view

After mapping

gsc - spt gsc - snail gsc - chd

chd’s expressionsnail's expression

spt’s expression gsc’s expression

Partial view 3Volume rendering of gsc , third channel, registered gsc and transformed third channel



Outline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Conclusions & Future work

• Goal achievedDesigned and implemented an image processing framework able to map different gene expression patterns on a common template (for a given developmental stage)

• Key points– Addition: Combine information from two channels– Initialization: Solves the problem of capture range for

optimization– Registration Method: Correlation Coefficient + Gradient

Descent

Summary - ConclusionsOutline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


• Advantages

– Modularity

– Configurability

– Semi-automated

• Future work

– More datasets -> more gene expression patterns

– Other developmental stages

– Validated with known gene regulatory networks

Conclusions-Future workOutline

Introduction

Motivation

Problem

Goal

ImageRegistration

Transformation

Interpolation

Similarity Measure

Optimization

Resampling


Concept

Overview



Tools

Implementation



Atlas


Thanks to…

Biomedical Image Technologies Laboratory (BIT)Technical School Of Telecommunications Engineering (ETSIT)

Technical University of Madrid (UPM)

Thank you for your attention

thesis, image registration methods

Technology