segmentation of medical images –techniques & application...

João Manuel R. S. Tavares

[email protected] www.fe.up.pt/~tavares

October 31, 2018

Segmentation of Medical Images – Techniques & Application Cases

Outline• Introduction• Image Segmentation

– Introduction– Accuracy Assessment– Techniques & Applications

• Conclusions• Research Team• Publications & Events

@2018 João Manuel R.S. Tavares Segmentation of Medical Images - Techniques & Application Cases 2

Introduction

Presentation• Associate Professor at FEUP (DEMec)• Senior Research and Projects Coordinator of the

Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI)

• Habilitation in Mechanical Engineering from the University of Porto

• PhD and MSc degrees in Electrical and Computer Engineering from FEUP

• BSc degree in Mechanical Engineering from FEUP• Research Areas: Image Processing and Analysis,

Medical Imaging, Biomechanics, Human Posture and Control, Product Development

@2018 João Manuel R.S. Tavares 4Segmentation of Medical Images - Techniques & Application Cases

@2018 João Manuel R.S. Tavares 5

> 6,000 undergraduate and master students> 750 doctoral students and 360 researchers> 550 faculty (396 full-time equivalent) and 350 technical staff

> 500 international students

Presentation

(in 2016)

Segmentation of Medical Images - Techniques & Application Cases

Presentation


FEUP: 50,000 people working in science, engineering andtechnology in a square kilometer


Presentation


PORTO: 1 million inhabitants, one of the biggest 33 European Cities


Introduction• The researchers of Image Processing and

Analysis aim the development of algorithms to perform fully or semi-automatically tasks performed by the (quite complex) human vision system

Original images Computational 3D voxelized and poligonized models built

Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369@2018 João Manuel R.S. Tavares 8Segmentation of Medical Images - Techniques & Application Cases

Introduction• Image processing and analysis are topics of the

most importance for our Society• Algorithms of image processing and analysis are

frequently used, for example, in:– Natural Sciences– Sports– Biology– Industry– Engineering– Medicine


Introduction• Examples of common tasks involving algorithms

of image processing and analysis:– Noise removal– Geometric correction– Segmentation, recognition (2D-4D)– Motion and deformation tracking and analysis,

including matching, registration and morphing– 3D reconstruction– Assisted medical diagnosis and intervention


• Image Acquisition: a sensor captures the energy reflected or emitted by the imaged object

Introduction

http://what-when-how.com/introduction-to-video-and-image-processing/image-acquisition-introduction-to-video-and-image-processing-part-1@2018 João Manuel R.S. Tavares 11Segmentation of Medical Images - Techniques & Application Cases

• Image: a matrix with n rows and m columns (and lslices in 3D), being each basic element known as pixel (or voxel in 3D)

Introduction


• Image Processing: by applying mathematical operations/rules using the values of the image pixels (or voxels in 3D) in the Cartesian or in other domain

Introduction

∗−1 −2 −10 0 0+1 +2 +1

⎡

⎣

⎢⎢⎢

⎤

⎦

⎥⎥⎥= Gy

∗−1 0 +1−2 0 2−1 0 +1

⎡

⎣

⎢⎢⎢

⎤

⎦

⎥⎥⎥= Gx

Gx2 +Gy

2 = G

Sobel operator

∗( denotes convolution)


• Difficulties: noise, artifacts, occlusion, poor

illumination, reflections, complex objects and

backgrounds

Introduction

https://rahaddadi.files.wordpress.com/2011/05/face_black_and_white_optical_illusion_cool-s453x562-92306-5803.jpghttp://s1.cdn.autoevolution.com/images/news/the-longest-traffic-jam-in-history-12-days-62-mile-long-47237_1.jpg


Introduction

Image(s) enhancement /

correction

Image(s) segmentation / features extraction

tracking

matching

classification

Image(s)

registration

image (pre)processing

image analysis /computational

vision

3D vision

computer vision

Usual Pipeline


Introduction• (Pre)processing of noisy images using an

intelligent worm

@2017 João Manuel R.S. Tavares Image Processing and Analysis in Biomechanics 16

Original images (a), noisy corrupted images (b) and smoothed images using different smoothing methods (c-h)

Araujo et al. (2014) Expert Systems with Applications 41(13):5892-5906

Image Segmentation

Segmentation

• It is intended to identify in a fully or semi-

automatically manner structures presented in static images or in image sequences (2D/3D/4D)

• It is one of the most usual operations involved in

the computational analysis of structures in images

• The most usual methodologies are based on thresholding, region growing, template matching, statistical, geometric or physical modeling, or artificial classifiers

• Frequent problems: noise, artifacts, low resolution,

reduced contrast, shapes not previously known,

occlusion, multiple structures, etc.


Accuracy Assessment • To analyze the accuracy of the computational

techniques, the computational segmentations are compared against:– Segmentations made manually by experts – health

professionals– Segmentations obtained by other computational

techniques – state of the art• Datasets for benchmarking are become more and

more common, e.g. https://grand-challenge.org

• Several accuracy measures have been proposed@2018 João Manuel R.S. Tavares 19Segmentation of Medical Images - Techniques & Application Cases

https://grand-challenge.org/

Accuracy Assessment • Examples of accuracy measures

– Region based• Jaccard measure (JM) - ratio between the size of

the intersection and the size of the union of the two regions (Sauto, Smanual)

• Dice coefficient (DC) - overlap between the two regions (Sauto, Smanual)

JM, DC ∈ [0, 1]: 0 - no overlap; 1 - perfect overlap@2018 João Manuel R.S. Tavares 20Segmentation of Medical Images - Techniques & Application Cases

Accuracy Assessment • Examples of accuracy measures

– Contour based• Polyline distance (Ds) - average minimum distance

between the points of the two contours (B1, B2)

• Hausdorff distance (HD) - maximum distance between the greatest distances between the points of the two contours (Ca, Cm)

N(B1), N(B2) - number of points in each contourA, b - points in each contourd(a, b) - Euclidean distance between points a and b


Techniques & Application Cases• Image segmentation by thresholding (binarization)

@2018 João Manuel R.S. Tavares 22Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246


• Example: segmentation of contours in dynamic pedobarography (Otsu’s method, morphologic dilation, xor operation)


Original images(pedobarographic image sequence)

Segmented images

Bastos & Tavares (2004) Lecture Notes in Computer Science 317e9:39-50

camera

mirror

contact layer + glass

reflected light glass

pressure opaque layer

lamp

lamp transparent layer

Optical pedobarographydevice


Techniques & Application Cases

• Image segmentation by region growing


Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246Segmentation of Medical Images - Techniques & Application Cases


• Example: segmentation of ear structures (region

growing)


Region Growing, x=215; y=254

Segmentation obtained(bony labyrinth)

Barroso et al. (2011) CNME 2011Ferreira et al. (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(8):888-904

X: 254 Y: 214Index: 116.7RGB: 0.459, 0.459, 0.459

Original Image



• Segmentation of objects based on deformable templates

Carvalho & Tavares (2006) CompIMAGE 2006, 129-134Carvalho & Tavares (2007) VipIMAGE 2007, 209-215

Example of a deformable template

(for the eye)



• Example: segmentation of eyefeatures (deformable geometric template)

Original image and associated force (or energy) fields

Segmentation of the iris using adeformable template (a circle)

Segmentation of an eye using an

deformable templateCarvalho & Tavares (2006) CompIMAGE 2006, 129-134Carvalho & Tavares (2007) VipIMAGE 2007, 209-215@2018 João Manuel R.S. Tavares 27Segmentation of Medical Images - Techniques & Application Cases


• Statistical modeling of objects (point distribution models)

Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241@2018 João Manuel R.S. Tavares 28Segmentation of Medical Images - Techniques & Application Cases


• Segmentation of based on active shape models(point distribution models, optimization)

@2018 João Manuel R.S. Tavares 29Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241



• Example: analysis of the vocal tract during

speech production from MR images (active shape model)


Originalimage

Finalsegmentation

Vasconcelos et al. (2011) Journal of Voice 25(6):732-742Segmentation of Medical Images - Techniques & Application Cases


• Example: analysis of the tongue shape during

speech production from MR images (active shape model)

Originalimages

Finalsegmentations

Delmoral et al. (2018) Journal of Engineering in Medicine 232(3):271-281@2018 João Manuel R.S. Tavares 31Segmentation of Medical Images - Techniques & Application Cases


• Segmentation of objects based on active appearance models (statistical models, optimization)

Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241@2018 João Manuel R.S. Tavares 32Segmentation of Medical Images - Techniques & Application Cases


• Example: analysis of the vocal tract shape during speech production from MR images (active appearance model)

Initialsegmentation

Finalsegmentation

Vasconcelos et al. (2011) Journal of Engineering in Medicine 225(1):68-76Vasconcelos et al. (2012) Journal of Engineering in Medicine 226(3):185-196@2018 João Manuel R.S. Tavares 33Segmentation of Medical Images - Techniques & Application Cases


• Segmentation of objects based on active contours (i.e. snakes – parametric models)


Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220

Esnake = Eints=0

1

∫ (v(s))+ Eext (v(s))ds

Eint =α (s)dv(s)ds

2

+ β(s) d2v(s)ds2

2



• Example: segmentation of medical images(active contours - snakes)


Initial contour Final contour

Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55



• Segmentation of objects based on the level-set method (geometrical models)


Ma et al. (2010) Medical Engineering & Physics 32(7):766-774Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

∂φ∂t

+ F Δφ = 0

Typical form of the motion equation:



• Example: segmentation of the carotid bifurcation in a Doppler image (active contour / level-set model)

Segmentation using a contour active method (Yessi’s model)

Segmentation using a level-set method (Chan-Vese’s model)

Silva et al. (2011) VipIMAGE 2011, 117-122Santos et al. (2013) Expert Systems with Applications 40(16):6570-6579Jodas et al. (2016) Expert Systems with Applications 46:1-14@2018 João Manuel R.S. Tavares 37Segmentation of Medical Images - Techniques & Application Cases



• Example: segmentation of the carotid artery lumen in MR images (K-means, classification indexes, active contour model)

Jodas et al. (2016) Computers in Biology and Medicine 79(1):233-242

1. Original image 2. K-means

4. Lumen identification (classification indexes) 5. Final result

3. Cluster with low intensity pixels (inverted) Pre-processing

Noise reduction Contrast improvement

Median filter Adaptive Gamma Correction

K-means clustering with subtractive

clustering

Region growing

To obtain all regions separately

Obtain pixels with low values

Segmentation

Lumen identification

Extraction of the contour pixels

Calculate classification

indexesMean roundness, irregularity and centre indexes

Active contourSelect the region with maximum

roundnessRefinement of the

contour



• Example: segmentation of the carotidartery contour in PDW MR images(pre-segmented lumen, active contour model)

Red contours - automatic segmentations; Blue contours -manual delineationsJM - Jaccard measure; DC - Dice coefficient; PD - Polyline distance; HD - Hausdorff distance; PD and HD in pixels

0.900.950.872.24

JM: 0.89DC: 0.94PD: 0.91HD: 2.00

0.870.930.932.00

0.820.901.072.00

@2018 João Manuel R.S. Tavares 39Jodas et al. (2017) Medical Image Analysis 40:60-79



• Example: segmentation of skin pigmented lesions in dermoscopic images (color spaces, level-set model)


Segmentation examples under different imaging conditionsand different types of skin pigmented lesions

Illustration of the segmentation process

Filho et al. (2015) Journal of Medical Systems 39(11):177Ma & Tavares (2016) IEEE Journal of Biomedical and Health Informatics 20(2):615-623Oliveira et al. (2016) Expert Systems with Applications 61:53-63




• Example: segmentationof 3D CT lung images(3D level-set model)

Filho et al. (2017) Medical Image Analysis 35:503-516Segmentation of Medical Images - Techniques & Application Cases


• Segmentation of objects based on the level set method with a prior knowledge


Ma et al. (2010) Medical Engineering & Physics 32(7):766-774Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246



• Example: segmentation simultaneously of threeorgans of the female pelvic cavity in MR images (level-set method, a prior knowledge)


Ma et al. (2013) Computers in Biology and Medicine 43(4):248-258Ma et al. (2012) The Int. Journal for Numerical Methods in Biomedical Engineering 28(6-7):714-726

Segmentation of the bladder, vagina and rectum in pelvic cavity images(3 examples)



• Example: segmentation of the bladder walls in MR images (level-set method, a prior knowledge)

Ma et al. (2011) Annals of Biomedical Engineering 39(8):2287-2297

Segmentation of the interior and external walls of the bladder (3 examples)



• Example: segmentation of the atrial region in CT images (active contour models, competitive strategy)

Morais et al. (2017) Medical Image Analysis 42:102-116

Semi-manual segmentation of the atrial region based on a competitive strategy

Examples of segmentations obtained in 3D and 2D (aortic tract, right and left atriums)



• Segmentation of objects based on artificial classifiers (neuronal network)

Albuquerque et al. (2010) Journal of Microscopy 240(1):50-59

Original image with training pixels (metallographic image)

Image segmented



• Segmentation of objects based on artificial classifiers (deep learning, convolutional neuronal network)

Delmoral et al. (2018) RecPad 2018, October 2018, Coimbra, Portugal

Used CNN architecture

Original CT image, segmented liver: manually (in red) and automatically (in yellow)



Conclusions

Conclusions

• The area of Image Segmentation is very complex and demand, but of raised importance

• Numerous hard challenges exist, for example, due to

adverse conditions in the image acquisition process,

occlusion, complex and unpredicted shapes

• Considerable work has already been developed, but the

proposed solutions are mainly developed on an application driven basis

• Many important and complex goals still to be reached• Methods and methodologies of other research areas,

as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching toward the development of more competent image segmentation solutions



AcknowledgmentsThe work presented has been done with the funding of the research projects and Institutions:

– NORTE-01-0145-FEDER-000022– PTDC/BBB-BMD/3088/2012– PTDC/SAU-BEB/102547/2008– PTDC/SAU-BEB/104992/2008– PTDC/EEA-CRO/103320/2008– UTAustin/CA/0047/2008– UTAustin/MAT/0009/2008– PDTC/EME-PME/81229/2006– PDTC/SAU-BEB/71459/2006– POSC/EEA-SRI/55386/2004


Research Team(Computational Vision)

Research Team (Computational Vision)


• Post-Doc students (5):

– Finished: Alexandre Carvalho, Simone Prado, Mercedes Filho, Pedro Predosa, Zhen

Ma

• PhD students (15):

– Finished: Zhen Ma, Francisco Oliveira, Teresa Azevedo, Daniel Moura, Sandra Rua,

Maria Vasconcelos, Roberta Oliveira, Danilo Jodas

– In course: João Nunes, Alex Araujo, Carlos Gulo, Pedro Morais, Andre Pilastri,

Domingos Vieira, Jessica Delmoral

• MSc students (37):

– Finished: João Martins, Diogo Sousa; Pedro Amorim, Joana Veiga, Rafaela Pinto,

Frederico Junqueira, Jessica Delmoral, Ricardo Le, Raquel Alves, Carolina Tabuas,

Jorge Pereira, Luis Ribeiro, Luis Ferro, Rita Teixeira, Liliana Azevedo, Diana Cidre,

Célia Cruz, Priscila Alves, Pedro Gomes, Nuno Sousa, Diogo Faria, Elisa Barroso,

Ana Jesus, Frederico Jacobs, Gabriela Queirós, Daniela Sousa, Francisco Oliveira,

Teresa Azevedo, Maria Vasconcelos, Raquel Pinho, Luísa Bastos, Cândida Coelho,

Jorge Gonçalves

– In course: Tiago Roque

• BSc students (2)

– Finished: Ricardo Ferreira, Soraia Pimenta


Publications & Events


Taylor & Francis journal “Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization”

www.tandfonline.com/tciv

Indexed in: Scopus, DBLP and Thomson ReutersEmerging Sources Index ©



Indexed in

Springer book series “Lecture Notes in Computational Vision and Biomechanics (LNCV&B)”Editors: João Manuel R. S. Tavares, Renato Natal Jorge

www.springer.com/series/8910


Events


João Manuel R. S. [email protected] www.fe.up.pt/~tavares

October 31, 2018

Segmentation of Medical Images – Techniques & Application Cases

Obrigado!

segmentation of medical images –techniques & application...

Documents