a computer vision system for the detection of cancer in barretts
TRANSCRIPT
A Computer Vision System for the Detection of
Cancer in Barretts Oesophagus
James Swainston
BSc Computer Science with Artificial Intelligence2007/2008
The candidate confirms that the work submitted is their own and the appropriate credit has been given
where reference has been made to the work of others.
I understand that failure to attribute material which is obtained from another source may be considered
as plagiarism.
(Signature of student)
Summary
Advances in digital imaging have made it possible to automatically analyse slices of human tissue
with the aid of a computer. This project investigates ways in which cancer of the oesophagus can be
automatically detected by a machine.
i
Acknowledgements
Firstly I’d like to thank Dr. Derek Magee, my project supervisor. He has been a great support
throughout this project and I would not have been able to achieve what I have done without his guidance.
He has been most willing to help, promplty replying to my many emails with helpful suggestions for
my project.
I would also like to thank Dr. Darren Treanor for the time and effort he has put into my project.
He has done a fantastic job of helping be to understand the medical background to the project. I would
also like to thank him for the time he spent counting nuclei for the evaluation of my nuclei counting
algorithm.
I would like to thank Dr. Matthew Hubbard for the very valuable feedback that he gave for my
mid-project report. It was highly useful to have such detailed feedback. I would also like to thank him
for the feedback given in the project demonstration.
ii
Contents
1 Project Introduction 1
1.1 Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Minimum Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Project Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Medical Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Histopathology in the Digital Age 4
2.1 Histolpathology and the Diagnosis of Cancer . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Virtual Pathology Slides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Applications of digital slides . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Barrett’s Oesophagus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3.1 Inadequacies with Current Grading System . . . . . . . . . . . . . . . . . . . 7
2.3.2 Indicators of Barrett’s, Dysplasia and Cancer . . . . . . . . . . . . . . . . . . 8
2.4 Motivations for Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Background to Machine Vision and Pattern Recognition 10
3.1 Image representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1.1 Colour Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 A Generalised Framework for Pattern Recognition Systems . . . . . . . . . . . . . . . 12
3.2.1 Image Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.3 Learning Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.4 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
iii
3.3 Relevant Image Processing Techniques and Concepts . . . . . . . . . . . . . . . . . . 16
3.3.1 Neighbourhood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.2 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3.3 Gaussian Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.4 Connected Components Analysis . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.5 Texture Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4 Current Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Design and Implementation 20
4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2 Review of Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.1 Matlab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.2 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.3 C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2.4 OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3.1 Image Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.4 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.5 Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.6 Image Pre-Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.7 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.7.1 Segmenting Nuclei . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.7.2 Separating joint Nuclei . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.7.3 Finding Nuclei Centre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.7.4 Computing Edge Lengths of Delaunay Triangulation . . . . . . . . . . . . . . 34
4.8 Learning Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.8.1 A new grading scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.8.2 Learning Class Representations . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.9 Classification Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5 Results Analysis and Evaluation 37
5.1 Detection of Nuclei . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
iv
5.1.1 Locating Nuclei Centres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.2 Classification of Slides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.3 Success of Class Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.4 Quality of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.5 Future enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.5.1 Exploration of New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.5.2 Testing of a variety of classifiers . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Bibliography 44
v
Chapter 1
Project Introduction
1.1 Aims
The aim of this project is to be able to classify areas on a virtual pathology slide as being either, normal,
dysplasia or cancer.
1.2 Objectives
• Research histopathology and the use of virtual slides in the field.
• Investigate how computer vision techniques could be used to diagnose cancer that has developed
from the condition Barrett’s Oesophagus.
• Develop a solution that will classify an area of a virtual slide as either being normal, dysplasia or
cancer.
• Compare the systems classifications with the human pathologists’ and provide statistical analysis
of the results.
• Carry out scientific evaluation of the success of the solution.
1
1.3 Minimum Requirements
The minimum requirements are:
• To understand the problem domain.
• To classify an area on a virtual pathology slide as either being normal, dysplasia or cancer.
1.4 Project Management
The table in appendix B shows the schedule that was made for the completion of this project. The major
milestones were identified which indicated areas of the project that needed to be completed before it
could progress any further. The project was managed to meet the internal deadlines set in the table in
appendix B. It was decided that as this project is a piece of scientific research rather than a traditional
software development project, following a specified design methodology such as the waterfall model
was not appropriate. However, it is still important that there is a method followed and this is discussed
further in section 4.1.
1.5 Medical Terminology
This section provides a reference for medical terms that occur frequently in the project. It is provided
as it assumes the reader has no background knowledge of human biology.
• Nucleus - The Nucleus is the part of the cell that contains the majority of the genes. Cells are
stained purple after tissue has been dyed with haematoxylin.[17]
• Cytoplasm - Cytoplasm is the substance that surrounds the nuclei that is composed of mainly
enzymes and water. Cytoplasm is stained pink after tissue as been dyed with eosin.[17]
• Histopathology - Histopathology is an area of medicine concerned with diagnosing disease by
examining human tissue samples.
• Immunohistochemistry - Immunohistochemistry is the process of staining tissue with certain
chemicals that make certain proteins visible in tissue.[19]
• Biopsy - A Biopsy is the process of removing human tissue for examination.
2
• Cytology - Cytology is the study of cells.[19]
3
Chapter 2
Histopathology in the Digital Age
2.1 Histolpathology and the Diagnosis of Cancer
Histopathology is concerned with the diagnosis of samples of human tissue. Cancer is traditionally
detected by pathologists who study slides of human tissue under a microscope. However, with the
advent of the digitisation of images, there have been an increasing amount of studies into machine
detection of various forms of cancer[7, 12, 30, 27]. Pathologists largely make their diagnosis based on
the distribution of different cells of varying appearance across the tissue and the internal structure of
the cells[7]. However, the diagnosis of a sample can be subject to significant inter- and intra-observer
variability[28]. This is due to the fact that a pathologists diagnosis is a subjective opinion and is based
on previous experience of recognising the condition[7]. This makes it attractive to be able to introduce
some quantitative techniques into the diagnosis process. Through use of image processing and computer
vision techniques, software systems can be developed to aid pathologists in their diagnosis of tissue
samples by introducing these quantitative measures.
4
2.2 Virtual Pathology Slides
This project utilises images produced by scanning glass slides of human tissue. This is achieved by
using extremely high resolution scanners produced by Aperio that are currently being used at St. James’s
hospital in Leeds. The scanners can currently capture images at up to 40X magnification. This enables
pathologists to analyse cellular structures at a very fine level of detail.A 20mm x 20mm specimen that
has been scanned at 40X magnification would produce an image with 80,000 x 80,000 pixels at a size
of 19.2GB, uncompressed. JPEG2000 compression is used on the servers to reduce the file size of the
images. It should be noted that 80,000 x 80,000 pixels is not a regular size as the images are more often
rectangular in shape, having been produced from rectangular glass slides.
Prior to virtual slides being produced there must be a glass slide with the human tissue specimens
available. The first step in the process of producing these slides is to extract the human tissue from
patients by means of biopsy or excision.The tissue is then treated with a solution to make it more rigid
and easier to slice. It is then cut into sections with a microtome to produce slices that are 3 micrometres
in thickness, although this size may vary. The tissue is then dyed using the chemicals haematoxylin and
eosin (or other chemicals depending on what is trying to be visualised), the former stains cell nuclei
purple and the latter stains cytoplasm pink [8]. This is shown in figure 2.1 The staining of the tissue
renders it visible under the microscope or in a scanned image of the slide. It is the patterns that these
colours produce that provide the primary means of the analysis of these histological images.
Figure 2.1: Nuclei are stained purple and cytoplasm is stained pink
It is important to note that the process of producing glass slides isn’t entirely perfect. Tissue is very
delicate and is prone to tearing. This can produce artefacts on the digital slides that will have to be
accounted for as they will have an effect on the automatic analysis of the images. Although Aperio state
5
that there scanning techniques produce virtually no artefacts[2], there still may be imperfections in the
original glass slides themselves. This will have to be taken into account when designing the software.
Figure 2.2: A digitised glass slide with a tissue slice highlighted in red
Figure 2.2 shows a typical virtual pathology slide that has a number of slices of human tissue on
display. The area highlighted by a red rectangle shows a single slice of oesophageal tissue. Each tissue
specimen on a slide has been taken from the same patient.
2.2.1 Applications of digital slides
The introduction of these digital slides could totally revolutionise the practice of pathology,[20] foresees
major impacts in the areas such as clinical diagnostics, research and education. In education, skills could
more easily be transferred from academics to students as points of interest on slides can be so easily
highlighted and referenced. The need for duplicating microscope slides in clinical diagnostics has been
eradicated. Pathologists from different institutions could simply share the same virtual slide on a single
server. This could vastly improve waiting times for diagnostic results. This project uses virtual slides as
a tool for research. The computational power of the computer is used to analyse the images and see if
anything new can be learnt from them.
The slides can be viewed using Aperio’s impressive ImageScope software, which introduces many
advanced techniques for working with them. The software provides much of the functionality of a
traditional microscope, allowing users to zoom in as far as the native resolution and pan across the
image to see different sections of the slide[29]. A particularly useful feature is that pathologists can
make there own annotations on the slide. This could allow pathologists to transfer their thoughts and
6
knowledge about different sections of the slides to other colleagues more easily.
2.3 Barrett’s Oesophagus
Barrett’s Oesophagus is a condition in which atypical cells develop on the lining of the Oesophagus[4].
It is caused by acid reflux from the stomach. It is thought that 6-15% of patients that undergo endoscopy
for this symptom have Barrett’s Oesophagus[14]. Before becoming cancerous, tissue goes through var-
ious phases of dysplasia in which the architecture of the tissue becomes more and more distorted[15].
It is a pre-invasive condition which can be diagnosed by examining the oesophageal tissue with a mi-
croscope. A six category scale is currently used to grade areas of tissues on a slide. As the grade of
dysplasia becomes higher the the structural abnormality of the tissue becomes intensified[25].
The Six Category Scale
1. Negative
2. Indefinite (Probably negative)
3. Indefinite (Probably dysplastic)
4. Low grade dysplasia
5. High grade dysplasia
6. Intramucosal carcinoma
2.3.1 Inadequacies with Current Grading System
The grade of dysplasia that a patient is diagnosed with has a direct effect on the clinical management of
the patient[30]. It is therefore of the utmost importance that a correct diagnosis can be achieved, whether
it is given by human or machine. However, dysplasia is traditionally difficult to grade and notoriously
based on subjective opinion. In [3] it is discussed that cancer was found in 11-73% of cases after surgical
resection of the oesophagus following the diagnosis of high-grade dysplasia. If the success rate of the
detection of cancer in removed sections of the oesophagus is so variable then clearly the diagnostic
practice must be questioned. It is thought that with quantitative measurement of the differentiation
between the tissue from each of the classes the diagnostic variability can be reduced[30].
7
2.3.2 Indicators of Barrett’s, Dysplasia and Cancer
[10] Discusses diagnostic features that are visible in dysplastic tissue and used by pathologists to give a
classification. Architectural abnormalities may include nuclear crowding and atypically shaped glands.
There are also cytological abnormalities in which the nuclei themselves change. They may become
larger in size and vary in shape. Another identifier of dysplasia is an increased nuclear-to-cytoplasmic
ratio, in which the nucleus dominates the cell body. The two extremes of the condition are shown in 2.3.
There are clear differences that can be seen in these images, the most notable being the crowding of the
nuclei in 2.3(b).
(a) Tissue with Barrett’s Oesophagus without Dysplasia (b) Tissue with Cancer
Figure 2.3: Showing the two extremes of the scale between Barrett’s Oesophagus and Cancer
2.4 Motivations for Research
Computer systems are currently being used in practice to aid Radiologists in detecting cancer. In one
study, a computer- aided diagnosis (CAD) system marked areas on a mammogram that warranted further
evaluation by the radiologist. It was found that there was an increase of 19.5% of cancers detected
when the CAD system was used compared to when the radiologist analysed the images without its
assistance[12]. It is important to note that even if the system does not prove to be as accurate as a
8
pathologist, it could indicate areas of suspicion on a virtual slide that could then be further investigated.
Although the exact figure is uncertain, it is thought that as many as 12% of patients that are diag-
nosed with Barrett’s Oesophagus go on to develop adenocarcinoma[15]. As with any type of cancer, the
chances of curing the disease are greatly increased if it is identified in its infancy[7].
9
Chapter 3
Background to Machine Vision and
Pattern Recognition
The classification of objects is a task that is central to this problem. The main requirement of the system
is to be able to classify areas of a virtual slide as belonging to a specific class. A set of images that
have been classified by a pathologist has been provided. Careful examination of these images is needed
to recognise features that are unique to each of these object classes. It is on the basis of these features
that a classification is able to be made. As [22] points out it is the recognition of patterns within images
that gives us knowledge about an object class [22]. This chapter focuses on machine vision and pattern
recognition techniques that are appropriate to the problem of analysing and classifying histological data.
3.1 Image representation
An image is a source of information that captures the current state of part of the world. The histological
images that are used in this project capture the state of the human body and provide pathologists with
the information that allows them to make an assessment of the well-being of a patient. An image is
essentially a multi dimensional array of numbers that represents the varying colour and intensity of the
light that is reflected from objects within a scene. Each number is a representation of the pixel at the
10
corresponding location in the image. It is these numbers that can be manipulated and extracted for use
in various computer vision systems.
3.1.1 Colour Space
In its simplest form an image is a two dimensional array of numbers that represent the intensity at each
pixel. However, there are also a number of different ways in which the colours in an image can be
represented. It is important to understand these representations as it greatly effects the way in which the
images can be used. Two of the most prevalent colour representations are discussed below.
The RGB Colour Model is based on the tristimulus theory of vision that suggests that the human eye
perceives colour through the stimulation of three visual pigments in the retina. The three pigments are
most sensitive to red, green and blue light[16]. This theory is represented in the RGB model, in which a
colour is defined by a mixture of green, red and blue. The RGB colour model could be used to represent
the histological images that are being worked with. One possible drawback with this model is that for
a single specific component of the images such as a nuclei, there may be much variation in the colour
values associated with it. This variation in colour is inherent from the process of creating the glass slides
with the tissue specimens. The slicing of tissue is a manual process performed by a technician so the
thickness of the tissue is prone to variation. If a tissue is cut too thickly then it will soak up more of
the dye resulting in colours with greater intensity. Similarly, if the tissue is cut too thinly then the tissue
will not soak up as much dye and the colours will be less intense. Variations in colour can also arise if
the tissues are left to soak in the dye for different amounts of time. This variance in colour could make
it difficult to segment individual components such as nuclei based on colour information alone.
The HSV colour model is thought to be more intuitive to work with[16]. Instead of mixing three
primary colours, A hue (colour) is first assigned and then the purity (saturation) of this colour and its
intensity (value) can be specified. The HSV colour model may be a better representation for the images
being worked with as it allows for the intensity of the colours to be represented in a separate channel,
namely the value channel. Therefore, to reduce the colour variation due to the differences in intensities,
the value channel could be discarded and the images could just be represented by the hue and saturation
of each pixel.
11
3.2 A Generalised Framework for Pattern Recognition Systems
The purpose of a pattern recognition system is to be able to assign a class to an object. The objects of
this project are the areas of human tissue on the virtual pathology slides and the classes are the varying
grades of dysplasia. The component of the system that actually assigns a class to an object is called a
classifier. However, before a classifier can make a decision about an object there are a number of steps
that must be taken to generate a formal description of each class. There are two main stages in the
process towards classification of the images. Firstly, figure 3.1 outlines the learning stage. This is the
stage in which pre-classified images are fed into the system so that a representation of each class may
be learnt. Figure 3.2 shows the various steps that an unseen image goes through before it is classified.
Figure 3.1: Learning class representations
12
Figure 3.2: Classifying unseen images
3.2.1 Image Pre-processing
Image pre-processing is concerned with manipulating the image so that features that may be useful in
further analysis are enhanced[22]. It is often desirable to remove or suppress noisy data in images that
would otherwise make it difficult to extract the information that is needed. Histological slides may have
noise in the form of pale areas, tears and anomalous patterns cause by the slicing process[8]. Smooth-
ing filters have been used in previous works to remove such noise [28, 18]. Other image processing
techniques such as edge detectors can be used to delineate areas of interest in an image.
3.2.2 Feature Extraction
To be able to recognise an object in an image there has to be some numerical way of describing a
feature of the object that makes it different from other objects that occur in the image. These numerical
descriptors are called features and in computer vision, a feature vector as shown in equation 3.1 is used
to describe an object class [22]. X is the set of all features for an object (the feature space) and each
member of X in some way numerically describes the object.
X = {x1,x2,x3, . . . ,xn} (3.1)
13
A salient point in an image is one that can be found in many images of the same object. It should be
invariant as possible to factors such as lighting, viewpoint, scale, rotation and intra-class variation. It is
important that these points are unique enough that they represent an object class effectively and can be
used to discriminate between classes. Salient points indicated areas of an image that a feature could be
extracted from.
Features can be based on different things in an image such as colour of interesting objects and edges
that define objects. Different types of features that could be used to describe histological images are
discussed below.
Morphological features provide information about the shape and structure of the cell and nucleus
itself[7]. A possible feature could be the area of a nucleus, which in computer vision terms would
translate as the total number of pixels within the boundary of the nucleus. For this measurement to be
computed accurately the boundary of the nucleus would have to be defined exactly. In [30] this was
achieved by manually tracing the nuclei border.
Textural features provide information about the variations in the intensity of a surface in an image.
Texture can be thought of a connected region of similar or identical pixels that occurs regularly in an
image[7]. It may be interesting to take square image patches around nuclei centres. Image patches from
the same class could then be clustered to form prototypical patches that provide a representation of the
class, as is performed in [13] to represent different object categories such as cars and faces. This feature
would reflect the changes in the nuclear-to-cytoplasm ratio (NRC) that occur when tissue goes through
the plastic transformations. In tissue without cancer, the NRC is smaller so there would be less change
in intensity across the patch. In cancerous tissue the NRC is larger so there would be a greater change
in intensity. (might be nice to show small patches showing this)
Topological features provide information about the spatial relationship between different structures
within the tissue. One such relationship could be defined by finding a triangulation between the nu-
clei. The Delaunay Triangulation between nuclei centres has been used in [18] to grade cervical intra-
epithelial neoplasia (CIN). The Delaunay triangulation presents a number of features that could be used
such as the mean edge length for each class and the mean triangle area. This feature reflects the fact that
pathologists grade CIN based on the degree of nuclear crowding [18].If the mean edge length is small
then there is likely to be significant nuclear crowding. Similarly if the mean edge length is large then
the nuclei are more spread out and the tissue is less likely to be cancerous. As nuclear crowding is an
indicator of high-grade dysplasia and cancer finding the Delaunay Triangulation may provide powerful
14
features that could discriminate between the different grades of dysplasia.
3.2.3 Learning Stage
Once the features that could discriminate between classes have been discovered learning algorithms can
be used to estimate the parameters of some classification model. A training set of images is used to
provide examples of each object class. Features are extracted from each of these training examples so
that a model for each particular class may be built[9]. A model to represent a class could be as simple
as a histogram of the colours in an image. Colour histograms could be extracted from each training
instance from each class, these could then be averaged to give an average colour histogram for each
class. Two different types of learning are discussed below.
Supervised learning is a type of learning in which each training instance has already been classified
by a teacher[24]. It is then the job of the learning algorithm to discover a function that separates the
classes based on the training instances.
Unsupervised learning is a type of learning in which patterns in the training instances are learnt
without them having been given a category from the teacher.This approach to learning can be used if
not much is known about the problem domain. Natural clusters in the data are found that may be able
to be used to discriminate between object classes.
3.2.4 Classification
Nearest Neighbour classifiers work under the assumption that the data points that represent a class lie
in a close region together[24]. If the data input to the classifier lies within this same region then it can
be classified as being in the same class.
Neural Networks are a computational model based on the biological computation of the brain. The
neuron is the elementary processing unit in a neural network and most neural approaches to classifica-
tion problems are based on the combinations of these individual units[22].A neuron takes a number of
weighted inputs and generates a single output. The output vector of a neural network may be used for
classification. If it is m-dimensional, binary and only containing a single one, the position of the one
classifies the input pattern into one of m categories[22]. A neural network may be used for tasks such
as pattern recognition[6].
[23] presented the performance of different types of neural networks on 2-d pattern recognition
problems. The features used for classification were based on the shape of the object. The features
15
were extracted from the Fourier and wavelet transformations of the image data[23]. The objects they
were classifying were simulated shapes of different types of 2-d airplanes. No amount of image pre-
processing such as edge detection was needed because just the border of the airplanes were given.
Three types of neural classifiers were tested, the multilayer perceptron (MLP), Kohonen self-organizing
network and a Hybrid neural network. The results showed that the classifier with the highest success rate
was the Kohonen self-organizing network with wavelets being used as descriptors of the pattern[23]. A
similar approach could be taken in this problem where the edges of nuclei are detected rather than the
edges of 2-d airplanes.
Support Vector Machines (SVM) have recently been shown to have very high classification rates in
pattern recognition problems[13]. They improve on single-layer neural networks as efficient algorithms
can be used to train them and they can represent complex nonlinear functions[24]. They are useful
in situations where a problem is known not to be linearly separable, meaning that a function can’t be
found that would easily split the image data into two separate classes. A recent study into the automatic
diagnosis of breast cancer showed a SVM to be the must successful classifier in that case[27]. The
classifiers that it was tested against were a multilayer perceptron neural network, a combined neural
network, a probabilistic neural network and a recurrent neural network. The SVM performed extremely
well, classification accuracy of 99.54 was achieved[27].
3.3 Relevant Image Processing Techniques and Concepts
To be able to detect patterns and extract features in images, there are many underlying computer vision
techniques that are relied upon. The following methods reviewed have been considered relevant to the
project and warrant further investigation.
3.3.1 Neighbourhood
The neighbourhood of a pixel is usually defined in one of two ways. The 4-neighbourhood of a pixel
is defined as the pixels that are adjacent to it directly above, to the right, below and to the left. The
8-neighbourhood of a pixel is defined similarly but includes pixels that are adjacent in the diagonals
also.
16
3.3.2 Segmentation
It is often highly important to be able to extract specific regions of an image that may constitute an object
such as a face or a cow. Segmentation is often achieved by dividing the image into regions that are alike
with respect to a property such as colour or texture[22]. With regards to this project it may be of great
use to be able to segment the nuclei so that they are separate from the rest of the image. If individual
nuclei can be segmented then morphological features may be measured to provide information that is
useful for classification. A number of different type of segmentation techniques are discussed below.
Intensity based segmentation can be used on greyscale images to extract regions within a certain
range of intensities. A threshold T can be chosen so that if a pixel p ≥ T it is labelled as an object pixel,
otherwise it is a background pixel[22]. As nuclei are stained purple they are likely to be darker than the
surrounding matter. K-means clustering can also be used to identify natural clusters of intensity values
in the image. This is used in [28] to find an intensity value that could segment potential nuclei.
Colour information can also be used for segmentation. Again, a clustering algorithm such as k-
means can be used to identify natural clusters of colours in the image that may represent different object
regions. Histograms of colour values in the image can be used to analyse the distribution of colour
across the image. The histogram bin with the highest frequency represents the dominant colour range
in the image
Graph-cuts segmentation treats each pixel in an image as a vertex of a graph. Edges are defined
between neighbouring pixels. Weights can be added to the edges to represent the similarity between
the two pixels. These weights could be based on properties such as the difference in brightness or the
difference in colour values. The problem then becomes to partition the graph so that the sum of the edges
cut is minimised. A graph G = (V,E) can be partitioned into a number of disjoint sets X1∩X2∩Xn = /0
simply by removing edges [26]. These edges that are removed are the edges that have been cut[26].
cut(A,B) = ∑u∈A,v∈B
w(u,v) (3.2)
Equation 3.2 defines the size of the cut that separates a graph into the two partitions A and B as being
the sum of the edges that cut the graph. This theory could be used in this project to separate nuclei from
the background matter. As all nuclei are a similar colour, edge weights should represent the difference
in colour between pixels. Edges between two similar pixels such as two nuclei pixels should have small
weights. This way as the graph is partitioned to minimise sum of the edge weights cut, theoretically the
17
nuclei should be labelled as separate from the background.
3.3.3 Gaussian Smoothing
It is often desirable to blur an image to remove unwanted information in the image. New pixel values
are determined by averaging the intensity values in a neighbourhood[22]. A kernel (small matrix) with
values representing a discrete approximation of the Gaussian distribution is used as a mask over the
image. The kernel is moved across the image in a sliding window fashion, where each pixel in the
image is multiplied by the corresponding value in the kernel. A smoothing technique such as this could
be used to discard noisy data in the histological images.
3.3.4 Connected Components Analysis
Once a binary image has been produced representing foreground and background pixels it is often
desirable to identify connected regions. These connected regions can be discovered by looping through
an image and labelling adjacent pixels, where adjacency is defined by either the 4- or 8- neighbourhood
of a pixel. It can be useful for identifying different regions in a binary image after segmentation.
3.3.5 Texture Analysis
As discussed before in section 3.2.2 textual features provide information about the variations in intensity
across the surface of an image. Features such as a grey-level co-occurrence matrices and grey level
run lengths have been used before to describe the intensity variations on the surface of colon tissue
slices[11]. These features were then used to automatically locate plastic areas in the images.
3.4 Current Solutions
There are currently no fully automated software systems that can analyse areas of oesophageal tissue
and give a classification relating to the six category scale used by pathologists. There have been attempts
to grade the level of dysplasia by using interactive software tools to measure different features of the tis-
sue and use these for classification purposes. In [30] an interactive system name QPRODIT R©was used
to manually trace round over 100 nuclei. Measurements such as the mean nuclear area and the relative
position of the nucleus in the epithelium (stratification index) were then taken. Immunohistochemistry
was then used to highlight the areas that contained the proteins p53 and ki67. It was found that the best
18
combination of features when discriminating between no Dysplasia and Low-grade dysplasia was strat-
ification index with the percentage area of p53, with an accuracy score of 89%. The best combination of
features when discriminating between low-grade dysplasia and high-grade dysplasia was stratification
index with the percentage area of ki67, with an accuracy score of 91%. Whilst these are relatively high
accuracy scores they have only been achieved when discriminating between two classes. The system
would also have limited use as it requires a pathologist to manually delineate areas of interest. A more
attractive solution would automatically extract the features used for classification, and would be able to
accurately discriminate between a greater number of classes.
19
Chapter 4
Design and Implementation
4.1 Methodology
This project has been a piece of scientific research rather than a traditional software engineering project.
The main focus of the project has been to discover ways in which cancer could be detected automatically.
For this reason, it was concluded that rather than following a specific design methodology, it would be
more appropriate to follow the framework outlined before in figures 3.1 and 3.2. This enabled the
problem to be broken down into more manageable sections, these being image pre-processing, feature
extraction, machine learning and classification. Computer vision techniques for each section were first
researched by reviewing relevant literature. Techniques learnt were then tested out on the images and
evaluated as to whether or not they would be of any use in the project.
4.2 Review of Technologies
As this project was to be highly experimental, it was decided that the main criterion for choosing soft-
ware tools should be that they allow one to rapidly prototype and evaluate different ideas. This would
mean if a particular project direction failed, new ideas could be implemented and tested quickly. The
following section evaluates the different software tools that were considered for use.
20
4.2.1 Matlab
Matlab is an integrated development environment with its own inbuilt high-level programming language.
It is often used for scientific research and solving highly technical problems. The main advantage of
Matlab is that it provides the tools to be able to solve complex problems faster than with traditional
programming languages. As Matlab is a high-level language, it is possible to develop solutions more
rapid than would be possible using a language such as C. Matlab has many inbuilt functions for working
with images. This makes it a particularly useful tool for working on this project as pathological images
are the major source of input data. Another major advantage of Matlab is that it provides simple tools to
visualise data. This would be very useful for producing graphs and images to support any findings that
arise in the proceedings of the project.
The only considerable disadvantage of using Matlab is that being as it is a high-level language,
processor intensive task may take longer to execute than if they had been coded in a lower-level language
such as C.
4.2.2 Java
Whilst Java is another high-level language that would make it easier to rapidly prototype ideas, it lacks
the inbuilt functions that make Matlab an easier tool to use. One benefit of Java is that it has the very
useful Swing library that can be used to develop a GUI(graphical user interface). Each Matlab installa-
tion has a JVM(Java Virtual Machine) which means that Java code can be run from within Matlab. A
candidate solution could be to use Matlab for the image processing work and Java for the GUI.
4.2.3 C
C is a low-level language that gives the programmer access to the computers underlying hardware. It
allows the programmer to manage their own memory and therefore has a speed advantage over languages
that have there own garbage collection services such as Java. However, this in turn means that there is
more effort on the programmers part. Whilst a program written in C may be more efficient than one
written in a higher-level language, a programmer could write the same program more quickly in a
language such as Matlab or Java.
21
4.2.4 OpenCV
OpenCV is a computer vision library that can interface with languages such as C++,C and Python. It is
a relatively mature library that includes over 500 functions and can be very efficient as it is implemented
in C and C++. Using this library in conjunction with C is an attractive candidate solution as it provides
both the performance benefits of using a low-level language and the functionality of having a library of
readily available functions that would be useful in the project.
4.3 Performance Considerations
In medical applications there is a trade off between performance and accuracy. When diagnosing disease
it is extremely important to be as accurate as possible as there is the welfare of the patient to consider.
However, in the case of diagnosing cancer it is also important to be able to diagnose a patient quickly.
Treating the condition early on in its development can often significantly increase chances of survival[7].
Therefore, if the software was to be of any use it would need to be both fast and accurate.
Speed is a major performance consideration if the product is to be useful to a Pathologist. After
discussion with Dr. Treanor it was found that pathologists can spend between 5 and 20 minutes diag-
nosing for Barrett’s and dysplasia under a microscope. They are sent between 2 and 30 pieces of tissue
to examine. 3 sections will be looked at from each tissue slice up to 90 images may be reviews in total.
If the system was fast enough then there would be the potential for more tissue sections to be examined
and so the diagnostic process could be made more efficient.
4.3.1 Image Size
The images being worked on are extremely large and can be up to 30GB in size. It is therefore important
to write as efficient code as possible. In their digital form images are simply matrices of numbers. As
Matlab is a development environment that has been optimized to work with matrices this makes it a very
useful tool for processing images efficiently.
On the platform that this project is being carried out on there is sufficient processing power for nu-
merical computations not to be too much of an issue when it comes to performance. However, thousands
of pixels per image are being processed so it is still important that code is written to be efficient.
A major factor affecting the speed of the program is the retrieval of images from the hospital server.
After conducting a small experiment with 20 images of size 500x500 pixels it was found that it takes an
22
average of 0.61 seconds to retrieve an image and load it into Matlab. This network latency could be rid
of by storing the images on a local machine. However, it has not been possible to duplicate the images
as it is sensitive patient data. The resources are also not available as duplicating the images would mean
having to have an extremely large amount of disk space on a local machine.
4.4 Proposed Architecture
After careful consideration and analysis of the different tools and performance criteria, it was decided
that the best architecture to use for the project would be to use Matlab for development and to use
the Aperio image server interface for the retrieval of images. Matlab is both efficient when processing
images and allows the programmer to develop more quickly. It was felt that whilst the combination of C
and OpenCV would produce computationally efficient code, it could hinder the progress of the project
as the development time would be longer.
The platform being worked on is Linux with 2GB RAM and an Intel Core 2 Duo processor. Matlab
is designed to use only one core at a time, each core is 2.13GHz. The application is to be developed in
Matlab it is easily portable to a Windows machine with a Matlab installation.
4.5 Image Retrieval
There is an issue with the reliability of Aperio’s image server. It can’t be guaranteed that images of the
correct size will be returned if the requested image is greater than 500x500 pixels. For this reason the
images were requested in 500x500 blocks and then concatenated together. The code to achieve this was
provided by [1]. The URL for each image was provided by the pathologist.
4.6 Image Pre-Processing
It was found that if a Gaussian filter was used to smooth the original image then the segmented nuclei
become more clearly defined. Figure 4.10 shows the difference between a histological image before and
after the filter has been applied. Figure 4.1(b) was achieved by applying a Gaussian filter with a 7x7
kernel and a standard deviation of 9.
23
(a) Original Image (b) Smoothed Image
Figure 4.1: Showing the difference between images after Gaussian smoothing
A clear difference in the segmentation of the nuclei in figure 4.2(a) and figure 4.2(b) can be seen.
In figure 4.2(a) some of the nuclei segmentations are not fully connected, having with missing regions.
There is also a larger amount of noise in figure 4.2(a), which can be seen as the very small connected
regions.
(a) Nuclei segmented from original image (b) Nuclei segmented from smoothed im-
age
Figure 4.2: Showing the difference between nuclei segmentation after smoothing
24
4.7 Feature Extraction
After careful review and analysis of the images it was decided that a potentially powerful feature would
reflect the fact that the nuclei in the tissue become more crowded as the grade of dysplasia becomes
higher, as shown below in figure 4.3. It was therefore decided that a Delaunay triangulation for the set
of nuclei centres should be found. A Delaunay triangulation for a set of points in the same plane, is a
triangulation such that no point is in the circumcircle of any triangle in the Delaunay triangulation[5].
The edge lengths between the nuclei would then give information about how crowded a section of tissue
is. The following section discusses the steps that were taken to find a Delaunay triangulation and to
compute the lengths of the edges.
Figure 4.3: Increased nuclear crowding is visible from left to right
4.7.1 Segmenting Nuclei
The first step to finding the Delaunay triangulation was to segment the nuclei from the background
matter. This could be done using a thresholding technique. To choose the correct threshold a preliminary
experiment was conducted. Firstly, nine different images were selected, three from each class, normal,
dysplastic and cancerous. The Gimp photo editing software was then used to colour areas of the pictures
that were clearly nuclei bright green (shown in figure 4.4(a)). Bright green was chosen, as this is a colour
that doesnt naturally occur in the images.
25
(a) Nuclei Coloured Green (b) Binary Mask (c) Nuclei Segmented
Figure 4.4: Manually segmenting Nuclei to extract HSV values
The bright green area was automatically then segmented to produce a binary image (shown in figure
4.4(b)) that indicated the locations of the nuclei on the original image. This area was simple to segment
as the HSV values of it were already known and were clearly separable from the background values.The
locations of the ones in the binary image indicated which locations were nuclei pixels. Each value in the
original image is then multiplied by the corresponding value in the binary mask. This leaves the nuclei
segmented as shown in figure 4.4(c).
Once the nuclei were segmented the HSV values were extracted for each pixel. These values were
then plotted to produce the graphs shown below in figures 4.5, 4.6 and 4.7. The graphs were then used
to find suitable threshold values to be able to extract the Nuclei.
Saturation against Value
As can be seen from Figure-4.5 the nuclei are not separable from the background when using the
Saturation and Value channels. Whilst there is clearly a cluster of nuclei HSV values, the spread of
the data is far too large. A significant amount of background data points lie in the same range. If the
segmentation was attempted with these channels there would be a lot of background noise in the images
and the nuclei would not have been totally separated from the background.
26
Figure 4.5: Saturation vs. Value
Hue against Value
As is the case with the Saturation and Value channels, Figure-4.6 shows that the Hue and Value
channels also can’t be used together to segment the nuclei from the background. Whilst there is a
significant cluster of nuclei data points in comparison to Figure-4.5, the range is again too large. There
are too many background data points present in the same region, meaning that again the nuclei wouldn’t
be separable from the background when using this specific combination of channels.
27
Figure 4.6: Hue vs. Value
Hue against Saturation
The graph displayed in figure 4.7 shows that the hue and saturation channels are the most suitable
for segmenting the nuclei from the background matter. There is a significant, dense cluster of nuclei
data points that are aren’t too spread. Whilst there is some overlap of the nuclei and background data
points, it isn’t as much as in figures 4.5 and 4.6.
28
Figure 4.7: Hue vs. Saturation
After deciding that the hue and saturation channels provided the best separation of nuclei from
background matter, a suitable threshold was chosen by using the graph in figure-4.7. Nuclei hue and
saturation values were determined to be in the following ranges.
0.7≥ Hue≤ 0.85
0.4≥ Saturation≤ 0.8
To automatically segment nuclei a binary mask (shown by figure 4.8(b)) was created by thresholding
the original image (shown by figure 4.8(a)) with the inequalities defined above. The white areas of the
binary mask indicate the locations of pixels of the original image that are potential nuclei.
29
(a) Original Image (b) Binary Mask (c) Nuclei Segmented
Figure 4.8: Automatic segmentation of Nuclei using thresholding on hue and saturation channels
4.7.2 Separating joint Nuclei
Once a binary image indicating areas of the original image was found, connected components analysis
was used to find regions of adjacent pixels. The pixels in each different connected component are
labelled with different numbers to indicate that they are separate contiguous regions. For example, the
function used labels pixels in the largest connected components with a 1, the pixels in the second largest
components are labelled with a 2 and so on. These adjacent regions should in theory represent the nuclei
found in the original image. However, there was a problem with using this technique as some nuclei in
the image were too close together to be separated, and were treated as a single large region. As can be
seen from figure 4.10(a), there are large regions of connected pixels (white areas that touch each other)
in which there are clearly more than a single nucleus. This would make the calculation of the edge
lengths between nuclei centres inaccurate and may render it useless as a classification feature.
To know how many different regions a component needed to be partitioned into, a small experiment
was carried out to estimate the average size of a nuclei in pixels. Individual nuclei were extracted in
the same way in which they were to carry out the nuclei colour experiment described in section 4.7.1.
Connected components analysis was performed on the binary image that resulted from the thresholding.
This resulted in a labelled binary image in which each different component was labelled with a different
number. A histogram was then drawn (figure 4.9) to show the frequency of different nucleus sizes across
the images used in the experiment.
30
Figure 4.9: A histogram to show the distribution of nucleus sizes
This histogram shows that the majority of nuclei at 40x magnification are between 200-250 pixels in
size. However, the average size of a nucleus was found to be 421 pixels. It was decided that connected
components less than 100 pixels could be ignored as nuclei were rarely found to be smaller than this.
Equation 4.1 was used to estimate the number of nuclei that were in a connected region.
Number of nuclei in connected region =Size of connected region(in pixels)
x (average size of nuclei at 40x)(4.1)
To separate the nuclei that had been grouped together a graph-cuts based segmentation method was
used. Each pixel within each connected component was represented as a node of a graph. The following
algorithm was then used to determine whether or not there should be an edge between two pixels.
1. For each pixel in a connected component loop through its 4-neighbourhood
31
2. If the neighbouring pixel is a foreground pixel (Part of the Nuclei) then define an edge between
the two.
A graph representation for each connected component (potential nucleus) was saved as a text file.
The first two numbers on the first line of the file represented the total number of nodes and the total
number of edges. In each subsequent line the first number was the graph node, and each number on
the same line (delimited by whitespace) were the other nodes that it was connected to. This was the
format accepted as input by the graph partitioning software package Metis that was used to divide
the graph into separate connected regions. Metis is a graph partitioning program that uses multilevel
partitioning algorithms to divide a graph into a specified number of partitions. These algorithms work
by first reducing the size of the graph by collapsing vertices and edges and finding a partition[21]. The
size of the graph is then increased incrementally and the partition is improved at each level. After Metis
computed the partitions for each connected component it output a text file indicating which node (pixel)
belonged to which partition. This text file was used to relabel the original binary image with the new
partitions.
for each pixel in connected component
if partition = 0
keep original label;
else
new label = total number of connected components + partition number;
end
add number of new partitions to total number of connected components;
Figure 4.10(b) shows the output of the relabeling algorithm. The connected components have been
seperated into different regions as shown by the colouring. Each region indicates the area in which the
centre of a nucleus will lie.
32
(a) Multiple nuclei in components (b) Components separated to indicate
nuclei
Figure 4.10: Showing the connected components after graph partitioning and relabeling
4.7.3 Finding Nuclei Centre
To find the centre of the nucleus the mean x and y coordinates of the foreground pixels were taken for
each connected component. The centres of each of the nuclei were then stored in a vector so they could
be used in calculating the Delaunay triangulation. An example of the nuclei centres that are found by
the system is shown in figure 4.11.
Figure 4.11: A tissue section with the nuclei found automatically
33
4.7.4 Computing Edge Lengths of Delaunay Triangulation
Matlab provides an inbuilt function called Delaunay to calculate the Delaunay triangulation of a number
of points. The parameters that the function takes are an X vector and a Y vector that correspond to a set
of data points. In this case they represent the centre of each nuclei in the image. A calculated Delaunay
triangulation is shown in figure 4.12.
Figure 4.12: The Delaunay triangulation of nuclei centres plotted over the original tissue section
Pythagoras’s theorem (equation 4.2 shows it solved for c) was then used to calculate the edge lengths
in the triangulation. Each edge was treated as the hypotenuse (c) of a right angled triangle. The lengths
of the other two edges(a and b) were found by computing the difference in X and the difference in Y
for the two edge endpoints (nuclei centres).
c =√
a2 +b2 (4.2)
4.8 Learning Stage
This section discusses the processes of learning a representation of each separate class from the training
images. A new grading scale is first proposed.
34
4.8.1 A new grading scale
Currently a six category scale is used to grade dysplasia. However, as discussed before it is notoriously
hard to grade and when the tasks is performed by humans there is not a high percentage of agreement
between different pathologists. For this reason a three category scale was used for learning and classifi-
cation purposes.
The Three Category Scale
1. Barrett’s but negative for dysplasia
2. Positive for dysplasia
3. Intramucosal carcinoma
Grades 1 and 2 from the original scale were treated as negative for dysplasia. Grades 3 and 4 from
the original scale were treated as dysplastic and grades 5 and 6 were treated as Intramucosal carcinoma.
This new category scale was deemed acceptable as the clinical outcomes of being diagnosed with any
of the three categories would be the same as if the grading system had been unchanged. If a patient
is diagnosed as category 1 they are likely to undergo endoscopic surveillance. If they are graded as
category two then the surveillance of the tissue would be increased. If they are diagnosed with category
3 then esophageal resection is likely.
4.8.2 Learning Class Representations
The images that were provided by the pathologist were separated into a training set and test a test
set for each of the three classes. For each training instance a histogram was computed that showed
the frequency of the edge lengths in the Delaunay triangulation found. For each class, an average
histogram was then computed which was the sum of all the training histograms in that class divided
by the total number of histograms in the class. It was hypothesised that the histogram representing the
negative for dysplasia class would have a low frequency of short edges and a higher frequency of longer
edges, reflecting the fact that nuclei are more spread out across the tissue. The histogram representing
Intramuscosal carcinoma would have a higher frequency of short edges and a lower frequency of longer
edges, reflecting the fact that nuclei are crowded together. The histogram representing the positive for
dysplasia class would be somewhere in-between the two extreme classes.
35
4.9 Classification Stage
A simple nearest neighbour classification method was used to try and separate between the classes and
assign a grade from the three category scale to previously unseen images. Firstly, the test image would
go through the same stages as the training images so that a histogram representing the frequency of
different edge lengths could be computed. The absolute difference between the test image’s histogram
and each of the class’s representation histograms was then computed. The lowest computed difference
gave the class that the test image was most similar to, and so the test image was then classified as being
in this class type.
36
Chapter 5
Results Analysis and Evaluation
This chapter presents an analysis of the results of the cancer detection algorithm. The accuracy of the
classifier is able to be evaluated against the classifications of the test data given by the pathologist.
5.1 Detection of Nuclei
The accurate detection of nuclei centres was a very important stage in the classification algorithm as it
formed the basis of finding a Delaunay triangulation between the nuclei. The success of this stage of
the cancer detection is discussed below.
5.1.1 Locating Nuclei Centres
In conjunction with the classification of the test data, the number of nuclei detected on each panel
was recorded. Sixteen cases were sent to the pathologist so that the accuracy of the nuclear detection
algorithm could be reviewed. Two images were given for each case, the first was the original image
without any markings and the second was the nuclei with their centres marked and a count of the total
number of nuclei detected. The algorithm proved to be very effective at detecting nuclei, on average
there were only 2.6% Type II errors (false negatives), meaning that only 2.6% of nuclei in each area
were not detected. However, on average there were 8% Type I errors (false positives), meaning that the
37
number of nuclei in an area was overestimated. The possible reasons for these errors is discussed below.
The Type I errors in which a point in the image was accepted as a nuclei centre when it isn’t resulted
from the algorithm overestimating the number of nuclei in a given area. An improvement to the accuracy
by using a more complex model to estimate how many nuclei may fit within a given area.
The Type II errors in which a point in the image was rejected as a nuclei centre when it should not
have been could have resulted from the nucleus not being detected at the segmentation stage. These
nuclei were not detected because they didn’t fit within the threshold. A particular problem that was
noticed is that very pale nuclei are often missed.
5.2 Classification of Slides
The results show that the system was most success full at classifying images from class 1(Barrett’s but
no dysplasia), achieving a score of 76%. Images from class 3 (Cancerous) were recognised 66% of the
time. When classifying images from class 2 (Low grade dysplasia), only a recognition rate of 53% was
achieved. The confusion matrix shown by figure 5.1 shows the results of the classification. The total
accuracy achieved was 67%.
Pathologist
1 2 3
1 23 4 2
Machine 2 4 16 8
3 3 10 20
Table 5.1: Confusion matrix
Figure 5.1 shows that the main area of confusion when classifying was between classes 2 and 3.
8 out of the 10 incorrect classifications of class 3 images were classified as class 2 by the system.
Similarly, 10 out of the 14 incorrect classifications of class 2 images were classified as class 3 by the
system.
38
5.3 Success of Class Representation
Figures 5.1,5.2 and 5.3 show the average histograms that were calculated for each class from the set of
training images that were given. They show the frequency of different edge lengths for each class. As
was stated previously, the images were broken down into 500*500 pixel sub-images so that they could
be stored in memory. This means that the graphs represent the frequencies of different edge lengths
found in the average 500*500 pixel sub-image of the areas indicated by the pathologist for each class.
The x axis bins are in order of edge length from left (smallest) to right (largest).
Figure 5.1: The representation of class 1 (Barrett’s but no dysplasia)
39
Figure 5.2: The representation of class 2 (Low grade dysplasia)
As can be seen from figure 5.1, class 1 is fairly distinguishable from classes 2 and 3. The frequencies
of the shorter lengths are significantly less than in classes 2 and 3. However, figures 5.2 and 5.3 show
that classes 2 and 3 are less distinguishable from each other in this representation. This could imply
that by the dysplastic stages the nuclei are as crowded as they will ever be. It may be some other
feature that can distinguish between classes 2 and 3. This is the reason that the system is less accurate
at discriminating between classes 2 and 3. An overall accuracy rate of 67% is not acceptable. If the
system is to be of any use to a pathologist then a better representation of each of the classes is needed to
be found.
40
Figure 5.3: The representation of class 3 (High grade dysplasia and cancer)
5.4 Quality of Data
A problem with evaluating the accuracy of the classifier is that there is variation between two different
pathologists in the diagnosis of the slides. As a pathologists diagnosis is subjective, it cannot be guaran-
teed that it is 100% accurate, but in this project it has been taken as a “gold standard.” The only figure
of accuracy that can be given is the agreement with the pathologist.
5.5 Future enhancements
This section suggests a number of possible enhancements that could be made to the project. The main
areas of improvement that have been identified are the accuracy of the classifier and the usability of the
41
system.
5.5.1 Exploration of New Features
A number of possible features that could have been used for classification in this project were identified
in section 3.2.2(Feature Extraction). The feature of the images that was exploited in this project was
the fact that the nuclei become more crowded in the tissue with increasing severity of dysplasia and are
at their most crowded when the tissue becomes cancerous. To improve the accuracy, different features
could be explored and are suggested below.
• One area that wasn’t explored was the fact that the nuclear-to-cytoplasm ratio in the cells increases
as the tissue becomes more and more dysplastic and eventually cancerous. As the centre of a
nucleus can already be found, a small patch around the centre could be taken to look at the texture
across a cell. In a cancerous cell there should be a greater amount of purple as the NCR is higher.
In a non-cancerous cell there should be a greater amount of pink as the NCR is lower.
• Another area that wasn’t tested was the morphology of the nuclei and cells themselves. It was
noticed that as tissue becomes cancerous, the nuclei loose there usual round shape and become
distorted. This fact could possibly be exploited to discriminate between the classes. The boundary
of the nuclei could be detected, if there are steep changes in the curvature then this could represent
a distorted nuclei, as regular nuclei should have a smooth curvature.
5.5.2 Testing of a variety of classifiers
Only a nearest neighbour classifier was used for recognition in this problem. Nearest neighbour classi-
fiers work best when a problem is linearly separable. It is clear that when using edge length histograms
as a representation of each class, the problem is not linearly separable. A support vector machine could
be used to try and classify with the current representations. They work by projecting the data of each
class into an n-dimensional space and trying to find a hyperplane that separates them. A support vector
machine could possibly improve accuracy results whilst still using the current representation.
42
5.6 Conclusion
Virtual Pathology is an exciting field that has the potential to introduce huge benefits for both patients
and pathologists. This project has shown that there is potential for an automated system to be able
to distinguish between cancer and Barrett’s Oesophagus. It has also been shown that it is particularly
difficult to detect the in between stages of dysplasia.
The most success full aspect of this project is the novel approach taken to locating the centre of
nuclei. This resulted in an excellent nucleus detection algorithm. It confirmed that the exact shape or
position of a nucleus is not needed to be known to approximate its centre.
For the project to be taken further, new features that distinguish between the classes need to be
explored.
43
Bibliography
[1] Alexander Wright. The Automated Counting of Cancer Cells Using Computer Vision. Technical
report, School of Computing, Leeds University, 2007. Final year project.
[2] Aperio Image Analysis Overview, 2008. http://www.aperio.com/
productsservices/prod-imageanalysis.asp [5th January 2008].
[3] Hugh Barr. High-grade dysplasia in barretts oesophagus: The case against oesophageal resection.
2006. Published by the Royal College of the Surgeons of England in the series “controversial
topics in surgery”.
[4] Barrett’s Oesophagus : Cancerbackup, 2007. http://www.cancerbackup.
org.uk/Cancertype/Gulletoesophagus/Pre-cancerousconditions/
Barrettsoesophagus [15th December 2007].
[5] Adrian Bowyer and John Woodwark. Introduction to computing with geometry. Winchester, 1993.
[6] Alison Carling. Introducing Neural Networks. Sigma Press, 1992.
[7] Cigdem Demir and Bulent Yener. Automated cancer diagnosis based on histopathological images:
a systematic survey. Technical report, Department of Computer Science ,Rensselaer Polytechnic
Institute, 2005.
[8] Darren Treanor. Advice for people doing research with virtual pathology slides. Technical report,
School of Computing, Leeds University, 2007.
[9] Richard O. Duda, Peter E. Hart, and David G. Stork. Pattern Classification. Wiley-Interscience,
second edition, 2001.
44
[10] B. J. REID et al. Observer variation in the diagnosis of dysplasia in barrett’s esophagus. Human
Pathology, 19:166–178, 1988.
[11] PW Hamilton et al. Automated location of dysplastic fields in colorectal histology using image
texture anaylsis. Journal of Pathology, 182:68–75, 1997.
[12] Timoth W. Freer and Michael J. Ulissey. Screening mammography with computer-aided detection:
Prospective study of 12,860 patients in a community breast center. Radiology, 220:781 – 786, Sep
2001.
[13] Gabriella Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski and Cdric Bray. Visual
Categorization with Bags of Keypoints. Technical report, Xerox Research Centre Europe, 2004.
[14] K Geboes and P Van Eyken. The diagnosis of dysplasia and malignancy in barrett’s oesophagus.
Histopathology, 37:99–107, 2000.
[15] Rodger C. Haggitt. Barrett’s esophagus, dysplasia, and adenocarcinoma. Human Pathology,
25:982–993, 1994. Issue 10.
[16] Donald Hearn and M. Pauline Baker. Computer Graphics with OpenGL. Prentice Hall, third
edition, 2004.
[17] Mary Jones and Geoff Jones. Human Biology for AS Level. Cambridge University Press, first
edition, 2004.
[18] Stephen J. Keenan, James Diamond, W. Glenn McCluggage, Hoshang Bharucha, Deborah Thomp-
son, Peter H. Bartels, and Peter W. Hamilton. An automated machine vision system for the his-
tological grading of cervical intraepithelial neoplasia (cin). Journal of Pathology, 192:351–362,
2000.
[19] Abraham L. Kierszenbaum. Histology and Cell Biology: An Introduction to Pathology. Mosby,
second edition, 2007.
[20] M Lundin, J Lundin, and J Isola. Virtual microscopy. Journal of Clinical Pathology, 57:1250–
1251, 2004.
[21] Metis Manual, 2008. http://glaros.dtc.umn.edu/gkhome/fetch/sw/metis/
manual.pdf [12th February 2008].
45
[22] Milan Sonka, Vaclav Hlavac and Roger Boyle. Image Processing, Analysis, and Machine Vision.
PWS publishing, second edition, 1999.
[23] Stanislaw Osowski and Do Dinh Nghia. Fourier and wavelet descriptors for shape recognition
using neural networksa comparative study. Pattern Recognition, 35:1949–1957, 2002.
[24] Stuart Russell and Peter Norvig. Artificial Intelligence A Modern Approach. Prentice Hall, second
edition, 2003.
[25] Edmond Sabo, Andrew H Beck, Elizabeth A Montgomery, Baishali Bhattacharya, Patricia Meit-
ner, Ji Yi Wang, and Murray B Resnick. Computerized morphometry as an aid in determining the
grade of dysplasia and progression to adenocarcinoma in barretts esophagus. Laboratory Investi-
gation, 86:12611271, 2006.
[26] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TRANSACTIONS
ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 22:888–905, 2000. No. 8.
[27] Elif Derya Ubeyli. Implementing automated diagnostic systems for breast cancer detection. Expert
Systems with Applications, 33:1054–1062, 2007. Issue 4.
[28] University of Warick and Pakistan Institute of Engineering and Applied Sciences. Detection
of Nuclei by Unsupervised Manifold Learning. Medical Image Understanding and Analysis
(MIUA’2007), 2007.
[29] Aperio ImageScope Users Guide, 2008. www.aperio.com/documents/rel8/Aperio
ImageScope Quickstart Guide.pdf [17th January 2008].
[30] Johanna W. van Sandick et al. Computerized quantitative pathology for the grading of dysplasia
in surveillance biopsies of barrett’s oesophagus. Journal of Pathology, 190:177–183, 2000.
46
Appendix A
I have thoroughly enjoyed working on this problem and feel that I have gained a great deal of confidence
in my own analytical and programming skills. I knew it was going to be a difficult problem to solve as
Dr. Treanor had made this clear in the weekly meetings. Working in an area such as pathology which
I was originally unfamiliar with has given me very valuable experience and I now have the confidence
and skills to break down and tackle problems in areas unknown to me. I am particularly pleased with
the success of the nucleus detection algorithm. With a little fine tuning it is something that could be a
very useful tool for pathologists and for use in future projects and research.
One of the most valuable lessons that I have taken from this project is the importance of time
management. With other coursework deadlines and a busy extra curricular callender it was extremely
important that I planned my time well and tried to stick to internal deadlines that I had set myself. Whilst
i did stick to my deadlines and worked to schedule I feel that the orignal plan that I drew up was not
adequate. I planned to design and implement a solution to the problem between the dates of 21/01/08
and 18/02/08. In hindsight I feel that more effort should have been put into the design of the solution
in the first semester. I would encourage any future student to start experimenting with different designs
early on and to try and implement a proof of architecture before the christmas holidays. This gives
plenty of time for any improvements that need to be made to the solution.
To students that are attempting a computer vision project, I would suggest that they try and under-
stand the basic image processing concepts and techniques early on in the project. It is also important
to start experimenting with these methods early on. Whilst I was a confident programmer at the be-
ginning of the project and was comfortable with Matlab, I feel that I didn’t do enough to experiment
47
with the many different types of vision techniques. With vision problems there are often many ways of
approaching a problem. If you are able to learn the skills early on, you will have the time to try different
approaches to the problem and select the best one.
During the second year I spoke to a finalist who advised me to choose my third year modules so that
I had 50 credit to 30 credit split for tought modules. This seemed like sound advise as it would give me
plenty of time in the second semester to compile my project report, which seemed the most sizeable task
that needed to be completed for the project. However, in hindsight I felt that this was a bad decision.
I found the workload unbearable in the first semester as I had chosen modules with large courseworks
to be completed. This also meant that I fell slightly behind in the project. Even though I had more
time to work on the project in the second semester, I don’t feel like I benefited from it. I would advise
future students that the final year project needs their full attention accross the whole year. Even though
theoretically the same amount of time should have been spent on the project as if I had chosen a 40/40
credit split, I feel that my project would have benefited if I had reached a deeper level of understanding
at an earlier stage. For this reason I would advise future students to give their project the full attention
it deserves from the outset of semester one.
I was lucky enough to have the guidance of a pathologist, Dr. Darren Treanor. I was often in contact
with Darren and was able to ask him questions to help me understand the medical condition I was
working on. The main correspondance I had with Darren was via email and also in weekly meetings.
When contacting Darren I always maintained a polite and professional manner. I feel that taking this
attitude when in contact with third parties is extremely important as students behaviour reflects on the
School of Computing. When evaluating my project I sent Darren the output images of my nucleur
detection algorithm with the nuclei marked with a green cross.This was so that he could score the
accuracy of the algorithm. When asking Darren to complete this task, I felt it was neccessary to respect
his time and the fact that he was likely to be very busy. For this reason I made sure I gave them to
Darren in good time so that he had the opportunity to plan it into his own schedule. Luckily Darren was
extremely helpful and reliable, so it was never a worry that I would not recieve his feedback in time.
However, other students cannot assume that other third parties will be so helpful and reliable. Most third
parties are likely to be busy with their own working lives and it is important that students are respectful
of this. Students should make sure they plan to give third parties plenty of time to give feedback.
48
Appendix B
49
Milestone Start End Description
19/10/2007 26/10/07 Research histopathology and the use of vir-
tual slides
26/10/2007 02/11/07 Research Barrett’s Oesophagus condition
1 02/11/2007 30/11/07 Investigate computer vision techniques that
could be applied to the problem
30/11/07 05/12/07 Draft mid-project report
06/12/07 07/12/07 Final mid-project report
07/12/07 06/01/08 Revise for semester one examinations
07/01/08 18/01/08 Semester one exam period
2 21/01/08 18/02/08 Design and Implement solution to problem
18/02/08 03/03/08 Revise solution to problem
03/03/08 18/04/08 Compile project report
07/03/08 Submit table of contents and draft chapter
3 07/03/08 21/03/08 Analysis of solution and comparison with
pathologists results
14/03/08 Demonstration of progress meeting
4 21/03/08 04/04/08 Scientific evaluation of success of solution
18/04/08 22/04/08 Proof read report
23/04/08 Submit final report
Table 5.2: A table showing the schedule of the project.
50