a machine learning-based approach for fault detection of railway track...
TRANSCRIPT
A Machine Learning-Based Approach for
Fault Detection of Railway Track and its
Components
Johnny Asber
Maintenance Engineering, master's level (120 credits)
2020
Luleå University of Technology
Department of Civil, Environmental and Natural Resources Engineering
1
Acknowledgments
The last six months will always be remembered as the time when I learned how to
search for suitable industrial solutions as well as how to conduct academic research
in the right way.
First of all, I would like to thank Prof. Uday Kumar, the head of the Division of
Operation and Maintenance, for giving me this opportunity to join this division for my
master's program. I would also like to express my sincere gratitude to Prof. Matti
Rantatalo, Examiner, for this master's thesis. Without his guidance and persistent
help, this work would not have been possible at all. Huge thanks to Johan Odelius,
my supervisor and program coordinator, for his valuable supervision, keen interest,
and inspiration at various stages of my thesis period. My sincere thanks to my other
supervisor, Praneeth Chandran, PhD student in the Division of Operation and
Maintenance; without him, my project would lack much direction. Many thanks to
Bombardier Transport for supporting the project and RailDoc for providing the images
used in this thesis.
I want to extend my deepest gratitude to all of the professors, teachers, and PhD
students that I met during my two-year master's program in Maintenance
Engineering. The knowledge you provided me has enriched my experience and
equipped me for sure with excellent skills to face the upcoming future industrial
challenges.
Each person in the Division of Operation and Maintenance has helped or encouraged
me to do my best in different ways. It was great to be part of such a great group. I
would also like to thank Laila Moussallik, a classmate, as well as an opponent for my
master's thesis. Thank you for sharing this journey!
Last but not least, I would like to express my appreciation for the support and great
love from my family for making me the person I am. They kept me going, and this
work would not have been possible without their support. Endless thanks to the
person who has always supported and guided me throughout my life’s journey, Uncle
Kheder Mekha.
I hope you enjoy reading this thesis. In case of comments or questions, please feel
free to contact [email protected]
2
Table of Content Acknowledgments ...................................................................................................................... 1
Abstract ...................................................................................................................................... 4
1 Introduction.......................................................................................................................... 5
1.1 Research Gap .................................................................................................................. 5
1.2 Research Problem............................................................................................................ 6
1.3 Research Objectives ........................................................................................................ 6
1.4 Research Questions ......................................................................................................... 7
1.5 Limitations ........................................................................................................................ 7
2 Theoretical Framework ........................................................................................................ 8
2.1 Condition Based Maintenance ......................................................................................... 8
2.2 Condition Based Maintenance in Railway ....................................................................... 8
2.3 The Importance of Rrailway Inspection ........................................................................... 9
2.4 Machine Learning ........................................................................................................... 12
2.5 Machine Learning Techniques ....................................................................................... 12
2.6 Selecting The Appropriate Algorithm ............................................................................. 13
2.7 Most Popular Image Classifiers ..................................................................................... 14
2.8 Transfer Learning ........................................................................................................... 15
2.9 The Used Different Techniques of Automated Fasteners Detection............................. 16
3 Method................................................................................................................................ 19
3.1 Stage One: Images Processing (All Rail Components)................................................. 20
3.1.1 Raw Images ......................................................................................................... 20
3.1.2 Image Merging ..................................................................................................... 21
3.1.3 Sleepers Positioning ............................................................................................ 21
3.1.4 New Framing........................................................................................................ 22
3.1.5 Rail Positioning .................................................................................................... 23
3.1.6 Image Segmentation............................................................................................ 23
2.2 Stage Two: Labelling (Processing Only The Fasteners) ............................................... 25
2.2.1 Fasteners labelling............................................................................................... 25
2.3 Stage Three: Machine Learning (Processing Only The Fasteners) .............................. 27
2.3.1 Machine Learning (ResNet-50) ........................................................................... 27
2.3.2 Training and Testing ............................................................................................ 28
2.3.3 Image Augmentation............................................................................................ 28
2.3.4 Validation ............................................................................................................. 29
4 Results and Discussions .................................................................................................... 30
3
4.1 Group 1 ........................................................................................................................... 30
4.2 Group 2 ........................................................................................................................... 33
5 Model Validation and Final Decision ................................................................................. 36
5.1 Noise Test ...................................................................................................................... 36
5.2 Illumination Test ............................................................................................................. 36
5.3 Speed Test ..................................................................................................................... 37
6 Conclusion.......................................................................................................................... 38
7 Future Work........................................................................................................................ 39
8 References ......................................................................................................................... 40
4
Abstract
The hard equation of railway safety versus the high commercial profits can only be achieved through the use of new inspection methods supported by modern
technologies. The track and its components can have different types of troubles, such as rail surface defects, broken sleepers, missing fasteners, and irregular ballast
levels. Each component of the track infrastructure plays a significant role, where the failure or the absence of any of them can pave the way to undesired situations. The rail is designed to carry and direct the train, the sleepers are meant to maintain the
level of the rail, and the ballast mission is to keep all components floating on the surface of the ground. The fasteners are used to fasten the rail to the sleepers, and
therefore too many missing fasteners can lead to sever unsteady tracks, which can, in turn, result in derailment. Therefore, there is a high demand for advanced inspection methods to monitor the railway track and its components continuously.
The presence of such advanced inspection models would help the railway industry avoid obstacles such as high operation and maintenance costs, dangerous
accidents, and uncomfortable passenger's experience. This master thesis aims to present an efficient method to classify the track and its
components by combining image processing techniques and deep learning algorithms. This method was able to detect the missing fasteners in the set of images
captured by a line camera, continuously monitoring the rail and its associated fasteners. The experimental results obtained in this thesis showed that the proposed method is efficient and robust for detecting the track and its components in complex
environments. The thesis also discusses the idea of building one complete model that can process and classify all track components at once. The image processing
technique was employed to extract different components of the track, individually: fasteners, rail, ballast, and sleepers. The model was trained and used to classify the state of the fasteners. Classification of other components of the track will be a part of
the future work.
5
1 Introduction
In a competitive and fast-growing global market, railway companies face many difficulties in staying productive. To obtain a high level of productivity, companies
need to have maximum availability of its assets, and maintenance is required to ensure maximum availability. Therefore, maintenance is essential for any industry,
but it remains a challenge that slows the pace of progress for all ambitious companies. Billions of euros are spent on maintenance each year to achieve the required goals, and the numbers are still getting higher every year [1]. Maintenance
actions occurrence would always increase with time, as the meantime to failure becomes less and less due to the natural degradation of materials [2].
Railway companies are seeking to improve passenger satisfaction and comfort by
reducing train delay times and increasing safety. Passenger delays are mainly
caused by railway track issues such as maintenance activities and reconstructions of
old networks [3]. Safety is strongly affected by manual inspection, where lack of
human concentration can lead to massive disasters. Also, the high costs of
maintenance and inspection activities are a significant concern for the rail industry.
Therefore, decreasing time for inspection activities and covering the weaknesses in
human capabilities could be one of the best solutions to achieve the current goals of
the railway companies.
Recently, the need for high-speed railways has prompted all companies to improve
their inspection methods to meet unprecedented challenges [4]. Therefore, the
automated detection method based on the use of machine vision has become the
main interest of all companies [5][6]. Many researchers worked on this area, and
many ideas have been implemented during the past few years. Each new method
covers some weaknesses that had not been discussed in previous researches. The
visual inspections of railway components have dominated for many past years, but
the detection techniques have varied over time [7]. However, there are many as yet
unresolved challenges in the previously published models, such as detection in
different illumination conditions, detection time, and detection of all components of
the track in one model.
This thesis proposes one model that can simultaneously detect faults in the track and
its components. By following this method, there will be an opportunity for
maintenance personnel to implement multi-tasking maintenance procedures and
bring in necessary tools and spare parts as well.
1.1 Research Gap
For automated visual inspections of track components, there are still several
problems to solve. Based on the review of previous research, the two main
challenges are
6
1- Dust, surface erosion, rusting, snow, stones, and heavy rain are considered a
tough challenge for all published methods, including the neural networks.
2- Need for optimum illumination conditions to obtain excellent detection results.
Some researchers are trying to overcome these issues by heading towards non-
destructive methods such as eddy current techniques, which seems a promising
solution. However, the visual detection methods are considered the closest to the
human performance attitude, and this area is improving continuously.
The latest published papers [8] that had used the Convolutional Neural Networks
were showing positive results to an extent. However, none of these papers were
considering having a single model that can classify all track components
simultaneously. Further, none of the researchers had previously employed the
ResNet-50 in detecting the track and its components to verify how the residuals
approach performs in the detection of the track and its components.
1.2 Research Problem
Inspection and monitoring of railway track and its components are critical to ensure
railway safety. Using a fast, easy, and cheap monitoring system not only reduces
inspection costs, but also increases the safety and reliability of track and its main
components. Today most of the inspection time is spent on components such as
weld joints, rail surfaces, switches, crossings, and fastening systems [9]. Automated
monitoring systems for track and its components, can provide the information needed
to carry out the right maintenance procedures at the right time.
1.3 Research Objectives
This thesis aims to develop an automated system for detecting faults in the track and
its components from images acquired during visual inspection. Current
methodologies for inspecting the track and its components and identifying the
limitations associated with the same are also discussed briefly in this thesis. Having
an efficient model that can detect all the different parts in the acquired images during
the visual inspection can enhance the detection time as well as help in organizing the
maintenance activities each day. This thesis would make use of Image processing
techniques and machine learning algorithms to facilitate real-time detection.
The suggested model in this thesis has an input of row images, which have
dimensions of (2048 x 2000 x 3). The final results would be small cropped images
classified into Healthy fasteners, Missing Fastener right, Missing Fastener Left,
Missing Fasteners, Healthy Rail, Damaged Rail, Healthy Sleeper, Bad ballast, and
Good ballast.
7
1.4 Research Questions
Looking over the past researches in the field of the automated visual inspection, the
research questions considered in this thesis were
1- Is it possible and practical to identify rail and track components by using image
processing techniques?
2- How does the ResNet-50 performs in detecting faults in track and its components.
1.5 Limitations
This thesis work had some limitations, where for example we did not look into each of
the following
1- Look into all possible optimization techniques of the ResNet-50.
2- Label and train the other track components that were cropped into small,
separated images in the first stage of this thesis.
3- Check the performance of other neural networks.
4- Train the model to detect different types of fasteners, where we were just able to
do image processing and labelling for one type of fasteners due to lack of time.
8
2 Theoretical Framework
2.1 Condition Based Maintenance
Preventive maintenance can be divided into predetermined maintenance PM and
condition based maintenance CBM. CBM build its decision on the actual stat of the
equipment, where the PM based its ones on pre-set schedules for maintenance,
which had set by some historical data for similar equipment [4]. For example,
changing oil filter in an engine in fixed run time intervals based on historical data for
similar filters. Instead with CBM one can use the filter until its whole life length. In this
example, CBM builds its decision by collecting the information from the oil flow
sensor, which would periodically keep checking and recording the flow condition
before and after the filter. By using this technique, the oil filter would be used for
more extended periods. Thus, this approach will increase the engine availability and
reduce maintenance costs.
Condition-based maintenance CBM can be described as a maintenance program
that can help to build decisions based upon the obtained information that is collected
utilizing condition monitoring tools or techniques [4]. CBM decisions led to
rescheduling the unnecessary maintenance actions by the ones that are needed.
Condition-based maintenance CBM can provide all needed to enhance maintenance
actions, procedures, safety, and strategies [4]. Acceptance of communication
procedures, upgraded knowledge of failure mechanisms, progresses in monitoring
and sensor devices, developments in failure forecasting techniques, improvements in
diagnostic and prognostic software, developments in maintenance software use, and
all different computer networking tools were the reason for the tremendous
improvements in the area of the CBM [10]. CBM is a technology that allows for more
accurate planning.
2.2 Condition Based Maintenance in Railway
Maintenance plays a significant role in determining the life of the track and all its
various components. The railway faces difficult tamping challenges that require
proper maintenance actions such as grinding and tapping. One example is squats, a
general category of Rolling Contact Fatigue RCF that accelerates the rail degradation
process, which can lead to derailment if not handled correctly [11]. Squat can be
treated effectively by grinding. However, grinding is considered as a viable option for
the cases that are at the initial stages. A complete replacement of the rail is only
performed when the squats have developed into a critical stage. Ballast degradation
led to unsteady sleepers, rail buckling, and derailment [12].
In the past few decades, many companies across Europe have shifted their focus
from using reactive solutions to proactive actions [13]. Thus, CBM began to be the
9
center of the interests for many industries; CBM is a proactive maintenance program,
helps in creating a decision model that uses the collected information for an asset
condition.
CBM is widely used in aviation and automobile industries. There is potential to
increase the use of these powerful methods in the railway industry. [14]. Advanced
monitoring techniques can provide all the necessary solutions for the railway
industry. CBM increase not only the safety level but also increase the availability and
reliability [14].
2.3 The Importance of Railway Inspection
A reliable and profitable inspection method of rail tracks is vital for ensuring the
safety operations [15]. Railway companies perform regular inspection actions on their
tracks to keep them in good condition, which ensures a high safety level and well-
organized operation. Besides, railway companies have to do these inspection tests
regularly to follow the mandatory procedures and regulations that are required from
the Federal Railroad Administration FRA. Even though these procedures result in
substantial annual operating expenses and have various other limitations such as
quality, speed, objectivity, and scope, they are very critical and should be done
correctly [16].
When it comes to mentioning the most inspected parts, one should first mention the
most exposed parts to be damaged. For example, the broken rails are considered as
the leading cause of the freight-train derailments [17]. Thus, the inspections of the
rails to detect the defects or the missing components that can cause the broken rails
are usually the most inspected parts in the rail industry.
Today most of the inspection times are spent on components such as weld joints, rail
surfaces, switches, crossings, and fastening systems [9].
10
Over the last few decades, all inspection actions were carried out manually, where
experts have to cross by the truck and its components to check their conditions and
record some necessary information. Nevertheless, the most used methods for
inspection in the rail industry of today are those that are carried out by using the
ultrasonic testing (UT) instruments. The location of the defects is mainly being done
by the test trains and hi-rail vehicles. The process of the verification and sizing steps
are then performed by suitable personnel by using hand-held devices. These
procedures are, to some extent, satisfied.
Nevertheless, this manual UT inspection is considered by many companies as a
labor-intensive and time-consuming method that needs experienced workers with
confirmed talent to understand the acquired signals. Also, the manual UT inspection
has a limitation when it comes to decide the proper time for the inspection, where it
only can be used in the night time or maintenance time due to the traffic conditions.
Therefore, we can say that the manual UT inspection cannot be considered as a
cost-effective method, and it also requires significant skilled human resources to
implement the needed actions.
Figure 1: Rail Components [18]
Figure 2a: NDT method Figure 2b: Manual Inspection
11
Lately, a machine vision method is being established to automate the inspection
process, but only for specific components that belong to the track structure [19]. The
machine vision method is considered as a useful tool for a speedy inspection
method. The machine vision system comprises a video acquisition system, which is
intended to record digital images of the track and its components, and by using a
designed algorithm, these images being processed to detect defects and
symptomatic conditions [19].
Figure 3: Automated Detection [20]
12
2.4 Machine Learning
According to the latest definitions, machine learning algorithms use computational
methods to learn information directly from the data without relying on a pre-defined
equation as a model [21]. Algorithms adaptively increase their performance while
increasing the number of samples there is for learning [21]. Machine learning
contains two types of techniques, supervised learning and unsupervised learning.
[22] The most common challenges to machine learning are the following
Datasets can be incomplete, in a variety of formats, and chaotic
The need for special knowledge and tools to do data pre-processing
Choosing the right model can take much time
2.5 Machine Learning Techniques
Supervised machine learning is used to build a model that makes predictions based
on evidence in the existence of uncertainty [21]. Typically, supervised learning
algorithms take a known data set with known responses in the output. These
responses are then used to train a model to generate predictions for response to the
new dataset. Moreover, the supervised learning technique develops predictive
models by using regression and classification techniques.
Classification models classify input data into categories
Regression techniques predict continuous responses
Unsupervised learning works in such a way that it finds hidden patterns in data. It is
used to extract conclusions from the dataset containing unlabelled input data. The
most common unsupervised learning technique is the clustering technique [21].
Clustering is the task of grouping a data set into groups of similar objects
Figure 4: Machine learning techniques [21]
13
2.6 Selecting The Appropriate Algorithm
There are several algorithms in machine learning that are grouped as unsupervised
and supervised algorithms. Each one of these algorithms takes a different learning
approach. There is no best technique or one size that can fit all types of different data
sets [21]. Thus, in order to find the correct algorithm, which fits the data set being
used, a series of trial and error steps should be performed. Nevertheless, choosing
the correct algorithm also depends on all of the following [21]
The type of data set used
The size of data set used
Insights needed to be extracted from the data set used
How those extracted insights will be used
The following Figure 5 shows different machine learning techniques that can be
considered in the process of selecting the right algorithm that can fit the data set
used.
Figure 5: Select algorithm Process [21]
14
2.7 Most Popular Image Classifiers
The task of extracting classes of information from a multi-scale bitmap is called
Image Classification [23]. Raster data generated from image classification can be
used to create thematic maps [23]. Image classifiers are usually added to the top of
feature extractors in order to classify them. The most common image classifiers are
Support Vector Machines, Decision Trees, Artificial neural networks, and
Convolutional neural networks.
Support Vector Machines (SVM): Support Vector Machines is a supervised
machine learning method that is used for both classification and regression problems
[22]. SVM uses the linear boundary method to perform classification. The function of
the kernel is the reason why these types of classifiers work efficiently. The most
commonly used kernels are Linear Kernel, Gaussian Kernel and Polynomial Kernel.
Decision Trees: Decision Trees are also a supervised machine learning technique.
This classifier uses if/else statements on the features selected [23].
K Nearest Neighbor: This algorithm is considered one of the most straightforward
[23]. This algorithm depends on the distance between new data points and feature
vectors by finding the most common class.
Artificial Neural Networks (ANN): [24] Artificial neural networks are computational
algorithms that operate like that of the human nervous system [23]. Artificial Neural
Networks are implemented as a system of connected processing elements, which
are called nodes. These nodes mimic the biological neurons of the human brain. The
links between every two nodes have values called weights, and by logically changing
these values, Artificial Neural Networks are fully capable of approximating the wanted
function. The hidden layers in the ANN learn separately as single feature detectors.
[24] By taking this approach, hidden layers will be able to recognize challenging
patterns in the data set as they are propagated across the network. Today, there are
many types of artificial neural networks. These types of networks are implemented
using mathematical tasks and a set of parameters needed to define the output [25].
The most common artificial neural networks are Feedforward Neural Network –
Artificial Neuron, Radial basis function Neural Network (RBF), Kohonen Self
Organizing Neural Network, Recurrent Neural Network (RNN) – Long Short Term
Memory, Convolutional Neural Network, Modular Neural Network [26].
Convolutional Neural Networks (CNN): Convolutional neural networks are a
particular architecture case of ANN. Convolutional neural networks are consist of two
simple components, namely the pooling layers and the convolutional layers CNNs
perform very well in machine vision tasks [23].
15
2.8 Transfer Learning
Transfer learning aims to enhance learning in the target task by leveraging
information from the source task [27]. Also to decrease the total time required to train
a new model. Therefore, this approach can be seen as the one that paved the way
for solving very complex problems. [27] There are three clues that transfer learning
may improve the learning process. The following Figure 6 shows how the transfer
learning offers the following
1) Higher model performance, through the use of transferred knowledge.
2) The learning curve is sharper with transfer learning (learning faster)
3) Over time, higher model performance can be obtained
Transfer learning is classified into transductive transfer learning, inductive transfer
learning, and unsupervised transfer learning [28]. Transfer learning can be beneficial
in situations where there is little data available. Further, using transfer learning with
extensive data set can increase model performance [29].
Figure 6: Transfer Learning Performance [27]
16
2.9 The Used Different Techniques of Automated Fasteners
Detection
On May 16th, 2000, after the approval of the patent of Trosino by the National
Railroad Passenger Corporation, Washington, DC, the machine vision method had
adopted by the railway companies as a new technology that can be used for
inspecting the different types of trucks [30][31]. The invented automated technique
by Trosino was able to collect and store the images of rail for later review "offline
analysis technique". The applications of the machine vision method increased safety
and reduced operation and maintenance costs compared to traditional methods.
[32][33].
In 2006 Singh used the edge density to identify the exact location of the clips and
detect the missing fasteners clips, then detect the recently replaced blue clips based
on the color information that existed in the positioned area [5].
In 2007, Marino introduced the VISyR system. The VISyR system was a fully
automatic and configurable FPGA-based vision system designed for real-time track
inspection. The system was able to classify the fasteners that have the bolts type and
analyze track defects. The VISyR system was effectively able to identify the missing
fasteners [34][35]. In 2009 Babenko from the University of Central Florida used an
image-based detection device containing two industrial laser range scanners
mounted on the train, one for each rail, to detect the missing or the defective
fasteners [36]. This method has used a separate filter (OT-MACH) for each type of
available fasteners in the industry [37].
Then in the same year (2009), De Ruvo initiates a solution by finding the desired
fastener window based on the prior geometric relationship of the rail and its
components [38]. De Ruvo and Marino also found a way to detect the missing
fasteners by using a multilayer perceptron neural network classifier based on wavelet
function [38][39].
Mazzeo and Stella detected the missing fasteners with a back-propagation neural
network and a radial basis function neural network classifier based on wavelets and
principal analyst of the component [40][41]. Xia, in 2010 worked on dividing each
fastener into four main parts. He trained each part of the divided fasteners
individually by using AdaBoost based on Haar-like features [32]. In 2015, Liu built an
algorithm based on a sparse demonstration to identify two symmetrical fasteners by
using a pyramid histogram of the oriented gradients [42].
Yang, in 2011, extracted the directional field for fasteners as a feature, where he
used that extracted feature to detect the missing fasteners through a comparison
process with the weighted template [43]. Further, Li was able to trace the fastener
region with the help of Hough transform, and then detected all the missing fasteners
by using edge features [44].
17
In 2014, Feng trace the fastener region based on the geometric relationship between
the sleeper, the fastener, and the rail. By using the technique of a probabilistic topic
model based on Haar-like features, he was able to classify the fasteners into healthy,
partially missing, and completely missing fasteners [45]. In 2016 Liu Jiajia, Xiong
Ying, Li Bailin, and Li present an image classification and recognition method based
on image fusion feature and Bayesian compressive sensing (BCS) [46].
In 2017, Manikandan, Balasubramanian, and Palanivel published a new technique
based on machine vision. This method could detect and classify the missing
fasteners by using the Support Vector Machine (SVM) classifier. The published new
system consists of pre-processing, transformation, feature extraction, and
classifications. The resizing of the images was implemented during the pre-
processing stage. Also, the Gabor transform was used as a transformation technique
in the new proposed system. The system makes uses of many different types of
features such as Local Binary Pattern (LBP), Grey Level Co-occurrence Matrix
(GLCM), and Discrete Wavelet Transform (DWT). Support Vector Machine (SVM)
was used as a classifier in the proposed method to categorize the images into either
fastener or missing fasteners [47].
In 2018 Bailin Li solved the problem of different levels of illumination and conditions
that led to ineffective automated visual detection of the fasteners. The proposed new
technique uses the line local binary pattern encoding, which considers the bond
between the middle point and its higher and lower neighborhoods [48].
The most commonly used techniques for modeling and detecting fasteners between
2004 and 2014 were each of the following: Gabor filters [49], edge-detection methods
[35], and support vector machines (SVMs) [50]. Also, conventional generative
models were used for the same purposes, which include structure topic models
(STMs) and latent Dirichlet allocation (LDA) [51].
In 2019, an advanced approach was presented by Xiukun Wei, Ziming Yang, Yuxin Liu, Dehua Wei, Limin Jia, and Yujie Li. This new approach incorporated image
processing tools and deep learning networks. They used the Dense-SIFT features, spatial pyramid decomposition, and BOVW techniques that are relies on interest
point detection in order to achieve better performance than any time before. After that, they trained the VGG16 for fastener detection and classification. This method revealed that the time for fasteners detection and classification is only taking one-
tenth of the other available methods in the market with a classification accuracy of 99.26%. Also, they performed Faster Region-based Convolutional Neural Networks
(R-CNN) to improve the detection rate and consume less time for each image. Finally, this method, at the time, was the first one that introduced the deep convolutional neural network (DCNN) for fastener defect detection and classification.
[8].
Junbo Liu, Yaping Huang, Qi Zou, Mei Tian, Shengchun Wang, Xinxin Zhao, Peng Dai, and Shengwei Ren solves the issue of the imbalanced data by using a novel vision-based fastener detection system (VFIS) that is inspired by few-shot learning.
18
This new method can collect a massive number of fastener images by using the suggested online template matching-based sorting technique. This new method also works on a minimal number of fastener templates. Furthermore, they built a deep
network based upon the similarity to solve the issue of the imbalanced dataset [52].
19
3 Method
The practical part of this research is divided into four main stages, where each stage
has a different code. Each stage comprises of a series of steps depending on its
primary purpose. The first stage includes Image Processing techniques, while the
second stage was to facilitate our Labelling task. The third stage is about using the
pre-trained neural network ResNet-50 to do the classification mission. Tests and
validation were implemented in stage four in order to evaluate the efficiency of the
model and validate its ability to deal with different real-life scenarios. The detailed
steps are shown in the figure below Figure 7.
Figure 7: Flowchart that describes the method of the thesis
20
3.1 Stage One: Images Processing (All Rail Components)
3.1.1 Raw Images
The dataset used in this thesis consists of 19000 images, which are separated into
12 files, and contains the following challenges.
Many images with one and a half sleeper
Different illumination levels
Very few Missing fasteners
Figure 8: Raw images
21
3.1.2 Image Merging
The dataset used in this thesis has images of size (2048 * 2000 * 3). The only
problem is that many of the images contain one and a half sleeper in one image or
two sleepers in one image, which increase the error in detection in the other stages
Figure 8. Thus, all images were merged into one big longitudinal image in order to
produce new framed images Figure 9. The new framed images have the following
characteristics
Each image has the same original size as the raw images
Each image contains only one sleeper
3.1.3 Sleepers Positioning
This step aims to produce new framed images that contain only one sleeper in the
center of the image. The used techniques for this purpose was by applying a series
of filtering steps and find peaks function on the filtered big longitudinal image so that
all sleepers could be located. An example of some of the original images after the
filtering stage is shown in Figure 10. First, a long line of pixels was captured from the
big filtered image, with the ones and zeros represent the white and black areas,
respectively. Second, the sum of all pixels was taken to enhance the clarity of the
peaks. The detected peaks, using find peaks and smooth functions, represent the
higher peak of each sleeper, as shown in Figure 10.
Figure 9: Four Images that are merged into one big Image
22
Thus, in order to extract the sleeper's center, there was a need to perform
mathematical functions in both positions of the peaks and width of the peaks.
3.1.4 New Framing
After collecting the positions of the peaks of both rail and sleepers, the cropping
function was implemented using these peaks positions as the row numbers (rows
parameters). The final cropping was done after adding the required dimensions for
the new framing. The new framed images have the same original image size, Figure
11. All new framed images have a single sleeper in the center of the image.
Figure 10: Sleepers detection
23
3.1.5 Rail Positioning
To generate small images that represent each component separately, another unique
technique was needed to provide more accurate image cropping (columns
parameters). The used technique here was the same as that used before in the
process of finding sleepers positioning. The only adjustment here is that the
coordinates have changed. Next, the center of the rail was determined by finding the
peak from a single-pixel line from the big filtered longitudinal image. Moreover, the
sum of all pixels was taken to enhance the clarity of the peaks. The detected position
of the peak is representing, in this case, the center of the rail.
3.1.6 Image Segmentation
After the new images frame, sleeper positioning, and rail positioning, every newly
framed image was further cropped into smaller images. Image cropping coordinates
were determined for each component using the center of the sleepers and the center
of the rail (rows and columns parameters). Each small image represented a specific
component: Fasteners, Rail, Sleeper Left Side, Sleeper Right Side, Upper Right
Ballast, Upper Left Ballast, Bottom Right Ballast, and Bottom Left Ballast.
Figure 11: New Framing Process
24
Also, image processing techniques have been performed on another type of fastener.
Regarding the short time for this thesis, the Machine Learning stage for this type of
fasteners has been canceled.
Figure 13: Other type of Fasteners – Fast Clip
Figure 12: Image Segmentation Process
25
2.2 Stage Two: Labelling (Processing Only The Fasteners)
2.2.1 Fasteners Labelling
Labelling is a process that requires a lot of repetitive actions, which can result in a lot
of time loss. Therefore using a unique technique to implement the labelling process
can reduce the time required to inspect thousands of images. The technique used is
a series of threshold numbers that take into account the difference in the dark around
the fasteners clamp. The darker the area of the clamps, the more likely the fasteners
will be present in Figure 14. The results obtained were cropped fasteners images that
are classified into Healthy Fasteners and Missing Fasteners in two separate files.
This technique only had one problem, because some images classified as missing
fasteners were healthy fasteners. Hence, there was a need to move these images to
the Healthy Fasteners file manually.
Figure 14: Fasteners Labelling
26
The code consists of four layers of fasteners detection. The first is for the general
detection that separates the images that do not have much noise to be detected as
healthy, with 100% accuracy. The second layer is to process the rest of the images
with new threshold parameters so that the algorithm can extract more healthy images
with an accuracy of 95%. The rest of the undetected images will pass through the
third and fourth layers for more thresholding numbers that are designed for the
images that have lots of noise. These layers can eliminate the healthy fasteners that
have lots of noise and keep the missing fasteners only to be saved in one file.
With this technique, only the file that was classified as Missing Fasteners was
required to perform a manual check. The manual part was to check the Missing
Fasteners file and separate them into Missing Right, Missing Left, Missing Both.
27
2.3 Stage Three: Machine Learning (Processing Only The
Fasteners)
2.3.1 Machine Learning (ResNet-50)
Figure 15 shows the structure of the residual neural network used in this thesis.
Building a new CNN and training it would take a very long time. Hence, the idea of
using a Pre-trained Neural Network to save time was used in the testing and
validation process of the model.
Figure 15: The residual neural network (ResNet-50) [53]
28
After checking the architectural structures of many neural networks that are available
to be used, such as AlexNet, VGG16, VGG19, Inception or GoogleNet,….etc, the
decision was to go with the residual neural network (ResNet-50).
ResNet-50 is a pre-trained model trained by using the ImageNet database. ResNet-
50 also won the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)
competition in 2015. Moreover, ResNet-50 was trained using more than a million
images [54]. ResNet-50 has five stages with a convolution and Identity block. Each
one of the convolution blocks has three convolution layers, as well as each of the
identity blocks has three convolution layers. ResNet-50 has in total of 177 layers (50
layer residual networks. The ResNet-50 has over 23 million trainable parameters and
can classify images into 1000 object classes [54].
The ResNet-50 was selected as the model skips the connections work to mitigate the
problem of vanishing gradient. The solution provided in this model is by using a
shortcut path for the gradient to flow through. Thus, the model can learn an identity
function that guarantees that the high layers would perform at least as good as the
deep layers, and not worse [54].
2.3.2 Training and Testing
The data set, obtained from the labelling stage, was divided into two separate
groups, one for training and the other for testing (70% and 30%, respectively). The
training process of the model was based on extracting features from the layer that
comes directly before the classification layer.
The generated data set after the labelling stage has a massive number of Healthy
fasteners but very few numbers of missing fasteners. To overcome this data
imbalance problem, the decision was to run two separate tests. The first test
considers the original data set only. In contrast, the other considers the original data
set mixed with a large number of augmented images for all classes.
2.3.3 Image Augmentation
The image augmentation process can defiantly increase the number of missing
fasteners. Despite a large number of healthy fasteners available in the data set used,
the decision was to apply augmentation technique to some healthy fasteners as well.
The Image augmentation was performed using each of the following image
augmentation functions (Color Transformations / Brightness Jitter / Saturation Jitter /
Shear / Contrast Jitter / Brightness and Contrast Jitter of Grayscale Images /
Synthetic Noise / Synthetic Blur).
29
Figure 16 shows some of the results after the image augmentation step. Most of the
images were augmented to higher or lower illumination levels, while only a few of the
images have been augmented in a way to contain some artificial noise.
2.3.4 Validation
The validation in this thesis was implemented to ensure that the model will produce
reliable and repeatable results within the pre-determined conditions. The validation
planned in this thesis took into account the following areas; training speed, detection
speed, noise test, and illumination test. Table 1 shows all the types of objects that
were considered noise in this validation process.
Noise test illumination test
Rocks, paint or strange objects that are placed on the top of the
fasteners.
High or low illumination degree levels
Table 1: Validation Information for the model
Figure 16: Examples of the augmentation images
30
4 Results and Discussions
After the labelling was carried out, all newly generated small images of fasteners,
training, and testing of the model were conducted. The training and testing process
occurred in two separate ways to take into consideration all possible scenarios of the
real world. Hence, the results were grouped into two parts; the first discusses the
results of training and testing after using only the original data set, while the second
includes the results obtained after taking into consideration the augmented images
for all four classes.
4.1 Group 1
Table 2 shows the obtained results from this test. This group considers training the model to classify the fasteners into four groups Healthy Fasteners, Left Missing, Right Missing, and Missing Fasteners. This group of training and testing processes
contained only the original labelled data set of the fasteners. The original images of fasteners have a vast number of healthy fasteners, but only a few of missing
fasteners, as shown in Table 3. Accuracy results
Attempt Number Accuracy
1 0,9639
2 0,9278
3 0,8452
4 0.8726
5 0,9004
6 0,9718
7 0,9087
Average Accuracy 0,9129
Table 2: Accuracy results for seven run times
Healthy Fasteners
Right Missing fastener
Left Missing Fastener
Both Missing Fasteners
Total images of Fasteners
2042 31 23 14 2110
Table 3: Detailed numbers of fasteners that are involved in the first group of training and
testing
The obtained results from this group of training were not suitable due to the absence
of a good number of images for each of the Missing Fastener classes. It is
reasonable to have such results, where the traditional convolutional neural networks
need a sufficient number of images to be able to make a proper detection.
The code is designed to randomly select training and testing sets (70% training set
and 30% testing set). However, because each of the training and test was randomly
selected, there was a need to collect results for seven runs and only consider
average accuracy Table 4.
31
Confusion matrix
Attempt Number Classes Confusion Matrix
1
Healthy Right Missing
Left Missing Both Missing
2
Healthy Right Missing
Left Missing Both Missing
3
Healthy Right Missing
Left Missing Both Missing
4
Healthy
Right Missing Left Missing Both Missing
5
Healthy Right Missing
Left Missing Both Missing
6
Healthy Right Missing
Left Missing Both Missing
7
Healthy
Right Missing Left Missing Both Missing
Table 4: Confusion matrix results for seven run times
Healthy
Fasteners
Missing
Fastener - Right
Missing
Fastener - Left
Missing
Fastener - Both
Healthy
Fasteners
1.0000 0 0 0
Missing Fastener -
Right
0 1 0 0
Missing Fastener -
Left
0.1429 0.1429 0.7143 0
Missing
Fastener - Both
0.3333 0 0 0.6667
Table 5: The confusion matrix results in only the third run of Model 1
32
The results collected from this random splitting led to the conclusion that the model
can make an appropriate classification whenever there is a large data set. The more
images of missing fasteners, the better the distinction will be in the model. For
instance, as shown in Table 4 and Table 5, in the third run attempt results of the
confusion matrix, the healthy fasteners have high detection results. However, when it
comes to Both Missing Fasteners classes, it is clear that the wrong detection
percentage is weighed high risk (33.33%). This is considered very high to the railway
companies and can lead to high risk in real-world use.
The mentioned example can happen for any of the trained classes: Healthy, Missing
Left, Missing Right, Missing Both. The reason for this is that every time the model
was executed, the code separates each of the classes into two groups randomly
(70% training set and 30 % test set). This means that every time the model considers
new images that have different conditions. Following this method, it was clear how
the model interacts with the different fasteners conditions such as noise, dust, rocks,
illumination,….etc. Therefore, there was a need to go for further investigation to see
how the model would perform just in case that we can provide, in the future,
extensive data set of Missing Fasteners classes. The only solution that is available
for the time being is to generate new images by applying the augmentations
functions on the original images so that we can get different missing fasteners
artificial conditions as well as increase the data set images to a trustable level.
33
4.2 Group 2
Table 6 shows the obtained results from this test. This group considers processing
the original labelled data set images mixed with the generated augmented images.
Increasing the data set images by using the augmentation technique was necessary
to further investigation of model performance. However, the question is which
augmentation functions should be used so that the results can be close to reality
without any exaggeration.
After many tests and discussions, the final choice was to consider only the functions
that reasonable in real-life scenarios. The used image augmentation functions are as
follows Color Transformations / Brightness Jitter / Saturation Jitter / Shear / Contrast
Jitter / Brightness and Contrast Jitter of Grayscale Images / Synthetic Noise /
Synthetic Blur). However, only a sufficient number of augmented images of the
healthy fasteners were used to avoid overfitting of the model.
The model was trained to classify the fasteners into four groups Healthy Fasteners,
Left Missing, Right Missing, and Missing Fasteners, and the number of fasteners for
each class shown in Table 7.
The obtained results after seven-run times were so close to each other, and the
average number of accuracy was 98.62 %. Table 8 shows the model performance,
where the model randomly split the data set for training and testing in each run time
so that we can simulate the different conditions.
Accuracy results
Attempt Number Accuracy
1 0.9857
2 0.9891
3 0.9894
4 0.9989
5 0.9756
6 0.9752
7 0.9898
Average Accuracy 0.9862
Table 6 Accuracy results for seven run times
Healthy Fasteners
Right Missing fastener
Left Missing Fastener
Both Missing Fasteners
Total images of Fasteners
2182 503 350 237 3272
Table 7 Detailed numbers of fasteners that are involved in the second group of training and
testing
34
Confusion matrix
Attempt Number Classes Confusion Matrix
1
Healthy Right Missing
Left Missing Both Missing
2
Healthy
Right Missing Left Missing
Both Missing
3
Healthy Right Missing
Left Missing Both Missing
4
Healthy Right Missing
Left Missing Both Missing
5
Healthy Right Missing
Left Missing Both Missing
6
Healthy
Right Missing Left Missing Both Missing
7
Healthy Right Missing Left Missing
Both Missing Table 8 Confusion matrix results for seven run times
Healthy
Fasteners
Missing
Fastener - Right
Missing
Fastener - Left
Missing
Fastener - Both
Healthy Fasteners
0.9954 0 0.0031 0.0015
Missing Fastener -
Right
0 1.0000 0 0
Missing
Fastener - Left
0 0 1.0000 0
Missing
Fastener - Both
0 0 0 1.0000
Table 9 The confusion matrix results in only the fourth run of Model 2
35
To investigate the results more deeply, we recorded the results of the confusion
matrixes after each run time. The results of the confusion matrix in Table 8 above
explain clearly how the model is getting stronger after including the augmented
images with the original data set. This led us to conclude that the used number of
augmented images for each class was acceptable, which also means that we do not
have overfitting in the process of detecting the fasteners classes.
Moreover, in the fourth-run Table 9, the accuracy was 0.9989, and there was no
misclassification for each of the missing classes compared to the results of the first
group. In this example, only 0.15% and 0.31% of the healthy fasteners were detected
as missing fasteners, while the detection of the missing fasteners was very high.
36
5 Model Validation and Final Decision
5.1 Noise Test
The group 2 model was trained on many images with lots of noise, but we kept those
that have more complex noise levels to validate and evaluate the detection accuracy.
The following figures Figure 17 shows some examples of those images that have
complicated environments. Each of these images was successfully detected by the
model.
5.2 Illumination Test
Through the training process, the group 2 model was trained with many different
illumination levels so that the model can operate in extreme illumination conditions.
Thus the model developed for group 2 can be operated regardless of the time that
the images were captured. The following images were used to validate the
performance of the group 2 model, and the results were accurate.
Figure 17: Different scenarios that had been detected correctly
Figure 18: Images that have different illumination levels
37
5.3 Speed Test
The results were outstanding, where the required detection time was the same for
group1 and group2. The only difference that was found related to training time, as the
number of images involved in model 2 was higher than the other.
Model Group 1 Group 2
Training Time 59 second 120 second
Detection Time 0.5 second 0.5 second
Table 12 Comparison of the model after training and testing in group1 and group 2
38
6 Conclusion
Using the automated visual inspection is becoming crucial in the railway industry of
the future. Over the past decade, researchers have been trying to develop suitable
models that can facilitate the mission of the automated visual detection for the rail
track and its component. All previously published researches were focusing on
building a model that can classify only one type of the rail component. However,
there is no specific model that can detect the rail track and its component together in
one model.
This thesis shows how an automated detection system can be developed for railway
tracks and its components. The development of machine learning models for
classification of images was carried out for one component only, namely fasteners.
The model was able to classify the fasteners successfully into four groups Healthy
Fasteners, Missing Fastener-Left, Missing Fastener-Right, and Missing Fasteners-
Both. The results of the testing and validation process were high, where the model
was performing with an accuracy of 98.62%. The training time is 120 seconds and
the time required for detection is 0.5 seconds. As the results were described in the
previous section of the report, the model was able to tackle severe conditions such
as different illumination levels, track covered in dust, and rocks.
Finally, we believe that using a more critical data set that contains a more
considerable amount of missing fasteners would also enhance its performance and
make the model performance very high. The results obtained from the fasteners
classification, using the ResNet-50, indicate that classification for the other discussed
parts of the rail could possibly give high results also.
39
7 Future Work
Our strategy for the future is to take into consideration each of the following ideas
1- Build an overall system to monitor all track components simultaneously Figure
(12).
2- Identify the position of the detected fasteners and pinpoint the location of missing
clamps, damaged rail, and broken sleeper by using GPS, sensors, or other ideas.
3- Incorporate more types of fasteners.
4- Enhance the data set for training the algorithm
5- Train the model to detect loose clamps.
The intent of this future work is to see how the model would respond to the tiny
movements of the fasteners clamps. The available data set, in this thesis, has only
one example for each. The right fastener clamp is out of its correct position about 2
cm Figure (21). It is expected that after training the model to detect such cases, the
future maintenance teams can be informed that those fasteners are about to fail.
Thus it can be scheduled to be repaired once the maintenance is performed near this
site.
Figure 20: Other type of Fasteners – Fast Clip
Figure 21: Difficult conditions
40
8 References
[1] B.S. Dhillon, B. S. (2002). Engineering maintenance: a modern approach. cRc
press.
[2] Su, Z., Jamshidi, A., Núñez, A., Baldi, S., & De Schutter, B. (2017). Multi-level
condition-based maintenance planning for railway infrastructures–A scenario-based
chance-constrained approach. Transportation Research Part C: Emerging
Technologies, 84, 92-123.
[3] [7] Patra, A. P. (2009). Maintenance decision support models for railway
infrastructure using RAMS & LCC analyses (Doctoral dissertation, Luleå tekniska
universitet).
[4] Jardine, A. K., Lin, D., & Banjevic, D. (2006). A review on machinery diagnostics
and prognostics implementing condition-based maintenance. Mechanical systems
and signal processing, 20(7), 1483-1510.
[5] Singh, M., Singh, S., Jaiswal, J., & Hempshall, J. (2006). Autonomous rail track
inspection using vision based system. In 2006 IEEE International Conference on
Computational Intelligence for Homeland Security and Personal Safety (pp. 56-59).
IEEE.
[6] Singh, M., Singh, S., Jaiswal, J., & Hempshall, J. (2006). Autonomous rail track
inspection using vision based system. In 2006 IEEE International Conference on
Computational Intelligence for Homeland Security and Personal Safety (pp. 56-59).
IEEE.
[7] Fan, H., Cosman, P. C., Hou, Y., & Li, B. (2018). High-speed railway fastener
detection based on a line local binary pattern. IEEE Signal Processing Letters, 25(6),
788-792.
[8] Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019). Railway track fastener
defect detection based on image processing and deep learning techniques: A comparative study. Engineering Applications of Artificial Intelligence, 80, 66-81.
[9] Chandran, P., Rantatalo, M., Odelius, J., Lind, H., & Famurewa, S. M. (2019).
Train-based differential eddy current sensor system for rail fastener
detection. Measurement Science and Technology, 30(12), 125105.
[10] Amari, S. V., & McLaughlin, L. (2004). Optimal design of a condition-based
maintenance model. In Annual Symposium Reliability and Maintainability, 2004-
RAMS (pp. 528-533). IEEE.
[11] Sandström, J., & Ekberg, A. (2009). Predicting crack growth and risks of rail
breaks due to wheel flat impacts in heavy haul operations. Proceedings of the
41
Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, 223(2),
153-161.
[12] He, Q., Li, H., Bhattacharjya, D., Parikh, D. P., & Hampapur, A. (2015). Track
geometry defect rectification based on track deterioration modelling and derailment
risk assessment. Journal of the Operational Research Society, 66(3), 392-404.
[13] Al-Douri, Y. K., Tretten, P., & Karim, R. (2016). Improvement of railway
performance: a study of Swedish railway infrastructure. Journal of Modern
Transportation, 24(1), 22-37.
[14] Goodall, R. M., & Roberts, C. (2006). Concepts and techniques for railway
condition monitoring. In 2006 IET international conference on railway condition
monitoring (pp. 90-95). IET.
[15] Papaelias, M. P., & Lugg, M. (2012). Detection and evaluation of rail surface
defects using alternating current field measurement techniques. Proceedings of the
Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit, 226(5),
530-541.
[16] Resendiz, E., Molina, L. F., Hart, J. M., Edwards, J. R., Sawadisavi, S., Ahuja,
N., & Barkan, C. P. L. (2010). Development of a machine-vision system for inspection
of railway track components. In 12th World Conference on Transport Research,
Lisbon, Portugal (pp. 1-22).
[17] Liu, X., Lovett, A., Dick, T., Rapik Saat, M., & Barkan, C. P. (2014). Optimization
of ultrasonic rail-defect inspection for improving railway transportation safety and
efficiency. Journal of Transportation Engineering, 140(10), 04014048.
[18] Faris, M., Núñez, A., Su, Z., & De Schutter, B. (2018). Distributed Optimization
for Railway Track Maintenance Operations Planning. In 2018 21st International
Conference on Intelligent Transportation Systems (ITSC) (pp. 1194-1201). IEEE.
[19] Hart, J. M., Ahuja, N., Barkan, C. P., & Davis, D. D. (2004). A Machine Vision
System for Monitoring Railcar Health: Preliminary Results. Technology Digest: TD-
04-008, Association of American Railroads, Pueblo, Colorado.
[20] Melander, M.(2016). Computer vision based solution for sign
detection.https://www.globalrailwayreview.com/article/30202/computer-vision-based-
solution-sign-detection/
[21] Introducing Machine Learning - MathWorks.
https://la.mathworks.com/content/dam/mathworks/tag-
team/Objects/i/88174_92991v00_machine_learning_section1_ebook.pdf
[22] Getting Started with Machine Learning.
https://www.mathworks.com/content/dam/mathworks/tag-
team/Objects/i/88655_93014v00_machine_learning_section2_ebook.pdf
42
[23] Rashidi, H. H., Tran, N. K., Betts, E. V., Howell, L. P., & Green, R. (2019).
Artificial intelligence and machine learning in pathology: the present landscape of
supervised methods. Academic pathology, 6, 2374289519873088.
[24] Gershenson, C. (2003). Artificial neural networks for beginners. arXiv preprint
cs/0308031.
[25] Jain, A. K., Mao, J., & Mohiuddin, K. M. (1996). Artificial neural networks: A
tutorial. Computer, 29(3), 31-44.
[26] Rojas, R. (2013). Neural networks: a systematic introduction. Springer Science &
Business Media.
[27] Torrey, L., & Shavlik, J. (2010). Transfer learning. In Handbook of research on
machine learning applications and trends: algorithms, methods, and techniques (pp.
242-264). IGI global.
[28] Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions
on knowledge and data engineering, 22(10), 1345-1359.
[29] Dai, W., Yang, Q., Xue, G. R., & Yu, Y. (2007). Boosting for transfer learning.
In Proceedings of the 24th international conference on Machine learning (pp. 193-
200).
[30] Trosino, M., Cunningham, J. J., & Shaw III, A. E. (2000). U.S. Patent No.
6,064,428. Washington, DC: U.S. Patent and Trademark Office.
[31] Trosino, M., Cunningham, J. J., & Shaw III, A. E. (2002). U.S. Patent No.
6,356,299. Washington, DC: U.S. Patent and Trademark Office.
[32] Xia, Y., Xie, F., & Jiang, Z. (2010, November). Broken railway fastener detection
based on adaboost algorithm. In 2010 International Conference on Optoelectronics
and Image Processing (Vol. 1, pp. 313-316). IEEE.
[33] Liu, J., Huang, Y., Zou, Q., Tian, M., Wang, S., Zhao, X., ... & Ren, S. (2019).
Learning visual similarity for inspecting defective railway fasteners. IEEE Sensors
Journal, 19(16), 6844-6857.
[34] Marino, F., Distante, A., Mazzeo, P. L., & Stella, E. (2007). A real-time visual
inspection system for railway maintenance: automatic hexagonal-headed bolts
detection. IEEE Transactions on Systems, Man, and Cybernetics, Part C
(Applications and Reviews), 37(3), 418-428.
[35] De Ruvo, P., Distante, A., Stella, E., & Marino, F. (2009). A GPU-based vision
system for real time detection of fastening elements in railway inspection. In 2009
16th IEEE International Conference on Image Processing (ICIP) (pp. 2333-2336).
IEEE.
[36] Babenko, P. (2009). Visual inspection of railroad tracks.
43
[37] Mahalanobis, A., Kumar, B. V., Song, S., Sims, S. R. F., & Epperson, J. F.
(1994). Unconstrained correlation filters. Applied Optics, 33(17), 3751-3759.
[38] De Ruvo, P., Distante, A., Stella, E., & Marino, F. (2009). A GPU-based vision
system for real time detection of fastening elements in railway inspection. In 2009 16th IEEE International Conference on Image Processing (ICIP) (pp. 2333-2336). IEEE.
[39] Marino, F., Distante, A., Mazzeo, P. L., & Stella, E. (2007). A real-time visual
inspection system for railway maintenance: automatic hexagonal-headed bolts detection. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(3), 418-428.
[40] Mazzeo, P. L., Nitti, M., Stella, E., & Distante, A. (2004). Visual recognition of
fastening bolts for railroad maintenance. Pattern recognition letters, 25(6), 669-677. [41] Stella, E., Mazzeo, P., Nitti, M., Cicirelli, G., Distante, A., & D'Orazio, T. (2002,
September). Visual recognition of missing fastening elements for railroad maintenance. In Proceedings. The IEEE 5th International Conference on Intelligent
Transportation Systems (pp. 94-99). IEEE. [42] Liu, J., Li, B., Xiong, Y., He, B., & Li, L. (2015). Integrating the symmetry image
and improved sparse representation for railway fastener classification and defect recognition. Mathematical Problems in Engineering, 2015.
[43] Yang, J., Tao, W., Liu, M., Zhang, Y., Zhang, H., & Zhao, H. (2011). An efficient direction field-based method for the detection of fasteners on high-speed
railways. Sensors, 11(8), 7364-7381.
[44] Li, Y., Otto, C., Haas, N., Fujiki, Y., & Pankanti, S. (2011). Component-based track inspection using machine-vision technology. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval (pp. 1-8).
[45] Feng, H., Jiang, Z., Xie, F., Yang, P., Shi, J., & Chen, L. (2013). Automatic
fastener classification and defect detection in vision-based railway inspection systems. IEEE transactions on instrumentation and measurement, 63(4), 877-888.
[46] LIU, J., XIONG, Y., LI, B., & LI, L. (2016). Research on automatic inspection algorithm for railway fastener defects based on computer vision. Journal of the China
Railway Society, 38(8), 73-80. [47] Manikandan, R., Balasubramanian, M., & Palanivel, S. (2017). Machine vision
based missing fastener detection in rail track images using SVM
classifier. International Journal on Smart Sensing and Intelligent Systems, 10(4),
574-589.
[48] Fan, H., Cosman, P. C., Hou, Y., & Li, B. (2018). High-speed railway fastener
detection based on a line local binary pattern. IEEE Signal Processing Letters, 25(6),
788-792.
44
[49] Mandriota, C., Nitti, M., Ancona, N., Stella, E., & Distante, A. (2004). Filter-based
feature selection for rail defect detection. Machine Vision and Applications, 15(4),
179-185.
[50] Mazzeo, P. L., Ancona, N., Stella, E., & Distante, A. (2003). Visual recognition of
hexagonal headed bolts by comparing ICA to wavelets. In Proceedings of the 2003
IEEE International Symposium on Intelligent Control (pp. 636-641). IEEE.
[51] Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal
of machine Learning research, 3(Jan), 993-1022.
[52] Liu, J., Huang, Y., Zou, Q., Tian, M., Wang, S., Zhao, X., ... & Ren, S. (2019). Learning visual similarity for inspecting defective railway fasteners. IEEE Sensors
Journal, 19(16), 6844-6857. [53] Peng, J., Kang, S., Ning, Z., Deng, H., Shen, J., Xu, Y., ... & Huang, J. (2020).
Residual convolutional neural network for predicting response of transarterial
chemoembolization in hepatocellular carcinoma from CT imaging. European
radiology, 30(1), 413-424.
[54] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image
recognition. In Proceedings of the IEEE conference on computer vision and pattern
recognition (pp. 770-778).