From Hand-Crafted to Deep Learning-based CancerRadiomics: Challenges and Opportunities

Parnian Afshar†, Student Member, IEEE, Arash Mohammadi†, Senior Member, IEEE, Konstantinos N.Plataniotis‡, Fellow, IEEE, Anastasia Oikonomou∗, and Habib Benali>, Member, IEEE

†Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montreal, Canada

‡ Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada∗Department of Medical Imaging, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, Canada

>PERFORM Centre, Electrical and Computer Engineering Department, Concordia University, Montreal, Canada

Abstract—Recent advancements in signal processing and ma-chine learning coupled with developments of electronic medicalrecord keeping in hospitals and the availability of extensiveset of medical images through internal/external communicationsystems, have resulted in a recent surge of significant interestin “Radiomics”. Radiomics is an emerging and relatively newresearch field, which refers to extracting semi-quantitative and/orquantitative features from medical images with the goal ofdeveloping predictive and/or prognostic models, and is expectedto become a critical component for integration of image-derivedinformation for personalized treatment in the near future.The conventional Radiomics workflow is typically based onextracting pre-designed features (also referred to as hand-craftedor engineered features) from a segmented region of interest.Nevertheless, recent advancements in deep learning have causedtrends towards deep learning-based Radiomics (also referred toas discovery Radiomics). Capitalizing on the advantageous ofthese two approaches, there are also hybrid solutions developedto exploit the potentials of multiple data sources. Consideringthe variety of approaches to Radiomics, further improvements re-quire a comprehensive and integrated sketch, which is the goal ofthis article. This manuscript provides a unique interdisciplinaryperspective on Radiomics by discussing state-of-the-art signalprocessing solutions in the context of cancer Radiomics.

Index Terms: Radiomics, Deep Learning, Hand-CraftedFeatures, Medical Imaging.

I. INTRODUCTION

The volume, variety, and velocity of medical imaging datagenerated for medical diagnosis are exploding. Generallyspeaking, medical diagnosis refers to determining the sourceand etiology of a medical condition. Diagnosis is typicallyreached by means of several medical tests, among of whichare biopsy and diagnostic imaging, in case of suspected cancer.Although biopsy can be very informative, it is invasive and bybeing focal, may not represent the heterogeneity of the entiretumor, which is crucial in cancer prognosis and treatment. Incontrast to biopsy, diagnostic imaging is not invasive and canprovide information on tumor’s overall shape, growth overtime, and heterogeneity, making it an attractive and favoredalternative to biopsy. Interpretation of such a large amount of

Corresponding Author is Arash Mohammadi, email:arash.mohammadi@concordia.ca

diagnostic images, however, highly depends on the experienceof the radiologist and due to the increasing number of imagesper study can be time-consuming.

Referred to as “Radiomics” [1], the ability to process suchlarge amounts of data promises to decipher the encodedinformation within medical images; Develop predictive andprognostic models to design personalized diagnosis; Allowcomprehensive study of tumor phenotype [2], and; Assesstissue heterogeneity for diagnosis of different type of cancers.More specifically, Radiomics refers to the process of extract-ing and analyzing several semi-quantitative (e.g., attenuation,shape, size, and location) and/or quantitative features (e.g.,wavelet decomposition, histogram, and gray-level intensity)from medical images with the ultimate goal of obtainingpredictive or prognostic models.

Although several challenges are in the way of bringingRadiomics into daily clinical practice, it is expected thatRadiomics become a critical component for integration ofimage-driven information for personalized treatment in thenear future. The first comprehensive clinical application ofRadiomics was performed by Aerts et al. [2] with involvementof 1019 lung cancer patients. More than 400 different intensity,shape, texture, and wavelet features were extracted from Com-puted Tomography (CT) images and used together with clinicalinformation and gene expression data to develop Radiomicsheat map, which shows the association between Radiomics anddifferent clinical outcomes such as cancer stage. This clinicalstudy has illustrated/validated effectiveness of Radiomics fortumor related predictions and showed that Radiomics has thecapability to identify lung and head-and-neck cancers from asingle-time point CT scan, which is significantly promising.Consequently, there has been a recent surge of interest [3]–[7] on this multidisciplinary research area as Radiomics hasthe potential to provide significant assistance for assessing therisk of recurrence of cancer; Evaluating the risk of radiation-induced side-effects on non-cancer tissues, and; Predicting therisk for cancer development in healthy subjects.

The key underlying hypothesis in the Radiomics is thatthe constructed descriptive models (based on biological and/ormedical data) are capable of providing relevant and beneficialpredictive, prognostic, and/or diagnostic information. In this

arX

iv:1

808.

0795

4v1

[cs

.CV

] 2

3 A

ug 2

018

2

regard, one can identify two main categories of Radiomics.Conventional pipeline based on Hand-Crafted Radiomic fea-tures (HCR) that consists of the following four main pro-cessing tasks: (i) Image acquisition/reconstruction; (ii) Imagesegmentation; (iii) Feature extraction and quantification, and;(iv) Statistical analysis and model building. On the other

hand, the Deep Learning-based Radiomics (DLR) pipeline hasrecently emerged which differs from the former category sincedeep networks do not necessarily need the segmented RegionOf Interest (ROI), and their feature extraction and analysisparts are partially or fully coupled. We will elaborate on theseproperties in section IV.

Information Post I: Radiomics Supporting Resources

Several potential medical resources provide information to theRadiomics pipeline, some of which are directly used to extractRadiomics features and some serve the decision making process,as complementary information sets. Below we review the mostimportant data resources for Radiomics.

Screening Technologies: The Radiomics features can be ex-tracted from several imaging modalities, among which the fol-lowing are the most commonly used modalities:

• Computed Tomography (CT) Scans: The CT is the modal-ity of choice for the diagnosis of many diseases in dif-ferent parts of the body, and by providing high resolutionimages [8] paves the path for extracting comparable Ra-diomics features. Nonetheless, the CT imaging performancedepends on different components of the utilized protocolincluding the following three main properties: (i) Slicethickness, which is the distance in millimeter (mm) betweentwo consecutive slices; (ii) The capability for projectingthe density variations into image intensities, and; (iii) Re-construction algorithm, which aims at converting tomo-graphic measurements to cross-sectional images. AlthoughCT protocols for specific clinical indications are usuallysimilar across different institutions, there is still a significantdemand to design Radiomics features that are insensitive toCT protocols [9]. CT images are typically divided into twocategories [10]: screening and diagnostic. While screeningCT uses low dose images, diagnostic CT utilizes high doseand is of higher quality and contrast.

• Positron Emission Tomography (PET) Scans: The PET isa nuclear imaging modality that evaluates body function andmetabolism [8], and since its performance depends on notonly the scanner properties, but also the doze calibration,it is more difficult to standardize PET protocols acrossdifferent institutions. Furthermore, glucose level at the timeof scanning can also affect the properties of PET images [9].

• Magnetic Resonance Imaging (MRI): Unlike CT, proper-ties of MRI images are not directly associated with tissuedensity and specific methods are required to obtain the so-called signal intensity. Besides, other parameters such asslice thickness and field strength affect the properties of theMR images [9] which should be consistent across differentinstitutions.

Complimentary Data Sources: In addition to imaging resources,the following clinical data sources are typically combined withRadiomics features:

• Gene expression: The process of converting DNA to func-tional product to have a global insight of cellular function.

• Clinical characteristics: Patient’s characteristics such age,gender, and past medical and family history [9].

• Blood Bio-markers: Measurable characteristics from thepatient’s blood such as glucose level, cholesterol level andblood pressure.

• Prognostic Markers: Markers to evaluate the progress ofthe disease, response to treatment or survival, such as size,tumor stage, tumor recurrence, and metastasis.

Radiomics Image Databases: Large amount of data is typicallyrequired to get reliable results about tissue heterogeneity based onRadiomics [9]. Table I below introduces a few publicly availableimaging data source that can be used to develop and test newRadiomics approaches:

TABLE I: Popular data sets for performing Radiomics.

Data Set Reference Year Imaging Modality Application

LIDC-IDRI [15] 2015 CT Lung TumorNSCLC-Radiomics [2] 2014 CT Lung Tumor

NSCLC-Radiomics-Genomics [2] 2014 CT, Gene expression Lung tumorLGG-1p19qDeletion [45] 2017 MRI Brain TumorHead-Neck-PET-CT [46] 2017 PET, CT Head-and-Neck Cancer

BRATS2015 [50] 2015 MRI Brain Tumor

More clinical studies are being approved and conductedto further investigate and advance the unparalleled oppor-tunities the Radiomics posed to offer for clinical applica-tions. Information Post I provides an overview of differentscreening technologies used within the Radiomics pipeline

along with supporting data sources and available deatasets todevelop Radiomics-based predictive/prognostic models. WhileRadiomics consists of a wide range of (partially intercon-nected) research areas with each individual branch possiblyworth a complete exploration, the purpose of this article is

3

to provide an inclusive introduction to Radiomics for thesignal processing community, as such, we will focus on theprogression of signal processing algorithms within the contextof Radiomics. In brief, we aim to present an overview of thecurrent state, opportunities, and challenges of Radiomics fromsignal processing perspective to facilitate further advancementand innovation in this critical multidisciplinary research area.As such, the article will cover the following four main areas:

(i) Hand-crafted Radiomics, where we introduce and in-vestigate different feature extraction, feature reduction,and classification approaches used within the context ofRadiomics.

(ii) Deep learning-based Radiomics, where we provide anoverview of different deep architectures used in Ra-diomics along with interpretability requirements.

(iii) Hybrid solutions developed to simultaneously benefitfrom the advantages of each of the above two mentionedcategories.

(iv) Challenges, Open Problems, and Opportunities, wherewe focus on the limitations of processing techniquesunique in nature to the Radiomics, and introduce openproblems and potential opportunities for signal process-ing researchers.

Fig. 1 shows the increasing interest in Radiomics within theresearch community. Although there have been few recentarticles [11], [12] reviewing and introducing Radiomics, tothe best of our knowledge, most of them are from outsidethe signal processing (SP) community. References withinthe SP society such as the work by J. Edwards [13] haveinvestigated recent advancements in medical imaging devicesand technologies without reviewing the role of Radiomicsin medical applications. Other existing papers outside SPcommunity (e.g., [12]) have failed to clearly describe theunderlying signal processing technologies and have narroweddown their scope only to hand-crafted Radiomics and itsdiagnosis capability. While Reference [11] has briefly touchedupon the deep learning pipeline as an emerging technology thatcan extract Radiomics features, it has not studied applicabilityof different deep architectures [14] and left the interpretabilitytopic untouched. All these call for an urgent and timely questto introduce Radiomics to our community especially since SPis one of the main building blocks of the Radiomics.

The reminder of this article is organized as follows: InSection II, we will discuss several applications of Radiomicsin cancer-related fields, followed by Hand-Crafted Radiomics(HCR) solutions in Section III. The Deep learning-basedRadiomics (DLR) is presented in Section IV, where severalaspects of DLR is investigated. In Section V, we explaindifferent hybrid solutions to Radiomics, which aim to leveragethe advantageous of both DLR and HCR. In Section VIvarious challenges and opportunities of Radiomics, especiallyfor SP community, is discussed. Finally, Section VII concludesthe paper.

II. APPLICATIONS

In recent years, Radiomics has been applied to a plenty ofhealth-care applications, among which cancer-related topics

Fig. 1: Increasing interest in Radiomics based on data from GoogleScholar.

have been the main focus of interest. Below we brieflyintroduce and define different cancer-related applications inwhich Radiomics has been proved to be successful:

1) Cancer diagnosis, which refers to confirming the pres-ence or absence of the cancer, is one of the most criticaland sensitive decisions that has to be made as earlyas possible. However, most of the times cancers arediagnosed in late stages reducing the chance of receivingeffective treatment, as there are typically few clinicalsymptoms in the early stages of cancer. Nevertheless,Radiomics has the potential to improve the accuracy ofcancer early diagnosis.

2) Tumor detection refers to the identification of thoselesions that are malignant, which is very important inorder to guide targeted local treatment. For instance,Radiotherapy, the process of killing cancerous cellsusing ionizing radiation, can cause irreparable damagesnot knowing the exact location of tumor. Drug delivery,i.e., having an exact plan to deliver the drug to thetarget area, is another problem that requires preciseinformation about the abnormality location.

3) Tumor classification and attribute scoring: Tumor clas-sification refers to determining the type of the tumor.Typically, cancer is classified into the following mainclasses: (i) benign; (ii) primary malignant, and; (iii)metastatic malignant. Besides, tumors are associatedwith different attributes such as their border and spheric-ity. Analyzing these attributes contribute to a betterunderstanding of the tumor’s shape and behavior.

4) Survival prediction: The knowledge of the expectedsurvival of a specific disease with or without a specifictreatment is critical both for treating physicians and thepatients. Physicians need to choose the best treatmentplan for their patients and patients need to know theirpredicted survival time in order to make their ownchoices for the quality of their life. Radiomics can addsignificant information about patient’s survival based onimage properties and heterogeneity of the tumor and thishas attracted a lot of attention recently.

5) Malignancy prediction: Tumors can be either malignant

4

TABLE II: Applications of Radiomics.

Reference Year Imaging Modality Information Sources Application Domain Radiomics MethodZhang et al. [3] 2017 CT - Prediction of lung can-

cer recurrence and death.Hand-crafted Radiomics(HCR)

Aerts et al. [2] 2014 CT Gene-expression andClinical data.

Lung, and head & neckcancer survival predic-tion.

HCR

Griethuysen et al. [4] 2017 CT - Lung cancer benign andmalignant classification.

HCR

Oikonomou et al. [1] 2018 CT, PET Standardized uptakevalue (ratio of imageand body radioactivityconcentration).

Lung cancer survivalprediction.

HCR

Kumar et al. [48] 2015 CT - Lung cancer benign andmalignant classification.

Deep Learning-basedRadiomics (DLR)

Kumar et al. [21] 2017 CT - Lung cancer benign andmalignant classification.

DLR

Huynh et al. [22] 2016 Mammogram - Classification of breastcancer: benign or malig-nant.

HCR, DLR, Combina-tion of HCR and DLR

Li et al. [23] 2017 MRI - IDH1 enzyme mutationprediction.

DLR

Sun et al. [5] 2017 CT - Lung cancer benign andmalignant classification.

HCR, DLR

Jamaludin et al. [24] 2016 MRI - Disc abnormality classi-fication.

DLR

Liu et al. [28] 2017 MRI - Prostate cancer diagnosis. HCR, DLR

Oakden-Rayner et al. [25] 2017 CT - Longevity prediction. HCR, DLR

Paul et al. [26] 2016 CT - Lung cancer short/long-term survival prediction.

Combination of HCRand DLR

Fu et al. [27] 2017 CT - Lung tumor detection. Combination of HCRand DLR

or benign based on several factors such as their abil-ity to spread to other tissues. Benign tumors usuallydo not spread to other organs but may need surgicalresection because occasionally they may grow in size.Pre-invasive lesions may be indolent for years, however,they may transform to aggressive malignant tumorsand therefore need to be monitored closely or evenbe treated with lower dose of anti-cancer regimens.Malignant tumors are life threatening and may spreadto distant organs, requiring more complicated treatmentssuch as Chemotherapy. Prediction of tumor malignancylikelihood with noninvasive methods such as Radiomicsis, therefore, of paramount importance.

6) Recurrence prediction: Even the treated cancers havethe potential to grow and reappear, which is referredto as “cancer recurrence”. As the cancerous regionis supposed to be removed or treated, there are notstrong landmarks or evidences helping with predictingthe recurrence. However, recently Radiomics is beingemployed to assist with such issue and has shownpromising initial results.

7) Cancer staging: Cancers may be diagnosed in differentstages, e.g., they may be in an early stage meaning thatthey are remaining in the tissue they have first appearedin, or they can be in an advanced stage, meaning thatthey are spread in other tissues. Knowing the stage of the

5

TABLE III: Applications of Radiomics (Continued)

Reference Year Imaging Modality Information Sources Application Domain Radiomics MethodLao et al. [29] 2017 MRI Clinical risk factors. Brain cancer survival

prediction.Combination of HCRand DLR

Shen et al. [30] 2015 CT - Lung cancer benign andmalignant classification.

DLR

Liu et al. [31] 2018 MRI - Alzheimer’s disease di-agnosis.

DLR

Antropova et al. [32] 2017 Mammogram,Ultrasound, MRI

- Breast cancer benignand malignant classifi-cation.

Combination of HCRand DLR

Chen et al. [34] 2017 CT - Lung cancer featuresscoring.

Combination of HCRand DLR

Liu et al. [33] 2017 CT - Lung cancer benign andmalignant classification.

Combination of HCRand DLR

Wang et al. [35] 2017 CT, PET Standardized uptakevalue.

Lung cancer benign andmalignant classification.

HCR, DLR

Shen et al. [36] 2017 CT - Prediction of lung tu-mor malignancy likeli-hood.

DLR

Liu et al. [37] 2017 CT - Lung cancer benign,primary malignant, andmetastatic malignantclassification.

DLR

Emaminejad et al. [49] 2016 CT Genomics bio-markers Lung cancer recurrencerisk prediction

HCR

Sun et al. [7] 2016 CT - Lung cancer benign andmalignant classification.

HCR, DLR

Kim et al. [38] 2016 CT - Lung cancer benign andmalignant classification.

Combination of HCRand DLR

Shen et al. [39] 2016 CT - Lung cancermalignancy probabilityestimation.

DLR

Ciompi et al. [40] 2017 CT - Lung tumor classifica-tion as solid and non-solid.

DLR

Afshar et al. [17] 2018 MRI - Brain tumor type classi-fication.

DLR

tumor has significant impact on the choice of requiredtreatment.

Based on the above categories, Tables II and III summarizedifferent application domains of Radiomics introduced inarticles that are cited in our work, along with their associatedRadiomics method (HCR, DLR, or the combination of both).These tables also provide information on any complementary

data source that has been utilized in combination with Ra-diomics.

III. STATE-OF-THE-ART IN HAND-CRAFTED RADIOMICS

In clinical oncology, tissue biopsy, which refers to theremoval of a small focal part of the cancerous tissue (tumor), isconsidered as the state-of-the-art approach for diagnosing can-cer. Although tissue biopsy has several diagnostic properties

6

and is widely used for detecting and investigating cancerouscells, its reliability is limited by the fact that tumors arespatially and temporally heterogeneous, and as a result, biopsycannot capture all the available information that is necessaryfor an inclusive decision. Besides, most of the biopsies areinvasive which restricts the number of times this procedurecan be performed, or sometimes biopsy is not an option due tothe high risk for complication that pertains to specific patients.Alternatively, Radiomics extraction from medical images is anon-invasive technique and can capture intra-tumoral hetero-geneity. Furthermore, Radiomics extraction can be performedin a real-time fashion and has the ability to reduce cancermortality and cancer cost by providing early stage diagnosis.Although it may seem conflicting, Radiomics can even be usedto facilitate biopsy by detecting more suspicious locations [12].

In this section, we focus on the state-of-the-art researchon hand-crafted Radiomics (HCR). Studies on hand-craftedRadiomics features [1], [8], [9], typically, consist of thefollowing key steps:

1. Pre-processing, introduced to reduce noise and artifactsfrom the original data and typically includes imagesmoothing and image enhancement techniques.

2. Segmentation, which is a critical step within the HCRworkflow, as typically HCR features are extracted fromsegmented sections and many tissues do not have distinctboundaries [12]. Although manual delineation of thegross tumor is the conventional (standard) clinical ap-proach, it is time consuming and extensively sensitive tointer-observer variability [9], resulting in a quest to de-velop advanced (semi) automated segmentation solutionsof high accuracy that can also generate reproducibleboundaries. Although there is no segmentation methodthat works for all medical images, the most commonapproaches include level sets, region growing (seed-based), graph cuts, and contour-based methods [9]. Sincenon of these approaches work for all regions, the beststrategy is to adopt a majority voting segmentation,to leverage the potentials of all the aforementionedmethods [50].The most important metric for evaluating a segmentationmethod is to calculate its accuracy according to aground truth. However, since ground truth is not alwaysavailable for medical images, reproducibility metrics areoften used to assess the performance of the segmentationalgorithm [9], [12]. For instance, reference [9] has useda similarity metric based on the overlap of generatedsegments resulting in a better average for automaticmethods compared with manual delineation.

3. Feature extraction, which is the main step in Radiomicsworkflow and will be discussed in details in Sub-section III-A.

4. Feature reduction, is another critical step in Radiomicsas although a large number of quantitative features canbe extracted from the available big image datasets, mostof the features are highly correlated, irrelevant to the taskat hand, and/or contribute to over-fitting of the model.To address these issues, Radiomics feature reduction

techniques are discussed in Sub-section III-B.5. Statistical analysis, which refers to utilizing the ex-

tracted Radiomics features in a specific applicationas outlined in Section II. We will further elaborateon such Radiomics-based statistical analysis in Sub-section III-C.

In the reminder of this section, we focus on Steps 3-5 in Sub-sections III-A-III-C, respectively, starting by reviewing thekey feature extraction methodologies recently used in HCR,followed by a review of the main feature reduction techniquesand Radiomics-based statistical analytics.

A. Radiomics Feature Extraction

During the feature extraction step within Radiomics work-flow, different types of features are extracted that can begenerally classified into three main categories: (1) First order(intensity-based and shape-based features) [3]; (2) Secondorder (texture-based features) [3], and; (3) Higher order fea-tures [12]. Table IV provides a summary of different potentialfeatures. It is worth mentioning that HCR features are notlimited to this list and can exceed hundreds of features (e.g., inReference [2] 400 HCR features are initially extracted beforegoing through a feature reduction process). Below, we furtherinvestigate the most commonly used categories of hand-craftedfeatures:

1. Intensity-based Features: Intensity-based methods convertthe multi-dimensional ROI into a single histogram (describingthe distribution of pixel intensities), from which simple andbasic features (e.g., energy, entropy, kurtosis, and skewness)are derived. Intensity features allow us to investigate propertiesof the histogram of tumor intensities such as sharpness,dispersion, and asymmetry. These features are, however, themost sensitive ones to image acquisition parameters such asslice thickness [10] (discussed in the Information Post I).Therefore, designing intensity-based features need special careand pre-processing. Among all intensity features, entropy anduniformity are the most commonly used in Radiomics and willbe briefly outlined below [44].

Generally speaking, entropy measures the degree of ran-domness within the pixel intensities, and takes its maximumvalue when all the intensities occur with equal probabilities(complete randomness). Considering H as the pixels intensityhistogram within a specific ROI and B as the number ofhistogram bins, entropy can be simply defined as follows

Entropy = −B∑i=1

H(i) log2H(i). (1)

Uniformity, on the other hand, estimates the consistency ofpixel intensities, and takes its maximum value when all thepixels are of the same value. Uniformity can be defined asfollows

Uniformity =

B∑i=1

H(i)2. (2)

Although intensity-based features are simple to calculate andhave the potential to distinguish several tissues such as benign

7

TABLE IV: Different categories of HCR features commonly used within the context of Radiomics.

Category Description Sub-categoryFirst Order Radiomics Concerned with the distribution of pixel

intensities and use of elementary metrics tocompute geometrical features.

• Shape Features Quantify the geometric shape of region orvolume of interest [9]

Size of the Region of Interest (ROI);Sphericity; Compactness; Total volume;Surface area, Diameter, flatness and;Surface-to-volume ratio [9], [10].

• Intensity Features Derived from a single histogram generatedfrom the 2D region or the whole 3D vol-ume [9].

Intensity Mean; Intensity Standard Devi-ation; Intensity Median; Minimum of In-tensity; Maximum of Intensity; Mean ofPositive Intensities; Uniformity; Kurtosis;Skewness; Entropy; Normalized Entropy;Difference of Entropy; Sum of Entropy, and;Range [9], [10].

Second Order Radiomics (Texture Features) Concerned with texture features and rela-tions between pixels to model intra-tumorheterogeneity. Texture features are gener-ated from different descriptive matrices [9].

• Gray Level Co-occurrence (GLCM) GLCM [10] is a matrix that presents thenumber of times that two intensity levelshave occurred in two pixels with specificdistance.

Contrast; Energy; Correlation; Homogene-ity;Variance; Inverse Difference Moment;Sum of Average; Sum of Variance; Differ-ence of Variance; Information Measure ofCorrelation; Autocorrelation; Dissimilarity;Cluster Shade; Cluster Prominence; ClusterTendency, and; Maximum Probability.

• Gray Level Run-Length (GLRLM) GLRLM [44] is a matrix that presents thelength of consecutive pixels having the sameintensity.

Short run emphasis; Long run emphasis;Gray Level Non-Uniformity; Run lengthnon-uniformity; Run percentage; Low graylevel run emphasis, and; High gray level runemphasis [9].

• Neighborhood Gray Tone DifferenceMatrix (NGTDM)

NGTDM [10] is concerned with the inten-sities of neighboring pixels instead of thepixel itself.

Coarseness; Contrast; Busyness; Complex-ity Texture Strength.

Higher Order Radiomics Use of filters to extract patterns from im-ages.

Wavelets; Fourier features [10]; Minkowskifunctionals; Fractal Analysis [12], and;Laplacian of Gaussian (LoG) [4].

and malignant tumors [44], they suffer from some drawbacks.First, the selected number of bins can highly influence suchfeatures, as too small or too large bins can not resemble theunderlying distribution correctly, and as such these featuresare not always reliable representatives. Besides, optimizing thenumber of histogram bins can also be problematic, because itleads to different number of bins for different ROIs, and makesit difficult to compare the results of various studies.

2. Shape-based Features: Shape-based features describe thegeometry of the ROI and are useful in the sense that theyhave high distinguishing ability for problems such as tumormalignancy and treatment response prediction [10]. Althoughradiologists commonly use shape features (also referred to as“Semantic Features” or “Morphological features”), the aim ofRadiomics is to quantify them with computer assistance [12].These features are extracted from either 2D or 3D structuresof the tumor region to investigate different shape and size

characteristics of the tumor.Among different shape-based features, volume, surface,

sphericity, compactness, diameter, and flatness are more com-monly used in Radiomics. For instance, sphericity measuresthe degree of roundness of the volume or region of interest andit is specially useful for the prediction of tumor malignancy,as benign tumors are most of the times more sphere comparedto malignant ones. Compactness is itself defined based onsphericity and as such, these two need not to be calculatedsimultaneously, and one of them will be probably excludedby the feature selection methods, which are targeting featureredundancy.

3. Texture-based Features: Shape-based and intensity-basedfeatures fail to provide useful information regarding correla-tions between different pixels across a given image. In thisregard, texture-based features are the most informative ones,specially for problems where tissue heterogeneity plays an

8

important role, because texture-based features can catch thespatial relationships between neighboring pixels [10]. In Ra-diomics, typically, texture-based features are extracted basedon different descriptive matrices, among which gray levelco-occurrence matrix (GLCM), gray level run length matrix(GLRLM), and neighborhood gray tone difference matrix(NGTDM) are the most commonly used ones [44], which aredefined below:• The GLCM, models the spatial distribution of pixels’

intensities and can be calculated by considering thefrequency of the occurrence of all pairs of intensityvalues. Features extracted from GLCM are the mostcommonly used textural features in Radiomics [44]. EachGLCM is associated with two predefined parameters θand d, where θ ∈ {0◦, 45◦, 90◦, 135◦}, and d is anyinteger distance admissible within the image dimensions.Therefor, GLCM can be defined as follows

GLCMθd(i, j)=

∣∣∣{((r, s), (t, v)) :I(r, s)= i, I(t, v)= j}∣∣∣ ,(3)

for (1 ≤ i, j ≤ Ng), where Ng is the number of allavailable intensities within the specific image, and I(·) isa function mapping image locations into intensities. Term| · | denotes the cardinality of a set. Terms (r, s) and (t, v)are any pixel pairs that have the following relations witheach other based on the pre-defined θ and d

(t, v) =

r + d, s θ = 0◦

r + d, s+ d θ = 45◦

r, s+ d θ = 90◦

r − d, s+ d θ = 135◦

(4)

Finally GLCMs can be calculated for different possiblevalues of θ and d allowing the calculation of hundreds oftextural features.

• The GLRLM, defines the number of adjacent pixelshaving the same intensity value, e.g., the (i, j) elementof the GLRLMθ matrix determines the number of timesintensity value i has occurred with run length j, indirection θ.

• The NGTDM, which is based on visual characteristicsof the image, is a vector whose kth element is definedas the summation of differences between all pixels withintensity value k and the average intensity of their neigh-borhood (size of which is determined by the user). TheNGTDM is calculated as follows

NGTDM(k) =∑

(i,j)|I(i,j)=k

|k −Avg(i, j)|, (5)

for (1 ≤ k ≤ Ng), where, Avg(i, j) is the average inten-sity of the pixels within the neighborhood of pixel (i, j).

4. Higher Order Radiomics Features: Higher order featuressuch as Wavelet and Fourier features capture imaging bio-markers in various frequencies [10]. Wavelet features arethe mostly used higher order features in Radiomics. Waveletcourse and fine coefficients represent texture and gradientfeatures respectively, and is calculated by multiplying theimage by a matrix including complex linear or radial “waveletmother functions”. Fourier features can also capture gradient

information. Minkowski Functional (MF) is another commonhigher order feature extractor considering the patterns of pixelswith intensities above a predefined threshold.

In brief, the MFs are computed by initially forming a binaryversion of the ROI through utilization of several thresholdswithin the minimum and maximum intensity limits. Althoughthe number of utilized thresholds is a free parameter, forbetter results, it should be identified through a selectionmechanism (typically empirical tests are used). Based on thebinarized ROI, different MFs such as area and perimeter arecomputable as follows

MFarea = ns, (6)and MFperimeter = 4ns + 2ne, (7)

where ns and ne are the total number of white pixels (abovethe threshold) and edges, respectively. This completes ourcoverage of feature extraction methods used in Radiomics.

B. Radiomics Feature Reduction Techniques

Feature reduction is another critical step in Radiomicsas although a large number of quantitative features can beextracted from the available image datasets, most of thefeatures are highly correlated, irrelevant to the task at hand,and/or contribute to over-fitting of the model (making ithighly sensitive to noise). Feature reduction techniques thatare used in Radiomics can be classified into supervised andunsupervised categories [3], as summarized in Table V. Su-pervised approaches, such as filtering and wrapper methods,take the discriminative ability of features into account andfavor features that can best distinguish data based on a pre-defined class. Unsupervised methods, on the other hand, aim toreduce feature redundancy and include Principle ComponentAnalysis (PCA), Independent Component Analysis (ICA) andZero Variance (ZV) [3].

In summery, various objectives can be defined when re-ducing the feature space in Radiomics. The following keycharacteristics can be defined for feature selection purposes[9], [12]:• Reproducibility: Reproducible features (also referred to

as “stable features”) are the ones that are more robustto pre-processing and manual annotations. These featureswill be discussed in Sub-section III-D.

• Informativeness and Relevancy, which can be definedas features that are highly associated with the targetvariable [10]. For instance a χ2-test, calculates the chi-squared statistic between features and the class variable,and consequently features with low impact on the targetare discarded. Another selection approach is a Fisherscore test, where features with higher variance are treatedas the more informative ones.

• Redundancy: Non-redundant features are the ones withsmall correlation with each other. Feature redundancy isdefined as the amount of redundancy present in a par-ticular feature with respect to the set of already selectedfeatures.

Bellow, supervised and unsupervised techniques commonlyused in Radiomics are further discussed.

9

TABLE V: Feature reduction techniques commonly used within the Radiomics literature.

Category Description MethodsSupervised Considers the relation of features with the

class labels and features are selected mostlybased on their contribution to distinguishclasses.

• Filtering (Univariate) Test the relation between the features andthe class label one by one.

Fisher score (FSCR); Wilcoxon rank sumtest; Gini index (GINI); Mutual informationfeature selection (MIFS); Minimum redun-dancy maximum relevance (MRMR), and;Student t-test [44].

• Wrapper (Multivariate) Considers both relevancy and redundancy. Greedy forward selection, and Greedy back-ward elimination.

Unsupervised Does not consider the class labels and itsobjective is to remove redundant features.

• Linear Features have linear correlations. Principle Component Analysis (PCA), and;Multidimensional scaling (MDS)

• Nonlinear Features are not assumed to be lied on alinear space.

Isometric mapping (Isomap), and; Locallylinear embedding (LLE).

1. Supervised Feature Selection Methodologies: Supervisedmethods are generally divided into two categories as outlinedbelow:

• Filtering (Univariate) Methods: These methods considerthe relation between the features and the class label oneat a time without considering their redundancy. Amongall filtering approaches, Wilcoxon test based methodhas been shown to be more stable, resulting in morepromising predictions in the field of Radiomics [44]. AWilcoxon test is a nonparametric statistical hypothesestesting technique that is used to determine dependenciesof two different feature sets, i.e., whether or not they havethe same probability distribution.

• Wrapper (Multivariate) Methods: Filtering methods havethe drawback of ignoring relations between featureswhich has led to development of wrapper techniques. Incontrary to the filtering methods, wrapper methods inves-tigate the combined predictive performance of a subsetof features, and the scoring is a weighted sum of bothrelevancy and redundancy [44]. However, computationaldifficulties prevent such methods from testing all thepossible feature subsets.Wrappers methods include greedy forward selection andgreedy backward elimination. In a forward feature re-duction path, selection begins with an empty set and thecorrelation with class label is calculated for all featuresindividually. Consequently, the most correlated featureis selected and added to the set. In the next step, theremaining features are added, one by one, to this set totest the performance of the obtained set, and the processcontinues until no further addition can increase the pre-dictive performance of the set. A backward selection pathworks in contrary to the forward one, beginning with a setincluding all the available features, and gradually reducesthem until no further reduction improves the performance.

Since supervised methods are based on class labels, they aresubject to over-fitting and can not be easily applied to differentapplications once trained based on a given feature set.

2. Unsupervised Feature Selection Methodologies: Unsuper-vised approaches try to reduce the feature space dimensionalityby removing redundant features (those who are correlatedand do not provide any additional information). Althoughthese methods are not prone to over-fitting, they are notguaranteed to result in the optimum feature space. Unsuper-vised techniques can be divided into linear and non-linearmethods, where the former assumes that features lie on a linearspace. Because, in the field of Radiomics, commonly, verysimple and basic forms of unsupervised techniques, such asPCA, are used, they are not covered in this article. However,this presents an opportunistic venue for application of moreadvanced statistical-based dimensionality reduction solutionsrecently developed within signal processing literature.

C. Radiomics Statistical Analysis

Statistical analysis refers to utilizing the extracted Ra-diomics features in a specific task such as cancer diagno-sis, tumor stage classification, and survivability analysis, asdescribed in Section II. Although most statistical methods,initially, treat all the features equally and use the sameweights over all predictors, in the area of Radiomics, the mostsuccessful methods are the ones that use a prior assumption(provided by experts) over the meaning of features [12]. Onebasic approach to analyze the Radiomics features adoptedin [2], [4] is to cluster the extracted features and look forassociations among clusters and clinical outcomes. For in-stance, patients belonging to one cluster may have similardiagnosis or patterns. Observations show that image bio-markers are associated with clinical outcomes such as tumormalignancy. Hierarchical clustering is most commonly used

10

TABLE VI: Common analysis methods in Radiomics.

Purpose Description MethodClustering Similar patients are grouped together based

on a distance metrics.Hierarchical, Partitional

Classification Models are trained to distinguish patientsbased on their associated clinical outcome.

Random Forest (RF); Support Vector Ma-chine (SVM); Neural Network (NN); Gen-eralized linear model (GLM); Naive Bayes(NB); k-nearest neighbor (KNN); MixtureDiscriminant Analysis (MDA); Partial LeastSquares GLM (PLS), and; Decision Tree(DT).

Time-related analysis The survival time or the probability of sur-vival is calculated based on the available setof data from previous patients.

Kaplan-Meier survival analysis; Cox pro-portional hazards regression model [1], and;Log-Rank Test.

in Radiomics [9]. However, clustering techniques are not basi-cally trained for target forecasting purposes, which necessitatesthe use of prediction tools that are specially trained basedon a predefined class label. Prediction tools in Radiomics arecategorized as either:(i) Classification and Regression Models that are mostly

similar to other multi-media domains, trying to foreseea discrete or continues value. Random Forest (RF), Sup-port Vector Machine (SVM) and Neural Network (NN)are among the most common regression and classifi-cation techniques used to make predictions based onRadiomics [3].

(ii) Survivability analysis: Also referred to as time-relatedmodels, mostly try to predict the survival time associatedwith patients. These models are also useful when testingthe effectiveness of a new treatment.

Table VI presents a summary of different Radiomics analysistechniques. As predictors belonging to the former categoryare also common in other multi-media applications, they arenot covered in this article. Survivability analysis (the lattercategory), however, is more specific to Radiomics, and assuch, below, we discuss the three mostly used techniques fromthis category, i.e., Kaplan-Meier Survival Curve (KMS), CoxProportional Hazards (regression) Model (PHM), and Log-Rank Test.

1. Kaplan-Meier Survival Curve (KMS): The KMS curve [2],[3] represents a trajectory for measuring the probability ofsurvival S(t) in given points of time t, i.e.,

S(t) =Number of patients survived until t

Number of patients at the beginning. (8)

The KMS curve can be calculated for all Radiomics featuresto assess the impact of different features on patients’ survivalas follows:

1) A desired feature, for which the KMS curve is supposedto be calculated, is selected.

2) Based on the selected feature, one or more thresholdsare considered that can partition patients into, e.g., lowand high risk cancer subjects. Patients are then groupedbased on whether their associated feature lies above orbelow the threshold.

3) The KMS curve is calculated for all the obtained groups,and the result can be used to compare the survivabilityamong patients with, e.g., low and high risk cancer. Forinstance, in Reference [2] high heterogeneity featuresare associated with shorter survival time, while highcompactness features are associated with longer survival.

2. Cox Proportional Hazards (Regression) Model (PHM) [2],is commonly used in medical areas to predict patient’s survivaltime based on one or more predictors (referred to as covariates)such as Radiomics features. The output of the PHM modeldenoted by h(t) is the risk of dying at a particular time t,which can be calculated as follows

h(t) = h0(t)× exp∑Nc

i=1 bixi (9)

where xi, for (1 ≤ i ≤ Nc), are predictors (covariates); birepresent the impacts of predictors, and h0(t) is called thebase-line hazard. The exponent term in Eq. (9) is referred to asthe “Risk” and is conventionally assumed to be a linear com-bination of the features (covariates), i.e., Risk ,

∑Nc

i=1 bixi.The Risk coefficients (bi, for (1 ≤ i ≤ Nc)) are then computedthrough a training process based on historical data. Morerealistically, the risk can be modeled as a general non-linearfunction, i.e., Risk , f(x), with the non-linearity beinglearned via deep learning architectures, which has not yet beeninvestigated within the Radiomics context.

3. Log-Rank Test [2], which is used for comparing thesurvival of two samples specially when these two samples haveundergone different treatments. This test is a non-parametrichypothesis test assessing whether two survival curves varysignificantly. One limitation associated with the Log-Ranktest is that the size of the groups can influence the results,therefore, larger number of patients should be included to fromequal sized groups.Evaluation of HCR: In summary, having a successful hand-crafted Radiomics pipeline requires an accurate design tochoose the best combination of feature extraction, featurereduction, and analysis methods. The effects of these designchoices are recently investigated in [3] through an analysis ofvariance, where it has been shown that, e.g., feature selectioncan significantly influence the final accuracy. Finally, it should

11

be noted that reporting accuracy is not as informative measurein Radiomics as it is in other multi-media domains. Because,in medical areas, making mistakes in, for instance, classifyingpositive and negative samples are not equal. Therefore, inRadiomics, measures which are capable of distinguishingbetween False Positive (FP) and False Negative (FN) errorsare more favored. Such one measure is the area under Re-ceiver Operating Characteristic (ROC) curve, which allows forinvestigating the impact of different decision thresholds onFP and FN rates. Confusion matrix is another common anduseful technique to report the performance of a Radiomicsclassifier in terms of its FP and FN rates. In practice, most ofthe decisions in medical areas cannot be made with a completecertainty, and physicians consider several factors such as harmsand benefits of a specific judgment, when forming thresholdsfor their decisions. However, these factors are not quantifiedand utilized in Radiomics, which calls for a broad investigationinto potential solutions to incorporate the factors commonlyused by physicians.

D. Radiomics StabilityAn important aspect of Radiomics is the stability of the

extracted features, which quantifies the degree of dependencybetween features and pre-processing steps. Stability in Ra-diomics is generally evaluated based on either of the followingtwo techniques:

1) Test-Retest: In this approach, patients undergo an imag-ing exam more than once and images are collectedseparately. Radiomics features are then extracted fromall the obtained sets and analyzed. Here, being invariantacross different set of images illustrates stability ofRadiomics features.

2) Inter-observer reliability, which is referred to an ex-periment where multiple observers are asked to delineatethe ROI from the same images, and Radiomics featuresare extracted from all different delineations to test theirstability for variation in segmentation [4]. Here, beinginvariant across different segmentations illustrates sta-bility of Radiomics features.

Different stability Criteria are used to find robust features inRadiomics as briefly outlined below:

1) Intra-class Correlation Coefficient (ICC): One criteriato measure the stability of Radiomics features, which isused for both the aforementioned categories (i.e., test-retest and inter-observer setting) is referred to as theintra-class correlation coefficient (ICC) [2]. The ICC isdefined as a metric of the reliability of features, takingvalues between 0 and 1, where 0 means no reliability and1 indicates complete reliability. Defining terms BMSand WMS as mean squares (measure of variance)between and within subjects, which are calculated basedon a one-way Analysis of variance (ANOVA), for a test-retest setting, the ICC can be estimated as

ICCTest-Retest =BMS −WMS

BMS + (N − 1)WMS, (10)

where N is the number of repeated examinations. Bydefining EMS as residual mean squares from a two-way ANOVA and M as the number of observers, for aninter-observer setting, the ICC can be calculated as

ICCInter-Observer =BMS − EMS

BMS + (M − 1)EMS. (11)

2) Friedman Test: The Friedman test, which is speciallyuseful for assessing the stability in an inter-observersetting, is a nonparametric repeated measurement thatestimates whether there is a significant difference be-tween the distribution of multiple observations, and hasthe advantage of not requiring a Gaussian population.Based on this test, the most stable features are the oneswith a stability rank of 1 [2].

In [2], it is declared that Radiomics features with higherstability have more prognostic performance, therefore, stabilityanalysis can be interpreted as a feature reduction technique.According to Reference [4], Laplacian of Gaussian (LoG),intensity-based, and texture features are more stable for lungCT images, while wavelet and shape-based features are sensi-tive to variation in segmentation.

To summarize our finding on HCR and elaborate on itsapplications, we have provided an example in InformationPost III-D, where the problem of Radiomics-based lung canceranalysis is investigated.

12

Information Post II: Radiomics for Lung Cancer Analysis

Lung cancer is among the deadliest cancers all over the world,and despite the advancements in cancer treatments, lung canceris still rarely cured as it is most of the times detected at advancedstages. Studies have shown that use of computer aided systemscan significantly help with the early diagnosis and detection ofthis disease and consequently lead to dramatic decrease in lungcancer-related mortality [48]. For these computer aided systemsto be successful in lung cancer early prognosis, distinguishingRadiomics features have to be extracted from the CT images,which are the main detectors of lung cancer. Radiomics canhelp with different tasks related to lung cancer such as: (i) Lungcancer patient survival prediction; (ii) Lung cancer malignancyprediction; (iii) Forecasting the patient’s response to treatment;and (iv) Prediction of the stage of the cancer.As illustrated in the figure bellow, the first stage in lung cancerRadiomics pipeline is commonly the segmentation of left andright lungs, followed by the segmentation of the tumor whichcan be performed automatically for tumors with high intensitieslocated in low intensity backgrounds. However, segmentation canbecome problematic when lung tumors are attached to vesselsor walls which makes automatic segmentation fail to generatereproducible and accurate outcomes. Due to this reason lungtumor segmentation is still an open problem [9].The next step after segmenting the tumor, is to extract Radiomicsfeatures. Types of features to extract depend on the problem athand. For instance, according to [10], intensity features are highlycorrelated to lung cancer patient survival, while shape featuresare beneficial for lung cancer malignancy prediction and alsoforecasting the patient’s response to treatment. More importantly,texture features are reported to be the most influential oneson lung cancer outcomes [12]. Selected features will then gothrough a feature selection process, where redundant and non-relevant features are excluded. Feature selection is followed bya statistical analysis step to perform the aforementioned tasks.

To Further elaborate on the use of Radiomics in Lung canceranalysis, we have grouped 157 patients, from NSCLC- Radiomicsdataset [2], into two categories: Long-term survival, referring topatients who have survived more then 500 days, and Short termsurvival, referring to those who have survived less than 500 days.Thirty first-order and second-order Radiomics features are thenextracted from lung tumors, among which, based on a χ2 featureselection, the surface area and volume of the tumors are the mostcorrelated ones to the survival outcome.The following figure, shows the relation between survival and thetwo aforementioned features. As it can be inferred from this fig-ure, although patients with short-term survival can be associatedwith various tumor sizes, those who have survived longer havesmaller tumors in terms of surface area and volume. In otherwords, being small in size seems to be necessary for long-termsurvival, however, as so many other factors influence survival, notall small tumors lead to long survivals. For instance, the locationof the tumor can have significant impact on survival, nevertheless,it can not be assessed based on hand-crafted Radiomics features,as they are extracted from the segmented ROI, without taking itslocation into account.

E. RadiogenomicsRadiomics is typically combined with Genomic data, of-

ten referred to as Radiogenomics [12]. In other words, Ra-

diogenomics refers to the relationship between the imagingcharacteristics of a disease and its gene expression patterns,

13

Fig. 2: Radiogenomics analysis using GSEA. Genes are first sorted based on their association with Radiomics features, andthe final heat map is formed according to the pattern of gene sets within the ordered gene list.

gene mutations and other genome-related characteristics. Po-tential association between imaging outcome and moleculardiagnostic data can serve as a predictor of patient’s responseto the treatments and provide critical support for the decisionmaking tasks within the clinical care setting. In other words,Radiogenomics has the potential to investigate the Genomicsof the cancer, without utilizing invasive procedures such asbiopsy. Various association mining and clustering methods areused to identify the relationships between gene expressionsand Radiomics, e.g., in [12] it was found that just 28 Ra-diomics features were able to reconstruct 78% of the globalgene expressions in human liver cancer cells.

To assess the association between gene expression anddiscrete classes such as benign and malignant tumors, genesshould be first sorted based on their discriminative ability.However, the goal of Radiogenomics is to find associationsbetween gene expression and Radiomics features, therefore,discriminative ability is indefinable. Spearman’s Rank Cor-relation Coefficient (SRC) [18] can be used to measure thecorrelation between a specific Radiomics feature and geneexpression. Genes are then sorted based on their SRC coeffi-cient instead of their discriminative ability. The ordered genesare typically stored in a list L, and to extract meaning fromthis list, the traditional approach is to focus on the top andbottom genes in list L, representing genes with the strongestpositive and negative correlations, respectively. This approach,however, is subject to several limitations, such as the difficultyin biological interpretation, which has led to the introductionof Gene-set Enrichment Analysis (GSEA) [19]. The GSEAis one of the mostly used Radiogenomics approaches [2],and is based on the definition of gene sets. Each gene setS is a group of genes that are similar in terms of theprior biological knowledge, such as involvement in common

biological pathways. The goal of the GSEA is to find outwhether the members of a given set S tend to occur in thetop or bottom of the list L. In this case, the expression of thisgene set is associated with the specific Radiomics feature. Theresult of Radiogenomics analysis using GSEA is a heat maprepresenting the degree of association between all gene setsand Radiomics features as shown in Fig. 2.

The main role of Radiogenomics is that it allows imperfectdata sets, where clinical outcomes are difficult to be collectedor require an extended period to be collected, to be leveragedbased on prior knowledge of the relationship between clinicaloutcomes and Genomics, in order to draw new conclusions.For example in a research project there may have been imagingdata and genome-related data but no clinical outcomes. If thereis prior knowledge of the relationship of the Genomics withcertain clinical outcomes, then by correlating the imaging datawith the Genomics, new conclusion can be drawn about therelationship of the imaging data with the clinical outcomes. Inthis scenario, Radiogenomics can fill the gaps in knowledge.

IV. STATE-OF-THE-ART IN DEEP LEARNING-BASEDRADIOMICS

Deep learning-based Radiomics (DLR), sometimes referredto as “Discovery Radiomics” or “Radiomics Sequence Dis-covery” with “Sequence” referring to features, is the processof extracting deep features from medical images based on thespecifications of a pre-defined task including but not limitedto disease diagnostics; Cancer type prediction, and; SurvivalPrediction. In brief, the DLR can be extracted via differentarchitectures (stack of linear and non-linear functions), e.g.,Convolutional Neural Network (CNN) or an Auto-Encoder, tofind the most relevant features from the input [48]. Fig. 3 illus-trates the schematic of extracting deep features. The extracted

14

Fig. 3: Extracting deep Radiomics. The input to the network can be the original image, the segmented ROI, or the combination of both.Extracted Radiomics features are either utilized through the rest of the network, or an external model is used to make the decision based onRadiomics.

features can then either go through the rest of the deep net foranalysis and making decisions or they may exit the networkand go through a different analyzer such as an SVM or aDecision Tree (DT). Commonly used deep architectures forRadiomics will be discussed in details later in Section IV-C.Benefits of DLR vs. HCR: An important advantage of DLRover its hand-crafted counterpart is that the former doesnot need any prior knowledge and features can be extractedin a completely automatic fashion with high level featuresextracted from low level ones [48]. Moreover, deep learningnetworks can be trained in a simple end-to-end process, andtheir performance can be improved systematically as they arefed with more training samples [41]. Another key benefit ofusing DLR instead of HCR is that the input to the deepnetworks to extract Radiomics features, can be the raw imagewithout segmenting the region of interest, which serves theprocess in two ways:

(i) Eliminating the segmentation step can significantly re-duce the computational time and cost by taking theburden of manual delineation off the experts and radiol-ogists, besides, manual annotations are highly observer-dependent, which makes them unreliable sources ofinformation, and;

(ii) Automatic segmentation methods are still highly errorprone and inaccurate to be used in a sensitive decisionmaking process.

Furthermore, the input to a deep network can also be thecombination of the original and segmented image along withany other pre-processed input such as the gradient image(referred to as “multi-channel” input), all concatenated alongthe third dimension [5]. The variety of input types can even go

further to include images from different angles such as coronaland axial [40].

Generally speaking, studies on DLR can be categorizedfrom several aspects including:

(i) Input Hierarchy: The input to the deep net can be thesingle slices, the whole volume, or even the wholeexaminations associated with a specific patient. Each ofthese cases require their own strategy, e.g., in case ofprocessing the whole volume simultaneously, one shouldthink of a way to deal with the inconsistent dimensionsize, as patients are associated with different number ofslices. One common architecture that allows for utiliza-tion of inputs with variable sizes, such as various numberof slices, is the Recurrent Neural Network (RNN), whichwill be briefly discussed in Section IV-C;

(ii) Pre-trained and Raw Models: Depending on the size ofthe available dataset and also the allocatable time, pre-trained models can be fine-tuned or raw models can betrained from the scratch. This will be analyzed morespecifically in Section IV-B, and;

(iii) Deep Learning Network Architectures: Choice of thedeep network is the most important decision one shouldmake to extract meaningful and practical DLR, whichwill be discussed in Section IV-C.

In the reminder of this section, we will review the state-of-the-art in deep Radiomics from different perspectives such asinput hierarchy, pre-trained vs. raw models, and deep learningnetwork architectures. Fig. 4 illustrates a taxonomy of differentDLR approaches providing a guide to the rest of this section.

15

Fig. 4: Taxonomy of Deep Learning-based Radiomics (DLR).

Fig. 5: Input hierarchy for one patient. In the top row the slice-level input is shown where the patients went through K examination visitsduring each of which N (i), for (1 ≤ i ≤ K), number of slices is captured. The second row shows the volume-level where all slices associatedwith one visit is provided simultaneously as the input to the network. Finally, the third row shows the Patient-level analysis, where a singleinput consisting of all the volumes is provided.

A. Input Hierarchy

As shown in Fig. 5, input images for DLR studies canbe divided into three main categories: Slice-level; Volume-level, and; Patient-level. Slice-level classification refers toanalyzing and classifying image slices independent from eachother, however, this approach is not informative enough aswe typically need to make decisions based on the labelsassigned to the entire Volume of Interest (VOI). Shortcomingsof slice-level classification leads to another approach referredto as volume-level classification, where either the slice-level

outputs are fused through a voting system, or the entire imageslices associated with a volume is used as the input to theclassifier. Finally, patient-level classification refers to assigninga label to a patient based on a series of studies (such as CTimaging follow-ups). For example, in Reference [39], patient-level classification is explored with the goal of estimating theprobability of lung tumor malignancy based on a set of CTstudies. To achieve this goal, initially, a simple three layerCNN is trained to extract DLR from tumor patches associatedwith individual CT series (volume-level classification) with theobjective of minimizing the difference between the predicted

16

malignancy rate and the actual rate. Then, by adopting apreviously trained CNN, the malignancy rate is calculated forall the series belonging to the patient and the final decisionis made by selecting the maximum malignancy rate. In otherwords, a patient is diagnosed with malignant lung cancer if atleast one of the predicted rates is above a pre-determined ratefor malignancy.

B. Pre-trained or Raw Models

Similar to the other medical areas, the DLR can be extractedbased on either of the following two approaches:

Training from scratch: Training a deep network from thescratch for extracting DLR has the advantage of havinga network completely adjusted to the specific problem athand. However, performance of training from scratch couldbe limited due to couple of key issues, i.e., over-fitting andclass imbalance. Adhering to patients’ privacy and need forexperts to provide ground truth typically limits the amountof medical datasets available for extracting DLR resulting inover-fitting of the deep nets. The second issue is the problem ofclass imbalance, i.e., unequal number of positive and negativeclasses. This happens as number of patients diagnosed withabnormalities is commonly less that the amount of data avail-able from healthy subjects. More specifically, class imbalancein medical areas is due to the fact that typically numberof positive labels is less than the number of negative ones,making the classifier biased toward the negative class, which ismore harmful than the other way around because, for instance,classifying a cancerous patient (positive label) as healthy(negative label) has worse consequences than classifying ahealthy patient as cancerous [6]. The following strategies canbe adopted to address these two issues:

(i) Data Augmentation, where different spatial deformations(such as rotation [21]) are applied to the existing datain order to generate new samples for training purposes.Sub-patch Expansion [5] is another form of augmentationcommonly adopted in Radiomics to handle the inadequatedata situation via extracting several random fixed-sizedsub-patches from the original images.

(iii) Multitask training is another method introduced to han-dle class imbalance and inadequate data [14], which isachieved by decreasing the number of free parametersand consequently the risk of over-fitting. For instance,this approach is adopted in [24] for spinal abnormalityclassification based on MRIs through training a multitaskCNN. Multitask in this context refers to performingdifferent classification tasks simultaneously via the sameunified network (e.g., the network tries to classify diskgrading and disk narrowing at the same time). The lossfunction is defined as the weighted summation of allthe losses associated with different tasks. One importantdecision to make in multitask learning is the point thatbranching begins, e.g., in Reference [24], a unified CNNis trained, where all Convolutional layers are shared forperforming different tasks and tasks are separated fromthe point that fully connected layers begin.

(iv) Loss function modification: Another common approachspecific to handling class imbalance for DLR extractionis to modify the loss functions by giving more weight tothe minority class [24].

Transfer Learning via a Pre-trained Network: A differentsolution to class imbalance and inadequate training data is“transfer learning” followed by “fine tuning” [5], [14], [22].The transfer learning phase refers to training the deep net usinga natural image data set, and then in the fine tuning phase, thetrained network will be re-trained using the desired medicaldataset. This strategy is adopted in Reference [22], where apre-trained CNN is used for breast cancer classification basedon mammographic images. The pre-trained CNN used is anAlexnet which is too complicated and prone to over-fitting forsmall datasets. Therefore, this network is first pre-trained usingImageNet database which consists of more than one millionnatural images. Pre-trained CNN based on ImageNet is alsoadopted in [26] for lung cancer survival prediction.

C. Deep Learning Architectures in Radiomics

Radiomics features can be extracted through both discrimi-native and/or generative deep learning networks. As is evidentfrom its name, discriminative deep models try to extractfeatures that make the classes (e.g., normal or cancerous)distinguishable, and thus these models can directly classifyinstances from the extracted features. On the other hand,generative models are unsupervised, meaning that they aretrained without considering the class labels. Generally, the goalof these models is to learn the data distribution in a way thatenables them to generate new data from the same distribution.In other words, generative models can extract the natural andrepresentative features of the data, which can then be used asinputs to a classifier. Furthermore, in the field of Radiomics, itis common [41] to train a generative model and use the learnedweights as initial weights of a discriminative model. Below,an introduction to widely used discriminative and generativedeep models in Radiomics is provided.

1. Discriminative Models: Deep discriminative models tryto extract features capable of distinguishing class labels, andthe objective is to minimize the prediction error. Below,we will review the Convolutional Neural Networks (CNNs)and Recurrent Neural Networks (RNNs), which are the mostpopular discriminative architectures in Radiomics. Later, wewill introduce a recently designed deep architecture referredto as the Capsule network (CapsNet) [16] and explain howthis new architecture can contribute to the Radiomics.

1.1. Convolutional Neural Networks (CNNs): CNN is a stackof layers performing Convolutional filtering combined withnonlinear activation functions and pooling layers [14]. Thefact that CNNs have recently resulted in promising outcomeshave made them the mostly used architecture in medical areasincluding Radiomics. CNNs are more practical in the sensethat shared weights are utilized over the entire input, whichreduces the number of trainable parameters. Unlike extractinghand-crafted features, kernels used in Convolutional layers are

17

not pre-determined and are automatically learned through thetraining process. This property makes CNNs suitable methodsfor extracting DLR features as they are flexible and can beapplied without requiring a prior knowledge. In [36], it hasbeen shown that the DLR extracted from a CNN can visuallydistinguish benign and malignant lung tumors when projectedinto a 2D space, while the original pixel values completelyfail to provide such distinction.

Generally speaking, in a CNN with N layers, the outputY (l−1) of layer l− 1, for (2 ≤ l ≤ N ), serves as the input tothe layer l resulting in the associated output Y (l) as follows

X(l)i,j =

M−1∑a=0

M−1∑b=0

WabY(l−1)i+a,j+b (12)

Y(l)i,j = σ(X

(l)i,j ), (13)

where X(l) is the pre-nonlinearity output, M is the size ofConvolutional filter, W is the matrix containing the CNNlearned weights, and σ(·) denotes the activation functionadopted to apply nonlinearity to the output. Convolutionallayers are typically followed by fully connected nodes (layers),commonly, leading to a final classification and/or regression(SoftMax) layer. Output of the fully connected layers istypically treated as DLR features. These features are theneither used within the original CNN to provide the desired(classification and/or regression) output such as cancer type,or exist the network to be provided as the input to therest of the of the Radiomics pipeline. As an example, inReference [26] DLR are extracted from the layer just beforethe classification (SoftMax) layer of the CNN with the goalof lung cancer survival prediction. These features are referredto as the “preReLU” and “postReLU” features as they areextracted both before and after applying the ReLU activationfunction. The DLR features are then used as inputs to fourclassifiers (i.e., Naive Bayes, Nearest Neighbor, Decision treeand Random Forest) after going through a feature selectionalgorithm.

The CNN architectures used in Radiomics can be dividedinto three main categories: (i) Standard architectures; (ii)Self-designed architectures, and; (iii) Multiple CNNs. Below,we describe each of these categories with examples fromRadiomics:

1.1.1. Standard CNN Architectures: As the name suggests,standard architectures are those that have been previouslydesigned to solve a specific problem, and due to theirsuccess are now being adopted in the Radiomics. Twoof such architectures that have been used in Radiomicsare LeNet and AlexNet. The LeNet is one of the sim-plest CNN architectures, having a total of 7 layers, thathas been used in Radiomics. However, researchers havesome times modified this network to achieve higherperformance, e.g., the CNN used in Reference [5] isa LeNet architecture with a total of 9 layers including3 Convolutional layers, 3 pooling layers and one fullyconnected layer followed by the classification layer toclassify lung tumors as either benign or malignant.

Another commonly used standard architecture in Ra-

diomics is the 11 layers CNN called Alexnet, which hasbeen adopted in [22] to extract DLR features from breastmammographic images. Features are extracted from all 11layers and used as inputs to 11 support vector machine(SVM) with the goal of classifying breast tumors as eitherbenign or malignant. Since it is not obvious which set(output of which of the 11 underlying layers) of DLRfeatures are more practical, these SVMs are comparedand the one with the largest area under the curve is chosenfor predictive analysis of breast cancer. The results of [22]concluded that the features extracted from the 9th layer(a fully connected layer before the last fully connectedlayer and the classification layer) are the best predictors ofbreast cancer and they are of lower dimension comparedto previous ones, which reduces the computational cost.In other words and in contrary to [26], the output of thelast Convolutional layer, right before the fully connectedlayer, is selected as the DLR features.

Although AlexNet is a powerful network, it has too manyparameters for small datasets and is, therefore, proneto over-fitting. As a result, Reference [35] has used anAlexnet with number of layers reduced to 5 in order toavoid the over-fitting problem. The input to this networkis a combination of CT and PET images, each having3 channels: One slice corresponding to the center ofthe lung nodule, specified by an expert, and the twoimmediate neighbors. The goal of this article is to classifylung tumors as benign or malignant, and although ithas been shown that the adopted CNN does not resultin significantly higher accuracy than classical methods(HCR), it is more convenient as it does not require thesegmented ROI.

1.1.2. Self-designed CNNs: As opposed to researchers that haveused standard CNNs with or without modifications, somehave designed their own architectures based on the spec-ification of the Radiomics problem at hand. For example,Reference [21] has used a CNN with three Convolutionallayers to extract DLR features, and although the CNNitself is trained to use these features for classifying benignand malignant tumors, they are used as inputs to a binarydecision tree.

In a similar fashion, Reference [23] has used a CNNwith 6 Convolutional layers and one fully connectedlayer for DLR extraction in the problem of brain tumorclassification. The designed network, however, is differentfrom previously mentioned articles as it is developed fortumor segmentation, and features are extracted from thelast Convolutional layer since they are more robust toshifting and scaling of the input. In other words, theCNN was designed for segmentation and once trained, theoutput of the last Convolutional layer is used as the DLRfeatures. The claim here is that the quality of extractedfeatures depends on the accuracy of segmentation, andwhen segmentation is precise the quality of Radiomicsfeatures is guaranteed. Finally an SVM is trained on theextracted features to classify brain tumors.

18

Fig. 6: Different angles of lung CT scan along with tumor crops inthree different scales.

Lung cancer detection using CNNs is also investigatedin [27], with the difference that the input to the network isnot only the original image but also the nodule-enhancedand vessel-enhanced images, stating that providing thenetwork with more information on tumor and vesselsreduces the risk of misplacing these two by the network.The main focus here is to reduce the false positive ratewhile keeping the sensitivity high, therefore, a significantnumber of nodule candidates are selected at the begin-ning. Use of CNNs is further investigated in [25], wherea 7 layer architecture is fed with down-sampled volu-metric CT images along with their segmentation masksfor longevity prediction. In [28] an architecture calledXmasNet is provided that can maximize the accuracyof prostate cancer diagnosis. This network consists of 4Convolutional layers, 2 fully connected layers, 2 poolinglayers and one SoftMax layer for cancer prediction. Theinputs to this network are 3D MRI images.

In summary, self-designed CNNs are developed by vary-ing the depth of the network (number of the Convolu-tional and non-Convolutional layers); the order the layersare cascaded one after another; the type of the input to thenetwork (e.g., single channel or different form of multi-channel), and/or; the layer whose output is treated as theDLR features.

1.1.3. Multiple CNNs: Beside using single standard or self-designed CNNs, some researchers have proposed to usemultiple networks, which has the advantage of benefitingfrom multiple inputs, having various modalities, scalesand angles as shown in Fig. 6, or different architectureswith different properties.

“Scale” is a significant factor to consider when designingthe input structure. For example to distinguish tumorsfrom vessels, a large enough region should be includedin the input patch, while to differentiate between solidand non-slid tumors, the nodule region should be themain core of the patch. Having this in mind, Refer-ence [40] has designed a CNN architecture for lung tumorclassification, where inputs are patches not only from

different angles (sagittal, coronal, and axial) but also indifferent scales. Following a similar path, Reference [30]has also designed a multiple CNN architecture, whereeach CNN takes a lung tumor patch at a specific scale(illustrated in Fig. 6) as input and generates the associatedDLR features. Features extracted from all the CNNs arethen concatenated and used for lung tumor malignancyprediction through a conventional classifier (SVM). Theidea here is that segmenting the tumor regions is notalways feasible. Furthermore using a tumor patch pro-vides information on not only the tumor itself but alsothe surrounding tissues, and since tumor sizes can varysignificantly among patients, using multi-scale patchesinstead of the single ones will improve the overall per-formance of the extracted DLR features. An interestingproperty of such multiple CNN architecture is that sincethe constituent CNNs share parameters, training can beperformed in a reasonable time. Another benefit of usinga multiple CNN architecture is that the network becomesrobust to addition of small noise to the input.

Similar to the work in [30], Reference [37] has designeda CNN called “Multi-view CNN”, which uses 7 patchesat different scales as inputs, with the difference that thesepatches are resized to have the same dimension, andtherefore, a single CNN can be used instead of multipleCNNs. This work has also extended the binary lungtumor classification to a ternary classification to classifylung tumors as benign, primary malignant, and metastaticmalignant. Furthermore, this article has adopted anothervalidation approach called “separability” besides the com-mon terms such as accuracy and AUC (area under curve).Separability refers to the extend that different classesare distinguishable based on the learned features, andaccording to the aforementioned article, the proposedmulti-view CNN has a higher Separability compared toa single scale CNN. In addition to that, as the layers godeeper, features with higher separability are learned.

The idea of using multi-scale image patches is further ex-panded in Reference [36] through designing a novel CNNarchitecture called “Multi-crop CNN”, where instead oftaking inputs in various scales, multi-scale features areextracted through parallel pooling layers, one of whichapplies pooling to a cropped version of the input fromthe previous layer. Features from multiple pooling layersare then concatenated and fed to the next layer. 3D lungCT images are inputs to this network, and since multipleCNNs are replaced with one single CNN, the trainingcan be performed in a more time effective manner. Besideforecasting the lung tumor malignancy, this work has alsopredicted other attributes associated with tumor such asdiameter, by replacing the final SoftMax layer with aregression one. It is worth mentioning that this networkis not performing all the assigned tasks simultaneously.Instead they are performed one after another, whichdistinguishes this network from a multitask training onediscussed in section IV-B.

Radiomics through multiple CNNs is further explored

19

recently in [31] for Alzheimer’s disease diagnosis usingMRI, where in the first stage several landmarks aredetected based on the comparison between normal andabnormal brains. These landmarks are then used to extractpatches (separately around each individual landmark),and consequently each CNN is trained taking patchescorresponding to a specific landmark position as input.Final decision is made based on a majority voting amongall the CNNs. Here, the idea behind using a multiplearchitecture is the fact that detecting Alzheimer’s diseaserequires the examination of different regions of the brain.

In summary, multiple CNNs methods developed for DLRfeature extraction are designed by either fusing the out-puts of several CNNs which are trained based on aspecific input, or multi-path layers are embedded within asingle network to modify the output from previous layersdifferently.

One challenge shared among all the aforementioned CNNarchitectures is that they do not take the spatial informationbetween objects into account. As an example, they may failto consider the location of abnormality within the tissue asan indicator of its type. The newly proposed deep architecturecalled CapsNets, described next, is introduced to overcomethis drawback.

1.2. Capsule Networks: Although CNNs are the state of theart in many medical and non-medical classification problems,they are subjected to several drawbacks including their lowexplainability and their negligence in preserving the spatialrelationships between elements of the image leading to miss-classification. Besides, CNNs have low robustness to sometypes of transformation. Loss of spatial relation information,which is associated with the pooling layers, is resolved by thenewly proposed Capsule networks (CapsNets) [16] consistingof both convolutional and capsule layers that can handle moretypes of transformation. These deep architectures have theability to consider the relationships between the location ofobjects and tolerate more types of transformation, throughtheir routing by agreement process, which dictates that anobject will not be classified as a specific category unless thelower level elements of this object agree on the existence ofthat category. Another important property of CapsNets is thatthey can handle smaller datasets, which is typically the casein most medical areas. Here we explain the architecture ofCapsule networks, as illustrated in Fig. 7, and their routing byagreement process.

Capsules are group of neurons whose activity vectors consistof various instantiation parameters, and the length of theactivity vectors represent the probability of a specific instancebeing present. Each Capsule in the primary capsules layertries to predict the outcome of all the capsules in the nextlayer (parent capsules), however, these predictions are sentto the next layer with different coefficients, which is basedon how much the actual output agrees with the prediction.This process of looking for agreement between capsules beforecoupling them is called routing by agreement and it is the mostimportant property of capsule networks, making them considerspatial relations among objects, therefore, being robust to

several types of transformations such as affine transformationand rotation. Defining ui as the output of capsule i in theprimary capsules layers, the prediction for capsule j, uj|i iscalculated as follows, where Wij is the weight matrix to belearned in back propagation

uj|i = Wijui. (14)

Consequently Cij , the coupling coefficient of capsules i andj, is calculated based on the degree of conformation betweenthese two capsules, based on the idea that if two vectors agreethey will have a bigger inner product, and the output of parentcapsule j, sj , is estimated as follows

sj = Cijuj|i. (15)

Finally, a non-linear function is applied to sj to always keep itslength equal or smaller than one as this length should representprobability of an object being present.

Below we further investigate the initial utilization of Cap-sNets in Radiomics [17], for the first time. In [17] we haveexplored various CapsNet architectures to select the one thatmaximizes the prediction accuracy resulting in a network thathas fewer convolutional filters (64 filters) compared to theoriginal Capsule network which has 256 filters. This archi-tecture consisting of one convolutional, one primary capsule,and a classification layer results in 86.56% accuracy.

Furthermore, separate networks are trained based on twotypes of inputs: the original brain images and the segmentedtumor regions, observing that CapsNet performs better whenbeing fed with tumor masks, probably because the brainimages have miscellanies backgrounds, distracting the networkfrom extracting important distinguishing features. Neverthe-less, Capsule network is of higher accuracy for both of theinput types compared to the CNN, which has 78% accuracyfor tumor images. Several factors may have enabled theCapsNet to provide a better performance including its abilityto handle small datasets, and being robust to transformationsand rotations, resulted from the routing by agreement process.

1.3. Recurrent Neural Networks: Most of the deep networkarchitectures need fixed-sized inputs, which makes them inef-fective for Radiomics analysis of volumetric images (volume-level classification), i.e., when the whole volume is needed tobe processed at once (such as tumor classification based onthe 3D volume). In these scenarios, the RNNs can be adoptedas they are capable of processing sequential data such as CTor MR slices, and they take both the present image slice andresult of processing the previous ones as inputs. RNNs are alsouseful to monitor the medical images resulted from follow-upexaminations (patient-level classification).

Since RNNs are associated with the vanishing gradientproblem, a new type called long-short-term-memory (LSTM)is proposed which has the ability to decide what to storeand what to forget. Although it seems that RNNs and LSTMare computationally more expensive than other architectures,their training time and cost is greatly reduced by using thesame weights over the whole network [14]. Use of LSTMsis explored in Reference [43] for prostate cancer benign and

20

Fig. 7: Capsule network architecture. A convolutional layer is used to form the primary capsules, and the decision is made based on theagreement among these capsules.

malignant classification based on sequences of ultrasoundimages, where it has been shown that the predictive accuracyof this sequential classification is higher than making decisionbased on independent single images.

This completes the overview of deep discriminative modelswith application to Radiomics. Next, we briefly review thegenerative models.

2. Generative Models: The objective of most of the deepgenerative models is to learn abstract yet rich features fromthe data distribution in order to generate new samples fromthe same distribution. What makes these models practical inRadiomics is the fact that the learned features are probably thebest descriptors of the data, and thus have the potential to serveas Radiomics features and contribute to a consequent taskssuch as tumor classification. Auto-encoder networks, deepbelief networks, and deep Boltzmann machines are among thedeep generative models that have been utilized in Radiomicsworks as outlined below:

2.1. Auto-Encoder Networks: An auto-encoder network con-sists of two main components: An encoder which takesas input Ns medical images denoted by f (i), for (1 ≤ i ≤Ns), and converts each into a latent space φ(W f (i)+b),i.e., Radiomics features. The second component, thedecoder, takes the latent space and tries to reconstructthe input image with the objective of minimizing the dif-ference between the original input and the reconstructedone φ(W Tφ(W f (i) + b) + c) [48] given by

minW ,b,c

Ns∑i=1

||φ(W Tφ

(W f (i) + b

)+ c)− f (i)||, (16)

where φ(·) is the network’s activation function; W de-notes the weight matrix of the network used by both theencoder and the decoder; Term b denotes the encoder’sbias vector; c is the decoder’s bias vector, and; superscriptT denotes the transpose operator. The reason that theencoded variables can be treated as Radiomics featuresis that they are the most important representatives of theinput image that can be used to reproduce it. Although anauto-encoder can be trained completely in an end-to-end

manner, to begin training with good initial weights andthus avoid the vanishing gradient problem, one can firsttrain layers one by one, and use the obtained weightsas the auto-encoder starting point [14]. Depending onthe application, Auto-encoders have several extensionsincluding:

2.1.1. Denoising Auto-Encoders (DAEs): To make auto-encoders capture more robust features of the input,one common strategy is to add some noise to theinput. This kind of auto-encoder is called a denois-ing auto-encoder (DAE) [14]. Reference [38] hasadopted DAE for extracting Radiomics features thatare fed to an SVM to classify lung tumors as benignor malignant. Reference [5] has also adopted a fivelayer denoising auto-encoder which takes the cor-rupted lung images as inputs and tries to recover theoriginal image. In particular, 400 Features extractedby the encoder part of this network are treated asRadiomics to train another neural network for lungcancer classification (identify the type of the tumorsuch as benign or malignant).

2.1.2. Convolutional Auto-Encoders (CAEs): This typeof auto-encoders are specially useful for Radiomics(image type inputs) as the spatial correlations aretaken into account. In these networks, nodes shareweights in a local neighborhood [14]. A CAE with5 Convolutional layer is adopted in Reference [6]for lung cancer diagnosis (identify the presence ofcancer).

There are two common strategies to leverage Auto-encoders in Radiomics:

– The first and most frequent approach is to directlyuse the extracted features to train a classifier. Forinstance, [48] has extracted Radiomics features usinga 5 layer auto-encoder, which receives the segmentedregion of interest as the input. These features gothrough a binary decision tree in the next stepto produce the output which is the classified lungnodule in this case.

– Auto-encoders can also serve as a pre-training stage

21

to make the network extract representative featuresbefore trying to perform the actual classification.For instance, Reference [41] has first trained a DAEbased on resized (down-sampled images to facilitatetraining) lung CT patches. In the next stage, a classi-fication layer is added to the network and the wholenetwork is re-trained taking both resized images andthe resizing ratio as inputs.

2.2. Deep Belief Networks (DBNs): DBNs are stack ofRestricted Boltzmann Machines (RBMs) on top of eachother where the RBM is an unsupervised two layerstochastic neural network that can model probabilisticdependencies with the objective of minimizing the re-construction error. More importantly RBM is a bipartitegraph allowing value propagation in both directions. Al-though DBNs are composition of RBMs, only the top twolayers have undirected relations. DBNs are first trainedin a greedy fashion meaning that RBM sub-networks aretrained individually followed by a fine-tuning phase [14].In Reference [5], a DBN consisting of 4 hidden layers isdesigned with the goal of extracting the DLR from the toplayer which has 1600 nodes. This last layer is connectedto an external neural network to classify lung nodules.Besides, to have multi-channel input (original image,segmented tumor, and gradient image), these channels areconcatenated vector wise before being fed to the network.

2.3. Deep Boltzmann Machine (DBMs): DBMs are alsobased on RBMs, but they differ from DBNs in the sensethat DBMs include undirected relations between all layerswhich makes them computationally ineffective, thoughthey are trained in a layer wise manner [14]. Due tothe two-way relations, however, RBMs can capture com-plicated patterns from the data [42]. DBMs are adoptedin [42] for Alzheimers disease diagnosis. In this work, aclassification layer is added to the last layer of the DBMallowing to extract not only hierarchical (generative) butalso discriminative features.

This completes an overview of different deep discriminativeand generative models used within the Radiomics workflow.Next, we consider a critical drawback of such architectures,i.e., acting as a black-box.

D. Explainability of Deep Learning-based Radiomics

Explainability of deep networks refers to revealing aninsight of what has made the model to come into a specificdecision, helping with not only improving the model by know-ing what exactly is going on in the network, but also detectingthe failure points of the model. No matter how powerful DLRare, they will not be utilized by physicians, unless they canbe interpreted and related to the image landmarks used bythe experts. Besides, not even a single mistake is allowed inmedical decisions as it may lead to a irreparable loss or injury,and having an explanation of the logic behind the outcomeof the deep net is the key to prevent such disasters. Thissubsection will present an overview on recently developedtechniques to increase the explainability of deep Radiomics.

One simple approach to ensure the accuracy of the au-tomatic prediction, is to double-check the results with anexpert. For instance, Reference [25], which has used a CNNfor longevity prediction using CT images, has reviewed theoutcomes with experts leading to the fact that people pre-dicted with longer lives are indeed healthier. However, thisapproach is time consuming and needs complete supervisionand investigation, which is in conflict with the concept ofautomatizing and personalized treatment, which is the wholepoint of Radiomics. Therefore, nowadays several criteria arebeing presented to reduce the time and complexity of ex-plaining deep Radiomics. One of these approaches is “featurevisualization” which tries to gain knowledge on the networkbehavior by visualizing what kinds of features the network islooking for. This technique can be applied to different layersof the model. For example, to visualize the first layer features,the associated filters are applied to the input and the resultingfeature maps are presented. However, as the last layer is themost responsible one in the network’s output, paying attentionto the features learned in this layer is more informative. Forinstance, Reference [5] has visualized the final wights of aDBN, showing that the network is looking for meaningfulfeatures such as curvity. Nevertheless, these features are notas meaningful as they are for simple image recognition tasksas clinicians themselves are sometimes unsure about the dis-tinctive properties of the images.

One other method to provide the user with an explanation onthe decision made by a deep architecture is called “sensitivityanalysis” referring to generating a heat-map highlighting theimage regions responsible for the output [24]. In the heat-map,the brighter areas are the ones that have influenced the predic-tion. This can be achieved by determining and measuring theeffect of changing each individual input pixel on the output. Ina CNN this effect can be estimated by determining the weightassociated with each input pixel through back propagation.This approach can discover the cause for the prediction [24],however, the drawback of this approach is that not all thedetected pixels through the heat-map are necessarily the onesleading to the specific decision, and besides, as the depth andcomplexity of the deep net increases, it becomes more difficultto measure the contribution of each individual pixel on theoutput.

A third proposed approach to understand the learned fea-tures is to project the high-dimensional feature space from thedeep network to a bi-dimensional plane. Reference [40] hasadopted this strategy by using t-Distributed Stochastic Neigh-bor Embedding (t-SNE) algorithm to visualize the featureslearned by a CNN for lung tumor classification. The resultedplane presents clearly defined clusters of lung tumors, whichshows that the networks has successfully learned discriminat-ing features. However, although this method can verify theaccuracy of the network, it does not provide information onthe exact reason behind making the decision.

The interpretability of meaningless weights is improved inthe newly proposed Capsule networks through reconstructingthe input image based on the features learned by the network.CapsNet includes a set of fully connected layers that take thefinal calculated features, based on which the final classification

22

Fig. 8: Effect of tweaking the final feature vector on the reconstructed brain tumor image. Each row corresponds to a single feature whichis tweaked 11 times.

is made, as inputs, and reproduce the original image with theobjective of minimizing the difference between the originaland the reconstructed image. This objective function is addedto the classification loss with a smaller weight not to distractnetwork from extracting discriminative features. If the trainedCapsNets is not only of high accuracy but also capableof resembling the input image, it has been successful inextracting representative features. Besides, visualizing thesefeatures provides insight on the explainability of the model.Interestingly, CapsNets are equipped with a powerful featurevisualization technique through their input reconstruction part,which works as follows:

1. If CapsNet is in the training phase, the feature vectorassociated with the true class label is selected, otherwise,the one with the higher probability is used.

2. The selected feature vector is tweaked, meaning thatsmall numbers are added to or subtracted from thefeature values leading to a slightly changed new featurevector.

3 The new feature vector is fed to the reconstructionpart and the input image is reproduced. However thisreconstructed image is not supposed to exactly resemblethe input image as it is generated using the tweakedfeatures not the actual ones learned by the network.

4. By repeating the process of tweaking and reconstruct-ing process over and over again, one can understandwhat features are learned by observing the influence ofchanging them on the generated images.

This strategy is adopted for explaining the output of a CapsNettrained to classify brain tumors, where it is shown that thenetwork is probably making the decision based on featuressuch as size and deformation. Fig. 8 shows the reconstructedbrain tumor images based on tweaked feature vectors helpingto gain insight on the nature of features.

This completes our discussion on deep learning-based Ra-diomics. Next, we consider hybrid solutions in which hand-crafted and deep Radiomics are jointly used.

V. HYBRID SOLUTIONS TO RADIOMICS

To summarize our findings on HCR and DLR features,Tables VII(a) and (b) provide different comparisons betweenthese two categories from various perspectives. In scenarioswhere neither of the above two mentioned categories arecapable of providing informative Radiomics features with highpredictive capacity, one can resort to hybrid strategies. Here,

potential hybrid solutions to Radiomics [49] are reviewed fromdifferent point of views including combination of Radiomicswith other data sources and combination of HCR and DLRfeatures.

A. Combination of Radiomics and Other Data Sources

Physicians, normally, do not rely on a single input for theirdiagnosis of diseases and disorders. To come into a conclusivedecision, inputs from different sources are compared and com-bined including Radiomics (image bio-markers from differentimaging modalities); Blood bio-markers; Clinical outcomes;Pathology, and; Genomics results [8]. In Information Post I,we have provided an overview of various Radiomics’ imagingmodalities and data sources which are typically combined withRadiomics features. Below, we discuss two different waysto fuse/combine Radiomics with other available resources ofinformation along with the rationales and potentials behindsuch combinations:

i. Extracting Radiomics from Different Imaging Modal-ities: As stated previously, Radiomics can be extractedfrom different imaging modalities each of which can onlycapture/provide particular information on tissues’ proper-ties. For instance, although the CT scan is among the mostcommon and informative imaging modalities allowing toobserve the body internal organs, the CT can not provideinformation on body function and metabolism. This typeof information is available through PET scan, which callsfor studying the effect of combining Radiomics extractedfrom different modalities. For example, in [1] Radiomicsfeatures are extracted from both CT and PET images,and the concatenated feature vector is fed to a classifierfor long cancer survival prediction, resulting in a higheraccuracy compared to each modality separately.

Combining different imaging modalities is also tested onbrain tumor classification in [23]. Since MRI can outputdifferent images varying mostly in terms of their contrastand relaxation parameters, these images can be fusedto provide complementary information. Based on [23],extracting Radiomics from this combination of MRIsoutperforms the single modal classifier for brain tumorclassification.

ii. Integration of Extracted Radiomics with Other DataSources: Radiomics features are combined with otherresources only after the extraction process. The bestdescriptive or predictive models in the field of Radiomics

23

TABLE VII: A Comparison between hand-Crafted and Deep Radiomics.Hand-Crafted Radiomics (HCR) Deep Radiomics (DLR)Needs a prior knowledge on types of features to extract. Can learn features on its own and without human intervention.

Features are typically extracted from the segmented ROI. does not necessarily require a segmented input.

It is generally followed by a feature selection algorithm. Feature selection is rarely performed.

As features are defined independent from the data, does not requirebig datasets.

Requires huge datasets, since it has to learn features from the data.

Processing time is not normally significant. Can have high computational cost depending on the architectureand size of the dataset.

Since features are pre-designed, they are tangible. The logic behind the features and decisions is still a black box.

are the ones that utilize not only imaging bio-markers,but also other information such as Genomics patterns andtumor histology [12]. In [2], it is reported that combiningRadiomics features with lung cancer staging information,which is obtained based on the tumor location and dis-persion, can improve the prognostic performance of Ra-diomics alone or staging alone. In other words, Radiomicscan provide a complementary information for lung cancerprognosis [9]. It is also shown that combining Radiomicswith other prognostic markers in head-and-neck cancerleads to a more inclusive decision. Combining Radiomicswith clinical data is further investigated in [29] for braincancer survival prediction. The interesting output of thiswork is a nomogram based on both Radiomics andclinical risk factors such as age that can be used tovisually calculate survival probability. Developing such anomogram is further investigated in references [20] and[47] for prediction of Hepatitis B virus and lymph nodemetastasis, respectively.In brief, the first step to build a Radiomics-based nomo-gram, is calculating a linear combination of selectedRadiomics features based on a logistic regression, whichresults in a Radiomics score to exploit further for thedesired prediction task. Consequently, by training a mul-tivariate logistic regression, Radiomics score is fusedwith other influential factors to make the final predic-tion. Fig. 9 presents the nomogram introduced in [47]along with an example illustrating how the lymph nodemetastasis prediction is made.

Reference [49] has adopted a fusion approach based onboth Radiomics and Genomics bio-markers for predictingthe recurrence risk of lung cancer. In this study, 35hand-crafted features are extracted from segmented lungCT images and reduced to 8 after a feature selectionphase. These features are then used to train a NaiveBayesian network. The same classifier is also trainedusing two Genomics bio-markers, and the outputs of twoclassifiers are fused through a simple averaging strategy.Results demonstrate that the combination of classifiersnot only leads to higher prediction accuracy comparedto individual ones, but also resembles the Kaplan-Meierplot of survival more precisely.

B. Fusion of HCR with DLR (i.e., Engineered Features Cou-pled with Deep Features)

As mentioned in Table VII, engineered (hand-crafted) anddeep Radiomics both have their own advantages and disadvan-tages. As a result, combining these features has the promiseof benefiting from both domains and incorporating differenttypes of features [27] potentially results in significantly im-proved performance. As shown in Fig. 10, the following twocategories of data fusion have been used in Radiomics mostrecently:

1. Decision-level Fusion: One common approach to combineHCR with DLR is to first use them separately to train separateclassifiers and then adopt a kind of voting between the outputsto make the final decision. The voting or fusion approaches inRadiomics include:

(i) Soft Voting, which is combining the probability outputs,for instance, through a simple averaging. Soft votingis adopted in [22], where two individual SVMs aretrained on hand-crafted and deep Radiomics features(extracted using a pre-trained CNN), and consequentlybreast cancer prediction is performed based on aver-aging the output probabilities. Results of this articleshows that the combined features are associated withhigher prediction accuracy. Fusion of separately trainedclassifiers through soft-voting based on deep and hand-crafted Radiomics is also examined in [32] for breastcancer classification, where it has been shown that thecombined SVM model outperforms individual classifiersin term of accuracy for mammogram, ultrasound, andMRI images.

(ii) Hard Voting, which is combining outputs, for example,through a majority vote.

(iii) Adaptive Voting, where a weight of importance for eachmodel (HCR and DLR) is learned for example using aseparate neural network. In Reference [33], a differentkind of voting is adopted for lung cancer classification.This voting is based on the idea that not all the classifierscontribute equally to the final decision, and contributionweights are parameters that should be optimized througha learning process. The aforementioned article has firsttrained a CNN and several traditional classifiers such

24

Fig. 9: Radiomics-based nomogram to predict lymph node metastasis. Tumor position is considered as extra information to assist withmaking a more reliable prediction.

Fig. 10: Combining deep and hand-crafted Radiomics through feature-level or decision-level fusion.

as SVM and logistic regression to independently predictthe type (benign or malignant) of lung cancer. Predic-tions are consequently utilized to train a second-stageclassifier to generate the final outcome. Any classifiersuch as SVM and NN can be used as the second-stageclassifier.

2. Feature-level Fusion: Second widely used approach tocombine deep and hand-crafted Radiomics is to first concate-nate the feature vectors and then feed them to a classifier,referred to as feature-level fusion [38]. Reference [26] hasshown that this combination lead to the highest performancein lung cancer survival prediction using Random Forest andNaive Bayes classifiers. The efficiency of this approach is alsoverified in [27] for lung tumor detection. Although mixingdeep and hand-crafted Radiomics has several advantageous

such as ensuring the heterogeneity of the features, it maycause over-fitting as the number of training data is relativelyless than the number of features. Therefore, Reference [29]has examined this large set of features in terms of stability,informativeness, and redundancy leading to a dramatic dimen-sion reduction and increase in the accuracy of brain cancersurvival prediction. To further reduce the number of features,a Cox regression is adopted that can determine the impact offeatures on survival, and as a result, those with small weightscan be removed as effectless.

Reference [34] has leveraged the idea of concatenatingdeep and hand-crafted features for lung tumor attributes (suchas spiculation, sphericity and malignancy) scoring through amulti-task learning framework. For extracting deep features, 9CNNs corresponding to each of the 9 task at hand, and a 3layer DAE are trained, where each CNN generates 192 Ra-

25

diomics features extracted from the last fully connected layerbefore the SoftMax layer, and DAE results in 100 features.Deep features are further combined with hand-crafted featuresconsisting of Haar and Histogram of oriented gradients (HoG)features, and the resulting vector is used as input to a multi-task linear regression model, which can consider the inter-task relations, in order to calculate the score of each of the 9lung cancer attributes. This completes our discussion on hybridRadiomics.

VI. CHALLENGES, OPEN PROBLEMS, ANDOPPORTUNITIES

In this section, we will focus on the limitations of process-ing techniques unique in nature to the Radiomics, and thenintroduce open problems and signal processing opportunitiesas outlined in the following subsections.

A. Challenges of Had-Crafted Radiomics

Extraction of hand-crafted features in Radiomics is moreproblematic in comparison to multimedia domains as, typi-cally, very limited distinct visual variation exists to differenti-ate, for example, cancerous tissues. To address this issue, thecommon approach in Radiomics is to first extract hundredsand hundreds of different low level and high level featureswithout devising a systematic mechanism and taking intoaccount the end goal. Then, to resolve the resulting curse ofdimensionality, simple and basic reduction techniques (e.g.,basic principal component analysis (PCA)) are used. Anothermajor issue with existing hand-crafted features in Radiomics(opportunity for signal processing researchers) is that they areextracted without using the information that can be obtainedfrom other sources of data such as gene expressions andclinical data, which further limits their discriminative abilitiesfor cancer prediction.

More importantly, most of the hand-crafted Radiomicsrequire the segmented ROI. Providing annotations might notbe a significant problem in other multi-media domains, asthis project can be easily crowd sourced. However, when itcomes to Radiomics, only experts have the ability to providesegmentations, which is both time and cost ineffective.

B. Challenges of Deep Radiomics

Although deep Radiomics has several advantages includingits generalization capability and its independence from the su-pervision of experts, it is also associated with some advantagessuch as its need for huge data sets, and its lack of robustnessto some kinds of transformation. Besides, another importantchallenge with deep learning-based Radiomics is that there isstill no strategy to choose the optimum architecture.

Establishing the requirements of an appropriate deep ar-chitecture for Radiomics is a main challenge in developmentof DLR and another venue for future works. In particular,performing sensitivity analysis is a critical step to explain theconnection between the designed choices and the achievedresults. It is essential to identify different conditions underwhich the results are obtained. In short, what is happening

when a specific architecture is used? For example, considerthe simple issue of choosing the size of the input image. Thequestion here is if one should provide the original image sizeor downsize the image and why? One simple answer could bethat downsizing is applied for computational savings. Anotherintuitive answer could be that down sampling makes the datainvariant to translation and rotation, i.e., when the image isdown sampled, we take averages and as such the model will beless prone to outliers. The latter intuition could be a reasonableidea for natural images but is this the case for Radiomics-basedfeatures obtained from medical images?

Intuitively speaking, a fundamental challenge and an openproblem for development of DLR is that the models andsolutions developed for natural images should be modifiedbefore being applied to medical images as the nature of theseimages are totally different (e.g., consider MRI images). Asthe nature of the input signal is different, parts of the modelhas to change but this issue has not yet been investigatedsystematically and thoroughly.

It is worth mentioning that while hand-crafted Radiomicsnormally need less amount of data, the number of imagesrequired for effective training of deep architectures, suchas the CNNs, depends on the complexity of the underlyingmodel. In other words, the more the number of trainableparameters, the more amount of training data needed. In thecase of lung cancer diagnosis, e.g., the LIDC Dataset [15]is widely used, which consists of 244,527 CT images of1010 patients. Normally, training complex deep models basedon such dataset cannot be performed in a timely fashionusing common processors and one needs to resort to one ormore GPU processors. One common approach to reduce thecomputational cost in deep learning methods is to crop theinput image to include just the object of interest. Although thisapproach does not cause any information loss in non-medicalareas, it is harmful for classifying medical images as size isan important discriminator feature of, for instance, normal andabnormal tissues. This problem is investigated in [7], where itis shown that the miss-classified lung tumors based on featuresextracted from a DBN are 4% larger than correctly classifiedones, possibly because the cropped images do not containsize information. On the other hand, although limited accessto training data is a common problem in other multimediadomains, it becomes significantly more critical in Radiomicsas typically access to patients data is subject to several ethicalregulations making it hard to collect the required amount ofdata for training purposes.

Finally, interpretability of deep Radiomics is of paramountimportance as human lives are at stake, and without providingappropriate explainability, utilization of deep-Radiomics inclinical practice will remain limited. The relation between deepfeatures and genetic patterns is also not established yet andrequires further studies.

C. Open Problems and Signal Processing OpportunitiesDespite recent advancements in the field of Radiomics and

increase of its potential clinical applications, there are stillseveral open problems which require extensive investigationsincluding:

26

1. Most Radiomics models a need rich amount of trainingimages, however, due to strict privacy policies, medicalimages are usually hard to collect.

2. Even without considering the privacy issues, it is difficultto find the required amount of data with similar clinicalcharacteristics (e.g., corresponding to the same cancerstage).

3. Radiomics analysis need ground truth which is scarceas labels can only be provided by clinical experts (thisis in contrary to other multimedia domains). This callsfor development of weakly or semi-supervised solutionstaking into account the specifics of the Radiomics do-main.

4. Properties of medical images such as their contrastand resolution varies significantly from one institute toanother (from one dataset to another), because eachinstitute may use different types of scanners and/or usedifferent technical parameters. Development of noveland innovative information fusion methodologies to-gether with construction of unifying schemes are crit-ical to compensate for lack of standardization in thisfield and produce a common theme for comparing theRadiomics results. Furthermore ground truth and annota-tions provided by different experts can vary significantlyas experts, depending on their area of specialty (suchas oncology and surgery), may consider and look fordifferent details and landmarks in the image.

5. Unbalanced data refers to a problem where classes arenot equal in a classification problem rendering the classi-fier biased toward the majority class. This is almost all ofthe time the case for Radiomics analysis as the numberof positive classes (existence of disease) is typicallysmaller than the negative ones. Therefore, proper careis needed when working with medical data. Althoughseveral solutions, such as modifying the metric functionto give more weight to minority class, are provided todeal with the aforementioned issue, it is still an unsolvedproblem that needs further investigations.

6. Dealing with image noise is another challenging prob-lem, which is common in all multi-media domains,but it is more severe in Radiomics as there may bemore unpredictable sources of variation in medicalimaging. As an example, patient’s breathing in the CTscanner can cause change of lung tumor location inconsecutive slices bringing about difficulty in extractingstable Radiomics features. Therefore to achieve reliablepersonalized diagnosis and treatment, careful strategiesshould be developed to address the effects of these kindsof variations.

7. The biggest challenge in combining various data sources(such as imaging and clinical) is that not all data isprovided for all the patients. In other words, Radiomicsanalysis model should be equipped with the ability towork with sparse data [12]. Besides, the currently usedfusion strategies within the Radiomics are still in theirinfancy and development of more rigorous fusion rulesis necessary. For instance, feature-level fusion results ina vector and how to sort/combine the localized feature

vectors is an open challenge. Giving the superiority ofthe initial results obtained from hybrid Radiomics, thisissue becomes an urgent matter calling for advancedmultiple-model solutions.

VII. CONCLUSION

During the past decades, medical imaging made significantadvancements leading to the emergence of automatic tech-niques to extract information that are hidden to human eye.Nowadays, the extraction of quantitative or semi-quantitativefeatures from medical images, referred to as Radiomics, playan important role in clinical care especially for cancer diag-nosis/prognosis. There are several approaches to Radiomicsincluding extracting hand-crafted features, deep features, andhybrid schemes. In this article, we have presented an integratedsketch on Radiomics by introducing practical applicationexamples; Basic processing modules of the Radiomics, and;Supporting resources (e.g., image, clinical, and genomic datasources) utilized within the Radiomics pipeline, with the hopeto facilitate further investigations and advancements in thisfield within signal processing community.

REFERENCES

[1] A. Oikonomou, F. Khalvati, et al., “Radiomics Analysis at PET/CTContributes to Prognosis of Recurrence and Survival in Lung CancerTreated with Stereotactic Body Radiotherapy,” Scientific Reports, vol.8, no. 1, 2018.

[2] H.J. Aerts, E.R. Velazquez, et al., “Decoding Tumour Phenotypeby Noninvasive Imaging Using a Quantitative Radiomics Approach,”Nature Communications, vol. 5, 2014.

[3] Y. Zhang, A. Oikonomou, et al., “Radiomics-based Prognosis Analysisfor Non-small Cell Lung Cancer,” Scientific Reports, vol. 7, 2017.

[4] J. Griethuysen, A. Fedorov, et al. “Computational Radiomics Systemto Decode the Radiographic Phenotype,” Cancer Research, vol. 77, no.21, pp. 104-107, 2017.

[5] W. Sun, B. Zheng, et al. “Automatic Feature Learning using Multichan-nel ROI based on Deep Structured Algorithms for Computerized LungCancer Diagnosis,” Computers in Biology and Medicine, vol. 89, no. 1,pp. 530-539, 2017.

[6] O. Echaniz, and M. Grana, et al., “Ongoing Work on Deep Learningfor Lung Cancer Prediction,” Biomedical Applications Based on Naturaland Artificial Computing, vol. 10338, pp. 42-48, 2017.

[7] W. Sun, B. Zheng, et al., “Computer Aided Lung Cancer Diagnosiswith Deep Learning Algorithms,” Proc.SPIE, vol.9785, 2016.

[8] P. Lambin, E. Rios-velazquez, et al., “Radiomics: Extracting moreInformation from Medical Images using Advanced Feature Analysis,”European Journal of Cancer, vol. 48, no. 4, pp. 441-446, 2012.

[9] V. Kumar, Y. Gu, et al., “Radiomics: The Process and the Challenges,”Magnetic Resonance Imaging, vol. 30, no. 9, pp. 1234-1248, 2012.

[10] R. Thawani, M. McLane, et al., “Radiomics and Radiogenomics in LungCancer: A Review for the Clinician ,” Lung cancer, vol. 115, pp. 34-41,2017.

[11] J. Tian, D. Dong, et al., “Radiomics in Medical Imaging-Detection,Extraction and Segmentation,” Artificial Intelligence in Decision SupportSystems for Diagnosis in Medical Imaging, vol. 140, pp 267-333, 2018.

[12] R. Gillies, P. Kinahan, et al., “Radiomics: Images Are More thanPictures, They Are Data.,” Radiology, vol. 278, No. 2, pp. 563-577,2016.

[13] J. Edwards, “Medical Optical Imaging: Signal Processing Leads to NewMethods of Detecting Life-Threatening Situations [Special Reports],”IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 17-20, 2017.

[14] D. Ravi, C. Wong, et al., “Deep Learning for Health Informatics,” IEEEJournal of Biomedical and Health Informatics, vol. 21, no. 1, pp. 4-21,2017.

[15] S. Armato, G. McLennan, et al., “Data From LIDC-IDRI. The CancerImaging Archive”

[16] S. Sabour, N. Frosst, G.E. Hinton, “Dynamic Routing Between Cap-sules,” Conference on Neural Information Processing Systems (NIPS),pp. 3859-3869, 2017.

27

[17] P. Afshar, A. Mohammadi, K.N. Plataniotis, “Brain Tumor TypeClassification via Capsule Networks,” Accepted in IEEE InternationalConference on Image Processing (ICIP), 2018.

[18] C. Parmar, “ Machine learning applications for Radiomics: Towardsrobust non-invasive predictors in clinical oncology ,” MaastrichtUniversity, 2017.

[19] A. Subramanian, P. Tamayo, et al., “ Gene Set Enrichment Analysis: Aknowledge-based Approach for Interpreting Genome-Wide ExpressionProfiles,” Proceedings of the National Academy of Sciences, vol. 102,no. 43, pp. 15545-15550, 2005.

[20] J. Peng, J. Zhang, et al., “ A radiomics nomogram for preoperativeprediction of microvascular invasion risk in hepatitis B virus-relatedhepatocellular carcinoma,” Diagnostic and Interventional Radiology,vol. 24, no. 3, pp. 121-127, 2018.

[21] D. Kumar, A. Chung, et al., “ Discovery Radiomics for Pathologically-Proven Computed Tomography Lung Cancer Prediction,” Image Anal-ysis and Recognition, pp. 54-62, 2017.

[22] B. Huynh, H. Li, et al., “ Digital Mammographic Tumor ClassificationUsing Transfer Learning from Deep Vonvolutional Neural Networks,”Journal of Medical Imaging, vol. 3, no. 3, 2016.

[23] Z. Li, Y. Wang, et al., “ Deep Learning Based Radiomics (DLR) andIts Usage in Noninvasive IDH1 Prediction for Low Grade Glioma,”Scientific Reports, vol. 7, no. 1, 2017.

[24] A. Jamaludin, T. Kadir, et al., “ SpineNet: Automatically PinpointingClassification Evidence in Spinal MRIs,” International Conference onMedical Image Computing and Computer-Assisted Intervention, vol.9901, pp. 166-175, 2016.

[25] L. Oakden-Rayner, G. Carneiro, et al., “ Precision Radiology: PredictingLongevity Using Feature Engineering and Deep Learning Methods in aRadiomics Framework,” Scientific Reports, vol. 7, no. 1, 2017.

[26] R. Paul, S. Hawkins, et al., “ Deep Feature Transfer Learning inCombination with Traditional Features Predicts Survival Among Patientswith Lung Adenocarcinoma,” Tomography: a journal for imagingresearch, vol. 2, no. 4, pp. 388-395, 2016.

[27] L. Fu, J. Ma, et al., “ Automatic Detection of Lung Nodules:False Positive Reduction Using Convolutional Neural Networks andHandcrafted Features,” Proc.SPIE, vol. 10134, 2017.

[28] S. Liu, H. Zhengr, et al., “ Prostate Cancer Diagnosis Using DeepLearning with 3D Multiparametric MRI,” Proc.SPIE, vol. 10134, 2017.

[29] J. Lao, Y. Chen, et al., “ A Deep Learning-Based Radiomics Model forPrediction of Survival in Glioblastoma Multiforme,” Scientific Reports,vol. 7, no. 1, 2017.

[30] W. Shen, M. Zhou, et al., “ Multi-scale Convolutional Neural Networksfor Lung Nodule Classification,” International Conference on Informa-tion Processing in Medical Imaging, 588-599, 2015.

[31] M. Liu, J. Zhang , et al., “ Anatomical Landmark based Deep FeatureRepresentation for MR Images in Brain Disease Diagnosis,” IEEEJournal of Biomedical and Health Informatics, 2018.

[32] N. Antropova, B. Huynh, et al., “ A Deep Feature Fusion Methodologyfor Breast Cancer Diagnosis Demonstrated on Three Imaging ModalityDatasets,” The International Journal of Medical Physics Research andPractice, vol. 44, no. 10, pp. 5162-5171, 2017.

[33] S. Liu, Y. Xie, et al., “ Pulmonary Nodule Classification in Lung CancerScreening with Three-Dimensional Convolutional Neural Networks,”Journal of Medical Imaging, vol. 4, no. 4, 2017.

[34] S. Chen, J. Qin, et al., “ Automatic Scoring of Multiple SemanticAttributes With Multi-Task Feature Leverage: A Study on PulmonaryNodules in CT Images,” IEEE Transactions on medical imaging, vol.36, no. 3, pp. 802-814, 2017.

[35] H. Wang, Z. Zhou, et al., “ Comparison of Machine Learning Methodsfor Classifying Mediastinal Lymph Node Metastasis of Non-Small CellLung Cancer from 18F-FDG PET/CT Images,” EJNMMI Research, vol.7, no. 1, 2017.

[36] W. Shen, M. Zhou, et al., “ Multi-crop Convolutional Neural Networksfor Lung Nodule Malignancy Suspiciousness Classification,” PatternRecognition, vol. 61, pp. 663-673, 2017.

[37] K. Liu, G. Kang, “ Multiview Convolutional Neural Networks for LungNodule Classification,” International journal of imaging systems andtechnology, vol. 27, no. 1, pp. 12-22, 2017.

[38] B. Kim, Y. Sung, et al., “ Deep Feature Learning for Pulmonary NoduleClassification in a Lung CT,” 2016 4th International Winter Conferenceon Brain-Computer Interface (BCI), 2016.

[39] W. Shen, M. Zhou, et al., “ Learning from Experts: DevelopingTransferable Deep Features for Patient-Level Lung Cancer Prediction,”International Conference on Medical Image Computing and Computer-Assisted Intervention, pp, 124-131, 2016.

[40] F. Ciompi, K. Chung, et al., “ Towards Automatic Pulmonary NoduleManagement in Lung Cancer Screening with Deep Learning,” ScientificReports, vol. 7, 2017.

[41] J. Cheng, D. Ni, et al., “ Computer-Aided Diagnosis with DeepLearning Architecture: Applications to Breast Lesions in US Imagesand Pulmonary Nodules in CT Scans,” Scientific Reports, vol. 6, 2016.

[42] H. Suk, S. Lee, et al. “ Hierarchical Feature Representation and Multi-modal Fusion with Deep Learning for AD/MCI Diagnosis,” Neuroimage,vol. 101, 2014.

[43] S. Azizi, S.Bayat, et al., “ Deep Recurrent Neural Networks for ProstateCancer Detection: Analysis of Temporal Enhanced Ultrasound,” IEEETransactions on Medical Imaging, 2018.

[44] V. Parekh, M. Jacobs, “ Radiomics: a new application from establishedtechniques,” Expert Review of Precision Medicine and Drug Develop-ment, vol. 1, no. 2, pp. 207-226, 2016.

[45] Z. Akkus, I. Ali, et al., “ Predicting Deletion of ChromosomalArms 1p/19q in Low-Grade Gliomas from MR Images Using MachineIntelligence,” Journal of Digital Imaging, vol. 30, no. 4, pp. 469-476,2017.

[46] M. Vallieres, E. Kay-Rivest, et al., “ Radiomics Strategies for RiskAssessment of Tumour Failure in Head-and-Neck Cancer,” ScientificReports, vol. 7, no. 1, 2017.

[47] C. Shen, Z. Liuet al., “Building CT Radiomics Based Nomogramfor Preoperative Esophageal Cancer Patients Lymph Node MetastasisPrediction,” Translational Oncology, vol. 11, no. 3, pp. 815-824, 2018.

[48] D. Kumar, A. Wong, et al., “Lung Nodule Classification Using DeepFeatures in CT Images,” International Conference on Computer andRobot Vision, 2015.

[49] N. Emaminejad, W. Qian, et al. “Fusion of Quantitative Image andGenomic Biomarkers to Improve Prognosis Assessment of Early StageLung Cancer Patients,” IEEE Transactions on Biomedical Engineering,vol. 63, no. 5, pp. 1034-1043, 2016.

[50] B. Menze, A. Jakab, et al., “The Multimodal Brain Tumor ImageSegmentation Benchmark (BRATS),” IEEE Transaction on MedicalImaging, vol. 34, no. 10, 2015.

Parnian Afshar is a Ph.D. candidate at Concordia Institute forInformation System Engineering (CIISE). Her research interestsinclude signal processing, biometrics, image and video processing,pattern recognition, and machine learning. She has extensive re-search/publication record in medical image processing related areas.

Arash Mohammadi is an Assistant Professor with Concordia In-stitute for Information System Engineering (CIISE), at ConcordiaUniversity. He is the Director-Membership Services of IEEE SignalProcessing Society (SPS), General co-chair of the Symposium on“Advanced Bio-Signal Processing and Machine Learning for MedicalCyber-Physical Systems,” under IEEE GlobalSIP 2018, and is theorganizer of the 2018 IEEE SPS Video and Image Processing (VIP)Cup. He was the General co-chair of the Symposium on “AdvancedBio-Signal Processing for Rehabilitation and Assistive Systems,” un-der IEEE GlobalSIP 2017, and General Co-Chair of 2016 IEEE SPSWinter School on “Distributed Signal Processing for Secure CyberPhysical Systems”. His research interests include statistical signalprocessing, biomedical signal processing, image/video processing,and machine learning.

Konstantinos N. Plataniotis Bell Canada Chair in Multimedia, is aProfessor with the ECE Department at the University of Toronto. Hiscurrent research interests are: machine learning, adaptive systems andpattern recognition, image and signal processing, communicationssystems, and big data analytics. He is a registered professional engi-neer in Ontario, Fellow of the IEEE and Fellow of the EngineeringInstitute of Canada. Dr. Plataniotis was the IEEE Signal ProcessingSociety inaugural Vice President for Membership (2014-2016) andthe General Co-Chair for the IEEE GlobalSIP 2017 (November2017, Montreal, Q.C.). He co-chairs the 2018 IEEE InternationalConference on Image Processing (ICIP 2018), October 7-10, 2018,Athens Greece, and 2021 IEEE International Conference in Acous-tics, Speech & Signal Processing (ICASSP 2021), Toronto, Canada.

Anastasia Oikonomou, MD, Ph.D., is the head of the CardiothoracicImaging Division at Sunnybrook Health Science Centre; Site Director

28

of the Cardiothoracic Imaging Fellowship program at University ofToronto, and an Assistant Professor with the Department of MedicalImaging at the University of Toronto. Her research interests includeimaging of pulmonary malignancies and interstitial lung diseases,Radiomics and machine learning methods in imaging of pulmonarydisease.

Habib Benali is a Canada Research Chair in “Biomedical Imagingand Healthy Aging” supported by the Natural Sciences and Engi-neering Research Council of Canada (NSERC). He is the InterimScientific Director of the PERFORM Centre, and a Professor withthe Department of Electrical and Computer Engineering at ConcordiaUniversity. Dr. Benali was the director of Unit 678, Laboratory ofFunctional Imaging of the French National Institute of Health andMedical Research (INSERM) and Paris 6 University (UPMC). Hewas also the director of the International Laboratory of Neuroimagingand Modelisation of the INSERM-UPMC and Montreal University.His research interests include biological signal processing, neurologyand radiology, biomedical imaging, biostatistics, and bioinformatics.