1

Dynamic radiomics: a new methodology toextract quantitative time-related features from

tomographic imagesFengying Che, Ruichuan Shi, Jian Wu, Haoran Li, Shuqin Li, Weixing Chen, Hao Zhang, Zhi Li, and

Xiaoyu Cui, Member, IEEE

Abstract—The feature extraction methods of radiomics are mainly based on static tomographic images at a certain moment, while theoccurrence and development of disease is a dynamic process that cannot be fully reflected by only static characteristics. This studyproposes a new dynamic radiomics feature extraction workflow that uses time-dependent tomographic images of the same patient,focuses on the changes in image features over time, and then quantifies them as new dynamic features for diagnostic or prognosticevaluation. We first define the mathematical paradigm of dynamic radiomics and introduce three specific methods that can describe thetransformation process of features over time. Three different clinical problems are used to validate the performance of the proposeddynamic feature with conventional 2D and 3D static features.

Index Terms—dynamic radiomics, static radiomics, feature extraction, breast cancer, gene mutation, neoadjuvant chemotherapy

F

1 INTRODUCTION

IN the past decade, tomography imaging technologies(computed tomography (CT), magnetic resonance (MR),

and positron emission tomography (PET)) have been widelyused in clinical diagnosis, treatment planning and prognosisevaluation [1, 2]. Traditional experience-based diagnosis iseasily influenced by subjective factors [3]. Thus, the quan-titative analysis method based on radiomics has receivedwidespread attention, through which a large number ofhigh-throughput quantitative imaging features can be ex-tracted and analyzed to carry out clinical decision making[4-6].

The core of radiomics is the extraction of high-dimensional feature data to quantitatively describe theproperties of the region of interest (ROI). In general, ex-tracted features mainly include first-order statistical fea-tures, texture features, shape features, and wavelet features.First-order statistics are derived from the histogram of voxelintensities [7], and statistical information such as the mean,median, skewness, and kurtosis can be calculated [8]. Tex-ture features are global features that describe the structuralproperties between voxels. For example, the gray-level co-

• F. Che, H. Li, S. Li, W. Chen and X. Cui are with College of Medicine andBiological Information Engineering, Northeastern University, Shenyang,Liaoning, China. X. Cui is with Key Laboratory of Intelligent Computingin Medical Image, Ministry of Education. J. Wu is with Key Laboratoryof Data Analytics and Optimization for Smart Industry, NortheasternUniversity, Shenyang, Liaoning, China. R. Shi, Z. Li are with Departmentof Medical Oncology, the First Hospital of China Medical University,Liaoning 110001, China. H. Zhang is with Department of Breast Surgery,Liaoning Cancer Hospital and Institute, Cancer Hospital of China MedicalUniversity, Shenyang 110042, Liaoning, China.

• F. Che and R. Shi have the same contribution to this paper.

• X. Cui(E-mail: cuixy@bmie.neu.edu.cn) and Z. Li(Email:zli@cmu.edu.cn) are corresponding authors.

occurrence matrix (GLCM) [9], gray-level run length matrix(GLRLM) [10], gray-level size zone matrix (GLSZM) [11],and neighborhood gray-level different matrix (NGLDM)[12] have been proposed. Shape features have two represen-tations: one is the contour feature, and the other is the re-gion feature, which includes the area, perimeter, roundness,centroid, smallest rectangle containing the area of the mass,etc. Wavelet features refer to the characteristics of differentfrequency bands extracted from the wavelet decompositionof the image [3].

Acquiring more radiomics features is helpful to identifypotential influencing factors that may be effective in pre-dicting results. Until now, there has been a sustained effortto identify, define, and extract more radiomics features. Wuet al. [13] developed a sparse representation-based featureextraction method that exploits the statistical characteristicsof the lesion area, which could be used for the outcomeprediction of higher-grade gliomas in the future [14]. Basedon graph theory, the features extracted by Zhou et al.[15] can specifically represent PET image characteristics.Moreover, the extraction of tomography image features canalso be extended to the three-dimensional domain [16]. Forsome specific applications, researchers have found that 3Dfeatures can achieve better performance than 2D features[17]. Meanwhile, 2D and 3D features are often used incombination and have shown a better performance than thatfrom 2D or 3D alone [18]. These features can potentially beextracted from individual habitats, thereby yielding thou-sands of data elements with which to describe each volumeof interest, with many volumes of interest available for eachpatient [19].

Regardless of which feature extraction technology isadopted, the data processed by the existing methods aremainly based on static tomographic images at a certainmoment. However, the occurrence and development of tu-

arX

iv:2

011.

0045

4v2

[ee

ss.I

V]

21

Dec

202

0

2

mors is a dynamic process that cannot be fully reflectedby only static characteristics. In some clinical applications,time-series images or images from multiple periods arerequired for diagnosis or prognosis. On the one hand,current research analysis shows that pharmacokinetics issuitable for the clinical analysis and prediction of a varietyof cancers [20-22]; thus, dynamic contrast-enhanced mag-netic resonance imaging (DCE-MRI) is commonly used toanalyze the metabolic processes of the tumor in patientssuspected of having breast cancer or prostate cancer [23,24].On the other hand, the size and shape of the tumor canbe detected by tomography directly; thus, imaging changesat different treatment stages are also used to assess theefficacy of radiotherapy or neoadjuvant chemotherapy incancer patients [25]. Recently, Carvalho et al. proposed“delta radiomics”, which can express the rate of change of aradiomics feature over time [26]. This approach can provideadditional information to identify, quantify, and potentiallypredict treatment-induced changes during treatment andhas been shown to have potential in evaluating the efficacyof colorectal [25], liver [27], pancreatic [28] and lung cancers[29]. However, these time-related diagnostic procedures arestill in the initial stage, so a rigorous model is still lackingand the procedures have not been well established in theexisting radiomics techniques.

In this study, we propose a novel dynamic radiomicsfeature extraction workflow that can take advantage oftime-related tomography images of the same patient bypaying attention to the changes in image features over timeand then quantifying them into new dynamic features fordiagnosis and prognosis assessment. First, we introduce thebasic framework of static radiomics feature extraction andthen propose the definition of dynamic radiomics features.In addition, we also introduce three specific methods thatcan describe the transformation process of features overtime. A total of 241 samples from three different clinicalproblems are used to validate the performance of the pro-posed feature extraction method.

The first problem is the prediction of axillary lymphnode metastasis (ALNM) in breast cancer, which is oneof the most important elements that can affect prognosis[30]. Many efforts have been made to develop a predictionmodel based on static radiomics features, such as usingT2-weighted images of MRI [31] or the first phase of T1-DCE images [32]; however, it is difficult for the predictiveaccuracy to meet clinical requirements. In this paper, weexploit the improved prediction performance based on thecomplete time-series images of DCE-MRI by using dynamicradiomics features.

The second problem is the prediction of gene mutationstatus as a noninvasive strategy. Previous studies [33] havereported that resistance to the anti-EGFR antibody cetux-imab was caused by a clone with a pre-existing KRASmutation and/or a mutation generated during treatment.Both mechanisms of resistance may occur concomitantly.Radiomics features can be used for detection multiple timesduring the whole treatment process, especially at each keytreatment node. In this way, the changes in gene mutationstatus can be tracked at any time to track the changes intumor heterogeneity and guide clinical decisions. In thisstudy, we aimed to construct a model based on radiomic

features obtained from multiple phases to improve the non-invasive assessment of RAS and BRAF mutations in patientswith colorectal cancer liver metastasis (CRLM) prior to anytreatment.

The third problem is the prediction of neoadjuvantchemotherapy. Clinically, it is of great significance to predictthe efficacy of neoadjuvant chemotherapy, and the abilityto identify nonresponders early may allow the selection ofpatients who may benefit from a therapy change. Currentstudies [34] have shown that some pharmacokinetic param-eters are significantly related to the efficacy of neoadjuvantchemotherapy. In addition, its curative effect can be wellpredicted through traditional static radiomics methods [35].In this study, we used public data [36] to verify the dynamicradiomics method.

The remainder of the paper is organized as follows.Section II provides some background on the static ra-diomics features. Section III presents the proposed time-related dynamic radiomics feature extraction method, whichcontains discrete time-related feature extraction, integratedtime-related feature extraction and parameter fitting featureextraction. Section IV reports the experimental results of theproposed method for differentiating sentinel lymph node(SLN) metastasis of breast cancer, for estimating liver cancergene mutations and for predicting the effects of neoadjuvantchemotherapy. Section V concludes the proposed method.

2 STATIC RADIOMICS FEATURESAt present, radiomics is mostly based on medical tomo-graphic images at a particular moment to extract the char-acteristics of the ROI. Therefore, in this study, we describethe existing radiomics technology as static radiomics, andits corresponding radiomics features can be specifically ex-pressed as:

F = {ψ(x(t)) ∈ R|x(t) ∈ Rm×n×p, t ∈ R+} (1)

where x(t) represents the area of interest extracted fromthe tomographic images collected at time t, the function ψdenotes the feature extraction method (including first-orderstatistical features, texture features, shape features, complexfeatures, etc.), and p represents the number of layers in thetomographic images. When p is equal to one, 2D featuresare extracted; when p is greater than one, 3D features areextracted. Suppose the number of extracted features is q.The feature extraction process of traditional static radiomicsis essentially a mapping process from m×n×p dimensionaltensor to q dimensional space.

3 DYNAMIC RADIOMICS FEATURESIn this study, we propose the concept of dynamic radiomicsfor the first time. Its purpose is to construct new time-related features that can describe the change rule by takingadvantage of the static characteristic changes at differenttime points, which can be expressed as:

φ(ψ(x(t1)), ψ(x(t2)), · · · , ψ(x(tk))) (2)

where φ(·) denotes the transformation from Rk to Rd;here, d is the number of dynamic features that can be

3

extracted. According to the number of time points collectedand the method of feature extraction, we propose threekinds of methods for calculating time dimension features,namely, integrated feature, discrete feature and parameterfeature.

3.1 Integrated Feature

According to the statistical analysis model, this method usesthe characteristics of each time point to build a model thatcan describe the overall change rule of the characteristicsas the characteristics of the time dimension. For example,φ(x1, x2, . . . , xk) can denote the statistical function of a sam-ple, such as the mean, variance, or coefficient of variation.

When

φ(x1, x2, · · · , xk) =1

k

k∑i=1

xi (3)

after the transformation of the function φ(·), the dynamicfeatures can be expressed as the average of the features atall times:

φ(ψ(x(t1)), ψ(x(t2)), · · · , ψ(x(tk))) =1

k

k∑i=1

ψ(x(ti)) (4)

When

φ(x1, x2, · · · , xk) =1

k

k∑i=1

|xi −1

k

k∑i=1

xi| (5)

the dynamic features can be express as:

φ(ψ(x(t1)), ψ(x(t2)), · · · , ψ(x(tk)))

=1

k

k∑i=1

|ψ(x(ti))−1

k

k∑i=1

ψ(x(ti))|(6)

Here, φ(·) can also be other types of statistical functions.In the process of tumor diagnosis or treatment, differentimaging features change constantly with drug metabolismor treatment progress. Integrated features are used to quan-titatively describe the overall law of these changes. Forexample, delta radiomics uses variance to evaluate the rateof change, which belongs to the category of integratedfeatures.

3.2 Discrete Feature

If φ(·) denotes the transformation from Rk to Rd, whered = k(k − 1)/2, then the transformation can be representedby the following operator:M = (mij)k×k , which is a matrixof k × k dimensions; g(·, ·)R2 → R1 is a function, where

mij = g(ψ(x(ti)), ψ(x(tj))), 1 ≤ i, j ≤ k (7)

Define P as an operation that takes the characteristicelements of the upper triangular region that do not containdiagonals of M and then straightens them; thus, we havePM ∈ Rk(k−1)/2, which can be used as dynamic features.For example, when g(x, y) = |x−y|/y, the dynamic featurescan be expressed as:

φ =(|ψ(x(t1))− ψ(x(t2))|

ψ(x(t2)),|ψ(x(t1))− ψ(x(t3))|

ψ(x(t3)),

· · · , |ψ(x(tk−1))− ψ(x(tk))|ψ(x(tk))

)

(8)

This method uses the interaction effect of the characteris-tics of two points in time, constructs the function to describethe characteristic change rule of two points, and runs thetwo points through the whole time point. Thus, the discretefeature can make up for the deficiency of the integratedfeature extraction method in detail feature extraction andquantify the characteristic change between each time point.

3.3 Parameter Feature

For the feature data corresponding to k moments, the corre-sponding data of k groups can generally be obtained as:

{(ti, ψ(x(ti))), i = 1, 2, · · · , k} (9)

To describe the variation rule of extracted features withtime at different moments, the common parameter modelin statistics is used, and corresponding parameters can alsobe used as extracted features. For example, when using thefollowing parametric model

ψ(x(ti)) = m(ti, θ) + εi, i = 1, 2, · · · , k (10)

where θ ∈ Rd represents unknown parameters of themodel, m(·) is a given function, εi represents the randomerror, and Eεi = 0, based on the least square estimationmodel, the datasets {(ti, ψ(x(ti))), i = 1, 2, . . . , k} can besolved by following optimization problem:

θ = argminθ∈Θ

k∑i=1

(ψ(x(ti))−m(ti, θ))2 (11)

Here, the dynamic radiomics features are the parameterθ. Below are some examples of models and their correspond-ing dynamic radiomics features:

m(ti, θ) =A·(1−e−α·t)q·e−β·t·(1+e−γ·t)

2 , θ = (A,α, q, β, γ)T

m(ti, θ) =∑7i=1 ai · ti, θ = (a1, a2, · · · , a7)

T

m(ti, θ) =(P2+(P5·t))

(1+e−P4(t−P3))+ P1, θ = (P1, P2, · · · , P5)

T ,

(12)The parameter feature refer to the idea of pharmacoki-

netics, and the characteristic values at each moment arefitted with a specific curve, and the fitted parameters areused as the results. The information content of this methodis between the integrated characteristics and the discretecharacteristics. It can describe both the global change andthe local change, which is suitable for data analysis withmany time points.

4

TABLE 1Equipment Parameters of Different Cohorts. Cohort 1 is the DCE-MRI data of breast cancer patients from the First Affiliated Hospital of China

Medical University. Cohort 2 is the DCE-MRI data of breast cancer patients from Shengjing Hospital of China Medical University. Cohort 3 is theCT data of patients with liver metastases from bowel cancer. Cohort 4 is the DCE-MRI data of patients treated with neoadjuvant chemotherapy

from a public database.

Cohort 1 Cohort 2

Manufacturers: Siemens 3.0 T MRI Manufacturers: Philips 3.0 T MRITR: 4.46 ∼ 7.80ms TR: 4.1msTE: 1.54 ∼ 4.20ms TE: 2.1msNo interval scanning No interval scanningSlice thickness: 2.0mm Slice thickness: 2.0mmStages in the scan: 8 Stages in the scan: 8Interval between stages: 1 min Interval between stages: 1 min

Cohort 3 Cohort 4

Manufacturers: Toshiba, GE, Phillips and Siemens Manufacturers: Siemens 3.0 T MRITube voltage: 120 kVp (range 100− 140kV p) TR: 6.2 msSlice thickness: 2.0 mm TE: 2.9 msMatrix: 512× 512 FOV: 30 ∼ 34cmTube current: 333 mA (range 100–752 mA) Slice thickness: 1.4 mmExposure time: 751 ms (range 500–1782 ms) In-plane matrix size: 320× 320

TE: echo time, TR: pulse repetition time, FOV: field of view

TABLE 2The Parameters of Cohort 1.

Characteristics Metastasis (n = 30) Nonmetastasis (n = 27) P value

Histological grade1 1 02 25 263 4 1

Stage 1.212 ∗ e− 8I+II 7 26III+IV 23 1

Molecular subtypeLuminal A 4(13.33%) 8(29.63%)Luminal B 18(60%) 15(55.56%)HER2-like 1(3.33%) 1(3.7%)Triple-negative 7(23.33%) 3(11.11%)

Age 0.1135≤ median 12(40%) 17(62.96%)> median 18(60%) 10(37.04%)

Tumor size, cm 0.001059≤2 3 14>2 27 13

The exact P values and Kruskal-Wallis test were used to determine whether the clinicopathological variables differed significantly between themetastasis and nonmetastasis sets..

TABLE 3The Parameters of Cohort 2.

Characteristics Metastasis (n = 34) Nonmetastasis (n = 33) P value

Stage 0.0003732I+II 23 33III+IV 11 0

Molecular subtypeA 3 7B 24 19Her2+ 3 2TNBC 4 3unknown 0 2

Age 0.8086≤ median 18(52.94%) 16(48.48%)> median 16(47.06%) 17(51.52%)

Tumor size, cm 0.02803≤2 11 20>2 23 13

The exact P values and Kruskal-Wallis test were used to determine whether the clinicopathological variables differed significantly between themetastasis and nonmetastasis sets.

5

TABLE 4The Parameters of Cohort 4.

Characteristics Mutant type (n = 64) Wild type (n = 43) P value

Microsatellite 0.4205no 39(60.94%) 35(81.4%)yes 25(39.06%) 8(18.6%)

Extrahepatic Meta 0.006no 38(59.38%) 37(86.05%)yes 26(40.62%) 6(13.95%)

Sex 0.202female 30(46.88%) 14(32.56%)male 34(53.12%) 29(67.44%)

Age 1≤60 33(51.56%) 23(53.49%)>60 31(48.44%) 20(46.51%)

Tumor siteleft 19(29.69%) 11(26.19%)rectum 26(40.62%) 19(45.24%)right 14(21.88%) 6(14.29%)unknown 5(7.81%) 6(14.29%)

“Extrahepatic Meta” refers to the presence of metastatic lesions other than the liver and regional lymph nodes. “Microsatellite” refers to a singlelarge lesion surrounded by multiple small lesions. The exact P values and Kruskal-Wallis test were used to determine whether the

clinicopathological variables differed significantly between the metastasis and nonmetastasis sets.

TABLE 5Model built using data from breast cancer patients, with the prediction accuracy based on various dimensions of FNN, SVM and LDA. The RCR,RACA, SD, DC and Ploy’s formulas are in Appendix I, where 2D* represents 2D dynamic features and 3D* represents 3D dynamic features. The

static feature refers to separately using 2D and 3D features for modeling and analysis, where 2D Multi and 3D Multi refer to feature set analysis atmultiple time points.

FNN SVM LDA

Integrated Feature 3D∗ RCR 1.000 0.952 0.8333D∗ RACR 0.952 0.857 0.9292D∗ RCR 0.881 0.857 0.8572D∗ RACR 0.857 0.881 0.976

Discrete Feature 3D∗ SD 0.667 0.592 0.6193D∗ DC 0.714 0.643 0.6192D∗ SD 0.810 0.738 0.7862D∗ DC 0.786 0.738 0.738

Parameter Feature 3D Ploy 0.425 0.450 0.6252D Ploy 0.300 0.275 0.325

Static Feature 3D Multi 0.857 0.595 0.8102D Multi 0.833 0.881 0.8333D 0.600 0.628 0.6842D 0.500 0.356 0.453

4 EXPERIMENT AND ANALYSIS OF RE-SULTS4.1 Data acquisition

This study used three types of data according to the threedifferent clinical problems, as shown in Table 1. For thefirst problem, the data of breast cancer patients came fromtwo hospitals: 57 patients were from the First AffiliatedHospital of China Medical University, and 67 patients werefrom Shengjing Hospital of China Medical University; theresolutions of the images from the two hospitals were384*384 and 560*560, respectively. The data were dividedinto two groups (64 SLN-positive cases and 60 SLN-negativecases). For the second problem, the data of 107 patients(63 men and 44 women, median age of 60 years, ageranging between 35 and 78 years) from the First Hospitalof China Medical University were collected for the CRLMstudy. Based on the status of the RAS and BRAF genes,the patients were classified into two groups: the mutant

group and the wild-type group. Patients with any mutationsin the RAS or BRAF gene were classified into the mutantgroup (N = 64), while others were classified into the wild-type group (N = 43). For the third problem, an open dataset was used, and data was gathered prior to the start oftreatment (V1) and after the first cycle of treatment (V2)from 10 patients: 3 pathological complete responders (PCRs)and 7 non-PCRs (https://wiki.cancerimagingarchive.net/display/Public/QIN+Breast+DCE-MRI). The resolution ofthe images was 320*320. The goal of the challenge was toevaluate variations in the DCE-MRI assessment of breastcancer response to neoadjuvant chemotherapy. The scan-ning parameters are listed in Tables 2-4. A pathological di-agnosis was made and confirmed by authoritative doctors.This study was approved by the Ethics Committee of theFirst Affiliated Hospital of China Medical University andthe Ethics Committee of the Shengjing Hospital of ChinaMedical University.

6

Fig. 1. The ROC curves of FNN, SVM and LDA models for each dimension.

4.2 Work pipeline

The image biomarker standardization initiative (IBSI) [37]was regarded as reference and taken into consideration inour data processing, images feature, and biomarker selec-tion procedure. The experimental processing flow of thisstudy is summarized as follows: (1) acquisition of imagedata; (2) tumor area calibration; (3) tumor area segmenta-tion; (4) multiperiod static feature extraction and quantifi-cation; (5) dynamic feature generation and dimensionalityreduction; and (6) dynamic classification model establish-ment and prediction. The following is a corresponding in-troduction to the process and challenges. First, we obtainedmulticenter experimental data and used the ROI outlinedby the clinician to segment the tumor region. Furthermore,static features such as first-order statistical features, tex-ture features, and shape features were extracted from thesegmented ROI in the image. Then, we used the dynamicalgorithm we designed to transform the static features intodynamic features and used the least absolute shrinkage andselection operator (LASSO) method for feature selection.Finally, the features after dimensionality reduction wereused for modeling, this process we use ’Wu Kong’ platform(https://www.omicsolution.com/wkomics/main/) for rel-ative analysis. And methods such as receiver operatingcharacteristic (ROC) curve were used for model evaluation.Among them, we used the holdout method to divide thedata into a training set and a test set at a ratio of 2:1.

Fig. 2. The accuracy of FNN and LDA models for each dimension. a1is the accuracy of the model using the data from the First Hospitalof Medical University as the training set and the data from ShengjingHospital as the test set. b1 is the accuracy rate of the model establishedby interchanging the training set and the test set.

4.3 Prediction of ALNM in breast cancer and com-parison of the predictive effect on data from differentsources

In this part of the work, we first conducted experimentsusing data from breast cancer patients. The prediction modelwas established by using three classifiers (feedforward neu-ral network (FNN), support vector machine (SVM) and Lin-ear Discriminant Analysis (LDA)) with the best classificationeffect. We compared and analyzed three types of dynamicmodels and one static model in various dimensions. In allmodel building processes, we used two-thirds of the data asthe training set and one-third of the data as the test set. Theresults are shown in Table 5.

It can be seen from the information in Table 5 that com-pared to the static model, the Integrated Feature predictionaccuracy rate in the dynamic model is greatly improved.Among them, for the 3D∗ RCR algorithm, the FNN modeland the SVM model have the best prediction results. For the3D∗ RACR algorithm, the FNN model and the LDA modelhave the best results. In contrast, the 2D∗ RCR algorithmdid not perform very well under the three classifier models.The 2D∗ RACR algorithm performed better under theLDA model. For the static models, the analysis and pre-diction results using single-period 2D and 3D features arerelatively general, and the analysis and prediction resultsusing multiperiod 2D and 3D features are better but stillworse than the prediction results of the Integrated Featurein the dynamic model. However, in the dynamic model,the Discrete Feature and Parameter Feature have generalresults in breast cancer prediction. In general, the modelsestablished by dynamic 2D and 3D radiomics methodsare significantly better than traditional static 2D and 3Dradiomics models. At the same time, dynamic 3D predictionmodels have higher accuracy than dynamic 2D models.This finding suggests that dynamic 3D features based onmedical images are more suitable than dynamic 2D featuresto describe the heterogeneity of the lesion area and can morefully describe the metabolic changes of the lesion. For theintegrated feature in the dynamic model, which had the bestresults, we used the ROC curve to evaluate the performanceof the models built by these two algorithms in differentdimensions. The ROC curve results are shown in Figure 1.

Since the data of breast cancer patients were from amulticenter data set, there were some differences in the

7

data obtained. These differences are mainly reflected inthe difference in the resolution of DCE-MRI, which alsodirectly affects the dynamic analysis of radiomics features.Therefore, in this section, we compare and analyze the dataof the two hospitals to evaluate the generalization of thenew algorithm, which is named the Integrated Feature, ondata from different sources. This part also uses two modelsof FNN and LDA. The data from the two hospitals areseparately used as the training and test sets for experiments.The results are shown in Figure 2.

Fig. 3. FNN(a) is the ROC curve of the FNN model for each dimension,which uses the data from the First Hospital of Medical University as thetraining set and the data from Shengjing Hospital as the test set. FNN(b)is the ROC curve of the FNN model for each dimension, which uses thedata from Shengjing Hospital as the training set and the data from theFirst Hospital of Medical University as the test set.

The analysis and comparative experiments of differentdata sources show that for the FNN model, the modelestablished by dynamic 2D and 3D radiomics algorithmscan eliminate the influence of different data sources to acertain extent. For data from two data sources, regardless ofwhich set of data is used to train the model and which isused to test the model, the prediction results of the modelare almost the same. The algorithm with the best perfor-mance among the four dynamic algorithms is 3D∗ RCR.In contrast, the accuracy and stability of traditional static 2Dand 3D models are not very good. In the LDA model, amongthe four dynamic algorithms, 3D∗ RACR, 2D∗ RCR and2D∗ RACR have better stability. The stability of 2D fea-tures in the static method is better, but the accuracy is stillnot as good as that of the dynamic algorithm. In general,models built using traditional static 2D and 3D features arenot suitable for predicting data from different data sources.In other words, static 2D and 3D radiomics methods aremore suitable for predicting image data with no differencein resolution, while dynamic 2D and 3D radiomics methodsare more suitable for data with differences. For the FNNmodel, which had the best results, we used the ROC curve toevaluate its prediction performance. The ROC curve resultsare shown in Figure 3.

4.4 Prediction of gene mutation status in patients withliver metastases from bowel cancer

In addition, we also used data from patients with livermetastases from bowel cancer to experiment with this dy-namic radiomics method. Radiomic features were extractedfrom the portal phase (PP), arterial phase (AP), portalvenous phase (PVP) and delay phase. The results of theexperiment are shown in Table 6.

In this part of the experimental results, the 3D∗ DCalgorithm in the dynamic method named Discrete Featureperforms better. In contrast, the Integrated Feature, whichperformed best in predicting breast cancer, does not performwell in this problem. This means that for different clinicaldiagnoses, different characteristics will show a higher cor-relation, rather than a certain type of characteristic beingapplicable to all clinical diagnoses. The predictive effect oftraditional static 2D and 3D methods in this part of theclinical data is not very good.

4.5 Joint analysis and algorithm validationIn the previous part of the experiment, we realized that asingle type of algorithm feature may not be suitable forclinical prediction in more fields, so in this part of thework, we conducted a joint modeling analysis on multi-ple algorithm features. We used a total of three data setswith data from breast cancer patients, data from patientswith liver metastases from bowel cancer, and data froma public data set for evaluating the efficacy of neoadju-vant chemotherapy. In addition, in this part of work, wealso introduce a time-dependent RNN model which namedLSTM [38], and conduct a comparative analysis with thedynamic model designed by us. The experimental data ofRNN model are of two types, one is the original image, theother is the extracted image features. The processing methodof the original image is CNN+RNN, and the extracted imagefeatures are directly analyzed by using the RNN model. Theaccuracy results are shown in Figure 4. Each box graphmodel randomly sampled the training set and test set tentimes.

Fig. 4. The accuracy of dynamic model, static model and RNN model.Among them, the dynamic model and the static model respectively useFNN.

From the experimental results of these three kinds ofdata, the performance of the dynamic analysis model is stillbetter than that of the static analysis model. In the predictionof the three types of diseases, the dynamic method hasthe best performance in the prediction of breast cancer andthe performance of the model is relatively stable, followedby the prediction of bowel cancer and the performance ofthe model is relatively good. In the prognostic evaluationof neoadjuvant chemotherapy, the predictive effect of themodel is poor, which may be due to the small amountof data. In addition, it can be seen from the comparative

8

TABLE 6Model built using data from patients with liver metastases from bowel cancer, with the prediction accuracy based on various dimensions of FNN,SVM and LDA. The RCR, RACA, SD, DC and Ploy’s formulas are in Appendix I, where 2D* represents 2D dynamic features and 3D* represents

3D dynamic features. The static feature refers to separately using 2D and 3D features for modeling and analysis, where 2D Multi and 3D Multi referto feature set analysis at multiple time points.

FNN SVM LDA

Integrated Feature 3D∗ RCR 0.556 0.500 0.5563D∗ RACR 0.528 0.583 0.5562D∗ RCR 0.611 0.528 0.7222D∗ RACR 0.694 0.667 0.778

Discrete Feature 3D∗ SD 0.639 0.556 0.6113D∗ DC 0.806 0.806 0.8332D∗ SD 0.500 0.500 0.7002D∗ DC 0.528 0.528 0.806

Parameter Feature 3D Ploy 0.619 0.619 0.6902D Ploy 0.548 0.500 0.476

Static Feature 3D Multi 0.528 0.556 0.6112D Multi 0.639 0.500 0.5283D 0.681 0.639 0.6512D 0.528 0.611 0.527

(a) (b)

Fig. 5. a is the GLRLM mean feature change in eight periods and b isthe Eccentricity feature change in eight periods, where the red curverepresents patients with sentinel lymph node metastasis, and the bluecurve represents patients without sentinel lymph node metastasis.

test of the RNN model that the performance results forbreast cancer and bowel cancer are not very good. Thismay be because the RNN model fails to dig out the timecorrelation between images well. The performance resultsin the prognostic evaluation of neoadjuvant chemotherapyare acceptable and the model is stable. For breast cancerdata, we also used the CNN+RNN model to analyze andpredict the original image with an accuracy rate of 0.819.The experimental results performed well, but there was stilla certain gap compared with the dynamic analysis model.We also found that among these extracted features, tex-ture features have the greatest impact on the experimentalresults. The specific metabolic process of the pathologicaltissue is mainly reflected in the texture change of the image.Moreover, through the screening of dynamic 2D and 3Dfeatures, we found that only the relative change rules ofcertain features at specific time points have an importantimpact on the specific information describing the lesion.Therefore, in the breast cancer analysis with the highest al-gorithm applicability, we extracted the two most predictivefeatures, named GLRLM mean and Eccentricity, which isthe eccentricity of an ellipse with the same second-ordermoment as the mass area. In addition, we use Figure 5 toshow how these two features change over time.

5 DISCUSSION

From the algorithm design of dynamic radiomics, we cansee that the dynamic change information of the image isdescribed in detail. This dynamic analysis includes thedifference between any two time-series images. In otherwords, any small changes in the lesion area will be cap-tured by the algorithm that we designed because the al-gorithm of dynamic radiomics is based on the pixel level.In the field of dynamic analysis, the most popular researchis pharmacokinetics. Compared with dynamic radiomics,pharmacokinetics contains little dynamic information aboutimage changes. Because some of the parameters describedby pharmacokinetics are too simple to change the state ofthe image, a large amount of image dynamic informationhas not been tapped. In addition, in the process of analysisand prediction using medical images, we found that toofew features used in modeling will not obtain good pre-diction results, even if these features are more meaningfulfeatures. The reason why the prediction accuracy of imagemodeling and analysis using pharmacokinetics alone is notparticularly high may be that its number of parameters isnot very large. Thus, the meaningful information in theimages cannot be mined to a great extent, which resultsin the pharmacokinetic parameters having significance inthe clinical analysis and prediction, but the results are notperfect. However, some of the purely mathematical modelsof pharmacokinetics that we have summarized can still beintroduced into dynamic radiomics analysis, although thesemodels currently do not fit our extracted dynamic featureswell, but in the future, there may be more suitable entrypoints.

The dynamic radiomics proposed by this research hasthe advantages of static image analysis and includes thefield of dynamic image analysis, which traditional 2D and3D radiomics cannot accomplish. At the same time, thefield of dynamic analysis makes up for the shortcomingthat pharmacokinetics cannot mine a large amount of imageinformation, and it also greatly improves the accuracy ofclinical analysis and prediction. In addition, to a certain

9

extent, the differences between different data sources areresolved, providing an effective method for clinical analysisand decision-making. Importantly, dynamic radiomics is notonly suitable for breast cancer research but can also be usedfor imaging studies of other organs with dynamic features.

6 CONCLUSION

The dynamic radiomics proposed in this field can assistdoctors in the dynamic analysis of images and providestheoretical support for clinical diagnosis. At the same time,it also provides a new way to obtain information that cannotbe accomplished with static analysis.

From the perspective of clinical application, the time-varying features extracted by dynamic radiomics can in-directly reflect the specific metabolic process of the lesiontissue, thereby establishing a relative relationship betweenmetabolic changes and pathological changes by means ofcomputer-assisted diagnosis. This method not only expandsthe analysis scope of traditional radiomics but also realizesthe transition from static analysis to dynamic analysis of theimage and provides an effective solution for current doctorsto analyze the impact of dynamic images due to subjectivity.

7 ACKNOWLEDGMENTThanks to Professor Liang Dong, Shenzhen Institute ofAdvanced Technology, Chinese Academy of Sciences, forhis comments and suggestions to improve the quality of thispaper.

REFERENCES

[1] H. R. Schelbert, “Nuclear Medicine at a Crossroads,” Journal ofNuclear Medicine, vol. 52, pp. 10S-15S, Dec, 2011.

[2] J. F. Eary, “Nuclear medicine in cancer diagnosis,” Lancet, vol. 354,no. 9181, pp. 853-857, Sep, 1999.

[3] H. Aerts, E. R. Velazquez, R. T. H. Leijenaar, C. Parmar, P. Gross-mann, S. Cavalho, J. Bussink, R. Monshouwer, B. Haibe-Kains, D.Rietveld, F. Hoebers, M. M. Rietbergen, C. R. Leemans, A. Dekker,J. Quackenbush, R. J. Gillies, and P. Lambin, “Decoding tumourphenotype by noninvasive imaging using a quantitative radiomicsapproach,” Nature Communications, vol. 5, Jun, 2014.

[4] V. Kumar, Y. H. Gu, S. Basu, A. Berglund, S. A. Eschrich, M. B.Schabath, K. Forster, H. Aerts, A. Dekker, D. Fenstermacher, D. B.Goldgof, L. O. Hall, P. Lambin, Y. Balagurunathan, R. A. Gatenby,and R. J. Gillies, “Radiomics: the process and the challenges,”Magnetic Resonance Imaging, vol. 30, no. 9, pp. 1234-1248, Nov, 2012.

[5] R. J. Gillies, P. E. Kinahan, and H. Hricak, “Radiomics: Images AreMore than Pictures, They Are Data,” Radiology, vol. 278, no. 2, pp.563-577, Feb, 2016.

[6] P. Lambin, E. Rios-Velazquez, R. Leijenaar, S. Carvalho, R. vanStiphout, P. Granton, C. M. L. Zegers, R. Gillies, R. Boellard, A.Dekker, and H. Aerts, “Radiomics: Extracting more informationfrom medical images using advanced feature analysis,” EuropeanJournal of Cancer, vol. 48, no. 4, pp. 441-446, Mar, 2012.

[7] R. Larue, G. Defraene, D. De Ruysscher, P. Lambin, and W. VanElmpt, “Quantitative radiomics studies for tissue characterization:a review of technology and methodological procedures,” BritishJournal of Radiology, vol. 90, no. 1070, 2017.

[8] I. El Naqa, P. W. Grigsby, A. Apte, E. Kidd, E. Donnelly, D. Khullar,S. Chaudhari, D. Yang, M. Schmitt, R. Laforest, W. L. Thorstad, andJ. O. Deasy, “Exploring feature-based approaches in PET images forpredicting cancer treatment outcomes,” Pattern Recognition, vol. 42,no. 6, pp. 1162-1171, Jun, 2009.

[9] G. Thibault, J. Angulo, and F. Meyer, “Advanced Statistical Matricesfor Texture Characterization: Application to Cell Classification,”IEEE Transactions on Biomedical Engineering, vol. 61, no. 3, pp. 630-637, Mar, 2014.

[10] R. M. Haralick, K. Shanmugan, I. Dinstein, ”Textural features forimage classification”, IEEE Transactions on System Man Cybern., vol.3, no. 6, pp. 610-621, Nov. 1973.

[11] M. M. Galloway, ”Texture classification using gray-level runlengths”, Computer Graphics and Image Processing, vol. 4, pp. 172-179, June 1975.

[12] T. P. Coroller, P. Grossmann, Y. Hou, E. R. Velazquez, R. T. H.Leijenaar, G. Hermann, P. Lambin, B. Haibe-Kains, R. H. Mak, andH. Aerts, “CT-based radiomic signature predicts distant metastasisin lung adenocarcinoma,” Radiotherapy and Oncology, vol. 114, no. 3,pp. 345-350, Mar, 2015.

[13] Wu G , Chen Y , Wang Y , et al. ”Sparse Representation-BasedRadiomics for the Diagnosis of Brain Tumors”. IEEE Transactions onMedical Imaging, 2018, 37(99):893.

[14] Arnaud A , Forbes F , Coquery N , et al. ”Fully Automatic LesionLocalization and Characterization: Application to Brain TumorsUsing Multiparametric MRI Data”. IEEE Transactions on MedicalImaging, 2017:1-1.

[15] Lifeng, Yang, Jingbo, et al. ”Development of a radiomics nomo-gram based on the 2D and 3D CT features to predict the survival ofnon-small cell lung cancer patients”. European Radiology, 2019.

[16] Cirujeda P , Cid Y D , Henning Muller, et al. ”A 3-D Riesz-Covariance Texture Model for Prediction of Nodule Recurrence inLung CT”. IEEE Trans Med Imaging, 2016, 35(12):2620-2630.

[17] R. Ortiz-Ramon, A. Larroza, S. Ruiz-Espana, E. Arana, and D.Moratal, “Classifying brain metastases by their primary site oforigin using a radiomics approach based on texture analysis: afeasibility study,” European Radiology, vol. 28, no. 11, pp. 4514-4523,Nov, 2018.

[18] Lifeng Yang, Jingbo Yang, Xiaobo Zhou, ”Development of a ra-diomics nomogram based on the 2D and 3D CT features to predictthe survival of non-small cell lung cancer patients”, European Radi-ology, vol. 29, pp. 2196–2206, 2019.

[19] Gillies R J , Kinahan P E , Hricak H . ”Radiomics: Images AreMore than Pictures, They Are Data”. Radiology, vol. 278, no. 2, pp.563-577, 2016.

[20] R. Fusco, M. Sansone, S. Maffei, N. Raiano, A. Petrillo, “Dynamiccontrast-enhanced mri in breast cancer: A comparison betweendistributed and compartmental tracer kinetic models,” J. BiomedicalGraphics and Computing, vol. 2, no. 2, pp. 23-36, 2012.

[21] J. Kallehauge, K. Tanderup, C. Duan, et. al., “Tracer kineticmodel se-lection for dynamic contrast-enhanced magnetic reso-nance imaging of locally advanced cervical cancer,” Acta Oncologica,vol. 53, pp. 1064-1072, 2014.

[22] N. Mayr, Z. Huang, Z. Jian, et al., “Characterizing tumor hetero-geneity with functional imaging and quantifying high-risk tumorvolume for early prediction of treatment outcome: Cervical canceras a model,” Int J Radiat OncolBiol Phys, vol. 83, no. 3, pp. 972-979,2011.

[23] N. M. Braman, M. Etesami, P. Prasanna, C. Dubchuk, H. Gilmore,P. Tiwari, D. Pletcha, and A. Madabhushi, “Intratumoral and per-itumoral radiomics for the pretreatment prediction of pathologicalcomplete response to neoadjuvant chemotherapy based on breastDCE-MRI,” Breast Cancer Research, vol. 19, May, 2017.

[24] H. Li, Y. T. Zhu, E. S. Burnside, K. Drukker, K. A. Hoadley, C. Fan,S. D. Conzen, G. J. Whitman, E. J. Sutton, J. M. Net, M. Ganott, E.Huang, E. A. Morris, C. M. Perou, Y. Ji, and M. L. Giger, “MRImaging Radiomics Signatures for Predicting the Risk of BreastCancer Recurrence as Given by Research Versions of MammaPrint,Oncotype DX, and PAM50 Gene Assays,” Radiology, vol. 281, no. 2,pp. 382-391, Nov, 2016.

[25] Boldrini L , Cusumano D , Chiloiro G , et al. ”Delta radiomicsfor rectal cancer response prediction with hybrid 0.35T magneticresonance-guided radiotherapy (MRgRT): a hypothesis-generatingstudy for an innovative personalized medicine approach”. SpringerOpen Choice, vol. 124, no. 2, 2019.

[26] Carvalho, S. et al. ”Early variation of FDG-PET radiomics featuresin NSCLC is related to overall survival - the ‘delta radiomics’concept”. Radiotherapy and Oncology 118, S20–S21 (2016).

[27] Rao S X , Lambregts D M , Schnerr R S , et al. ”CT textureanalysis in colorectal liver metastases: A better way than sizeand volume measurements to assess response to chemotherapy?”.United European Gastroenterology Journal, 2015, 4(2).

[28] Nasief H , Zheng C , Schott D , et al. ”A machine learning baseddelta-radiomics process for early prediction of treatment responseof pancreatic cancer”. npj Precision Oncology.

10

[29] Cunliffe, Alexandra, Armato, Samuel G, Castillo, Richard, et al.”Lung Texture in Serial Thoracic Computed Tomography Scans:Correlation of Radiomics-based Features With Radiation TherapyDose and Radiation Pneumonitis Development”. International Jour-nal of Radiation Oncology, Biology, Physics, 2015, 91(5):1048-1056.

[30] Weigel M T , Dowsett M . Weigel MT, Dowsett M, ”Current andemerging biomarkers in breast cancer: prognosis and prediction.Endocr Relat Cancer 17: R245-R262”. Endocrine Related Cancer, 2010,17(4):R245-62.

[31] Cui X , Wang N , Zhao Y , et al. ”Preoperative Prediction ofAxillary Lymph Node Metastasis in Breast Cancer using RadiomicsFeatures of DCE-MRI”. Scientific Reports, 2019, 9(1).

[32] Han L , Zhu Y , Liu Z , et al. ”Radiomic nomogram for predic-tion of axillary lymph node metastasis in breast cancer”. EuropeanRadiology, 2019.

[33] Arena S , Bellosillo B , Siravegna G , et al. ”Emergence of Multi-ple EGFR Extracellular Mutations during Cetuximab Treatment inColorectal Cancer”. Clinical Cancer Research, 2015, 21(9):2157-2166.

[34] Ahsee M L , Makris A , Taylor N J , et al. ”Early changes in func-tional dynamic magnetic resonance imaging predict for pathologicresponse to neoadjuvant chemotherapy in primary breast cancer”.Clinical Cancer Research An Official Journal of the American Associationfor Cancer Research, 2008, 14(20):6580.

[35] Braman N M , Etesami M , Prasanna P , et al. ”Intratumoral andperitumoral radiomics for the pretreatment prediction of patho-logical complete response to neoadjuvant chemotherapy based onbreast DCE-MRI”. Breast Cancer Research, 2017, 19(1):57.

[36] Huang W , Chen Y , Fedorov A , et al. ”The Impact of Arte-rial Input Function Determination Variations on Prostate DynamicContrast-Enhanced Magnetic Resonance Imaging PharmacokineticModeling: A Multicenter Data Analysis Challenge”. Tomography AJournal for Imaging Research, 2016, 2(1):56.

[37] Zwanenburg A , Martin Vallieres, Abdalah M A , et al. ”TheImage Biomarker Standardization Initiative: Standardized Quanti-tative Radiomics for High-Throughput Image-based Phenotyping”.Radiology, 2020, 295(2):191145.

[38] S. Hochreiter, and J. Schmidhuber. ”Long short-term memory”.Neural Computation, 1997, 9(8):1735.