Machine Learning for Computational Modeling of TongueMechanics

Aniket A. Tolpadi1, Arnold D. Gomez2, Jerry L. Prince2

1 Department of Bioengineering, Rice University, Houston, TX, US 770052 Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore,

MD, US 20218

ABSTRACT

Muscular contraction is fundamental for transport processes associated with tissue motion. However, direct non-invasive measurement of contraction patterns is currently impossible, and indirect estimation using optimization-based inverse biomechanical models produces results heavily influenced by necessary regularization approaches,limiting their application. This research frames the inversion problem as an extension of machine learning, as hasbeen demonstrated in medical imaging applications. As such, we constructed a random-forest regression schemewhere model information is transferred from the finite-element domain to a volumetric imaging lattice. Once inthe lattice, contraction patterns are decomposed into wavelet coefficients, which are manipulated and then re-constructed to yield new contraction patterns that are leveraged as training data. Results show the image-basedapproach to generating training data is flexible and produces datasets that reduce contraction estimation errorin test cases where past approaches have failed.

1. INTRODUCTION

Tissue motion is crucial to many fundamental life processes, including peristalsis in the digestive system, loco-motion with the limbs, and speech production and mastication in the tongue. Such motion can be initiated bothactively or passively, and muscular activation is a key means of eliciting movement. It follows that muscularactivationparticularly as it relates to generating forces and inducing tissue motionis critical to survival. Sincethe tongue is one of few body parts in which muscular activation and passive forces act to induce motion withlittle assistance from bone movement,1 it is one of the most intriguing regions in which to study tissue motion.Clinically, contractile dysfunction is observed in diseases such as orofacial myofunctional disorders and apraxiaof speech.2 In other common diseases, treatments include excising diseased or obstructing portions of the tongue(glossectomy). Sleep apnea, for example, sees the constriction of airways, and is treated by the excision of excesstissue. In head, neck, and oropharyngeal cancers, tumors are found directly on or near muscles of the tongue,and can be deadly or result in severe disabilities. These are also treated by total or partial glossectomy.3,4 Theincidence rate of head and neck cancers continues to increase, and the incidence rate of oropharyngeal cancerremains steady despite a dramatic decline in the number of smokers.5,6 In these cases, an effective activation-based model of tongue motion can serve to simulate prospective excisions and ultimately select a treatment thatminimizes the functional impact of tissue loss (i.e. a treatment that does not fundamentally impede the motionnecessary for common speech sound would be preferable over one that does). While current models of tonguemotion have been effective in modeling the healthy tongue, the lack of sufficient tissue motion data from impairedand diseased tongues has prevented accurate modeling of unhealthy tongues. As such, methods to study andmodel the abnormal tongue not only would have great clinical utility, but also help further the study of oral andneck cancers, sleep apnea, and speech disorders in which normal tongue movements are not always observed.

To leverage mechanical models of the tongue as guides for speech therapy, surgery, and other means to improvelives, simulations must accurately model motion in the abnormal anatomy and be patient-specific. At a funda-mental level, the model establishes links between muscle activations and tongue tissue motion, and its accuracyis determined by agreement of simulated outputs against experimental data. Means of tracking tongue motionare fairly well-defined: researchers are capable of measuring tongue displacements through the use of medicalimaging techniques, such as tagged-MRI and similar analyses.7 Unfortunately, the means of tracking muscleactivations in the tongue are fundamentally limited. For instance, electromyography has been demonstrated toestimate muscular activity in the tongue and other organs, but this approach suffers from lack of reliability.8 An

alternative means of estimating muscular activity patterns within the tongue is through inverse finite-element(FE) models, in which an activation pattern is estimated that yields a simulated deformation that most closelymatches that of the measured deformation. The inverse problem has been solved in the field by minimizinga similarity metric between experimental and simulated results9 or as a machine-learning problem where themodel provides training data.10 To reduce the computational intensity of either approach, however, past solu-tions to this inverse problem have implemented physiological constraints on how muscles must interact: that is,the tongue has been regularized into functional components that are assumed to synchronously and uniformlycontract in approximately the same directions.8,11 Due to this regularization, however, these approaches failin cases of disease when the assumed anatomical distribution (a healthy tongue) is not tractable, such as whenregions of the tongue are excised and replaced with tissue grafts.12 This research involves the extension of amachine-learning-based approach that will give its users the capability to estimate contraction partners withoutexplicit assumptions about the anatomical placement of tongue tissues. Such an approach will facilitate the studyof contraction based on experimental observations of motion in both healthy individuals and patients before andafter glossectomy.

1.1 Tongue Anatomy

The tongue comprises eight muscles, each of which are classified as intrinsic muscles and extrinsic muscles.Intrinsic muscles anchor themselves entirely within the tongue, while extrinsic muscles anchor themselves outsidethe body of the tongue. The four intrinsic muscles are as follows: the superior longitudinal (SL), inferiorlongitudinal (IL), and transverse and vertical (TV). Similarly, the four extrinsic muscles are as follows: thegenioglossus (GG), hyoglossus (HG), styloglossus (SG), and palatoglossus. In some literature, the palatoglossusmay not be considered a tongue muscle, and instead may be considered a palatal muscle. The spatial locationsof these muscles in a right side view of the tongue can be seen in Fig. ??.

Figure 1. Schematic of the following key tongue muscles: styloglossus (SG), hyoglossus (HG), superior longitudinal(SL),transverse and vertical (TV), inferior longitudinal (IL), and genioglossus (GG).14

Each of these muscles have unique muscle fiber orientations that allow them to execute various tongue movements.These orientations can be used to intuitively determine the contraction patterns responsible for bringing aboutdifferent types of tongue motion. A brief summary of some of the permissible movements that result fromindividual muscle contractions can be seen in Fig. ??.

Figure 2. Muscle fiber orientations and associated deformations that can be induced when the muscles contract individually.

1.2 Description of Current Machine-Learning-Based Inversion Pipeline

Earlier work has applied machine learning to solve the inversion problem (estimating activations from deforma-tions) and has centered around regularizing the tongue in accordance with the muscle locations and orientations.We seek to develop a model of motion and contraction that advances this work. In particular, this projectinvolves advancing a previous mechanics-based approach10 where a FE model consisting of 6 contractile muscles

was used to demonstrate strain-to-activation regression with 10.1% error. However, due to the lack of trainingdata, this method failed when the strain input included a small non-contractile flap.

Our lab has previously developed a FE model of the human tongue composed of N=262 contractile elements,and assigned to M=9 contractile groups (individual muscles and intersections of muscles) associated with thegenioglossus, superior longitudinal, inferior longitudinal, transverse, vertical, and the styloglossus.

Figure 3. Original forward-simulating FE model which was based off of 6 muscle groups: SL, IL, T, V, GG, and SG.

The FE simulations are carried out using Finite Elements for Biomechanics (FEBio) software.13 In the originalmodel, elements in the same muscle group contract uniformly and synchronously. The FEBio software runssimulations based on a configuration file in extensible markup language (xml) format, which specifies informationabout materials, elements, and nodes in the model, among others. To generate training data in the previouswork, simulations were produced by assigning different levels of contractions to the muscle groups (or groupsof elements). The assignment processes was equivalent to sampling from a M-dimensional space, which wasquantized from 0 to 50% contraction in steps of 10%. The samples were acquired at random yielding a total of1974 simulations, which made up the training data.

The success of this machine-learning approach in past work likely occurred because the strain vectors thatresulted from simulating various activations would form a relatively smooth (M-dimensional) hyperplane withinthe set of all possible contractions (an N-dimensional space). This smoothness is a crucial concept; thus, futureregression and machine learning approaches should follow be a defined relationship between element activationsand local strains that resembles the distributions in the original set of training data. We will advance past workby deregularizing our labs existing model, but in light of this necessary smoothness, deregularization necessitatesa means of generating data distributed in a manner that is similar to the original set.

2. METHODS

The goal of this research is to make key technical improvements to the existent pipeline, so that a more compre-hensive training data set can be obtained. The improvements will include developing the ability to:

1. Contract elements independently of one anothers in their muscle group;

2. Generate element activations that take advantage of independent contractile capability;

3. Use of machine learning to estimate activations from tongue deformation measurements.

2.1 Finite-Element Simulations

For the purposes of this study, this model was adapted to produce a new FE model in which each element canact independently of others. In practice, this means that material properties and the degree of contraction ofeach element can be adjusted separately. Modifying the previous model into one usable for this project thereforerequired the automatic generation of multiple sections of the setup xml file, and a new means of inputting desiredelement contraction data into the model. Briefly, the Materials Section of the xml file had to be expanded suchthat one material existed per contractile elements, the Elements Section had to be reconstructed to assign eachelement to a different material. Finally, the Load Curves section specifies a balance between internal and externalforces exerted in the model, and thus, manipulates the degree to which elements contract. This section had tobe reworked to allow the user to specify the maximum loads applied (i.e. contraction) for each element; afterreconfiguration, this section is now rewritten by a MATLAB script every time a simulation is run. Altogether,these changes yielded a model in which individual elements and their properties could easily be manipulated.

Figure 4. Original forward-simulating FE model (6 independently-contracting muscles) and the new forward-simulatingmodel (262 independently-contracting elements).

2.2 Data Generation and Sampling Experiments

After establishing a forward-simulating FE model of the tongue capable of simulating tongue motion and defor-mations in response to individual element activation, the next step was to generate data that could be used todevelop a machine-learning model of the inverse problem.

The necessity of developing a systematic means of generating smooth data is a direct result of the transition froma muscle-based to element-based model. On the order of 262 elements, this relationship is not guaranteed to lieon a hyperplane, because the assumption of uniform and synchronous contraction of elements in given musclegroups has been relaxed. Hence, an alternative means of preserving this smoothness must be developed to ensurelater regression will produce results on which regression can be employed effectively. At the same time, havingindependent element contractions results in a large number of element activation combinations. However, mostof these are effectively noise, and new training data must be related to the existent muscular distributions.

The solution that we pursued to simultaneously relax the“uniform and synchronous assumption while preservingsmoothness utilizes the following logic: we can take element activations that were used in generating trainingdata in the previous model (without the assumption being relaxed) and perturb them slightly, such that the

overall vector of element activations is similar to the original by some similarity metric, but still differ in somefundamental way. These slight differences should allow the elements to act independently, but, in theory, willpreserve the smoothness necessary in the machine learning phase.

Thus section, is broken down into two subsections: developing the algorithms to generate data, and samplinggenerated data to produce training data for machine learning.

2.2.1 Data Generation Mechanism

1. The geometry of the forward-simulating model is projected onto a lattice, giving each element in the modelsome corresponding lattice points. One array, Loc, is consequently generated to hold information regardingthe spatial locations of each element in the lattice. Another array, Act, is then constructed in which latticepoints that correspond to an element are assigned that element’s activation. Act is manipulated as follows:

• A 3rd order wavelet decomposition of Act is performed.

• The mean of the absolute value of all wavelet coefficients is defined as the critical coefficient value.All coefficients whose magnitude is less than or equal to the critical coefficient value are reset to zero.

• All nonzero low-pass approximation coefficients G are manipulated as follows, where β is a tunablecoefficient and R is a random variable uniformly distributed in [0,1]: G = G+ βGR

• All nonzero third-order coefficients H are manipulated as follows, where β is a tunable coefficient andR is a random variable uniformly distributed in [0,1]: H = H + βHR

• Act is reconstructed using a 3rd order wavelet reconstruction on the set of wavelet coefficients.

2. New element activations are calculated after Act is manipulated. They are calculated in accordance withthe following steps for each element, where Xn

1 , Xn2 , X

n3 are the lattice centroid coordinates of element n.

• Using Loc, all lattice points in Act that correspond to element n are found.

• The distance each lattice point lies from Xn1 , X

n2 , X

n3 is calculated and stored in an array D.

• Activations are assigned to element n as follows, where Actp is the vector of perturbed elementactivations:

Actpi =

∑i=1

Acti(2 − Di

max(D) )∑i=1

2 − Di

max(D)

(1)

Values of β were tuned for different applications of the data perturbation system. These values will be mentionedas these different experiments are described.

2.2.2 Sampling

Having developed a means of generating element activation profiles, the next step is to select and generatetraining data. The concepts of distance and the median critical distance of the existing training data set carryover from the previous sections. In this context, the critical distance acts as a threshold distance. Existing datafrom the previous model is entirely transferred into this models training data set, and the new training data setis grown in accordance with the procedure outlined below:

1. For a given vector of element activations from the previous training data set, use the Data GenerationMechanism described earlier (Section 1) to obtain a vector of perturbed element activations

2. Run the FE simulation using the element activation vector to obtain data on local strains

3. Calculate the minimum distance between these element activations and the current training data

4. If the distance is above the threshold distance along both orientations, add this simulation to the trainingdata; if not, add it to a set of verification data (verification data will be used later on to test the model)

Figure 5. Schematic of data generation mechanism (perturbation system).

5. When 5 total simulations result in data being added to the verification data and not the training data,move to the next entry in the original training data set and repeat

6. Repeat 1-5 for all entries in the original training data set.

It is believed that the original training data set will grow in a manner that preserves smoothness but introducesnew information onto the learning basis.

2.3 Regression

After finalizing a set of training data, the final step is to pass the training database onto the random forestregression framework and verify the results.

3. EXPERIMENTS & RESULTS

3.1 Perturbation System Demonstration

The perturbation system perturbs original element activations such that the perturbed activations are spatialsimilar with the original, but still exhibit slight differences (Figure 6). The degree of spatial similarity is controlledby β: as the magnitude of β increases, this degree of spatial similarity decreases. As such, the mechanism iscapable of synthetically generating element activations that are rooted in known patterns, but can stray fromthem in a tunable manner. This could yield data more suited to model diseases and patient-to-patient variability.

3.2 Random Forest Prediction of Unhealthy Mechanics

The objective of this experiment was to design and test a training data generation scheme for predicting contrac-tion patterns with areas exhibiting contractile deficiency, which may result after tumor removal.3 The schemewas designed to alter contraction patterns in the “original training data” (described in the previous section),creating perturbations designed to recreate areas of low contraction. To do so, the forward model was modifiedto support heterogenous activation using 262 independent elements (Fig. ??c), and a “seed” contraction patternwas selected. The model’s geometry and the contraction information were projected onto an imaging latticeforming in image of the seed contraction pattern. Then, the image underwent a third-order wavelet decompo-sition to isolate geometrical features of different length scales. A subset of wavelet coefficients associated with

Figure 6. Application of perturbation mechanism to a contraction pattern with different values of β. As β increases, thedegree of spatial similarity perturbed contraction patterns have with the original decreases.

the size of target contractile deficiencies was manually identified and modified using a random distribution. Thenew image was projected back to the model for generation of new contraction-deformation training data for RFregression.

As a proof of concept, 3 RF regression schemes were used to approximate contractions of 5 manually created‘test flaps’. Training data consisted of the same subset of the original training data (50 identical contraction-deformation pairs), which was augmented by: (1) perturbing one of the test flaps or perturbed flaps, (2) perturbinga single, randomly selected entry from the original data or perturbed original, and (3) adding unperturbed entriesof the original data at random dissimilar to flap. The deformation information (strain along the local fiberdirections) was used to obtain contraction patterns using each of the 3 RF regression schemes, and comparedto the contraction test data. Fig. 7a shows the evolution of error as a function of the relative size of thetraining database. As expected, adding random data dissimilar to the flap (green) results in no improvementin regression accuracy. However, augmenting the training data with perturbed flaps (red) has a negative trendin approximation error (50% reduction), suggesting that perturbation of one of the test flaps can increaseregression accuracy in other flap cases. Generating perturbations of the original data (blue) has no effect on theerror trends. These results suggest that perturbation-based training can expand the overall regression capabilityprovided adequate seed patterns are chosen to synthesize new data.

Figure 7. Regression accuracy under different training regimes. Augmenting the training data with perturbed flapsimproves regression accuracy (a). This trend does not occur when adding perturbations of the original data, or randomcontraction-deformation pairs. The improvement is present in all 5 test cases, resulting in reduced error near the flap (b).

4. CONCLUSION

This work has established a means of synthetically generating training data that, on a small scale, can be usedto significantly improve the contraction predictions in noncontractile tissue mechanics, as one would see whena tumor is excised and replaced with a “flap” of tissue. This method is based off of parameters that allow it

to be tuned, thus allowing for the synthetic generation of contraction patterns that balance plausibility andindependence of elements in a forward-simulating model. Given that tumor locations can vary from patient-to-patient and given that feasible contractions will vary between patients even in the absence of disease, this datageneration mechanism is a necessary step in making the leap between existing data and data more representativeof all prospective patients. After some additional refinement in the months to come, this mechanism will be usedto generate full sets of training data on the large scale, thus yielding a tongue model more capable of predictingelement activations in both healthy and unhealthy cases.

REFERENCES

[1] Takemoto H. Morphological Analyses of the Human Tongue Musculature for Three-Dimensional Modeling.(2001) J Speech Lang Hear Res;44(1):95-107.

[2] Schroter-Morasch H, Ziegler W. Rehabilitation of impaired speech function (dysarthria, dysglossia). (2005)GMS Curr Top Otorhinolaryngol Head Neck Surg;4.

[3] Edge SB, Compton CC. The American Joint Committee on Cancer: the 7th edition of the AJCC cancerstaging manual and the future of TNM. (2010) Ann Surg Oncol. Jun;17(6):1471-1474.

[4] Patel SG, Shah JP. TNM staging of cancers of the head and neck: striving for uniformity among diversity.(2005) CA Cancer J Clin. Jul-Aug;55(4):242-58.

[5] Sturgis EM, Cinciripini PM. Trends in head and neck cancer incidence in relation to smoking prevalence:an emerging epidemic of human papillomavirus-associated cancers? (2007) Cancer Oct;110(7):1429-35.

[6] Taturu D, Mak V, Simo R, Davies EA, Gallagher JE. Trends in the epidemiology of head and neck cancerin London. (2017) Clin Otolaryngol Feb;42(1):104-14.

[7] Parthasarathy V, Prince JL, Stone ML, Murano EZ, Nessaiver M. Measuring tongue motion from taggedcine-MRI using harmonic phase (HARP) processing. (2007) J Acoust Soc Am. Jan;121(1):491-504.

[8] Stavness I, Lloyd JE, Fels S. Automatic prediction of tongue muscle activations using a finite element model.(2012) J Biomech 45:2841-2848.

[9] Harandi NM, Woo J, Stone ML, Abugharbieh R, Fels S. Variability in muscle activation of simple speechmotions: A biomechanical modeling approach. (2017) J Acoust Soc Am. Apr; 141:2579-2590.

[10] Gomez AD, Jog A, Stone ML, Prince JL. Machine Learning for Estimation of Activation Patterns in Compu-tational Models of the Tongue. (2017) paper presented to SBC2017: Summer Biomechanics, Bioengineeringand Biotransport Conference, Tucson, 21-24 June.

[11] Kajee Y, Pelteret JP, Reddy BD. The biomechanics of the human tongue. (2013) Int J Numer MethodBiomed Eng. 29(4):492-514.

[12] Al-Ghamdi KB, Bakhsh ZA. Partial glossectomy and floor of mouth (FOM) defect repair with biologicaldural graft: A case report. (2005) Int J Surg Case Rep. 11;78-82.

[13] Maas SA, Ellis BJ, Ateshian GA, Weiss JA. FEBio: Finite elements for biomechanics. (2011) J BiomechEng. 2012 Jan;134(1):011005.

[14] Sanders I, Mu L. A three-dimensional atlas of human tongue muscles. (2013) Anat Rec (Hoboken)Jul;296(7);1102-14.

5. APPENDIX

5.1 Research Ethics

The idea of ethics was never explicitly addressed in our lab, but it was certainly understood: whenever wecame across key results, we would always make sure to repeat the experiments multiple times, and if possible,have multiple people check over results to ensure reasonability. As much as possible, we tried to ensure thatour conclusions were not being made based off of fradulent results and data, and using the many checks weimplemented, we did succeed in doing so.

5.2 Value of Program

I found this program to be a fantastic experience for me on the whole: for one, I was exposed to biomechanics,machine learning, and modeling in an engaging and stimulating manner. These are all fields that had intriguedme from afar, but I was not quite sure if they were things that I would be comfortable pursuing in the future:after this summer, however, I am quite positive that these would all be viable and interesting avenues of futurework for me as I pursue a Ph.D. My fundamental goal from this summer was to gain some clarity on preciselywhich field I would like to research in the future: I have had some work in bioinformatics in the past, andregenerative medicine has always intrigued me, but so too has biomechanics. After this experience, I am nowmore of the mindset that biomechanical modeling would be among my one or two most preferred fields as I lookto the future. Aside from all of this, I have never worked in a lab that was so structured and have never workedaround so many intelligent people at once; I was pushed intellectually in a way that I simply have not been inthe past. This will likely have the most lasting impact of all aspects of this program.

5.3 Overview of Program

I certainly would recommend this program to my friends, as the research opportunities it provides—and theextent to which this program will push you intellectually—are second to no other experience I have had. I dofeel, however, that within the research itself, however, that the students can be “let into the deep end” earlierthan they actually are: in my experience, it was not until approximately week 5 or 6 that my mentor begantrusting me to take on more significant challenges, which was a bit frustrating at first. Aside from that, I wouldrecommend that there are fewer and more carefully-selected trips: while some of them certainly added to theexperience, I did think that there were so many that they ultimately interfered with our ability to make progressin our work. Still though, the research as a whole and the visit to the medical center were the highlights of thisexperience and made it more than worthwhile for me.