Avai lable onl ine at www.sc iencedirect .com

journal homepage: www.elsevier .com/ locate / jva l

ISPOR TASK FORCE REPORTS

Selecting a Dynamic Simulation Modeling Method for HealthCare Delivery Research—Part 2: Report of the ISPOR DynamicSimulation Modeling Emerging Good Practices Task ForceDeborah A. Marshall, PhD1,*, Lina Burgos-Liz, MSc, MPH, BSc Ind Eng2, Maarten J. IJzerman, PhD3,William Crown, PhD4, William V. Padula, PhD, MS5, Peter K. Wong, PhD, MS, MBA, RPh6,Kalyan S. Pasupathy, PhD7, Mitchell K. Higashi, PhD8, Nathaniel D. Osgood, BS, MS, PhD9,10, the ISPOREmerging Good Practices Task Force1Health Services & Systems Research, Department of Community Health Sciences, Cumming School of Medicine, University ofCalgary, Calgary, AB, Canada; 2Cumming School of Medicine, University of Calgary, Calgary, AB, Canada; 3Department of Health Technology & ServicesResearch, University of Twente, Enschede, The Netherlands; 4Optum Labs, Boston, MA, USA; 5Section of Hospital Medicine,University of Chicago, Chicago, IL, USA; 6HSHS Illinois Divisions and Medical Group, Hospital Sisters Health System, Belleville, IL,USA; 7Health Care Systems Engineering Program, Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health CareDelivery, Rochester, MN, USA; 8GE Healthcare, Barrington, IL, USA; 9Department of Computer Science, University of Saskatchewan,Saskatoon, Saskatchewan, Canada; 10Department of Community Health & Epidemiology and Bioengineering Division, Saskatoon,Saskatchewan, Canada

A B S T R A C T

In a previous report, the ISPOR Task Force on Dynamic SimulationModeling Applications in Health Care Delivery Research EmergingGood Practices introduced the fundamentals of dynamic simulationmodeling and identified the types of health care delivery problems forwhich dynamic simulation modeling can be used more effectivelythan other modeling methods. The hierarchical relationship betweenthe health care delivery system, providers, patients, and other stake-holders exhibits a level of complexity that ought to be captured usingdynamic simulation modeling methods. As a tool to help researchersdecide whether dynamic simulation modeling is an appropriatemethod for modeling the effects of an intervention on a health caresystem, we presented the System, Interactions, Multilevel, Under-standing, Loops, Agents, Time, Emergence (SIMULATE) checklist con-sisting of eight elements. This report builds on the previous work,systematically comparing each of the three most commonly useddynamic simulation modeling methods—system dynamics, discrete-event simulation, and agent-based modeling. We review criteria forselecting the most suitable method depending on 1) the purpose—type of problem and research questions being investigated, 2) theobject—scope of the model, and 3) the method to model the object toachieve the purpose. Finally, we provide guidance for emerging goodpractices for dynamic simulation modeling in the health sector,covering all aspects, from the engagement of decision makers in themodel design through model maintenance and upkeep. We concludeby providing some recommendations about the application of these

methods to add value to informed decision making, with an emphasison stakeholder engagement, starting with the problem defini-tion. Finally, we identify areas in which further methodologicaldevelopment will likely occur given the growing “volume, velocityand variety” and availability of “big data” to provide empiricalevidence and techniques such as machine learning for parameterestimation in dynamic simulation models. Upon reviewing this reportin addition to using the SIMULATE checklist, the readers should beable to identify whether dynamic simulation modeling methods areappropriate to address the problem at hand and to recognize thedifferences of these methods from those of other, more traditionalmodeling approaches such as Markov models and decision trees. Thisreport provides an overview of these modeling methods and examplesof health care system problems in which such methods have beenuseful. The primary aim of the report was to aid decisions as towhether these simulation methods are appropriate to address specifichealth systems problems. The report directs readers to other resour-ces for further education on these individual modeling methods forsystem interventions in the emerging field of health care deliveryscience and implementation.Keywords: decision making, dynamic simulation modeling, health caredelivery, methods.

& 2015 Published by Elsevier Inc. on behalf of International Society forPharmacoeconomics and Outcomes Research (ISPOR).

1098-3015$36.00 – see front matter & 2015 Published by Elsevier Inc. on behalf of International Society for Pharmacoeconomics and

Outcomes Research (ISPOR).

http://dx.doi.org/10.1016/j.jval.2015.01.006

E-mail: damarsha@ucalgary.ca.

* Address correspondence to: Deborah A. Marshall, Department of Community Health Sciences, Cumming School of Medicine,University of Calgary, 3280 Hospital Drive NW, Rm 3C56 Health Research Innovation Centre, Calgary, AB, Canada T2N 4Z6.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0

Introduction

The translation of evidence into policy and clinical care throughimplementation in the health care system is a core issue facinghealth care delivery system transformation around the world.Evidence-based practices can be implemented through the aid ofoperations research methods to redesign health care deliverysystems and improve patient outcomes and health system per-formance [2]. In a previous article [3], the ISPOR Task Force onDynamic Simulation Modeling Applications in Health Care DeliveryResearch Emerging Good Practices introduced the fundamentals ofdynamic simulation modeling by defining complexity and healthcare systems interventions and identifying the types of health caredelivery problems for which dynamic simulation modeling can beused. The article introduced three dynamic simulation modelingmethods most commonly used—system dynamics (SD), discrete-event simulation (DES), and agent-based modeling (ABM)—andreviewed where they differ from models more typically used ineconomic evaluation such as Markov models and decision trees.Finally, the System, Interactions, Multilevel, Understanding, Loops,Agents, Time, Emergence (SIMULATE) checklist was developed andpresented as a tool to help researchers decide whether dynamicsimulation modeling is an appropriate method for modeling the

effects of a particular policy or health care intervention on a healthcare system. The SIMULATE checklist identifies eight elements(System, Interactions, Multilevel, Understanding, Loops, Agents,Time, Emergence) that characterize problems that could beaddressed more effectively using dynamic simulation modelingmethods rather than other modeling methods.

This report builds on this work by systematically comparingeach of these three dynamic simulation modeling methods, and byidentifying criteria for selecting the most suitable method amongthese three alternative methods depending on the type of problembeing addressed. In cases in which different dynamic simulationmodeling methods may be used for the health care deliveryproblem, several specific elements were identified for differentiat-ing the methods such as the perspective, the origin of dynamicinteractions in the system, and resource requirements in terms ofmanpower and costs. Following the description of the threemodeling approaches, we provide emerging good practices differ-ent from guidance for other modeling studies reported elsewhere,covering all aspects from the engagement of decision makers inthe model design through to model maintenance and upkeep. Weconclude with recommendations about how to apply these meth-ods in practice to inform decision making and by identifying areasfor continued methodological development in applying dynamicsimulation models to health care delivery research.

Background to the Task Force

In October 2013, the ISPOR Health Science Policy Councilrecommended to the ISPOR Board of Directors that an ISPOREmerging Good Practices for Outcomes Research Task Force beestablished to focus on dynamic simulation modeling methodsthat can be applied in health care delivery research andrecommendations on how these simulation techniques canassist health care decision makers to evaluate interventions toimprove the effectiveness and efficiency of health care delivery.The Board of Directors approved the ISPOR Dynamic SimulationModeling Emerging Good Practices Task Force in November2013.

The task force leadership group is composed of experts inmodeling, epidemiology, research, systems and industrial engineer-ing, economics, and health technology assessment. Task forcemembers were selected to represent a diverse range of perspec-tives. Theywork in hospital health systems, research organizations,academia, and the pharmaceutical industry. In addition, the taskforce had international representationwithmembers fromCanada,The Netherlands, Colombia, and the United States.

The task force met approximately every 5 weeks by tele-conference to develop an outline and discuss issues to beincluded in the report. In addition, task force members met inperson at ISPOR International meetings and European con-gresses. All task force members reviewed many drafts of thereport and provided frequent feedback in both oral and writtencomments.

Preliminary findings and recommendations were presentedin forum and workshop presentations at the 2014 ISPOR AnnualInternational Meeting in Montreal and ISPOR Annual EuropeanCongress in Amsterdam. In addition, written feedback wasreceived from the first and final draft reports’ circulation to the190-member ISPOR Modeling Review Group. Comments werediscussed by the task force on a series of teleconferences andduring a 1.5-day task force face-to-face consensus meeting. Allcomments were considered, and most were substantive andconstructive.

Comments were addressed as appropriate in subsequentversions of the report. All written comments are published at

the ISPOR Web site on the task force’s Webpage: http://www.ispor.org/TaskForces/Simulation-ModelingApps-HCDelivery.asp. The task force report and Webpage may also be accessedfrom the ISPOR homepage (www.ispor.org) via the purpleResearch Tools menu, ISPOR Good Practices for OutcomesResearch, heading: Modeling Methods.

In the course of task force deliberations, in response tospecific comments and suggestions from reviewers, and agrowing concern about length, it became apparent that thematerial would need to be covered in two task force reports tobe thorough, covering the essential points, yet keep the reportreadable and digestible. With permission from the editors ofValue in Health, the material was split into two articles.

The first article “Applying Dynamic Simulation ModelingMethods in Health Care Delivery Research—The SIMULATEChecklist: Report of the ISPOR Dynamic Simulation ModelingApplications in Health Care Delivery Research EmergingGood Practices Task Force,” is a primer on how dynamicsimulation modeling methods can be applied to healthsystem problems. It provides the fundamentals and defini-tions, and discusses why dynamic simulation modelingmethods are different from typical models used in economicevaluation and why they are relevant to health care deliveryresearch. It includes a basic description of each method(system dynamics, discrete-event simulation, agent-basedmodeling), and provides guidance on how to ascertainwhether these simulation methods are appropriate for aspecific problem via the SIMULATE checklist that wasdeveloped by the task force.

This second report provides more depth, delving into thetechnical specifications related to the three dynamic simulationmodeling methods. It systematically compares each methodacross a number of features and provides a guide for emerginggood practices for outcomes research on dynamic simulationmodeling. This report concludes by providing recommendationson the application of dynamic simulation modeling methods toadd value to informed decision making, with an emphasis onproblem definition and stakeholder engagement and identifiesareas where further methodological development will likelyoccur given the growing “volume, variety, velocity” of “bigdata” [1].

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0148

Overview of Commonly Used Dynamic SimulationModeling Methods

The first task force report [3] has identified three modelingapproaches commonly used—SD, DES, and ABM.

System Dynamics

SD is a simulation modeling method used for representing thestructure of complex systems and understanding their behaviorover time (dynamic). It captures complex and nonlinear relation-ships between components of a complex system dynamically. It isrooted in “industrial dynamics” and was developed by Jay For-rester, Massachusetts Institute of Technology, in the 1950s [4–6].

SD is a “top-down” learning approach that informs under-standing of the dynamic behavior of the system being studied.The process of SD modeling includes gathering insights, valida-tion, revisiting the results, and updating the model to reflectthese learnings. SD has been used to model problems in variousfields and understand the systems better [7].

SD is based on the core assumption that the behavior of thesystem is a consequence of the system structure and not externalforces or factors [8]. The structure of the system can be under-stood as the feedback loop structure, and the structure ofaccumulations and rates, which generate the behaviors (Fig. 1).

At a more technical level, SD models involve 1) a higher levelof aggregation than do other dynamic simulation modelingmethods, 2) quantities that change over time and can be formu-lated mathematically in continuous time as differential equa-tions, and 3) feedback loops (balancing or reinforcing).

SD models traditionally aggregate the population in states andsubpopulations rather than analyzing at the individual level.Therefore, SD models provide a deterministic cross-sectionalview of a system by counting over time the number of peopleexhibiting particular combinations of characteristics or in partic-ular (e.g., health) states. Hence, actions taken in one time periodinfluence the actions taken in subsequent periods [8].

Quantities that change over time are called variables [9].Variables can be one of three types—stock, flow, or auxiliary(Fig. 1). The state of the system is described by the stock variables.Stocks are accumulations or aggregations of something, forexample, people, beds, or oxygen.

Stocks (also known as state variables) are accumulations ofinflows and outflows over a period of time. When the system isstopped for an instant, stocks will have a value that determinesthe state of the system at that instant. The flow variables (alsoknown as rates of change) change the accumulations of thestocks and control the rates of flow. Flows (rates) feed in and outof stocks and have the same units of stocks per time unit, forexample, people per hour, beds per year, or oxygen per minute.

The assumption used to build the SDmodel is that the structurecan be represented using a series of stock and flow variables [4].The flow variables determine how fast a system is changing. Therate equation recognizes the system’s goal, compares the goal withthe current state of the system, and makes corrections to narrowthe discrepancy and get closer to the goal [4,10]. The stock and flow

variables are interlinked with a series of cause and effect relation-ships that determine the underlying flows of matter and informa-tion within a system. These relationships and the flow bring thevarious components together as a single holistic entity as opposedto having multiple individual components [9].

Feedback processes describe the circular relationshipsbetween variables in the system. These processes include reac-tions of actors and the system to decisions that affect them andtheir goals. An important concept in SD is nonlinearity. Thisconcept is tied to the existence of feedback processes, and itmeans that an effect is seldom proportional to the cause. Ourdecisions and actions today affect our actions and decisionstomorrow in a nonlinear way as the system and other conditionschange [7,11]. Results from decisions may be immediately appa-rent or may be dormant and become apparent after a delay. Thisdelay is due to the accumulation dynamics and the feedbackstructures in complex social systems. Social systems containfeedback processes both reinforcing and balancing [11,12].

SD can be used for policy analysis and design for problems incomplex social, managerial, economic, and ecological systems.Any dynamic system is characterized by interdependence,mutual interaction, and feedback. Most applications can becategorized as 1) recognition and identification of behavioralpatterns in a system, for example, in an organization; 2) gaininsight into the processes of a system and the consequences ofdecisions; 3) identification of leverage points and/or structures inthe system to generate change and foster system redesign; and 4)reproduction of a given behavior (reference mode)[7].

As an example, Milstein et al. [13] used SD to study andevaluate the US health system reform that included three mainstrategies: coverage, care, and protection. The model wasdesigned to address questions around the impact of thesestrategies nationwide, individually and together. This is a typicalexample of a broad problem with systemwide implications thatrequires a holistic perspective with attention to dynamic proc-esses within the system and its structure. The modelers esti-mated the relative and combined effects of the three strategiesfrom 2000 to 2010 and asked what might have happened had theUnited States taken decisive action in these three areas duringthat decade in terms of reducing avoidable deaths and loweringhealth care costs for Americans. Results and simulated scenariosshow that all three strategies have the potential of savingmillions of deaths while offering good economic value. Beyondthe 10-year horizon, however, protection yields the best result bysaving more lives and money. The model offers a useful way ofobserving how the US health care system tends to respond tolarge-scale interventions. Scenarios let planners compare thesemajor interventions regarding direction, timing, costs, and bene-fits. The interpretation of these results is as follows: 1) a 10-yearhorizon tends to obscure the full effect of interventions;2) protective interventions could effectively complement cover-age and care by ensuring that people stay healthy for longer,hence reducing excess demand on the health care system; and3) because population-based prevention policies take longer toyield their full economic and health benefits, they should not bepostponed until positive effects are seen from coverage and care.

Populationwith severe

osteoarthritis(Stock)

Referral rate tospecialist (Flow)

Average time toreferral (Auxiliary)

Population withsevere

osteoarthritisunder specialist

care (Stock)

Population withsevere osteoarthritis

(Stock)Referral rateto specialist

(Flow)

Feedback Loop

Fig. 1 – Basic structure of a system dynamics model. Vensim Personal Learning Edition® (Vensim Software, Harvard, MA) usedto create Figure 1.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 149

Model outputs and level of insight are varied and dependenton the purpose of the model and the type of problem. In generalterms, SD can produce patterns and trends, as well as meanvalues. SD allows for the elicitation of “mental models” fromstakeholders involved in the discussions and also from thoseinvolved in the model-building process. A mental model is anexplanation of the stakeholder’s thought process about howsomething works in the real world [7,14]. This methodologygenerates a high level of insight about the problem and thesystem under study at strategic and policy levels.

Interpretation of outputs also depends on the type of problemand the purpose for which the model is designed. The model willnot give a unique answer or optimal answer to a problem.Instead, the model allows experimentation to test alternativestrategies (“what-if scenarios”) for system intervention andobserving their potential outcomes to inform decision makingbefore implementing a particular strategy.

Discrete-Event Simulation

DES is used to represent processes at an individual level wherepeople may be subject to events, whether they be decisions oroccurrences over time. DES is a simulation method that capturesindividual-level heterogeneity and is used to characterize andanalyze queuing processes and networks of queues where thereis an emphasis in the utilization of resources [15].

Core concepts in DES are events, entities, attributes, queues,and resources [3] (Fig. 2). Figure 2 illustrates these core conceptsin the context of a simplified emergency department (ED) processto triage and assess patients. Patients are individual entities withparticular characteristics that flow through the processes of“triage and admission” and “consult and procedure,” both ofwhich take a certain amount of time and require resources suchas a triage nurse and a physician. Patients wait in queues for bothprocesses, proceed through them, and are finally discharged.

Although DES has been applied for health economic modeling[16], most problems or questions that DES can help analyze are thoseregarding resource utilization and queues, that is, waiting times. Inaddition, in health care specifically, DES can be useful to analyzeeffects on health-related outcomes. DES is also useful for problemsfor which it is particularly relevant to be able to capture the changingattributes of entities, for example, patients, and for which theprocesses to be characterized can be described by events [16].

An example of a problem that can be addressed with DES is tofacilitate decision making for a health system to invest inexpansion of ED and/or intensive care units (ICUs) based onvariable patient flow. The flow of patients into a hospital istypically limited by ED capacity; ICUs also limit flow at timeswhen admissions are high, or patient flow increases from otherparts of the health system such as the ED, surgery, or decom-pensated patients in general medicine [17]. Thus, future patientsrequiring critical care are held in the ED for longer times, andthose who may have had scheduled high-revenue appointmentssuch as surgery have to be cancelled and rebooked. The lack ofbed availability in the ED prohibits additional patients from beingaccepted at a facility. This classical case leaves many healthsystems constantly investigating whether to expand ED capacity,as well as downstream units such as the ICU, to enhance flow.

This case and proposed expansion has several consequences.Expansion can increase revenue for a facility when higher

volumes of patients are flowing efficiently through the ED andthe ICU; however, investment in facility expansions that areunderutilized on a regular basis may not be cost-effective. Therate of surgical procedures could be limited by bed capacityafterwards. In addition, patients scheduled for high-revenuesurgical procedures should have beds on reserve.

DES is a flexible, yet data-intensive modeling approach.Flexibility is defined by the flexibility in building the modelstructure representing the processes, and the different sourcesof inputs and data formats that may be used, as well as the easeof model structure modification and upkeep. DES also allowscumulative probability functions for variables in the model,allowing modeling at the patient level. Outputs of DES can bepoint estimates as well as mean values and distributions ofvalues. Events are traceable because individual entities arefollowed throughout processes. Results of DES scenarios andexperimentation can be interpreted or used for system perform-ance indicators such as resource utilization, waiting times,number of entities in queues, and throughput of services orproducts.

Agent-Based Modeling

ABM is a simulation method for modeling dynamic, adaptive, andautonomous systems [18]. It is useful to discover patterns oremergence by using “deductive” and “inductive” reasoning. Incontrast to SD or DES models, which begin with a “top-down”approach of mapping a system or process, ABM begins with a“bottom-up” approach. The foundation of an ABM model beginswith individual objects and describes their local behavior withlocal rules. At the core of an ABM model, these “autonomous”and “interacting” objects are called agents. Agents are social andinteract with others, they live in an environment, and their nextactions are based on the current state of the environment. Inaddition, an agent senses its environment and behaves accord-ingly on the basis of simple decision rules. Agents may haveexplicit goals to maximize or minimize and may learn and adaptthemselves on the basis of experience. The definition of agentbehaviors uses a range of simple to complex mathematical logicoperators.

The three core concepts that form the basis for ABM areagency, dynamics, and structure [19]. Agency means that agentshave goals and beliefs and can act. Examples of agents caninclude patients, providers, and administrative staff. Theseagents can move through space and time, interact with eachother, learn, and disseminate new learnings to other agents intheir social network. Dynamics means that both the agents andtheir environment can change, develop, or evolve over time(Fig. 3). Structure is emergent from agent interaction. Forinstance, how populations of people tend to aggregate in certainlocations on the basis of predefined behaviors is an example ofagent interaction.

ABM has been applied to various modeling scenarios: marketforecasting, human migration and movement patterns, urbandesign, resource management (e.g., water), political mobilization,health, and epidemiology [20]. As a general rule, the more activethe objects (e.g., people, vehicles, and products), the moresuitable ABM is to apply as a modeling technique. ABM modelscan help address problems that involve both deterministic andstochastic processes. For example, when patient A becomes sick,

Fig. 2 – Basic structure of a discrete-event simulation model. AnyLogic used to create Figure 2.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0150

she might always go to hospital B to seek care because it is closestto her home (deterministic) but the likelihood that she willcomply with the prescribed drug therapy is specified by aprobability distribution (stochastic).

ABM is a rapidly maturing health modeling technique wellsuited to addressing public health planning, policy needs, andhealth care infrastructure investment decisions. The attainmentof specific population health goals can be simulated at thepopulation level, and the specifics of investments needed toachieve these goals can be investigated in more detail. Primarygoals can be defined by disease outcomes, efficiency measures,return on investment, or costs [21].

An example of ABM is the study published by Macal et al. [22]in which they model the community-associated methicillin-resistant Staphylococcus aureus (CA-MRSA) epidemiology in Chi-cago to identify target interventions to reduce transmission. Theydeveloped an agent-based model to represent heterogeneity inpopulation locations, behavior, and contact patterns, which arerelevant for transmission and control. Several sources of datawere used including national survey data and a comprehensiveliterature review to establish transmission probabilities. Themodel represents variation in sociodemographic characteristics,locations, behaviors, and physical contact patterns. The ABMgenerates temporal and geographic trends in CA-MRSA incidencesimilar to Chicago from 2001 to 2010. Colonized agents ratherthan infected agents were shown to be the source of 95% oftransmission events. This is an important finding because cur-rent paradigms in MRSA control in the United States focus on theinfected population, which are unlikely to have a populationwideimpact. The Chicago CA-MRSA ABM included places such ashouseholds, workplaces, schools, gymnasiums, nursing homes,hospitals, jails, and college dormitories. Each agent in the ABMhas a “daily activity profile” that determines the times he or sheoccupies each location. Social contact between agents occurswhen multiple agents occupy the same location at the same time.Depending on age, for example, some agents are assigned toschools. Also, households are assigned visits to other householdswithin the same census area and other areas. Gymnasiums andhospitals are assigned using the geographically closest one.Algorithms consider transmission probabilities depending onthe activity; for example, athletic activities are considered tohave a higher risk.

ABM is able to model various outcomes, such as epidemio-logical disease burden, population sociodemographic character-istics, health status, system utilization, patient preferences,health care provider preferences, behaviors, and costs [21,23].

Agents can be located in a geospatial environment defined bya geographic information system map, with real latitudeand longitude coordinates, thus enabling the modeler to con-ceptualize proximity and distance that agents have in relation toeach other. When agents are connected according to their closestand most frequent interactions (e.g., family members), theresulting network can yield insight into how information iscommunicated throughout a community or diseases spread.These properties make ABM well suited to generate insights intopatterns of health and behavior of large populations over time.

The strength of interpretation of ABMs lies in the results ofsensitivity analyses. ABM adds a new dimension to traditionalsensitivity analyses by enabling the modeler to test a range ofassumptions about human behavior: how people learn, how theydisseminate information to their peers or families, and how theychange their behavior in response to new information, incen-tives, or penalties. For example, introducing a new diabetesprevention program versus lowering the co-pay for diabetesmedications will produce different behaviors among patientsubgroups. ABMs can be a powerful tool to test assumptionsabout human behavior, assist planning, and forecast the effectsof different health system scenarios on population health.

Comparison of Dynamic Simulation Modeling Methods

Table 1 summarizes the relevant aspects to be considered whencomparing the three dynamic simulation modeling methods—SD, DES, and ABM. This table guides researchers and decisionmakers in determining which of the three methods is appropriateto address the problem at hand and will meet the purpose of themodeling endeavor. The table identifies 14 aspects to comparethese methods and differentiate them.

We highlight five key aspects for the initial selection of adynamic simulation modeling method. Emphasis must be on thetype of problem to be addressed [3,24] and the perspective requiredto answer the research questions. SD is better suited for problemsat the strategic level and for which a systemwide perspective isrequired (top-down), whereas DES focuses on process-centeredproblems (top-down) and operational/tactical questions. ABM canbe appropriate for problems at multiple levels, that is, strategic,operational, and tactical; however, ABM is most suited toindividual-level problems focused on how individual interactions(bottom-up) generate emergent system behaviors and structures.

The individual resolution characteristic of ABM is shared byDES, as well as its handling of time, that is, discrete, yet, the origin ofthe dynamics is quite different. In ABM, active heterogeneous

Fig. 3 – Basic elements of an ABM model. ABM, agent-based modeling. © The AnyLogic Company 2015. Reprinted withpermission.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 151

Table 1 – Comparison of dynamic simulation modeling methods [19,25–28].

Method

Aspect System dynamics Discrete-event simulation Agent-based modeling

Type of problems Strategic, operational Operational, tactical Strategic, operational, tactical

Perspective System-oriented, emphasis on dynamiccomplexity (top–down)

Process-oriented, emphasis on detail complexity (top–down)

Individual-oriented, dynamic and detailcomplexity (bottom–up)

Resolution Homogeneous entities, continuouspolicy pressures and emergentbehavior

Individual heterogeneous passive entities, attributes, andevents

Individual heterogeneous active agents, decisionrules

Origin of dynamics Deterministic endogenous fixedstructure

Stochastic endogenous fixed processes Agent–agent, agent–environment interactions andadaptive behavior of agents

Handling of time Continuous Discrete Discrete

Approach Exploratory and explanatory Explanatory Exploratory and explanatory

Basic building blocks Feedback loops, stocks, and flows Entities, events, queues Autonomous agents, decision rules

Data sources Broadly drawn: qualitative andquantitative

Numerical with some judgmental elements Broadly drawn: qualitative and quantitative

Unit of analysis Feedback loops and stocks’ dynamics Queues, events Decision rules, emergent behavior

Mathematical formulation Differential equations Mathematically described with logic operators Mathematically described with logic operators anddecision rules

Outputs Understanding of structural source ofbehavior modes, patterns, trends,relevant structures, aggregate keyindicators

Point predictions, performance measures Detailed and aggregate key indicators,understanding of emergence due to individualbehavior, point predictions

Model maintenance Upkeep may require large structuremodifications, global

Upkeep may require process modifications, global. Allowsfor local modifications regarding individualheterogeneity

Upkeep may require simple local modifications

Development time Dependent on the problem, purpose, andscope of the model; these models mayrequire less time to be developed

These models are more data intensive. This requires moretime regarding obtaining data and data analysis toprepare model inputs. Programming and calibration areusually very time consuming

These models can be data intensive, whichrequires data analysis and time to obtain thedata. Programming and calibration are usuallyvery time consuming

Cost In general, SD is less costly than are DESand ABM. This involves datarequirements, and skill sets needed

Because of costs associated with data and skill setsrequired, these methods tend to be more costly than isSD

If the model is data intensive or requires primarydata collection, costs may increase. Skill setsrequired may also increase the costs

ABM, agent-based modeling; DES, discrete-event simulation; SD, system dynamics.

VALUE

INH

EALTH

18

(2015)147–160

152

agents interact with each other, make decisions, and adapt tochanges in the environment, whereas entities in DES are hetero-geneous passive objects that are “worked-upon” by specificresources in a particular stochastic process. In comparison, inSD models, homogeneous populations flow through a determin-istic system structure in continuous time.

Technical aspects such as mathematical formulation and units ofanalysis are useful in understanding where the differences liebetween methods and how they work. Nonetheless, these are ofsecondary consideration because they should not drive the initialselection of a method. The remaining comparative aspects arediscussed throughout the report.

Criteria for Selecting a Dynamic Simulation ModelingMethod

Selection of a dynamic simulation modeling method for some orall components of a model will generally take into accountvarious considerations. We provide an overview of these criteriahere and identify additional resources for further information[29–33]. Figure 4 provides a high-level summary of criteria forselecting an appropriate modeling method.

The most central consideration is model purpose, that is, whywe are building the model—the problem or research questionbeing investigated [34]. This focus on model purpose reflectsthree facts. First, all models—like maps—are abstractions that are“wrong” in the sense that they omit myriad details. Second,selection reflects the fact that although the modeling methodsdiscussed here vary in the details of the formalisms, they differeven more fundamentally in terms of their aims and the ques-tions that they prioritize, that is, what we are modeling—object ofstudy (scope) [34]. For example, SD modeling emphasizes repre-sentations and processes that help shift stakeholders’ mental

models. ABM emphasizes agent-agent and agent-environmentinteraction and multiscale insights. DES emphasizes insights intothe impact of resource availability—and sometimes location—onprocess efficiency, workflow, and throughput. Finally, and as willbe discussed further below, although it might be possible to useany of these methods to model a wide range of problem types,the different methods differ in their ease and capacity to addressspecific types of questions. For example, the implications inher-ent to SD model assumptions and structures are that SD modelsare not ideally suited to inform reasoning about variability inindividual interactions with systems characterized by highlyheterogeneous populations, or evaluating interventions struc-tured around network dynamics or individual history becauseSD models are typically aggregate. Similarly, it is infeasible to useABM source code as a discussion tool for interactive stakeholderparticipation, feedback, and refinement. Additional specifics arediscussed below.

Also of critical importance in method selection is the degreeto which one is seeking to capture agent interactions, theavailability of requisite skill sets (e.g., recourse to softwareengineering expertise for ABM), the available level of process-related knowledge and empirical data, what time duration ofsimulation is viewed as acceptable, the degree of flexibilitysought in model scope (e.g., types of heterogeneity incorporated—more flexible for individual-based models than for SD/com-partmental models), the nature of interventions or counterfactualsituations to be represented, the character of outputs of interest,and the importance of differences between individuals accordingto characteristics, history, and spatial network context, theimportance of insights at multiple scales, the need to supportscaling to large or highly heterogeneous populations, the degreeto which one is seeking the simulation to reproduce statisticalvariability, and whether one seeks to use tools to assist in modelanalysis or to reason about the possible behavior of the

Fig. 4 – High-level summary of criteria for selecting a dynamic simulation modeling method [34].

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 153

simulation over broad ranges of parameters (rather than con-cerning particular discrete scenarios).

Modeling methods that individualize system agents are nota-bly attractive for capturing information with regard to imple-menting interventions or governing processes that depend onagent history (e.g., on an individual’s past care pathways) andlearning and memory effects. The capacity to maintain suchlongitudinal information raises particular opportunities for cali-bration and validation against data from individual-level sources.Such individual-based models also excel in representing largeamounts of heterogeneity, in contrast to the lower levelof granularity that occurs in aggregate models such as SD.Such attribute-based disaggregation scales poorly as the numberof distinctions by which we wish to capture heterogeneityincreases [31].

As a result, individual-based models confer substantialadvantages in capturing not only diverse, continuous, and dis-crete attributes (e.g., sex, income, body mass index, birth weight,and preferences), but especially evolving conditions such ascomorbidities, but also (possibly dynamic) situational context(spatial or network position and connections, with associatedexposures, localized perception, resource availability, choice sets,influencing local factors). The individual-based character of ABMand DES models supports not only scalability but also flexibilityin evolving representations of both discrete and continuousheterogeneity. Adding—or removing—a new dimension of heter-ogeneity for an individual-based model is a simple, modularoperation.

This contrasts with the situation in aggregate models, inwhich a similar change to the heterogeneity captured by a modelis a more complicated operation affecting the structure extendingacross much of the model. As a result, ABMs support morenimble experimentation with the degree to which heterogeneityis considered. Of particular note in ABMs is the capacity tocapture empirically grounded, rich models of individual decisionmaking (e.g., using elements of discrete choice theory), which canaid in endogenously capturing behavioral responses of thepopulation to interventions.

The ability to capture such heterogeneity can aid in not onlycapturing behavioral variability in underlying processes but alsoevaluating targeted interventions in specific populations. More-over, the actor-centric character of ABM supports the straightfor-ward and transparent construction of multilevel models, whosestructure—captured with nested or network actors at differentlevels of scale—mirrors that of the external world. Such modelscan then be used to characterize emergent behavior at multipledistinct levels of a system, opening opportunities for not onlyunderstanding intervention impact on and across multiple levelsof intervention but also enhancing the flexibility of calibrationand validation.

The stochastic character of most individual-based modelsrepresents both an asset and a liability. On the positive side,stochasticity supports substantial analysis insights, such asexplaining empirical variability [35] and testing of interventionsand scenarios under expected uncertainties. Stochasticity, how-ever, also imposes a substantial performance burden in additionto the already heavy computational demands of individual-basedmodels, particularly because of the need to run the model manytimes as part of different types of Monte Carlo simulations. In asimilar fashion, the flexibility of agent-based models is a double-edged sword, permitting a tremendously wide repertoire of possiblemodel designs, but in a way that current ABM modeling environ-ments require software engineering expertise [36]. For larger models,this can require ongoing involvement of individuals with program-ming (preferably, software engineering) background in model con-struction, maintenance, debugging, and quality assurance.

Although SD models can be applied at a wide variety oflevels of aggregation—with some classic models involvingindividual-level use of such models [4]—in health applications,they are most commonly used as compartmental models, inwhich the model categorizes an underlying population into a setof internally homogenous states. In contrast to agent-basedmodels, smaller SD models are often far faster to construct,maintain, share, and discuss with stakeholders, and—often—tounderstand. Both because they involve fewer “moving parts” andare not stochastic, such models are also typically faster toexecute than are ABMs and DES models. Because of SD models’reliance on the creative use of a small modeling vocabulary—most centrally, stocks and flows—less computationally special-ized skill sets are required for working with them.

Because of the lower software engineering burden, more timecan be spent learning from a smaller model, rather than main-taining it, and learning from model changes is considerablyfaster. The formal aspects of SD model design are readily under-standable, which not only facilitates model building but alsodistinguishes the technique in terms of its support for participa-tory processes.

By virtue of SD’s use of both qualitative and quantitativemechanisms that can be viewed and understood by those withdiverse backgrounds and training, and from the inception of aproject, it supports refined and time-tested processes for use ingroup-model building. The consequent benefits include helpingto elicit stakeholder and community mental models and breakdown barriers to effective communication among stakeholders,and this can aid greatly in securing buy-in and energizing astakeholder group. The low execution burden associated withsmaller models not only lowers the learning curve but alsosupports participatory settings that leverage rapid interactionand adaptation with scenarios formulated by stakeholders,community members, or other nontechnical participants.Although many system dynamics practitioners do not seek toconduct formal mathematical analyses with their models, SD isfurther distinguished by its capacity to support closed-formanalysis.

In addition to benefits accruing to DES as an individual-basedmethod, it offers strengths in the context of defined workflows,associated with multiple stages of processing for some class ofindividual entities (patients, vials of vaccine, etc.), where suchprocessing is typically contingent on the availability of limited-capacity resources of one or more types, and where such aresource can be in use only by one entity at any given time.Such resources may be fixed in space (a magnetic resonanceimaging scanner, an examination room), portable (a blood pres-sure cuff, wheelchair, or an intravenous drip), or be mobile withsome limited agency (a clinician, a nurse’s aide, etc.). DES caneffectively and concisely model situations with passive entities,particularly capturing queuing behavior, the impact of resourceavailability, arrival time distributions, and so forth on waitingtimes, throughput, queue length, resource utilization, qualityof care, and other outcomes of interest in health servicesresearch.

By mapping entities to space, DES can further represent theimpact of the physical environment— for example, facility layoutand resource placement—on such outcomes as travel time as anemergent phenomenon. Although DES—like ABM—is character-ized at an individual-based level, it would require relatively moretime, effort, and software engineering skills to build an ABMcharacterizing such interactions between resources, agents, spa-tial layout, queuing, and so forth. Although DES shines inrepresenting comparatively more passive entities that are “oper-ated upon” by processes, it offers far less flexibility for represent-ing situations when entities need to interact in a flexible fashion

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0154

with each other, with the environment, or otherwise exert a highdegree of agency.

Dynamic Simulation Modeling Good Practices

Although operations research methods are widely used in indus-trial and business operations to improve effectiveness andefficiency of processes, they are still relatively new in healthapplications [7]. Yet, the feasibility and relevance of thesemethods to inform health care delivery system planning anddecision making for improving system efficiency have beendemonstrated [37]. Developing good practice guidelines in simu-lation modeling methods is important to the scientific field andwill advance the application of dynamic simulation modelingmethods in health [38]. A combined ISPOR-SMDM taskforce hasrecently published a series of seven articles providing an over-view of good practices for modeling studies to inform health caredecisions [38]. Also recently, the Consolidated Health EconomicEvaluation Reporting Standards checklist provides an overview oftechnical issues to report with regard to health economic model-ing studies [39]. There is, however, a lack of clear and accessibleguidance for selecting and using dynamic simulation modelingmethods to evaluate interventions in health care delivery sys-tems. This task force report discusses some of the considerationsspecific to dynamic simulation modeling studies and providesguidance to modelers, researchers, and decision makers. Theguidance is well aligned with the SIMULATE checklist intended toguide the investigator in choosing dynamic simulation modeling[3] and the criteria for selecting the appropriate dynamic simu-lation modeling method as presented in this article.

In principle, key differences between dynamic simulation andother modeling methods are as follows: 1) the complex combina-tion of interactions and scenarios in the model required by theproblems; 2) their capability of capturing emergent behaviors; and3) the continuous stakeholder engagement in the iterative proc-esses of definition and testing of model scope and assumptions,model verification, and calibration. The emerging good practicesdiscussed here highlight the basics. For more detailed guidance ondesigning and building each type of dynamic simulationmodel, thereader is referred elsewhere throughout the text.

Model Design and Assumptions in Dynamic SimulationModels

The most fundamental decision involving design of a givenmodel version is that related to the problem to be addressed,the model purpose and scope [24], particularly with regard to thereflective delineation of factors falling into each of three catego-ries: 1) endogenous factors calculated as part of model operation,and generally exhibiting emergent behavior; 2) exogenous factorsrepresented in the model, but according to prespecified assump-tions using constant values or time series; and 3) factors that areconsciously ignored. Data availability may constrain the scope ofthe model; however, at this stage, it is important to have acomplete definition of the problem regardless of data limitations[24,38]. Other fundamental decisions to be made upfront includethe scope of the model population, temporal and spatial scales,including time horizon, spatial extent and topology, and anydiscretization imposed.

It is advisable to conceptualize the model incrementally whenmaking decisions involving model scope. Such discipline isparticularly important for the ABM method, whose very flexibilityraises the risk of scope creep and overly casual inclusion ofadditional factors. When decision makers are deeply involved inthe process, preliminary models become tools for discussion thathelp better define the scope of the problem at hand and

assumptions and elicit ideas, solutions, and interventions [40].Being engaged in this iterative process, decision makers find theirmental models and preconceived ideas about the system chal-lenged and are obliged to think broadly about the problem andreflect on the system in which it is embedded [41,42]. Hence,decision makers are forced to engage in operational thinking anddevelop intuition about the system, thinking about the nuts andbolts of the system and how it really works, including humanbehavior assumptions, thereby informing the design of thesystem and interventions realistically and more accurately [12].

Iterative Model-Building Strategy and Data Requirements

Simulation models use empirical data in two primary capacities.The first use is for model parameterization (with the data beingincorporated directly or indirectly—for instance, via backing out—into model formulation). The second use lies in model calibra-tion, where the data is used as evidence to match againstemergent behavior of a model. For the first of these uses,appropriate documentation is particularly important, and proba-bilistic sensitivity analysis can be of high value in examining theresponse of model outputs—and particularly cross-interventiontrade-offs—in light of uncertainties concerning parameter values.We emphasize again the importance of maintaining metadataconcerning the provenance of model parameters as a routinecomponent of model documentation.

Given the counterintuitive behaviors frequently associatedwith simulation models, modelers are advised to build andpopulate the model incrementally and adaptively, cyclingthrough steps of adding small pieces to a model, running themodel for insight cross-checked for invariant behavior [43,44]. Inaddition, the incremental development [45] enhances modelquality by helping to ensure that latent defects are spotted asquickly as possible (both due to visible behavior and whenrunning suites of formal automated tests) [43,44].

Dynamic Simulation Model Validation, Verification, andCalibration

Validation, verification, and calibration processes are key. Validationfocuses on the correspondence between a model and the real-world phenomena under investigation or to be addressed,whereas verification seeks to understand the extent to which themodel is true to its original design.

Traditional model validation approaches vary widely amongthe different modeling methodologies discussed here, with DEShaving a particularly strong body of practice on the subject [46–49].Many models also undergo calibration processes, whereby simu-lation model parameters are adjusted such that the emergentbehavior of a simulation model compares most closely withempirical data or reference modes. For such processes, informationshould at least be provided regarding the calibrated values ofmodel parameters. Of particular interest are 1) opportunities tocompare model estimates against outcomes from prospectiveinterventions or natural experiments, 2) testing of predictivevalidity of results obtained, and 3) the role of domain experts inproviding feedback on model assumptions and outputs—particu-larly visualization output—to help identify either quantitative orqualitative discrepancies from their experience [47].

Model verification draws heavily on principles and practices ofsoftware quality assurance [50,51]. Some of the most importantfactors promoting model quality have to do with process commit-ments on the part of the modeling team. Examples includeadherence to regular peer review in both its informal varieties(e.g., pair modeling and peer desk check of models) and to formalmodel inspection, widely acknowledged as a powerful bestpractice in software development. Where a model is being

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 155

worked on by more than one party, such best practices furtherinclude practices such as pair modeling (an adaptation of pairprogramming) [52], buddy testing (where one party tests the modelcomponents contributed by another party) [44,53], version control,continuous integration efforts and their associated smoke tests,automated testing, stylistic analysis, and so forth. Although littleapplied in modeling, formal strategies for estimating the occurrenceof latent model implementation defects can confer considerablevalue for larger model development efforts. It is highly valuable tointegrate model validation throughout the iterative model develop-ment process noted above, rather than viewing it as a gatingexercise to be undertaken only at the end of the modeling [47].

Important technical good practices for verification include theuse of assertions to check model assumptions. For instance,assumptions that stocks of physical quantities are non-negative,that one model variable is strictly less than another, or that severalquantities sum to unity. Where possible, the use of techniques suchas unit testing and mocking is highly desirable for enhancing thetestability of ABMs. Models that include considerable levels ofalgorithmic specification will also benefit from adherence to qualitycoding standards, architectural principles, and possible use ofaspect-oriented mechanisms to capture cross-cutting concerns[54]. These models will also benefit greatly from periodic refactoringof code, improving the clarity, modularity, transparency, flexibility,and generality of the implementation code without changing itsbehavior [55].

Finally, model calibration seeks to match emergent modeloutput against empirical data and often provides much addi-tional confidence into model suitability and fitness for purpose.Both calibration and cross-validation processes have particulartexture for individual-based models, as they can leverage addi-tional types of data (e.g., longitudinal data, spatial and topologicalpatterns, and patterns at multiple scales), but the data are almostnever sufficient to unambiguously estimate model state in suchmodels. Inability to meet validation criteria is best recognized notas a failure of the modeling project, but as an opportunity torefine the model and the assumptions behind it (recognizing themodel’s role as a tool that helps one learn quickly and reliably byspotting inconsistencies between one’s theories of the world andavailable empirical evidence).

Validation of dynamic simulation models in their entirety canbe challenging because of model complexity and lack of com-parative data. As such, the iterative model-building process indynamic simulation modeling is critical to build the confidence ofthe modeling team as the model is continuously cross-checkedby all stakeholders through observation.

Analysis of Outputs and Sensitivity Analysis

A key use of models lies in the analysis of outputs associatedwith model scenarios. Sensitivity analysis— structure andparameters (including one-way, multiway parameter sweepsand probabilistic analysis)—is highly recommended in under-standing the variability of a model’s outputs in response todifferent assumptions. The details associated with model analy-sis differ considerably between modeling types, with stochasticmodels typically requiring different types of Monte Carlosimulation-based analyses. Principles of experimental design[56,57] and dimensional analysis [58] can be of particular valueto allow for the most judicious scenario selection, selecting thevariables to be varied so as to maximize learning.

Reporting and Documentation before, during, and after theModel Building

Reporting of model results should place a premium on reprodu-cibility. Given the important role that reproducibility of scientific

results has traditionally played in scientific research, it is desir-able to publish a sufficient degree of detail on a model’sformulation, including the fundamental specification (“sourcecode”) of the model, the framework in which it was built, modelparameters, any external sources of data used, and initial con-ditions [59,60]. Beyond this basic criterion, it can be highlyvaluable to specify the building blocks of the model whosesemantics is precisely understood, and which characterize it interms of what is represented (object), rather than all the details asto how that is captured. Most importantly, for the growingnumber of models whose implementation relies more heavilyon algorithms and computer code, this entails specificationbeyond the associated code, preferably in terms of mathematicalformalisms (e.g., finite state machines, hybrid probabilisticautomata, and flowcharts) [61].

Maintaining model documentation is essential not only forcommunicating and sharing models but also for avoiding modeldefects, enhancing model transparency, reducing the work asso-ciated with model changes, supporting new members of theteam, and facilitating model evolution. In addition to the cleardelineation of both data sources and parameter assumptions, it isbest to clearly document the formulations used to derive suchmodel parameter values. Although there are many commonthemes in reporting guidelines for dynamic modeling in general,we refer the reader to methodology-specific reporting best prac-tices [48,62–65].

During model construction, that is, model building, a clearrecord of model changes (as maintained manually or by versioncontrol systems) [44,66,67] can be instrumental in resolvingmodel defects. During model construction the output componentsof particular model scenarios or scenario collections should bedocumented, including discussion of aggregation, summariza-tion, statistical methods applied, and their parameters. Wherepossible, both input factors (e.g., preprocessing code, databases,text files, and spreadsheets) and output factors (e.g., code, syntaxfiles, scripts, spreadsheets, and database queries) should becross-linked to outputs, copied or otherwise made available inan immutable fashion, and placed under version control for laterreference. It is also common practice to routinely report metadataas part of or cross-linked to scenario exploration. Most critically,this includes model version and parameter assumptions, withoutknowledge of which thorough interpretation of such results isoften impossible. Additional information could include modelrun time data, hardware and software platform used to run thescenarios, model software settings (e.g., specifying prioritizationof handling of simultaneous equal-prioritized events, time steps,and numerical integration routines used), and random numberseeds used.

Model Maintenance and Upkeep

Many models seek to serve as persistent assets, contributingto ongoing deliberations concerning policy trade-offs andallocation of health care resources, and serve as “learningtools” by comparing model results against empirical data andpreconceived mental models. For ongoing maintenance, it isimportant that a model be periodically updated to reflect thelatest evidence, so as to take into account changes in thesystem being modeled, in resources availability, and in inter-ventions of interest. Such updates often require revisitingsteps of the modeling process. Most commonly, some of themodel parameters will need to be updated with new estimates,and very often new calibration and validation steps arerequired. In this regard, DES and ABM may be more flexiblebecause they allow for local modifications, usually operation-ally simple, whereas SD may require larger structural mod-ifications, that is, global, operationally more complicated. For

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0156

situations in which data arrive frequently, such as with dailyreports on bed occupancy and patient length of stay, orincident case counts, more automatic methods are advisedsuch as modern sequential Monte Carlo techniques such asParticle Filtering [33]. It is also often necessary to reexamineother assumptions within the model, such as those capturedin the model structure and in terms of decisions concerningmodel scope.

Recommendations for Applying Dynamic SimulationModeling in Practice to Add Value to InformedDecision Making

Planning health care delivery services is complex due to theinvolvement and interactions of people, facilities, processes, andtechnology. Making evidence-based decisions while customizingcare to the needs of individual patients and family is critical todelivering patient-centered care efficiently and effectively. Inmany instances, the interactions are not only among patientsand providers but also with other system levels such as patientsand payers, and payers and governments.

Compared with Markov models and decision trees usedcommonly in health technology assessment, dynamic simulationmodeling methods allow for estimation of the consequences ofunforeseen interactions (emergence) and can become prescrip-tive in nature [3]. In particular, these models are useful forprescribing actions/interventions based on scenarios testedthrough “what-if” experiments. As such, dynamic simulationmodeling is often a better and more effective method forevaluating interventions in the context of complex systems.

In light of the advantages and limitations of these methods,we have six key recommendations for applying dynamic simu-lation modeling in practice with the goal of adding value toinformed decision making and improving patient-centeredcare.

1. First, plan to invest sufficient time upfront with stake-holders to define the problem clearly. The problem you areattempting to address needs to be well defined to guidethe purpose of the model and the object (scope). Consider-ation of the perspective is essential, that is, system-oriented,process-oriented, or individual-oriented. Without a clearunderstanding of the system issues to be modeled and thequestions to be answered, the model design may fail to fullyincorporate essential system elements and interactions thatdescribe the problem such as feedback, queuing, and individ-ual behaviors.

2. Second, dynamic simulation models should be consideredroutinely for health service delivery planning and process andperformance improvement. The feasibility and relevance ofdynamic simulation modeling methods to inform health caredelivery system planning and decision making for improvingsystem efficiency have been demonstrated [37]. To increasemodel relevance and usefulness, appropriate model verifica-tion, validation, and calibration are crucial. This includes,among others, checking model design assumptions andparameter assumptions, stakeholder feedback, peer review,and calibration of estimates against empirical data.

3. Third, the SIMULATE checklist [3] is a tool for researchers anddecision makers to facilitate assessment of the relevance andappropriateness of dynamic simulation modeling methodscompared with more traditional models used in health tech-nology assessment to address the problem in question.Although we suggest that dynamic simulation models shouldbe considered routinely for health service delivery planningand process and performance improvement, not all problems

require dynamic simulation modeling methods. In selecting aspecific modeling method, consider which method canaddress the problem most effectively and efficiently.

4. Fourth, decision makers and other stakeholders should beconsidered an integral part of the modeling team, and as such,their participation throughout the modeling process, includ-ing the problem definition, the model design, and beyond toinclude model-building and validation processes, is essential.Modeling team members with the technical expertise must betransparent with model details and documentation, and bewilling to communicate clearly to other modeling teammembers. This allows stakeholders to gain the appropriateknowledge of underlying model assumptions, data require-ments, limitations, and applications of the model to make themodel relevant and useful for them. Furthermore, decisionmakers must be prepared to assess the plausibility of each ofthe model assumptions. Sensitivity analysis and scenariotesting are vital for decision makers so that the full plausiblerange of interactions and emergent behaviors resulting fromdecisions can be reviewed.

5. Fifth, detailed model documentation is necessary due to theiterative and incremental nature of the model-building proc-ess and building strategy. Documentation should include theproblem definition, research questions, assumptions, datasources, model version, structure and clear record of modifi-cations, parameter values, calibration and validation proce-dures, and scenario outputs. This enhances transparencywithin the modeling team, improves communication,increases confidence in the model, facilitates model main-tenance and upkeep, and fosters model uptake by decisionmakers.

6. Sixth, consider the data requirements for the model and thefeasibility of maintaining the model over time. Like anymodeling effort, dynamic simulation models require a con-siderable investment of time and resources to design andbuild, and are often data intensive. Sufficient qualitative andquantitative research is critical to establish data requirementsand availability. Therefore, it is preferred to use data that areroutinely generated and captured by health care deliverysystems (e.g., claims and electronic medical records) andreadily accessible to reduce the data burden. Although datafrom across the clinical continuum of care can be useful inassessing the processes and outcomes of health care delivery,it can be challenging to obtain routinely and may requiresubstantial investments to make it practical to use. Forexample, 70% to 80% of the clinical content captured byelectronic medical records typically remains in unstructuredphysician notes rather than structured data fields despite thestructured fields often being available in the database. Insome instances, primary data collection will be necessary,especially regarding human behaviors, and this reduces theability to update the model routinely for sustainable applica-tion. The modeling team must consider these factors whendesigning the model and balance the advantages of includingeach variable in the model (scope).

In summary, application of dynamic simulation modelingmethods can not only enrich decision and policymaking proc-esses through scenario analyses depicting the what-if questionsand answers, but will also improve outcomes of decisions aroundsystem redesign, such as facility planning, reducing wait time forservices, and alternative models of care delivery. System redesignis an essential step to achieving sustainable implementation ofevidence-based practice interventions across the care contin-uum. Dynamic simulation modeling can be useful to estimatethe clinical, economic, and humanistic outcomes associated withprogression of disease, consequences of interventions, and the

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 157

complexity of interactions within the system. Hence, thesemethods can inform the adoption of evidence-based patient carepractices by considering both intended and unintended conse-quences of health system interventions through model testingbefore implementation. Despite these potential benefits of apply-ing dynamic simulation models to improve patient-centered care,it must be remembered that models do not solve problems.Models can only inform potential solutions to problems, andthe ultimate decision responsibility rests with decision makers.

Areas for Continued Methodological DevelopmentApplying Dynamic Simulation Models in Health CareDelivery Research

The dynamic simulation modeling methods described in thisarticle are not new; for example, SD models have been with us formore than 50 years [68]. The application of dynamic simulationmodeling to better understand the impact of different interven-tions on health care systems has, until recently, however, beenlargely confined to traditional operations research problems suchas minimizing transportation costs and patient wait times. Suchapplications are perfectly appropriate, but the power of dynamicsimulation methods is especially evident when tackling broaderproblems such as the simulation of health care reforms [69,70].The rapidly growing health care literature on dynamic simulationmethods testifies to their increasing application and accessibility[30] for understanding the effects of policy and medical inter-ventions in complex health care systems [37].

There will undoubtedly continue to be refinements in meth-odology as dynamic simulation models are applied to health careproblems, such as the increasingly popular, powerful, and flexiblehybrid models that combine multiple dynamic simulation mod-eling methods [29]. This report guides the selection of a dynamicsimulation modeling method that is most suitable for a givenproblem, recognizing that more than one method could be used,but may be more or less effective and efficient. Nonetheless, inhealth care delivery, the problem to be addressed often involvesdifferent levels (e.g., strategic, operational, and tactical) simulta-neously; that is, subproblems require distinct perspectivesbecause of their nature and research questions, thereby requiringa multimethod or hybrid approach to effectively combine modelswith different purposes and objects [71–73].

The most fundamental advances will likely be driven byfactors such as the growing availability of electronically sourcedand cross-linked data and attendant advances in data science.For more than 20 years, health care data—particularly medicaland drug claims—have been used to analyze the safety and real-world effectiveness of alternative treatments. In an effort toimprove access to such data to support health care research,large-scale data warehouses are being assembled around theglobe [74,75]. In many cases, these databases contain not onlyinformation regarding traditional patient interactions with thehealth care system reflected in claims or medical records but alsoprimary data collection on patient perspectives. The most recentdevelopments include the advancement of feeds from personalmobile devices such as Apple’s iWatch directly into electronicmedical records systems [76]. The “3 Vs” of such data—its greatervolume, variety, and velocity [1]—bring more extensive empiricalevidence as variable inputs to dynamic simulation models, suchas those involving actor behavior, exposures, and preferences.

Of equal significance is the growing capacity to use machinelearning together with dynamic simulation. Because of thecomplexity and scale of data systems created by such diversedata feeds, as well as the speed with which they are beingupdated, machine learning methods [77] are now starting to beused to build predictive models of health care interventions [78].

Such methods support much more robust and powerful forms ofparameter estimation in dynamic simulation models [79,80], andtechniques for keeping such models routinely and automaticallyupdated with incoming data feeds, thereby improving the accu-racy of the data and the model [33,81]. Although some compo-nents of machine learning methodology, such as cross-validation, already enjoy longstanding application in dynamicsimulation modeling, the combination of machine learning anddynamic simulation has been far more recent. Nevertheless, it isalready demonstrating strong benefits in other health simulationsubdomains, and seems likely to soon see broad and powerfulapplication leveraging big data in the analysis of health caresystems.

Finally, like dynamic simulation methods, there is a growingrecognition of the applicability of optimization methods fromoperations research to health care problems. Optimization meth-ods from operations research were originally developed in WorldWar II, so they are similar to dynamic simulation models in theirmaturity. Optimization methods such as linear, nonlinear, inte-ger, and dynamic programming are already used in health carefor traditional applications (minimizing wait times or transpor-tation costs). In the DES example of emergency room expansionconsidered earlier, it would be a logical next step to considerestimation of the optimal solution once the problem, inputs, andsystem constraints were sufficiently well understood through theDES modeling process. Optimization methods have only recentlybegun to be applied to outcomes research problems such asfinding the optimal treatment pathway for a particular type ofpatient within constraints imposed by the system (insurancecoverage, access to certain types of facilities, etc.) [82]. In eithercase, it is apparent that dynamic simulation models would be animportant precursor to optimization, greatly improving ourunderstanding of the system necessary to formulate the optimi-zation problem. This creates further opportunities to transferlearnings from fields outside of health care to the optimization ofhealth care systems.

Acknowledgments

We thank the following reviewers for valuable written feedbackon earlier drafts of this report: Brian Denton, Beth Devine, ChrisJones, Jonathan Karnon, Nishkarsh Likhar, Andriy Moshyk, DavidC. Norris, Martin O’Leary, Luke Rudmik, and Amir Viyanchi. Wealso very much appreciate the comments, expertise, and insightof those who provided oral comments during presentations ofour work to date. ISPOR member comments contribute to thehigh-quality consensus nature that characterizes ISPOR goodpractices for outcomes research reports. Finally, the steady andcapable support of our ISPOR staff liaison, Elizabeth Molsen, isgenuinely appreciated.

R E F E R E N C E S

[1] Laney D. 3-D Data Management: Controlling Data Volume. Velocity andVariety, META Group Original Research Note, 2001.

[2] Berwick DM. What ‘patient-centered’ should mean: confessions of anextremist. Health Aff (Millwood) 2009;28:w555–65.

[3] Marshall DA, Burgos-Liz L, IJzerman MJ, et al. Applying dynamicsimulation modeling methods in health care delivery research—theSIMULATE Checklist: report of the ISPOR Simulation ModelingEmerging Good Practices Task Force. Value Health 2015;18:5–16.

[4] Forrester J. Industrial Dynamics. Cambridge, MA: The MIT Press, 1961.[5] Forrester J. Urban Dynamics. Walthum, MA: Pegasus Communications,

1969.[6] Forrester J. World Dynamics. Walthum, MA: Pegasus Communications,

1971.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0158

[7] Sterman JD. Business Dynamics: Systems Thinking and Modeling for aComplex World. New York: McGraw-Hill, 2000.

[8] Richardson G, Pugh A, III.. Introduction to System Dynamics Modeling.Cambridge, MA: MIT Press, 1981.

[9] Roberts EB. System dynamics—an introduction. In: Roberts EB, ed.Managerial Applications of System Dynamics. Cambridge, MA. MITPress; 1978.

[10] Forrester J. Principles of Systems. Cambridge, MA: MIT Press, 1968.[11] Sterman JD. Learning from evidence in a complex world. Am J Public

Health 2006;96:505–14.[12] Richmond B. Systems thinking: critical thinking skills for the 1990s and

beyond. System Dyn Rev 1993;9:113–33.[13] Milstein B, Homer J, Briss P, et al. Why behavioral and environmental

interventions are needed to improve health at lower cost. Health Aff2011;30:823–32.

[14] Senge PM. Mental models. Strategy Leadersh 1992;20:4–44.[15] Siebers PO, Macal CM, Garnett J, et al. Discrete-event simulation is

dead, long live agent-based simulation! J Simul 2010;4:204–10.[16] Karnon J, Stahl J, Brennan A, et al. Modeling using discrete event

simulation: a report of the ISPOR-SMDM Modeling Good ResearchPractices Task Force-4. Med Decis Making 2012;32:701–11.

[17] Troy PM, Rosenberg L. Using simulation to determine the need for ICUbeds for surgery patients. Surgery 2009;146:608–17.

[18] Gunal MM. A guide for building hospital simulation models. HealthSystems 2012;1:17–25.

[19] Borshchev A, Filippov A. From system dynamics and discrete event topractical agent based modeling: reasons, techniques, tools. Presentedat: 22nd International Conference of the System Dynamics Society.Oxford, England, July 25–29, 2004.

[20] IPLER. Information Products Lab for Emergency Response (IPLER). 2008.Available from: http://ipler.cis.rit.edu/node/1?_sm_au_=iVV0HQk2ZV57r4kR. [Accessed January 10, 2014].

[21] Kruzikas DT, Higashi MK, Edgar M, et al. Using agent-based modeling toinform regional health care system investment and planning.Presented at: Computational Science and Computational Intelligence(CSCI), 2014 International Conference. Las Vegas, NV, March 10-13,2014.

[22] Macal CM, North MJ, Collier N, et al. Modeling the transmission ofcommunity-associated methicillin-resistant Staphylococcus aureus:a dynamic agent-based simulation. J Transl Med 2014;12:124.

[23] Sobolev B, Harel D, Vasilakis C, Levy A. Using the Statecharts paradigmfor simulation of patient flow in surgical care. Health Care Manag Sci2008;11:79–86.

[24] Roberts M, Russell LB, Paltiel AD, et al. ISPOR-SMDM Modeling GoodResearch Practices Task Force. Conceptualizing a model: a report of theISPOR-SMDM Modeling Good Research Practices Task Force–2. MedDecis Making 2012;32:678–89.

[25] Brailsford S, Hilton N. A comparison of discrete event simulation andsystem dynamics for modelling health care systems. In: Riley J, ed.Planning for the Future: Health Service Quality and Emergency Accessibility.Operational Research Applied to Health Services (ORAHS). Glasgo:Caledonian University, 2001.

[26] Behdani B. Evaluation of paradigms for modeling supply chains ascomplex socio-technical systems. Presented at: 2012 Winter SimulationConference (WSC). Simulation Conference IEEE. Berlin, Germany, 9–12,2012.

[27] Schieritz N, Milling PM. Modeling the forest or modeling the trees.Presented at: Proceedings of the 21st International Conference of theSystem Dynamics Society. New York, NY, July 20–24, 2003.

[28] Jun GT, Morris Z, Eldabi T, et al. Development of modelling methodselection tool for health services management: from problemstructuring methods to modelling and simulation methods. BMC.Health Serv Res 2011;11:108.

[29] Gao A, Osgood ND, An W et al. A tripartite hybrid model architecturefor investigating health and cost impacts and intervention tradeoffs fordiabetic end-stage renal disease. Presented at: 2014 Winter SimulationConference (WSC). Savannah, GA, December 7–10, 2014.

[30] Osgood N. Using traditional and agent based toolsets for systemdynamics: present tradeoffs and future evolution. Presented at: 25thInternational Conference of the System Dynamics Society and 50thAnniversary Celebration. Boston, MA, July 29–August 2, 2007.

[31] Osgood N. Representing heterogeneity in complex feedback systemmodeling: computational resource and error scaling. Presented at:22nd International Conference of the System Dynamics Society.Oxford, England, July 25–29, 2004.

[32] Osgood N. Representing progression and interactions of comorbiditiesin aggregate and individual-based systems models. Presented at:Proceedings of the 27th International Conference of the SystemDynamics Society. Albuquerque, New Mexico, July 26–30, 2009.

[33] Osgood N, Liu J. Towards closed loop modeling: evaluating theprospects for creating recurrently regrounded aggregate simulationmodels. Presented at: 2014 Winter Simulation Conference (WSC).Savannah, GA, December 7–10, 2014.

[34] Lorenz T, Jost A. Towards an orientation framework in multi-paradigmmodeling. Presented at: Proceedings of the 24th InternationalConference of the System Dynamics Society. Nijmegen, TheNetherlands, July 23–27, 2006.

[35] Dyck R, Osgood N, Gao A, Stang MR. The epidemiology of diabetesmellitus among First Nations and non-First Nations children inSaskatchewan. Can J Diabetes 2012;36:19–24.

[36] Vendrov I, Dutchyn C, Osgood Frabjous ND. A declarative domain-specific language for agent-based modeling. Social Computing,Behavioral-Cultural Modeling and Prediction. Washington, DC: Springer,2014.

[37] Brailsford SC, Harper PR, Patel B, Pitt M. An analysis of the academicliterature on simulation and modelling in health care. J Sim2009;3:130–40.

[38] Caro JJ, Briggs AH, Siebert U, Kuntz KM. ISPOR-SMDM Modeling GoodResearch Practices Task Force. Modeling good research practices–overview: a report of the ISPOR-SMDM Modeling Good ResearchPractices Task Force-1. Med Decis Making 2012;32:667–77.

[39] Husereau D, Drummond M, Petrou S, et al. ISPOR Health EconomicEvaluation Publication Guidelines-CHEERS Good Reporting PracticesTask Force. Consolidated Health Economic Evaluation ReportingStandards (CHEERS)—explanation and elaboration: a report of theISPOR Health Economic Evaluation Publication Guidelines GoodReporting Practices Task Force. Value Health 2013;16:231–50.

[40] Vennix JAM. Group model-building: tackling messy problems. SystemDyn Rev 1999;40:379–401.

[41] Pasupathy KS. Transforming healthcare: leveraging thecomplementarities of health informatics and systems engineering. Int JHealthcare Deliv Ref Initiatives 2010;2:35–55.

[42] Brown GD, Patrick TB, Pasupathy KS. Health Informatics: A SystemsPerspective. Chicago, IL: Health Administration Press, 2013.

[43] Axelrod RM. The Complexity of Cooperation: Agent-Based Models ofCompetition and Collaboration. Princeton, NJ: Princeton University Press,1997.

[44] Osgood N, Tian Y. 15 things system dynamics can learn from softwaredevelopment. Available from: http://systemsmodelbook.org/viewmodel.aspx?id=3a2b3049-db06-49d3-95a4-31cb4832dec6.[Accessed February 25, 2015].

[45] Duvall PM, Matyas S, Glover A. Continuous Integration: Improving SoftwareQuality and Reducing Risk. Crawfordsville, IN: Pearson Education, 2007.

[46] Forrester JW, Senge PM. Tests for Building Confidence in SystemDynamics Models. Cambridge: System Dynamics Group, Sloan Schoolof Management, Massachusetts Institute of Technology, 1978.

[47] Law AM. How to build valid and credible simulation models. Presentedat: 2009 Winter Simulation Conference (WSC). Simulation ConferenceIEEE. Austin, TX, December 13–16, 2009.

[48] Eddy DM, Hollingworth W, Caro JJ, et al. ISPOR-SMDM Modeling GoodResearch Practices Task Force. Model transparency and validation: areport of the ISPOR-SMDM Modeling Good Research Practices TaskForce–7. Med Decis Making 2012;32:733–43.

[49] Barlas Y. Formal aspects of model validity and validation in systemdynamics. Syst Dyn Rev 1996;12:183–210.

[50] McConnell S, Johannis D. Code Complete. Vol. 2. Redmond: MicrosoftPress, 2004.

[51] Dasso A, Funes A. Verification, Validation and Testing in SoftwareEngineering. Hershey, PA: IGI Global, 2007.

[52] Beck K. Extreme Programming Explained: Embrace Change. Boston,MA: Addison-Wesley Professional, 2000.

[53] Craig RD, Jaskiel SP. Systematic Software Testing. Norwood, ME: ArtechHouse, 2002.

[54] Kiczales G, Lamping J, Mendhekar, et al. Aspect-Oriented Programming.Springer, 1997.

[55] Fowler M. Refactoring: Improving the Design of Existing Code. Presentedat: 11th European Conference. Jyväskylä, Finland, June 9–13, 1997.

[56] Kleijnen JP. Banks J ed. Experimental Design for Sensitivity Analysis,Optimization, and Validation of Simulation Models, in Handbook ofSimulation: Principles, Methodology, Advances, Applications, andPractice. Hoboken, NJ: John Wiley & Sons, Inc., 1998.

[57] Kleijnen JP. Design and Analysis of Simulation Experiments. Vol. 111.New York, NY: Springer, 2007.

[58] Szirtes T. Applied Dimensional Analysis and Modeling. Burlington, MA:Butterworth-Heinemann, 2007.

[59] Xue Y, Osgood N, Gutwin C. SILVERVIZ: extending SILVER forcoordination in distributed collaborative modeling. Presented at: The29th International Conference of the System Dynamics Society.Washington, DC, July 24–28, 2011.

[60] Osgood N. Silver: software in support of the system dynamics modelingprocess. Presented at: The 27th International Conference of the SystemDynamics Society. Albuquerque, NM, July 26–30, 2009.

[61] Fowler M. UML Distilled: A Brief Guide to the Standard Object ModelingLanguage. Boston, MA: Addison-Wesley Professional, 2004.

[62] Rahmandad H, Sterman JD. Reporting guidelines for simulation‐basedresearch in social sciences. System Dyn Rev 2012; 28:396–411.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0 159

[63] Müller B, Bohn F, Dreßler G, et al. Describing human decisions in agent-based models–ODDþ D, an extension of the ODD protocol. EnvironModel Software 2013;48:37–48.

[64] Grimm V, Berger U, Bastiansen F, et al. A standard protocol fordescribing individual-based and agent-based models. Ecolog Model2006;198:115–26.

[65] Grimm V, Berger U, DeAngelis DL, et al. The ODD protocol: a review andfirst update. Ecolog Model 2010;221:2760–8.

[66] Pilato CM, Collins-Sussman B, Fitzpatrick BW. Version Control withSubversion. Sebastopol, CA: O’Reilly Media, Inc., 2008.

[67] Loeliger J, McCullough M. Version Control with Git: Powerful Tools andTechniques for Collaborative Software Development. Sebastopol, CA:O’Reilly Media, Inc., 2012.

[68] Forrester J. System dynamics—a personal view of the first fifty years.Syst Dynam Rev 2007;23:345–58.

[69] Glouberman S, Zimmerman B. Complicated and Complex Systems: WhatWould Successful Reform of Medicare Look Like. Discussion paper number8. Commission on the Future of Health Care in Canada, 2002.

[70] Milstein B, Homer J, Hirsch G. Analyzing national health reformstrategies with a dynamic simulation model. Am J Public Health2010;100:811–9.

[71] Viana J, Rossiter S, Channon A, et al. A multi-paradigm, whole systemview of health and social care for age-related macular degeneration.Presented at: 2012 Winter Simulation Conference (WSC). Berlin,Germany, December 9–12, 2012.

[72] Chahal K, Eldabi T. Applicability of hybrid simulation to differentmodes of governance in UK healthcare. Presented at: WinterSimulation Conference (WSC). Miami, FL, December 7–10, 2008.

[73] Brailsford SC, Desai SM, Viana J. Towards the holy grail: combiningsystem dynamics and discrete-event simulation in healthcare.

Presented at: 2010 Winter Simulation Conference (WSC). Baltimore,MD, December 5–8, 2010.

[74] Curtis LH, Brown J, Platt R. Four health data networks illustrate thepotential for a shared national multipurpose big-data network. HealthAff (Millwood) 2014;33:1178–86.

[75] Wallace PJ, Shah ND, Dennen T, et al. Optum Labs: building a novelnode in the learning health care system. Health Aff (Millwood)2014;33:1187–94.

[76] Diamond D. iPhone 6: Apple and Mayo Clinic partnership could besmart medicine. 2014. Available from: http://www.forbes.com/sites/dandiamond/2014/09/09/iphone-6-apple-and-mayo-clinic-partnership-could-be-smart-medicine-2/. [Accessed January 20, 2015].

[77] Hastie T, et al. The Elements of Statistical Learning. Vol. 2. Springer,2009.

[78] Crown W. Potential application of machine learning in healthoutcomes research and some statistical cautions. Value Health2015; 18:137–40.

[79] Dorigatti I, Cauchemez S, Pugliese A, Ferguson NM. A new approach tocharacterising infectious disease transmission dynamics from sentinelsurveillance: application to the Italian 2009–2010 A/H1N1 influenzapandemic. Epidemics 2012;4:9–21.

[80] Osgood N, Liu J. An introduction to using Markov chain Monte Carloapproaches with dynamic models. In: Analytical Handbook for DynamicModelers. Cambridge, MA: MIT Press, 2015.

[81] Ong JB, Chen MI, Cook AR, et al. Real-time epidemic monitoring andforecasting of H1N1-2009 using influenza-like illness from generalpractice and family doctor clinics in Singapore. PloS One, 2010;5:e10036.

[82] Zhang J, Denton BT, Balasubramanian H, et al. Optimization ofprostate biopsy referral decisions. Man Serv Operations Manag2012;14:529–47.

V A L U E I N H E A L T H 1 8 ( 2 0 1 5 ) 1 4 7 – 1 6 0160