Review ArticleInformatics Metrics and Measures for a Smart Public HealthSystems Approach: Information Science Perspective

Timothy Jay Carney1 and Christopher Michael Shea2

1The Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, 1101-C McGavran-Greenberg Bldg.,CB 7411, Chapel Hill, NC 27599-7411, USA2The Gillings School of Global Public Health, The University of North Carolina at Chapel Hill, 1101-F McGavran-Greenberg Bldg.,CB # 7411, Chapel Hill, NC 27599-7411, USA

Correspondence should be addressed to Timothy Jay Carney; tcarney@unc.edu

Received 14 September 2016; Revised 30 November 2016; Accepted 4 December 2016; Published 10 January 2017

Academic Editor: Alejandro Rodrıguez-Gonzalez

Copyright © 2017 T. J. Carney and C. M. Shea. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

Public health informatics is an evolving domain in which practices constantly change to meet the demands of a highly complexpublic health andhealthcare delivery system.Given the emergence of various concepts, such as learning health systems, smart healthsystems, and adaptive complex health systems, health informatics professionals would benefit from a common set of measures andcapabilities to inform our modeling, measuring, and managing of health system “smartness.” Here, we introduce the concepts oforganizational complexity, problem/issue complexity, and situational awareness as three codependent drivers of smart public healthsystems characteristics. We also propose seven smart public health systems measures and capabilities that are important in a publichealth informatics professional’s toolkit.

1. Introduction

Public health informatics is an evolving domain in whichpractices constantly change to meet the demands of a highlycomplex public health and healthcare delivery system. Thetypical definition for a variety of domains of informatics (e.g.,public health, population health, nursing, clinical, medical,health, consumer, and biomedical) centers on the “applicationof information science and information technology to [aspecific domain of] practice, research, and training” [1, 2].This definition of informatics relies on a technical view ofthe health system. A technical view of informatics largelyidentifiesmore tangible products such as databases, decision-support tools, information systems, web portals, and mobiledevices as the primary means of addressing complex healthissues, improving care, and reducing health disparities.

Public health informatics systems expressed as a func-tion of intelligence can be understood in terms of twocodependent pathways of (1) generating health information

technology (HIT) policies that ensure our ability to Gov-ern Intelligence as a byproduct and (2) allowing innova-tions in HIT to shape and inform public health systemspolicy and practice to ensure that we Govern Intelligently.In the former case, public health informatics professionalsendeavor to generate HIT policy to guide national, state, andlocal information architecture, information infrastructure,and information integration efforts that ultimately guidehow public health meets the needs of stakeholder/agentssuch as patients/families/health consumers, communities,providers/healthcare organizations, researchers, policymak-ers, and disease-centric communities of practice throughthe meaningful supply of intelligence. Such intelligencecan inform stakeholder understanding about the burdenof disease, spread of an outbreak, health alerts and foodrecalls, disease clusters, community needs assessments, andhealth risk assessments. In the latter case, public healthinformatics professionals seek to find innovative ways toleverage HIT to improve the way we govern by seeking ways

Hindawi Publishing CorporationComputational and Mathematical Methods in MedicineVolume 2017, Article ID 1452415, 12 pageshttp://dx.doi.org/10.1155/2017/1452415

2 Computational and Mathematical Methods in Medicine

Other sectorsOther sectors

Other sectors

Information architecture

Personal health, populationlevel assurance and

environmental health servicesprovided by local health

departments (LHD)

Information infrastructure

(i) Moving from paper-based to electronic health records(EHRs)

Healthcare and public healthresponsibilities of state health

departments (SHD)Information integration

(i) Information exchange(ii) IT optimization

Domains, stakeholders,functions, services &

interventions, data sources &public

Patients, healthconsumers, &

familiesPolicyResearchersProviders and

healthcareorganizations

Communities

How public health informsstakeholders (governing

intelligence)

How public health is informed bystakeholders (intelligently

governing)

Diseasesectors

(i) IT design (ii) Collection of database

systems

Figure 1: Public health informatics systems intelligence perspectives.

to streamline processes that positively impact cost, quality,safety, and overall health outcomes. Figure 1 highlights theserelationships in the context of public health practice domains.

Although useful for fostering greater levels of adop-tion and use of technical measures, this technical view ofpublic health informatics (1) does not highlight the changingknowledge needs of these system agents over time, (2)fails to capture the full array of interaction among agentsin a dynamic environment, and (3) cannot maintain pacein adapting to an ever-increasing complex environment.In other words, the purely technical approach does noteffectively highlight the full spectrum of knowledge, commu-nication, and learning that is needed to keep all types of healthsystem stakeholders—including individuals, organizations,or collections of individual and organizational networks (e.g.,coalitions, collaborations, consortiums, and taskforces)—informed and able to respond to environmental changes atall stages of the healthcare continuum.

In this era of informatics where the emphasis is on lesstangible cognitive capacities (e.g., learning health systems,intelligent and smart systems, and complex adaptive systems),a new public health informatics analytics approach may berequired that is less information technology-driven andmoreknowledge-driven and defines new ways of demonstratingthe added value of informatics in shaping health systemsperformance [3]. Specifically, stakeholders (hereafter referredto as individual- or organizational-level agents) need concise,accurate, and objective analytic measurements of abstract

concepts, such as empowerment, which previously has beendescribed as a function of knowledge for the purposes ofachieving a quantifiable metric for computational analysis ofperformance.

Such a view of public health informatics may focus onabstract constructs like actionable intelligence as the primaryinformatics-centric outcome [3]. Such a strategy should yieldobjective operational measures and capabilities designed toensure that individual agents, organizations, and networkshave sufficient knowledge to mount an intelligent responseto solve complex public health problems. In other words,the strategy should support development and maintenanceof smart health systems, that is, a system that “incorporatesfunctions of sensing, actuation, and control in order todescribe and analyze a situation, andmake decisions based onthe available data in a predictive or adaptive manner, therebyperforming smart actions. In most cases the ‘smartness’ ofthe system can be attributed to autonomous operation basedon closed loop control, energy efficiency, and networkingcapabilities” [4].

The purpose of this paper is to propose a set of measuresfor tracking the development and sustainability of smartpublic health systems. Specifically, we introduce the conceptsof organizational complexity, problem/issue complexity, andsituational awareness as three codependent drivers of smarthealth systems. We then describe seven smart health systemsmeasures. This discussion is important for public healthinformatics professionals responsible for specifying metrics,

Computational and Mathematical Methods in Medicine 3

overseeing information systems housing data for the metrics,and evaluating the performance of smart public healthsystems.

2. Factors Shaping SmartAgents and Organizations

The underlying objective of any agent or actor withina given public health system is to maximize the use ofdata, information, and knowledge as strategic resources. Aninformatics-biased view of a public health system focuses onthe sum of data, information, knowledge systems, people,practices, policies, and cultural factors that operates tosupport some predefined intelligence strategy, organizationalmission, or other event [5]. In these terms, the publichealth system can be understood as a functional knowledgeculture. We have also used related terms such as knowledgeenvironments, information or knowledge ecosystems, andinformation or knowledge ecologies to represent this idea.Here, we use knowledge culture and knowledge environ-ment interchangeably. We argue that any defined systemboundary that contains the formal or informal governanceof critical strategic and shared knowledge resources can becalled a knowledge environment. The primary purpose ofany knowledge environment is best understood in termsof the essential need to leverage data, information, andknowledge in managing individual or collective uncertainty[6–8].

The way in which we engage in information and knowl-edge seeking, organize ourselves into collectives of varyingunit configurations (e.g., workgroups, project teams, task-forces, departments, divisions, networks of organizationalcoalitions, and consortiums), and/or apply the use of toolsor technology indicates the basic need to manage any and allforms of uncertainty [9]. We organize ourselves in responseto external and internal drivers/stressors and increasing envi-ronmental complexity as a means of reducing or removingany impediments toward fast, reliable, and pertinent data,information, and knowledge resources [10]. This impera-tive to organize for the sake of becoming smarter is bestobserved in our introduction of three primary drivers thatwe argue are interdependent in any knowledge environment.By describing these factors as interdependent, we are statingthat as one type of driver category increases or decreases bysome set of circumstances or events, corresponding changescan occur in one or both of the other areas. These areasinclude organizational complexity, problem/issue complexity,and situational awareness (see Figure 2). Essentially, each ofthese three primary driver categories shapes our overall data,information, and knowledge strategy within any knowledgeenvironment. The primary objective of an informatician indesigning and maintaining a smart public health knowledgeenvironment is then to understand the basic predictorsof change in any or all of these categories, as well as toaccount for the correspondingmediation/moderation factorsthat can shape continued data, information, and knowledgemaximization for agents within any public health knowledgeenvironment.

Organizationalcomplexity

Problem/issuecomplexity

Situationalawareness

Figure 2: Knowledge environment factors of influence.

2.1. Organizational Complexity Factors Shaping Public HealthKnowledge Environments. We use a variety of organizationalstructures to facilitate interaction, communication, andknowledge representation in our quest to manage changes inour environment. Generally, the levels of organization mayvary from amicro- to macrocontinuum that starts with orga-nizational agents/individuals, components/subunits, a singleentity/facility, and a multiunit of systems/collaborations/coalitions/networks/taskforces/consortiums [11, 12]. Typically, the level of complexityinherent in the public health challenge or crisis event deter-mines the corresponding level of organizational complexityrequired in the response [7, 13]. Challenges or crisis eventsthat are short-term or relatively minor may only requireminimal, ad hoc, or temporary organizational responses.Within the modern healthcare environment, these can repre-sent informal partnerships or formal structures appearing asshort-term project teams or workgroups. More involved andlong-term problems may require increasing levels of com-plexity within the organizational response. These long-termor complex organizational responses may be represented inthe form of permanent departments or divisions within anorganization, or theymay even extend beyond organizationalboundaries to include coalitions, collaborations, taskforces,and interagency network arrangements.

One common public health system problem-solvingstrategy used throughout the US and worldwide involvesformulating networks of individuals and organizations to co-ordinate global-, national-, state-, regional-, county-, city-, oreven community-level responses to health threats to indi-viduals or populations. Such networks (e.g., coalitions, col-laborations, consortiums, and taskforces) present oppor-tunities to define common goals, shape strategy, achieveeconomies-of-scale through the sharing of resources and faci-litate the centralized monitoring and measuring of progresstoward stated objectives. However, one challenge for thepublic health informatics professional involves ensuring thatthe data, information, and knowledge needs of networksof stakeholders—ranging from patient advocates, healthorganizations, providers, community groups, public healthdepartments, policy makers, and researchers—are all metwith efficiency and effectiveness. The issues surrounding

4 Computational and Mathematical Methods in Medicine

timely intelligence were on full display during the recentEbola virus and Zika virus outbreaks.

Currently, there are no consistent measures or metricsto evaluate the efficiency and effectiveness of the abilityof “smart” health networks—of any size or configuration—to leverage data, information, and knowledge to produceactionable intelligence from their efforts [3, 14]. In otherwords, there is no quantifiable set of standardizedmeasures orstandard operational definitions of what a smart or learninghealth network is now or what it should be in the future [15].Within any public health knowledge environment, a widevariety of network structures can be assumed. The organi-zation is viewed as a dynamic, complex, and adaptive entitywhose size, structure, and other organizational determinantsmust be constantly evaluated to promote its ability to respondto internal and external challenges, threats, and opportunitiesthat will impact individuals and/or the collective leveragingof actionable intelligence to ensure success in health systemmanagement [16–18].

2.2. Problem/Issue Complexity Factors Shaping KnowledgeEnvironments. An analogy for problem/issue identificationand response within any public health knowledge envi-ronment is the human immune response in which thehuman immune system assesses threats on a constant basisand determines if a foreign agent is a “friend” or “foe.”Once identified in a healthy immune system, the properimmune response is triggered. For a friend response, facili-tation/proliferation strategies ensue, and, for a foe response,elimination/mitigation strategies ensue. Two critical compo-nents in the overall immune response system are the abilityto retain a memory of this encounter and to demonstratesystem learning to prepare for future encounters of a relativelysimilar nature.The same sort of dynamic occurswithin a pub-lic health system network among its various organizationalcomponents, actors/agents, and events. Once a phenomenon(i.e., circumstance/event/activity/occurrence) is identified asa potential problem, either threatening or nonthreatening,system or collective memory is vetted for familiarity [18]. Ifsufficient memory of the phenomenon or something similaris found, the ideal response algorithm(s) (set of instructions)is/are identified, outlining the appropriate response mecha-nism. If no memory exists, a response must be determinedon an ad hoc basis. Clinical or public health events/activitiesthat allow favorable health outcomes (e.g., diffusion of bestpractices, strategic summits, introduction of new technology,disease screening and awareness campaigns, and new fund-ing announcements) may be considered targets for facilita-tion/proliferation, whereas unfavorable events/activities (e.g.,disease outbreaks, health or food recalls,medical errors, devi-ations from guideline concordant care, risk behaviors linkedto disease spread, budget shortfalls, and staff layoffs) may betargets for elimination/mitigation. In either case, sufficientmemory must be generated of the response algorithms (pro-cess/workflows, policies/procedures) that contributed to theevent(s), pathways toward emergence, and/or remediationstrategy to eliminate the threat. Learning in this contextpresents the ability to circumnavigate potentially harmful

events that have the potential for reoccurrence or the abilityto repeat/reinforce positive events that are beneficial [19].Hence, the ability to extract actionable intelligence fromstored memory is essential to overall public health systemperformance and an effective knowledge environment [3].Two factors that shape this dynamic of event, memoryevaluation, and learning within a knowledge environmentare familiarity and preparedness, borrowed from the field ofemergency preparedness [20].

Within any knowledge environment, issues/problemcomplexity and relative familiarity (stored memory) largelyshape the level of “shock” or environmental stress to thepublic health system, which creates what Burton termedan organizational design misfit [16]. In the presence of anorganizational design misfit, the goal is to seek to restoresomemeasure of equilibrium [17].The level of shock broughtby the introduction of a problem/issue into any public healthknowledge environment and its corresponding impact on thepublic health system can be thought of in terms of two factors:(1) the degree to which the event was expected to occur and(2) the degree to which the environment was prepared for itsoccurrence. Figure 3 highlights the relationships of these twofactors, where the green represents a highly desirable state ofsystem and organizational readiness (operationally definedhere as the agents’—within the public health knowledgeenvironment—ability to process the event and determinean appropriate response), yellow represents less desirablestates of organizational readiness, and red represents the leastdesirable state of organizational readiness and the highestlevel of vulnerability from both internal and external threats.

Although most public health systems are prepared todeal with any event; some noticeable changes can occur inthe face of uncovered vulnerabilities introduced by shockevents. Such adjustments on the organizational side maypresent as unexpected leadership shifts, sudden changes inorganizational command structures, abrupt shifts in policyand procedures, new stratums of research funding to inves-tigate and solve problems, or the addition or elimination ofstaff and key personnel [16]. On the public health knowledgeenvironment side, such adjustments can take the form ofwide-scale data integration or health information exchangeefforts, the formation of new database solutions, the demandfor new technology to monitor and track the problem,surveillance protocols, information systems, knowledge por-tals, decision-support systems, and changes in informationresource-management protocols [12].The level of complexityin both the problem/issue and the capability of the publichealth knowledge environment to process the event andmount an appropriate response heavily shapes the level oforganization (or in some cases reorganization) required tomediate the threat or exploit the opportunity. Additionally,these changes—and more importantly the rate of changes inthe organization in particular and the public health knowl-edge environment in general—may serve as proxy indicatorsfor overall public health knowledge environment maturityin managing uncertainty. In other words, a health system orpublic health agency that has undergone frequent leadershipchanges, high staff turnover, frequent redrafting of strategicplans, and reorganizations in a relatively short span of time

Computational and Mathematical Methods in Medicine 5

UnexpectedPrepared

ExpectedPrepared

UnexpectedUnprepared

ExpectedUnprepared

Level of expectedness

(More prepared)

(More expected)(Less expected)

(Less prepared)Level of

preparedness

Figure 3: Problem/issue complexity factors.

serves as a strong indicator of the lack of overall public healthknowledge environment maturity [12]. Such a public healthknowledge environment characteristically remains in a loopmoving from crisis-to-solution to a new or remerging crisis-to-solution. In contrast, a mature public health knowledgeenvironmentwill seek to identify and understand the patternsof organizational complexity and problem/issue complexityemergence and response. Properly stored, organized, andreadily accessible systemmemory can greatly aid in achievinga more mature public health knowledge environment.

2.3. Situational Awareness Factors Shaping Information Envi-ronments. Previously, we stated that organizational complex-ity is shaped by external or internal factors in a given publichealth knowledge environment, requiring different levels offormal or informal organizational structures to manage theirenvironmental challenges.We alsomentioned that the level ofcomplexity inherent in problems/issues and the correspond-ing systemmemory and preparedness will shape system-levelresponses to control andmitigate any perceived threats. Here,we formally define the term situational awareness (SA) as “theability to make sense of an ambiguous situation. It is the pro-cess of creating [situational awareness] and understandingto support decision-making under uncertainty—an effort tounderstand connections among people, places, and events inorder to anticipate their trajectories and act effectively” [21].Endsley elaborated the definition for situational awareness(SA), stating that SA is comprised of three subdomains thatshape individual understanding of some phenomena. Theseinclude (1) Situation Perception (defining the current publichealth condition), (2) Situation Comprehension (definingthe relative public health threat or opportunity), and (3)Situation Projection (forecasting the public health outcomesof hypothesized trajectories) [22].

Within situational awareness, the two elements of orga-nizational complexity and problem/issue complexity arecombined and serve as contributing factors that determine

the degree to which organizational structure and functionare properly suited to facilitate unencumbered informationprocessing. Previous organizational theories have describedthe organization, functioning within a given environment,as an information-processing entity (IPE) [16]. From thisperspective, organizations are seen as sophisticated infor-mation processing and decision-making machines that actas if they have preprogrammed subroutines in managingthe loop stages of information flow and organizationalprocesses (model → input → transformation → output→ feedback) [4]. Within the IPE view of an organization,we must understand information processing as a means ofshaping organizational and individual decisions, behaviors,and communication patterns [18]. The flow of informationand knowledge is codependent on our constant need tolearn and share knowledge, largely shaping the structure ofour social and organizational network arrangements [23].Therefore, the need to know or cognitive demand of bothindividuals and organizations becomes a primary driver ofIPE activity [18, 23]. Information processing for the sakeof storing mountains of data, information, and knowledgeresources as an end itself is meaningless in the context ofefficiency, effectiveness, and viability inmeeting public healthsystem organizational missions, goals, and objectives. Moreprecisely, the primary function of any level of IPE—fromsimple department units to complex multiorganizationalnetworks or health information exchanges—is to respondto what agents/actors need to know, when they need toknow it, and to support the choices/decisions that must bemade as shareholders navigate through the health system,defined here as actionable knowledge or intelligence [3]. Thepublic health organizational IPE will seek to leverage SAto maximize readiness to meet public health threats fromthe environment and to maximize public health knowledgeenvironment agent/actor individual and/or collective intel-ligence in the performance of core public health tasks andfunctions. Therefore, public health-centric SA serves as acomprehensive measure of public health system smartness

6 Computational and Mathematical Methods in Medicine

Smart healthsystems

measures andcapabilities

Knowledgediscovery

rate (KDR)Agent-

specific andsystem-level

memory

Agent-specific and

system-learning

Knowledgeabsorptionrate (KAR)

Agent-specific andsystem-level

cognitivedemand

Cognitivemapping

Aberrantdetectionanalytics

Figure 4: Smart health systems measures and capabilities.

and is essential for any standardized assessment of publichealth system performance from a public health informaticsperspective.

3. Smart Systems Vulnerability Index

In this section, we propose seven smart health systemmeasures and capabilities appropriate for helping to man-age public health organizational complexity, problem/issue/complexity, and situational awareness for public health sys-tems networks and public health knowledge environments.Figure 4 lists these seven measures and provides a briefdescription of a smart public health system and our rationalefor its use. Although other measures may be available inthe literature, we believe these seven effectively capturethe key concepts discussed above. Of course, public healthinformatics professionals may need to use discretion whenapplying measures based on the context, the purpose ofmeasurement, and any constraints hindering measurement.

3.1. Knowledge Discovery Rate (KDR). Thomas Davenportdescribed knowledge as data and information imbued withmeaning and relevance. In this way, knowledge is seen asa continued aggregation and refinement that begins withraw data elements. For example, a set of 10 numerical digitsconstitutes raw and meaningless data. At the stage of infor-mation, it can be recognized as a telephone number.

These same digits can be viewed as knowledge when thatnumber is contextualized as a conduit to satisfy some indi-vidual or organizational cognitive demand to support healthdecision-making or address some issue or problem. Thistelephone number can be viewed as a source of knowledgewhen, for example, it represents a nurse hotline for patientnavigation. The informatics professional managing/building

a mature public health knowledge environment works withstakeholders to develop a comprehensive knowledge inven-tory (formerly referred to as an ontology) of all products usedto inform public health stakeholder decision-making [23].

The knowledge discovery rate (KDR) represents the rateat which knowledge is generated from new or existingdata and information resources. In other words, the KDRshows how long it may take someone to (1) realize these10 digits represent a telephone number, (2) process thatthis telephone number is connected to a patient navigatingservice, and (3) realize the nursing navigation service hasadditional resources to maximize the care experience. Inpublic health, KDR may be expressed as the time it takeshealth officials to recognize a pattern in seemingly unrelatedhealth events (e.g., ER traffic, school/work absence, providercase reporting, news reports, food plant inspection reports,and grocery store sales), indicating a disease threat in theform of a potential food-borne illness. It should be notedthat display or interface is an essential component as well.Endsley explores how the interface and display of informationcan greatly deter the uptake of knowledge and consequentlyimpair overall situational awareness [22]. For example, willeveryone read these representations in the same manner—“1234567890” and “123.456.7890” and “(123) 456-7890”—intheir communication exchange?

A key component in shaping the rate of knowledge dis-covery involves comprehensively assessing the presentationand display of data, information, and knowledge throughoutkey stages of any healthcare delivery (e.g., clinical pathway) orpublic health process. To that end, KDR involves understand-ing how knowledge is packaged for consumption in the formor paper or electronic tangible (explicit) knowledge productsor as less tangible (tacit) knowledge products to informdecision-making. This measure examines the productioncurve of knowledge from generation, presentation, selection,and consumption, as well as the qualitative assessment ofknowledge’s relevance to agent-specific choice.

KDR may be particularly pertinent in settings in whichpattern recognition depends on a coordination of data,information, and knowledge from a highly heterogeneousnetwork of sources and stakeholders. KDR becomes essen-tial in public health situations where intelligence has tobe collated across multiple agencies (e.g., school, hospital,retail, and corporate), wide geographic boundaries (e.g.,multiple regional metropolitan area health departments), ormultiple categories of stakeholders (e.g., patients, providers,and health administrators). In such cases, KDR representsa measures of timeliness and operates as a key indicator ofpublic health outcomes.

3.2. Organizational (Agent- or Systems-Level) Memory. Ear-lier we described the matrix of system shock as a functionof preparedness and expectedness in response to internaland external stimuli (events). Organizational or systems-level memory can be described as the degree to which thehistory of these encounters, responses, and the relative degreeof success or failure of those responses are catalogued andstored for future use by other agents/actors in the future.This

Computational and Mathematical Methods in Medicine 7

can be operationally understood as the “repeatability” level,commonly referred to as level two of five in the capabilitymaturity model (CMM) [24]. Systems memory simply asksto what degree are phenomena captured and labeled asfavorable or unfavorable and response algorithms developedand made available for expedient consumption by the sameand/or other agents within the knowledge environment. Alack of repeatability represents a high level of unnecessary “adhoc” or CMM level-one responses [24] and may result in aninordinately high level of shock to the system for events thatif properly catalogued could have been relegated to the realmof routine with minimal system-shock value.

Here, the primary measure is to determine the levelof completeness, sophistication, and use of knowledge-bases that represent the sum of public health knowl-edge stored for current and future public health decision-making. This can be expressed as basic knowledge inven-tories, resource guides, policy and procedure manuals, andintranet/Internet lessons or best practices. It can also beexpressed as highly sophisticated knowledge ontologies thatcapture and display public health knowledge, tasks, events,and procedures in complex electronic tools to supportnetwork modeling, information flows, and critical com-munication pathways. Public health knowledge portals canbe constructed to identify public health stakeholder querydemand more easily, as well as access, retrieve, display,and analyze knowledge use in any public health knowl-edge environment. This capability is essential to the properuse and maximization of organizational memory. In theabsence of standardized knowledge memory management,a public health organization remains in a perpetual ad hocresponse mode to each new or reoccurring public healthcrisis.

3.3. Agent-Specific and System Learning. There is a growingbody of literature of the evolution of a “learning healthsystem” [25]. Our study contributes to this concept byproviding a conceptual definition of both agent-specific andsystem-level learning from the perspective of a public healthinformatics professional managing/building a public healthknowledge environment. Here, learning is understood as thewisdom level of the informatics continuum [26]. We haverefrained from using this concept throughout this discussion,but at this point it is appropriate to recognize that some infor-matics literature describes the informatics continuum (earlierreferred to as the data progression) as data to informationto knowledge to wisdom [26]. Typically, finding objectivemeasures of wisdom is not easy or universally accepted.However, we have chosen to substitute wisdom for decisionand outcomes. As a result, our data progression extends tothe following sequence: data to information to knowledge todecisions to outcomes. The difference is that choices, whenproperly linked to specific outcomes and their correspondingconsequences, provide opportunities for learning. As a resultof this substitution, we are now able to define the conceptof wisdom operationally as the degree to which choice—informed by relevant knowledge products—can lead to morehighly desirable decisions, beneficial outcomes, and positive

consequences for the overall health and wellbeing of agentsand the system.

We extend our definition of wisdom to incorporateintelligence, simply understood as the display of wisdom overtime. In our model, learning acts as a measure of differentialwisdom and intelligence over time (the difference measuredat two distinct points in time). In other words, this equationinvolves individual or organizational wisdom displayed ormeasured at some endpoint (𝑡2)minus the individual or orga-nizational wisdom displayed at some starting point (𝑡1). Theorganizational IQ in a learning health system is then under-stood as the measure of differential wisdom displayed overtime toward some set of decisions/choices, actions/tasks, orother health phenomena. Learning represents ameasurementof agent-specific or system-level discernment (the ability toleverage situational awareness in comprehending threat level,as well as leveraging stored or new knowledge in choosingbetween differing options). To this end, learning is construedas the means of refinement in the art of discernment orwisdom acquisition.

In public health terms, the operational construct of thismeasure of learning is still evolving, and little literature existson applying this construct in public health practice. Wesuggest thatmeasures of learning—presented here as ameansof leveraging knowledge resources in a wise manner—arelargely dependent on the previousmeasures of organizationalmemory. In the absence of a well-designed public healthknowledge-base that captures history or practice, learningfrom such experience becomes extremely episodic and anec-dotal in nature. For example, we can only speculate onhow much stored memory has been gathered with respectto the Ebola crisis that may more easily mitigate anotheroutbreak or similar outbreaks of other diseases in relatedconditions.The emergence of rapid learning health networksdeals with some aspects of this challenge by streamlining theprocessing of research evidence into practice and gatheringknowledge stores of what works best in achieving betterhealth outcomes. However, global implementations of theseresearch-to-practice and comparative effectiveness networksare still in the early stages of development.

3.4. Knowledge Absorption Rate (KAR). Carley et al. des-cribed how the sum of knowledge within a given systemboundary can be quantified in terms of knowledge bits [27–29]. According to this concept, knowledge represented in itsvarious forms can be deconstructed into quantifiable units[29, 30]. The number of knowledge units or bits that maycomprise a discreet package of knowledge is determinedby the level of complexity of the decisions or tasks thisknowledge is designed to inform [29, 30]. As such, a directrelationship exists between the number of knowledge bits andthe level of the complexity in related decisions and tasks. Thegreater the level of task or decision complexity/criticality ordecision, the larger the knowledge complement (or numberof knowledge bits) associated with the management, storage,display/representation, diffusion, use, and comprehension ofknowledge [29, 30]. This perspective assumes more knowl-edge bits are needed to saturate or carry out a complex

8 Computational and Mathematical Methods in Medicine

task or make a critical choice than to implement a moresimplified/less complex task or choice.

In essence, this concept of knowledge bits suggests thatthroughout the knowledge environment, agents/actors canhave either 0% saturation of knowledge (or 0 bits) up to 100%saturation of knowledge (or all bits available in the knowledgeenvironment). The consumption rate or absorption of theseknowledge bits over time, in the performance of core taskperformance, can then be evaluated using a variety of sta-tistical and computational modeling methods. To carry thisout, a value or weight is assigned to every piece of knowledgerepresented within a knowledge inventory (also referred toas ontology). The weight given to a knowledge productrepresents both the degree of value assigned by consumersof that knowledge product (elasticity-of-demand) and themagnitude of importance of the respective decision(s) it isintended to inform (criticality). The curve of a knowledgeproduct’s elasticity-of-demand and criticality of decisions isevaluated in the context of a core set of tasks to be performedat the agent or system level.

In public health settings, KAR represents a concrete wayof measuring overall application of knowledge to perfor-mance. In previous studies, we examined the knowledgeabsorption rate of community health clinical staff with regardto breast, cervical, and colorectal cancer screening policies,guidelines, and protocols as derived from the use of electronicclinical decision-support (CDS) [31].We examined the extentto which CDS use and corresponding knowledge absorptionrates would be correlated to organizational performance forcancer screening [31]. We demonstrated that KAR was, infact, a predictor of organizational performance in meetingprocess-of-care outcomes in cancer care [31]. Hence, wesuggest that KAR can serve as an effective measure of HITimpact on performance by focusing on end-users’ abilityto access key knowledge by interacting with HIT tools andapplying this knowledge to healthcare delivery and publichealth practice.

3.5. Agent-Specific and System-Level Cognitive Demand.Within any given knowledge environment, agents/actors atmany levels perform key tasks, make decisions, and engage ina series of activities that can be described by a set of processalgorithms [28]. The constant factor governing this activityis the principle of a cognitive demand for information. Theprinciple of supply and demand borrowed from the fieldof economics applies somewhat to the field of informat-ics with respect to agents’/actors’ need for information tosupport decision-making and task performance. Here, wefocus on the metric as a measure of the relative demand fordata, information, and knowledge resources by agents/actorsoperating at all levels of the multilevel model, as well asthe corresponding supply of data, information, and knowl-edge resources available for consumption. We refer to maindriver of the interplay between the supply and demand ofdata, information, and knowledge resources as the cognitivedemand or simply the “need to know.”

This “need to know” or cognitive demand shapesinformation-seeking behaviors of the agents/actors withinthe system and may govern the amount effort they are

Highly criticalLow supply

Highly criticalHigh supply

Low criticalLow supply

Low criticalHigh supply

Criticality(elasticity of

demand)

Supply

Knowledge gaps Knowledge parity

Knowledge parity Knowledge surplus

Figure 5: Criticality (elasticity-of-demand) and the supply ofknowledge.

willing to expend in acquiring the data, information, orknowledge resources. The level of importance or criticalityof information to the agent is measured by the elasticity-of-demand (a borrowed term) for that information.Themeasureof elasticity coupled with the relative supply of informationcan be used to measure relative states of “informedness” ofthe agents/actors within the system. According to the formaldefinition of elasticity, in an elastic demand, the change inquantity demanded due to a change in price is large [32].In contrast, an inelastic demand is one in which the changein quantity demanded due to a change in price is small[32]. Cognitive demand can serve as a core measure inidentifying knowledge-related vulnerabilities within a systemor the relative degree to which the cost of the knowledgerequired is acceptable or not acceptable.

We understand that in the context of health systems, theconcept of price with respect to knowledge can be measuredin terms of access, affordability (time and effort), overallopportunity-cost (ease-of-use, processing, comprehension,and understanding), and relevancy. Figure 5 lists that fourdistinct states agents/actors can assume within any publichealth knowledge environment based on the level of criti-cality (elasticity-of-demand) and the supply of knowledge.When the cognitive demand for knowledge is highly criticaland the relative supply is limited, knowledge gaps emerge.Such knowledge gaps may result from a variety of scenarios,including (1) the information or knowledge product does notexist, resulting in a need for innovation; (2) the resourceexists, but access is in some way limited or encumbered;(3) the resource exists with abundant access but is noteasily processed or consumed because of literacy challenges,content presentation, or other reasons; and (4) the supply ischallenged by other competing priorities and is intentionallyundeveloped or underdeveloped.The two states we refer to asparity conditions represent areas where the level of criticalityis adequately met by the level of knowledge supply. In suchcases, the main strategy is to employ continuous monitoring

Computational and Mathematical Methods in Medicine 9

to ensure balances remain within desired ranges of accept-ability in conjunction with the need for balance in the overallpublic knowledge environment. A state of knowledge surplusresults when the level of information or knowledge productsupply exceeds the relative level of importance placed onthe information or knowledge products (also understoodas relevancy). This state represents an opportunity for theelimination of outmoded or underused knowledge resources,information system redesign/upgrade, or other informationtechnology strategic efforts to ensure long-term relevance ofinformation resources, information systems, and knowledgeproducts.

The public health application of this measure of rela-tive cognitive demand or the need to know is illustratedby the recent Zika outbreak. It became evident that thepopulation at greatest risk for the disease was pregnant orsoon-to-be pregnant women. The demand for informationregarding protective measures, travel restrictions, the rateof transmission, the relative threat to fetal health, and signsand symptoms once infected created large pockets of healthconsumer uncertainty, stress, and anxiety. Given the severelevel of risk to pregnant women and their developing babies,the demand for knowledge of what to do for protection washighly critical. The window of transmission of the Zika virusfrom mother to fetus was highly uncertain, the effectivenessof preventive measures was hard to measure, and travel deci-sions to affected areas were rather unclear, resulting in condi-tions of highly critical and low-resourced knowledge profilesfor many health consumers and stakeholders at all levels.Rapid research was needed to identify proven measuresagainst the invading mosquito population. Public healthdepartments throughout the affected areas were scramblingto model the spread of the disease, measure the impact ofthe preventive measures, and manage the reports of newscases. Meanwhile, the public was constantly demanding newanswers and updates on a daily basis. This was compoundedby the timing of the 2016 Summer Olympic games in Riothat sparked highly publicized athletes refusing to travel tothe region to participate in the event. Highly critical/lowsupply-resourced conditions are probably the most difficultto manage. In any public health knowledge environment, acontinual assessment of stakeholder cognitive demand mustbe done—relative to the capabilities of the existing or evolvingknowledge-base—as a means of satisfying current and/orprojecting anticipated demands.

3.6. Cognitive Mapping. Once the knowledge inventory andrelative measures of importance are assessed and the corre-sponding process and information flows have been identified,the public health informatics professional can now engagein the process of creating cognitive maps or models ofboth existing and emerging knowledge and communicationpathways.These pathways can bemodeled for specific agents,for the system as a whole, or any combination of thetwo. Here, the public health informatics professional is notsimply asking who uses what information or examiningthe use of computerized information resources; instead, thegoal involves trying to model the cycle of information and

knowledge within the public health knowledge environment.This information cycle is best understood as starting withraw materials, in this case raw and at times unformatted dataelements, which are assembled into chunks of information(e.g., electronic databases or information systems). Theseinformation chunks are either coordinated in the formationof meaningful knowledge products or presented to usersof information to coordinate based on their specific needs(structured queries), which can be thought of as off-the-shelf knowledge products or ad hoc user-defined knowledgeproducts to support decision-making (ad hoc queries). Werefer to this cycle as knowledge refinery.

Analytic measures of knowledge refinement consist ofexamining the pace of knowledge development and exploringsystem responsiveness as expressed by the supply of anddemand for data, information, and knowledge resources[33]. The basic elements of analysis consist of the totalknowledge in any given public health knowledge environ-ment (knowledge entropy) relative to the amounts of usedkinetic knowledge and unused potential knowledge [33].Thiscan also be expressed in terms of the amount of knowl-edge/information gained or loss in an effort to maximizeperformance [33]. Additionally, the public health informaticsprofessional could examine existing and emerging pathwaysthat are developed through the examination of patterns ofuse, which is closely linked to the concept of plasticity[34]. In the field of neuroscience, the term neuroplasticityrefers to the human brain’s ability to change in responseto behavioral, environmental, and neural processes [35]. Inthe human brain, these pathways, after repeated stimulationand reinforcement, are actually carved into the brain tissue[35]. Like neural pathways are carved into the human brain,IPEs, as described earlier, may examine how public healthknowledge environments respond to changes in behavior,environmental conditions, or agent-specific or system-levelcognitive demands [34, 36].

Pathways of changes in public health knowledge environ-ment cognition can be modeled using a variety of conceptualand visualization techniques [37]. Such pathways, whenobserved and modeled, can yield repeated patterns, whichmay be canonized as permanent or semipermanent cognitivepathways toward system-level knowledge and learning healthsystems [37]. Within our model of public health knowledgeenvironments, highly intelligent health systems have theability to manage such cognitive pathways in response tocognitive demands [37]. Where old or unused pathwaysexist, data and information systems (and the correspondingknowledge products) will likely be considered outdated ornot useful. Where current cognitive pathways are robust andfrequented, data and information systems are likely to beconsidered essential to decision-making, and where new andemerging cognitive pathways are observed or predicted, thelikelihood exists for innovation and systems developmentto support emerging communities of practice, workgroups,department/divisions, and other formal or informal organi-zational structures [38].

The public health application lies in understanding thepublic health system as an evolving complex network ofindividuals, organizations, groups, and knowledge resources.

10 Computational and Mathematical Methods in Medicine

Here, the public health informatics professional may findknowledge, skills, and abilities in modeling social networksand organizational networks that are essential in establishingcurrent state network diagrams (baseline) and future statediagrams designed to guide the visualization of a publichealth knowledge environment. In this context, a largelibrary of network measures can be employed to supportthe analysis of a public health system and its respectiveknowledge environment, like measures of network densityagents, closeness/connectedness of agents to each other orto other knowledge resources, patterns of clustering andcliques behaviors, knowledge-sharing practices, and more.The application of social and organizational network analysisin public health is growing at a rapid rate and is expected tocontinue moving forward.

3.7. Aberrant Detection Analytics. Arguably the most impor-tant analysis within this discussion involves being able todetect subtle changes within the public health knowledgeenvironment that may pose a threat to one or more agents orthe system overall. Here, we discuss the ability of intelligentanalytics or telemetry as part of a Public Health SituationRoom that can be used to detect subtle changes within thepublic health knowledge environment. The public healthinformatics professional relies heavily on the use of probesand sensors as part of any surveillance and monitoringsystem to gather intelligence. Similarly, physicians and nursesrely on telemetry to monitor patient vital signs, driversuse dashboards to detect changes in automobile status, andinvestors use tickers to track global investments.These formsof monitoring and tracking systems have one feature incommon: they all make use of a grid system or networkof core indicators as validated predictors of overall systemhealth.

Any sensor network used to monitor and track activ-ity within a given public health Public Health SituationRoom must recognize several information-specific concepts.First, information does not always travel along predefinedorganizational departmental or process pathways. Instead,information exchange may occur along a multiplicity ofpathways, some predictable and others highly unpredictable.While organizational constructs of departments and divi-sions may account for some of communication and informa-tion exchange, they do not account for all activity. Therefore,the placement of public health data/event collecting sensorswithin a given public health knowledge environmentmust bea fluid network that is highly adaptive and capable of cap-turing activity in different settings, wherever the informationchannels may lead.

Second, the Public Health Situation Room sensor gridmust be able to identify and track activity by both inter-nal agents and system components, as well as by externalagents and system components that may interact with theenvironment. No public health knowledge environment iscompletely closed. As a result, sensors must capture intel-ligence from portals through which information travels inand out of the system in all forms. Finally, the level ofcompleteness must be defined to determine what representsan adequate level of coverage. A poorly designed or partial

Public Health Situation Room sensor grid that allows largelevels of undetected activity would not be useful on long-term. A sensor grid should be viewed as a living and growingnetwork of data/event collection activities that changes withevolving needs and priorities, and the level of granularityor specificity of detection must also be capable of changingwithin the grid as strategic priorities shift. The full list ofindicators drawn from public health knowledge environmentfactors, stakeholder-levels, and unique views will largelyshape the types of sensors, density of the network, andlevel of sensitivity needed for meaningful aberrant detectionalgorithms and monitoring system development.

Public Health Situation Rooms are used at all levels of thepublic health system throughout national and internationalhealth settings, the US Department of Health and HumanServices, various public health agencies, and healthcaredelivery settings across the globe. However, there is nostandardized model for this type of monitoring capability.Public Health Situation Room needs and priorities varywidely by organization and may include but not be limitedto disease management, outbreaks investigation, emergencypreparedness, disaster response, community health assess-ments, and even healthcare access, equity, and quality. Theuse of electronic performance, strategic, operational, andclinical dashboards are typical of such PublicHealth SituationRooms. We argue the primary challenge of the public healthinformatics professional in the design and execution of PublicHealth Situation Rooms is to develop the underlying smartinfrastructure (knowledge-base) and array of analytic mea-sures described in this discussion to ensure the maximumimpact on desired outcomes.

4. Conclusion

As we enter a public health informatics era with terms likelearning health systems, smart health systems, and adaptivecomplex health systems, we must identify a common set ofanalytic measures and capabilities to inform our modeling,measuring, and managing of public health “smartness.” Sucha set of measures must take into account the full spectrumof sociotechnical factors that make up a public health systemand shape performance, including technical, organizational,and human contributions. It is essential that we understandthe basic drivers of smart systems, expressed in this dis-cussion as simply the need to know or cognitive demand.This basic need to know and our corresponding effort toleverage data, information, and knowledge resources towardsome individual or collective set of goals and objectivesform the basic parameters of any smart system. In thecontext of a public health system, public health informaticsprofessionals stand poised to redefine the benefit of smarterhealthcare delivery and public health practice. A common setof analytic measures and capabilities that can drive efficiencyand viable models can demonstrate how incremental changesin smartness generate corresponding changes in public healthperformance. Here, we introduced the concepts of organiza-tional complexity, problem/issue complexity, and situationalawareness as three codependent drivers of smart publichealth systems characteristics. We also propose seven smart

Computational and Mathematical Methods in Medicine 11

health systems measures and capabilities that are consideredessential in a public health informatics professional’s toolkit.Because this area of research and practice is still in itsformative stages, the intent of this discussion is to buildon the developing body of literature seeking to establishstandardized measures for smart, learning, and adaptivepublic health systems.

Disclosure

This work represents the opinion of the author and cannotbe construed to represent the opinion of the US FederalGovernment. Timothy Jay Carney is the founding partnerof the Global Health Equity Intelligence Collaborative, LLC,Durham, NC (2014).

Competing Interests

The authors declare that they have no conflicts of interests.

Authors’ Contributions

Timothy Jay Carney participated in the conceptual modeldevelopment, design, and literature review of themanuscript.Christopher Michael Shea contributed to the review andsubstantial redesign of the manuscript for resubmission.

Acknowledgments

The manuscript is supported by The University of NorthCarolina Gillings School of Public Health andThe LinebergerComprehensive Cancer Center and The Carolina Commu-nity Network Center to Reduce Cancer Health DisparitiesDiversity Supplement 3U54CA153602. Special thanks are dueto Hannah M.L., Elance/Upwork consultant, for outstandingediting, content review, and consultation on manuscriptdevelopment.

References

[1] R. Kukafka andW. A. Yasnoff, “Public health informatics,” Jour-nal of Biomedical Informatics, vol. 40, no. 4, pp. 365–369, 2007.

[2] W. A. Yasnoff, P. W. O’Carroll, D. Koo, R. W. Linkins, and E.M. Kilbourne, “Public health informatics: improving and trans-forming public health in the information age,” Journal of PublicHealth Management and Practice, vol. 6, no. 6, pp. 67–75, 2000.

[3] C. E. Hsu, W. C. Chambers, J. R. Herbold, J. C. Calcote, R. S.Ryczak, and R. F. DeFraites, “Towards shared situational aware-ness and actionable knowledge—an enhanced, human-centeredparadigm for public health information system design,” Journalof Homeland Security and Emergency Management, vol. 7, no. 1,2010.

[4] J. G. S. H. A. March, Organizations, Blackwell, Cambridge,Mass, USA, 1993.

[5] T. H. Davenport and L. Prusak, Information Ecology: Masteringthe Information andKnowledge Environment, OxfordUniversityPress, New York, NY, USA, 1997.

[6] A.V.D. Roux, “Complex systems thinking and current impassesin health disparities research,” American Journal of PublicHealth, vol. 101, no. 9, pp. 1627–1634, 2011.

[7] K. H. Lich, E. M. Ginexi, N. D. Osgood, and P. L. Mabry, “Acall to address complexity in prevention science research,”Prevention Science, vol. 14, no. 3, pp. 279–289, 2013.

[8] M. Arndt and B. Bigelow, “Commentary: the potential of chaostheory and complexity theory for health services management,”Health CareManagement Review, vol. 25, no. 1, pp. 35–38, 2000.

[9] R. Kling, “Organizational analysis in computer science,” TheInformation Society, vol. 9, no. 2, pp. 71–87, 1993.

[10] A. Bandura, Social Learning Theory, Prentice Hall, EnglewoodCliffs, NJ, USA, 1977.

[11] R. Kling, H. Rosenbaum, and S. Sawyer, Understanding andCommunicating Social Informatics: A Framework for Study-ing and Teaching the Human Contexts of Information and Com-munication Technologies, 2005, http://public.eblib.com/choice/publicfullrecord.aspx?p=3316144.

[12] N. M. Lorenzi and R. T. Riley, Organizational Aspects of HealthInformatics: Managing Technological Change, Springer, NewYork, NY, USA, 1995.

[13] R. A.Thietart and B. Forgues, “Chaos theory and organization,”Organization Science, vol. 6, no. 1, pp. 19–31, 1995.

[14] United States Government Accountability Office, Public HealthInformation Technology Additional Strategic Planning Needed toGuide HHS’s Efforts to Establish Electronic Situational AwarenessCapabilities: Report to Congressional Committees, United StatesGovernment Accountability Office, Washington, DC, USA,2010.

[15] C. Grossmann, W. A. Goolsby, and L. Olsen, Engineering aLearning Healthcare System: A Look at the Future: WorkshopSummary, 2011, http://public.eblib.com/choice/publicfullrecord.aspx?p=3378793.

[16] R. M. Burton and B. Obel, “The Dynamics of the ChangeProcess,” in Strategic Organizational Diagnosis and Design, vol.4 of Information and Organization Design Series, pp. 385–420,Springer US, Boston, Mass, USA, 2004.

[17] I. Nonaka, “A dynamic theory of organizational knowledgecreation,” Organization Science, vol. 5, no. 1, pp. 14–37, 1994.

[18] M. Popper and R. Lipshitz, “Organizational learning mech-anisms: a structural and cultural approach to organizationallearning,” Journal of Applied Behavioral Science, vol. 34, no. 2,p. 161, 1998.

[19] M.M. Crossan, H.W. Lane, and R. E.White, “An organizationallearning framework: from intuition to institution,”TheAcademyof Management Review, vol. 24, no. 3, pp. 522–537, 1999.

[20] L.Warren andT. Fuller,Contrasting Approaches to Preparedness,2011.

[21] G. Klein, B. Moon, and R. R. Hoffman, “Making sense of sense-making 1: alternative perspectives,” IEEE Intelligent Systems, vol.21, no. 4, pp. 70–73, 2006.

[22] M. R. J. D. G. Endsley, Designing for Situation Awareness: AnApproach to User-Centered Design, CRC Press, Boca Raton, Fla,USA, 2012.

[23] S. Bowen, Ontology (knowledge representation in informationscience), 2012.

[24] K. M. Dymond, A Guide to the CMM: Understanding the Capa-bility Maturity Model for Software, Process Inc US, Annapolis,Md, USA, 1995.

[25] C. P. Friedman, A. K. Wong, and D. Blumenthal, “Achievinga nationwide learning health system,” Science TranslationalMedicine, vol. 2, no. 57, 2010.

12 Computational and Mathematical Methods in Medicine

[26] P. F. Smith and D. A. Ross, “Information, knowledge, andwisdom in public health surveillance,” Journal of Public HealthManagement and Practice, vol. 18, no. 3, pp. 193–195, 2012.

[27] K. Carley, “Adaptive organizations and emergent forms,” inProceedings of the 3rd International Conference on Multi AgentSystems (ICMAS ’98), Paris, France, July 1998.

[28] D. Krackhardt and K. M. Carley, PCANS Model of Structurein Organizations, Carnegie Mellon University, Institute forComplex Engineered Systems, Pittsburgh, Pa, USA, 1998.

[29] C. Schreiber, S. Singh, and K. M. Carley, “Construct—AMulti-Agent Network Model for the Co-Evolution of Agents andSocio-Cultural Environments,” 2004, http://handle.dtic.mil/100.2/ADA460028.

[30] B. R. Hirshman, K. M. Carley, and M. Kowalchuck, Speci-fying Agents in Construct, 2007, http://handle.dtic.mil/100.2/ADA500804.

[31] T. J. Carney, G. P. Morgan, J. Jones et al., “Using computationalmodeling to assess the impact of clinical decision support oncancer screening improvement strategieswithin the communityhealth centers,” Journal of Biomedical Informatics, vol. 51, pp.200–209, 2014.

[32] conomicsonline.co.uk/, Price elasticity of demand. Economics,http://www.economicsonline.co.uk/Competitive markets/Priceelasticity of demand.html.

[33] A. Golan and E.Maasoumi, “Information theoretic and entropymethods: an overview,” Econometric Reviews, vol. 27, no. 4–6,pp. 317–328, 2008.

[34] D. Scarborough andM. J. Somers,Neural Networks inOrganiza-tional Research: Applying Pattern Recognition to the Analysis ofOrganizational Behavior, American Psychological Association,Washington, DC, USA, 2006.

[35] I. MedicineNet,Definition of Neuroplasticity, MedTerms™, 2016,http://www.medicinenet.com/script/main/art.asp?articlekey=40362.

[36] DARPA Neural Network Study (U.S.), Lincoln Laboratory, andUnited States. Air Force. Systems Command, “DARPA neuralnetwork study final report,” 1989, http://books.google.com/books?id=ruRGAQAAIAAJ.

[37] T. Hammad, “Computational intelligence: neural networksmethodology for health decision support,” in Health DecisionSupport Systems, J. K. H. Tan and S. B. Sheps, Eds., AspenPublishers, Gaithersburg, Md, USA, 1998.

[38] K. Carley, “Smart agents and organizations of the future,” inHandbook of New Media: Social Shaping and Consequences ofICTs, L. A. L. S. M. Lievrouw, Ed., pp. 206–220, SAGE, London,UK, 2002.