decision-making with heterogeneous sources of information

Risk Analysis 3-3963 Art. 374 Jodi

Risk Analysis, Vol. 19, No. 1, 1999

Decision-Making with Heterogeneous Sourcesof Information

Donald Richards1 and William D. Rowe2

In any model the values of estimates for various parameters are obtained from differentsources each with its own level of uncertainty. When the probability distributions of theestimates are obtained as opposed to point values only, the measurement uncertainties inthe parameter estimates may be addressed. However, the sources used for obtaining thedata and the models used to select appropriate distributions are of differing degrees ofuncertainty. A hierarchy of different sources of uncertainty based upon one’s ability tovalidate data and models empirically is presented. When model parameters are aggregatedwith different levels of the hierarchy represented, this implies distortion or degradation inthe utility and validity of the models used. Means to identify and deal with such heterogeneousdata sources are explored, and a number of approaches to addressing this problem is pre-sented. One approach, using Range/Confidence Estimates coupled with an InformationValue Analysis Process, is presented as an example.

KEY WORDS: Parameters; probability distributions; validity.

1. INTRODUCTION

In any decision model, particularly decisions in-volving risks and benefits, estimates for the variousmodel parameters are obtained from differentsources each with its own level of uncertainty. Whena parameter in such a model is characterized by aprobability or a likelihood distribution for which notmuch is known, the aggregation of these parametersin a decision model, along with parameters whosedistributions are better characterized, must be ad-dressed. For example, if a critical parameter and itsdistribution are being estimated without much infor-mation, the method of estimation may be the keydeterminant of the uncertainty in the decision, lead-ing to a biased (perhaps, even unrecognized) conclu-sion. Conversely, if the parameter is not crucial, anyreasonably acceptable conclusion may suffice without

1 University of Virginia, 112 Olsson Hall, Charlottesville, Vir-ginia 22903.

2 RREA Inc.

690272-4332/99/0200–0069$16.00/1 1999 Society for Risk Analysis

significant bias. Therefore the aggregation of modelswhose parameters have such heterogeneous sourcesof uncertainty must be addressed with some rigor,and strategies and processes developed to providepractical approaches to such aggregation.

As an initial approach, these heterogeneoussources of uncertainty can be organized in hierarchi-cal form based upon one’s ability to validate bothdata and models empirically. This structure can pro-vide insight into the manner in which parameters inmodels may be aggregated with different levels inthe hierarchy represented. The types and extent ofuncertainty in the parameters to be aggregated, theuses of the models, and the techniques that are usedto aggregate the information all have implications asto the validity of the result and the expression of thecombined uncertainties in the result.

The purpose of this paper is not only to presentthe hierarchy, but also to identify the sources of un-certainty that may be present, how the models areused, and some approaches for making better deci-sions for problems of this type. First the hierarchy is


70 Richards and Rowe

presented along with some examples. Different usesof models and their associated needs for differentsources of information are discussed; these are ad-dressed in terms of their need for empirical valida-tion. Next, the sources of uncertainty are examinedusing a pre-existing framework. Then several ap-proaches are given to manage the informationneeded in these types of decisions. First a value ofinformation approach to acquiring new data to re-duce uncertainty is presented. Recognizing that ex-pressing uncertainty to decision makers, the publicand technical experts is a major problem, severalclassical and nonclassical approaches are describedto address this problem. Since range estimates ratherthan point estimates are more informative means todescribe uncertainty, the implications for varioustypes of these estimates are discussed. Lastly, oneapproach, Range/Confidence Estimation, for hand-ling information that uses margins of safety is pre-sented along with an example to illustrate how theuncertainties may be addressed directly, using thestructures described in the paper.

2. HIERARCHY OF DATA SOURCES BASEDUPON EMPIRICAL VALIDATION

The following hierarchy is one way of categoriz-ing data sources based on the degree to which thedata can be validated empirically at the top to unvali-dated data at the bottom :

1. Standard Distribution. Empirical data vali-dated by many different investigations and generallyagreed upon as a standard. For example, the distribu-tion of adult weight in the United States.

2. Empirical Distribution. Empirical data vali-dated for specific instances. For example, the distri-bution of arsenic concentration in 20 samples forten wells.

3. Validated Model. A model that has been vali-dated empirically in repeated experiments. For exam-ple, determining altitude from local pressure instru-ments from surface pressure and humiditymeasurements.

4. Unvalidated Model. A model which has notbeen empirically validated. For example, inferenceabout the extrapolation of effects on human cancerat low doses from high dose data in animals. Valida-tion may or may not be possible.

5. Alternate Models. Alternate unvalidatedmodels to those selected above that seem to be lessreasonable, but possible, at the time that the model

is used in a decision process. For example, a thresholdmodel for cancer. Alternate models may actually besuperior to the primary model from differing perspec-tives.

6. Expert Value Judgment. In the absence of em-pirical data the judgment of experts may be used.Since the information is unvalidated, the selection,framing and bias of experts becomes dominant.Sometimes consensus may be important; otherwisediversity may be important and retained. For exam-ple, should benign tumors that do not progress tomalignancy be counted in mouse exposure data?

7. Best Guess Estimate. A form of presumablyexpert judgment based upon reasonable expectationand the framework in which the decisions are asked,but acknowledging the lack of integrity in the esti-mates at the outset.

8. Test Case. Values used to test models for theirsensitivity, critical variables and the value of addi-tional information.

There are degrees between each step in the hier-archy, and one may add or delete levels; but the listprovides a means to differentiate between sourcesof uncertainty.

Different uses require different levels of the hi-erarchy. Presumably, one always wants to start withthe best scientific data (the top of the hierarchy).However, if the policy decisions involved do not re-quire this precision, then the associated costs in timeand resources may be exorbitant. There are a varietyof uses such as Scientific Information—empiricaldata stressed, Regulatory Decisions—mix of empiri-cal and value data, Economic Decisions—cost andvalue data stressed, and Public Policy—values arethe drivers.

3. TYPES OF UNCERTAINTY3

Different types of uncertainty are inherent ateach level. How do they impact the process at eachlevel.

Temporal. Uncertainty in future states and un-certainty in past states.

Structural. Uncertainty due to complexity(modal uncertainty).

Metrical. Uncertainty in measurement.Translational. Uncertainty in explaining uncer-

tain results.All four types occur in any situation, but de-

pending on the situation one or more dominates.

3 This material is derived from Rowe (1994).


Decision-Making with Heterogeneous Sources of Information 71

Table I. Parameters of the Types of Uncertainty

Uncertainty Unknown Discrimination Valuationclass information parameter parameter Methods

Temporal Future Probability Luck PredictionTemporal Past Historical data Correctness RetrodictionStructural Complexity Usefulness/likelihood Confidence ModelsMetrical Measurement Precision Accuracy StatisticsTranslational Perspective Goals/values Understanding Communication

Although these types are not necessarily indepen-dent, the nature of each type is quite different. Eachclass can be addressed separately, and then their in-teraction may be examined.

Each of these types of uncertainty has a set ofassociated parameters that define their role. Table Isummarizes them. The left column of the table liststhe types where Temporal has two entries, one forthe future and one for the past. The first set of param-eters to the right of the type shows the particularinformation that is unknown and uncertain in termsof single word descriptions. The second parameterto the right, the discrimination parameter, describesthe parameter used to differentiate among differentlevels of uncertainty. This parameter is often de-scribed in reverse; that is, in terms of the informationused to reduce the uncertainty. For, example, for thetemporal (future) type of uncertainty, high probabil-ity indicates increased certainty. The third parameterindicates the degree to which the uncertainty hasbeen successfully addressed. For example, in the tem-poral (future) case, if a desirable, but low probabilityevent occurs, one has had good luck. The right mostcolumn shows the primary methods used for handlingthe uncertainty.

Future uncertainty uses the concept of probabil-ity as a discrimination parameter, and is based on amodel (structural uncertainty) which presupposesthat the future will behave similarly to past experi-ence. The model may be expressed as an a priori, afrequentist, or a subjectivist approach. The outcomeis valued by ‘‘luck.’’ Before an event, a probabilitymodel of the likely outcomes can be constructed.After the event occurs, one considers a desirableoutcome as ‘‘lucky,’’ and an undesirable one as ‘‘un-lucky.’’ Selecting a case where an undesirable out-come has low probability makes good sense, but theundesirable outcome can still occur.4

4 This case might be considered as ‘‘competent error,’’ the unluckyoutcome of a rare event, as opposed to ‘‘negligent error,’’ know-ingly or unknowingly choosing a high probability for an un-wanted outcome.

Past information is subject to different interpre-tations as requirements change or ‘‘political correct-ness’’ imposed. The past is not always a useful pro-logue to the future. The degree to which historicaldata were recorded, stored, and made available, in-cluding the biases and resources of the collectors,differentiates between adequate and inadequate in-formation. The correctness of the interpretation, tak-ing into account quality and quantity of the historicaldata and the biases of both the collector and inter-preter, determines the value of the result. Measure-ment error occurs when the data needed are not thehistorical data that were collected or were inade-quately collected.

Structural uncertainty, as a result of complexity,depends on both the number of model parametersand their interaction as well as the degree to whichmodels of the complex situation are useful. If themodels used also correspond to reality to a suitabledegree, they tend to have higher value. This is partic-ularly the case when the models can be validatedempirically. The idea of likelihood of membership isone method for differentiating among model parame-ters; for example, the plausibility that model A ismore likely to be valid than model B. Note that bothprobability and likelihood use similar mathematicalapproaches, but each address a different set of con-cerns, the former the prediction of future event out-comes, and the latter degree of membership to a setof model conditions. The confidence in the estimatesin terms of validation is the valuation parameter.

Measurement uncertainty lies in our inability tomeasure; that is, the limitations in measuring instru-ments. The ability of a measuring device to discrimi-nate among values of the parameter being measuredis its precision. Accuracy addresses errors made inusing the device, that is, how good was the measure-ment. (For example, if a ruler is only precise to centi-meters, but a five centimeter line is measured as sevencentimeters, then the measurement is inaccurate bytwo centimeters). Statistical sampling methods areused to detect and control the measurement errorsof this type.


1 Line Long (exception grid 9998)


Table II. Differing Perspectives of Uncertainty in Risk EstimationLeading to Translational Uncertainty

Approach Limitations

Scientific risk approach Tend to measure that whichProvide the best estimate is measurable rather than

of risk and the ranges of that which is critical butuncertainty above and be- difficult to measure.low the best estimate.

Regulatory risk approach Very high margins of safety,Assure, with a given degree may have high cost and

of confidence, that the ac- preclude some beneficialtual risk does not exceed activities.the risk estimate.

Design engineering approach Very high margins of safety,Use conservative, proven may have high cost, and

designs to produce engi- are subject to unwelcomeneering structures with surprises when applied inminimum liability. new environments

Performance management ap- Risks cannot be measured,proach only modeled. EmpiricalRisks are one parameter verification of perfor-

used in balancing risk, mance is often very dif-costs, benefits and the ficult.performance of engi-neered structures. Uncer-tainty is addressed in allfour parameters.

Translational uncertainty differs from the otherthree types in many respects. It occurs after the firstthree have been considered. People have differentperspectives, goals and values, as well as differentcapabilities and levels of training. These affect theinterpretation of results of analyses; and, thereby,interact with these interpretations, often contributingto the uncertainty. Translational uncertainty arisesfrom differing perspectives of users, and differinggoals and values lead to different interpretations ofboth qualitative and quantitative results. Better un-derstanding, perhaps, through enhanced communica-tion, often can reduce this type of uncertainty. TableII illustrates four differing perspectives of dealingwith risks, all valid for their purpose, but with differ-ent biases.

4. VALUE OF INFORMATION (TO THEDECISION)

Several techniques provide a process for analyz-ing the value of new information.5

5 Mathematics of structure vs mathematics of content must beaddressed. If one is looking for an ultimate methodology withstructural completeness, its application may be so general as tolimit the effectiveness of valuing information. If one is willing toconstrain the methodology to specific cases with definitive con-tent, the information content itself lends itself to valuation.

1. Identify the most critical parameters and de-termine the range of uncertainty in them. In multipli-cative and raised power models, those parameter esti-mates with greatest measures of dispersion will bedominant. In additive models, the dispersion in theparameters with largest values will dominate. Gener-ally speaking, great attention should be paid to pa-rameters having estimates with large standard devia-tions.

2. In terms of the decision alternatives, deter-mine what data will be sufficient and non-confound-ing to resolve the decision. A sensitivity analysis usingtest data may provide this information. One approachto performing sensitivity analyses is by way of the‘‘expected value of perfect information’’ approach(Morgan and Henrion, 1990, pp. 197, 263), which isa measure of the sensitivity of a decision analysisto uncertainty.

3. Determine the costs and likelihood of ob-taining sufficient information. Can the costs beavoided or offset by simultaneously accepting moreuncertain data in noncritical parameters. That is, canone move down the hierarchy for noncritical vari-ables without affecting the outcome of the decision?

4. If the costs in time and/or resources are exor-bitant, or if the uncertainty is unresolvable, are thereother parameters and methods that may be used toretreat from a totally unresolvable decision?

5. Where differences exist in expert judgementover choice of models (model uncertainty) or in val-ues assigned to parameters (measurement uncer-tainty), consider preserving the diversity to the endof the decision-making process where the implica-tions of these differences can be ascertained by thosemaking the decisions as well as the stakeholders. Asimportant in postponing a final decision on the choiceof models and/or parameters can be the providentialarrival of further information or data which can helpto clarify the choice of model. An example of this isthe well-known story of the drug thalidomide in theU.S. : Because of disagreements among pharmaceuti-cal and medical experts, approval to sell this drug inthe U.S. was delayed; after the terrible birth defectscaused by the drug started to appear in Europe, at-tempts to issue the drug were halted.

4.1. Identifying Critical Parameters

The identification of critical parameters may be-gin with an examination of the range of uncertaintyin each of a model’s parameters. As will be describedsubsequently, an uncertainty index based upon range


1 Line Long (exception grid 9999)


is used with Range/Confidence estimates, and theseindices may be used directly for this purpose. Thewider the range or the larger the value of an uncer-tainty index, the higher the range of the underlyingdistribution (known or unknown). Focusing on pa-rameters with higher uncertainty provides a first cutat identifying critical parameters.

Next, the parameters may be separated into themajor categories that dominates each parameter :

Metrical Uncertainty. These parameters are themost amenable to uncertainty reduction by acquiringbetter information through improved measurementsand/or additional data points.

Temporal Uncertainty. Better historic data doesnot always provide better forecasts. Mitigation ofpossible events (at additional cost for increased capa-bility) may be a better way to constrain the range ofconsequences.

Structural Uncertainty. Better validation of mod-els are necessary for uncertainty reduction in thiscategory with costs associated with the validation.Development of alternate or more complex modelswon’t help if they can’t be validated. Constrainingparameter ranges can be useful in limiting the deci-sion in some cases.

Translational Uncertainty. Finding better meansto communicate and present uncertainties to decisionmakers and stakeholders may be very cost-effectivefor reducing this type of uncertainty.

The costs of uncertainty reduction may also beaddressed for each parameter, as well as the likeli-hood that further information will, indeed, bring clo-

Fig. 1. An information value analysis process flow diagram.

sure as opposed to confounding the results. Cost-effectiveness of uncertainty reduction and elasticitymeasures may be good measures of closure.

Some possible closure measures based upon theuncertainty index (addressed later); Ix, the cost ofbetter information; C; and the likelihood of closure,p; are :

Percentage Change in Uncertainty Index : Q 5100 *DIx/Ix

Cost Effectiveness of Index Reduction : C/E 5DQ/DC

Elasticity of Index Reduction : E 5 d(DQ)/dC(DC)

Probability of Successful Reduction Factor :F 5 p * Q

Probability of Successful Cost/Effectiveness :P(C/E) 5 p* D Q/DC

Probability of Elasticity Reduction : P(E) 5 p*EBoth the cost and likelihood estimates can them-selves be Range/Confidence Estimates and the jointconfidence in the measures calculated and displayed.

4.2. Flow Chart of an Information Analysis Process

One approach to an Information Analysis Pro-cess is shown in Fig. 1. After making an initial range/confidence estimate, a test is made to see if the deci-sion can be resolved with the present level of informa-tion. If not, the steps outlined above are condensedinto the functional flow in the figure. After each cycle,



a test is made to see if the decision can be resolvedin the presence of normally distributed data.

5. MEANS OF EXPRESSING UNCERTAINTY

In analyzing the data, it is important to give acareful review of the important uncertainties existingwithin the framework of the conclusions. Thereforewe will review briefly some methods for expressingthe uncertainties contained within an expertjudgment.

5.1. Spread

The spread inherent to any random dataset willpreclude the expert from stating conclusions of com-plete certainty. Several classical point estimates ofthe spread or dispersion of a dataset are well-knownto be powerful statistical tools for expressing, in aconcise numerical or graphical manner, the uncer-tainties caused by the variability of the observed data.

5.2. Range

The ‘‘range’’ of a dataset, which is the differencebetween the largest and smallest observations, is easyto compute, but suffers from a lack of robustness topotential outliers. The ‘‘sample standard deviation’’is the quintessential measure of dispersion; althoughthis measure is also not robust in the presence ofoutliers, it is well-known to possess superb inferentialproperties. The ‘‘coefficient of variation’’ measuresthe dispersion of a dataset relative to the mean ofthe data; in the presence of heteroskedastic data,the coefficient of variation often is superior to thestandard deviation.

5.3. Measures of Confidence

Although point estimates of dispersion are use-ful for assessing uncertainty, more fundamental toolsfor the measurement of uncertainty must be utilized.To review some of these procedures, we considerthree general setups :

5.3.1. Empirical Data

In this situation, the expert will review the datafrom an empirical standpoint. For instance, the extent

to which the sampling procedure is complete mustbe ascertained, and the methods of analysis must besuited to the manner in which the data were collected.For instance, it is a not-uncommon error to collectdata by means of a stratified random sampling schemebut then to attempt to analyze the data using thestatistical procedures proper only for simple randomsampling schemes. Similarly, heteroskedastic pro-cesses cannot be analyzed correctly by means ofmethods developed for homoskedastic processes. Toavoid this type of error, the expert must develop adeeper understanding of the underlying process.

An analysis of the shape of the distribution orhistogram is important to the process of determiningwhether or not the data appear to be drawn from,say, a symmetric population. One- and two-sided con-fidence intervals and regions also are well-knownprocedures for estimating parameters of the process.Confidence intervals also capture succinctly the de-gree of confidence, hence also the uncertainty, whichthe expert may have in parameter estimates. Hypoth-esis tests and P-values, some of which may be derivedfrom confidence intervals or regions, also form a well-studied set of procedures for capturing the degree ofuncertainty in hypothetical statements about aprocess.

5.3.2. Models

At the center of the analysis is the expert’s goalto identify a class of models which correctly describethe process. A proposed model must be capable ofvalidation by the procedures applicable to empiricaldata, as described above. For instance, a probabilisticmodel for a data set must be well-suited to describethe shape and standard deviation of the observedhistogram; and the parameters in the model shouldbe capable of accurate estimation by probabilisticmethods. Models should withstand simple tests ofreasonableness and common-sense, and should becomplete in the sense that they are able to describefuture data sets which may be moderately differentfrom the data already collected.

The degree of complexity of a proposed modelshould be minimized; here we are reminded of Oc-cam’s razor and of information-theoretic proceduresfor minimizing the number of parameters in a pro-posed model. Finally, statistical tests (e.g., chi-squaregoodness-of-fit tests) for the correctness of proposedmodels need to be performed.



5.3.3. Expert Judgment

An expert judgment is simply an expression ofconfidence by individuals who have demonstratedexpertise in their chosen field. It is important to re-member that although a consensus of experts is usu-ally a happy occasion, consensus does not imply orguarantee validity. In short, a consensus of expertsis an expression of their joint agreement as to thecorrectness of their opinion. Thus it is as importantto carefully review a consensus opinion as it is impor-tant to review a diversity of opinions.

Computer-based expert systems (e.g., neuralnetworks or other fully-automated methods), for ana-lyzing data are now an important part of the industrialand technological scene. In this situation, severalcomputing machines are connected in a ‘‘voter sys-tem’’; with individual machine computations subjectto approval by all machines. If sufficiently many com-puters ‘‘vote’’ in favor of a given computation thenthe results are passed on to the next level of thesystem. These systems are especially useful when thedata are generated by very-high-speed processes andare of a technically routine nature; an example is abusy trading day on the NYSE.

6. IMPLICATIONS FOR VARIOUS TYPES OFRANGE ESTIMATES

6.1. Application to Monte Carlo AnalysisTechniques

6.1.1. Stochastic Simulation

During the past two decades the use of simula-tion has grown enormously, with uses ranging over :aircraft design and pilot training; construction of masstransportation systems, analysis of nuclear explo-sions; inference about the effect of new drugs onhumans based upon testing on laboratory animals;models for the spread of pollution of undergroundwater; reliability testing of double-hulled oil tankers;design of communication systems for simultaneoustransmission of video, voice and data; and analysisof incomplete multivariate data.

Simulation methods provide powerful, cost-ef-fective routes to analyzing the reliability of complexstructures or processes, allowing the engineer tostudy random phenomena while accounting for theprobability of every outcome. As computer time/storage costs continue to decrease, it becomes more

feasible to perform extensive simulation of com-plex processes.

In constructing simulation procedures, a basicproblem is to simulate the probability distributionsof random variables. There are several methods ofsolving this problem : the standard Monte Carlo tech-nique, Markov chain methods, Gibbs sampling proce-dures. These procedures have found wide-rangingapplications in the engineering sciences.

6.1.2. Resampling Techniques

Resampling methods, such as the bootstrap andjackknife procedures, have revolutionized some as-pects of statistics since the 1980’s (Shao and Tu, 1995).These techniques allow the scientist to repeatedlysample the observed data, and to estimate accuratesolutions to complex inference problems. Resam-pling methods can provide accurate answers to somecomplex, multidimensional probabilistic problems.These techniques require large amounts of computertime and memory, and they have become increasinglypopular as computers have become more efficient.

Classical statistical techniques have demon-strated how to estimate the population mean andstandard deviation, and to measure the accuracy ofthose estimates. An important advantage of the jack-knife and bootstrap resampling methods (over classi-cal techniques) is that they allow us to estimate anyparameter and to evaluate the accuracy of the esti-mating procedure.

6.2. Other Approaches

6.2.1. Classical Methods

Despite the success of the modern, computer-intensive methods, it is worthwhile to remember thatthe classical methods remain important. The classicalmethods are generally less computer-intensive thantheir modern counterparts, so they are easier to applyin field locations where extensive computational facil-ities are unavailable. Moreover, statistical estimationthrough, say, confidence/prediction intervals or re-gions are relatively simple to interpret. For instance,an engineer under deposition in a complex lawsuitover an expensive engineering failure stands a muchbetter chance of explaining to a jury of lay personsthe concept of a prediction interval than he/she wouldhave of explaining a Gibbs sampling simulation of



the Gaussian distribution. Thus, it is important forengineers to continue to require a proper groundingin classical statistical techniques as part of the entryfee for the profession (Nelson, 1990).

These methods generally are used to addressmeasurement uncertainty through measures of cen-tral tendency and higher moments. Factor analysis,ANOVA and correlation approaches address uncer-tainty due to complexity. Probability theory is usedto address future uncertainty.

6.2.2. Multivariate Models

Multivariate models also play a central role inthis area. These models are crucial to studyingmultifactor processes in which correlations betweendifferent factors are of concern. Multivariate reliabil-ity models are fundamental to analyzing the patternof observed failures of a system and to gaining insightinto the reasons underlying the failures. Multivariatelogistic models have proved important in analyzingthe velocity of automobile occupants during acci-dents, and to identifying those factors (e.g., occupantweight, velocity of automobile at the time of crash,weight and size of the automobile, usage/non-usageof airbags and/or seatbelts) which are of primaryimportance in explaining the level of severity of injur-ies. Multivariate survival models have been appliedto study : the failure of electrical cable insulation;dosage levels of radiation in energy-generating com-plexes; failure times of small electrical appliances(Johnson and Wichern, 1992). In these problems, itis the multivariate nature of the models which giveinsight into the interdependence among the factorsunderlying the observed data. These methods partic-ularly address structural uncertainty and conflictinggoals.

6.2.3. Time Series Methods

Approaches based on time series methods areof primary importance when the data are continuousin nature. Autoregressive or moving average modelsmay be useful in the study of data sets in whichobservations at neighboring time points are corre-lated. In a study of signal-to-noise ratios for noise-detection or mine-detection systems, time seriesmodels may greatly contribute to an understanding ofthe important explanatory variables. These methods

particularly address uncertainty of past states; and tothe extent that past behavior is likely to continue inthe future, can be a predictor of future behavior.

7. RANGE/CONFIDENCE ESTIMATES

Range/Confidence Estimation is a methodologywhere three values and their respective confidencelevels are used to study the distributions of inputvariables, providing ranges rather than point esti-mates. The combinations of joint values and jointconfidence levels are then ordered and displayed.

This approach is useful for certain classes ofproblems : (1) when data are insufficient for a morecomprehensive probabilistic treatment, (2) when aninitial analysis is needed rapidly for developing astrategy for further data acquisition, (3) when thedata are from widely different sources with differingdegrees of precision, (4) when safety factors are usedto provide margins of safety, and (5) as a means toprovide screening for uncertainty analysis.

The approach provides for direct use and evalua-tion of heterogeneous data sources. An uncertaintyindex, based upon the range of the estimates, pro-vides a direct measure of the uncertainty inherent inthe estimates in conjunction with a confidence ex-pression about those estimates. The initial resultsmay then be subjected to an Information Value Anal-ysis Process (IVAP) which provides a strategy forcost-effective acquisition of further information toreduce critical uncertainties. In addition, criteria fordeciding when to stop the process can be provided.

7.1. Basis for Range/Confidence Estimates

7.1.1. Range/Confidence Estimating Protocol

There are many ways to make range estimates.However, the method of making range estimates usedhere is a purposely one-sided approach for elicitinginformation on each input parameter from heteroge-neous data sources.6 For risk and costs an overesti-mate is made of how confident is the analyst or data

6 One-sided distributions, either high or low, as well as two-sideddistributions can be addressed. For simplicity here only the one-sided approach for eliciting information for each input parameteris addressed here, and the one-sided distribution direction (over-estimate) is selected to provide consistency throughout the deci-sion model example used.



source is that the actual risk or cost lies below theestimate. For benefits and performance it is the re-verse, that is an underestimate is made. Three esti-mates of a parameter (e.g., probability of an eventwith a given consequence magnitude) are made usingalternative models or alternative levels of parameterswithin a model.

Bare Estimate. A ‘‘bare estimate’’ is made withno margins of safety. This is, by definition, a bestestimate of what is likely to occur, but the actual riskthat is being estimated may be either higher or lower.It represents an estimate where all margins of safetyhave been removed. This type of estimate is oftenvery difficult to make for engineers who are trainedto build in margins of safety. However, it is necessaryto insist on this type of analysis if the margins ofsafety are to be made subsequently visible and usedas a tool for addressing uncertainty. A central valueof the estimate (mean, median, mode) may be usedas appropriate here; for example, in the case of anormal distribution, the sample mean is the appro-priate choice.

Conservative Estimate. A ‘‘conservative esti-mate’’ is made using margins of safety high enough toprovide confidence (a reasoned subjective judgment)that actual risk lies below the estimate. This type ofestimate is consistent with conservative engineeringdesign. For example, if the underlying process was anormal distribution then an illustrative example isthe 95th percentile.

Worst Case Estimate. A ‘‘worst case estimate’’attempts to make sure that the actual risk is alwaysbelow the model estimate. The estimate takes intoaccount all reasonable, imaginable excursions. In anormal distribution it might exceed the 99th per-centile.

These three (or more) estimates make up a rangeestimate as shown in the illustrative example in Fig.2, represented by the solid line connecting the three.Any one of the three estimates cannot be taken outof context with the other two. To do so invalidatesthe framework.

The estimate for each parameter is independentof the other parameters, and represents the state ofknowledge and confidence about knowledge that theestimator has at the time of the estimate. If the totaldistribution of all the parameters were known to thesame degree, other methods such as Monte Carlowould be appropriate. Conversely, only gross valuejudgments can be made about the critical parameters,the total distribution of noncritical parameters haslittle value to the decision.

Fig. 2. Example of a range/confidence estimate.

7.1.2. Uncertainty Indices

The distance between the bare and the worst-case estimate of a parameter is a measure of its uncer-tainty, and an uncertainty index may be derived bydividing the result of the worst-case estimate by thatof the bare case :

Uncertainty Index (UIX)5 Worst Case Result/Bare Case Result

The magnitude of the index is used as a meansto make comparisons about the relative uncertaintyof range estimates. A higher index implies higheruncertainty in the range estimate.

An important aspect of these estimates is therecognition that many of the explicitly expressed con-fidence levels are subjective. They represent the sub-jective evaluation of the estimator, and are subjectto challenge, requiring documentation of the assump-tions used to make the estimates. The subjectivityemployed has always been there with point estimates,but was normally invisible to the decision maker.This process makes it visible and manageable.

Challenges to particular subjective estimates arebest addressed by sensitivity analysis when both therisk estimates and confidence level estimates are com-bined to make a range estimate for an achievementstandard. Experience has shown that increasing rangevalues at high levels of confidence are often relativelyinsensitive to the decision when a number of rangeestimated parameters are combined in risk models.



7.1.3. Combining Range/Confidence Levelsin Models

Different types of risk analysis use differentmodels and parameters suitable for the problem be-ing addressed. In a Range/Confidence Estimate allparameters in models are estimated using ranges,and are combined according to model requirements.Since there are three estimates for each parameterand all combinations of estimates must be made thereare many more computations to be made. For exam-ple, if there are two parameters, nine estimates ofrisk are required. In general, the number of risk calcu-lations to be made (N) is determined by N 5 3n wheren is the number of variables.

Since all the calculations are arithmetic, they areeasily automated. Spreadsheets work very well anda number of templates have been designed for thispurpose.

The risk level and combined confidence level (aswell as costs when appropriate) are calculated foreach of the n combinations. These Range/ConfidenceEstimates are presented as a plot of increasing riskon the abscissa vs. the expressed combined confi-dence level on the ordinate for all combinations ofrisk and confidence levels. The resulting curve usuallyhas a very sharp knee, indicating very high levels ofover conservatism, and provides a totally new per-spective on deciding on achievement levels and ac-ceptable confidence levels. Moreover the very wideranges of uncertainty in the estimates are shown di-rectly to the decision makers. Based upon combinedconfidence levels, a new combined range estimatemay be made with a much smaller uncertainty index,eliminating unnecessary margins of safety.

In a Monte Carlo analysis, random combinationsof values of each parameter are used to exercise themodel, leading to an array of joint probability values.The range/confidence approach uses selected sam-ples of the distributions rather than the whole distri-bution, and computes both the joint probability andthe joint confidence for the joint probabilities.

7.1.4. Reporting Results

Range estimates, as opposed to point estimates,will be made for all parameters in risk/benefit orcost/benefit analyses. For risk analyses both the prob-ability of occurrence of events and the magnitude ofconsequences are addressed. As a minimum, a bare,conservative and worst case estimate will be made

of each event probability along with the confidencelevel for each estimate. Note that a Range/Confi-dence estimate is a compound estimate, and no indi-vidual estimate may be taken out of context with theother two. To do so is invalid, until a managementdecision is made, using and specifying a given level ofconfidence that the actual risk lies below the estimate.

The confidence in a parameter value is assignedby the estimator, and represents his or her subjectiveevaluation. For empirical data, the estimate is muchless subjective than for a value estimate (e.g., whatpoints on the empirical distribution to use). Con-versely, the confidence of a ‘‘guess estimate’’ may betotally subjective.

7.2. Templates for Risk Analysis

Each generic type of risk analysis, using Range/Confidence Estimates requires a template which pro-vides the risk analysis structure and models peculiarto the type of problem on hand. An example shownbelow for arsenic in drinking water is an example ofa template, in this case it addresses health risks froma RCRA perspective, using U.S. EPA RegulatoryAssessment Guides (RAGs) as a basis. A number ofsuch templates have already been used successfully.

2.4.1 Example—Risk Analysis of Arsenic inWell Water

An example of a multiplicative model is the de-termination of the risk of ingestion of a chemical(arsenic) considered as a carcinogen in well water.Arsenic is found in well water in varying concentra-tions with a data base of 20 samples from each of 40wells. A maximum exposed individual uses one wellfor drinking, and the pathway is direct ingestion ofwell water. Exposure is controlled by periodic moni-toring on a monthly, yearly or lifetime (no monitor-ing) basis, and assumes that if concentration levelsexceed a given value in any period that alternatewater sources will be used. Table III shows the valuesof each of seven parameters. For example, for con-centration, the base case is the mean of all the sam-ples, the conservative case, the 90th percentile, andthe worst case, the maximum value found. The expo-sure days represent the time between monitoringsamples for different monitoring strategies. The po-tency-slope are EPA extrapolations from high to lowdose at various confidence levels. The next three arefrom standard or empirical distributions with onlythree points chosen from the distribution as shownin Table III.



Table III. Range/Confidence Estimates of Input Parameters for Arsenic in Well Water

Bare Cons. WorstParameter case case case B. Conf. C. Conf. W. Conf. Units UIX Shape Soruce

Concentration 0.3 1.2 32 50% 90% 99.00% eg/liter 107 Convex Empirical distributionExposure daysa 30 365 25550 50% 95% 99.99% days/lifetime 852 Convex Unvalidated modelPotency slopeb 0.15 15 30 50% 95% 98.00% 1/(mg/kg/day) 200 Convex Unvalidated modelDaily liquid intake 2.1 3.3 3.8 50% 90% 95.00% liters/day 1.8 Concave Standard distributionTap water portion 0.62 0.8 1 50% 90% 99.99% fraction 1.6 Concave Empirical distributionAdult weight 70 51 45 50% 95% 98.00% kilograms 1.6 Concave Standard distributionRetention factor 0.6 0.8 1 50% 90% 99.99% fraction 1.7 Concave Unvalidated model

a Based upon [BC] monthly monitoring, [CC] yearly monitoring, [WC] no monitoring (lifetime).b Estimated for BC and WC.

The risk is calculated by the following model :

Lifetime Risk 5 I 3 T 3 C 3 D 3 P3 R/(W 3 1000 3 25550)

where I 5 Ingestion in liters/day (standard distribu-tion), T 5 Tap water portion of ingested water (em-pirical distribution), C 5 Concentration in ug/liter(mean, 90th percentile, and highest value of 400 ac-tual samples), D 5 Days of exposure (based onmonthly, yearly, or no monitoring and substitution ofsupply when contaminated), R 5 Biological retentionfactor (unvalidated model), P 5 Potency-slope valuein 1/(mg/kg/day) (unvalidated U.S. EPA model),W 5 Adult weight in kg (standard distribution), 1000is the conversion factor of 1000 mg/eg, 25550 is thenumber of days in 70 years.

The uncertainty index in Table III shows therelative uncertainty ranges for each parameter withthe exposure days and the potency slope with thegreatest ranges of uncertainty. The shape parameter

Fig. 3. Graphic display of range/confidence calculations for arsenic in well water.

determines whether the conservative case is aboveor below a linear distribution. The data source foreach parameter is also shown. Note also that theadult weight parameter is in the opposite directionof the others since it appears in the denominator ofthe model.

Figure 3 shows a graphical display of results withthe probability of lifetime cancer on the abscissa andthe joint confidence in the estimate on the ordinate.The abscissa is shown as a distorted scale in order todistinguish the points at the lower probabilities. Thejoint confidence is computed by calculating the 37

(2187) vertices in probability space and placing thesein confidence level bins.7 Note that the scale coversover five orders of magnitude, and this uncertainty

7 The calculations are performed automatically by established com-puter programs and are not described in detail because of spacelimitations. The details of this methodology have been publishedelsewhere. (Rowe and Burnham, 1996).



may be reduced by examining the critical variablesand reducing their ranges using an Information ValueAnalysis Process if adequate information to make adecision is lacking.

If one examines the results in Fig. 3, it is obviousthat the lifetime risk to an individual is below 1.5 31027 for a confidence level of over 99.99%. In thiscase, a decision on acceptable risk can be made atthe very first test of the process in Fig. 1. Otherwise,the uncertainty indices in Table III indicate that thedays of exposure is the parameter with the largestrange (852). This parameter can be constrained bysampling more often, at increased cost, for example,monthly, annually and biannually such that the bestcase is 30 days, the conservative case is 365 days, andthe worst case 730 days, with an uncertainty indexof 24.3.

7.3. Some Implications

Range/Confidence Estimation results provideboth the joint values of the model and the joint con-fidence for each value. When ordered as shown inFig. 3, for example, the margins of safety can beaddressed directly. The choice is how much marginof safety, as expressed in the confidence that the jointrisk lies below the estimate, does the decision makerwant. Combined with cost estimates of the alterna-tives, the cost of increasingly larger margins of safetycan be ascertained, that is, the cost of increased credi-bility as expressed by larger margins of safety ismade visible.

Admittedly, Range/Confidence Estimates rep-resent only a small sample of the data that may beavailable for a parameter (the whole distribution maybe known empirically). However, if the critical vari-ables are unvalidated models or judgment of experts,the loss of this noncritical information is diminished.Any judgments made on these critical parametersmay be no better than the Range/Confidence Esti-mates made using the protocol outlined here. Thesituation determines what methods are appropriate.

When confidence levels for risk values are avail-able, one can address the uncertainty in risk estimatesdirectly by specifying the level of confidence accept-able rather than just a range of uncertainty in therisk estimates or at worst, a single value for risk.In many cases, more information and more detailedanalyses may be required, but the Information ValueAnalysis Process provides a means to address these

further studies in a more focused and cost-effectivemanner.

8. CONCLUSIONS

Analyze your models before performing MonteCarlo analyses. Identify the critical variables andconcentrate on those. Once a variable is determinedto be noncritical, don’t worry too much about it.Address the data corresponding to critical variablesso you can focus your resources on these data,particularly if the data available about these criticalvariables are based upon unvalidated models orexpert judgment. Do not expend resources on vari-ables that can be easily measured and validated,but are not critical. Providing data because youcan easily obtain it for noncritical values only tendsto obfuscate the analysis and the uncertainties in-volved.

Keep in mind the uncertainty in your resultsand do not take the results to mean more thanthey do. One should not only calculate the jointvalues for a Monte Carlo analysis, but the confi-dence limits for these joint values as well. Knowingthe confidence allows better judgments to be madeby decision-makers since these confidence levelsprovide insight into the reliability of the resultsand the margins of safety available along with thecost of these margins.

Critical parameters which use unvalidated mod-els, have competing alternative models, or dependupon the judgment of experts, dictate the limitationsof your results. Evaluate what you do not know andmake it explicitly visible for the evaluator of your re-sults.

Methods such as Range/Confidence Estimationprovide means to rapidly assess data from highlyuncertain sources as well as for known distributions,determine if enough information is available for adecision. Regardless of the methodology used forestimation, an Information Value Analysis Processcan provide cost-effective data acquisition whenbetter information is necessary.

ACKNOWLEDGMENTS

The authors wish to acknowledge the reviewersof the original paper for their insightful commentsand their help in focusing the direction of the paper.



REFERENCES

Berger, J.Statistical Decision Theory and Bayesian Analysis, 2ndEd. (Springer-Verlag, New York, 1987).

Hofer, E.‘‘On Some Distinctions in Uncertainty Analysis,’’ Meth-ods for Treatment of Different Types of Uncertainty, PSAC/DOC (90) 11 (OECD Nuclear Energy Agebcy, Paris,1990).

Howard, R. A. ‘‘Information Value Theory,’’ IEEE Trans. Syst.Sci. Cybernet. SSC3 1, 22–26 (1996).

Johnson R. A., and Wichern, D. W. Applied Multivariate StatisticalAnalysis (Prentice-Hall, Englewood Cliffs, NJ, 1992).

Karlsson P, and Haimes, Y. Y.‘‘Probability Distributions and TheirPartitioning,’’Water Res. Res. 24 (1), 21–29 (1988).

Morgan, M. G., and Henrion, M. Uncertainty: A Guide to Dealingwith Uncertainty in Quantitative Risk and Policy Analysis(Cambridge University Press, New York, 1990).

Nelson, W. Accelerated Testing : Statistical Models, Test Plans andData Analysis (Wiley, New York, 1990).

Rowe, W. D., and Burnham, M.‘‘Risk Analysis of Levee ClosuresUsing Range/Confidence Estimates’’, Y. Y. Haimes, D. A.Moser, E. Z. Stakhiv, (eds.), Risk-Based Decision Making inWater Resources VII (American Society of Civil Engineers,New York, 1966).

Rowe, W. D.‘‘Understanding Uncertainty,’’ Risk Anal. 14(5), 743–749 (1994).

Shao, J., and Tu, D. The Jackknife and Bootstrap (Springer-Verlag,New York, 1995).

decision-making with heterogeneous sources of information

Documents