bette meek-2013-a framework for fit

7/22/2019 Bette Meek-2013-A Framework for Fit

1/7

Workshop Report

A framework for fit-for-purpose dose response assessment

M.E. Bette Meek a,, Michael Bolger b,1, James S. Bus c,2, John Christopher d, Rory B. Conolly e, R. Jeffrey Lewis f,Gregory M. Paolini g, Rita Schoeny h, Lynne T. Haber i, Amy B. Rosenstein j, Michael L. Dourson i

a R. Samuel McLaughlin Centre for Population Health Risk Assessment, University of Ottawa, One Stewart Street, Suite 309, Ottawa, ON, Canada K1N 6N5b Center for Food Safety and Applied Nutrition, U.S. Food and Drug Administration, 5100 Pain Branch Parkway, College Park, MD 20740, USAc Toxicology and Environmental Research and Consulting, The Dow Chemical Company, Building 1803 Washington St., Midland, MI 48674, USAd Independent Consultant, 8173 Suarez Way, Elk Grove, CA 95757, USAe U.S. Environmental Protection Agency, 109 TW Alexander Dr., Research Triangle Park, NC 27711, USAfExxonMobil Biomedical Sciences, Inc., 1545 Route 22 East, Annandale, NJ 08801, USAg Risk Sciences International, 325 Dalhousie Street, Ottawa, ON, Canada K1N 7G2

h U.S. Environmental Protection Agency, 1200 Pennsylvania Ave., Washington, DC 20460, USAi Toxicology Excellence for Risk Assessment, 2300 Montana Ave., Suite 409, Cincinnati, OH 45211, USA

j Independent Consultant, 18 Ames Ave., Lexington, MA 02421, USA

a r t i c l e i n f o

Article history:

Received 11 September 2012

Available online 6 April 2013

Special note: This paper is dedicated to the

memory of Dr. Randall Manning of the

Georgia Department of Natural Resources.

Dr.Manning wasan earlyand avid supporter

of this collaborative effort, making

constructive comments during its 3rd

meeting, and listening well and supportingothers in their efforts to find common

scientific ground on seemingly intractable

risk assessment issues. His humor, kindness

and intellect are sorely missed.

Keywords:

Mode of action

Fit-for-purpose

Problem formulation

Framework

Tier

Endogenous

Methods compendium

a b s t r a c t

TheNRC report Science and Decisions:Advancing RiskAssessmentmadeseveral recommendations to improve

chemical risk assessment, with a focus on in-depth chronic doseresponse assessments conducted by the

U.S. Environmental Protection Agency. The recommendations addressed two broad elements: improving

technical analysis and utility for decision making. To advance the discussions in the NRC report, in three

multi-stakeholder workshops organized by the Alliance for Risk Assessment, available and evolving risk

assessment methodologies were considered through the development and application of case studies. A

key product was a framework (http://www.allianceforrisk.org/Workshop/Framework/ProblemFormula-

tion.html) to guide risk assessors and managers to various doseresponse assessment methods relevant

to a range of decisioncontextsrangingfromprioritysettingto fullassessment,as illustrated by casestudies.

It is designed to facilitate selection of appropriate methodology for a variety of problem formulations and

includes a variety of methods with supporting case studies, for areas flagged specifically by the NRC com-

mittee for consideration e.g., susceptible sub-populations, population variability and background. The

framewok contributes to organization and communication about methodologies for incorporatingincreas-

ingly biologically informedand chemical specific knowledge into doseresponseanalysis, which is consid-

ered critical in evolving fit-for-purpose assessment to address relevant problem formulations.

2013 Elsevier Inc. All rights reserved.

1. Introduction

In 2009, the National Research Council of the National Academy

of Sciences (NAS) released a report entitled Science and Decisions:

Advancing Risk Assessment (NRC, 2009 also known as the Silver

Book). Recommendations encompassed two broad elements: (1)

improving technical analysis, namely developing and using scien-

tific knowledge and information to promote more accurate charac-

terization of risk; and (2) ensuring that risk assessments provide

meaningful support to allow discrimination among risk

management options. Specifically, recommendations addressed

the following areas: design of risk assessments, uncertainty and

variability, selection and use of defaults, a unified approach to

doseresponse assessment, cumulative risk assessment, improving

0273-2300/$ - see front matter 2013 Elsevier Inc. All rights reserved.http://dx.doi.org/10.1016/j.yrtph.2013.03.012

Abbreviations: ARA, alliance for risk assessment; CPF, chlorpyrifos; MOA, mode

of action; NAS, National Academy of Sciences; NGO, non-governmental organiza-tion; NRC, National Research Council; PBPK/PD, physiologically-based pharmaco-

kinetic/pharmacodynamic; RfD, reference dose; VOI, value of information. Corresponding author. Address: Chemical Risk Assessment, McLaughlin Centre

for Population Health Risk Assessment, University of Ottawa, One Stewart Street,

Suite 309, Ottawa, ON, Canada K1N 6N5. Fax: +1 613 562 5380.

E-mail addresses: [email protected] (M.E. Bette Meek), [email protected]

(M. Bolger), [email protected] (J.S. Bus), [email protected] (J. Christopher),

[email protected] (R.B. Conolly), [email protected] (R.J. Lewis),

[email protected] (G.M. Paolini), [email protected] (R. Schoeny),

[email protected] (L.T. Haber), [email protected] (A.B. Rosenstein), dourson@-

tera.org (M.L. Dourson).1 Current address: Center for Chemical Regulation and Food Safety Exponent, 1150

Connecticut Ave. NW, Washington, DC 20036, USA.2 Current address: Senior Managing Scientist, Center for Toxicology and Mecha-

nistic Biology, Health Sciences Group, Exponent, USA.

Regulatory Toxicology and Pharmacology 66 (2013) 234240

Contents lists available at SciVerse ScienceDirect

Regulatory Toxicology and Pharmacology

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / y r t p h
http://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://dx.doi.org/10.1016/j.yrtph.2013.03.012mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://dx.doi.org/10.1016/j.yrtph.2013.03.012http://www.sciencedirect.com/science/journal/02732300http://www.elsevier.com/locate/yrtphhttp://www.elsevier.com/locate/yrtphhttp://www.sciencedirect.com/science/journal/02732300http://dx.doi.org/10.1016/j.yrtph.2013.03.012mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://dx.doi.org/10.1016/j.yrtph.2013.03.012http://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.html


2/7

the utility of risk assessment and stakeholder involvement and

capacity-building within the U.S. Environmental Protection Agency

(U.S. EPA).

As illustrated in Figure S-1 ofNRC (2009), the authors of the re-

port expanded the risk assessment paradigm of NRC (1983), prin-

cipally through inclusion of a problem formulation step including

framing of the assessment to address specific risk management op-

tions, explicit consideration of stakeholder input, and confirmation

that the assessment addressed the issues identified in the problem

formulation. The report (see Figure 58) provided additional guid-

ance on considerations for doseresponse assessment, including

endpoint assessment, assessment of mode of action (MOA), vulner-

able populations and background exposure, conceptual model

selection, and doseresponse method selection.

In response to recommendations of this report and other NRC

and international initiatives (e.g., NRC, 2007; IPCS, 2006, 2007;

Meek and Armstrong, 2007; Meek et al., 2011), improvement of

risk assessment practice continues to be explored in a series of ini-

tiatives, including the workshops described here, organized by the

Alliance for Risk Assessment (ARA, a coalition of non-profit

organizations).

The purpose of the workshop series, entitled Beyond Science

and Decisions: From Problem Formulation to DoseResponse

Assessment (henceforth, the ARA workshop series) was to extend

these discussions, with the goal of developing a practical compen-

dium of dose response assessment methods for fit-for-purpose

doseresponse analysis and potentially in future, other compo-

nents of risk assessment. While not referenced in the NRC report,

the concept of fit-for-purpose assessment has been widely adopted

recently in legislative mandates requiring greater efficiency in con-

sideration of much larger numbers of substances (see for example,

Meek and Armstrong, 2007) andin research initiatives, for example

in Lee et al. (2006), who described a fit-for-purpose approach for

biomarker method development and validation. Fit-for-purpose

doseresponse analysis encourages application of a level of rigor

commensurate with the intended purpose and use of an assess-

ment. As recommended by the NRC (2009) report, the nature andextent of the assessment needs to be considered in the problem

formulation stage, with level and complexity to be no greater than

that needed to identify the best choice among risk management

options (i.e., fit for purpose). In practice, this is accomplished

by having a variety of available tools (e.g., tools for acute vs.

chronic exposures) and using tiered approaches, proceeding down

the tiering only as far as necessary to set an issue, exposure or

chemical aside (as not of concern) or to target it for further assess-

ment and/or management.

Three multi-stakeholder workshops were held in 2010 and

2011. The workshops explored available and evolving methodolo-

gies through the development and application of case studies.

While these case studies covered a number of important aspects

of the NAS text, particular attention was focused on problem for-mulation, use of information on MOA and endogenous and back-

ground exposure during solicited speaker presentations and

panel discussions. This paper summarizes the outcome of the

ARA workshops and the resulting ARA fit-for-purpose dose re-

sponse assessment methods framework. This framework, which

is illustrated by case studies, is designed for use by risk managers

and scientists in a variety of settings (e.g., government agencies,

industry), for a range of applications and/or levels of analysis

including distributional, non-threshold methods for estimating

risk-specific doses for toxic effects other than cancer. Case studies

were selected to be illustrative of various approaches rather than

as assessments for any specific environmental contaminant. How-

ever, the scope and variety of included case studies are anticipated

to assist in the determination of appropriate assessment strategiesand relevant risk management options. Additional case studies are

also being sought for consideration in the context of the

framework.

2. Description of the workshop series

2.1. Workshop objectives and structure

The DoseResponse Advisory Committee (DRAC), which in-cludes state, federal, industry, and NGO representatives, organized

the workshop series on behalf of the now more than 50 workshop

sponsors. The DRAC determined the agendas in consultation with

the Science Panel. The Steering Committee of the ARA, which in-

cludes representatives from state, tribal, the federal government,

academia, and environmental NGOs (www.allianceforrisk.org/

ARA_Steering_Committee.htm) provided oversight of the work-

shop series.

The workshops were designed to address technical aspects

(methods development) based on robust process (stakeholder

engagement), as described in Table 1. Important aspects included

(1) broadly advertising the workshops; (2) providing for web-

based participation; (3) posting all workshop-related materials

on the web; (4) providing an open process for interested partiesto develop and submit case studies; and, (5) sponsorship by a

group of more than 50 diverse organizations.

The first workshop included two primary elements. About half

of this workshop was devoted to presentations by thought leaders

from various sectors on activities related to issues raised in the

NRC (2009) report, as well as perspectives on the NRC report.

The other half of the workshop was devoted to brainstorming

and evaluation of the impact for the NRC recommendations of 27

submitted proposals for case studies developed by volunteer teams

of scientists from numerous organizations. Some of the case stud-

ies reflected previously published work, while others were de-

signed to evolve specific methodological issues identified in the

NRC (2009) Science and Decisions report, such as approaches for

low-dose extrapolation. Based on the recommendations from

Workshop 1, case studies were developed and presented to the Sci-

ence Panel at Workshop 2 for their review, recommendations, and

consideration for incorporation into the Framework (see

Section 3.1).

Workshop 3 was organized primarily around three cross-cut-

ting topics identified by the Science panel: (1) problem formula-

tion, (2) use of mode of action information, and (3) endogenous/

background exposure. Discussion of each of these themes was ini-

tiated by a presentation by an expert on the topic, followed by

Science Panel discussion in the context of the case studies

presented.

Presentations, meeting material and reports from all three of

the workshops are available at, http://www.allianceforrisk.org/

ARA_Dose-Response.htm.

2.2. The science panel

Following an open nomination process, the ARA Steering Com-

mittee selected a Science Panel designed to reflect a range of affil-

iations, perspectives, and expertise (e.g., biology, risk assessment,

modeling). Particular effort was made to include representatives

from the NRC Science and Decisions committee and environmental

NGOs. Invitations were sent to 27 nominees, with 13 individuals

accepting the invitation. The Science Panel members for

Workshops 2 and 3 are listed at http://www.allianceforrisk.org/

Workshop/Panel.htm. Science Panel members provided input on

the utility of the case study methods to address specific problem

formulations, and identified areas for additional development ofthe case study and/or method. After the first three workshops, a

M.E. Bette Meek et al. / Regulatory Toxicology and Pharmacology 66 (2013) 234240 235
http://www.allianceforrisk.org/ARA_Steering_Committee.htmhttp://www.allianceforrisk.org/ARA_Steering_Committee.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/Workshop/Panel.htmhttp://www.allianceforrisk.org/Workshop/Panel.htmhttp://www.allianceforrisk.org/Workshop/Panel.htmhttp://www.allianceforrisk.org/Workshop/Panel.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Steering_Committee.htmhttp://www.allianceforrisk.org/ARA_Steering_Committee.htm


3/7

similar open nominations process was followed, and the ARA

Steering Committee selected a standing Science Panel to serve for

23 years.

3. Results and discussion

This section describes issues related to three cross-cutting

topics that were the focus of the third workshop: (1) problemformulation, (2) use of mode of action information, and (3)

endogenous and background exposure. In addition, it describes

the development of the framework to organize and provide ac-

cess to dose response assessment methods, as illustrated by the

case studies.

3.1. Framework for identifying context specific methods for dose

response analysis

A key product of the ARA workshops was a framework to

guide risk assessors to different dose response methods relevant

to a variety of decision contexts ranging from priority setting or

screening to full assessment, and illustrated by case studies. The

framework is designed for use by risk managers and scientistsat the problem formulation stage, to aid in selecting appropriate

dose response assessment methodology, based on the objectives

of any specific assessment taking into account factors such as

time and resource constraints. This central resource for access

to methods and guidance from many organizations was

considered helpful as a basis to promote better planning for dif-

ferent levels of analysis required for specific risk assessment

applications.

The framework is available on-line at http://www.alliancefor-

risk.org/Workshop/Framework/ProblemFormulation.html and on

the National Library of Medicines Enviro Health Links suite of

databases at http://sis.nlm.nih.gov/enviro/toxweblinks.html . The

current version is derived from the general structure of the

doseresponse portion of the risk assessment framework pre-sented in the NRC (2009) report (Figures S-1 and 58 of the

NRC report). Based on need, the user chooses from amongst a

number of methods options for different decision contexts (e.g.,

priority setting, screening and full assessment); the ARA frame-

work also references case studies illustrating methods that can

address the topics and questions in the NRC report. Case studies

were selected based on the Science Panels evaluation of the util-

ity of the method to address a practical application. Inclusion of a

method or case study as an illustration of a useful technique does

not imply Science Panel consideration or acceptance of the chem-

ical-specific outcome.

It is expected that the framework will continue to be updated

with new and evolving methods. For example, areas where addi-

tional methods or tools exist are identified, even if case studies

have not yet been presented for consideration. Additional case

studies illustrating these and other methods are expected to be

added to the framework as they are submitted and considered by

the Science Panel.

3.2. Major themes discussed at workshop 3

Although the NRC (2009) report focused primarily on ap-

proaches to doseresponse analysis for in-depth assessments ofhazard from chronic exposure, the case studies addressed a broad-

er range of methods relevant to a variety of different decision con-

texts, including those for short term exposures and screening

assessments (e.g., see Case Study #1 below).

In addition to populating the framework with individual case

studies, the Science Panel focused its discussions in the final work-

shop on three cross-cutting topics: problem formulation, use of

MOA information, and the issue of how to address endogenous/

background exposures. Discussions concerning those cross-cutting

topics are summarized here and illustrated by representative case

studies. Additional details on these case studies including

strengths, weaknesses and minimum data needs, are available

at: http://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.

html. The case studies presented here address some of the majorthemes from Workshop 3, as well as illustrating the range of variety

in data requirements, approaches, and sophistication of analysis.

3.2.1. Theme 1: Problem formulation consideration of the risk

management Objectives and Options

The NRC (2009) report emphasized that the level and complex-

ity of a risk assessment should be no greater than what is needed

to identify the best choice among risk management options. Thus,

in order to optimize requirements for data generation and/or iden-

tification, available risk management options need to be consid-

ered at the earliest stage of an assessment and re-considered in

an iterative fashion throughout the process, based on additional

complexities uncovered during initial phases involving consulta-

tion with stakeholders.The extent of formal problem formulation in risk assessment

varies among different programs, stakeholders and institutions,

depending on decision context.

Value-of-information (VOI) analysis is a decision analytic meth-

od that characterizes the relative contribution (in a specific deci-

sion context) of specific information in reducing various

uncertainties in an assessment (Yokota and Thompson, 2004).

VOI analysis can be part of the problem formulation step, as well

as contributing to consideration of whether or not more informa-

tion would meaningfully inform an assessment. While there are a

number of challenges in instituting formal VOI analysis (including

the need to specify prior distributions and the need to know the

sensitivity of decision-makers choices to risk assessment out-

comes), the panel considered that informal (qualitative) VOI anal-ysis could potentially assist in focusing resources on issues and

Table 1

Workshop Objectives.

General workshop objectives:

Additionally develop the content of the NAS (2009) report on improving

the risk assessment process to develop a compendium of practical, prob-

lem-driven approaches for fit for purpose risk assessments, linking

methods with specific problem formulations (e.g., prioritization, screen-

ing, andin-depth assessment) for use by risk managers at a variety of lev-

els (e.g., states, regional managers, people in a variety of agencies, and in

the private sector).

Implement a multi-stakeholder approach to share information, ideas and

techniques in support of developing practical problem-driven risk assess-

ment methods compendium.

Specific workshop objectives:

Identify useful doseresponse techniques for specific issues, including

consideration of relevant data, characterization of assumptions, strengths

and limitations, and how the techniques address key considerations in

the doseresponse.

These techniques should appropriately reflect the relevant biology

(including the biology of thresholds), and mode of action information,

at a level of detail appropriate for the identified issue.

Provide methods to explicitly address human variability in cancer assess-

ment, and enhance the consideration of human variability in noncancer

assessment, including explicit consideration of underlying disease pro-

cesses, as appropriate for the relevant risk assessment context.

Identify methods for calculating the probability of response for noncancer

endpoints, as appropriate for the relevant risk assessment context. Develop a risk methods compendium that will serve as a resource for reg-

ulators and scientists on key considerations for applying selected dose

response techniques for various problem formulations, with suggested

techniques and resources.

236 M.E. Bette Meek et al. / Regulatory Toxicology and Pharmacology 66 (2013) 234240
http://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://sis.nlm.nih.gov/enviro/toxweblinks.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://-/?-http://-/?-http://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://sis.nlm.nih.gov/enviro/toxweblinks.htmlhttp://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.htmlhttp://www.allianceforrisk.org/Workshop/Framework/ProblemFormulation.html


4/7

analyses that are fit-for-purpose for the relevant decision context

as a basis for increasing efficiency. Since there are few published

examples of informal VOI analysis the panel recommended devel-

opment of a case study to explore this avenue.

The panel noted that tiered approaches, such as summarized in

Case Study #1 below (see the full framework for further details),

also contribute to efficiency, with assessment proceeding to the

next tier only if needed to inform the risk management decision.

Prior problem formulation is helpful, then, in deciding which of a

series of approaches incorporating increasingly biologically in-

formed and chemical specific knowledge would meaningfully ad-

dress the choice among potential risk management options,

taking into account the potential impact of the decision and time

and resource constraints.

Case Study Box #1

Tiered Approach to Screening Level Development

Key point: A tiered approach appropriate for the decision

context, namely to set screening levels for acute exposures

for as many air contaminants as possible, for air permit-

ting for emissions from new or modified facilities.

Method: For chemicals with limited toxicity data, interim

screening levels for acute exposures can be derived using

a tiered approach. This includes application of either

default screening levels or derivation of generic health-

based screening levels, depending on the availability of

toxicity information and time and resource constraints.

Tier I assessments are based on a Threshold of Regulation

approach, using a default value of 2 lg/m3 (TCEQ, 2012;

the original case study was based on the then-current

default of 1 lg/m3). Tier II assessments can be based on

either (1) categorizing the chemical based on its LC50 and

identifying an appropriate Threshold of Concern for that

chemical category; or (2) using a conservative estimate

of a NOAEL-to-LC50 ratio based on a distributional analysis

of such ratios for known chemicals to extrapolate from the

LC50 for the chemical of interest to a health-protective

limit (Grant et al., 2007). Tier III assessments are based

on a relative toxicity/potency approach to extrapolate

from related chemicals (TCEQ, 2012).

Conclusions: Tiered strategies provide flexible and efficient

approaches, taking into account the decision context, data

availability and resource constraints. Screening assess-

ments use conservative default approaches with the goal

of providing health-protective results where the assess-

ment is resource-limited and/or the screening indicates

that further action and analysis are not needed. An impor-

tant aspect of using such tiered approaches is that if a

potential problemis indicated in a lower tier, and if impor-

tant for risk management, then the assessment is advancedto a subsequent, more informed, tier.

Consideration of the nature of decisions to be made in the

problem formulation is also critical to selection of the appropri-

ate form of expression of hazard for health-related endpoints;

that is whether or not to calculate a dose corresponding to a

specified risk level, such as the daily dose that, over a lifetime,

would result in a 1 105 population risk. The calculation of

risk-specific doses for any suitable endpoint (not just cancer

incidence) was noted by NRC (2009) as desirable, and of high

importance for cost-benefit analyses. However, the panel recom-

mended that this might be most useful when the measured or

predicted exposure approaches recommended limits such as a

Reference Dose (RfD). In other words, the panel recommended

that calculating a risk-specific dose may be helpful if exposure

is near an RfD or similar value, but it may be of much less

interest if the measured or estimated exposure is well below

(or well above) a safe dose, or if the application considered in

problem formulation requires the estimation of an RfD or similar

value.

Case Study #2 summarizes an approach developed by Hattis

et al. (2002) that could be used potentially to predict risk levels

at any dose of interest, if risk management options are best in-

formed by such analyses (e.g., cost benefit considerations). This

approach also draws on non-chemical-specific broadly based

information sources (i.e., for other chemicals and pharmaceuti-

cals) to improve the traditional uncertainty factor framework,

but draws on more generic (i.e., non chemical specific) informa-

tion and as such is likely to have greater relative uncertainty

than, for example, Chemical Specific Adjustment Factors (CSAF)

(IPCS, 2005).

Case Study Box #2

Use of Hattis Straw Man Approach for DoseResponse

Evaluation

Key point: Chemical-specific doseresponse data and gen-

eric distributional data are used to estimate risk for non-

cancer endpoints, based on the assumption of a

distribution of individual thresholds of toxicity.

Method: The Straw Man model provides a distribution of

risks at specified doses, and for illustrative purposes,

defines the reference value as the 5th percentile value of

the dose corresponding to a 1 in 100,000 increase in risk

of mildly adverse effects (Hattis and Lynch, 2007). Other

percentiles and risk levels could be used. The model ini-

tially estimates an uncertainty distribution for the effec-

tive animal dose expected to yield a 50% response

(animal ED50). It then applies a series of uncertainty distri-

butions based on specific chemical data, or empirical data

for other compounds to transform the animal ED50 to a

human ED50 uncertainty distribution (e.g., subchronic-to-

chronic, database deficiency, animal-to-human.) Next, sep-

arate distributions based on specific chemical data, or

empirical data for other compounds of human pharmaco-

kinetic and pharmacodynamic variability specific to the

organ system affected and the severity of effect are used

to account for human variability. The uncertainty distribu-

tions are combined in a Monte Carlo simulation to predict

a distribution of doses corresponding to a target risk level.

MOA data could be used in determining the relevant set of

reference chemicals for deriving the distributions.

Conclusions: The Straw Man approach draws on non-chem-

ical-specific broadly based information sources (i.e., forother chemicals and pharmaceuticals) to improve the tra-

ditional uncertainty factor framework. Science Panel mem-

bers recommended that further refinement would be

based on, for example, chemical-, endpoint- or route-spe-

cific MOA.

3.2.2. Theme 2: Fit-for-purpose MOA analysis for dose response

Most doseresponse assessments are based on effect levels

identified in animal or human doseresponse studies designed

principally to identify defined hazard response levels,

(e.g., NOAEL, LOAEL, etc.), followed by application of default

extrapolation approaches (e.g., division by uncertainty factors

or linear extrapolation). Only rarely are MOA data used directly

http://-/?-


5/7

in the doseresponse analysis and risk characterization. Lack of

integration of data on MOA limits predictivity, since results then

cannot be easily extrapolated to other chemicals and/or condi-

tions. Early focus on patterns of effects, taking into account

MOA data on toxicokinetics and dynamics, can be informative

in considering appropriate approaches for extrapolations

addressing interspecies differences and human variability. Thus

MOA analysis can be applied throughout an assessment, inform-

ing many aspects flagged as important in NRC (2009), including

but not limited to the approach for low-dose extrapolation. For

example, MOA is critical in identifying likely susceptible popula-

tions and the potential range of variability in response in both

the general and susceptible populations. Work to develop a

database of accepted MOAs as noted by Carmichael et al.

(2011) will additionally facilitate incorporation of MOA in risk

assessment to address issues raised within the NRC Science

and Decisions report.

The NRC (2009) noted that there is a need for increased

understanding in the risk assessment community of the basis

for defaults and their underlying assumptions, so that asses-

sors can evaluate if and when other approaches may be

appropriate. The NRC committee stressed that clear criteria

and guidance should be available for judging whether, in spe-

cific cases, data are adequate to support inference in place of

default, and what level of evidence is needed to justify use of

agent-specific data instead of a default. An important contribu-

tion of the workshops and framework relates to increasing

familiarity in an accessible and structured format, with inter-

national and existing relevant US EPA guidance and illustrative

case studies, addressing specifically the objective above cited

by the NRC committee (see, for example IPCS, 2005, 2006;

U.S. EPA, 1994, 2011).

Depending on the needs identified in the problem formulation,

there is a range of approaches incorporating increasingly biologi-

cally informed and chemical specific knowledge on MOA, into

doseresponse assessment (see Fig. 1 and associated guidance).

These approaches as considered here reflect a continuum ofincreasing degrees of understanding of how (the) chemical(s) in-

duce(s) critical adverse effect(s) and the implications of that infor-

mation for understanding dose response. Thus, problem

formulation should include consideration of the appropriate extent

of analysis of MOA in any assessment, depending on the expected

importance to potential risk management decisions. Transparency

in problem formulation regarding the expected value of the MOA

analysis also serve to provide up front incentives for valuing col-

lection of information on MOA.

Case Study Box #3 illustrates the potential of information on

MOA to inform more predictive and accurate quantification of

doseresponse in humans, including potentially sensitive popula-

tions; see also the case study on chemical-induced ovarian ef-

fects (Kirman and Grant, 2012). Implications of the mode ofaction analysis for this case study are relevant to other com-

pounds that act through cholinesterase inhibition, but for which

fewer data are available; the case study also illustrates how

MOA can be helpful in addressing one of the more generic meth-

odological issues raised in the NRC report (i.e., addition to back-

ground). In relation to the latter, the NRC (2009) report stated

that effects of exposures that add to background processes

and background endogenous and exogenous exposures can lack

a threshold if a baseline level of dysfunction occurs without

the toxicant and the toxicant adds to or augments the back-

ground process, an idea initially raised by Crump et al.

(1976). However, Case Study #3 provides an example of using

a data-based, computational modeling approach to inform the

low-dose response in the face of background biological activity,

rather than being constrained to either of two alternative default

approaches.

Case Study Box #3

Quantitative Assessment of Sensitivity and Variability in

Humans

Key points: This study addresses MOA, human kinetic anddynamic variability, background response variability, and

interaction with background exposures and predisposing

disease processes.

Method: A source-to-outcome model provided a quantita-

tive description of the relationship. between the amount

of dietary residues of chlorpyrifos (CPF) in food, and the

impact of the exposures on inhibition of cholinesterase in

exposed populations (Price et al., 2011; Hinderliter et al.,

2011). Longitudinal dietary exposure was modeled and

combined with a physiologically based pharmacokinetic/

pharmacodynamic (PBPK/PD) model of response. The

resulting model quantitatively predicts changes in the

activity levels of cholinesterase that occur as a result of

an oral (i.e., dietary) dose of CPF. Variability in dose andresponse was addressed through Monte Carlo analysis.

Conclusions: The approach illustrates the use of chemical-

specific MOA data to characterize the doseresponse for

a chemical at environmentally relevant exposures, includ-

ing consideration of variations in background biological

activity. It allows for identification of a dose that does

not cause a biologically meaningful change in a critical

precursor MOA key event (cholinesterase inhibition), and

by inference, one where increases in apical effects are

not expected, even for individuals who are at increased

susceptibility due to other stressors. This approach is an

example of how MOA can inform a population threshold

as defined by NRC (Conceptual model 2 as described by

NRC, 2009, page 141).

3.2.3. Theme 3: Endogenous and background exposures

The Science Panel identified several issues relevant to the dose

response implications of exogenous exposures that had similar

impact as an endogenous process. In such scenarios, population

variability in endogenous levels and associated response needs to

be considered, but high variability alone is an inadequate basis

to conclude that exogenous exposure may be a trivial contributor

to risk. Of critical importance is the quantitative difference be-

tween the magnitude of response associated with endogenous

exposures compared to that induced by exogenous exposures. In

particular, a key issue in this comparison is how close the biologi-

cal response to endogenous levels is to an adverse effect level (i.e.,how much of a margin exists between the level of exposure for the

biological response and the level of exposure for the adverse ef-

fect). It would be helpful to understand, for example, the relative

contribution to total dose of both endogenous and exogenous

sources, and any kinetic differences between the two sources in

the formation or disposition of the chemical that may affect its

toxic potential. This sort of information can better inform selection

of dose response models appropriate for assessing risks associated

with low-dose exogenous exposures, as a basis for informing risk

management options (see also the case study on background/

endogenous damage at http://www.allianceforrisk.org/Workshop/

WS3/CaseStudies WS3.html).

As an example, DNA damage originating from some exogenous

chemical exposures may be identical to that resulting from

http://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://www.allianceforrisk.org/Workshop/WS3/CaseStudiesWS3.htmlhttp://-/?-http://-/?-


6/7

endogenous biological processes. For other chemicals, these two

sources of DNA damage may be differentiated, but only by complex

laboratory testing. For example, Swenberg et al. (2011) showed

that low-level exogenous chemical exposures result in DNA ad-

ducts, the nature of which are indistinguishable from those result-

ing from endogenous background, and those resulting from

exogenous exposure only exceed endogenous levels at higher

doses. These cases can provide key generic insights into the poten-

tial shape of the dose response curve and the risk associated with

low-level exposures to some DNA-reactive substances.

3.3. Additional methods needed

The Science Panel identified the following priorities for future

addition to illustrate and populate the framework:

Risk assessment for combined, multiple, exposures including

aggregate and cumulative exposures.

Case studies on value of information.

Case studies that illustrate an entire risk assessment, from prob-

lem formulation to conclusion.

Case studies that illustrate in vitro to in vivo extrapolation.

The Science Panel also recommended investigating the poten-

tial utility of linking the ARA methods Framework to case studies

and examples that illustrate methods that have not undergone

ARA Science Panel review.

4. Conclusions and future work

There is a wide range of decision contexts, necessitating differ-

ent approaches to doseresponse analysis. Incorporation of

increasingly biologically informed and chemical specific knowl-

edge in MOA-based approaches has the potential to additionally

refine estimates of hazard based on issues identified by the NRC

(2009) committee, including, for example, consideration of suscep-tible subgroups, though this must necessarily be balanced against

resource and time constraints for completion of assessments to in-

form risk management.

Sharing and communicating in structured fashion (the devel-

oped framework to consider potential contribution of these meth-

odologies in a problem formulation context) is considered

important as a basis to increase understanding of the availability

of various methodologies incorporating increasingly biologically

informed and chemical specific knowledge. The ARA methods

framework is expected to contribute to increasing transparency

in the basis for selection of fit-for-purpose approaches to dosere-

sponse assessment as well as to promote continuing advancements

in that practice, as a basis to increase predictivity and efficiency.

Such transparency is anticipated to aid in distinguishing and com-municating science judgments (i.e., weighting of options based on

transparent consideration of available scientific support) from

science policy, (i.e., selection of options motivated by the desire

for increased public health protection). Subsequent revisions of

the ARA framework are intended to better enable interested risk

assessment scientists to consider a broad range of dose response

methods in the context of recommendations made in the NAS re-

port and perhaps more importantly, may facilitate selection by risk

managers of an appropriate dose response method through a re-

view of problem formulations offered in different case studies.

The ARA is facilitating an ongoing process to expand the repos-

itory of fit-for-purpose methods for doseresponse analysis. New

methods and revisions will be incorporated to keep the ARA meth-

ods Framework and materials evergreen. This approach includes

Science Panel review of methods on a regular basis, allowing for

updates to additionally illustrate the ARA risk assessment methods

framework. Additional enhancements could also include expansion

of the framework from the current focus on doseresponse meth-

ods to include other components of risk assessment.

Conflict of interest

The authors declare that they have no conflicts of interest.

Funding sources

The workshop series described in this paper was supported by

over 50 sponsorsand collaborators, includinggovernment agencies,

industry groups, scientific societies, non-profit organizations/con-

sortia, and consulting groups. These groups are listed at http://

www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htm . Some

government employees received travel reimbursement for their

participation, but no other compensation.

Disclaimer

The information in this document has been funded in part by

the U.S. Environmental Protection Agency. It has been subjected

to review by the National Health and Environmental Effects Re-

search Laboratory and approved for publication. Approval does

not signify that the contents reflect the views of the Agency, nor

does mention of trade names or commercial products constitute

endorsement or recommendation for use. This paper reflects the

views of the authors and does not necessarily represent views or

policies of the employers of the non-Agency authors. These views

should also not be construed to represent those of science panel

members who are not listed as contributing authors. Nor should

these views be construed to represent those of the projects 55

sponsors.

Acknowledgments

The authors thank the almost 50 sponsors and collaborators

includinggovernment agencies, industrygroups,scientificsocieties,

non-profit organizations/consortia, and consulting groups (http://

www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htm ). The

workshop series built upon the consideration of the evolution of

risk assessment of the authors of Science and Decisions (NRC,

2009). Results of the workshop series are available at http://

www.allianceforrisk.org/ARA_Dose-Response.htm . The authors

wish to thank all of the workshop participants for their contribu-

tions, particularly the case study authors, science panel members

who declined or were otherwise unavailable for authorship andOliver Kroner for his work in organizing the workshops.

Fig. 1. Increasingly biologically-informed and chemical-specific approaches.

http://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://-/?-http://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Dose-Response.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://www.allianceforrisk.org/ARA_Dose-Response_Sponsors.htmhttp://-/?-http://-/?-


7/7

References

Carmichael, N., Bausen, M., et al., 2011. Using mode of action information to

improve regulatory decision-making: an ECETOC/ILSI RF/HESI workshop

overview. Crit. Rev. Toxicol. 41, 175186.

Crump, K.S., Hoel, D.G., et al., 1976. Fundamental carcinogenic processes and their

implications for low dose risk assessment. Cancer. Res. 36, 29732979.

Grant, R.L., Kadlubar, B.J., et al., 2007. Evaluation of acute inhalation toxicity for

chemicals with limited toxicity information. Regul. Toxicol. Pharmacol. 47,

261273.Hattis, D., Lynch, M.K., 2007. Empirically observed distributions of pharmacokinetic

and pharmacodynamic variabilityin humansimplications for the derivation of

single point component uncertainty factors providing equivalent protection as

existing RfDs. In: Lipscomb, J.C., Ohanian, E.V. (Eds.), Toxicokinetics in Risk

Assessment. Informa Healthcare USA, Inc., pp. 6993.

Hattis, D., Baird, S., Goble, R., 2002. A straw man proposal for a quantitative

definition of the RfD. Drug Chem. Toxicol. 25, 403436.

Hinderliter, P.M., Price, P.S., et al., 2011. Development of a source-to-outcome

model for dietary exposures to insecticide residues: an example using

chlorpyrifos. Regul. Toxicol. Pharmacol. 61, 8292.

IPCS (International Programme on Chemical Safety), 2005. Chemical-specific

adjustment factors for interspecies differences and human variability:

guidance document for use of data in dose/concentrationresponse

assessment. Available at http://whqlibdoc.who.int/publications/2005/

9241546786_eng.pdf.

IPCS (International Programme on Chemical Safety), 2006. IPCS framework for

analysing the relevance of a cancer mode of action for humans and case studies.

Available at http://www.who.int/ipcs/methods/harmonization/areas/

cancer_mode.pdf.IPCS (International Programme on Chemical Safety), 2007. Assessment of Combined

Exposures to Multiple Chemicals: Report of a WHO/IPCS International

Workshop on Aggregate/Cumulative Risk Assessment. World Health

Organization, Geneva, Switzerland.

Kirman, C.R., Grant, R.L., 2012. Quantitative human health risk assessment for 1,3-

butadiene based upon ovarian effects in rodents. Regul. Toxicol. Pharmacol. 62,

371384.

Lee, J.W., Devanarayan, V., et al., 2006. Fit-for-purpose method development and

validation for successful biomarker measurement. Pharm. Res. 23, 312328.

Meek, M.E., Armstrong, V.C., 2007. The assessment and management of industrial

chemicals in Canada. In: Van Leeuwen, K., Vermeire, T. (Eds.), Risk Assessment

of Chemicals. Kluwer Academic Publishers, pp. 591615.

Meek, M.E., Boobis, A.R., et al., 2011. Risk assessment of combined exposure to

multiple chemicals: a WHO/IPCS framework. Regul. Toxicol. Pharmacol. 60, S1

S14.

NRC (National Research Council), 1983. Risk Assessment in the Federal

Government: Managing the Process. National Academy Press, Washington, DC.

NRC (National Research Council), 2007. Toxicity Testing in the 21st Century: Avision and a Strategy. National Academy Press, Washington, DC.

NRC (National Research Council), 2009. Science and Decisions: Advancing Risk

Assessment (NAS Final Report). Available at http://www.nap.edu/

catalog.php?record_id=12209.

Price, P.S., Schnelle, K.D., et al., 2011. Application of a source-to-outcome model for

the assessment of health impacts from dietary exposures to insecticide

residues. Regul. Toxicol. Pharmacol. 61, 2331.

Swenberg, J.A., Lu, K., et al., 2011. Endogenous versus exogenous DNAadducts: their

role in carcinogenesis, epidemiology, and risk assessment. Toxicol. Sci. 120,

S130145.

TCEQ (Texas Commission on Environmental Quality), 2012. TCEQ Guidelines to

Develop Toxicity Factors (proposed). Chief Engineers Office. RG-442 (revised).

U.S. Epa (United States Environmental Protection Agency), 1994. Methods for

Derivation of Inhalation Reference Concentrations and Application of Inhalation

Dosimetry. US Environmental Protection Agency, Washington, DC, EPA/600/8-

90/066F.

U.S. EPA (United States Environmental Protection Agency), 2011. Guidance for

Applying Quantitative Data to Develop Data-Derived Extrapolation Factors for

Interspecies and Intraspecies Extrapolation (external review draft); EPA/100/J-

11/001. Available at: http://www.epa.gov/raf/DDEF/index.htm.

Yokota, F., Thompson, K.M., 2004. Value of information analysis in environmental

health risk management decisions: past, present and future. Risk Anal. 24, 635

650.

http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0010http://refhub.elsevier.com/S0273-2300(13)00050-0/h0010http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0025http://refhub.elsevier.com/S0273-2300(13)00050-0/h0025http://refhub.elsevier.com/S0273-2300(13)00050-0/h0025http://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://whqlibdoc.who.int/publications/2005/9241546786_eng.pdfhttp://whqlibdoc.who.int/publications/2005/9241546786_eng.pdfhttp://www.who.int/ipcs/methods/harmonization/areas/cancer_mode.pdfhttp://www.who.int/ipcs/methods/harmonization/areas/cancer_mode.pdfhttp://www.who.int/ipcs/methods/harmonization/areas/cancer_mode.pdfhttp://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0045http://refhub.elsevier.com/S0273-2300(13)00050-0/h0045http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0060http://refhub.elsevier.com/S0273-2300(13)00050-0/h0060http://refhub.elsevier.com/S0273-2300(13)00050-0/h0060http://refhub.elsevier.com/S0273-2300(13)00050-0/h0065http://refhub.elsevier.com/S0273-2300(13)00050-0/h0065http://www.nap.edu/catalog.php?record_id=12209http://www.nap.edu/catalog.php?record_id=12209http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://www.epa.gov/raf/DDEF/index.htmhttp://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://refhub.elsevier.com/S0273-2300(13)00050-0/h0080http://www.epa.gov/raf/DDEF/index.htmhttp://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0085http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0075http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://refhub.elsevier.com/S0273-2300(13)00050-0/h0070http://www.nap.edu/catalog.php?record_id=12209http://www.nap.edu/catalog.php?record_id=12209http://refhub.elsevier.com/S0273-2300(13)00050-0/h0065http://refhub.elsevier.com/S0273-2300(13)00050-0/h0065http://refhub.elsevier.com/S0273-2300(13)00050-0/h0060http://refhub.elsevier.com/S0273-2300(13)00050-0/h0060http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0055http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0050http://refhub.elsevier.com/S0273-2300(13)00050-0/h0045http://refhub.elsevier.com/S0273-2300(13)00050-0/h0045http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0040http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://refhub.elsevier.com/S0273-2300(13)00050-0/h0035http://www.who.int/ipcs/methods/harmonization/areas/cancer_mode.pdfhttp://www.who.int/ipcs/methods/harmonization/areas/cancer_mode.pdfhttp://whqlibdoc.who.int/publications/2005/9241546786_eng.pdfhttp://whqlibdoc.who.int/publications/2005/9241546786_eng.pdfhttp://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://refhub.elsevier.com/S0273-2300(13)00050-0/h0030http://refhub.elsevier.com/S0273-2300(13)00050-0/h0025http://refhub.elsevier.com/S0273-2300(13)00050-0/h0025http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0020http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0015http://refhub.elsevier.com/S0273-2300(13)00050-0/h0010http://refhub.elsevier.com/S0273-2300(13)00050-0/h0010http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005http://refhub.elsevier.com/S0273-2300(13)00050-0/h0005

bette meek-2013-a framework for fit

Documents