Use of Microbial Risk Assessment in Decision-MakingDavid Vose Consultancy24400 Les LchesDordogneFrancewww.risk-modelling.com

EmailDavid Vose's secretary David VoseSlide show on: www.risk-modelling.com/firstmicrobial.htm

IntroductionApplying CODEX guidelines in realityDifficultiesOther ways of thinkingExperience with microbial modellingSome survey resultsThe Dutch experienceSome US experienceModelling challengesComparison of some complete modelsReviewing a model in context

Codex Alimentarius CommissionFAO/WHO (1995)Microbial risk assessment is a scientifically-based process consisting of four steps:

Hazard Identification The identification of known or potential health affects associated with a particular agent;Exposure Assessment The qualitative and/or quantitative evaluation of the degree of intake likely to occur;Hazard Characterization The quantitative and/or qualitative evaluation of the nature of the adverse effects associated with biological, chemical and physical agents that may be present in food For biological agents a dose-response assessment should be performed if the data is available;Risk Characterization Integration of Hazard Identification, Hazard Characterization and Exposure Assessment into an estimation of the adverse effects likely to occur in a given population, including attendant uncertainties.

OIE experienceOIE produced guidelines for animal import risk assessments (for the management of disease spread)Now in its second editionGuidelines were offered as a way to help member (including developing) countries understand how to perform a r.a.First Ed. guidelines were used too literally, both by analysts and lawyers, and found to be often impractical or irrelevant to the risk questionLesson: keep guidelines non-specific, encourage understanding rather than prescribing a formulaic approachPopular interpretation of CODEX guidelines suffer similarly

(1) Risk analysis uses observations about what we know to make predictions about what we dont know. Risk analysis is a fundamentally science-based process that strives to reflect the realities of Nature in order to provide useful information for decisions about managing risks. Risk analysis seeks to inform, not to dictate, the complex and difficult choices among possible measures to mitigate risks...Society for Risk AnalysisPrinciples for Risk Analysis

(2) Risk analysis seeks to integrate knowledge about the fundamental physical, biological, social, cultural, and economic processes that determine human, environmental, and technological responses to a diverse set of circumstances. Because decisions about risks are usually needed when knowledge is incomplete, risk analysts rely on informed judgment and on models reflecting plausible interpretations of the realities of Nature. We do this with a commitment to assess and disclose the basis of our judgments and the uncertainties in our knowledge. Society for Risk AnalysisPrinciples for Risk Analysis

Current modellingMicrobial QRA is a developing scienceWere making a lot of progress, but it is still in infancyMostly producing farm-to-forkModels the whole system but very poorlyNot designed to model any decision question wellOften relies on poor data, surrogates, and guessesAlmost never is a decision question posed beforehandAssessors have probably over-sold QRAs usefulnessManagers have expected too much

F2F Achilles HeelsVery little data available, system being modelled is hugely complex!Uncertainty, variability, inter-individual variablilityTake too long to complete, too easy to make mistakesF2F considers only pathogen on the food sourceE.g. not E.coli produced during life of animal, appearing in water, vegetables, farmers exposurePredictive microbiology still unreliableBroth data doesnt translate well to food (usually overestimate, but some data Tamplin, USDA shows lag period can be shorter, e.g. E.coli in ground beef, Listeria in processed hams)Models often not based on physical/biological ideas, so we dont learnAttenuation may not be death, and ignores reactivation of bacteriaD-R models inadequateDont describe variability observedP(ill|dose, infected) = P(ill|infected)?Feeding trial data dont match epi data can hugely underestimate the riskLittle cost-benefit analysis effort madeIncluding actions affecting several risk issuesRequires enormous resources impractical for many countries

Dutch observations on past QRAHavelaar, Jansen (2002)The lessons learnt from risk analysis experiences:Risk management has not always been an integral part of risk analysis so far;Risk managers should be trained to understand risk assessment, and risk assessors should be trained to explain their work;Available data are often of limited use for risk assessment and communication of data needs between risk assessors, food scientists and risk managers is a critical issue;The risk manager questions usually require rapid results, whereas (farm-to-fork) risk assessment projects require several years to complete. Solving this conflict requires open communication;Uncertainty is often large.

Our surveyInternet based, voluntary participation, 39 valid responses

Chart1

0.10256410260.23076923080.51282051280.05128205130.1025641026

0.15384615380.38461538460.33333333330.05128205130.0769230769

0.10256410260.23076923080.48717948720.12820512820.0512820513

0.17948717950.23076923080.33333333330.12820512820.1282051282

Very much

Much

Some

Little

Very little

Do you consider that risk analysis has brought about the following improvement to government decision-making?

Sheet1

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950

Dr. Christan StogbauerProff. Barry T. HartChristie DrewArie HavelaarEnda CumminsHelene BergeronPaul M. SchlosserChris HawkinsInge Dorthe HansenDavid YapSam BeckettClark GormleyRodolfo AvilaEdmund PeelerStephen L BrownVicki BierIvan TaylorRichardStefan FabianssonDale CooperKeith HayesCynthia FullerJoyce Van DonkersgoedDave EvansHector YepMark A. TurneoJames S. SmithAlexander LeinJack WattsRobin ZwartDavid PybusBruce NcNabLee-Ann JaykusTine HaldSanja SeparovicStuart MacDiarmidScott McEvenPaul VanderlindePhilip Berger

1

QuestionCan you give examples of proven benefits that have arisen from commissioning assessment, and acting approproately on its results?

AnswerNoxxxxxxxxNo, we carry out our own risk analysesx

Yes. Please explainregulatory requirements, giving incentive for better risk management by assessing and evaluating risk control environment.A recent ecological risk assessment for a paper mill discharging effuent to an estuary in Tasmania (Australia) resulted in the company being able to convince the state EPA that they were not causing environmental harm to the estuary..xYes, in the past, we used RA without calling it that way, but we find that the more structured and standardized gives credibility to our risk-decisions and helps supporting the regulations..Yes, Application of quarantine policy has been done on the basis of quantitative RAxOur import protocols are based on international standards or on risk assessments - indeed, all WTO Member countries must base any trade-restrictive measures on those provided in the relevant international standards (OIE and IPPC) or on scientific assessmeUse of hypochlorite instead of liquefied chlorine gas for chlorination of water, due to the safety risk for the public from a release of toxic chlorine gas.Helps to identify important sources of uncertainty. Good alternative when conservative estimations indicate a significant riskwork done in Australia on the risks associated with imported fish products failed to demonstrate a case for restricting the importation of wild caught salmon, and hence, after a ruling by the WTO, imports were allowed.Of course, the answer depends on what constitutes a "benefit". I have typically carried out risk assessments on behalf of "regulated" parties in recent years. From their point of view, the benefit is relief from excessive or unnecessI know of a number of situations in which nuclear power risk asessments have resulted in the identification and implementation of significant risk reductions at the power plants in question, often at extremely modest cost. References available on requestIt has helped the Canadian Forces prepare for multiple contingencies in military operationsIt reduces uncertainity and increases the confidence of decision makers in committing projectsQualitative BSE risk assessment fed into decision making process (fertilser, effluent)We reviewed a project risk assessment recently that showed benefits of USD 350,000 within 48 hours of completion, and further non-quantified benefits over the next few months as strategies were adjusted to deal with major indentified risks.Yes - but a tentative one - in ecological risk assessment the benefits are usually impossible to bank and therefore do not show up on the accounts - tangible benefits usually occur where the risk assessment allevies industry of regulation (eg the ballastRisk support for cost-effective re-development of impacted landchanges in bluetongue regulations in 1995 to allow entry of US feeder cattle during the nonvector seasonWe continue to hear and learn that clients who undertake our "project risk analysis" are able to mitigate cost and schedule during project construction and enhancement. Good risk analysis ensures that projects are on schedule and on budget..The improvement of regulations being promulgated by the US Department of Agriculture demonstrates the effectiveness of Risk Assessment as a support structure for decision-making. Within both APHIS and FSIS, regulations have been more focused and cost eff.More safe projects of new installations and reconstruction old, accommodation of the personnel and workers onsite, a choice of priorities and development of right actions for increase of safety.determination of the costrange of RD&D programmes and the consequent decission whether or not to continue them.Yes, risk of Bluetongue disease in US cattle imports to Canada, resulted in change in import regs (Cnd Fd Insp Agncy has done many import RAs to support or change import policies), 2) risk of clenbuterol residue in veal calf meat, supported placment ofI believe that in the U.S., the major benefit to food safety risk assessment has been largely related to the process of gathering all available information and organizing it in a comprehensive manner. This has facilitated the identification of critical aThe Danish Zoonoisis Centre uses a quantitative risk-based approach to estimate the contribution of the different animal-food sources to human salmonellosis. The results are published annually, and the authority react to these by for instance establishing.A risk assessment conducted looking at E. coli O157:H7 in Australian beef highlighted the problems with chilling carcases over weekends (a practice that is common in Australia but not in the US). As a result of these findings the industry was modified itsRisk asssessment input data typically are inadequate for decision making. Risk assessment is best used to guide decisions making by showing what parameters are insufficiently known so as to make unbiased decisions.

2* One from Canada and one from Ireland, RA not used there yet - "very little"

QuestionDo you consider that risk analysis has brought about the following improvement to government decision-making?Percentages

Very muchMuchSomeLittleVery little *Check:Very muchMuchSomeLittleVery littlecheck

AnswerFairnessSomeSomeSomeSomeSomeVery muchSomeVery muchSomeSomeVery muchMuchSomeMuchSomeSomeSomeSomeSomeVery littleSomeSomeSomeVery littleSomeMuchVery littleMuchSomeMuchMuchSomeMuchLittleVery muchMuchMuchLittleVery little4920243910%23%51%5%10%100%

RationalitySomeMuchSomeSomeVery littleVery muchMuchVery muchMuchSomeVery muchSomeSomeVery muchMuchSomeVery muchSomeMuchSomeMuchSomeVery littleVery littleMuchSomeSomeVery muchMuchMuchMuchSomeMuchMuchMuchMuchMuchLittleLittle61513233915%38%33%5%8%100%

ConsistencyMuchSomeSomeSomeVery littleMuchSomeVery muchSomeLittleVery muchSomeSomeMuchSomeLittleVery muchSomeSomeSomeMuchSomeLittleVery littleSomeSomeSomeVery muchMuchSomeMuchSomeSomeSomeMuchMuchMuchLittleLittle4919523910%23%49%13%5%100%

TransparencyVery muchMuchSomeSomeVery littleSomeLittleVery muchMuchLittleVery muchSomeVery littleMuchSomeSomeVery muchLittleMuchSomeSomeSomeVery littleVery littleSomeVery muchSomeMuchMuchLittleMuchSomeSomeLittleMuchVery muchVery muchMuchVery little7913553918%23%33%13%13%100%

Please explainThis is a difficult question because I'm not sure what your frome of reference is - imporvements compared to what or when? In my experience different agencies and circumstances have different experiences for all these "improvements". Also, some people mig.Risk analysis is only new in Ireland, can't say it has played any major part yet?If we compare to the time before RA, it is much better, less arbitrary, but there are still many problems and difficulties to overcome.More fair than without, but discrepancies remain; approach is fairly rational; but often case-specific; not very transparent to the general publicProcess is transparent, science based, & risk management undertaken on basis of risksI am not too sure of the answers as I am not close to this decition process.Believe that the structured process of risk assessment, as described in the relevant international standards, creates a format for decision making that greatly enhances each of the points above.In the past the real reason behind restrictions on the import of animal products has been to prevent free trade. The use of risk analysis ensures that defensible, transparent arguments have to be presented in support of trade restrictions.Risk analysis potentially can insert rationality into the search for an "efficient" tradeoff of the costs and benefits of regulation--the balancing of net benefits against net costs. It can also help identify who wins and loses with a change inThere is still enormous inconsistency between agencies, and in the regulation of different types of risks. However, I believe that overall, risk analysis has improved the process of government decision-making and regulation of risks.We have combined Multi-Criteria Decision Making with Risk Assessment to consider Risk along with Benefits and Costs in a transparent manner..Should do but probably still a way offMost Government procurement is based on processes designed to ensure fairness and transparency explicitly, with rationality and consistency as useful byproducts. Risk analysis supports these objectives, but it is not a dominant (or even necessary) featurMuch of the government risk assessments we deal with (eg import risk analysis) are qualitative - the decisions are not entirely transparent because of linguistic uncertainty - the process is, however, consistent and relatively rationale.Risk assessment is not really a science or art, but a tool, dependent on the assumptions and approaches used. Since there has been many changes to US/state risk assessment approaches over the years (I've been practicing for over 12 years in this field, tour federal government (CFIA) does the risk assessments in isolation of industry in spite of industrys request to participate.In Canada, governments rarely use or apply quantitative risk analysis. Other than in the area of environmental risk analysis to restate environmental compliance criteria and with energy related applications/approvals, only qualitative risk assessment is.The final decisions have changed some, but the most important result is the ability for others to clearly understand the trade-offs inherent in the decision..Any reasonable constructed process will enhance these qualities. More study of the process, such as this survey, can improve it further.In principle comparison of decissions on risks is fair and rational, however often the starting points are not the same (hence no consistency) or not known at all (no transparency)The risk based approach provides fewer opportunities for obfuscationI think we are moving in the right direction. I think it will imporve more with time, but will never be perfect. There are many forces that tend to push us "off track" (e.g. crises of the day, conflicting objectives), but sound, systematic RASince food safety risk assessment is still very much in developmental phases, we do not yet know its final value to US agencies. It is clear that there remain issues related to consistency as there are many different ways to approach a risk assessment.I had a very hard time answering the above, because it varies from one risk assessment to the other. Generally, when complex "stable-to-table" analysis are performed the improvement or benefit is difficult to see. However for smaller or more speTo make use of huge amount collected dataThe main advantage of the import risk analyses we carry out is that they make the decision making transparent. Objectivity is a slippery commodity, especially where data are sparse, but transparency should always be attainable.At the moment there are no apparent linkages between risk assessment and risk management. Risk assessments tend to be carried out in isolation simply as an exercise rather than as a tool for use by risk managers. There is a great need for improving the liThe more transparent risk assessment models are often too simplistic to be useful. Where bias is recognized, rational and consistent decision making occurs more oftenbut fairness required unbiased decision making.

3

QuestionDo the sponsors of risk asessment in general get good value for money?NumberPercentage

AnswerYesxxxxxxxxxxxxxxxxxxxxxx22Yes58%

No, mostly because0No, mostly because0%

They take too long to completexxxxx5They take too long to complete13%

They cost too much moneyxxxxxxx7They cost too much money18%

Their focus didn't address the problemxxxx4Their focus didn't address the problem11%

Other, please specifyI don't think I'm qualified to say,There is often a conflict between necessary resources for a good RA and available resources. So what is best value for money?They take too long to complete.Import Risk assessments can take longer to complete than might be thought at the outset.Being a practitioner, it would be difficult for me to answer no generally. For my regulated-party clients, the savings from less stringent regulation is often much greater than the cost. On the other hand, I'm reasonably certain that some risk assessmen.There are some issues with using qualitative data that have not been resolved..Results often too complexWe are consultants, and they pay us for risk assessment, so I have a slight bias here!!.Depends on the assessor. I've seen some really bad assessments performed.no input from industry who is affected by these risk assessments - thus, their tax dollars not being using fairly.The process is still evolving. Most of the time, the individuals ASKING for the risk assessment are not clear on what they want, nor do they understand what they will get. In addition, the perception that the risk assessment will "bind" the deAs a provider of risk assessment, my company survives because we provide "sponsors" with good value for the money, but then we often undercut our competition by providing a quality product for less than half the cost.RA must be only a part of improvements in whole management systemThe results are not validated.Depence on the knowledge of the sponsors ... some people just ask for anything and will also accept anything for any priceDo not commission external risk analysisdepends on point of view, there is much room to improve efficiency, should strive to optimise marginal return, must learn to walk before runI am not sure that the US federal agencies know what to do with their risk assessments after they are completed.Again, it depends on the specific analysis, but generally I believe, that the risk assessments commissioned by a government is not good value for money. They may help scientists to focus their future research, but the results are often too uncertain to re..All of the above to some degree but the main reason is that there is no communications between risk managers and risk assessors, therefore what is delivered is not what was expected. While the two should be separate there must be clear communication betweYes and no..if they risk assessment is dynamic (includes immune status and secondary spread) then they get good value. If they risk assessment is static, then the focus didn't address the problem adequately.38100%

4

QuestionIn general, what is your assessment of the quality of risk assessment?question too vague - where when? A better question might be what is your assessment of the quality of risk assessments that you have first hand knowledge of? And add a n/a categoryNot relevant to the way we work. We carry out our own risk analyses on behalf of government.NumberPercentage

AnswerExcellent.xxx3Excellent8%

Quite goodxxxxxxxxxxxxxx14Quite good38%

Fairxxxxxxxxxxxxxxxx16Fair43%

Poorxxxx4Poor11%

Unacceptable.0Unacceptable0%

537100%

QuestionWhat factors jeopardise the value of an assessment?NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1Insufficient human resources to complete the assessment323313334342342234342444432432322222311214100373%32%38%27%0%100%

2Insufficient time to comlplete the assessment432312334242341234422433422322421431412101003611%33%28%28%0%100%

3Insufficient data to support the risk assessment121112243221241114331332211332122221114137303738%35%19%8%0%100%

4Insufficient in-house expertise in the ares2242433434222424242334544324223342420148131360%39%22%36%3%100%

5Insufficient general scientific knowledge of the area2242442444224241443323244422232342512145141366%39%14%39%3%100%

6Poor inter-disciplinary or inter-agency/department/organisation collaboration22433342422334334331222434222144423131211100368%33%31%28%0%100%

7Other, please specify.Poor communication to stakeholders and the public.Arbitrary constraints such as default assumptionsInsufficient knowledge of risk analysis methods on the part of the analysts.Lack of senior management support and interest.no industry input from being of process to the end (poor risk communication skills), not transparent process.Disconnect with target audience..

Comments:in yellowLack of expertise and lack of data the two big ones..Risk assessments are not often subjected to any attempts at confirmation even in those rare cases in which it is scientifically possible to do so. So we don't know whether the field as a whole has much quality. It is certainly true that many individualI am not really in a position to know the answers to the above questions, since I am familiar with only a narrow spectrum of risk analyses, and most of those were not done by government agencies.Because of limited time we need to utilize expert judgement qualitative assessments.Risk assessor striving for accuracy at the expense of management understanding and simplicityA risk assessment can be designed around most limitations and achieve a useful (if not 'optimal') outcome, but senior support is a vital ingredient irrespective of other constraints.Government risk assessments that we review often show evidence of poor in-house expertise in RA methods and techniques outside of the usual qualitative approachesCompletely depends on the person doing the assessment..The biggest risk in a risk assessment is failure to properly frame the problem and fully understand the assumptions around the problem..Lack of understanding on the part of the individuals asking for and using the assessment.Risk assessment sucess is measured by the ability of the target audience to understand the content and conclusions..Communication of how to use the results, i.e., the link between risk assessment and risk management.uncertainty in given data, simply because they are totally unknown.It will take time for a critical mass of technicians and managers to change their mind set, acquire skills, become proficient, gain comfort and confidence, accept costs and accept findings that what has been done for years may have been sub-otimal (or.inadequate static risk assessment models

6

QuestionWhat factors jeopardise the approporiate implementation of a risk management decision?NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1Politics4223321223322231244241221223322224125197503614%53%19%14%0%100%

2Other issues take precedence333123343122242333321432333222334233111740359%31%49%11%0%100%

3Legal restrictions5433444253443243354443544424433425330411165360%11%31%44%14%100%

4Insufficient resources to implement the action232323344243343233433233344322434340817100350%23%49%29%0%100%

5Risk assessment too complicated233344444434342214442534242232224341108151353%29%23%43%3%100%

6Risk assessment not accepted as valid344344434444342353242423342232224220119141350%31%26%40%3%100%

7Insufficient time333423344344342333441424434422424142611160356%17%31%46%0%100%

8Other:Lack of awareness of available tools and expertise.Lack of senior management support and interest.A risk assessment that is not tailored to the needs of the project, but is performed with an academic slant.insufficient participation of stakeholders.Lack of understanding by decision-maker of risk and risk conceptsInappropriate application of risk assessment methodologies to site-specific issues..Usually, risk assessment does not address the concerns of manager

Comments:Many risk analysis projects are commissioned as an organisation wants to be SEEN to be doing something, if the results don't suit it gets shelved..Not too sure.A risk assessment can only be part of the story about what should be done; what constitutes "appropriate implementation" will always remain in the eye of the beholder whether or not the implementation matches the assessment.Same as above; I am not sure I am in a good position to comment in generalThere is a growing awareness of the need but lack of awareness of the capabilities of tools and sources of information..There is still a lack of understanding between assessors and managers and the communication of factsSimilar comments to Item 5In reality politics and economics seem to take precedance, particularly in qualitative assessments where low or negiligible are open to interpretation..Laziness is a factor too.....until things change or go wrong.Risk assessment is a tool of risk managers. The value of this tool in risk management decisions is dependent on a variety of intrinsic and extrinsic factors..What is appropriate? Is the question intended to include appropriate implementation of a bad decision, i.e., no implementation..The changes implied can be big. Any change is difficult. Expensive change that implies one was "doing it wrong before" is very difficult intertia to overcome..Risk assessment is a rather new discipline in food safety, and I believe many risk managers are uncomfortable about using the results. Especially, if it suddenly contradict how they would have reacted in case the risk assessment was not performed..Again on of the main issues is that risk assessors often do not know what management options are best suited to the situation and without proper guidance by risk managers will not address there issues in the risk assessment.risk assessment not adequate (static model)

7

QuestionAre the risk assessors made sufficiently aware of the purpose of the assessment?NumberPercentage

AnswerYesxxxxxxxxxYes, I have always felt that I knew the purpose; I have sometimes run into clueless risk assessorsIt all depends. In many cases, they are. However, in others, a company or government agency may commission a risk assessment as a "whitewash" of a particular topic, or to give "protective coloration" for a decision that has already bxxxYes. We tend to use AS/NZS 4360 as our process guide. Establishing the context is a critical component of the risk management process.xxxxxxxxxxx2359%

No, please explain:No, not always.No, usually not : it is often unclear for the managers themselves..The regulator, i.e. those who order the assessment, usually do not realy understand what they need.If they don't understand what they are doing, they shouldn't be doing it!.Yes & No, because sometimes the clients cannot describe or relate to the problem to be risked..Selodm, mainly because the decision maker may not completely know herself, or because political factors lead the decion maker to "hide the ball".Usually there is poor communication with decision makersDepends, mostly yes ... not always however.Yes and no, not always..... it is extremely important in any group effort that everyone understands adn buys into the same objectiveMany of the US risk assessments do not have a clear regulatory endpoint, nor are they focused enough. This leads to completed documents for which there is limited use with respect to public policy.Often, it is up to the risk assessors to decide upon the outcome, i.e. define the question(s) that needs answering.No, often the politics underlying the need for conducting a risk assessment are not specified and therefore the risk assessor does not always construct the risk assessment appropriately.

8

QuestionHave you been involved in a risk assessment that has reached a 'More data are required to produce a meaningful assessment' impasse?NumberPercentage

AnswerNoxxxxxxxxxxxxx1333%

Yes, In which casexxxxxxxxxxxxxxxxxxxxxxxxxx2667%100%

Do you think that the result could have been predicted beforehand?xxxxxx615%

Was the risk assessment still performed because:xxxxxxxxxxxxxx1436%

The purpose of the risk assessments was to point out areas where more information is needed?xxxxxxxx821%

One needed to demonstrate that a risk assessment had been attempted?00%

International guidelines (e.g. Codex, OIE) required it to be completed?00%

There is national or international legislation stating that risk assessment should be done before some decision could be made?xxxx410%

It was a specific request from the relevant authority?xxxx410%

It was a specific request from the government (i.e. a political decision)?xxx38%

Corporate rules required that the risk assessment be done anyway?x13%

The risk assessors were not asked for their opinion beforehand (or with held it and did the risk assessment knowing that 'more data is needed' would be an acceptable result?00%

Everybody else is talking about and performing risk assessments?00%

Other reasons? Please specify:General comment: the question is: When can you produce meaningful results? Given scientific standards, usually do not provide of enough data. However, in a situation with sparse data there is still potential for value-added simply by showing people that.Any of the above reasons can be involved. Even a severely limited assessment can be informative, usually more so than no assessment at allThere is always the possibility of producing a more meaningful assessment with more data. However, in my experience, a good risk assessment can still help to clarify the situation, by indicating exactly what IS known about the question of interest, whatIt was determined that some analysis was better than none..It was important to develop a structure (model) that could be discussed with industry to show what if scenarios.Unfortunately, risk assessments often are considered after all data are collected, with no prior input from the risk assessor. The risk assessor needs to be part of the project; this is not an effort that can be done in a "black box".An evaluation of data quality and useability will usually identify situations of "impasse." In other cases, a risk assessment may be required under applicable regulation, as a tool for determining risk-based remedial goals, or in a phased appro.There are multiple answers to this question. The format is not allowing them!Although some figures are unknown, relevant (sub)conclusions can be drawn.Essentially, versions of all of the above can happen (and likley have) in various organizationsThe need for "additional data" is fairly universal to food safety risk assessment. It should not be an excuse for not completing the assessment. US risk assessors seem resistant to using Bayesian techniques (and expert elicitation) to get arou.

9Several of the above in some circumstances, and most of them across a range of assessments. However, I have not been involved, either as an assessor or as an auditor of an assessment, where the assessors were not asked for their opinion (or withheld it),

QuestionAre you aware of risk assessments being commissioned where it was obvious from the beginning that the results were not going to be used, or a meaningful analysis could not be achieved?NumberPercentage

AnswerNoxxxxxxxxxxxxxxxxxxxxxxxx2462%

Yes. In which case, was the risk assessment still performed because:xxxxxxxxxxxx1231%

One needed to demonstrate that a risk assessment had been attemptedxx25%

International guidelines (e.g. Codex, OIE) required it to be completed?00%

There is national or international legislation stating that risk assessment should be done before some decision could be made?x13%

It was a specific request from the relevant authority?xxxx410%

It was a specific request from the government (i.e. a political decision)?xx25%

Corporate rules required that the risk assessment be done anyway?xx25%

The risk assessors were not asked for their opinion beforehand (or with held it and did the risk assessment knowing that 'more data is needed' would be an acceptable result?x13%

Everybody else is talking about and performing risk assessments?x13%

Other reasons? Please specify:I'm sure this does happen sometimes, for a variety of reasons, but do not believe that I have been in this situation myself.Sometimes we have done analysis to demonstrate the capability for future operations even though the analysis is not expected to impact the current operation..We try not to waste our clients' money! As consultants, we think it is not worth it in the long run (and has few benefits in the short run either).A cannot think of a specific case that I have personnaly been involved in where nothing was used or nothing was learned, but I would not be surprised if it has happened..required as part of benefit/cost analysis

10

QuestionDo decision makers:NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1) Understand and use the results of a risk assessment3223222222343342342323344424233432244501512101380%39%32%26%3%100%

2) Encourage involvement of stakeholders and external expertise?223323333223333234224544323343332313251111952383%29%50%13%5%100%

3) Encourage assessors to explain what may and may not be possible to achieve?52432343424444322422344244444434431325198182383%24%21%47%5%100%

4) Put a lot of emphasis on receiving comments at the planning stage of a risk assessment4333434334434223223533543334344324350517113360%14%47%31%8%100%

5) Involve risk assessors in planning how to communicate the risk assessment results?22334223243434324344434442444344314451710181373%19%27%49%3%100%

6) Involve stakeholders after completion of the assessment of the appropriate risk management action to take?53223232143434233323444433333232322251111672373%30%43%19%5%100%

7) Expect too much of a risk assessment?342223334422324323431112343231332222241413603711%38%35%16%0%100%

8) Expect too little of a risk assessment?33444544432444234343455232343233254540611155370%16%30%41%14%100%

9) Assign sufficient resources and time to complete the risk assessment?423344321342334232224144234234334434221012130375%27%32%35%0%100%

10) Encourage assessors to suggest alternative approaches to the assessment?4444244444443342424444442343434442425065251370%16%14%68%3%100%

11) Require the assessment to be simplified for ease of understanding, at the expense of technical accuracy?144244444432342213334333134442543235146111423711%16%30%38%5%100%

12) Make the results available only if it suits their purposes?25442343 / 4532333132343232344442454254412810124366%22%28%33%11%100%

13) Encourage assessors to produce an assessment to support a predetermined position?35252234554443422243423235443444425531198137383%24%21%34%18%100%

14) Involve risk assessors in the decision-making?23231322234243424443354433433342123413913111378%24%35%30%3%100%

15) Allow/expect the risk assessors to make the decision?4443343423434452454425344244335214435159175373%14%24%46%14%100%

Other comments on decision-makers' behaviour:There is not yet an understanding on what should be the threshold (quatitative) of an acceptable risk.Decision makers' behaviour is influenced by interim discussion and allusion to realistic expectations or RADo not dare to start a risk assessment, afraid of the publicity of the outcome.The distinction between risk assessors and decision makers is often blurred in the the context of import risk analysis.They exhibit a very wide distribution of behaviorsIn my experience, I have been pleasantly surprised at how rarely decision makers have encouraged assessors to produce assessments to support predetermined positions. In my view, the more serious problems are decision makers who do not understand the resuBecause of subjectivity of analysis decision makers will often discount results that do not satisfy there needs..Still not much drive to use risk assessment in many areas apart from import risk assessments to support WTO cases.don't have stakeholder involvement in the beginning of the risk analysis processMost decisions are based on "it seemed like a really good idea at the time".It is often a good business strategy for a "sponsor" to separate risk assessment and management activities since these two activities can have conflicting business goals..No real understanding of risk assessment and how it can and should be used. Expect a risk assessment to tell them the risk management options rather than using the risk assessment to evaluate alternative options.

11

QuestionIs room left for decision-makers to change or reverse the decision should new information appear that contradicts the risk assessment assumptions?NumberPercentage

AnswerNoxxxxxxNo, The monitor-review - reassess cycle is rarely implemented in my experience - this is one of the biggest failings of ecological risk assessment in my area.xxxxx1128%

Yes. Please explain how this is achieved:The fact new info contradicts risk assumptions makes people dubious about the risk assessment and its malidity to start with!! Can this not happen again, and again!!Yes, but managers are prudent. It is suggested in our framework for risk analysis but we have not experienced it yet.Risk assessments are periodically reviewed and revisedPremises on which RA is done are specified, and the assessment is clearly stated to be a dynamic entity which will require updating as events change - e.g. as the prevalence of a disease in an exporting state changes, so the risk to the importing state ch.Yes, the decision review board is reconvened, and outcome communicated to decision maker(s)Assessments are revised and updated systematically to take account of chengs in assumptions / inputs.the risk analysis clears states the data used, hence as more data becomes available it is possible to change positionsNot always, but sometimes a decision can be changed; in particular, new information on toxicity may shift a regulation toward more stringency, although the opposite is less likelyIn fact, there is one well-known and published example where decision-makers reversed the recommendation of a risk analysis, even WITHOUT new information that contradicts the risk assessment assumptions, and this is not generally considered to represent aIn general the subject matter experts are not considered to be unbiased and therefore their opinions can be discounted.xRisk assessment is just one input to decisions. Like any tool, it only support decisions, but managers are paid to manage, and that usually involves factors other than those included in simple tools..In our process, we always use a reality check against a calculated risk-based outcome. Head checks rule. This type of outcome relates to not fully understanding and/or quantifying the type and number of risks that will or may have impact on an outcome.A good risk assessment merely lays out the trade-offs. Other factors, including culture, values, politics and economics must be considered by the decision maker in maing the final decision.Regulatory relief.It must be described in the comnay policy.As long decissions still can be changed when new information is received (and considered)By identifying more information necessary to change the uninformed decision.Significnaly new information can lead to reassessment, new result, change in policy, BUT it all takes time..... changing regulations is often a long processHave not yet seen much decision making linked to risk assessment, at least in the food safety arena.Technically, yes, decision makers are often the authorities and can as such change the legislation. However, in practice it will be very hard to change a decision when first it has been made. I think most decision makers will be afraid to look untrustwort.Yes. A number of mechanisms exist for review of risk analyses. Indeed, it is explicit in our policies.Yes. By making it clear that decisions may be revised in light of new informationYes. Decisions can be reviewed but this usually takes more than 12-months. Therefore you are suck with decisions for more than 12-months, even bad decisionsYes. Decision making is easy developing a new or varying the rationale for a decision is very difficult and is usually inadequate

12

QuestionWhat has been the public perception (if appropriate) to the whole risk analysis process on a specific issue (or issues) you know about?

AnswerIssue 1:OpRisk quantification, too early to tell about the general perception.Ecol risk assessment for irrigation systems being done at the moment - public expectations highHe has a lot to say, if you are interested go to paper 3.Does it represent reality, how practice is it (as people keep asking me!!)In our experience,public perception is very sensible and variable. In some occasions, even with a solid science based risk assessment,when the media manipulates the public opinion, they don't listen because they think that a specific government representaI have seen some very negative feelings expressed by the environmental justice community who feel that RA is just a tool to justify exposing poor people to chemicals. Anecdotal evidence suggests that people with little economic power are excessively expoIn general, pleased with the transparency and consistency of decision making, and with the benefits of RA outcomes (e.g. saving producers the cost of addtional expensive testing for imported diseases)Complicated, theoretical methods.nuclear waste-poor.It has recently been reported that the public is more likely to believe an assessment if it finds a potentially serious problem than if it dismisses an issue. But of course the public is not uniform; perception of the quality of risk analyses depend stroIn the nuclear power area, the success of risk assessment in influencing public perception has been slim to none. Early on, the nuclear industry believed that favorable risk assessment results should translate directly into improved public perception, anThe public sees a need for risk assessment.Import risk assessments are often questioned by the group disadvantaged by the results irrespective of scientific accurracy (apples, salmon import to Australia).Salmon Import risk assessment - poor perception of the process due to several often conflicting assessments citing largely the same evidenceThey do not easily understand the technical basis of risk assessments and some of the nuances get lost. They look for certainty when it is all relative.animal health trade: RA poorly done, usually insufficient data or poor quality data used, too complicated, no stakeholder involvement, gov't process for not making decisions, gov't puts too much emphasis on risk assessment and doesnt' use common sense orThere is a failure to understand that risk analysis is not about risk, but about opportunity. The purpose of risk analysis is not only to mitigate risks but to identify, and act on opportunities..Public perception colored by popular news reports of specific chemical hazardRisk assessment more appropriate then classic consequence assessment for major accidents for public.A lot of "interesting" informationCoastal erosion issues now involve risk assessments and the coastal strip public have perceptions of their house disappearing soon.I have seen examples of stakeholders (and some regulatory managagers, overwhelmed with pressures) who think since RA is "sceince based" it must provide the one true exact answer to all our problemsPublic perception to federal food safety risk assessment in US seems to focus on the complexity of the process. The completed documents are not "user-friendly" to the general public, even though they are readily obtainable from the web. In sho.he "public" is not swayed one way or the other by risk assessments.skepticism in some cases;Listeria in ready-to-eat-foods. Public not really informed, but industry highly sceptical and uses its own & "experts" to criticise risk assessments. Basically use the lack of hard data as a way of devaluing the results of the assessment.

Issue 2:Public expectations to an ERA regarding pesticides in northern rivers region was much less favourable. Some saw this as a delaying tactic by govt. not wanting to fund actions..Will an assessment really be taken on board and used or is it just a nice academic exercise!!!.Some resistance by risk averse sectors of the community - see RA as betraying longstanding tried and proven methods (=closed borders)In principle most accept that there is no zero and the need for risk assessment.environmental inpacts-poor.The public sees a need for transparency in risk assessment.Risk analysis is also about the impacts of risks. There are many, many risks but it is only the risks that have a potential to impact that require assessment..Public perception colored by an inability to grasp basic scientific concepts necessary to understand risk assessment issues..I have also seen examples of stakeholders (and regulatory managers) who think it is all a complete waste of timeI believe that food processors wrongly assume that "quantitative" risk assessment has an important role in their quality assurance programs. Most food processors, even the large ones, do not have the resources (and expertise) to do such assessm.others respond & quot;

Issue 3:Is the data and assumprions used right..Great press interest in comparism between risks (Bjoern Lomborg).risks of accidents-poor.The public is skeptical about the motives of the risk assessors and decision makers..Risk analysis is not a rocket science calculation. It is a process. It is about getting a common understanding about the frequency and impact of both good and bad events..Public perception colored by local health issues..hmmmmmm, makes sense&quote; when hear about it

13

QuestionPlease comment on how you rank the usefulness of the following risk analysis codes?Not familiar with them.NumbersPercentages

AnswerIndispensible : Very useful : Quite useful : Fairly useless IrrelevantVery usefulQuite usefulFairly uselessIrrelevantTotal:Very usefulQuite usefulFairly uselessIrrelevantcheck

Codexvery usefulVery usefulCodex Code of Hygienic Practice for Milk and Milk Products , annex 2 (draft step 3/5).IrrelevantIndispensibleQuite usefulIrrelevantVery usefulFairly uselessVery UsefulQuite usefulQuite usefulQuite usefulFairly uselessQuite usefulVery useful55221436%36%14%14%100%

OIEirrelevantVery usefulQuite usefulVery usefulIrrelevantQuite usefulVery usefulVery usefulIrrelevantQuite usefulVery usefulQuite usefulQuite usefulQuite usefulQuite usefulQuite usefulFairly uselessIndispensibleVery useful68131833%44%6%17%100%

OtherHealth Canada, CFIAKaiser BMD, US EPA BMDSAS/NZS 4360:1999.IPPC - important not to forget plant health.@risk, analytica, other similar.IPPC International Standards for Phytosanitary measures - pest risk analysis for quarantine pestsNever heard of either of the above!regretfully, CFIA does not follow OIE risk analysis codes or recommendations.USDA Departmental Guidelines.RiskexDidn't know there were any?.

Sheet1

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

00000

Always

Usually

50:50

Seldom

Never

Do decision makers:

Sheet2

Always

Usually

50:50

Seldom

Never

What factors jeopardise the value of an assessment?

Sheet3

Always

Usually

50:50

Seldom

Never

What factors jeopardise the appropriate implementation of a risk management decisions?

Very much

Much

Some

Little

Very little

Do you consider that risk analysis has brought about the following improvement to government decision-making?

Very little *

Do the sponsors of risk assessment in general get good value for money?

Very little *

In general, what is your assessment of the quality of risk assessment?

0000

0000

Very useful

Quite useful

Fairly useless

Irrelevant

Please comment on how you rank the usefulness of the following risk analysis codes?

Chart4

0.0270270270.32432432430.37837837840.27027027030

0.11111111110.33333333330.27777777780.27777777780

0.37837837840.35135135140.18918918920.08108108110

00.38888888890.22222222220.36111111110.0277777778

0.05555555560.38888888890.13888888890.38888888890.0277777778

Always

Usually

50:50

Seldom

Never

What factors jeopardise the value of an assessment?

Sheet1

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950

Dr. Christan StogbauerProff. Barry T. HartChristie DrewArie HavelaarEnda CumminsHelene BergeronPaul M. SchlosserChris HawkinsInge Dorthe HansenDavid YapSam BeckettClark GormleyRodolfo AvilaEdmund PeelerStephen L BrownVicki BierIvan TaylorRichardStefan FabianssonDale CooperKeith HayesCynthia FullerJoyce Van DonkersgoedDave EvansHector YepMark A. TurneoJames S. SmithAlexander LeinJack WattsRobin ZwartDavid PybusBruce NcNabLee-Ann JaykusTine HaldSanja SeparovicStuart MacDiarmidScott McEvenPaul VanderlindePhilip Berger

1

QuestionCan you give examples of proven benefits that have arisen from commissioning assessment, and acting approproately on its results?

AnswerNoxxxxxxxxNo, we carry out our own risk analysesx

Yes. Please explainregulatory requirements, giving incentive for better risk management by assessing and evaluating risk control environment.A recent ecological risk assessment for a paper mill discharging effuent to an estuary in Tasmania (Australia) resulted in the company being able to convince the state EPA that they were not causing environmental harm to the estuary..xYes, in the past, we used RA without calling it that way, but we find that the more structured and standardized gives credibility to our risk-decisions and helps supporting the regulations..Yes, Application of quarantine policy has been done on the basis of quantitative RAxOur import protocols are based on international standards or on risk assessments - indeed, all WTO Member countries must base any trade-restrictive measures on those provided in the relevant international standards (OIE and IPPC) or on scientific assessmeUse of hypochlorite instead of liquefied chlorine gas for chlorination of water, due to the safety risk for the public from a release of toxic chlorine gas.Helps to identify important sources of uncertainty. Good alternative when conservative estimations indicate a significant riskwork done in Australia on the risks associated with imported fish products failed to demonstrate a case for restricting the importation of wild caught salmon, and hence, after a ruling by the WTO, imports were allowed.Of course, the answer depends on what constitutes a "benefit". I have typically carried out risk assessments on behalf of "regulated" parties in recent years. From their point of view, the benefit is relief from excessive or unnecessI know of a number of situations in which nuclear power risk asessments have resulted in the identification and implementation of significant risk reductions at the power plants in question, often at extremely modest cost. References available on requestIt has helped the Canadian Forces prepare for multiple contingencies in military operationsIt reduces uncertainity and increases the confidence of decision makers in committing projectsQualitative BSE risk assessment fed into decision making process (fertilser, effluent)We reviewed a project risk assessment recently that showed benefits of USD 350,000 within 48 hours of completion, and further non-quantified benefits over the next few months as strategies were adjusted to deal with major indentified risks.Yes - but a tentative one - in ecological risk assessment the benefits are usually impossible to bank and therefore do not show up on the accounts - tangible benefits usually occur where the risk assessment allevies industry of regulation (eg the ballastRisk support for cost-effective re-development of impacted landchanges in bluetongue regulations in 1995 to allow entry of US feeder cattle during the nonvector seasonWe continue to hear and learn that clients who undertake our "project risk analysis" are able to mitigate cost and schedule during project construction and enhancement. Good risk analysis ensures that projects are on schedule and on budget..The improvement of regulations being promulgated by the US Department of Agriculture demonstrates the effectiveness of Risk Assessment as a support structure for decision-making. Within both APHIS and FSIS, regulations have been more focused and cost eff.More safe projects of new installations and reconstruction old, accommodation of the personnel and workers onsite, a choice of priorities and development of right actions for increase of safety.determination of the costrange of RD&D programmes and the consequent decission whether or not to continue them.Yes, risk of Bluetongue disease in US cattle imports to Canada, resulted in change in import regs (Cnd Fd Insp Agncy has done many import RAs to support or change import policies), 2) risk of clenbuterol residue in veal calf meat, supported placment ofI believe that in the U.S., the major benefit to food safety risk assessment has been largely related to the process of gathering all available information and organizing it in a comprehensive manner. This has facilitated the identification of critical aThe Danish Zoonoisis Centre uses a quantitative risk-based approach to estimate the contribution of the different animal-food sources to human salmonellosis. The results are published annually, and the authority react to these by for instance establishing.A risk assessment conducted looking at E. coli O157:H7 in Australian beef highlighted the problems with chilling carcases over weekends (a practice that is common in Australia but not in the US). As a result of these findings the industry was modified itsRisk asssessment input data typically are inadequate for decision making. Risk assessment is best used to guide decisions making by showing what parameters are insufficiently known so as to make unbiased decisions.

2* One from Canada and one from Ireland, RA not used there yet - "very little"

QuestionDo you consider that risk analysis has brought about the following improvement to government decision-making?Percentages

Very muchMuchSomeLittleVery little *Check:Very muchMuchSomeLittleVery littlecheck

AnswerFairnessSomeSomeSomeSomeSomeVery muchSomeVery muchSomeSomeVery muchMuchSomeMuchSomeSomeSomeSomeSomeVery littleSomeSomeSomeVery littleSomeMuchVery littleMuchSomeMuchMuchSomeMuchLittleVery muchMuchMuchLittleVery little4920243910%23%51%5%10%100%

RationalitySomeMuchSomeSomeVery littleVery muchMuchVery muchMuchSomeVery muchSomeSomeVery muchMuchSomeVery muchSomeMuchSomeMuchSomeVery littleVery littleMuchSomeSomeVery muchMuchMuchMuchSomeMuchMuchMuchMuchMuchLittleLittle61513233915%38%33%5%8%100%

ConsistencyMuchSomeSomeSomeVery littleMuchSomeVery muchSomeLittleVery muchSomeSomeMuchSomeLittleVery muchSomeSomeSomeMuchSomeLittleVery littleSomeSomeSomeVery muchMuchSomeMuchSomeSomeSomeMuchMuchMuchLittleLittle4919523910%23%49%13%5%100%

TransparencyVery muchMuchSomeSomeVery littleSomeLittleVery muchMuchLittleVery muchSomeVery littleMuchSomeSomeVery muchLittleMuchSomeSomeSomeVery littleVery littleSomeVery muchSomeMuchMuchLittleMuchSomeSomeLittleMuchVery muchVery muchMuchVery little7913553918%23%33%13%13%100%

Please explainThis is a difficult question because I'm not sure what your frome of reference is - imporvements compared to what or when? In my experience different agencies and circumstances have different experiences for all these "improvements". Also, some people mig.Risk analysis is only new in Ireland, can't say it has played any major part yet?If we compare to the time before RA, it is much better, less arbitrary, but there are still many problems and difficulties to overcome.More fair than without, but discrepancies remain; approach is fairly rational; but often case-specific; not very transparent to the general publicProcess is transparent, science based, & risk management undertaken on basis of risksI am not too sure of the answers as I am not close to this decition process.Believe that the structured process of risk assessment, as described in the relevant international standards, creates a format for decision making that greatly enhances each of the points above.In the past the real reason behind restrictions on the import of animal products has been to prevent free trade. The use of risk analysis ensures that defensible, transparent arguments have to be presented in support of trade restrictions.Risk analysis potentially can insert rationality into the search for an "efficient" tradeoff of the costs and benefits of regulation--the balancing of net benefits against net costs. It can also help identify who wins and loses with a change inThere is still enormous inconsistency between agencies, and in the regulation of different types of risks. However, I believe that overall, risk analysis has improved the process of government decision-making and regulation of risks.We have combined Multi-Criteria Decision Making with Risk Assessment to consider Risk along with Benefits and Costs in a transparent manner..Should do but probably still a way offMost Government procurement is based on processes designed to ensure fairness and transparency explicitly, with rationality and consistency as useful byproducts. Risk analysis supports these objectives, but it is not a dominant (or even necessary) featurMuch of the government risk assessments we deal with (eg import risk analysis) are qualitative - the decisions are not entirely transparent because of linguistic uncertainty - the process is, however, consistent and relatively rationale.Risk assessment is not really a science or art, but a tool, dependent on the assumptions and approaches used. Since there has been many changes to US/state risk assessment approaches over the years (I've been practicing for over 12 years in this field, tour federal government (CFIA) does the risk assessments in isolation of industry in spite of industrys request to participate.In Canada, governments rarely use or apply quantitative risk analysis. Other than in the area of environmental risk analysis to restate environmental compliance criteria and with energy related applications/approvals, only qualitative risk assessment is.The final decisions have changed some, but the most important result is the ability for others to clearly understand the trade-offs inherent in the decision..Any reasonable constructed process will enhance these qualities. More study of the process, such as this survey, can improve it further.In principle comparison of decissions on risks is fair and rational, however often the starting points are not the same (hence no consistency) or not known at all (no transparency)The risk based approach provides fewer opportunities for obfuscationI think we are moving in the right direction. I think it will imporve more with time, but will never be perfect. There are many forces that tend to push us "off track" (e.g. crises of the day, conflicting objectives), but sound, systematic RASince food safety risk assessment is still very much in developmental phases, we do not yet know its final value to US agencies. It is clear that there remain issues related to consistency as there are many different ways to approach a risk assessment.I had a very hard time answering the above, because it varies from one risk assessment to the other. Generally, when complex "stable-to-table" analysis are performed the improvement or benefit is difficult to see. However for smaller or more speTo make use of huge amount collected dataThe main advantage of the import risk analyses we carry out is that they make the decision making transparent. Objectivity is a slippery commodity, especially where data are sparse, but transparency should always be attainable.At the moment there are no apparent linkages between risk assessment and risk management. Risk assessments tend to be carried out in isolation simply as an exercise rather than as a tool for use by risk managers. There is a great need for improving the liThe more transparent risk assessment models are often too simplistic to be useful. Where bias is recognized, rational and consistent decision making occurs more oftenbut fairness required unbiased decision making.

3

QuestionDo the sponsors of risk asessment in general get good value for money?NumberPercentage

AnswerYesxxxxxxxxxxxxxxxxxxxxxx22Yes58%

No, mostly because0No, mostly because0%

They take too long to completexxxxx5They take too long to complete13%

They cost too much moneyxxxxxxx7They cost too much money18%

Their focus didn't address the problemxxxx4Their focus didn't address the problem11%

Other, please specifyI don't think I'm qualified to say,There is often a conflict between necessary resources for a good RA and available resources. So what is best value for money?They take too long to complete.Import Risk assessments can take longer to complete than might be thought at the outset.Being a practitioner, it would be difficult for me to answer no generally. For my regulated-party clients, the savings from less stringent regulation is often much greater than the cost. On the other hand, I'm reasonably certain that some risk assessmen.There are some issues with using qualitative data that have not been resolved..Results often too complexWe are consultants, and they pay us for risk assessment, so I have a slight bias here!!.Depends on the assessor. I've seen some really bad assessments performed.no input from industry who is affected by these risk assessments - thus, their tax dollars not being using fairly.The process is still evolving. Most of the time, the individuals ASKING for the risk assessment are not clear on what they want, nor do they understand what they will get. In addition, the perception that the risk assessment will "bind" the deAs a provider of risk assessment, my company survives because we provide "sponsors" with good value for the money, but then we often undercut our competition by providing a quality product for less than half the cost.RA must be only a part of improvements in whole management systemThe results are not validated.Depence on the knowledge of the sponsors ... some people just ask for anything and will also accept anything for any priceDo not commission external risk analysisdepends on point of view, there is much room to improve efficiency, should strive to optimise marginal return, must learn to walk before runI am not sure that the US federal agencies know what to do with their risk assessments after they are completed.Again, it depends on the specific analysis, but generally I believe, that the risk assessments commissioned by a government is not good value for money. They may help scientists to focus their future research, but the results are often too uncertain to re..All of the above to some degree but the main reason is that there is no communications between risk managers and risk assessors, therefore what is delivered is not what was expected. While the two should be separate there must be clear communication betweYes and no..if they risk assessment is dynamic (includes immune status and secondary spread) then they get good value. If they risk assessment is static, then the focus didn't address the problem adequately.38100%

4

QuestionIn general, what is your assessment of the quality of risk assessment?question too vague - where when? A better question might be what is your assessment of the quality of risk assessments that you have first hand knowledge of? And add a n/a categoryNot relevant to the way we work. We carry out our own risk analyses on behalf of government.NumberPercentage

AnswerExcellent.xxx3Excellent8%

Quite goodxxxxxxxxxxxxxx14Quite good38%

Fairxxxxxxxxxxxxxxxx16Fair43%

Poorxxxx4Poor11%

Unacceptable.0Unacceptable0%

537100%

QuestionWhat factors jeopardise the value of an assessment?NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1Insufficient human resources to complete the assessment323313334342342234342444432432322222311214100373%32%38%27%0%100%

2Insufficient time to complete the assessment432312334242341234422433422322421431412101003611%33%28%28%0%100%

3Insufficient data to support the risk assessment121112243221241114331332211332122221114137303738%35%19%8%0%100%

4Insufficient in-house expertise in the ares2242433434222424242334544324223342420148131360%39%22%36%3%100%

5Insufficient general scientific knowledge of the area2242442444224241443323244422232342512145141366%39%14%39%3%100%

7Other, please specify.Poor communication to stakeholders and the public.Arbitrary constraints such as default assumptionsInsufficient knowledge of risk analysis methods on the part of the analysts.Lack of senior management support and interest.no industry input from being of process to the end (poor risk communication skills), not transparent process.Disconnect with target audience..

Comments:in yellowLack of expertise and lack of data the two big ones..Risk assessments are not often subjected to any attempts at confirmation even in those rare cases in which it is scientifically possible to do so. So we don't know whether the field as a whole has much quality. It is certainly true that many individualI am not really in a position to know the answers to the above questions, since I am familiar with only a narrow spectrum of risk analyses, and most of those were not done by government agencies.Because of limited time we need to utilize expert judgement qualitative assessments.Risk assessor striving for accuracy at the expense of management understanding and simplicityA risk assessment can be designed around most limitations and achieve a useful (if not 'optimal') outcome, but senior support is a vital ingredient irrespective of other constraints.Government risk assessments that we review often show evidence of poor in-house expertise in RA methods and techniques outside of the usual qualitative approachesCompletely depends on the person doing the assessment..The biggest risk in a risk assessment is failure to properly frame the problem and fully understand the assumptions around the problem..Lack of understanding on the part of the individuals asking for and using the assessment.Risk assessment sucess is measured by the ability of the target audience to understand the content and conclusions..Communication of how to use the results, i.e., the link between risk assessment and risk management.uncertainty in given data, simply because they are totally unknown.It will take time for a critical mass of technicians and managers to change their mind set, acquire skills, become proficient, gain comfort and confidence, accept costs and accept findings that what has been done for years may have been sub-otimal (or.inadequate static risk assessment models

6

QuestionWhat factors jeopardise the approporiate implementation of a risk management decision?NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1Politics4223321223322231244241221223322224125197503614%53%19%14%0%100%

2Other issues take precedence333123343122242333321432333222334233111740359%31%49%11%0%100%

3Legal restrictions5433444253443243354443544424433425330411165360%11%31%44%14%100%

4Insufficient resources to implement the action232323344243343233433233344322434340817100350%23%49%29%0%100%

5Risk assessment too complicated233344444434342214442534242232224341108151353%29%23%43%3%100%

6Risk assessment not accepted as valid344344434444342353242423342232224220119141350%31%26%40%3%100%

7Insufficient time333423344344342333441424434422424142611160356%17%31%46%0%100%

8Other:Lack of awareness of available tools and expertise.Lack of senior management support and interest.A risk assessment that is not tailored to the needs of the project, but is performed with an academic slant.insufficient participation of stakeholders.Lack of understanding by decision-maker of risk and risk conceptsInappropriate application of risk assessment methodologies to site-specific issues..Usually, risk assessment does not address the concerns of manager

Comments:Many risk analysis projects are commissioned as an organisation wants to be SEEN to be doing something, if the results don't suit it gets shelved..Not too sure.A risk assessment can only be part of the story about what should be done; what constitutes "appropriate implementation" will always remain in the eye of the beholder whether or not the implementation matches the assessment.Same as above; I am not sure I am in a good position to comment in generalThere is a growing awareness of the need but lack of awareness of the capabilities of tools and sources of information..There is still a lack of understanding between assessors and managers and the communication of factsSimilar comments to Item 5In reality politics and economics seem to take precedance, particularly in qualitative assessments where low or negiligible are open to interpretation..Laziness is a factor too.....until things change or go wrong.Risk assessment is a tool of risk managers. The value of this tool in risk management decisions is dependent on a variety of intrinsic and extrinsic factors..What is appropriate? Is the question intended to include appropriate implementation of a bad decision, i.e., no implementation..The changes implied can be big. Any change is difficult. Expensive change that implies one was "doing it wrong before" is very difficult intertia to overcome..Risk assessment is a rather new discipline in food safety, and I believe many risk managers are uncomfortable about using the results. Especially, if it suddenly contradict how they would have reacted in case the risk assessment was not performed..Again on of the main issues is that risk assessors often do not know what management options are best suited to the situation and without proper guidance by risk managers will not address there issues in the risk assessment.risk assessment not adequate (static model)

7

QuestionAre the risk assessors made sufficiently aware of the purpose of the assessment?NumberPercentage

AnswerYesxxxxxxxxxYes, I have always felt that I knew the purpose; I have sometimes run into clueless risk assessorsIt all depends. In many cases, they are. However, in others, a company or government agency may commission a risk assessment as a "whitewash" of a particular topic, or to give "protective coloration" for a decision that has already bxxxYes. We tend to use AS/NZS 4360 as our process guide. Establishing the context is a critical component of the risk management process.xxxxxxxxxxx2359%

No, please explain:No, not always.No, usually not : it is often unclear for the managers themselves..The regulator, i.e. those who order the assessment, usually do not realy understand what they need.If they don't understand what they are doing, they shouldn't be doing it!.Yes & No, because sometimes the clients cannot describe or relate to the problem to be risked..Selodm, mainly because the decision maker may not completely know herself, or because political factors lead the decion maker to "hide the ball".Usually there is poor communication with decision makersDepends, mostly yes ... not always however.Yes and no, not always..... it is extremely important in any group effort that everyone understands adn buys into the same objectiveMany of the US risk assessments do not have a clear regulatory endpoint, nor are they focused enough. This leads to completed documents for which there is limited use with respect to public policy.Often, it is up to the risk assessors to decide upon the outcome, i.e. define the question(s) that needs answering.No, often the politics underlying the need for conducting a risk assessment are not specified and therefore the risk assessor does not always construct the risk assessment appropriately.

8

QuestionHave you been involved in a risk assessment that has reached a 'More data are required to produce a meaningful assessment' impasse?NumberPercentage

AnswerNoxxxxxxxxxxxxx1333%

Yes, In which casexxxxxxxxxxxxxxxxxxxxxxxxxx2667%100%

Do you think that the result could have been predicted beforehand?xxxxxx615%

Was the risk assessment still performed because:xxxxxxxxxxxxxx1436%

The purpose of the risk assessments was to point out areas where more information is needed?xxxxxxxx821%

One needed to demonstrate that a risk assessment had been attempted?00%

International guidelines (e.g. Codex, OIE) required it to be completed?00%

There is national or international legislation stating that risk assessment should be done before some decision could be made?xxxx410%

It was a specific request from the relevant authority?xxxx410%

It was a specific request from the government (i.e. a political decision)?xxx38%

Corporate rules required that the risk assessment be done anyway?x13%

The risk assessors were not asked for their opinion beforehand (or with held it and did the risk assessment knowing that 'more data is needed' would be an acceptable result?00%

Everybody else is talking about and performing risk assessments?00%

Other reasons? Please specify:General comment: the question is: When can you produce meaningful results? Given scientific standards, usually do not provide of enough data. However, in a situation with sparse data there is still potential for value-added simply by showing people that.Any of the above reasons can be involved. Even a severely limited assessment can be informative, usually more so than no assessment at allThere is always the possibility of producing a more meaningful assessment with more data. However, in my experience, a good risk assessment can still help to clarify the situation, by indicating exactly what IS known about the question of interest, whatIt was determined that some analysis was better than none..It was important to develop a structure (model) that could be discussed with industry to show what if scenarios.Unfortunately, risk assessments often are considered after all data are collected, with no prior input from the risk assessor. The risk assessor needs to be part of the project; this is not an effort that can be done in a "black box".An evaluation of data quality and useability will usually identify situations of "impasse." In other cases, a risk assessment may be required under applicable regulation, as a tool for determining risk-based remedial goals, or in a phased appro.There are multiple answers to this question. The format is not allowing them!Although some figures are unknown, relevant (sub)conclusions can be drawn.Essentially, versions of all of the above can happen (and likley have) in various organizationsThe need for "additional data" is fairly universal to food safety risk assessment. It should not be an excuse for not completing the assessment. US risk assessors seem resistant to using Bayesian techniques (and expert elicitation) to get arou.

9Several of the above in some circumstances, and most of them across a range of assessments. However, I have not been involved, either as an assessor or as an auditor of an assessment, where the assessors were not asked for their opinion (or withheld it),

QuestionAre you aware of risk assessments being commissioned where it was obvious from the beginning that the results were not going to be used, or a meaningful analysis could not be achieved?NumberPercentage

AnswerNoxxxxxxxxxxxxxxxxxxxxxxxx2462%

Yes. In which case, was the risk assessment still performed because:xxxxxxxxxxxx1231%

One needed to demonstrate that a risk assessment had been attemptedxx25%

International guidelines (e.g. Codex, OIE) required it to be completed?00%

There is national or international legislation stating that risk assessment should be done before some decision could be made?x13%

It was a specific request from the relevant authority?xxxx410%

It was a specific request from the government (i.e. a political decision)?xx25%

Corporate rules required that the risk assessment be done anyway?xx25%

The risk assessors were not asked for their opinion beforehand (or with held it and did the risk assessment knowing that 'more data is needed' would be an acceptable result?x13%

Everybody else is talking about and performing risk assessments?x13%

Other reasons? Please specify:I'm sure this does happen sometimes, for a variety of reasons, but do not believe that I have been in this situation myself.Sometimes we have done analysis to demonstrate the capability for future operations even though the analysis is not expected to impact the current operation..We try not to waste our clients' money! As consultants, we think it is not worth it in the long run (and has few benefits in the short run either).A cannot think of a specific case that I have personnaly been involved in where nothing was used or nothing was learned, but I would not be surprised if it has happened..required as part of benefit/cost analysis

10

QuestionDo decision makers:NumbersPercentages

Answer1- Always. 2- Usually. 3- 50:50. 4- Seldom. 5- NeverAlwaysUsually50:50SeldomNeverTotal:AlwaysUsually50:50SeldomNevercheck

1) Understand and use the results of a risk assessment3223222222343342342323344424233432244501512101380%39%32%26%3%100%

2) Encourage involvement of stakeholders and external expertise?223323333223333234224544323343332313251111952383%29%50%13%5%100%

3) Encourage assessors to explain what may and may not be possible to achieve?52432343424444322422344244444434431325198182383%24%21%47%5%100%

4) Put a lot of emphasis on receiving comments at the planning stage of a risk assessment4333434334434223223533543334344324350517113360%14%47%31%8%100%

5) Involve risk assessors in planning how to communicate the risk assessment results?22334223243434324344434442444344314451710181373%19%27%49%3%100%

6) Involve stakeholders after completion of the assessment of the appropriate risk management action to take?53223232143434233323444433333232322251111672373%30%43%19%5%100%

7) Expect too much of a risk assessment?342223334422324323431112343231332222241413603711%38%35%16%0%100%

8) Expect too little of a risk assessment?33444544432444234343455232343233254540611155370%16%30%41%14%100%

9) Assign sufficient resources and time to complete the risk assessment?423344321342334232224144234234334434221012130375%27%32%35%0%100%

10) Encourage assessors to suggest alternative approaches to the assessment?4444244444443342424444442343434442425065251370%16%14%68%3%100%

11) Require the assessment to be simplified for ease of understanding, at the expense of technical accuracy?144244444432342213334333134442543235146111423711%16%30%38%5%100%

12) Make the results available only if it suits their purposes?25442343 / 4532333132343232344442454254412810124366%22%28%33%11%100%

13) Encourage assessors to produce an assessment to support a predetermined position?35252234554443422243423235443444425531198137383%24%21%34%18%100%

14) Involve risk assessors in the decision-making?23231322234243424443354433433342123413913111378%24%35%30%3%100%

15) Allow/expect the risk assessors to make the decision?4443343423434452454425344244335214435159175373%14%24%46%14%100%

Other comments on decision-makers' behaviour:There is not yet an understanding on what should be the threshold (quatitative) of an acceptable risk.Decision makers' behaviour is influenced by interim discussion and allusion to realistic expectations or RADo not dare to start a risk assessment, afraid of the publicity of the outcome.The distinction between risk assessors and decision makers is often blurred in the the context of import risk analysis.They exhibit a very wide distribution of behaviorsIn my experience, I have been pleasantly surprised at how rarely decision makers have encouraged assessors to produce assessments to support predetermined positions. In my view, the more serious problems are decision makers who do not understand the resuBecause of subjectivity of analysis decision makers will often discount results that do not satisfy there needs..Still not much drive to use risk assessment in many areas apart from import risk assessments to support WTO cases.don't have stakeholder involvement in the beginning of the risk analysis processMost decisions are based on "it seemed like a really good idea at the time".It is often a good business strategy for a "sponsor" to separate risk assessment and management activities since these two activities can have conflicting business goals..No real understanding of risk assessment and how it can and should be used. Expect a risk assessment to tell them the risk management options rather than using the risk assessment to evaluate alternative options.

11

QuestionIs room left for decision-makers to change or reverse the decision should new information appear that contradicts the risk assessment assumptions?NumberPercentage

AnswerNoxxxxxxNo, The monitor-review - reassess cycle is rarely implemented in my experience - this is one of the biggest failings of ecological risk assessment in my area.xxxxx1128%

Yes. Please explain how this is achieved:The fact new info contradicts risk assumptions makes people dubious about the risk assessment and its malidity to start with!! Can this not happen again, and again!!Yes, but managers are prudent. It is suggested in our framework for risk analysis but we have not experienced it yet.Risk assessments are periodically reviewed and revisedPremises on which RA is done are specified, and the assessment is clearly stated to be a dynamic entity which will require updating as events change - e.g. as the prevalence of a disease in an exporting state changes, so the risk to the importing state ch.Yes, the decision review board is reconvened, and outcome communicated to decision maker(s)Assessments are revised and updated systematically to take account of chengs in assumptions / inputs.the risk analysis clears states the data used, hence as more data becomes available it is possible to change positionsNot always, but sometimes a decision can be changed; in particular, new information on toxicity may shift a regulation toward more stringency, although the opposite is less likelyIn fact, there is one well-known and published example where decision-makers reversed the recommendation of a risk analysis, even WITHOUT new information that contradicts the risk assessment assumptions, and this is not generally considered to represent aIn general the subject matter experts are not considered to be unbiased and therefore their opinions can be discounted.xRisk assessment is just one input to decisions. Like any tool, it only support decisions, but managers are paid to manage, and that usually involves factors other than those included in simple tools..In our process, we always use a reality check against a calculated risk-based outcome. Head checks rule. This type of outcome relates to not fully understanding and/or quantifying the type and number of risks that will or may have impact on an outcome.A good risk assessment merely lays out the trade-offs. Other factors, including culture, values, politics and economics must be considered by the decision maker in maing the final decision.Regulatory relief.It must be described in the comnay policy.As long decissions still can be changed when new information is received (and considered)By identifying more information necessary to change the uninformed decision.Significnaly new information can lead to reassessment, new result, change in policy, BUT it all takes time..... changing regulations is often a long processHave not yet seen much decision making linked to risk assessment, at least in the food safety arena.Technically, yes, decision makers are often the authorities and can as such change the legislation. However, in practice it will be very hard to change a decision when first it has been made. I think most decision makers will be afraid to look untrustwort.Yes. A number of mechanisms exist for review of risk analyses. Indeed, it is explicit in our policies.Yes. By making it clear that decisions may be revised in light of new informationYes. Decisions can be reviewed but this usually takes more than 12-months. Therefore you are suck with decisions for more than 12-months, even bad decisionsYes. Decision making is easy developing a new or varying the rationale for a decision is very difficult and is usually inadequate

12

QuestionWhat has been the public perception (if appropriate) to the whole risk analysis process on a specific issue (or issues) you know about?

AnswerIssue 1:OpRisk quantification, too early to tell about the general perception.Ecol risk assessment for irrigation systems being done at the moment - public expectations highHe has a lot to say, if you are interested go to paper 3.Does it represent reality, how practice is it (as people keep asking me!!)In our experience,public perception is very sensible and variable. In some occasions, even with a solid science based risk assessment,when the media manipulates the public opinion, they don't listen because they think that a specific government representaI have seen some very negative feelings expressed by the environmental justice community who feel that RA is just a tool to justify exposing poor people to chemicals. Anecdotal evidence suggests that people with little economic power are excessively expoIn general, pleased with the transparency and consistency of decision making, and with the benefits of RA outcomes (e.g. saving producers the cost of addtional expensive testing for imported diseases)Complicated, theoretical methods.nuclear waste-poor.It has recently been reported that the public is more likely to believe an assessment if it finds a potentially serious problem than if it dismisses an issue. But of course the public is not uniform; perception of the quality of risk analyses depend stroIn the nuclear power area, the success of risk assessment in influencing public perception has been slim to none. Early on, the nuclear industry believed that favorable risk assessment results should translate directly into improved public perception, anThe public sees a need for risk assessment.Import risk assessments are often questioned by the group dis