FSIS’s Needs for Peer Review of FSIS’s Needs for Peer Review of Risk AssessmentsRisk Assessments

Carol Maczka, Ph.D.Senior Scientist for Risk Assessment

FSIS/USDA

September 30, 2003Presented at

JIFSAN/SRA/RAC SymposiumOn Peer Review of Risk Assessment

FSIS Peer Review Process to date

Lessons Learned

Needs and Challenges

Peer Review MechanismsPeer Review Mechanisms Salmonella Enteriditis Risk Assessment in Shell Eggs and

Egg Products (1995)

Inter- and Intra-Agency review by specialists

Independent review by select subject area specialists

Publications in peer review literature

E. coli 0157:H7 Risk Assessment in Ground Beef

Inter- and Intra-Agency review by specialists

NACMCF review

Independent review by select subject area specialists

Public comment

External peer review by National Academies of Science

Peer Review Mechanisms Peer Review Mechanisms (Continued)(Continued) Risk of E. coli 0157:H7 in Tenderized versus Non-Tenderized Steaks/Roasts

Inter-Agency review

NACMCF review

Harvard BSE Risk Assessment

Commission risk assessment (Independence)

Inter- and Intra-Agency review

External peer review using independent contractor

Listeria Risk Assessment - Product Contamination from Food Contact Surfaces

Inter- and Intra-Agency review

ORACBA review

Public comment

External peer review using independent contractor

ORACBA/OMB review

Peer Review Mechanisms Peer Review Mechanisms (Continued)(Continued)

Other Risk Assessments (Currently in Progress)

Perfringens Risk Assessment in RTE Meat and Poultry Products

Risk Assessment for Salmonella Enteriditis in Shell Eggs and Salmonella spp in Egg

Products

Salmonella Risk Assessment in Raw Beef and Poultry Products

Salmonella, E. coli 0157:H7 Risk Assessment in RTE Meat and Poultry Products

External Peer Review using Independent Contractor

FSIS Peer Review Process to DateFSIS Peer Review Process to Date

Intra-AgencyIntra-AgencyReviewReview

ORACBAORACBA

IndependentIndependentExternalExternal

Peer ReviewPeer ReviewInter-AgencyInter-Agency

ReviewReviewRARA

PublicPublicCommentComment

ORACBAORACBA

OMBOMB

RR

RRRRRR

R = Revised in response to commentsR = Revised in response to comments

Elements of a “Good” Peer Review ProcessElements of a “Good” Peer Review Process

Independent External Peer Review especially for major regulations or influential information

Access to broad range of scientific expertise (modelers, subject area experts)

Timely and cost-effective review

Appropriate staging of peer review and public comment to avoid unnecessary revisions

Reviewers are given a focused charge and are invited to provide general comments.

At a minimum, reviewers should be asked to comment on: Overarching logical structure of the

model; Validity and appropriateness of the data used, reasonableness of assumptions made, model’s

mathematics and equations, whether risks have been appropriately characterized; key sources of

variability and uncertainty identified; critical assumptions; data gaps. Computerized models should

be audited.

Reviewers are instructed to avoid policy determinations

Lessons LearnedLessons Learned

Elements of a “Good” Peer Review ProcessElements of a “Good” Peer Review Process

Reviewers are provided with relevant background information on potential sources

of controversy

Reviewers are held accountable

Review is balanced

Reviewers do not have a conflict of interest

Procedures for documenting response to comments; comment/response document

Outside firm (entity) supervises the review. Comments are provided to Agency

blinded

SOPs for conducting peer reviews

Lessons Learned Lessons Learned (Continued)(Continued)

Needs and ChallengesNeeds and Challenges Large pool of external peer reviewers (“expert registry”)

Requirements for Peer Reviewers

Scientific and Technical Expertise/Experience

Multiple areas of expertise (e.g., modelers, epidemiologists, microbiologists)

No real or perceived conflict of interests

No advocated positions (balance)

Have not conducted multiple peer reviews for the Agency in recent years

How can an “expert registry” be created ?

How and When are the pool of potential candidates screened for bias/conflict of

interests

Needs and Challenges Needs and Challenges (Continued)(Continued) Appropriate Mechanisms for Peer Review

Contract with Independent Body

National Academies of Science

Outside Firm

NACMCF

Risk Assessment Consortium

Consortium of Universities ?

Pro Bono

Others ?

Need to ensure Timely/Cost Effective/Independent Peer Review Mechanism

Standard Operating Procedures for Conducting Peer Review

Need to ensure consistency with other agencies conducting significant regulatory review

Summary of Major NeedsSummary of Major Needs

Pool of External Reviewers from which to select

Mechanisms for accessing peer reviewers

Establish SOPs for external peer review