Ethics In Evaluation: The ARECCI approach to enhancing how ethical risk is managed in g gyour evaluation practice 2008 CES Conference

May 11-14, 2008

Presentation to

2008 Conference of theCanadian Evaluation SocietyCanadian Evaluation Society

Quebec, PQ CanadaMay 12, 2008

By

Linda Barrett-Smith & Birgitta LarssonARECCI Project Lead PrincipalAlberta Heritage Foundation BIM Larsson & Associatesfor Medical Research

With Contributions by Laura L’HeureuxWith Contributions by Laura L Heureux

TodaySituating Ethics in Evaluation

Describe the Alberta ResearchDescribe the Alberta Research Ethics Community Consensus Initiative (ARECCI) journey

Background and Description• Background and Description

Present some deliverables• ARECCI Online Ethics

Screening Tool• ARECCI Ethics GuidelinesARECCI Ethics Guidelines

(Review Considerations) for QI and Evaluation

Forum for discussion

ARECCI Collaborating Partners• AHFMRAHFMR• Alberta Cancer Board• Alberta Mental Health Board• Aspen Health Region• Athabasca University University Research Services• Athabasca University, University Research Services• Canadian Evaluation Society, Alberta Chapter• Calgary Conjoint Health Research Ethics Board• Calgary Health Region

C it l H lth R i• Capital Health Region• Chinook Health Region• College of Physicians and Surgeons – Research Ethics• Community Research Ethics Board of Alberta• David Thompson Health Region• East Central Health Region• Health Quality Council of Alberta• Michael Smith Foundation for Health Research (BC)( )• Ministry of Health and Wellness, Government of Alberta• Northern Lights Health Region• Office of the Information and Privacy Commissioner of Alberta• Palliser Health Region• Palliser Health Region• Provincial Health Ethics Network of Alberta• SEARCH Canada• Southern Alberta Child and Youth Health Network

Why is Ethics an Issue?

Recent increase in volume ofRecent increase in volume of evaluation projects

Increasing scope andIncreasing scope and sophistication of evaluation projects

Lack of clear established guidelines and ethics oversight processprocess

Wide range of practitioners with various levels of skill leading thevarious levels of skill leading the projects

Evaluation Realities in Alberta

The Government AccountabilityThe Government Accountability Act

Policies of funders of health and social services, such as Health Canada, Alberta Alcohol and DrugCanada, Alberta Alcohol and Drug Commission, Regional Health Authorities and large foundations

Expected that a portion of the overall project budget will be allocated to evaluation

GuidelinesHealth Information Act (HIA) identifies the designated REBs for the province of Alberta:the province of Alberta:

–Requires ethics review of health information used for research purposespurposes

–Six REBs designated as able to perform such ethics review

Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS)

Interpretation that evaluation does not require formal ethics review

Research versus E l tiEvaluation

O l b hOverlap between the processes of knowledge generating projects including research and p j gevaluation which leads to grey areas and lack of clarity

Research focuses on defining best practice, Program Evaluation serves to advanceEvaluation serves to advance better practices

Evaluation Ethics Guidelines

CES Guidelines

AES Guidelines

Australasian Evaluation Society

EES

Professional GuidelinesProfessional Guidelines

FOIP

HIA

Who is Responsible?Who is Responsible?Evaluators

Project Leads

Clients

OrganizationOrganization

Funders

What to submit and to whom?

Consequences

Differences in deciding if a project is research & thereforeproject is research & therefore needs REB review

I i t / l k fInconsistency / lack of transparency in ethics review decisions

Avoidance of ethic review

Inconsistent consideration of ethical implications in non-research projectsresearch projects

Consequences q(cont’d)

Determining minimal risk

Who and how decisions areWho and how decisions are made

E d lt ibilit fEnd result – possibility of exposing participants to potential risks and disrespectp p

Issues that led to ARECCI work

• Lack of clarity related to what yprojects required ethical review

• Same projects sometimes• Same projects sometimes treated differently by separate research ethics boards

• Some projects unnecessarily going to designated REBsg g g

• Some projects not receiving review when neededreview when needed

Alberta Research Ethics Community Consens s Initiati eConsensus Initiative

• Collaborative undertakingCollaborative undertaking

• Joint initiative• Alberta Heritage Foundation for

Medical Research (AHFMR)• REBs in AlbertaREBs in Alberta• Health Authorities• Alberta Health and Wellness

• Created in 2003• Completed 3 phasesp p

ARECCI ObjectivesARECCI Objectives1. To develop a common understanding

and broad consensus on issues of ethics review

2 T i th l it i t2. To increase the clarity, consistency, transparency, and efficiency of ethics review processes in Alberta

3. To provide an approach to answering three questions:

h t t f i ti ti ?• what type of investigation?• what processes should it go through?• what level of review should occur?

ARECCI ObjectivesARECCI Objectives (cont’d)

4. To develop guidelines and tools for Alberta’s health researchersfor Alberta s health researchers, managers, ethics boards and other stakeholders to implement the recommendations

5. To inform regional, provincial and5. To inform regional, provincial and federal policy related to ethics review processes

Phase 1: Assumptions & P i i lPrinciples

• Protecting people involved in evaluation projects is in principle just as important asprojects is, in principle, just as important as protecting people participating in research investigations

• It is possible to pragmatically distinguish, for purposes of ethics review, between research and quality / evaluationand quality / evaluation

• If an investigation is determined to require ethics re ie it sho ld be re ie ed thro ghethics review, it should be reviewed through its own particular process of ethics review by a body having appropriate jurisdiction and

itcapacity

Phase 1: Assumptions & P i i l ( t’d)Principles (cont’d)

• Although REBs currently provide theAlthough REBs currently provide the highest level of ethics review across the province, there is no reason, in principle, that a similarly high level of ethics reviewthat a similarly high level of ethics review should not be developed and provided by other processes for quality or evaluation projectsprojects

• Each institution responsible for the protection of people and increasingprotection of people and increasing knowledge will comply with legislation, regulations, and policy application to its functionfunction

Phase 1: Results ase esu ts

• Five Recommendations for Ethics S i d R iScreening and Review

• Pragmatic Screening Tool to determine project primary purpose

Research or Quality Improvement / EvaluationEvaluation

• Risk Filters to determine level of riskResearchEvaluation and Quality Improvement

Phase 2: Field Test ResultsFinalized ARECCI document describing• Finalized ARECCI document describing underlying principles and concepts –“Protecting People While Increasing Knowledge” (Dec 05)

• Revised stand alone, paper-based tools One Screen - Determining Project PrimaryOne Screen - Determining Project Primary PurposeTwo Risk Filters

for Quality and Evaluation Projectsfor Quality and Evaluation Projectsfor Research Projects

• Ongoing connections with other stakeholdersg gInter-provincial, national, international

• Agreement to sustain momentum to enhance awareness & build ethics capacity across the system

Phase 3: Feb 2006 – Mar 2008Capacity Building - Decision SupportCapacity Building - Decision Support Guides

R i i AUTOMATION d f th• Revisions, AUTOMATION, and further testing of ethics screening tools

• Development of ARECCI Ethics Guidelines for QI and Evaluation

• Draft Process / Protocol for review of higher risk QI, QA and Evaluation projects

• Development of Project ethics education

• May 4-6, 2008 Inaugural National Conference on the issues

ARECCI Innovations to DateC t f “k l d ti ”• Concept of “knowledge generating”o To describe a range of activities: Research and

Non-research (i.e., evaluation, QI, QA)

• Concept of “Project Ethics” in relation to “Striking a balance in protecting/respecting people while increasing knowledge”o Ethics considerations across types of projects

• 5 Recommendations (Principles) - Published“Ethi S i T l” li i• “Ethics Screening Tool” – online, paper versions

• “Ethics Guidelines for QI & Evaluation Projects”• “Review Considerations” for higher risk• Review Considerations for higher risk

Evaluation/QI type projects• May 4-6, 2008 “Protecting People” Conference

o Fostered national dialogue and action

For more information:Linda Barrett-SmithARECCI P j t L dARECCI Project LeadAlberta Heritage Foundation for Medical ResearchEdmonton, Alberta CanadaLinda.Barrett-Smith@ahfmr.ab.ca

Birgitta LarssonPrincipal, BIM Larsson and AssociatesEdmonton, Alberta Canada,birgitta@bimlarsson.ca

Laura L’HeureuxSouthern Alberta Child & Youth Health NetworkSouthern Alberta Child & Youth Health NetworkAlberta Children’s HospitalCalgary, Alberta Canadalaura@panoptik.ca

Or visit the ARECCI website: www.ahfmr.ab.ca/arecci