Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 1
TWO APPROACHES TOEVIDENCE-BASED POLICYDEVELOPMENT FORPROFESSION REGULATORS
Evidence-Based Policy Making at the Collegeof Physicians and Surgeons of Ontario
Presentation Overview
• What is evidence?
• Why use evidence?
• CPSO Policy Review Process
• Types of evidence
• Case study
• Insufficient evidence – what to do?
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 2
Evidence-Based Policy
Evidence-basedEvidence-basedEvidence-based Evidence-informedEvidence-informedEvidence-informed
What is evidence?
Anything that can be used to bolster aparticular policy position.
Why Use Evidence?
Now what is theCollege making me
do?
I am trying to takecare of patients –do they just make
stuff up?
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 3
Why Use Evidence?
Why Use Evidence?
CPSO Policy Review Process
STEP 1: Research andPreliminary Consultation
STEP 1: Research andPreliminary Consultation
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 4
CPSO Policy Review Process
STEP 2: Analysisand Drafting
STEP 2: Analysisand Drafting
CPSO Policy Review Process
STEP 3:Consultation
STEP 3:Consultation
STEP 4: Re-draft and FinalizeSTEP 4: Re-draft and Finalize
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 5
What Types of Evidence• Research
Jurisdictional ReviewJurisdictional Review Legal /Case Law
Legal /Case Law
Literature ReviewLiterature Review
What Types of Evidence
• Consultation
– External
– Internal
• Public Opinion
Polling
Case Study• Physicians’ Relations with Industry: Practice,
Education and Research Policy
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 6
– Focus on 2 specific positions set out in thepolicy
• Receipt of Gifts byPhysicians
• Product Information /Marketing
What is the Evidence – Gifts?
Current College of Physicians and Surgeonsof Alberta standard:
“A physician must not accept any personal gift ofany monetary or other value from industry.”
Current College of Physicians and Surgeonsof Alberta standard:
“A physician must not accept any personal gift ofany monetary or other value from industry.”
Current College of Physicians and Surgeonsof Alberta standard:
“A physician must not accept any personal gift ofany monetary or other value from industry.”
Jurisdictional Review
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 7
Jurisdictional Review• Association of research-based
pharmaceutical companies (Rx&D)…
Rx&D Code of Ethics:
"Members must not offer toany Stakeholder... any gift – incash or in kind, or anypromotional aid, price,reward, or any other item asan incentive or reward..."
Rx&D Code of Ethics:
"Members must not offer toany Stakeholder... any gift – incash or in kind, or anypromotional aid, price,reward, or any other item asan incentive or reward..."
Includes common, low-value items such asagendas, mouse pads,stationary, etc…
Literature Review
Literature Review
University of PennsylvaniaScholarly Commons -Centre for Bioethics Papers(2003)
• Industry gifts of minimal valuehave the capacity to influencephysician behaviour, evenunconsciously.
• “Policies and guidelines thatrely on arbitrary value limits forgift giving or receipt of giftsshould be re-evaluated.”
• Industry gifts of minimal valuehave the capacity to influencephysician behaviour, evenunconsciously.
• “Policies and guidelines thatrely on arbitrary value limits forgift giving or receipt of giftsshould be re-evaluated.”
• Industry gifts of minimal valuehave the capacity to influencephysician behaviour, evenunconsciously.
• “Policies and guidelines thatrely on arbitrary value limits forgift giving or receipt of giftsshould be re-evaluated.”
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 8
Literature Review• Int. J. Radiation Oncology Biol. Phys. (2004)
• 96% of physicians admittedaccepting gifts.
• 5% agreed that they wereinfluenced.
• 33% agreed that otherphysicians were influenced.
• 96% of physicians admittedaccepting gifts.
• 5% agreed that they wereinfluenced.
• 33% agreed that otherphysicians were influenced.
• 96% of physicians admittedaccepting gifts.
• 5% agreed that they wereinfluenced.
• 33% agreed that otherphysicians were influenced.
Many Physiciansbelieve they are
immune to influence
Literature ReviewFamily Medicine (2012) JGIM (2011)
• Patients less likely totake a prescribedmedication ifphysician had recentlyaccepted a gift inreturn for listening toa pharmaceuticalrepresentative’spresentation.
• Patients whobelieve giftrelationships existreport lower levelsof physician trust &higher rates ofhealth care systemdistrust.
• Patients whobelieve giftrelationships existreport lower levelsof physician trust &higher rates ofhealth care systemdistrust.
• Patients whobelieve giftrelationships existreport lower levelsof physician trust &higher rates ofhealth care systemdistrust.
Accepting gifts risksundermining patient trust
Accepting gifts risksundermining patient trust
Accepting gifts risksundermining patient trust
Literature Review Conclusions:
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 9
What did We Put in the Policy?
New policy states:
“Physicians must not acceptpersonal gifts of any valuefrom industry or industryrepresentatives.”
What is the Evidence – ProductInformation / Marketing?
Literature Review• PLoS Med. (2010) • Studies of exposure to
information provided bypharmaceutical companieshad associations with higherprescribing frequency, highercosts, or lower prescribingquality or found no significantassociations.
• Authors found no evidence ofnet improvements inprescribing.
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 10
Literature Review• Journal of General and Internal Medicine
(2013)
• Serious adverse events were mentioned in 5-6%of promotions, although 45% were for drugswith FDA "black box” warnings of serious risks.
• Physicians judged the quality of scientificinformation to be good or excellent in 54% ofpromotions and indicated a readiness toprescribe 64% of the time.
• Serious adverse events were mentioned in 5-6%of promotions, although 45% were for drugswith FDA "black box” warnings of serious risks.
• Physicians judged the quality of scientificinformation to be good or excellent in 54% ofpromotions and indicated a readiness toprescribe 64% of the time.
What Did We Put in the Policy?
• New expectation around industry producedproduct information:
“Physicians must ensure that they criticallyevaluate any information provided by industryrepresentatives and do not solely rely on thisinformation when making clinical decisions.”
What if We Don’t Have Good Evidence?
Court Rulings
Patient and Physician Advocacy
Federal Legislation
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 11
Provisions in Marijuana forMedical Purposes Policy
• “While conclusive evidence regarding thesafety and effectiveness of dried marijuanaas a medical treatment is limited…”.
• “These expectations are grounded in theprinciples of medical professionalism set outin the Practice Guide, and take into accountthe best available evidence…”.
Thinking With Your Heart:Using Qualitative Evidence in
Policy Development @PEO
32
How Might We....? (Problem Statement)
• Increase % of online licence renewals?
• Improve customer service (in _____process)?
• Improve efficacy of CPD requirements?
• Improve practitioners’ ethical behaviour?
• Reduce #, types of complaints?
• Identify practitioner difficulties earlier?
• Think & operate more holistically?
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 12
Research on Regulatory Impacts
“...the most notable finding to emerge from this review isthe shortage of systematic knowledge on the mainresearch question...how does professional regulationaffect the behaviour of those subject to regulation?...”
Professional Standards Authority (UK)Rethinking Regulation, August 2015, p.6
34
What do we want to know?
Gain insight into perspectives, opinions,reasons of various users on
Regulatory tools & resources
Processes: Admissions, Renewals, Standards,CPD/QA, Complaints
Policies & Outcomes
35
Correlations vs. Causality
36
Source: “Spurious Correlations” website (www.tylervigen.com)
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 13
Quantitative
evidence
HOW THINGSHAPPEN
Qualitative
evidence
WHY THINGSHAPPEN
ThoughtsIdeas
Values, BeliefsMotivations
Opinions
Qualitative ToolsPROACTIVE
• Environmental Scanning• Strategic Planning• White Papers• Surveys/interviews• Social Media/Crowdsourcing• Focus Groups/Town Hall
Meetings• Advisory Boards/Councils• Communities of Interest/
Practice• Behavioural Insights/
”Nudging”• Human-Centred Design
REACTIVE• Stakeholder Feedback/
Consultation• Customer Service Audits• Compliance Audits• Complaints/Criticism• Respond to news stories/legal
cases
Behavioural Economics 101• People are irrational
decision-makers: “gut feel”,priming, habit, laziness,herding, biases, incompleteinformation, and otherinfluences (emotionalmemory, loss aversion, etc.)
• People can be ‘nudged’towards more “rational”decisions by understandingtheir feelings, values, andmotivations
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 14
“Take the Stairs instead” –4 Nudge approaches
Germany Japan SwedenSingapore
What Is Human-centred Design(Aka Design Thinking)?
HCD6 Key
Features
HCD6 Key
Features
Human-FocusedHuman-Focused
QualitativeQualitative
CollaborativeCollaborative
IterativeIterative
InteractiveInteractive
Visual/
Experiential
Visual/
Experiential
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 15
Regulatory Policy ModelsSystem-centred vs. Human-centred
Practitioner-Client/
Patient
Leg./Reg./Bylaw
Pro
cess
es
ITSyste
ms
Org. Structure
LicenceHolder/
Registrant
PRACTITIONER
CLIENT/
PATIENT/
FAMILY
44
Practitioner-Client RelationshipTECHNICAL INTERACTIONS
NON-TECHNICAL INTERACTIONS
Assessment/diagnosis/scope
Contract/treatment plan
Treatment/service/payment Feedback/Evaluation
Trust/Understanding
Compliance Expectations/Satisfaction
Communication/Dialogue/Feedback
Results/adjustments
Advertising/Choice
Safety/Legalrequirements
Comfort/reassurance
PRACTITIONER
TECHNICAL ZONE
Codes/Standards
PLI Provider
Builders/manufacturers
Suppliers (materials,software)
CLIENT/PATIENT/FAMILY
REGULATORY ZONE
Regulatory Body
Ministries, departments
Special Purpose bodies
Agencies, boards, commissions
PRACTICE ZONE
Employer
Consultants &Contractors
Allied Practitioners
FinancialEND
USERSWorkers
Public
45
GenericPractitioner’s
Ecosystem
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 16
One Size Does Not Fit All• Sole Practitioner
• Small/Medium/LargePractice
• Generalist/Specialist
• InterdisciplinaryFacility/Clinic
• Management
• Non-practising
46
Other Variables
• Location - Urban/rural
• Local/cross-state/global practice
• Age
• Gender
• # Years of Practice
• Domestic/foreign trained
• Adaptability to new technology
47
Typical Design ThinkingProcess
48
Personas
CustomerJourney
Maps
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 17
Ethnography – for empathy• Recorded fieldwork
on user experience by
– Observing/Shadowing
– “Walking in theirshoes”/simulation
– Active Listening
– Stories
– Interviews
Regulatory Service Channels Licensure (application)
Registration/inter-provincial/state transfers
Licence Renewal & Annual Reporting
CPD/QA
Professional Liability Insurance/Amenities
Practice Audits
Practice Standards/Advisory
Complaints & Discipline
Fitness to Practice
Elections
Practitioner-Centred Research (PCR)Project Purpose
• Through qualitative research, gain a deeperunderstanding of:– behaviours, attitudes, trends and challenges faced by engineers
offering service to the public throughout the province;
– efficacy of Professional Affairs tools; and
– values and drivers that motivate engineers in their work
• Identify how to improve Professional Affairstools and services and work towards a moreperson-centred and proactive PolicyDevelopment and delivery
51
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 18
Practitioner Centred Research Project | Phase 2: Practitioner Research | Final Report | May 27, 2016 52
RESEARCHSET-UP
Recruitment
Interview guide
Motivationsdeck
Screening plan
RESEARCH
12 interviews(recorded)
Transcripts frominterview audio
RESEARCH PROCESS: OVERVIEW
Transcriptsnippet sorting
12 practitionerstories
SYNTHESIS I
Themes fromtranscriptsnippets
Categories and
insightsclustered from
themes
SYNTHESISWORKSHOP
Profile patterns
Distill keyresearchfindings
Articulate 5personas
RESEARCH&
SYNTHESIS II
Validationsurvey: develop
and deploy
Understand keyresearchfindings
Identify keyopportunities
OPPORTUNITYWORKSHOP
Map potentialactivities
Personaposters
Summaryreport
RESEARCHPACKAGE
Next stepsplanning
Sample Engineer Persona
Problem Definition
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 19
Practitioner Centred Research Project | Phase 2: Practitioner Research | Final Report | May 27, 2016
Support employers toimprove their internal
training?
Leverage vastexperience to share
these experienceswith younger
audiences?
Provide practitionerswith technical
support?
Improve the public’sperception of
engineers?
Provide engineerstechnical resources?
Incorporate externalchange agents to help
current?
Provide engineersclarifications of
differences? (stds v.guidelines)
Get universities tounderstand what skills
and knowledgepractitioners really
need?
Inform engineersabout legalities all
regulatoryadministrative
structures (basics of
authority)?
Capitalize on theseprocesses to share
the knowledge base?(relying on peers,
colleagues, etc.)
Work with employersto identify entry level
job skills?
Determine engineersare valued in society?
Expand and improveguidance in ethical
issues?
Better explain thedifferences on the
website?
Obtain information onthe guidelines that are
being used?
Educate engineers toclarify PEO’s
mandate and role ofother organizations?
Engage with newlylicensed engineers to
reorient therelationship?
Inform practitionersabout organizations
that are moreappropriate for these
activities?
Improve PEO’swebsite?
Inform members ofchanging/overlapping
regulations?
Improve users
exposure to PEO
resources?
Change PAResources to make
them more accessibleto engineers?
Guide P. Engineers intraining and personal
development?
Promote regs andcodes and PA
Resources toengineers?
Help engineers tobetter understand
practice advisoryservices?
Evaluate the mostuseful formats for
learning?
Ensure youngerengineers know and
make use of practiceadvisory?
Clarify with PEO andOSPE about whose
role it is to promote &provide such
resources/ programs?
Discover theguidelines and topics
that engineers wouldfind useful?
Deal with youngengineers who are no
longer trained by theircompanies?
Better identifyemerging trends and
factors leading to newguidance?
Better targetcommunications to
promote responseand relevancy?
Connect more closelywith practitioners
above annual feepayment?
Create moreengagement between
engineers and PEO?
Educate council tofocus on regulatory
issues?
LOWIMPACT
HIGHIMPACT
How might we...
LOW EFFORT
HIGH EFFORT
IMPACT/EFFORT MATRIX CHALLENGESPotential challenges to take on as determined by the teamduring the Opportunity Finding workshop May 11, 2016
55
Professional Affairs (PA) Transformation
Overall Goal: Relationship between professional
engineers and Professional Affairs is more supportiveand collaborative
Issues to address via structuralrefinements:1. How to improve the navigation of PA services
2. How to increase awareness of PA service
3. How to make practice guidelines more useful
56
Professional Affairs (PA) Transformation
Overall Goal: Relationship between professional
engineers and Professional Affairs is more supportiveand collaborative
Issues to address through messaging &content:1. Engineers find it challenging to stay current with shifting and
overlapping regulations
2. Engineers find non-technical activities the most challenging intheir current roles
57
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 20
Reflections• very useful and eye-opening to have an
‘out- of-regulatory-body’ experience toget a more realistic, ground-levelperspective
• can be a humbling experience
• could be part of a different regulator-practitioner relationship, transformingthe organization
• challenge is how to continue andoptimize channels of dialogue 58
Evidence-based Culture Supports
• Critical importance of CEO/Board support &leadership – value of evidence-based decisions
• Gather resources and distribute/presentexamples of evidence-based policy (from othercolleges or government) if possible
• Audit your organization’s EBP readiness
• Introduce an evidence requirement intodecision documents
59
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 21
BONUS PACKAGE:ADDITIONAL RESOURCES
How EBP-Ready Are We?
All PD evidence-informed, we regularly dialoguewith stakeholders; include impact costs, andconsider alternatives to regulation
Some use of evidence, where convenient or
easily available; we talk with stakeholders at somepoint in the process
Evidence is rarely used, and usually after-the -factwhen solutions are proposed; we talk to
stakeholders to confirm direction at the end
What’s evidence?
62
Moving To An Evidence-informedCultureStaged approach
– Identify current data collection and researchresources; share data within org
– Start generating policy questions – what would youlike to know/understand better?
– Data analysis – risk factors > implications
– Pilot studies on policy initiatives to test/demonstrate value
– Baseline quantitative/qualitative data
– Discuss/promote/learning cycles
63
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 22
Selected Evidence-Based Policy Resources
• www.arnoldfoundation.org
• www.whatworkscities.bloomberg.org
• www.mitacs.ca
• www.horizons.gc.ca
• www.ahrq.gov/research/findings/evidence-based-reports
• www.sunlightfoundation.com
• www.results4america.org
Behavioural Insights UnitsWorldwide
• White House – social behavioral scienceteam
• Canada – Privy Council Office – CentralInnovation Hub
• Ontario – Treasury Board Secretariat
• UK – Cabinet Office (partnership)
• Australia – New South Wales – Premier &Cabinet
• World Bank – Global Insights Initiative
Evidence-based/informed policy –University Research Centres
• www.ohsu.edu/xd/research/centers-institutes/evidence-based-policy-center
• www.healthpolicyinstitute.pitt.edu/evidence-based-policy
• www.govlab.hks.harvard.edu/
• www.govex.jhu.edu/
• www.mowatcentre.ca
• www.inside.rotman.utoronto.ca/behaviouraleconomicsinaction
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 23
Selected DT ResourcesArticles:
• Design Thinking for Regulatory Policy, Parts 1 & 2 JordanMax, Engineering Dimensions, Jan/Feb., March/April 2016
• The Evolution of Design Thinking Harvard Business Review,September 2015
Reports:
• Design Thinking for Public Service Excellence, UNDP GlobalCentre for Public Service Excellence, 2014, Singapore
• Licence to Innovate: How Government Can Reward RiskWorking Paper 24, Institute for Competitiveness &Prosperity, Toronto, February 2016
67
Selected DT ResourcesBooks:
• Change by Design by Tim Brown
• A More Beautiful Question by Warren Berger
• Creative Confidence by Tom Kelley
68
Selected DT ResourcesWebsites and Blogs:• www.ideou.com, www.ideo.com (tools, case studies)
• www.dmi.org (articles, videos, slides)
• FastCo Design Thinking Blog
• Gov.uk Open Policy Making Toolkit (tools, case studies)
• MindLab (tools, case studies)
• Public Policy Lab (NYC)
• DIY Tool Kit (Nesta)(tools)
• Public Sector Innovation Toolkit (Gov't. of Australia)
69
Two Approaches to Evidence-Based PolicyDevelopment for Professional Regulators
Portland, OregonSeptember 16, 2016
Council on Licensure, Enforcementand Regulation 2016 Annual Educational Conference 24
US Federal Government DT Initiatives
• Lab@OPM
• HHS IDEA LAB
• DVA Center for Innovation
• USAID Global Development Lab
• US Digital Services