evaluation and policy: how are they connected? geoff nelson, psychology, wilfrid laurier university...
TRANSCRIPT
Evaluation and Policy: How Are They Connected?Geoff Nelson, Psychology, Wilfrid Laurier UniversityTim Aubry, Psychology, University of OttawaAimee Watson, Mental Health Commission of Canada
Agenda
2:45-3:00 – introductions and expectations for the workshop3:00-3:10 – policy 3:10-3:20 – evaluation3:20-3:30 – knowledge transfer3:30-3:50 – small group discussion3:50-4:00 – reporting back on small groups
PolicyThe renewal and repurposing of HPS• March 21, 2013, federal budget announcement, Economic Action Plan
2013 proposes $119 million per year over five years, nearly $600 million in total, to the Homelessness Partnering Strategy using a “Housing First” approach.
• Building on the successful At Home/Chez Soi project, the Government will continue to work in partnership with provinces and territories, communities, the private sector and other stakeholders, such as the Federation of Canadian Municipalities, to reduce homelessness. Economic Action Plan 2013 proposes $119 million per year over five years for the Homelessness Partnering Strategy using a Housing First approach. Over the coming months, the Minister of Human Resources and Skills Development will work with Government partners at the community level to confirm the implementation details of this approach.
PolicyThe renewal and repurposing of HPS• The evidence shows that the Housing First approach: can be
implemented across Canada; improves the lives of those who are homeless and have a mental illness; and makes better use of public dollars, especially for those who are high service users—as demonstrated by a reduction in the use of hospital in-patient care, emergency rooms, police detentions and justice services.
• The project will develop a body of evidence to help provinces and territories provide services to homeless people living with a mental illness. In addition, funding arrangements are being established to ensure that people participating in the project will have supports in place once the project is completed.
PolicyHow did this renewal and repurposing of HPS happen? Evaluation and trusted messengers
• At Home/Chez Soi research and MHCC knowledge translation• The conservative heartland – the Alberta experience and the
emergence of the Canadian Alliance to End Homelessness led by Tim Richter
• Looking south of the border – US research on Housing First, support from the G.W. Bush administration, development of 10-year plans to end homelessness in many US cities
PolicyHow did this renewal and repurposing of HPS happen? Political will
• Dissatisfaction with progress in reducing homelessness in Canada• Review of HPS provided an opportunity to refocus its approach• Desire to do a better job of reducing homelessness in Canada• An existing funding mechanism and program (HPS)
Program Evaluation (Rossi et al., 2004)Definition
“The systematic collection of information about the activities, characteristics, and outcomes of programs”
Why Evaluate?1. to make judgements about the program, 2. to improve program effectiveness, 3. to inform decisions about future programming”
PROGRAM MISSION PROGRAM PROCESSES
PROGRAM OUTCOMES
NEEDS INPUTS INITIAL OUTCOMES
GOALS & OBJECTIVES
VALUES & PRINCIPLES
ACTIVITIES
OUTPUTS
INTERMEDIATE OUTCOMES
LONGER-TERM OUTCOMES
RELATIONSHIP TO THE THREE MAIN TYPES OF PROGRAM EVALUATION
NEEDS ASSESSMENT
IMPLEMENTATION / PROCESS
EVALUATION
IMPACT / OUTCOME EVALUATION
Make-Up of a Program
Housing First Program Mission Program Processes Outcomes Individual:- Reduce HOUSING + Housing Stability Homelessness FIRST + Functioning + Quality of life Community: - Chronic & episodic
homelessness
Housing First: Assessment of Fidelity
Housing First:Assessment of Fidelity
Options1. Self-asessment of fidelity (Gilmer,
Stefanic, Sklar, & Tsemberis, 2013)2. External assessment of fidelity
(Mental Health Commisssion of Canada, 2014)
Outcome Monitoring“The ongoing collection and analysis of information on indicators of how well programs are performing with regard to the achievement of results” (Rossi, Freeman, & Lipsey, 1999).
Standardized self-report measures Simple counts of client status Collateral rating scales Consumer satisfaction measures
Types of Outcome Measures
Housing First: Outcome Indicators & Potential
Measures1. Housing status of clients (Simple count)2. Community functioning (Multnomah Community
Ability Scale – Collateral rating scale)3. Quality of Life (Lehman’s Quality of Life Interview
– Standardized self-report measure)4. Consumer Satisfaction Measure – (Service
Satisfaction Scale)
A PIT is a snapshot of sheltered and unsheltered homeless people in a community on a single night.
Provides a baseline Identifies needs of the populatioin Enables you to measure community level
outcomes
Helps identify priority populations
Community-Level Outcomes Point-in-Time Counts
Why do a count?
Canadian Point-in-Time Count
Methodology and Toolkit
Piloted in Seven Cities in AlbertaOctober 16, 2014
Who to Count?OPERATIONAL CATEGORY LIVING SITUATION COUNTED
1 Unsheltered o People living in public or private spaces without consent or contract ✔
1.2 People living in places not intended for permanent human habitation ✔
2 Emergency Sheltered
Emergency overnight shelters for people who are homeless ✔
2.2 Violence-Against-Women (VAW) shelters ✔
2.3 Emergency shelter for people fleeing a natural disaster or destruction of accommodation due to fires, floods etc.
3 Provisionally Accommodated
3.1 Interim Housing for people who are homeless ✔
3.2 People living temporarily with others, but without guarantee of continued residency or immediate prospects for accessing permanent housing.
3.3 People accessing short term, temporary rental
accommodations without security of tenure 3.4 People in institutional care who lack permanent
housing arrangements. ✔ 3.5 Accommodation / Reception centres for recently
arrived immigrants and refugees 4 At-Risk of
Homelesness
4.1 People at imminent risk of homelessness
4.2 Individuals and families who are precariously housed.
1
Mandatory questions
• Screening • Consent
• Gender• Age• Ethnicity• Migration• Immigration• Homelessness History• Veteran Status
Using Program Evaluation to Develop and Improve Housing First Programs
1. Conduct regular fidelity assessments 2. Integrate outcome monitoring into service delivery3. Conduct regular point-in-time counts4. Provide training of service providers on data
collection5. Provide regular feedback on collected data6. Keep analyses simple7. Make program improvements based on collected
data
Knowledge Exchange“Evidence of effectiveness alone is rarely enough to ensure adoption of interventions” (Leff & Mulkern, 2002)
What is Knowledge Exchange?The Canadian Institutes of Health Research defines KE as: a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system www.cihr irsc.gc.ca/e/29418.html‐
Knowledge Exchange
How do you do it?Moving Knowledge to Action:
• What?• Who?• How?
Synthesis
Dissemination
Exchange
- 1 and 3 pagers, summaries, early findings
- media, community events, social media
- Engagement, training, collaboration
Knowledge ExchangeInfluencing Policy
Lessons from the At Home/ Chez Soi experience:
Throughout At Home/Chez Soi, the project team engaged in an ongoing process to work with government. Strategies included:• Worked with all levels of government (municipal, provincial, federal)• Regularly held in-person meetings and briefings throughout course of
project• Developed briefing material- PowerPoint presentations, Early Findings
reports• Used key messages to ensure consistent and focused messaging• Resulted in strong government engagement in At Home project sites
and ongoing policy change
Focusing the Frame
- 1 and 3 pagers, summaries, early findings
- media, community events, social media
- Advisory panels, collaboration, meetings
A Participant Photography Project
My Kitchen__________________________I love to cook. On the street it's hard to get a decent meal. Now my fridge is full and I can choose what and when I eat. I'm so much healthier now, and I get so much satisfaction in doing something I love to do. I can even invite people over for a meal. Beats the hell out of a food kitchen.
View from my Window________________________
I am so grateful for this view. I am also grateful to not have anyone able to knock on my
window at all hours of the night. I have some sense of safe.
Focusing the Frame
- 1 and 3 pagers, summaries, early findings
- media, community events, social media
- Advisory panels, collaboration, meetings
A Participant Photography Project
Strings____________________
The name “chaos theory” comes from the fact that the systems that the theory
describes are apparently disordered. But Chaos Theory is really about finding the
underlying order in seemingly random data.
The “butterfly effect” is a sensitive dependence on initial conditions. Just a small change in the initial conditions can
drastically change the long term behaviour of a system...
I am the system
Small Group Discussion Questions• How can your communities incorporate evaluation into your Housing
First programs?• What knowledge transfer strategies could you use with policy-makers
in your jurisdictions to promote the adoption of Housing First