addictions foundation of manitoba developing an evaluation of an accountability process in a...

Addictions Foundation of Manitoba

Developing an evaluation of an accountability process

in a provincial addiction agency

David Patton, PhD.



Overview

Goals of the presentationInform processPresent a model for discussionOffer insights into our process


AFM structure

C orpora te R esources R esearch andQ uality

M onito ring

SM RR egional D irector

p lann ing team

W om en's ServiceAdult R ehab.

SM Rsupervisor

p lann ing team

Youth Servicescounse llingprevention

schoo l-based program sgam bling

com m unity education

Prevention& Education

SM RC onsu ltant

p lann ing team

Adult R ehab

SM Rprogram supervisor

p lann ing team

G am bling

C EO &M anagem ent

Team


Planning Teams

Developed to provide consistent direction for each program areaManagement representative (SMR), chair usually supervisor of the program, staffWork together to develop accountability model – one of their goals (logic model?)Research Unit used as facilitator/guide to the processReviewed by Management Team re: consistency with organizational goals


Developing Accountability Models

Begin by asking staff a series of questions about programs (to generate content of each model).

Who are the target clients?Assessment process (how do we determine that the program is appropriate?)Identify short-term goals (by the end of their program) How can we tell when a client has achieved these goals?What activities do staff engage in to achieve the short term goals?


Identify long-term goals (3 – 6 mths)How can we tell when a client has achieved these goals?How can we link short term goals with long term goals?How are these consistent with AFM’s mandate?

Developing Accountability Models


Model of Program AccountabilityModel of Program Accountability

Staff-Client Activities

Short-Term Objectives

Long-Term Goals


Logic model developed

Different from the accountability modelsNot program basedMore top-down (based on business plan)Link with strategic directions (staff also have input into this process)


Goals within planning teams

Identify expected outcomes for programsLink these to accountability modelsDevelop (with research group) measures/research protocolsContinuous Improvement loop (feedback & pilots)Use feedback/pilot data to improve programs – not here just yet


Identify expected outcomes for programs (examples)

Increased Knowledge about AODG (as appropriate)Better coping skillsRealistic rehab planReduced harmful use (may include abstinence)


Link these to activities

What is it that the staff do?Mostly counseling/educationSome planning around process in clients’ livesGroup sessions with speakers (e.g., financial planners for residential gambling program)Link with support groups


Develop measures/research protocols

How would be be able to tell if clients are half way towards meeting their goals? – staff can tell us.Sometimes standardized measures can be used, sometimes we create our own (e.g., SFI) NOT client satisfaction measures


Continuous Improvement loop


Use data to improve programs and services

Mostly pilot projects – evaluation reports used to convince partners to continue to fund (e.g., school-based projects, new gambling residential rehabilitation program)

Starting some work with family therapists

Services also being evaluated – e.g. library brochures, media links


Example of ModelExample of Model

Show video

Teaching Sessions

1 to 1 discussion

Understanding the impacts of

gambling

Increased Knowledge about how gambling

worksReduced

Harm from Gambling

Short-Term Objectives

Staff-Client Activities

Long-Term Goals


Example: Gambling Residential Program

Long-Term Goals, Short-Term Goals, and Program ActivitiesPre-and Post-Tests for Short-Term Goals

Several indicators for each main short-term objective

Benchmark for Long-Term GoalsThese are asked of the client again 3 to 6 months after program involvement


Benefits of Program Benefits of Program Accountability ModelsAccountability Models

Program development and Evaluation

Identify specific gaps in agency’s program offerings

Job descriptions are more clear

Referring agencies can be made aware of each program’s specific activities and objectives


Practical issues for counseling-based programsMechanism for ongoing program improvementContribution to Best Practices literature

Issues and challengesIssues and challenges


Summary

Management buy in to processPlanning teams (include staff, supervisors and management rep)Develop trust with line staffListen to their inputWork with the staff to develop models


Thank you

Comments/questions?

addictions foundation of manitoba developing an evaluation of an accountability process in a...

Documents