task and process: assessment and evaluation of outreach efforts
DESCRIPTION
Task and Process: Assessment and Evaluation of Outreach Efforts. Gary D Glass, Ph.D Duke University Forrest Seymour, MSW, LICSW Keene State College Teresa Michaelson-Chmelir , Ph.D, LMHC University of Central Florida Eva G. Schoen, Ph.D University of Iowa Chris Meno , Ph.D - PowerPoint PPT PresentationTRANSCRIPT
Task and Process: Assessment and Evaluation of Outreach Efforts
Gary D Glass, Ph.D Duke University
Forrest Seymour, MSW, LICSW Keene State College
Teresa Michaelson-Chmelir, Ph.D, LMHC University of Central Florida
Eva G. Schoen, Ph.D
University of Iowa
Chris Meno, Ph.D
Indiana University
Task and Process: Assessment and Evaluation of Outreach Efforts
Why evaluate?What are the challenges?(Dealing with the “UGH!”)
What are steps forward?
Why are we here?AUCCCO Research subcommittee
charge:Explore options in developing measures to assess the quality & value of outreach, particularly workshops & presentations.
Ongoing process & dialogue:We are looking for your input and feedback.Why are you here?
Our Frame and Quest…
Empires of Outreach
1. Identifying/Clarifying Goals of Outreach• Introducing therapeutic elements and
messages into the campus environment• Marketing Counseling Center • Providing skills and knowledge• Increasing Awareness of mental health
and student development issues• Responding to Acute Needs in
Community (crises/tragedy/trauma)
Our Frame and Quest…
2. Reviewing Modalities of OutreachDelivering PresentationsProviding Training to professional
colleagues and student para-professionals
Facilitating DiscussionsOrganizing Experiential/Interactive EventsProducing/Distributing Content (print,
electronic, multimedia, etc)Building/Maintaining Liaison Relationships
w/ constituenciesScreenings/Early Detection Activities
Our Frame and Quest…
3. Reviewing Methods/Tools of Conducting Assessment
• Printed Evaluation Forms• Online Evaluation Forms• Tablet/On-site computer
tools• Accessing institutional data
(including, obviously, counseling center
Our Frame and Quest…
UGH!
Our Players and Problem
The Players:◦ UCF: 60,000+ CAPS: Clinical Staff: 20 Outreach
Committee: 7 members (Psychologists, Mental Health, SW, and a doctoral intern); meeting bi-weekly
The Problem: ◦ We needed to better evaluate the effectiveness of
outreach◦ But there is just so much variation in what we define
as “outreach” (psychoeducational/developmental programs, tabling's, signature events, etc)
◦ The OTHER variables, WHAT do we want to measure? Does programming increase utilization of services? Did we teach new skills/awareness?? Did our outreach program improve retention???
Shifting from “UGH, there is too much!!” …to….”ok, we got this.”
As a committee, we decided to focus on one area of our outreach ….psychoeducational / developmental programming.
We began to think of aaaaalllllllll the program titles we have done, and coded them under topics headings
Initial action steps……
As you can see….we struggled with duplication. Some topics just seemed to fit under several areas….
A second look….We took another look at the
themes and decided that we could narrow them down a bit further AND thought more about how WE want to define our programs…. make it fit with our strategic plans, our mission, our whole existence!!!
We can define ourselvesOur overarching presentation
themes became: Academic SuccessSocial ConnectionsWellness and Life SkillsMental Health AwarenessDiversity Awareness
UCF CAPS Outreach Website
Now what? In order to evaluate our efforts,
we had to identify what we wanted to know about our services….◦Overall effectiveness of workshop in
achieving success at UCF and in life◦Topic specific value◦Likelihood of using the
skill/information in the future
Evaluation Questions: Strongly Strongly Agree Agree Neutral Disagree Disagree *I learned ___________ that will help me be
more successful at UCF or in other areas O O O O O of my life.
* I learned _______ to improve my ______ O O O O O *I intend to use what I learned in this
presentation. O O O O O
Learning Objectives:Overarching:
1. To learn in this workshop that will help me be more successful at UCF and in my life.
Topic Specific :
2. To learn to improve my (Academic Success, Social Connectedness, Wellness and Life Skills, Mental Health Awareness,
Diversity Awareness)
SkillsInformationSkills and Information
Task and Process: Identifying Variables Of Effectiveness
What would actually go on the forms that
we use to evaluate our programs?
Variables of Effectiveness:
1. Outcome Measures Related to Change In Attendees As a Result of the Program/Presentation
2. Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation
3. Macro-Level Outcome Measures (Related to Prevalence and Severity of issues directly or indirectly addressed by Counseling Center)
Our Process: From A Single Standardized
Form to a Series of Variables to
Choose From
Outcome Measures Related to Change In Attendees
Student reports Intention/Plan to Implement (specified) Change After Program
Student Gained Knowledge/Information (re: topic of program)
Student Gained/Improved Skills from Program
Student Increased/Gained Awareness or Insight (regarding Self and Self-in-the Context of his/her environment)
Student reports achieving SPECIFIED Desired Change in Experience or Behavior
Outcome Measures Related toChange In Attendees
Student reports achieving SPECIFIED Desired Change in Experience or Behavior◦ Student reports decrease in specified
problem indicators (i.e., stress, symptoms, behaviors, etc)
◦ Student reports increase in specified resilience or satisfaction indicators (i.e., positive self-concept, improved sleep, increased focus, etc).
◦ Student Reports Experiencing Connection With/Among Other Attending Students
◦ Student Reports Improved Efficacy in Being Change/Support Agent to peers and community
Outcome Measures Related to Change In Attendees
Students indicates change in attitude• Regarding topic/behavior (specified in
particular program)• Regarding counseling center• Regarding help-seeking in general
Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation
Overall Rating of Presenter Overall Quality/Value of ProgramOpen Ended Questions on Program (+ and –) Produced Desired Knowledge and Attitudes
of Counseling Center Assessment of whether program was Cross-
Culturally Competent/Attended to Diversity
Macro-Level Outcome Measures Overall Reduction in Symptoms/Incidence of dysfunction or
impairment in local clinical data (LCD) Overall Reduction in Prevalence of Problematic Behaviors in
campus data Overall Increase in presence of individual resilience or
satisfaction Factors in institutional data Increased utilization of counseling center resources for high
risk or populations that under-utilize services Decrease utilization of counseling center resources for low-
severity issues that are identified as linked to problematic campus environment norms.
Overall Increase in Campus Activity Toward Promoting/Creating Supportive Campus
Decrease in instances of bystander-apathy/increase in instances of bystander-intervention (whenever someone calls to consult concerned about a student they are worried about, how many were students calling about other students, etc.)
Outreach Messages Showing up in student circles
Where To Go From Here?Examining and Respecting Process so
Concrete Examples emerge Maintaining Flexibility and Creativity as
Organization to accommodate different staff and program/presentation styles with need for organizing structure (some “required” measures?)
Reminder not to allow structure to stifle creativity and expansion of what constitutes outreach
What do YOU think?Questions and Comments?Continue the Conversation
◦Interested in participating in the subcommittee?
◦Provide input on a google.doc for follow up comments and questions to this workshop? (sign email list and we’ll send you the
link!)https://docs.google.com/document/d/1lVyAtAQs0BVYitDzYNx6lailDn5VaOpSRM6RK288QTc/edit?
usp=sharing