task and process: assessment and evaluation of outreach efforts

Task and Process: Assessment and Evaluation of Outreach Efforts

Gary D Glass, Ph.D Duke University

Forrest Seymour, MSW, LICSW Keene State College

Teresa Michaelson-Chmelir, Ph.D, LMHC University of Central Florida

Eva G. Schoen, Ph.D

University of Iowa

Chris Meno, Ph.D

Indiana University

Task and Process: Assessment and Evaluation of Outreach Efforts

Why evaluate?What are the challenges?(Dealing with the “UGH!”)

What are steps forward?

Why are we here?AUCCCO Research subcommittee

charge:Explore options in developing measures to assess the quality & value of outreach, particularly workshops & presentations.

Ongoing process & dialogue:We are looking for your input and feedback.Why are you here?

Our Frame and Quest…

Empires of Outreach

1. Identifying/Clarifying Goals of Outreach• Introducing therapeutic elements and

messages into the campus environment• Marketing Counseling Center • Providing skills and knowledge• Increasing Awareness of mental health

and student development issues• Responding to Acute Needs in

Community (crises/tragedy/trauma)

Our Frame and Quest…

2. Reviewing Modalities of OutreachDelivering PresentationsProviding Training to professional

colleagues and student para-professionals

Facilitating DiscussionsOrganizing Experiential/Interactive EventsProducing/Distributing Content (print,

electronic, multimedia, etc)Building/Maintaining Liaison Relationships

w/ constituenciesScreenings/Early Detection Activities

Our Frame and Quest…

3. Reviewing Methods/Tools of Conducting Assessment

• Printed Evaluation Forms• Online Evaluation Forms• Tablet/On-site computer

tools• Accessing institutional data

(including, obviously, counseling center

Our Frame and Quest…

UGH!

Our Players and Problem

The Players:◦ UCF: 60,000+ CAPS: Clinical Staff: 20 Outreach

Committee: 7 members (Psychologists, Mental Health, SW, and a doctoral intern); meeting bi-weekly

The Problem: ◦ We needed to better evaluate the effectiveness of

outreach◦ But there is just so much variation in what we define

as “outreach” (psychoeducational/developmental programs, tabling's, signature events, etc)

◦ The OTHER variables, WHAT do we want to measure? Does programming increase utilization of services? Did we teach new skills/awareness?? Did our outreach program improve retention???

Shifting from “UGH, there is too much!!” …to….”ok, we got this.”

As a committee, we decided to focus on one area of our outreach ….psychoeducational / developmental programming.

We began to think of aaaaalllllllll the program titles we have done, and coded them under topics headings

Initial action steps……

As you can see….we struggled with duplication. Some topics just seemed to fit under several areas….

A second look….We took another look at the

themes and decided that we could narrow them down a bit further AND thought more about how WE want to define our programs…. make it fit with our strategic plans, our mission, our whole existence!!!

We can define ourselvesOur overarching presentation

themes became: Academic SuccessSocial ConnectionsWellness and Life SkillsMental Health AwarenessDiversity Awareness

UCF CAPS Outreach Website

Now what? In order to evaluate our efforts,

we had to identify what we wanted to know about our services….◦Overall effectiveness of workshop in

achieving success at UCF and in life◦Topic specific value◦Likelihood of using the

skill/information in the future

Evaluation Questions: Strongly Strongly Agree Agree Neutral Disagree Disagree *I learned ___________ that will help me be

more successful at UCF or in other areas O O O O O of my life.

* I learned _______ to improve my ______ O O O O O *I intend to use what I learned in this

presentation. O O O O O

Learning Objectives:Overarching:

1. To learn in this workshop that will help me be more successful at UCF and in my life.

Topic Specific :

2. To learn to improve my (Academic Success, Social Connectedness, Wellness and Life Skills, Mental Health Awareness,

Diversity Awareness)

SkillsInformationSkills and Information

Task and Process: Identifying Variables Of Effectiveness

What would actually go on the forms that

we use to evaluate our programs?

Variables of Effectiveness:

1. Outcome Measures Related to Change In Attendees As a Result of the Program/Presentation

2. Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation

3. Macro-Level Outcome Measures (Related to Prevalence and Severity of issues directly or indirectly addressed by Counseling Center)

Our Process: From A Single Standardized

Form to a Series of Variables to

Choose From

Outcome Measures Related to Change In Attendees

Student reports Intention/Plan to Implement (specified) Change After Program

Student Gained Knowledge/Information (re: topic of program)

Student Gained/Improved Skills from Program

Student Increased/Gained Awareness or Insight (regarding Self and Self-in-the Context of his/her environment)

Student reports achieving SPECIFIED Desired Change in Experience or Behavior

Outcome Measures Related toChange In Attendees

Student reports achieving SPECIFIED Desired Change in Experience or Behavior◦ Student reports decrease in specified

problem indicators (i.e., stress, symptoms, behaviors, etc)

◦ Student reports increase in specified resilience or satisfaction indicators (i.e., positive self-concept, improved sleep, increased focus, etc).

◦ Student Reports Experiencing Connection With/Among Other Attending Students

◦ Student Reports Improved Efficacy in Being Change/Support Agent to peers and community

Outcome Measures Related to Change In Attendees

Students indicates change in attitude• Regarding topic/behavior (specified in

particular program)• Regarding counseling center• Regarding help-seeking in general

Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation

Overall Rating of Presenter Overall Quality/Value of ProgramOpen Ended Questions on Program (+ and –) Produced Desired Knowledge and Attitudes

of Counseling Center Assessment of whether program was Cross-

Culturally Competent/Attended to Diversity

Macro-Level Outcome Measures Overall Reduction in Symptoms/Incidence of dysfunction or

impairment in local clinical data (LCD) Overall Reduction in Prevalence of Problematic Behaviors in

campus data Overall Increase in presence of individual resilience or

satisfaction Factors in institutional data Increased utilization of counseling center resources for high

risk or populations that under-utilize services Decrease utilization of counseling center resources for low-

severity issues that are identified as linked to problematic campus environment norms.

Overall Increase in Campus Activity Toward Promoting/Creating Supportive Campus

Decrease in instances of bystander-apathy/increase in instances of bystander-intervention (whenever someone calls to consult concerned about a student they are worried about, how many were students calling about other students, etc.)

Outreach Messages Showing up in student circles

Where To Go From Here?Examining and Respecting Process so

Concrete Examples emerge Maintaining Flexibility and Creativity as

Organization to accommodate different staff and program/presentation styles with need for organizing structure (some “required” measures?)

Reminder not to allow structure to stifle creativity and expansion of what constitutes outreach

What do YOU think?Questions and Comments?Continue the Conversation

◦Interested in participating in the subcommittee?

◦Provide input on a google.doc for follow up comments and questions to this workshop? (sign email list and we’ll send you the

link!)https://docs.google.com/document/d/1lVyAtAQs0BVYitDzYNx6lailDn5VaOpSRM6RK288QTc/edit?

usp=sharing