Taking Evidence Seriously: Assessment as a Tool to Tell a Story

Taé Nosaka, Assistant DirectorLory Ann Varela, Research Coordinator

Key Learning CommunitiesCenter for Advising & Student

Achievement

Key Communiti

es and Objectives

Outcomes-Based

Assessment

Telling the

Story

Results

Closing the

Loop

Key Communities

CLUSTER COURSES

RESIDENTIAL COMMUNITY

(Braiden Hall)

FEEDBACK/ CONNECTION

Peer Mentors

Key Communities Structure

Social Integration

Academic Integration

Intrusive Advising

EARLY

ORIENTATION

Objectives of Key Develop a strong community with an

academic focus Provide opportunity for shared learning Provide access to feedback, information,

and opportunities Foster a healthy, diverse communities

in which learning and academic achievement are encouraged and respected

Offer enriched opportunities for involvement and leadership development

Objectives (cont’) Promote high academic

performance (GPA) Promote high retention and

graduation rates Promote academic engagement and

campus involvement Promote student satisfaction with

their university experience

Outcomes-Based Assessment Outcome

› What do we want Key to accomplish? Strategy

› What is in place to achieve that outcome? Assessment Method

› How are we going to measure that outcome?

Criteria› How will you know if the “numbers” are

“good”?

Key Outcome Example #1

Key Communities Outcome #1› Students in the Key Communities will have higher

retention rates then similar students not participating in Key

Strategy› Students meeting 1:1 with Key Mentors three times per

semester, Mid-Semester Academic Progress Conferences, Taking Stock at Mid-Semester Early Warning, Academic Success Workshops (2 mandatory), Referrals to campus resources, Co-enrollment in cluster classes.

Assessment Method› First-Year Retention Data

Criteria› Key students will have higher first-year retention rates

Key Outcome Example #2

Outcome› Students in the Key Communities will have an increased

awareness of diversity in terms of: 1) Opportunities to learn about other cultures by participating in Key 2) Interacting with students from backgrounds different from their own, 3) Having a greater understanding of people different from self because of experience in Key 4) Attending cultural events outside of their own culture.

Strategy› Living in a diverse community, discussing topics around diversity

in KEY 192 classes, requiring students to attend at least 1 cultural activity per semester

Assessment Method› Student reports on Fall and Spring Program Evaluations

Criteria› At least 80% of students on all four goals

Key Outcome Example #3 Outcome

› Determine the components of the Key Academic Community that students find most beneficial in the fall semester in order to develop a structure for the spring semester that will help alleviate the “spring semester slump” that has been experienced

Strategy› 1) Co-enrolling in Cluster Classes with Key students› 2) Having a Key Mentor› 3) Living in Braiden Hall with Key students› 4) The Key Seminar Class› 5) The Plan for Excellence filled out during Key Orientation› 6) Key Academic Success Workshops (Time Management, Study Skills, etc)› 7) Moving in early› 8) Mid semester Academic Progress Conferences› 9) Resident Assistant› 10) Key Orientation

Assessment Method› Fall Program Evaluations for 2 years

Criteria› Determine top three most beneficial components

Telling the Key Story

Key

Student’s Experienc

e

Faculty Experienc

e

Key Mentor

Experience

Retention and Academic

Performance Data

Program Evaluations: the Students’ Story

Every Fall and Spring Semester Paper Copy given in Seminar Classes

(fall) Student Voice (spring) Specific questions addressing

outcomes Sample What do we learn from these?

Highlights from Fall 2007 Program Evaluations

92% response rate 82% of Key students were satisfied

with their decision to participate in Key

30% said that Key was a large factor in their decision to stay at CSU (retention)

Highlights from Fall 2007 Program

Evaluations 90% indicated that Key was

welcoming of students of their background and experience (sense of belonging)

89% noted that they had the opportunity to interact with students from backgrounds different from their own (diversity)

Online Survey: The Story of our Faculty

Fall 2007 using Student Voice Highlights

› 100% satisfied with their experience teaching

› Motivated by students, being asked, extra salary

› Most positive experience: class discussions, students in smaller groups, students’ growth

› Challenges: motivating students› Incentives: higher compensation

Key Mentor Focus Groups: The Mentors’ Story

Fall 2007 using 2 Focus Group with 17 staff (20 total) Highlights

› 94% indicated they would return to position› Valuable experiences include meaningful connections

with people, skills in relationship and advising, and personal growth and development

› Skill development reported with counseling, organizational, communication, balance, boundaries and expectations

› Challenges included a disconnect with T.A. position› Recommendations on training, advice for new staff on

developing relationship, and the meaningfulness of Peer Mentor role

Retention and Academic

Performance DataRetention Tool developed by CASA

Semester GPA information

Key Academic Community First Year Retention FA03-FA06 Cohorts

All FA03 FA04 FA05 FA06

83.16%

86.34%

83.24%85.14%

77.98%

81.57% 82.14% 82.03%81.03% 81.06%

Key Academic CSU

Key Academic Community First Year Retention, Students of Color

All FA03 FA04 FA05 FA06

84.93%88.24%

85.42% 86.05%

80.00%78.82% 77.48%

82.98%

77.44% 77.41%

Key Academic CSU

Key Service Community First Year Retention, Non-Residents

All FA05 FA06

79.98%81.82%

78.13%

73.92% 73.87% 73.97%

Key Service CSU

Key Plus Community

Persistence to 3rd FallAverage rates for Key Plus 2001-2003 Entering Classes and CSU

1998-2001 Entering Classes

95.3%

85.6%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Persistence from 2nd Fall to 3rd Fall

Key Plus CSU

Key CommunityAcademic Performance

Key Academic and Service GPA Compared to CSU

22.12.22.32.42.52.62.72.82.9

3

FA05 FA06 FA07

2.9 2.92

2.842.78

2.69 2.66

Key

CSU

Closing the Loop

Using the evidence to change or improve our programs

Examples of Program Improvement or Change

Key Academic Community = Spring Seminar

Key Plus Community = Options “Office Hour” to “Administrative Hour” Stool in the Key Mentor Office TA Training for Mentors Consolidating Key Communities in

Braiden Hall

Other sources for assessing Key

Taking Stock at Mid-Semester

National Study of Living Learning Programs (NSLLP)

Key Alumni Survey

What is the first thing you would do to start

assessing your program?

Worksheet

Questions?