student affairs assessment a comprehensive, centralized approach

STUDENT AFFAIRS ASSESSMENT

A comprehensive, centralized approach

Lori Varlotta, Vice President for Student AffairsSacramento State

WASC Conference on Assessment

Irvine, CA

October 16-18, 2008

Learning Outcomes for Today’s Participants

This session will help participants:• Differentiate outcome based assessment

from satisfaction based assessment• Articulate the potential benefits of

assessment in student affairs• Write direct learning outcomes that measure

learning goals• Identify useful types of assessment and

evidence-gathering tools

What is Assessment?

Sacramento State’s Division of Student Affairs sees assessment as an ongoing process primarily aimed at one of two things: 1) improving programs or services, and 2) understanding and eventually increasing student learning.

To reach those goals, the Division must:

• Make program objectives and student learning outcomes explicit and public

• Set appropriate criteria and high expectations; systematically gather, analyze, and interpret evidence to determine how well programs and services matches those expectations and standards

• Use the resulting data to document, explain, and improve programs, services, and student learning outcomes

Expanding the Definition of Student Learning

At Sacramento State University, the Division of Student Affairs “defines student learning as a comprehensive, holistic, transformative activity that integrates academic learning and student development, processes that have often been considered separate, and even independent of each other.

When we say learning, then, we do not mean exclusively or primarily academic instruction, the acquisition of disciplinary content, or classroom learning.”

(quoted directly from Learning Reconsidered: A Campus-wide Focus on the Student Experience)

Why Do It?

Assessment will allow us to:

• Demonstrate how Student Affairs contributes to student learning and student success

• Showcase our strengths

• Identify areas where we can improve

• Strategically align ourselves with – Chancellors’ Office Graduation Initiative– CSU Access to Excellence– University planning – Destination 2010– WASC requirements to demonstrate the impact of Student Affairs

on student learning

• Respond to public calls for increased accountability– (e.g. VSA College Portrait)

Other Good Reasons to do Assessment

• The Vice President of Student Affairs holds each director responsible for conceptualizing and implementing an assessment plan

• The directors’ annual evaluation includes commentary on assessment

• Assessment data has supported requests for additional staff and/or funding

Role of Institutional Research

To support the vision of the Vice President of Student Affairs and assist Student Affairs Directors (via on-going consultations, one-on-one meetings, survey development and data analyses) in developing departmental assessment plans that are aligned with Destination 2010 and the WASC re-accreditation.

Utilizing On Campus Expertise

Role of other Campus Experts

Why this Particular Assessment Approach is So Exciting

• A true collaboration with departments within and outside the Division of Student Affairs (Institutional Research, ASI, program units, Provost, academic departments)

• Comprehensive and centralized – a level of consistency throughout the Division and uniquely designed for each department

• Directly tied to the Chancellor’s Office Graduation Initiative, WASC requirements, Strategic Plan, etc.

• Access to Excellence• Moves beyond student satisfaction to student learning

Six Step Assessment Model

1. Mission— 2-3 sentences that articulate office name, primary purpose, primary activities, audience.

2. Goals— top 3-5 “planning-type” statements.

3. Program Objectives and Learning Outcomes— 3-5 specific (i.e. measurable) statements.

4. Measures—Direct measures can verify learning outcomes, attitudinal change or behavior modification. Indirect measures typically reflect student satisfaction and/or self-reported understandings.

5. Results — Include a brief narrative of findings and/or essential tables or graphs. The results and subsequent analysis should indicate the extent to which the program objective or student learning outcome was met.

6. Conclusion —Summarizes the collection and analyses of data and then "closes the loop" by identifying what decisions or program modifications are or will be made on the basis of these analyses.

Step 1 – Mission Statement

The departmental or program mission should be an expanded statement of either the institutional or divisional purpose.

Extract from University Mission Statement: California State University, Sacramento is dedicated to the life-altering potential of learning that balances a liberal arts education with depth of knowledge in a discipline. We are committed to providing an excellent education to all eligible applicants who aspire to expand their knowledge and prepare themselves for meaningful lives, careers, and service to their community..…

Academic Advising Center MissionThe Academic Advising Center offers new student orientation, mandatory freshman advising, and advising on General Education and graduation requirements for all students. The Center engages students in a developmental process that helps clarify and implement individual educational plans consistent with their skills, interests, and values. Through individual appointments, group advising sessions, and presentations the professional staff, faculty advisors, and student interns help students understand the university’s academic requirements as well as its policies and procedures. As a result, students are better prepared to take responsibility for their education and persist towards a timely graduation.

Primary Activities

Primary Purpose

Target Audiences

Office Name

Step 2 – Goals

Goals:

Are broad statements that describe the overarching long-range intended outcomes of an administrative unit.

Example: Help students clarify and implement individual educational plans which are consistent with their skills, interests, and values.

Are usually not measurable and need to be further developed as separate distinguishable outcomes.

Example: Prepare students to take responsibility for their education and persist towards a timely graduation.

Step 2 – Goals (cont’d)

Goals:

Are primarily used for general planning and areused as the starting point to the development andrefinement of outcomes.

Example: Help students gain an understanding of the university’s academic requirements as well as its policies and procedures.

Step 3 – Objectives

Objectives:

Are specific statements that describe desired outcomes derived from the goal statements of the unit.

Are typically one of two types: program objectives or student learning outcomes.

Example – Program Objective: All freshmen will participate in a three-phase comprehensive, proactive advising and major/career exploration program by Spring 2008.

Example – Student Learning Outcome: 70% of selected students will be able to demonstrate knowledge of their college preparatory requirements and the resources available to assist them.

Step 3 – Objectives (cont’d)

Objectives:

Program objectives help staff identify how aprogram, workshop, or activity can be modified or

improved; while student learning outcomes reveal the changes in attitudes or behaviors that a student user can describe or demonstrate after utilizing a service or program.

Components of Learning Outcomes

• Audience: For whom is the program aimed?

• Behavior: What do you expect the audience to know/be able to do?

• Conditions: Under what conditions or circumstances will the learning occur?

• Degree: How much will be accomplished, how well will the behavior need to be performed, and at what level?

Indirect Learning Outcomes

Indirect Learning Outcomes - self-reported statements/ comments that reveal a perceived increase in understanding or appreciation. The perception is not verified through any demonstration of knowledge acquisition or observed behavioral / attitudinal change.

Methods to Assess IndirectLearning Outcomes

• Satisfaction surveys • Program / evaluation surveys • Questionnaires• Inventories• Face-book responses• Informal peer-to-peer conversations (e.g. with

RAs, orientation leaders)

Direct Student Learning Outcomes

Direct Student Learning Outcomes – The abilities, information retention, knowledge acquisition, attitudinal or behavioral changes that students can demonstrate after participating in a program or utilizing a service

Methods to Assess DirectLearning Outcomes

• Portfolios• Capstones• Performances• Common assignments• Narratives with reflection• Tests• Competency observations• Observations of employee or student

behaviors• Juried art exhibits• National licensure examinations• Standardized tests• Work samples

Differentiating Direct and Indirect Learning Outcomes

Direct or Indirect?1) T / F This orientation session has helped me

understand the foreign language requirement at Sacramento State.

2) Which of the following examples fulfill the foreign language requirement necessary to graduate from Sacramento State?

A) Demonstrated fluency in a language other than English

B) Passed the AP foreign language exam with a score of 3 or higher

C) Successfully completed, with C- or better, 3 years of high school foreign language

D) All of the above

Changing Indirect Learning Outcomes to Direct Learning Outcomes

Distribute Orientation Survey Handout

Initiate group activity

Direct Student Learning Instrument

Distribute Post-Test Orientation

Assessment Handout

Step 4 – Measures

• SLO – Orientation Participants: Students who complete the orientation post test that measures their short term retention of information related to GE Advising, Academic requirements and campus resources will score 85% or better. The post test will be administered at the end of the 2-day orientation program.

• SLO – Orientation Leader: Orientation leaders observed in a twenty-minute structured role play exercise will earn ‘4’ or better in each of competency areas described on the rubric developed for this training.

• Program Objective – Orientation: 95% or students and parents will indicate on their program evaluation that they were ‘satisfied’ or ‘very satisfied’ with all aspects of the orientation.

Step 5 – Results

The results should highlight all significant findings and indicate the extent to which the program/service reached its intended outcomes.

Example: Freshman Orientation pre/post tests showed that out of the 11 questions asked, 6 of the questions fell short of the 85 percent goal of knowledge acquisition. However, of those 6 questions not meeting the goal, there still was significant improvement between the pre-test and post-test responses.

Step 6 – Conclusions

The conclusions should explain how the findings from data will be used to improve the program and/or increase student learning.

Example: The Orientation Coordinator will use the data collected to identify the areas that need to be emphasized in the student training sessions offered to orientation assistants.

Suggested Reading

1. The National Association of Student Personnel Administrators & The American College Personnel Association. (2004). Learning Reconsidered: A campus-wide focus on the student experience.

2. American College Personnel Association (1994). The student learning imperative: Implications for student affairs. Washington, DC.

3. American College Personnel Association. (1996) The student learning imperative: Implications for student affairs [on-line]. Available: www.acpa.nche.edu/sli/sli.htm

4. American College Personnel Association and National Association of Student Personnel Administrators (1997). Principles of good practice for student affairs [on-line]. Available: www.acpa.nche.edu/pgp/principle.htm

5. Joint task Force on Student Learning. (1998) Powerful partnerships: A shared responsibility for learning [on-line]. Available: www.aahe.org/teaching/tsk_frce.htm

6. National Association of Student Personnel Administrators (1987). A perspective on student affairs. Washington DC.

Other Resources

WASC website http://www.wascweb.orgCAS Standards http://www.cas.edu/gags.htmlKansas Study Website http://Kansasstudy.orgNational CC Benchmark Project http://www.NCCBP.orgPowerful Partnerships: A Shared Responsibility for Learning

http://www.aahe.org/assessment/joint.htmStudent Learning Imperative

http://www.acpa.nche.edu/sli/sli.htmAssociation of Theological Schools http://www.ats.eduSacramento State’s VSA http://www.csus.edu/checkusout/Sacramento State’s Student Affairs’ Assessment

http://www.csus.edu/wascaccreditation/Assessment.stm