2007 Assessment Symposium

WELCOME!

May 22, 2007

Study Abroad Center helping students gain global perspective

Lisa LobergDirector, Study Abroad

Continuous Improvement Project

Process• Purpose: – To review coordination between campus offices as

student participation increases– Examine systems to create new procedures

• Team Representatives:– Business Office Registrar– Institutional Research Study Abroad– International Admissions

• Outcomes: – Created Datatel codes for all programs and screen– Entirely new system of budgeting and accounting for

department– Building relationships and improving communication

• Review: – Team to review implemented changes and make new

recommendations as needed

Welcome Back Survey Purpose

Evaluation of:1. Specific Study Abroad Program

• Academics• Location• Housing• Activities• Overall experience

2. Study Abroad Center• Effectiveness • Suggestions for improvement

3. Impact of student experience • Evidence of student ability to

• Demonstrate Cultural Competency • Growth in Identity and Values• Appreciate Diversity

2006-07 Survey Design Goals

• Review previous data and questions

• Incorporate questions that measure Student Learning Outcomes

• Convert paper-based survey to electronic Flashlight survey

Design Chronology• Prior to 2004: No Study Abroad Experiences Survey.

• Spring 2005 1st version: – Paper survey with all open-ended questions. – Useful but difficult to tabulate results.

• Spring 2006 2nd version: – Paper survey with both open-ended questions and rank-order

scale.– Rank-order helpful; needs development.

• Fall 2006 3rd version: – Paper survey with open-ended questions and rank-order scale

with scale levels defined. – Provided more accurate information; needs collection

improvement.

• Spring 2007 Current version: – Electronic Flashlight survey with open-ended questions and

rank-order scale. Incorporated specific questions to measure student learning outcome goals.

– Results tabulated immediately. Facilitates reporting.

Analysis of DataStudy Abroad Emerging Themes• Experiences are life-changing. • Cultivates global perspective and respect for people of

diverse cultures. • Increases students’ ability to Demonstrate Cultural

Competency, Grow in Identity and Values, and Appreciate Diversity.

Lessons Learned• Creating surveys in Flashlight is simple and is effective. • Flashlight results are tabulated immediately and easily

reported. • Attention to survey design enriches quality of responses

and ability to use data.• Survey design can capture the achievement of Student

Learning Outcomes.

2007-08 Assessment Goal

• Every returning student completes a survey

• Incorporate additional questions relating to Student Learning Outcomes

• Add a demographics section to make national reporting of statistics even easier

Criminal Justice-Program Review

Robert J. MeadowsHelen Ahn Lim

COAS - Department of Criminal Justice

CJ Program Features• Approximately 100 majors • Offers B.S. in Criminal Justice

(42 units) • Offers Minor in Legal Studies

(20 units)• Comprehensive with

interdisciplinary focus and cross listing of courses (sociology, psychology, political science)

• One of fastest growing UG majors

• Internship requirement• Alpha Phi Sigma honor society

• Two full-time faculty Robert J. Meadows, Ph.D.,

Ed.D. ; Professor and Program Chair

Helen Ahn Lim, Ph.D.; Assistant Professor

• Three to four adjunct instructors

• Approximately 100 majors • About 10-12 classes each

semester with an average class size of 20 students

2006-07 Program Review Goal

To assess how the Program prepares students for employment or post graduate study, to identify graduate’s perceptions regarding the value of their degree and courses, and to learn of their education/employment progress through alumni surveys

Data CollectionBeginning September 2006, examined:

• Curriculum information and course syllabi

• Program goals• CJ and CLU Student

Learning Outcomes

• Trends (Enrollment and Faculty-Student Ratios)

• Library resources• Alumni survey data• Review by external reviewer

Alumni Survey Data Collection and Analysis

• Collected data from 2000-2006 CJ graduates

• 98 graduates identified (46 usable questionnaires)

• Questionnaire responses

1. Employment of alums (33/46 or 71% in jobs related to CJ) including law enforcement, corrections, law, security and Government jobs related to justice.

2. Alums attending or completing post graduate studies (14/46 or 38%) including degrees in Business (MBA), Psychology, Education, and Criminal Justice.

Alumni Survey Data Collection and Analysis

• Questionnaire responses

3. Degree preparation for career (36/46 or 78% believed the degree somewhat prepared or prepared them very much for their present vocation).

4. Degree preparation for graduate or professional studies. (All respondents attending post grad studies believed the degree prepared them for post graduate study).

5. Overall reflection of learning experience (39/46 or 85% indicated that their educational experiences and courses taken in criminal justice were either excellent or good).

Analysis of DataWhat did the data say?

• The degree provides a relevant and valuable education • Overall, curriculum is consistent with Program goals, CJ

and CLU Learning Outcomes • Higher scores than most other programs for Challenge and

Engagement in Arts and Sciences• Most diverse student profile in Arts and Sciences

What themes emerged?• Problem with understaffing given the number of majors

What was learned?• Alumni Survey: Most graduates are employed in relevant

field; several attend graduate or law school. Demonstrates the value of senior seminar course and internship requirement.

• Courses: Consistent with major criminal justice programs in other schools. However, Alumni Survey suggests that more courses would further enrich the program.

• CLU is the only private residential school offering a four-year degree in Ventura, Los Angeles and Santa Barbara Co.

Goals and Recommendations

Outcomes of Program Review and Measurable Goals

• Comprehensive and unique program (meets CJ goals and aligns with CLU Learning Outcomes)

• Experienced tremendous growth and is a highly marketable degree

• Need to hire more full-time faculty in the future (in order to expand curriculum, and further enhance the quality of the program)

• Provide more leadership opportunities for students (group projects, CJ honor society)

Recommendations for growth based on analysis

• Hire more full-time faculty• Improve facilities and offices

Connecting the Dots:School of Business Assessment

System An OverviewCharles MaxeyBonnie Johnson

Key Problem Points

•Continuous Improvement Model

•Institutional School or Department Program Course Assignment Assessment Data Collection Reporting, Discussion, Analysis Educational Effectiveness and Continuous Improvement

Professional Preparation

• Field Specific Knowledge and Experience• Ethical and Professional Judgment• Interpersonal and Teamwork Skills

Liberal Learning• Critical Thinking• Information Literacy• Written Communication• Oral Communication• Ability to comprehend issues from disciplinary and

interdisciplinary perspectives• Understanding of cultural and global diversity• Integration of knowledge with ethical reflection

Character and Leadership Development• Growth in Identity and Values• Appreciation of Diversity• Service to the Community• Principled Leadership

Professional Preparation• Field Specific Knowledge and Experience• Ethical and Professional Judgment• Interpersonal and Teamwork Skills, Principled

Leadership• Planning and Organization• Statistical Data Analysis

Liberal Learning• Creative and Critical Thinking• Information Literacy and Computer Skills• Written Communication• Oral Communication and Listening Skills• Evaluation

• Understanding of cultural and global diversity• Creation of Value

Character and Leadership Development

Student Learning OutcomesCalifornia Lutheran University and CLU School of Business

(Alignment and Expansion)

University: School of Business:

Standard Pages

Following the Process

•Sample Syllabus: Bus367 – Behavior in Organizations (taught by Dr. Susan Murphy)

•Sample Assignment Description from Bus367 – Personal Application Assignment

•Sample Rubric for Assessing Personal Application Assignment

•Sample Report for Further Departmental Discussion and Continuous Improvement

CLU’s Assessment System

Library Services Review

Cindy Campbell

Director of Library Operations and Systems Information Systems & Services

2006-07 Assessment Goal

To evaluate Pearson Library services for the purposes of improvement: – Library Services in general– Library collection– Library space– Information literacy instruction

Assessment Design

Measurements• LibQual Survey• MyCLU Portal Survey

Participants • Stratified random sampling • University population• Confidence interval of 90%

Data CollectionLibQual• Survey Levels and Scale

– Minimum, Desired, & Perceived– 1 to 9 with 9 being the most favorable

• 22 core items in 3 areas of service provided by the ISS Library Service group:

Affect of Service Library as Place Information Control

MyCLU Portal Survey

Analysis of Data with Resulting Goals

Emerging Themes

Resulting Goals

Community space

Evaluate how we are currently using the library spaceLibrary space

Lighting Evaluate lighting issues

Superiority = Perceived – Desired• Employees who are consistently courteous

• Giving users individual attention

Adequacy = Perceived – Minimum

Analysis of Data with Resulting Goals

Emerging Themes

Resulting Goals

Library Collections

Increase our on-line resources Update our print resources

Information Literacy

Publicize our information literacy program at a higher level

Website navigation

Redesign our website

Department of PhilosophyProgram Review and Evaluation

William Bersley, Chair and Associate Professor

Nathan Tierney, ProfessorXiang Chen, Professor

The Philosophy Department conceives of philosophy as an enterprise of both mind and spirit. The faculty are committed to providing the

student with the knowledge and skills of critical and reflective thinking, moral judgment, and understanding of major philosophical traditions and perennial philosophical questions. Such knowledge and skills not only enable the student to become a philosophically literate person, but facilitate the integration of this knowledge with each student’s process of moral, spiritual and intellectual growth.

2006-07 Program Review Goal

To compare departmental grade scores to mean grade distributions for Arts and Sciences and Humanities Division.

Assessment Design

Data collected and examined in Review

• Trends (Enrollment and Faculty-Student Ratios)

• Teaching Effectiveness• Scholarly Productivity• Challenge and

Engagement• Critique of Curriculum• Department Student

Learning Outcomes

• Demand data• Load data• Capacity data• Student evaluation

data Grade distribution data

• Comparison with similar programs

• Student Profiles• Curriculum Map

DataGrade Distributions (4 point scale)

Spring ‘05

Fall ‘04

Spring ‘04

Fall’03

Spring ‘03

Fall ‘02

Philosophy

3.21 3.15 2.70 2.98 3.25 3.25

TraditionalUndergrads (School)

3.23 3.16 3.22 3.15 3.25 3.17

TraditionalUndergrads (Division)

3.14 3.08 3.17 3.06 3.20 3.21

Analysis of Data• Departmental grade distributions were comparable

to both Arts and Sciences (School) and the

Humanities Division, with the exception of Spring ‘04 and Fall

‘03 which were somewhat below the averages of the

School and Division.

• External review of this (and other) data suggests creation of departmental/internal assessment strategy to improve quality, e.g., longitudinal evaluation of capstone writings and supplementary survey/questionnaires for core required courses to determine value to undergraduate education and skills transferable to other courses.

Goals and Recommendations• Provide faculty with current data concerning grade

distribution in order to prevent grade inflation (see previous data)

• Sustain highly evaluated “vital teaching” and articulate the centrality of Philosophy within the university

• Increase the role of Philosophy in the Honors Program

• Implement the use of Philosophy 400 to initiate a Philosophy Honors Program/Capstone Seminar (requires staffing and budgetary support)

• Develop an assessment strategy that authentically improves Philosophy on its own terms

• Increase mentoring of sustained student projects

• Implement the establishment of a Center of Global Ethics under the leadership of Dr. Nathan Tierney

The Academic Assistance Program

(The AAP) An essential piece of our retention strategies

Damien A. Peña, MSWSenior Director of Academic Programs

• Academic Assistance Program (AAP) • Retention strategy for Probation students

and Conditionally Admitted students• Modeled after student success plans in

Student Support Services• Evaluates each student’s academic

progress through administered assessments

• Self-funded (advising fee assessed to students)

What is the AAP?

2006 – 2007 Program Goals

To provide Academic Assistance (strategies and skills) to:

• Students on Probation– To be removed from probationary status

• Conditionally Admitted Students– To achieve academic success at CLU

• Self-selected Students– To stay “ahead”

Design and Data CollectionTriangulated Strategy • Exit Interview

– Gather student perceived performance as a result of being in the program

– 1 on 1 with the Academic Counselor – 5 questions– 93 responses out of 99

• Exit Flashlight Survey– Gather student perceived performance using anonymous

survey during the last session– Walked student to computer– 9 Open ended questions– 17 scale questions concerning services and the CAAR

Office– 93 responses out of 99

• Retention Data– Determine number of students removed from Probation– Data pending until grades submitted

Analysis of Data• Students’ perceived outcomes/goals

– Wanted to improve grades– Wanted to improve study habits

• Positive impact on students– Accountability– Connection with someone on campus

• Challenges of program– Communication with Faculty– Capacity issues– Library hours

2007-2008 Program GoalsAddressing Challenges• Communication with Faculty:• Capacity: Outside funding to address

capacity issues• Library Hours:

In Addition• Assess AAP performance using benchmarks

identified as indicators of success• Conduct longitudinal study using quantitative

and qualitative data (retention data & exit surveys)

• Expand services for self-selected students• Achieve 100% return rate for student

evaluations

Athletic Facility Scheduling

Dan Kuntz, Director of Athletics

Coaches, Assistant Coaches, Athletics Administrative Assistant

2006-2007 Assessment GoalGoal/ProblemTo develop and implement an Athletic Facility Master

Schedule incorporating internal and external requests for:Gilbert Arena Samuelson Aquatics CenterSoiland Recreation Center Tennis courtsGame and practice fields

To evaluate Effectiveness of Athletics Facility Scheduling Plan

Factors and limitations• 19 sports programs / nearly 400 student athletes / 6 practice sessions per

week per program / between August 15 and May 15 / overlapping seasons. • 18 sports / 4 to more than 25 home contests (No track facility for Track and

Cross Country home contests).• Single venues used by multiple sports teams limits schedule options.• Status of coaches and class schedules.• PT v FT - afternoon hours for practices.• Lack of lighted facilities for outdoor sports. • SCIAC schedules set days and times.

Qualitative and Quantitative Research Design

Methodology

• Data collected annually

• Collaborate with Athletic facility users to discover needs for facility use

• Data Collection Measures:Electronic facility request forms, e-mails, phone calls, meetings, observations, conference schedules, practice times, internal and external special event requests

• Population: Potential users, internal, communityAthletics Academics Student ActivitiesAdmissions ESSM Classes Alumni Relations Conferences and Events Renters Church Relations Residence Life Club sports Marketing and CommunicationsIntramurals Recreation Multi Cultural / International ProgramsOpen facility requests for students, faculty, staff, alumni and CLA community members

Qualitative and Quantitative Research Design

Variables and ConstraintsWeather, Academic Schedules of Students, Athletic Facility Maintenance, Student Employment Staffing, Special Events, Facility quantity /quality /design.

Practices and/or Games by Months:Baseball: January - May + OctoberM/W Basketball: October - MarchFootball: October – Nov + AprilM/W Soccer - August - Dec + AprilM/W Swim and Dive: Sept - MarchM/W Tennis: January - May

Softball: January - May + OctoberVolleyball: August - November + AprilNo track athletes off campus:M/W Cross Country: Sept - November Golf: February – MayM/W Track and Field: January - May

Data Analysis Prioritization Measures: Requests for scheduling time filtered

1. Academic2. Athletic3. Student Activities

4. General University Requests5. Outside University Requests

Analysis of DataData Reveals Emerging Themes• Requests growing from internal and external community users for access to

CLU Athletic facilities.• New emerging ideas for Athletic facility usage.

Additional factors• Maintenance costs are climbing.

Lessons Learned• There is confusion about scheduling needs and processes.• Diplomacy is necessary.• Education and communication is needed in the scheduling process-

tolerance. • Centralized scheduling is important.• Open access to Athletics facilities can be limited by rental of facilities.

Future Challenges• Class scheduling. Lack of lights on fields. Lack of sufficient quality and

quantity of fields. Weather variables. League requirements. Special access. • Demand for Athletic facility access and usage will likely grow internally and

externally.

2007-08 Assessment Goal2006 – 2007 Outcomes• Internal and external Athletic facility schedules are coordinated. • Users are satisfied with access.

Growth Recommendations 1. Centralize Athletic facility scheduling for efficiencies to potentially

reduce conflicts, miscommunication, and misunderstanding.2. Prioritize scheduling of Athletic facilities for CLU students, faculty, staff

and alumni despite growing pressures for revenue production from facility rentals.

3. Increase personnel resources. 4. Purchase scheduling software.5. Implement On-line scheduling.

Measurable goals for 2007 - 2008• Access and flexibility for CLU program offerings sustained.• Provide more student, faculty, alumni and staff access to Athletic

facilities.• Attract sports events and activities that bring prospective students and

families to CLU.• Develop innovative community outreach programs that attract visitors

to see the value of CLU programs.

Sharing Responsibility for Multiple Learning OutcomesNew partnerships across Academic Departments and Student

Affairs

Dr. Silva KarayanDirector, Center for Academic Service-Learning

Stine OdegardCoordinator, Community Service Center

Sharing Responsibility for Multiple Learning Outcomes

Purpose:To enhance CLU’s educational effectiveness in a global era, focus on the integration of multiple learning outcomes.

Service-learning pedagogy facilitates mission-focused integrative learning through intentional collaboration:• Academic Affairs Office• Academic Departments• Center for Academic Service-Learning (CASL)• Community Service Center (CSC)

Key Partnerships on Campus• Community Service and Co-curricular Service-learning

(CSC)– One coordinator, 4 student workers (10 hours/week)– 2 week-long trips, 4 weekend experiences, 35 day experiences– Co-curricular service-learning: action-reflection model

• Center for Academic Service-learning and SLO’s (CASL)– One director, many faculty partners from various departments– Ongoing faculty development opportunities & individualized mentoring – High quality service-learning project components:

– Paradigm Shift: From “Deficit Model” to “Empowerment Model”– Orientation Shift: From “Charity and benevolence” to “Learning and

making a difference”

• Integrative learning• High Service• Student voice

• Collaboration• Reflection• Evaluation

Assessment Goal and DesignSurvey Goal To determine student interest and discover the meaning of their

experience in the March 10, 2007 Community Service Day (extra-curricular activity)

Survey Design and Data Collection• Pre administered on-site, paper format (67 respondents)• Post survey (Flashlight) administered via email (42 respondents)• Review of BCSS/NSSE data• Qualitative and quantitative data

Demographics of 117 participants• M/F ratios: 23 males (17%), 94 females (80%) • Fields of study: (Bus.-12%, Comm.-12%, ESSM-8%, Lib. Studies-

5%)• Previous service involvement:

65% participated in service at CLU before85% participated in service during high school

Analysis of Data• Q4: “Overall, my involvement in Service Day was

important to me because:”

• Q3 and Q5: Student likes and needs:Structured events, social events, meaningful tasks

• BCSS(2004)/NSSE (2005):

Service to the CommunityHighLow

41%50%Low

8%1%HighE

ngag

emen

t

Expectation

2007-08 Assessment Goals• Measurable goals for 2007 – 2008

– Offer more integrated service opportunities (meet student expectations)

– Increase number of faculty incorporating service-learning pedagogy

– Provide additional faculty development and mentoring opportunities

– Expand number and variety of community partners

• Strategy to achieve goals– Intentional collaboration through shared SLO’s and integrative

learning• CSC: co-curricular and community service focus• CASL: academic service-learning focus

– Reach a greater number of faculty and support their efforts to incorporate service-learning pedagogy to achieve SLO’s

Ambassadors for a Peaceful Multicultural World

A Peer Facilitated Diversity Workshop Program

2005/2006 2006/2007

Our Team Amber Scott and Juanita Hall – program

coordinators & trainers (through Anti-Defamation League)

Our Student Ambassadors 11 (05/06); 12 (06/07)

2006-07 Assessment GoalsOur Goals:• To foster an appreciation for diversity • To improve the campus climate for diversity

Our Study : • To determine the effectiveness of the workshops in achieving

our goals– What student participants learned throughout 2 years of workshops – Did their learning fit our identified Learning Outcomes

• To discover which exercises achieved the identified learning outcomes.

Assessment DesignWe collected post-workshop reflections from

participants at the close of each workshop:• Students responded to the following prompt:

“Please tell us what you learned about yourself or others as a result of your participation in this diversity workshop.”

• Reflections separated by workshop and Ambassador teams: for the Ambassador’s own personal reflections and self-evaluation, and to match Learning Outcomes with particular exercises.

• Reflection packet was dated and labeled with faculty names, class, and facilitators.

Data Collection

Reflections (312) from:• 9 workshops in 2005/2006 (132)• 10 workshops in 2006/2007 (180)

Where: majority in freshmen seminar classes; others included leadership retreats, a student government meeting, and individual classes.

Analysis of DataProcess• Read for themes, coded, quantified • Inter-rater reliability (2 readers)

Results• Student reported their learning matched Learning

Outcomes– All 3 “Appreciate Diversity” Learning Outcomes – 4 of the 5 “Grow in Identity and Values”

What was learned?• Workshops are useful for the character development of

our students • Workshops help enhance the campus climate for

diversity • Certain diversity exercises were more effective than

others in promoting student growth and appreciation of diversity

Analysis of Data

2005/2006 Workshops 132 Student participants

Appreciate Diversity Grow in Identity and Values

Understand social importance of diversity

Awareness of similarities and differences

Increased respect & understandingfor people ofdifferent cultures

Learned nothing new (already knew)

Understand & express sense of responsibilityto community & world

Understand & express identity in terms of theirdiversity

Awarenessof self in relationship to own background & others

Compassion & commitment to justice

Code 1 Code 2 Code 3 Code 4 Code 5 Code 6 Code 7 Code 8

32 57 27 12 10 29 47 292006/2007 Workshops 180 student participants

41 63 18 9 3 39 47 31

2007-08 GoalsRecommendations • Ambassador workshops to occur in all

freshmen seminars• Use the most effective exercises in the

workshops

Assessment Goal• A greater # of students will report Learning

Outcomes:1. Understand and express an increased

understanding and respect for people of different cultures

2. Demonstrate both compassion toward others and a commitment to justice

3. Understand and express their sense of responsibility to their community and the world

2007 Assessment Symposium

Learning Together

May 22, 2007