kathy s. williams, director kathy a. krentler, associate director

Kathy S. Williams, DirectorKathy A. Krentler, Associate Director

2

Michelle Lopez, Assistant Dean

Division of Undergraduate Studies

Jill Esbenshade, Associate Professor

Department of Sociology

Teresa Cisneros-Donahue, Study Abroad Director

College of Business Administration

Do High Impact Activities Really Make an Impact?

What are High Impact Activities?

An investment of time and energy over an extended period that has unusually positive effects on student engagement in educationally purposeful behavior.

(George Kuh, Indiana University)

3

Examples of High-Impact Activities

1st Year Seminars and Experiences Common Intellectual Experiences Learning Communities Writing-Intensive Courses Undergraduate Research International Experiences Service Learning/Community-Based

Learning Internships Capstone Courses and Project

4

The Value of High-Impact Activities (SELF-REPORTED)

On Learning

General Gains

Personal Gains

Practical Gains

Learning Communities

Service Learning

Study Abroad

Undergraduate Research

Culminating Experiences

5

P < .001

George D. Kuh, High-Impact Educations Practices: What They Are, Who Has Access to Them, and Why They Matter; Washington , DC; AAC&U, 2008.

The Value of High-Impact Activities (As measured at SDSU by

Graduation Rate)STUDY ABROAD1st Time

Freshmen Fall 2005

Studied Abroad

No Study Abroad

4-Year Graduation Rate

32.9% 26.2%

5-Year Graduation Rate

78.0% 51.0%

6-Year Graduation Rate

90.2% 60.7%

6

The Value of High-Impact Activities (As measured at SDSU by

Graduation Rate)INTERNSHIPS1st Time

Freshmen Fall 2005

Internship No Internship

4-Year Graduation Rate

45.5% 24.6%

5-Year Graduation Rate

84.7% 50.4%

6-Year Graduation Rate

93.2% 59.4%

7

But the “next level” question is:What is the impact of HIP

on direct measures of student learning?

What student learning outcomes do we expect from High-Impact Activities?

How can we measure whether these outcomes are occurring?

8

Michelle Lopez

ACADEMIC SCHOLARSHI

PS

Academic/Merit

National Competitive Scholarships

Some require endorsement by SDSU

Required 3 – 5 faculty recommendation letters (depending on type)

Essays or Proposals

Require participation in various HIPS

Office of Academic Scholarships (OAS)

Establishing SLOs for the OASFirst, asked a few key questions…

What is the Office of Academic Scholarships trying to do and why?

What is my office supposed to accomplish?

What will I do to promote the kind of learning and development that our campus seeks?

How will student learning be influenced?

What do I want students to be able to do or know as a result of completing the scholarship or fellowship process?

11

Establishing OutcomesThrough interacting with the OAS, students will be able to:

1 Identify which HIP aligns with their personal/professional interests, to strengthen their overall scholarship/fellowship application

2 Locate and use appropriate resources, to develop scholarship ideas or projects

3 Explore long-term professional goals and connect these goals back to immediate scholarship/fellowship purpose

4 Write, evaluate, and rewrite written materials based on collaborative feedback

5 Interact with faculty, administrators, and other referrees in a timely manner

6 Demonstrate thoroughness and timeliness in completing all scholarship/fellowship applications

7 Translate overall significance of project in laymen's terms

12

After drafting outcomes…

Get feedback from others

How do we know that the OAS is contributing to student learning?Where is the content delivered that

will allow the SLOs to be met?How will we measure SLO

achievement?

13

What is being done to directly/indirectly facilitate student learning?

Outcome Info Sessions

IndividualAdvising

FacultyAdviser

LibraryCourse Guide

Identify which HIP aligns with their personal/professional interests, to strengthen their overall scholarship/fellowship application

X X X

Identify which HIP aligns with their personal/professional interests, to strengthen their overall scholarship/fellowship application

X X X

Explore long-term professional goals and connect these goals back to immediate scholarship/fellowship purpose

X X

Write, evaluate, and rewrite written materials based on collaborative feedback

X X

14

Importance of Assessing Student LearningGreat reminder to:

Reflect on the OAS mission and goals

Check to see how student learning outcomes align with national fellowship organization or SDSU’s mission and goals

Reflect on how OAS contributes to SDSU priorities

Informs prioritization on time spent, as well as other resources

Discuss how achievement is determined 15

Lived Experience

of Immigrants and RefugeesJill EsbenshadeSociology and Honors

Course Goals To give students an understanding of and an

ability to engage in conversation and debate about immigration and the lives of immigrants . . .

For students to develop a “sociological imagination.” That is for students develop the ability to apply academic concepts and theories to individual lives . . .

For students to improve their writing skills . . . For students to develop their abilities and

desire to use their knowledge to engage in public debate and action to improve their community or the broader society.

Class Structure Background: First Section Historical and Current

Flows (migration patterns) Numbers US policy US in comparative perspective

Focus: Experience and Identity

Disruptions, Changes and Adaptation/Coping in:

Section 2: Class and Labor Section 3: Gender and Family Section 4: Race and Assimilation

“Lived Experience” Sociological Concepts

Applied to Fiction and Non-fiction Movies Internship

Internship: International Rescue Committee Students Plus Program: Crawford High School

ESL Program Homework Tutoring CASHEE Prep Peacemakers College Prep

Higher Learning Navigator

Issues with Internship Agreement with SDSU

Hours (one component of course) Reduced 30 to 20

Scheduling conflicts Civic Action Option

Grade/Evaluation (25% of course) Assessment

Assessment Before Class Discussion

Feedback Session Wrap-Up Session

Papers (4 of 5) Incorporate internship into analysis or as

separate point

Journal Connect to class material

Obstacles to Assessment Connecting internship to class material

Type of interaction at internship Hesitancy to ask Tendency to describe rather than analyze

Strengths of Experiences Clear affect on student outlook:

Open-mindedness Sympathy Judging others

Building relationship Group outings

Current Assessment: Civic Capacities From CSL document (simplified and modified) Broad ideas about growth as citizen (rather

than CSL as service) Conscious development of capacities Journals

Connections with material or capacity Development of capacity

Most Common Capacitiesin Journal Check1. Build a sense of community, locally and

globally Work with people from different

backgrounds Develop curiosity and capacity to listen Cultivate open-mindedness and

appreciation of worth of each person Enhance concern for community and the

wellbeing of others

Also, capacities 2, 4 and 5

Teresa Donahue-Cisneros

STUDY ABROAD

College of Business Administration

Cross Cultural Training for US and EU students (CROCUS)Project

• 2009, Grant by the Department of Education• Required an evaluation plan about student’s

learning • Major specific objectives:

1. Increase participants’ cultural and2. language learning

College of Business Administration

Cross Cultural Training for US and EU students (CROCUS)Project

• 2009, Grant by the Department of Education• Required an evaluation plan about student’s

learning • Major specific objectives:

1. Increase participants’ cultural and2. language learning

College of Business Administration

Student Learning Outcomes of the CROCUS Project

1. Gain knowledge of other cultures.

2. Use knowledge, diverse cultural frames of reference, and alternate perspectives to think critically and solve problems.

3. Become conscious of one’s own cultural perspective.

4. Accept cultural differences and tolerate cultural ambiguity.

5. Improve target language competency in speaking, listening, reading, and writing.

Rubrics developed by American Council on Education (ACE) and Department of Education, FIPSE

College of Business Administration

Opportunities to Learn

1. Study abroad experience

2. Online intercultural communications course “Crossing the Bridge”

College of Business Administration

Assessing student learning in the CROCUS Project

1. Used assignments in the “Crossing the Bridge” Course

2. Exit interviews with students

College of Business Administration

Assignments in the CourseStudents were asked to complete written assignments

that demonstrated their accomplishment of the SLOs.

• SLO: “Gain knowledge of other cultures.”• Assignment: “Write a two-page paper (500 words or

more) about your host country. Accumulate facts about the country’s history, political situation, economy, religion, international relations etc. Ensure that you include current events in your paper (elections, economic ties etc.).”

College of Business Administration

Student Interviews• Students completed in-depth interviews

upon return from Study Abroad1. Describe what you learned about yourself?2. Describe what you learned about your host

country?

College of Business Administration

Assignments in the Course & Interviews

• Rubric developed to measure whether assignment demonstrated the SLOs

• Set benchmark – 75% of the students should be able to achieve a Moderate level (score of 3 in the rubric)

• 2 independent raters read each assignment and score it using the rubric.

• Scores reviewed for inter-rater reliability• Determination made as to whether the benchmark was

achieved.

Gain knowledge of other cultures 1 Inadequate

(descriptions

are inaccurate or

poorly developed)

2 Minimal

(Describes

basic points

accurately)

3 Moderate

(Compares and

contrasts perspectives;

uses examples to

illustrate)

4 Extensive

(Content

knowledge is extensive;

analyses are sophisticated)

1. Cultural practices (e.g., religious, secular, political, governmental, educational, family structures.)

College of Business Administration

Do They Know It?Interesting Outcome – Students scored

higher when assessed via interviews than when assessed via course assignments.

CHALLENGE: Achieving cross-measure reliability.

SLO1 SLO2 SLO3 SLO4 Average Coursework Score of Both Reviewers 2.65 2.80 2.75 2.94 Average Interviews Score of Both Reviewers 3.05 3.15 3.05 3.26

College of Business Administration

If they don’t know it, how do we fix the problem?

• Only 60% of students showed evidence of having gained cultural knowledge, below the 75% acceptable benchmark.

• Programmatic Assessment asks not “what’s wrong with these students?” but rather, “how can we strengthen the program to produce a better result?”

College of Business Administration

CLOSING THE LOOP1. Did the program deliver the content needed

to allow students to accomplish the SLOs?• Curriculum mapping can help to answer

this question.

2. How does the program need to change to improve things?

Goal Crossing the Bridge Course

Study Abroad Experience

Pre-Departure Workshop

Re-Entry Workshop

Gain Knowledge of other cultures.

Introduced, Reinforced

Reinforced, Applied Introduced Reinforced

Components of Crosus Program

40

The Cross Cultural Training for US and EU students (CROCUS) Assessment Rubricsi Knowledge Rubric The below items are designed to assess various knowledge areas that are evident in the student’s portfolio information and/or materials. Please note that each portfolio begins with a Portfolio Information Form. Ratings should be based upon the information provided by the student in the Portfolio Information Form as well as the actual items from the portfolio. Please base your ratings on the following scale: 1 = Inadequate; 2 = Minimal; 3 = Moderate; 4= Extensive.

Gain knowledge of other cultures 1 Inadequate

(descriptions

are inaccurate or

poorly developed)

2 Minimal

(Describes

basic points

accurately)

3 Moderate

(Compares

and contrasts

perspectives; uses examples

to illustrate)

4 Extensive

(Content

knowledge is extensive;

analyses are sophisticated)

1. Cultural practices (e.g., religious, secular, political, governmental, educational, family structures.)

2. Cultural beliefs and values

In addition to your ratings, please provide additional information regarding the student's performance in this area. Information regarding why you assigned the ratings you did—as well as specific examples from the portfolio—would be particularly useful.

Become conscious of one’s own cultural perspective

1. Self in cultural context (e.g., aware of one’s own origins, history, ethnic identity, communities, etc.).

2. The history of his or her own culture in relation to the history of other cultures.

In addition to your ratings, please provide additional information regarding the student's performance in this area. Information regarding why you assigned the ratings you did—as well as specific examples from the portfolio—would be particularly useful.

i Adapted from The ACE-FIPSE SPIF/e-Portfolio Assessment Rubric

41

Skills Rubric The below items are designed to assess various skill areas that are evident in the student’s portfolio information and/or materials. Please note that each portfolio begins with a Portfolio Information Form. Ratings should be based upon the information provided by the student in the Portfolio Information Form as well as the actual items from the portfolio. Please base your ratings on the following scale: 1 = Inadequate; 2 = Minimal; 3 = Moderate; 4= Extensive.

Uses knowledge, diverse cultural frames of reference, and alternate perspectives to think critically and solve problems.

1 Inadequate Proficiency

2 Minimal

Proficiency

3 Moderate

Proficiency

4 Extensive

Proficiency

1. Recognizing the importance and validity of others’ perspectives

2. Providing culturally-grounded evidence to make points (e.g., recognizes the cultural underpinning of evidence, opinion, and arguments).

In addition to your ratings, please provide additional information regarding the student's performance in this area. Information regarding why you assigned the ratings you did—as well as specific examples from the portfolio—would be particularly useful.

42

Attitude Rubric The below items are designed to assess various attitude areas that are evident in the student’s portfolio information and/or materials. Please note that each portfolio begins with a Portfolio Information Form. Ratings should be based upon the information provided by the student in the Portfolio Information Form as well as the actual items from the portfolio. Please base your ratings on the following scale: 1 = Inadequate; 2 = Minimal; 3 = Moderate; 4= Extensive.

Accepts cultural differences and tolerates cultural ambiguity. 1 Demonstrates

inadequate awareness

of, openness toward, or

engagement with…

2 Demonstrates

minimal awareness

of, openness toward, or

engagement with…

3 Demonstrates

moderate awareness

of, openness toward, or

engagement with…

4 Demonstrates

extensive awareness

of, openness toward, or

engagement with…

1. … the similarities and/or differences among cultures.

2. ... the process of reflecting upon his or her own thoughts and feelings toward different cultures.

3. ... the importance of interpreting cultural events and experiences "through the eyes of" individuals from different cultures.

4. ... the specific ways in which he or she has been changed and/or transformed as a result of cross-cultural experiences.

5. ... his or her own biases, prejudices, or stereotypes in relation to a different culture.

In addition to your ratings, please provide additional information regarding the student's performance in this area. Information regarding why you assigned the ratings you did—as well as specific examples from the portfolio—would be particularly useful.