peczalska-iu2u assessment presentation .pptx [read-only] · microsoft powerpoint - peczalska-iu2u...

10/18/2019

1

How to Assess IU2U?: A Pre-Arrival Orientation Program’s Assessment of

International Student Retention, Academic Performance and Experience

INDIANA UNIVERSITY BLOOMINGTON

Ania PeczalskaIndiana University Higher Education Ph.D. Candidate


Introduction

• 8 years of full-time student services employment at Indiana University Bloomington (IUB)

• IU Higher Education Ph.D. Candidate

• Conducted surveys and a focus group analysis

10/18/2019

2


Purpose• Employ a case study to show how assessment is

implemented in real life and how a program utilized the results

• The case study is a 2017 longitudinal assessment of IUB’s IU2U program


Presentation Learning Outcomes

IU2U assessment outcomes

How IU2U utilized the results

How IU2U learned to

longitudinally assess its program

10/18/2019

3


Source: Suskie (2009) Assessing student learning: A common sense guide p. 4

Table 1.1. What is Assessment

Assessment is the ongoing process of:

Establishing clear measurable expected outcomes of student learning

Ensuring that students have sufficient opportunities to achieve those outcomes

Systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches our expectations

Using the resulting information to understand and improve student learning


Suskie (2009) Assessment Cycle

10/18/2019

4


Presentation’s Use of the Term Assessment

Banta and Palomba (2015) see assessment as including program evaluation and administrative effectiveness


What is Good Assessment?

• Intentional assessment that is planned in advance of the study

• Identifying bottlenecks

• Implementing program improvements through identifying and collaborating with key partners who have responsibility for implementing the changes

(Blumberg, 2018)

10/18/2019

5


IU2U Program • Established in 2014 for incoming IUB international undergraduate

students

• Bring support and resources to students’ home countries through one day orientations for students and parents

• Faculty, staff, and domestic and international student presenters


2014-2017 Matriculating IUB Undergraduate Students

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

2014 2015 2016 2017

40 220 205 221

731 706 671 688

7716 7875 7673 8001

5% 32% 32% 33%

Nu

mb

er o

f M

atr

icu

lati

ng

Un

der

gra

du

ate

Stu

den

ts

YearIU2U ParticipantsTotal Matriculating Undergraduate International StudentsTotal Matriculating Undergraduate Students

% = Percentage of IU2U students compared to the total matriculating international student cohort

10/18/2019

6


Institutional Review Board (IRB)

• IRB approval allows for the research to be presented and published

• Submitted an IRB application and went through IRB review

• IRB Protocol 1703539614

• IRB approval is needed before the assessment begins


IRB Tips • Review what is needed to submit IRB as you begin planning

your study

• Look at templates and other examples as you prepare your IRB protocol and documents

• Submit your initial IRB protocol ideally about 8 weeks before beginning your study

• Allow time for IRB protocol revisions in case the IRB reviewers request them

10/18/2019

7


Initial 2017 IU2U Cohort Research Questions

• To what extent does participation in the IU2U program lead students preparing for acculturation a seamless transition to IUB?

• To what extent does IU2U build participant confidence and social support for IUB study?

• From the perspective of IU2U students, which areas of IU2U programming (ex. academic programming, campus engagement, health and safety) are covered well and which areas can be improved?


IU2U Assessment Method• Prepared through consulting with campus survey experts and

conducting a pilot survey with a few IU2U alumni

• Distributed 3 surveys on Qualtrics with Likert items and open-ended questions

May 2017

July 2017

October 2017

• Put surveys on easy to access Canvas platform and offered October survey respondents a chance to win a $20 gift card

10/18/2019

8


IU2U Survey Results

2017 IU2U cohort = 230

Pre-Program Survey = 37 (16%)

Post-Program Survey = 9 (4%)

First-Semester Survey = 30 (13%)

The assessment had a low response rate issue!

PA5


Discussion 1. How many of you have encountered low response

rates in your assessment work?

2. What are some solutions to survey low response rates?

PA5 Students in China may not be able to access the Pre-Program and Post-Program surveys on the Canvas platform. The students may have trouble accessing Canvas in China.Peczalska, Ania, 10/16/2019

10/18/2019

9


2018 Additional Assessment• Talked with Bloomington Assessment Research Lead Research

Analyst/Programmer Stefano Fiorini

• Decided to add quantitative data from 4 IU2U cohorts (2014-2017) and analyze retention and cumulative GPA for all 4 cohorts

• A control group was created by another analyst through utilizing a greedy distance matching approach based on the K-means methods

• Thematic analysis was conducted on survey responses for the qualitative research question

• Took 2 months (while working full-time) to analyze and write a short report summarizing the findings


Statistical Analysis Tips for Survey Data • Explore the data and conduct basic descriptive statistics before

doing your analysis

• Label variables with short names and create a codebook

• Save all statistical software output with dates (ex. 9.20.2017)

• Maintain a document that summarizes what you did each session and the results

10/18/2019

10


Final Quantitative Research Questions

• What is the degree of difference in academic performance (cumulative GPA) between IU2U students and the control group across all semesters of enrollment?

• What is the association between IU2U status and IU retention after one semester and one year?

• What differences exist between IU2U students and non-IU2U students for sense of belonging and comfort level in asking for help from professors, academic advisors, RAs and other residential staff, and other campus staff?


Final Qualitative Research Question

• How do IU2U students employ sense-making and meaning-in-life during their IUB study?

• Pan (2011) resilience framework

10/18/2019

11


RQ 1: What is the degree of difference in academic performance (cumulative GPA) between IU2U students and the control group across all semesters of enrollment?

IU2U students had a significantly higher end of the semester cumulative GPA (M = 3.21, SD = .67) compared to the control group (M = 3.07, SD = .84).


RQ 2: What is the association between IU2U status and IU retention after one semester and one year?

There was a significant association after the first semester (1, 1372) = 6.60, p = .01.

One Semester Two Semesters

Retained Departed Total Retained Departed Total

IU2U 676 10 686 433 32 465

Control 661 25 686 418 47 465

Total 1337 35 1372 851 79 930

PA6

PA6 The table data is from the beginning of the Spring 2018 semester. The second semester student numbers are lower because they do not include the Fall 2017 cohort.Peczalska, Ania, 10/16/2019

10/18/2019

12


RQ 3: What differences exist between IU2U students and non-IU2U students for sense of belonging and comfort level in asking for help from professors, academic advisors, RAs and other residential staff, and other campus staff?

44%

60%

64%

52%

44%

30%

10%

40%

30%

30%

Belonging

Professor Help

Academic Advisor Help

RA and Other Residental Staff Help

Other Campus Staff Help

Non-IU2U N = 10 IU2U N = 25

PA4


RQ 4: How do IU2U students employ sense-making and meaning-in-life during their Indiana University Bloomington study?

Sense-making

Explain why you liked a certain part of the workshop the most.

Response about Arrival to Campus programming: “It helped me have a vivid picture of what kinds of situations I will face when I come to IU.”

The student finds a way to make meaning and prepare for IUB study.

PA4 The chart shows the top response ("Definitely") of a four-point Likert scale for 5 October 2017 survey questions. Mann-Whitney U-Tests for Independent Samples showed that how comfortable students were approaching professors for help was significantly greater for the IU2U group (Median = 4.0) than the non-IU2U group (Median = 3.0).Peczalska, Ania, 10/16/2019

10/18/2019

13



Global meaning-in-life

Please explain why you think students are responsible to chose their courses and majors at IUB.

“Because we’re the person who knows ourselves best and we need to plan for our future. We need to learn many skills and have a lot of things to put in our resume and attract many good companies.”

The student finds meaning through examining the question from a global perspective (i.e. post-graduation plans).



Situational meaning

Please explain why you think students are responsible to chose their courses and majors at IUB.

“Different students are interested in different areas. They will enjoy university life only when they chooses [sic] the classes they really want to take. Their time will not be wasted.”

The student makes meaning of why a smaller objective, major selection, is important in the US context.

10/18/2019

14


How IU2U Utilized the Results

• 2019 IU2U cohort survey included 2017 survey items

• Data related to retention was shared with campus administrators who determine future IU2U funding

• Used data to determine number of 2020 IU2U U.S. domestic student peer mentors


November 2018 Presentation

Wrote a paper that was accepted for publication and presentation at the 2018 National Symposium on Student Retention.

10/18/2019

15


Summary of Presentation Learning Outcomes

IU2U assessment outcomes

How IU2U utilized the results

How IU2U learned to

longitudinally assess its program


Final Discussion In groups of 4 to 5 please discuss the following questions.

1. What are other ways IU2U can conduct program assessment?

2. How can you assess individuals from other non-English speaking countries, especially if you are surveying individuals from many different countries? Should it be done in English or the individual’s native language?

3. Every country has its own cultural nuances. Cultural nuances refer to how certain words are interpreted in a culture. For example “we will see about that” can mean “no” in Chinese culture. How do you account for cultural nuance in assessment?

10/18/2019

16

Questions?

Ania Peczalska

[email protected]


References

Banta, T.W., & Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass.

Blumberg, P. (2018). Two underused best practices for improvement focused assessments. Research & Practice in Assessment, 13, 77-83.

Pan, J.-Y. (2011). A resilience-based and meaning-oriented model of acculturation: A sample of Mainland Chinese postgraduate student in Hong Kong. International Journal of Intercultural Relations, 35(5), 592-603. doi: 10.1016/j.ijintrel.2011.02.009

Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.).San Francisco, CA: Jossey-Bass.

peczalska-iu2u assessment presentation .pptx [read-only] · microsoft powerpoint - peczalska-iu2u...

Documents