before & after: what undergraduates and alumni say about their college experience and outcomes
DESCRIPTION
Before & After: What Undergraduates and Alumni Say About Their College Experience and Outcomes Angie L. Miller, NSSE & SNAAP Research Analyst Amber D. Lambert, NSSE & SNAAP Research Analyst Becca Houghton, SNAAP Project Coordinator Assessment Institute October 30 th , 2012. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
Before & After: What Undergraduates and Alumni Say About Their College Experience and
Outcomes
Angie L. Miller, NSSE & SNAAP Research AnalystAmber D. Lambert, NSSE & SNAAP Research Analyst
Becca Houghton, SNAAP Project Coordinator
Assessment InstituteOctober 30th, 2012
IntroductionSurveys are a common means of
assessment in higher education
Student surveys are conducted on a variety of topics, from student engagement to use of campus resources to faculty evaluations
Alumni surveys are used to gather information about satisfaction, acquired skills, and career attainment
Research QuestionsQuestion 1: Are there differences in how
students and alumni perceive aspects of their institutional experiences and the skills and competencies that they acquire at their institutions?
Question 2: What discipline-specific strengths and weaknesses exist when exploring students’ reported development of skills and abilities?
What is SNAAP?
Strategic National Arts Alumni Project
On-line annual survey of arts graduates
Investigates educational experiences and career paths
Provides findings to educators and policymakers to improve arts training, inform cultural policy, and support artists
Who is Surveyed?Graduates of:
Arts schools, departments, or programs in colleges and universities
Independent arts collegesArts high schools
Both graduate and undergraduate degree recipients
All arts disciplines
SNAAP Questionnaire Topics
1. Formal education and degrees
2. Institutional experience and satisfaction
3. Postgraduate resources for artists
4. Career
5. Arts engagement
6. Income and debt
7. Demographics
SNAAP 2011 Administration InformationAdministered in Fall 2011
66 participating institutions 58 postsecondary and 8 high schools
Over 36,000 total respondents
What is NSSE?National Survey of Student Engagement
NSSE gives a snapshot of college student experiences in and outside of the classroom by surveying first-year and senior students
NSSE items represent good practices related to desirable college outcomes
Indirect, process measures of student learning and development
NSSE Purpose
NSSE annually gathers valid, reliable information on the extent to which students engage in and are exposed to proven educational practices that correspond to desirable learning outcomes.
• Results indicate how students spend their time and what they gain from college.
NSSE 2012 Administration Information Administered in Spring 2012
546 participating U.S. institutions
Over 285,000 total respondents
Each year, experimental item sets appended at end of core survey
Question 1: MethodologyParticipants from 6 institutions that
participated in both SNAAP11 and NSSE12
Senior NSSE respondents from arts majors in corresponding SNAAP participating programs (n = 222)
Alumni of undergraduate SNAAP programs from graduating cohorts of 2001-2010 (n = 593)
Question 1: Respondents
Characteristics NSSE SNAAP
Female 72% 61%
U.S. citizen 98% 98%
First Generation status 37% 30%
Race/Ethnicity*
White 68% 89%
Black 5% 4%
Latino/Hispanic 7% 3%
Asian 1% 5%
*SNAAP asks race/ethnicity in a check-all format; NSSE asks race/ethnicity in a forced-choice format
Question 1: SNAAP items
Question 1: NSSE items
Question 1: Results
Means comparison for overall rating of institutional experience (4-point scale from “Poor” to “Excellent”) suggests that alumni give higher general appraisals
Student
Mean
Alumni Mean
Sig. Effect
size (d)
Overall experience
3.27 3.38 * .17
*p<.05; **p<.01; ***p<.001
Question 1: Results (cont.)Means comparisons for satisfaction with aspects of time at institution (4-point scale after removing “Not Relevant” option) suggests that alumni give lower specific appraisals for certain aspects
Student
Mean
Alumni Mean
Sig.
Effect size (d)
Academic advising
3.00 2.79 ** .22
Career advising 2.79 2.40 *** .39
Opp. for internships
2.68 2.36 *** .31*p<.05; **p<.01; ***p<.001
Question 1: Results (cont.)
Means comparisons for amount of institutional contribution to acquired skills and competencies (4-point scale from “Not at all” to “Very much”) show a similar pattern, with alumni giving lower specific appraisals for certain skills
*p<.05; **p<.01; ***p<.001
Student
Mean
Alumni
Mean
Sig.
Effect size (d)
Research skills 3.29 3.10 ** .24
Clear writing 3.21 2.95 *** .33
Persuasive speaking
2.96 2.79 * .20
Project management
3.23 3.03 ** .23
Technological skills
3.25 3.08 * .21
Financial & business
2.24 1.92 *** .38
Entrepreneurial 2.23 2.00 ** .26
Networking 3.07 2.84 ** .27
Question 2: MethodologyParticipants from 37 institutions
that participated in NSSE12 and received the experimental items derived from the SNAAP survey
Senior NSSE respondents from all majors* (n = 23,726)
*Excluding “other” and “undecided” categories
Question 2: RespondentsCharacteristics
Female 64%
U.S. citizen 95%
First Generation status 40%
Race/Ethnicity
White 70%
Black 6%
Latino/Hispanic 8%
Asian 5%
Question 2: Respondents’ Majors
Primary Major Field
Arts 8%
Humanities 9%
Biological Sciences 12%
Business 18%
Education 8%
Engineering 10%
Physical Sciences 4%
Professional (other) 14%
Social Sciences 16%
Question 2: ResultsLooking at frequencies for students in
the different majors reporting that their institution contributed “Very Much” to acquiring certain skills and competencies reveals interesting patterns
Some patterns are expected; others highlight less apparent similarities among major fields
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
74%
21%7% 11% 14% 7% 9% 15% 11%
Artistic Technique
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%10%20%30%40%50%60%70%80%90%
100%
10% 8% 6%
65%
10% 9% 8% 14% 13%
Financial & Business Management Skills
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
33% 28%17% 19%
77%
15%27% 26% 19%
Teaching Skills
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
49%62%
51% 55% 52% 57% 58% 58% 61%
Critical Thinking
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%10%20%30%40%50%60%70%80%90%
100%
62% 58%46%
55% 59% 63% 57% 61% 57%
Creative Thinking
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
40%
61% 60%48% 48% 47% 54% 57% 64%
Research Skills
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%10%20%30%40%50%60%70%80%90%
100%
40%25% 28%
39% 36%
60%40% 39%
27%
Technological Skills
% Reporting Institution Contributed “Very Much” to Acquiring…
Arts
Hum
aniti
es
Biolo
gica
l Scien
ces
Busin
ess
Educa
tion
Engin
eerin
g
Phys
ical
Scien
ces
Prof
ession
al (o
ther
)
Social
Scien
ces
0%10%20%30%40%50%60%70%80%90%
100%
36% 32% 29%
53% 47%38% 30%
47%37%
Interpersonal Relations
Discussion
Question 1Alumni may be viewing their institutional
experience as a whole through rose-colored glasses when they think about “the good old days”
Post-graduation experiences in the workplace may better enable alumni to reflect on certain aspects of their time
Alumni may also learn that they needed to develop some skills more once they have gained work experience
Discussion (cont.)
Question 2The skills that students acquire while at
their institutions can vary greatly among different majors
Many of these differences are expected (and increase our confidence in the survey instrument)
Other patterns link seemingly dissimilar majors, such as arts and engineering, or business and education
LimitationsMay not represent ALL students and
alumni, data only available for those participating in both SNAAP and NSSE (and those receiving experimental NSSE items)
Difficulty of surveying alumni: response rates, tracking accurate contact information
Relies on self-reported data
ConclusionsImportant institutional information can be
gained through surveying both students and alumni
Students may be better able to provide information about affective components of their experience, while alumni may be better judges of specific things needed in the workplace
A variety of perspectives can also be gained through exploring differences by majors
http://3millionstories.com/March 7-9, 2013
Nashville, TN
Indiana University Center for Postsecondary Research
Strategic National Arts Alumni Project
[email protected] snaap.indiana.edu
National Survey of Student Engagement
Angie L. Miller [email protected] D. Lambert [email protected] Houghton [email protected]