NSSE: More Than Your Average Lake Monster

John Ryan Director, Office of

Institutional Research The University of

Vermont 1/25/13

Presentation Overview

§  An Introduction: NSSE & Student Engagement

§  NSSE Uses and Select Results at UVM

§  Questions & Discussion

§  Contact Information

An Introduction: NSSE and the Concept of Student Engagement

What is Student Engagement?

§  What students do -- time and energy devoted to educationally purposeful activities

§  What institutions do -- using effective educational practices to induce students to do the right things

§  Educationally effective institutions channel student energy toward the right activities

Time on task (Tyler, 1930s)

Quality of effort (Pace, 1960-70s)

Student involvement (Astin, 1984)

Social, academic integration (Tinto, 1987, 1993)

Good practices in undergraduate education (Chickering & Gamson, 1987)

College impact (Pascarella, 1985)

Student engagement (Kuh, 1991, 2005)

Foundations of Student Engagement

Seven Principles of Good Practice in Undergraduate Education

(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)

ü  Student-faculty contact ü  Active learning ü  Prompt feedback ü  Time on task ü  High expectations ü  Experiences with diversity ü  Cooperation among

students

How College Affects Students (Pascarella & Terenzini, 2005)

Because individual effort and involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. – p. 62

Student Success in College: Creating Conditions that Matter

(Kuh, Kinzie, Schuh, Whitt, and Associates, 2005)

Illustrates actual institutional policies, programs, and practices that promote student success. §  Comprehensive report of in-depth study

of successful educational practices across institutional types

§  Practical guidance on how to implement effective practice in a variety of contexts

§  Co-sponsored by leading higher education groups (CFAT, Pew Forum on Undergraduate Learning)

NSSE Survey Content

Student Behaviors in College

Institutional Actions And Requirements

Student Reactions to College

Student Background

Information

Student Learning & Development

NSSE Indicators of Effective Educational Practice

Level of Academic Challenge

Active & Collaborative

Learning

Enriching Educational Experiences

Student – Faculty Interaction

Supportive Campus

Environment

Survey Administration §  Administered to random

samples of first-year & senior students across U.S.

§  Administration occurs in the spring academic term

§  Paper & Web-based survey

§  Flexible to accommodate consortium questions

§  Multiple follow-ups to increase response rates

How Valid is Self-Reported Data? Research Confirms Self-Reported Data Validity

When Following Conditions Exist… §  Requested information is known to respondents

§  Questions are clear and unambiguous

§  Respondents take questions seriously and thoughtfully

§  Answering does not threaten, embarrass, or violate privacy or compel a socially desirable response

The NSSE survey instrument was designed to meet these conditions…but

there is ongoing debate

For more details, see: NSSE Psychometric Portfolio at http://nsse.iub.edu/html/NSSE_Psychometric_Portfolio.cfm

NSSE Uses and Select Results at UVM

NSSE 2011 Institution Response Rates

§  UVM’s response rate = 33%

§  Slightly more female, ALANA similar compared to

UVM student population; weighted for comparison

reports; participated 2005, 2008, 2011…2014 next

Average Institutional Response Rates

§  23% for Carnegie Class comparison institutions §  28% for NSSE 2011 Peer Group institutions

§  26% for Writing Consortium institutions

NSSE 2011 Select Results

§  Many strategies for analyzing the results

§  Comparisons with other universities/groups

§  “Change” over time at UVM (and relative to other schools)

§  First year v. senior respondents

§  Differences among colleges within UVM

§  Compare similar majors across universities

§  Predictive modeling for outcomes (learning, development,

satisfaction, retention, academic performance)

§  Cohort tracking (2008 first year respondents and as seniors

in 2011)

NSSE 2011

UVM Results §  Thinking about your overall experience at this

institution, how would you rate the quality of relationships with faculty and administrative personnel and offices (FY)?

§  What percentage of our first year students would say “quite a bit” or “very much” about the extent to which UVM provides the support they need to thrive socially?

Survey says?

Answer:

55% in 2008 and 48% in 2011 §  Thinking about your overall experience at this

institution, to what extent does the college provide the support you need to thrive socially (FY)?

Selected Peer Comparison

with UVM Results §  In thinking about your undergraduate program as a whole,

including your major, have you done a culminating senior experience (e.g., senior comprehensive exam, capstone course, thesis or project)?

NSSE 2011 UVM Findings

What percentage of our students (in comparison to selected peers) participated in community service or volunteer work?

Class UVM Carnegie Peers

First-Year 29% 39%

Seniors 64% 58%

Survey Says?

§  What percentage of our FY and SR students reported typically spending more than 15 hours per week studying/preparing for class?

NSSE 2011 UVM Findings

What percentage of UVM students spent more than 15 hours per week studying/preparing for class?

Class UVM Carnegie Peers

First-Year 55% 41%

Seniors 42% 42%

Using NSSE at UVM: Student Success and Satisfaction

"SENIORS" 2008 NSSE+Engagement+FactorsImpact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+over

Academic(ChallengeActive(Collaborative(LearningStudent(Faculty(InteractionEnriching(Educational(ExperiencesSupportive(Campus(EnvironmentSAT(MathSAT(VerbalFemaleEthnicity( HispanicRankMajor(Category BusinessTransferCampus(residenceMother's(educationFather's(education

2008 NSSE+Engagement+Factors"FIRST+YEARS" Impact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+overAcademic(ChallengeActive(Collaborative(LearningStudent(Faculty(InteractionEnriching(Educational(ExperiencesSupportive(Campus(EnvironmentSAT(Critical(ReadingSAT(Critical(WritingSAT(MathFemaleEthnicity(RankMajor(Category Professional,+Social+Science,+OtherTransferCampus(residenceMother's(education Attended+college+or+completed+degreeFather's(education Attended+but+did+not+complete+college Attended+but+did+not+complete+college

Student Success and Satisfaction "SENIORS" 2008 UVM.Engagement.Factors

Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.overInclusion)and)SupportActive)and)Enriching)LearningHigher8Order)Critical)ThinkingOpenness)to)New)IdeasSAT)MathSAT)VerbalFemaleEthnicity) HispanicRankMajor)Category Arts.and.Humanities,.Business,.ProfessionalTransferCampus)residenceMother's)education Attended.but.did.not.complete.collegeFather's)education Completed.associate.degree Completed.a.bachelor's.degree

2008 UVM.Engagement.Factors"FIRST.YEARS" Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.overInclusion)and)SupportActive)and)Enriching)LearningHigher8Order)Critical)ThinkingOpenness)to)New)IdeasSAT)Critical)ReadingSAT)Critical)WritingSAT)MathFemaleEthnicity) Caucasian,.HispanicRankMajor)Category Business,.Professional Business,.Professional,.Social.Science,.Other Arts.and.Humanities,.Business,.ProfessionalTransferCampus)residenceMother's)education Attended.college.or.completed.degree Attended.but.did.not.complete.collegeFather's)education Completed.a.bachelor's.degree

Student Persistence

Likelihood of Persistence Fall 2011-Fall 2012 ì ë Grade-Point Supportive

Average Campus Environment

Student Persistence ...and the predictors for the predictors… GPA Supportive Environment -SAT Score -Level of Academic -Active Learning Challenge -Female -Student-Faculty

Interaction -Not transfer student -Fewer hrs./wk. working off campus

Best Practice Benchmarking

Class Mean a Sig bEffect Size c Mean a Sig b

Effect Size c Mean a Sig b

Effect Size c

First-Year 63.0 .00 61.5 * .07 63.0 .00Senior 58.9 * -.10 57.4 -.02 59.1 ** -.11

Note: Each box and whiskers chart plots the 5th (bottom of lower bar), 25th (bottom of box), 50th (middle line), 75th (top of box), and 95th (top of upper bar) percentile scores. The dot shows the benchmark mean. See page 2 for an illustration. See pages 10 and 11 for percentile values.

Supportive Campus Environment (SCE) I temsStudents perform better and are more satisfied at colleges that are committed to their success and cultivate positive working and social relations among different groups on campus.

Mean a

62.957.1

a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.

Distributions of Student Benchmark Scores

NSSE 2011 Benchmark ComparisonsUniversity of Vermont

Supportive Campus Environment (SCE)

Mean Comparisons University of Vermont compared with:

UVMWriting

Consortium Carnegie Class NSSE 2011 Peer Group

0

25

50

75

100

UVM WritingConsortium

Carnegie Class NSSE 2011 PeerGroup

First-Year

0

25

50

75

100

UVM WritingConsortium

Carnegie Class NSSE 2011 PeerGroup

Senior

� Campus environment provides the support you need to help you succeed academically � Campus environment helps you cope with your non-academic responsibilities (work, family, etc.) � Campus environment provides the support you need to thrive socially � Quality of relationships with other students � Quality of relationships with faculty members � Quality of relationships with administrative personnel and offices

7

Class Mean a Sig bEffect Size c Mean a Sig b

Effect Size c Mean a Sig b

Effect Size c

First-Year 53.8 * .09 53.3 *** .12 54.5 .03Senior 57.1 .02 56.7 .05 57.5 -.01

NSSE 2011 Benchmark ComparisonsUniversity of Vermont

Level of Academic Challenge (LAC)

Mean Comparisons University of Vermont compared with:

UVMWriting

Consortium Carnegie Class NSSE 2011 Peer GroupMean a

54.957.4

a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.

Distributions of Student Benchmark Scores

Note: Each box and whiskers chart plots the 5th (bottom of lower bar), 25th (bottom of box), 50th (middle line), 75th (top of box), and 95th (top of upper bar) percentile scores. The dot shows the benchmark mean. See page 2 for an illustration. See pages 10 and 11 for percentile values.

Level of Academic Challenge (LAC) I temsChallenging intellectual and creative work is central to student learning and collegiate quality. Colleges and universities promote high levels of student achievement by emphasizing the importance of academic effort and setting high expectations for student performance.

0

25

50

75

100

UVM WritingConsortium

Carnegie Class NSSE 2011 PeerGroup

First-Year

0

25

50

75

100

UVM WritingConsortium

Carnegie Class NSSE 2011 PeerGroup

Senior

� Hours spent preparing for class (studying, reading, writing, doing homework or lab work, etc. related to academic program) � Number of assigned textbooks, books, or book-length packs of course readings � Number of written papers or reports of 20 pages or more, between 5 and 19 pages, and fewer than 5 pages � Coursework emphasizes: Analysis of the basic elements of an idea, experience or theory � Coursework emphasizes: Synthesis and organizing of ideas, information, or experiences into new, more complex interpretations and relationships � Coursework emphasizes: Making of judgments about the value of information, arguments, or methods � Coursework emphasizes: Applying theories or concepts to practical problems or in new situations � Working harder than you thought you could to meet an instructor's standards or expectations � Campus environment emphasizes: Spending significant amount of time studying and on academic work

3

Other examples of NSSE at UVM

§  “Strategic Value” §  College Key Performance

Indicators §  Board of Trustees

Dashboard Indicators §  College Portrait/VSA §  Student learning outcomes

(writing, diversity, …) §  Accreditation evidence

Examples from Other Universities

§  NSSE results are being used across all sectors and types of institutions.

§  Discovering and sharing ways student engagement results are being used is one of NSSE’s most important activities.

§  The following slides illustrate how NSSE data can inform educational policy and practice and provide examples of how specific institutions have used their NSSE results in productive ways.

Areas of Effective

Educational Practice

Areas for Institutional Improvement

Internal Campus Uses §  Gauge status of campus

priorities §  Examine changes in

student engagement between first and senior years

§  Assess campus progress over time

§  Encourage dialogue about good practice

§  Link with other data to test hypotheses, evaluate programs

§  Improve curricula, instruction, services

Institutional Improvement

Learning Communities 1ST Year

and Senior Experience

Academic Affairs

Learning Assessment

Faculty Development

Academic Advising

Peer Comparison

Student Affairs

Institutional Research

Enrollment Management

External Campus Uses §  Assess status vis-à-vis

peers, competitors

§  Identify, develop, market distinctive competencies

§  Encourage collaboration in consortia (e.g., statewide NSSE conference)

§  Provide evidence of accountability for good processes (while awaiting improvement in outcomes)

Public Accountability

Fund Raising

Governing Boards

Prospective Students

Alumni

State Policy

Makers Performance Indicators

Focus on Right Things

Accrediting Bodies

Media

Parents

Level of Academic Challenge UNIVERISTY OF MICHIGAN §  Finding: An item of discussion at

a meeting of the Provost’s Faculty Advisory Committee raised the question of students’ writing abilities and how much writing students have to do as U of M undergraduates.

§  Action: NSSE and FSSE results pertaining to academic challenge, i.e. those that focus on the amount of effort involved in class preparation and how much the environment emphasizes studying and academic work, were prepared and shared with faculty.

Active and Collaborative Learning WASHINGTON STATE UNIVERSITY §  Finding: Results indicated the

campus was not meeting student expectations for collaborative learning, student-faculty interaction, and enriching educational experiences.

§  Action: Freshman Focus learning communities were created to provide all incoming freshmen the opportunity to engage in an extensive living-learning community system.

Student – Faculty Interaction CALIFORNIA STATE UNIVERSITY -

FRESNO

§  Finding: NSSE results showed that student-faculty interaction was lower than expected.

§  Action: Student success task force identified ways to improve student success. Participated in Building Engagement and Attainment for Minority Students (BEAMS) program to develop Mentoring Institute. Now 200+ faculty members, staff and student mentors have been trained.

Enriching Educational Experiences

SIMONS ROCK COLLEGE OF BARD

§  Finding: NSSE results for out-of-class engagement were not as strong as results on pure academic measures.

§  Action: Student affairs staff members used NSSE results to guide discussions and develop new student activities and community engagement efforts. Have added a new professional position in Student Activities.

Supportive Campus Environment TOWSON UNIVERSITY

§  Finding: Wished to examine and enhance experience of female students.

§  Action: Women’s Center staff developed a narrative about the collegiate experience of women on their campus, strengthened programs that offer leadership opportunities and self-empowerment content and process.

Sharing NSSE Results Campus Wide CLAYTON STATE UNIVERSITY:

Discusses NSSE results at faculty counsel, presidential retreats, student success forums, and in various standing committees.

§  UNIVERSITY OF COLORADO AT COLORADO SPRINGS: Institutional Research Office publishes a series of one- to two-page research briefs on their NSSE Web site, for faculty and staff members to access. Each brief is dedicated to a specific aspect of NSSE, such as “Diversity” or the “Academic Experience.”

Improving Civic Engagement

JAMES MADISON UNIVERSITY §  Finding: First-year student

involvement in service-learning programs was lower than desired.

§  Action: Increased the number and quality of class presentations about service-learning; increased the number of contacts with new faculty members regarding the importance of emphasizing service-learning.

Improving General Education WORCESTER POLYTECHNIC

UNIVERSITY §  Finding: General education

courses provide foundation essential to numerous learning outcomes. A faculty-appointed committee defined 5 objectives to enhance core education.

§  Action: Have begun developing a new first-year general education curriculum featuring interdisciplinary, inquiry-based seminars, better integration of the disciplines, and broader, more engaging introductions to major areas of study.

Questions & Discussion

Contact Information

John F. Ryan, Ph.D.

jfryan@uvm.edu

656-4418

UVM Office of Institutional Research (OIR)

www.uvm.edu/~isis/

.