copyright information copyright david ernst 2006. this work is the intellectual property of the...

Copyright Information

Copyright David Ernst 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Student Evaluations: A Comparison of Online vs.

Paper Data CollectionDavid Ernst

Director, Academic TechnologyCollege of Education and Human Development

University of Minnesota

Student Evaluations of Teaching (SETs)

Student Evaluations of Teaching (SETs) Tenure

Promotion

Questions

1. Are there differences in the completion rate between evaluations completed online and those completed using paper and pencil?

Questions

2. Are there differences in the distribution of responses between evaluations completed online and those completed using paper and pencil?

Questions

3. Are there differences in the average ratings between evaluations completed online and those completed using paper and pencil?

Questions

4. What are common perceptions of each format?

Faculty Focus Groups

4used the online SET system

& taught large classes

5used the online SET system

& taught small classes

8never used the online SET

system

Student Focus Groups

• Biol. Sciences (7)• Prof. Schools (6)• Liberal Arts (7)• Technology (6)• Graduate School (8)

34 students

Instructor and StudentFocus Group Results

General SET Issues

Student MotivationGeneral SET Issues

Student MotivationExternal Motivation: Reward

“The only time I’ve ever filled out an evaluation for a graduate course is when the professor bribed us with an automatic five percent grade increase if we filled out the evaluation by a certain date.”

- Focus Group Student

General SET Issues


External Motivation: Punishment

“I had one class where … you’d have an incomplete until you filled out the course evaluation.”



External Motivation: Social Pressure

Internal Motivation: Making a Difference

“I had one professor who said “Okay, these are the top three things you said I can either improve on or are doing [well].” And he discussed whether they were even feasible to fix, or how we address it, so [students] saw that this actually meant something.”




“I’ve definitely strongly felt that my evaluation will do little good because I can’t imagine that the people ten years before me haven’t said the exact same thing and the professor just disregards it or chooses not to make any changes.”




“And the questions are so vague on that questionnaire that I don’t feel like me filling in a bubble is going to really help that teacher do better next semester.”



Internal Motivation: Strong Opinion

“I agree that I put more time and effort into [SETs] if I liked the professor a lot or didn’t like the professor a lot. If it’s somebody I don’t really care about I will go down, just mark satisfactory, satisfactory, satisfactory…”



AnonymityGeneral SET Issues

Class Size

Advisor

“The people in the department have a huge impact on your profession after you leave, after the degree and so many students feel the need to suck up to the professors and fill out the evaluations in such a way.”


AnonymityGeneral SET Issues

Instructor and StudentFocus Group Results

Issues of Format

Cost

“Time is money … There’s a lot of money to be saved online.”

-Focus Group Instructor

EfficiencyFormat Issues

=

Speed of Obtaining Results

“… the deadline [was] on a Sunday night mid-night … and there, low and behold, in an email on Monday or Tuesday I got all the results. That was fantastic …”



Convenience

“[Online SET comments] are in your computer and you can copy and paste them. I found that very useful for talking about student comments.”



“It’s just a bit odd to have to remind students emphatically to do their course evaluations … if you have to advertise it, it becomes kind of ‘Say nice things about me.’”


Instructor RoleFormat Issues

“The big concern I would have is the low [response] rate that you might experience. It becomes somewhat optional. It has an optional feel if the students are not presented with the form and encouraged to fill it out.”


Return RateFormat Issues

“What I would be afraid of is that you would get a split of two different populations. You’d get the populations who are very enthusiastic about the instructor and material ... and those who were not particularly impressed ... You might miss that 60 - 80% middle window …”


Characteristics of Likely CompletersFormat Issues

GoodBad

Most Likely to CompletePaper?

GoodBad

Most Likely to CompleteOnline?

“I want to get out of this class because it’s the last 15 minutes, last day, last semester. You know you just want to go; either you want to go home for Christmas or you want to go outside because it’s nice, or you just want to go home and go to bed. Whatever it is, it’s like ‘OK, I know these questions. Whatever. Blah, blah, blah.’”

-Focus Group Student

In-Class TimingFormat Issues

“… someone else would collect [the paper SETs] when everyone’s done, which can be a tad bit intimidating if you have a lot to say and they’re waiting for you to turn yours in. ”



“… you’re not in the middle of the class doing [the paper SET], and if you just had a bad lecture … that’s going to influence it. So, [online] you’re stepping back and you’re actually more critically thinking about how you’re actually responding.”



“… it takes time out of the class. That time can be used for teaching rather than filling out an evaluation …”


“The paper is more convenient for me because the teacher has to give you the time to do that. And, because I don’t have a lot of time.”



Handwriting

“I haven’t done the written part of it because … it’s been a small enough class that [instructors] would know my handwriting.”


Format Issues

Anonymity and Privacy

Handwriting

“… how many papers have we submitted that we actually write? None. We all type everything almost always …”


Format Issues


Handwriting

“Everything I’ve gotten from the students has been typed so I couldn’t identify their handwriting.”


Format Issues


Instructor Proximity

“… they know who took a long time filling it out versus who took a short time filling it out and then that tells them a lot about who wrote what …”


Format Issues


Computer Security

“With the computer one I have to log in so I’m thinking to myself that maybe this is tied to my log in so then, somehow someone could probably figure it out.”


Format Issues


Computer Security

“And yet, it’s not anonymous. If you’re filling out a form online, there’s a trace.”


Format Issues


Computer Security

“… are the teachers all going to really hack into that? I have teachers who don’t know how to send an attachment…”


Format Issues


“When I have someone sitting next to me, even if they’re my good friend and I really trust them and I don’t really have any issues with the professor, if I give the professor anything less than a six or a seven I feel like I have to justify it to my friend, like, ‘Well, you know there was the day when she did this.’ So it feels kind of stupid. Why should I have to feel that way?”


Format Issues

Accessibility

“… it’s more worth my time when I know that I have three things to do there instead of just one thing to do there.”


Format Issues

Completing multiple SETs

“… if you’re doing … five [evaluations] or so, by the time you get down to the last one, you’re just sick of doing these …I think three is the most you’d want to do at a time.”


Format Issues

Completing multiple SETs

“… [with] online, email messages, or instant messaging, people tend to blurt out what’s on their mind and use expletives that they might think twice about using if they were confronting you face-to-face or writing it down. They tend to be very blunt.”


Format Issues

Culture of Online Communication

“If you can get past the invectives that often slip in there, they actually do spend more time writing the comments out and there are more useful gems in there if you’re tough enough to apply the filter that they should have applied in the first place.”


Format Issues

Culture of Online Communication

Format Issues

Technical Difficulties

Summary of SET Data Analysis

693,816 student evaluations

693,816 student evaluations Courses taught 2+ times with same instructor• at least once using paper SET• at least once using online SET


54,006 student evaluations

Dr. Jones – BIOL 1001

Dr. Smith – BIOL 1001

Dr. White – PSY 5331

Dr. Wang – EDUC 4121

Dr. Blick – NURS 6472

Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4

Dr. Ryan – LIT 2301






Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4


Matching semesters






Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4


Similar class sizes






Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4


Chronologically close






Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4







Fal

l ‘0

2

Sp

rin

g ‘

03

Su

mm

er ‘

03

Fal

l ‘0

3

Sp

rin

g ‘

04

Su

mm

er ‘

04

Fal

l ‘0

4


No face-to-face/distance pairs or concurrent meetings withother courses


54,006 student evaluationsClass pairs (one paper, one online)

Removal of irregular pairs

14,154 student evaluations(314 pairs = 628 classes)

Selected Classes by Course Level

Selected Classes by Instructor Type

Students by Age

Student Ethnicity

Student Gender

SET Required Questions7-point scale ( 1 = “Very Poor” to 7 = “Exceptional”)

SET Required Questions

• How would you rate the instructor's overall teaching ability?

7-point scale ( 1 = “Very Poor” to 7 = “Exceptional”)



• How would you rate the instructor's knowledge of the subject matter?





• How would you rate the instructor's respect and concern for students?






• How much would you say you learned in this course?


How would you rate the instructor's overall teaching ability?

How would you rate the instructor's knowledge of the subject matter?

How would you rate the instructor's respect and concern for students?

How much would you say you learned in this course?

Paper Completion RatesMean=76.7%

Online Completion RatesMean=55.9%

Ratings


Online Paper

Mean Rating 5.645 5.673

p=0.685, significant if <.05

Ratings


Online Paper



Ratings

• How much would you say you learned in this course?

Online Paper



Findings

SET completion rates will be lower online

Findings


Students with strong opinions will be more likely to complete SETs online

Findings


Students with strong opinions will be more likely to complete SETs online

SET scores will be lower, on average, online

Other Findings

The online SET system

• is a more efficient process, reducing the time and effort that staff and instructors spend on SET issues

• provides better accessibility and confidentiality for seeing impaired students

• increases students’ perceived anonymity

Questions

David Ernst

[email protected]

copyright information copyright david ernst 2006. this work is the intellectual property of the...

Documents

focus group student

online set system slide

internal motivation

student focus groups

social pressure slide

faculty focus groups

course evaluation

difference ive