Reflections on Teaching

Course Improvement

• We can simply use the feedback to pat

ourselves on the back, documenting that

we are doing a great job

• Or we can use it to capitalize on strengths

and improve weak areas

• Either way, we can include some

information

Sources of Information

• Gut feelings

• Student performance on the content, in the

class or on external tests

• Informal feedback from students

• Peer visits and feedback

• Committee visits and feedback

• IDEA evaluations

Easy to interpret

• Recommendations from colleagues,

supervisors and students

• Strong student performance

• High evaluations in all categories

• The latter two don’t really suggest any

changes

• The first often comes easy to implement

Harder to Interpret

• Poor student performance

• Gut feelings

• IDEA forms

• The IDEA forms have a lot of information,

some of which can be useful

What is reliable anyway?

• This is simply a function of the number of

responses; lower numbers are considered

less reliable. 25 or more responses are

noted as “reliable”, 15?-24 are noted as

“fairly reliable”, 10-15 are noted as

“marginally reliable”, and 9 and below are

noted as “unreliable.”

And what is representative?

• This is the fraction of the class that filled

out the form; this is an attempt to ascertain

if the response is representative of the

class. Response rates of 65%-79% are

noted as “probably representative”;

response rates of 80% and higher are

noted as “representative.”

IDEA Raw vs. Adjusted

• The Raw score is the average of the

responses by the students.

• Why adjust this?

• There are factors that affect scores that

are independent of the instructor or the

class.

• An example of this is the students’ desire

to take the course, regardless of teacher

Factors to adjust for

• Class size (larger classes get lower ratings)

• Type of course (core requirements get lower

ratings)

• Student interest (interested students rate

courses higher, uninterested students rate them

lower)

• These are called “extraneous variables” and are

discussed in detail in IDEA Research Report #6

How are adjustments made?

• The raw scores are adjusted, based on

students’ answers to the questions

pertaining to interest and effort and based

on class size and level. In short, if most of

the students report a high desire to take

the course, the raw score will be adjusted

down a bit. For a large class, the ratings

adjust up a bit.

Why adjust scores at all?

• Once the scores are adjusted, it is possible to

make comparisons between classes, without

having to worry about class size, or the rest.

• You can compare your own ratings between

classes, or from one semester to another.

• We can compare the ratings (which are a

reflection of student perceptions, of course)

between faculty.

Key General Indicators

• Progress on Course Objectives

• Excellent Teacher

• Excellent Course

• These are always worth looking at as summary statistics. They do not provide much detail.

• Good for identifying serious issues, or for patting ourselves on the back.

Progress on Course Objectives

• This is a summary evaluation, based on a

number of questions in the form.

• It does reflect student perception on their

progress.

• The student responses that are used for

this indicator are those objectives (items

21-32) that the professor picked as

“important” or “essential” (counting double)

Excellent Teacher

• This is a summary evaluation, based on

Question #41 on the IDEA form.

• It does reflect student perception of their

teacher.

Excellent Course

• This is a summary evaluation, based on a

number of questions in the form.

• It does reflect student perception on their

progress.

• The question that is used for this indicator

simply asks the students if it was an

excellent course, question #42

Clear Indicators

• Good ratings in general, 4.0 or above in

the three summary numbers would mean

that the students are well satisfied with the

course.

• Poor ratings in general, 2.5 or below in the

three summary numbers would mean that

the students are very unhappy with the

course.

Mixed Results

• We saw this in FYEx 103. Several faculty received very high ratings as teachers, but the course ratings were very low. This was echoed by the written comments.

• This reflects student satisfaction with the professor, but not with the class. The rest of the evaluation can help us discern what about the class was a problem for them.

Information to modify a class

• The IDEA reports cannot substitute for

judgment on our part.

• They can give us information on what

students liked or didn’t like about a class.

Why do we care?

• Okay, students can’t tell if we are properly

qualified.

• They can tell if we seem competent.

• They do make decisions based on how

they respond to us.

What is this “converted” score?

• The converted scores are simply

normalized to an average of 50, with 63

being the upper 10%.

• This allows comparison between

questions, as well as between classes,

semesters, faculty, etc.

What information can be useful?

• Other than identifying student

dissatisfaction, some questions can

provide specific guidance.

• Specifically, the last page, statistical detail

can provide specific guidance

How Can We Use IDEA

Results to Improve

Instruction? https://theideaonline.org/idea/cs/in

dex.jsp

Teacher

Old Master

The “Dr. Fox”

Lecture In a well-known 1970 study,

a professional actor (Michael Fox)

was hired to deliver a non-substantive

and contradictory lecture, but in an

enthusiastic and authoritative style.

Naftulin, Donald H., John E. Ware, and Frank A.

Donnelly, "The Doctor Fox Lecture: A Paradigm of

Educational Seduction," Journal of Medical Education

48 (1973): 630-5.

Rice, Lee. "Student Evaluation of Teaching: Problems

and Prospects," Teaching Philosophy 11 (1988): 329-

44.

X

The “Dr. Fox” Lecture

http://ecclesiastes911.net/doctor

_fox.html

• The lecture was held at the University of Southern California School

of Medicine.

• The subject was "Mathematical Game Theory as Applied to

Physician Education."

• The speaker was introduced as Dr. Myron L. Fox from Albert

Einstein College of Medicine, a pupil of von Neumann and an

authority on the application of mathematics to human behavior. The

attendees were psychiatrists and psychologists (MDs and

PhDs) who were gathered for a training conference. They

listened to the lecturer with great interest, asked many questions,

and were satisfied with speaker's replies.

Let’s Watch A YouTube Excerpt

of Dr. Fox’s Lecture

• http://www.youtube.com/watch?v

=RcxW6nrWwtc

Dr. Fox’s Lecture

• Dr. Fox gave his talk on a Scientific American

article on “Game Theory. “ He borrowed from the article some

phrases, but not any sense. He mixed them up with allusions to unrelated

subjects, which on purely verbal level may appear relevant. He conducted

himself with great confidence and showed such a mastery of the

aforementioned allusions that the audience was convinced that a luminary is

standing before them.

• 100% had answered in the questionnaire that Dr. Fox had

stimulated their thinking, 90% said that he presented material

in a well organized form, and 90% said that Dr. Fox put his material

across in an interesting and factual manner.

• Some audience members even commented that they

• had read Dr. Fox’s published papers!

Effects of the “Dr. Fox” Study

Can a brilliant delivery technique of a talk so

completely “bamboozle” a group of experts

that they overlook the fact that the content

was nonsense?

The published study in 1973 cast serious

doubts on results obtained from evaluation of

teachers and lectures.

More modern studies have affirmed results

from the “Dr. Fox” study.

What does “Ms. Mentor” in The

Chronicle Say?

The Torment of Teaching

Evaluations • By Ms. Mentor

• Question: I work twice as hard at teaching as anyone I know, but I still can't get good student evaluations. I've tried outlines

and keywords on the board, handouts, individual meetings, midterm evaluations, peer observations, lecturing more, lecturing

less. Some student comments are so harsh and demoralizing that I put off reading my evaluations until school breaks, when I

have time to be depressed. Is it possible that my low evaluations stem from personality issues that I can't do anything about?

Am I alone?

• Answer: Certainly you are not solo. Ms. Mentor's mailbag is full of complaints about student evaluations -- none of them from

students. Faculty members feel that they are cheated, mistreated, and misunderstood. Often they're right, for most evaluation

forms are so vague and perfunctory ("Concerned about students -- rate 1 to 9") that they do nothing to improve teaching. More

often, they become weapons to get rid of untenured profs who have made enemies. No one has ever really agreed on what

makes a "good teacher," and Ms. Mentor still grieves for Socrates, who taught his students to question everything, and wound

up dead instead of tenured.

• But your risk is smaller. You just need good evaluations, and Ms. Mentor can tell you how. (Naive and idealistic souls may

wish to stop reading at this point.)

• Simplest of all, you can give higher grades, which do correlate with student ratings. You can use more hand gestures,

modulate your voice more, and walk while you talk. Students give higher evaluations to teachers who are good-looking or very

dramatic. This is called "the Dr. Fox effect," named for a hired actor who purported to be "Dr. Fox" and who gave a

nonsensical university lecture in a wildly entertaining style, and got outstanding student evaluations for his brilliance.

• In one notorious study, those who saw just a 30-second soundless video of a teacher in action gave him virtually the same

ratings as the students taking his course -- who'd spent a semester reading, writing, thinking, and talking with him. Smile

warmly for the first 30 seconds of the first class in January, and you'll get good scores in May.

Give Higher Grades, Use More

Hand Gestures… What to Do?

There are some more effective ways to

improve student learning in the classroom.

Mine Your IDEA Results and

Other Data

Using IDEA Results to Improve Instruction &

Learning: EXAMPLES

1. Use student PERCEPTIONS of general and

specific SLO progress.

2. Compare student PERCEPTIONS with

some measure of REALITY.

3. Show PROGRESS in various areas over time in

the same course from semester-to-semester.

4. Sift through student COMMENTS to find

relevant and useful information.

Student Perceptions of

Achieving Progress on SLOs

• General (Progress on Course Objectives) • This is a summary evaluation, based on a number of questions in the form.

• It does reflect student perception on their progress.

• The student responses that are used for this indicator are those objectives

(items 21-32) that the professor picked as “important” or “essential”

(counting double)

• Specific (Supplemental Questions

provided by the Instructor)

Indirect Measures-SLOs

• You can use information from your IDEA

results (supplemental questions) to

examine student perceptions of achieving

SLOs in your courses.

No.

students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7

SLO#8 SLO#9 SLO#10

Murphy

MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy

TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean

(Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

INDIRECT MEASURES OF STUDENT LEARNING OUTCOMES

Chemistry 105: General Chemistry I

Fall 2010

Even More…

• Once you have a good idea of student

perceptions on whether or not they

achieved the SLOs, compare their

perceptions with REALITY (direct

measures).

• Pick questions which directly measure

student achievement on a single SLO on a

test or examination at the end of the

course. Mine the data.

Compare Student Perception with

Reality: GEN CHEM I

• SLO #6 “After taking this course, I feel I am

able to balance chemical equations.”

No.

students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7

SLO#8 SLO#9 SLO#10

Murphy

MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy

TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean

(Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

From the ACS Gen. Chem. I Examination (questions #15 and #24) given right after

the IDEA Evaluations, 23/25 and 20/21 of the students could correctly balance a

chemical equation.

CHEM 105 Student Perception Direct Measure

MWF 8 4.5/5.0 23/25 (93%)

TR 8 4.8/5.0 20/21 (95%)

Compare Student Perception with

Reality: GEN. CHEM I

No.

students

SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7

SLO#8 SLO#9 SLO#10

Murphy

MWF 8

25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9

Murphy

TR 8

21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9

Mean

(Murphy)

23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9

From the ACS Gen. Chem. I Examination (questions #10 and #16) given right after

the IDEA Evaluations, 13/25 (52%) and 11/21 (53%) of the students could correctly

do this.

Student Perception Direct Measure

MWF 8 3.9/5.0 13/25 (52%)

TR 8 3.9/5.0 11/21 (53%)

• SLO #10 “After taking this course, I feel I am able to calculate the standard

enthalpy of reaction for a given series of reactions by using tabular data.”

SHOW TEMPORAL

PROGRESS

CHEMISTRY 105: General Chemistry I Fall Classes

IDEA Results: Must be reliable and representative

Progress on Objectives Excellent Teacher

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

0 2 4 6 8 10

Pro

gre

ss o

n O

bje

tiv

es

Year Since Started Teaching at Huntingdon

Progress on Objectives

3.8

4

4.2

4.4

4.6

4.8

5

0 2 4 6 8

Excellen

t T

eah

er

Rati

ng

fr

om

ID

EA

Fo

rms

Year Teaching at Huntingdon

Excellent Teacher

What If?

3

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

0 1 2 3 4 5 6Pro

gre

ss o

n C

ou

rse O

bje

cti

ves

-ID

EA

Teaching Year

Progress on Course Objectives

Think about the above results: 1. Same course objectives selected each year?

2. Changes in class size and/or population

3. Change in teaching techniques

4. Other

5. Supplement with additional measures

How Can Student

Comments Be Used

Effectively?

Sample Student Comments • Huntingdon College

• IDEA Survey Student Comments

• Spring 2011

• Murphy, Maureen – CHEM 106 – MWF 8:00

• · Great instructor!

• · Great instructor. Made concepts easy to understand.

• · Dr. Murphy was an outstanding instructor.

• · Left-handed teachers-step away from board so students on the right side of the classroom can see more!

• · Great class!!!

• Murphy, Maureen – CHEM 106 – TR 8:00

• · The course and approach is fair and straightforward. I believe all courses and

• institutions should have a Dr. Murphy.

• -Quit being so positive-heck, it is 8 am in the morning!

• · Great teacher!! Little hard but wiling to help!!

• · E for the cell= cathode – anode

• · Dr. Murphy has been my favorite teacher in my 2 years at Huntingdon and done

• so teaching one of my least favorite subjects! She’s the best!

• · Dr. Murphy puts a lot of her time in to help her students better understand

• chemistry. She has the most office hours out of all of my professors!

• Murphy, Maureen – CHEM 446 – MW 19:00

• · I really appreciate Dr. Murphy’s concern for her students. She is very open to

• helping whomever needs help and she wants the students to understand the

• material. Her classes are never boring and I always learn a lot. Overall I very

• much enjoyed the class and Dr. Muphy’s teaching and learned a lot.

• · You are an amazing teacher and I appreciate all that you do. If every teacher,

• faculty, and staff member were like you.

• · Dr. Murphy is the best teacher I’ve had in my entire life. She changed my life

• forever. I couldn’t begin to try to name everything she has done for me so I’m not

• going to try. I could never repay her or thank her enough. I love you Dr.

• Murphy, thank you.

• Gets distracted sometimes talking about her husband teaching chemistry also.

Murphy, Maureen – CHEM 409 – MWF 9:15

· Great teacher! Dr. Murphy really cares about seeing us

do well in her class as

well as our life. We each share our own personal

aademic relationship with her.

· Dr. Murphy is the greatest asset at Huntingdon

College. She is an incredible

teacher and cares infinitely about her students. Dr.

Murphy is the only reason I

am where I am today from an academic perspective.

She helped me when I

needed help, and pushed and dragged me through

when I needed dragging. I do

not agree with everything that Huntingdon does, but

having Dr. Murphy around

and involved is easily the smartest thing Huntingdon has

ever done. Thank you

Dr. Murphy, I will never be able to repay you for all that

you have given me.

·

COMMENTS IN BLUE MAY BE

HELPFUL IN IMPROVING

INSTRUCTION

Student Comments that

Deserve Attention • Tests are too hard and do not reflect the

level at which the professor teaches.

• Lets class out early every day-great class!

• Too fast. Slides are read to us in class

and we see 56-75 slides per 60 minutes.

• Cannot (or refuses to) answer our

questions in class.

• Not available or not there during office

hours.

SUMMARY • IDEA Student Ratings of Instruction contain

some useful information for faculty.

• Understanding what the ratings mean and how

they are constructed is important.

• Student perceptions can be useful, but are

limited.

• Supplemental questions can be effectively used

on IDEA forms, especially when paired with

direct assessment of progress on SLOs.

• Evaluation of ratings over time can be useful.

• “Sifted” comments from students can inform

future instruction.

YOUR QUESTIONS

??????????

Additional Resources

• http://chronicle.com/article/A-Tech-Happy-

Professor-Reboots/130741/