Name

School

Department

JUST IN TIME TEACHING

MDL WORKSHOP

JANUARY 17TH, 2015

JEFF LOATS

DEPARTMENT OF PHYSICS

WARM-UP: TEACHING HERITAGE

Thinking about the college instructors you've had

experiences with (including yourself), where do you

think their methods and attitudes come from? Why

do you think they teach the way that they do?

~67% → Imitation of our mentors

~33% → Teacher comfort/style

~25% → Experiences while teaching

~8% → Education classes

~17% → Workshop/observation tidbits

~17% → Learning styles

WARM-UP: TEACHING HERITAGE

“We tend to teach the way we were taught,

assuming that what went well for us has to go

well for our students.”

“I believe that instructors acquire their

experience mainly through contact with the

students and seeing what is effective and what is

not.”

WARM-UP: TEACHING HERITAGE

“[…] Same progression now, we are trying to

move to student-centered learning, but when the

students don't respond (or we can't figure out how

to motivate them to study), it's just back to the

old lecture format. It's easier to keep talking than

to find ways to get THEM to talk! It's easiest to

re-teach those who didn't study. And such a

vicious circle!”

WARM-UP: TEACHING HERITAGE

“Most instructors probably teach in a way similar

to how they have been taught - unless they have

been subjected to intense professional

development eradicating habit.

Or they teach to how they learn themselves. If

they are visual learners they may prefer to teach

by visual learning principles.”

ASIDE: LEARNING STYLES

“I think that many teachers teach in a way that

makes sense to them, according to their learning

style […]”

Best current evidence: Learning styles don’t exist

References:

• “The Myth of Learning Styles”

by Cedar Riener and Daniel Willingham

• YouTube: Learning Styles Don’t Exist

• Scholarly review: “Learning styles: Concepts

and evidence”, Pashler et al, 2008

WARM-UP: TEACHING HERITAGE

Thinking about the college instructors you've had

experiences with (including yourself), where do you

think their methods and attitudes come from? Why

do you think they teach the way that they do?

~67% → Imitation of our mentors

~33% → Teacher comfort/style

~25% → Experiences while teaching

~8% → Education classes

~17% → Workshop/observation tidbits

~17% → Learning styles

THE EVIDENCE STANDARD

Teachers can feel bombarded…

I strive to be a scholarly teacher …

• Apply the rigor we bring to our academic

discipline to the discipline of teaching.

• Choose teaching methods that are strongly

informed by the best empirical evidence

available.

Contrast teaching your subject with treating a

medical condition like diabetes

In your teaching do you have a method for holding

students accountable for preparing for class?

8% →Stern threats and/or playful pleading.

58% →Paper method (quiz, journal, others?)

8% →Digital method (clickers, others?)

0% →Just in Time Teaching.

25% →Some other method.

9

18%

49%

10%

5%

17%(𝑁 ≈ 180)

OVERVIEW

1. Motivation for change

2. Basics of Just-in-Time Teaching

3. Mock example

4. Evidence for effectiveness

5. Summaries

PHYSICS EDUCATION REVOLUTION

Eric Mazur, Physicist at Harvard:

“ALL SIMILARLY (IN)EFFECTIVE…”

University of

Washington

CU Boulder

University of Illinois

at Urbana-Champaign

TECHNIQUE & TECHNOLOGY

Technique:

Just-in-Time Teaching

Technology:

Online question & response tools

Learne

rTeacher

JUST-IN-TIME TEACHING

Online pre-class assignments

called WarmUps

First half - Students

• Conceptual questions, answered in sentences

• Graded on thoughtful effort

Second half - Instructor

• Responses are read “just in time”

• Instructor modifies that day’s plan accordingly.

• Aggregate and individual (anonymous) responses

are displayed in class.

Learne

rTeacher

Consider a typical day in your class. What fraction

of students did their preparatory work before

coming to class?

A) 0% - 20%

B) 20% - 40%

C) 40% - 60%

D) 60% - 80%

E) 80% - 100%

16

28%

33%

21%

14%

5%(𝑁 ≈ 206)

JITT STRUCTURE & RESPONSE

RATES

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

% R

esp

on

sed

Class #

Response Rate by Day

College Physics I, Fall 2013, N = 78

Worth 10% of final grade

Due 10 PM the night before class

Assignments available for prior 2-3 daysCollege Physics I

Students have developed a robot dog

and a robot cat, both of which can

run at 8 mph and walk at 4 mph.

A the end of the term, there is a race!

The robot cat must run for half of its

racing time, then walk.

The robot dog must run for half the

race distance, then walk.

A) The cat wins B) The dog wins C) They tie

18

WARM-UP: ROBODOG VS.

ROBOCAT

Predict which one will win the race, and explain

why you think so.

~60% → Robocat!

~0% → Robodog!

~30% → They tie!

~10% → Depends!

2 people gave no response

~20% → Good math

~10% →Bad math

~10% → Good reasoning

~50% → Bad reasoning

~10% → Invalid arguments

WARM-UP: ROBODOG VS.

ROBOCAT

“I was terrible at physics (my professor was a

really bad instructor and could never explain

difficult content well), so I checked out at 6th

grade :( I have a hunch that the cat may win

though.”

“The cat, if I program its racing time accordingly,

that is, a time shorter than the time it takes the

dog to get to the finish line.”

WARM-UP: ROBODOG VS.

ROBOCAT

“Both have the same running and walking

speeds, so they will cover the same distance

while running, therefore at the time of switching

to walking, they will still be next to each other

and walk together till the finish line, no matter

the distance nor the time that it takes. The cat

might though stop for some Friskies at Pet Smart

and the dog might be interested in a tree on the

way. "Run for half the race distance" or "Run for

half of its racing time" means the same to me.

These were my "one sentence or two".”

WARM-UP: ROBODOG VS.

ROBOCAT

“I think the cat will win. If the cat runs over half

the distance in the time it's racing, then it'll be

further that the dog when the time comes to walk.

I also like cats more. Thank god I teach french.”

“The Cat will win. In an 8 mile race, the dog will

run 4 mi and walk 4 mi. This will take 1.5 hours.

In the same time frame (1.5 hours) the cat will

run 45 min and walk 45 minutes, so the cat can

cover 9 miles in the same time that the dog can

cover only 8 miles.”

WARMUP QUESTIONS

• Every-day language

• Occasional simple comprehension question

• Mostly higher level questions (a la Bloom)

• I suspect any question is better than none

Connections to evidence:

–Pre-class work reduces working memory load

during class.

–Multimodal practice (not learning styles):

JiTT brings reading, writing and discussion as

modes of practice.

METACOGNITION

Two questions in every WarmUp:

First: “What aspect of the material did you find

the most difficult or interesting.”

Last: “How much time did you spend on the pre-

class work for tomorrow?”

Connections to evidence:

–Forced practice at metacognition:

Students regularly evaluate their own

interaction with the material.

JUST-IN-TIME TEACHING

A different student role:

• Actively prepare for class(not just reading/watching)

• Actively engage in class

• Compare your progress & plan accordingly

A different instructor role:

• Actively prepare for class with you(not just going over last year’s notes )

• Modify class accordingly

• Create interactive engagement opportunities

Learne

rTeacher

MAZUR AFTER 1 YEAR

ELSEWHERE?

Mean on 1-5 scale

Preparation for class 4.06

Engagement during

class 3.93

Learning the material 3.79

STUDENT SURVEY RESULTS

9% 10%

81%

10%18%

73%

10%

22%

68%

0%

20%

40%

60%

80%

100%

Harmful Neutral Helpful

How did WarmUps affect your...

Preparation Engagement Learning

N = 781

STUDENT SURVEY QUOTES

Physics:

“Initially, it was hard for me to get used to the

warm-ups. It seemed like along with the

homework assignments there was a lot of things

to do. Eventually I got used to it and ultimately

the warmups really helped me to learn the

material and stay caught up with the class.”

“If it weren't for warm ups, the amount of time I

spent reading the book would have dropped by

75%”

WHAT MIGHT STOP YOU?

In terms of the technique:

Time, coverage, not doing your part, pushback…

In terms of the technology:

Learning curve, tech. failures, perfectionism…

In any reform of your teaching:

Reinventing, no support, too much at once…

In the time we have left, which topic would you

most like to discuss?

A) Writing good JiTT questions

B) The evidence base for JiTT

C) Choosing a good JiTT implementation tool

D) “The sales pitch” (getting student buy-in)

31

FEATURES OF A GOOD QUESTION32

What would a “good” response look like?

– A paragraph? (too long)

– One word? (too short)

Make sure the reading is needed to respond (but a sentence straight out of the book shouldn’t work).

Make sure a beginner can take a crack at the question

Be concrete:

– “Explain in 2-3 sentences.”

– “Give two brief examples.”

– “Explain how you got your estimate.”

“Game out” their responses a bit.

WRITE A QUESTION AND SHARE...33

Imagine an introductory course in your discipline.

Imagine a topic you discuss early in that course.

Pick one type, write one question:

–A “low level” question (remember, understand):Terms: “Define, repeat” or “describe, explain”

–A “higher level” question (apply, analyze, evaluate)Terms: “Sketch, use” or “compare, estimate”

Write for a few minutes, then to trade and answer your neighbor’s.

STUDIED EFFECTIVENESS

Used at hundreds of institutions

Dozens of studies/articles, in many disciplines:

Bio, Art Hist., Econ., Math, Psych., Chem., etc.

– Increase in content knowledge

– Improved student preparation for class

– Improved use of out-of-class time

– Increased attendance & engagement in class

– Improvement in affective measures

JITT VS. FINAL GRADE

CORRELATIONS

College Physics I, Fall 2013

0

20

40

60

80

100

0 10 20 30 40 50 60 70 80 90 100

Cu

mu

lati

ve S

core

(w

ith

ou

t w

arm

-up

s)

WarmUp Score

WarmUps vs. Cumulative Score

Correlation r = 0.71

PROGRESSIVE EXAMS

CORRELATIONSCollege Physics I:

Important disclosure: This was not a hypothesis we were

testing, it appeared as we analyzed the data. Could be

spurious.

0.18

0.33

0.43

0.54

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Mini Exam(week 4)

Exam 1(week 7)

Exam 2(week 11)

Final Exam(week 16)

No

ne

We

akSt

ron

gM

od

erat

e

Correlations between Total WarmUp Score

and Sequence of Exams

JITT STRUCTURE & RESPONSE

RATES

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

% R

esp

on

sed

Class #

Response Rate by Day

College Physics I, N =

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

% R

esp

on

sed

Class #

Response Rate by Day

Intro. Sociology, N =

23

Worth 10% of final grade

Due 10 PM the night before class

Assignments available for prior 2-3 days

College Physics I Intro. to Sociology

Worth 5% of final grade

Due 10 PM the night before class

Assignments available for prior 2-3 days

FEATURES OF A GOOD JITT TOOL38

All student responses on one webpage

Auto-grading: 2/2 for anything by default.

Click to email students from the response page.

“Frequently sent responses” a bit automated.

List of responses is either randomized or tracked

to distribute instructor attention.

Other “modern” web amenities, like autosave,

time warnings, etc.

SMALL ASIDE: TEXT EXPANDER39

Every professor should have this!

You define a short text string, such as “ttyl”

When typed instantly replaced: “Talk to you later!”

Best FREE tools for Windows:

– Texter (simple with some advanced tools)

–AutoHotKey (advanced and can do much more)

Best tools for Mac:

– TypeIt4Me (30 days free, $5 after that. Worth it)

WHAT TOOLS TO USE?40

• CMS/LMS (Blackboard, D2L, Moodle, etc.)

Ready to use, tools range from ok to awful

• Free service from JiTTDL.org.

Designed just for JiTT, but extra login, and the

site has not been improved in ~5 years

• Students email responses

Easy! Usually overwhelming and awful

• Blogging tools (WordPress)?

• New tools (TopHat, Learning Catalytics)?

THE SALES PITCH

The way we talk to our students impacts

• How they approach the assignments

• How they feel about the work they do

OVERARCHING MESSAGE

Communicating with your students (humans)

• Message (explicit statements)

• Attitude (subtext, body language, etc.)

Consistent subtext:

"I am here to help you learn, and I have thought

about your learning trajectory carefully."

Consistent attitude:

I am comfortable and relaxed about my part of

this partnership.

DAY 1 – GENERAL

Describe components of the course

• How each one is graded, and why.

• How each one is important for learning and/or

assessment.

Keep justifications short and succinct

Be honest:

"This is my first time using this method, and

there is a lot of data on how and why this is

effective and what the best practices are."

DAY 1 – JITT

When discussing JiTT:

“Today is going to feel pretty ‘normal.’ You’ll get

to see how this works starting next time, after

you’ve done your first warm-up.”

It isn’t “more assignments = more work,” but

rather “working in smaller chunks is more

effective and more efficient.”

ALLOW TIME

They (probably) won’t “buy it all” on Day 1

Emphasize that you will be consistent and they

will get to see its value over time

“This class is different, and I will say that to you,

but it really is something you will get to

see/experience every day.”

DAY 2 – JITT

Discuss their first experience with warm-ups

Share how many did them

Remind them of structure:

Release/due times, course value, grading

Remind them of the purpose of warm-ups:

–Student preparation

–Instructor preparation

(“Which I’ll show you now!”)

DAY 3 – JITT BITS

A different role for you:

• Actively prepare for class by engaging and

being reflective.

(not just reading/watching)

• Be ready to actively engage with the material

in class.

• Take regular “readings” on your experience

with the material compared to classmates.

Make plans accordingly.

DAY 3 – JITT BITS

A different role for me:

• I will actively prepare for class by engaging

and focusing on you.

(not just going over last year’s notes )

• I will modify the class plan based on what I

see in your preparatory work.

• I will consciously create chances for you to

grapple with the material in an active way.

STUDENTS: BUSY-WORK

DETECTORS

K-12 represents more than 13,000 hours of class

Students are experts at detecting what really

matters to an instructor:

• What does the instructor do with class time?

• What does the instructor talk about?

• Does the instructor push against the usual

“invisible contract” of the classroom?

DEMONSTRATING VALUE IN JITT

Ideas for demonstrating that you value JiTT

• Thank those who do them for giving you

insight into their learning.

• Bring at least one “difficult/interesting” item

from WarmUp to class each day.

• Give non-verbal cues that you value

discussing WarmUps as much (more) than

other course components.

• Be consistent!

CONSISTENCY

Be consistent with:

• Assignment releases

• Assignment due dates/times

• Follow-up in class

• Exam questions that build on WarmUps

MY SUMMARY

JiTT may be among the easiest research-based instructional strategies that you can consistently integrate into your teaching.

From an evidence-based perspective, JiTT addresses often-neglected areas.

The sales pitch, demonstrating value and consistency can make the JiTT experience shine for both you and the students.

MSU Denver is doing good work in supporting innovative pedagogy, like JiTT. We have a solid JiTT “User’s Group.”

YOUR SUMMARY

If you want to implement JiTT, what is the most

important next step?

Please… get in touch!

We have an MSU Denver user’s group.

Email: Jeff.Loats@gmail.com

Slides: www.slideshare.net/JeffLoats

JITT REFERENCES & RESOURCES

Simkins, Scott and Maier, Mark (Eds.) (2010) Just in Time Teaching: Across the Disciplines,

Across the Academy, Stylus Publishing.

Gregor M. Novak, Andrew Gavrini, Wolfgang Christian, Evelyn Patterson (1999) Just-in-Time

Teaching: Blending Active Learning with Web Technology. Prentice Hall. Upper Saddle River NJ.

K. A. Marrs, and G. Novak. (2004). Just-in-Time Teaching in Biology: Creating an Active Learner

Classroom Using the Internet. Cell Biology Education, v. 3, p. 49-61.

Jay R. Howard (2004). Just-in-Time Teaching in Sociology or How I Convinced My Students to

Actually Read the Assignment. Teaching Sociology, Vol. 32 (No. 4 ). pp. 385-390. Published by:

American Sociological Association

Stable URL: http://www.jstor.org/stable/3649666

S. Linneman, T. Plake (2006). Searching for the Difference: A Controlled Test of Just-in-Time

Teaching for Large-Enrollment Introductory Geology Courses. Journal of Geoscience Education,

Vol. 54 (No. 1)

Stable URL:http://www.nagt.org/nagt/jge/abstracts/jan06.html#v54p18

ON-DEMAND SLIDES