inquiry into effective teaching with one hand tied behind your back

Inquiry into effective teaching with Inquiry into effective teaching with one hand tied behind your backone hand tied behind your back

22 November 2006UBC Institute for the Scholarship of Teaching and Learning

Dan Bernstein, University of [email protected]

Overview of session• Invited to discuss my research methods• Among methodological issues in research on teaching, design and measurement are key• Look at examples of some of my methods• Embed them in larger framework as I go• Inform some practices by research literature•Recommend a different path from my own• Propose a big tent, even for research done without the best possible methods

Research Methods Issues

Design and Analysis Measurement



Random AssignmentExperiment

DescriptiveStudy




DescriptiveStudy

Treatment Control




DescriptiveStudy

Treatment Control

EXAMPLE

Experimental Procedures

• Commercial and home-made multi-media • Used video and live lecture with same

content as multi-media• Created text samples with same content• Assessment of levels of understanding with

mixed methods of responding• Individual instruction with unlimited time• Participants from introductory psychology

Lecture, Reading, and Computer

0

10

20

30

40

50

60

<= 69 70-79 80-89 90-100

LectureReadingsComputer




DescriptiveStudy

Treatment Control

Analyze components




DescriptiveStudy

Treatment Control

Analyze components

EXAMPLE

Mean Instructional Time by Instructional Mode

0

5

10

15

20

25

30

Computer Lecture Readings

Instructional time*

Time on task (in minutes)

0

10

20

30

40

50

60

70

Computer Lecture Reading




DescriptiveStudy

Treatment Control

EXAMPLE

Upgraded the lecture component

• Live lecture to portion of introductory class• Topic is reactive measurement and

quantum mechanics• Matched to locally authored multi-media

and text -- also irrelevant text control• Motivation sustained by possibility of test• Assessment layered by type of

understanding and format of question

Distribution of total test performance resulting from four methods of

instruction

0%

10%

20%

30%

40%

50%

60%

<= 59

60-69

70-79

80-89

90-100

<= 5960-6970-7980-8990-100




DescriptiveStudy

Treatment Control

Statistical comparison




DescriptiveStudy

Treatment Control

Statistical comparison

EXAMPLE

Pushed envelope on assessment

• Looked for performance assessment• Topic was searching a jail cell• Verbal assessment is recognizing and

stating guidelines for complete search• Authentic assessment is use of guidelines

in searching a jail cell assembled for testing

• High end interactive program developed by NETV for Nebraska Sheriff’s Association

Authentic v. Verbal Assessment

010

2030

4050

6070

8090

InteractiveComputer

Lecture

VerbalAuthentic *




DescriptiveStudy

Treatment Control

¿Ethical questions?¿Permission?




DescriptiveStudy

Benchmarks of achievement




DescriptiveStudy


EXAMPLE

Criteria for oral interviewTopic Threshold

understanding

(5-9 pts)

Good understanding (10-15 pts)

Advanced understanding (16-20 pts)

Free will and determinism

Acknowledges that people can take position of free will and position of determinism; provides an example of someone’s beliefs that are influenced

Identifies some form of evidence that supports determinants of beliefs; identifies a form of evidence that supports free will in values; likely takes soft determinism view, that both are true

Provides strong evidence related to conventional criteria for free will or determinism; identifies and uses more than one criterion for either view; articulates criteria in reference to own beliefs or practices

Criteria for oral interviewTopic Threshold

understanding

(5-9 pts)

Good understanding (10-15 pts)

Advanced understanding (16-20 pts)

Scientific community

Recognizes the distinction between empirical claims and assertions of value; can give an example of a scientific analysis of a specific topic; thinks of psychology in broad ways of knowing

Can provide an example of scientific question and a non-scientific question; describes a procedure characteristic of an empirical analysis;

thinks of psychology as an empirical enterprise

Provides an original example of a scientific and non-scientific question; describes more than one analytic tool with understanding of its interpretation; recognizes that psychology involves value assertions as well as empirical questions




DescriptiveStudy


Movement ofachievement




DescriptiveStudy


EXAMPLE

Movement ofachievement

[Universal] Design for Success

• Presume students can learn

• Discount need to sort or differentiate

• Maximize overall course performance

Benjamin Bloom promoted mastery • Based on practice

and feedback• Divide course into

many smaller units• Take examinations

and get results• Require taking exam

again until high score• IFF 95% correct =>

study next unit

Fred Keller promoted mastery• IFF 95% correct =>

study next unit• Course grade is number

of units passed• No penalty for repeating

and learning• All who pass 12 units =>

grade of A• Do A work on 10 units

=> grade of C

Also taught conventional lecture

• Mastery Class

• 95% contingency

• No penalty for learning

• Immediate feedback

• No lecture required

• Lecture class

• Same exam questions

• Two attempts per test

• In class feedback

• No contingency

Total amount learned

• Nearly twice as many at the high end of learning• Virtually no one failed to learn• Maximized learning for many students

Showed in amount and accuracy

• Many more questions answered

• Took 12 15-item tests• Lecture was three tests

of 20 items each• Certify more learning• Overall percent correct

also higher

No magic -- students studied better

• They put in more time to their learning

• There was more work asked for by the course

• Note that they report doing the reading more

• Preparation for class is key issue (later also)

• Guideline in US -- 2 hours outside for every 1 hour in class




DescriptiveStudy

What do you think about the relative utility of these approaches?



Direct Indirect



Direct Indirect

Authentic Verbal surrogate

Cognitive Affective



Direct Indirect


Cognitive Affective

¿EXAMPLES?



Direct Indirect


Repeatknowledge

Abstract understanding

Contextualapplication

Affect

EXAMPLE

Added online questions on reading

Feedback until mastery of topics

Addition of EDU Out of Class

05

10152025303540

Percent Students

<=69 70-79 80-89 90-100

Levels of Achievement

New Questions Web-aided



Direct Indirect


Repeatknowledge



Affect

YOUR EXAMPLES



Direct Indirect


Repeatknowledge



Affect

EXAMPLE

Reasonably Conceptual Items• Describe the results of [Razran’s] second experiment on Describe the results of [Razran’s] second experiment on

semantically mediated generalization (numbers) done semantically mediated generalization (numbers) done with a 13-year old boy. Be sure to identify the with a 13-year old boy. Be sure to identify the respondent behavior, the training stimuli, and the test respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than stimuli. Why is this example semantically rather than physically mediated generalization?physically mediated generalization?

• What do chicken scratches, raccoon rubbings and pig What do chicken scratches, raccoon rubbings and pig rooting have in common? What stimuli have pre-existing rooting have in common? What stimuli have pre-existing relations these bits of behavior? What does the relations these bits of behavior? What does the “misbehavior of organisms” have to do with the generality “misbehavior of organisms” have to do with the generality of principles of learning? of principles of learning?

Transform into Problems• Old question:

– What were the reinforcing consequences in the Welsh, Bernstein & Luthans (restaurant) study? How were the consequences identified?

• Problem-based Assessment :– Suppose you were asked to implement a motivational

program in a business with 25 employees engaged in the following activities: production planning, inventory delivery, direct production, packaging, and marketing. Based on your understanding of the restaurant study, how would you improve the quality of the employees work by using access to activities as a motivator? Describe the costs and benefits of the program and make a recommendation about implementation.

• Old question:– The behavioral context into which punishment is

added must be considered in evaluating the likely effect of a given stimulus. What conditions are essential in this analysis?

• New question:– Suppose it is your task to decrease the frequency of

college students engaging in binge drinking or games involving high levels of intoxication. You have been asked to set up a punishment program to eliminate this program before someone is hurt. What context for binge drinking would you identify first? What punishing consequence would you use? What characteristics would you include in your punishing system to maximize effectiveness? Please give an example of an additional element you would need to include to make the plan maximally effective.

• Old question:– Describe the results of the experiment on semantically mediated

generalization done with a 13-year-old Soviet boy. Be sure to identify the respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than physically mediated generalization?

• New question:– Generate your own example of classical conditioning using the

neutral stimuli “idea” and “rock” along with the eliciting relation between a very loud noise and increased heart rate. Your example should include all of the following components: a description of a conditioning procedure that would produce differing reactions to the two stimuli, a description of the procedure that tests for the direct effects of conditioning, a description of a procedure that would test for physically mediated generalization, a description of a procedure that would test for semantically mediated generalization, and the likely results of the three test procedures.

Distribution on 1st Attempt

05

10152025303540

Percent Students

<= 69 70-79 80-89 90-100


Old questions

Change of Questions

05

101520253035404550

Percent Students

<= 69 70-79 80-89 90-100


Old questions New questions



Direct Indirect


Repeatknowledge



Affect

GeneralinterestIntent to act



Direct Indirect


Repeatknowledge



Affect

FISHBEIN


Prediction of action through self reports

• Huge problem in social psychology

• Three decades of poor correlations

• Martin Fishbein saved the day– Specific action– Context– Time– Target

• Resulted in a more limited construct

• Gone was the global attitude



Direct Indirect


Repeatknowledge



Affect

MISCHEL


Global traits also challenged

• Personality tests based on general traits

• Three decades of data showed modest correlations, largely ~.30 (10% of variance)

• Walter Mischel offered less global theory

• Interaction between persons and situations

• Not useful in predicting what people do

• Replies have all been more specific, less global



Direct Indirect

Bounded surveyof perception

Reflectionon learning



Direct Indirect



Amount oflearning

Process of learning



Direct Indirect



Amount oflearning

Process of learning

FISHBEIN NISBETT &WILSON

Introspection is the first problem• Academic psychology grew from this issue• More general problem of human judgment• Many complications lead to non optimal

performance• Richard Nisbett and Timothy Wilson

reviewed decades of work• In general people do notdo not accurately report

the why of consciousness, only the what• Alternatives from Simon, from Kahneman &

Tversky -- all stress practice and feedback• Open question is capacity given training



Direct Indirect



Amount oflearning

Process of learning

FISHBEIN NISBETT &WILSON

NISBETT & WILSON



Direct Indirect

WHAT DO YOU THINK ABOUT THE UTILITY OF DIRECT AND INDIRECT MEASURES FOR RESEARCH ON TEACHING? YOUR EXPERIENCES?




DescriptiveStudy

Treatment Control

THIS IS THE PATH I TOOK, STARTINGFROM EXPERIMENTAL PSYCHOLOGY



Direct Indirect


Repeatknowledge



Affect

I RECOMMEND MOST FACULTY MEMBERS START WITH RICH DESCRIPTION

What kind of research on their What kind of research on their teaching is appropriate for typical teaching is appropriate for typical

university faculty members?university faculty members?Is this properly called scholarship?Is this properly called scholarship?

Rigorous EducationalResearch

NarrativeDemonstration Of Learning

Inquiry intoPromotion ofLearning

Benchmark of Exceptional Achievement

All are Public





Range of Purposes

DemonstrationOf Quality

Discover NewWays to Teach





Range ofMethods

Social Science Humanities

http://www.cte.ku.edu

Your Insights?

Ernest Boyer:

The work of the scholar remains incomplete

until it is understood and used by others.

Three functions of grading

• Certification of learning

• Motivation for learning

• Differentiation among learners

• Each has a legitimate purpose

• No one system does all well

Variability in conventional course

• Students learn at different rates• When course ends, fast learners get best results• Very good at identifying fast learners (differentiate)• Less good at motivating for more work

Variability in a mastery course

• Everyone learns until material is mastered• Reward is for work; subjective probability of success• Very good at certification of learning• Provides incentive for studying, no penalty if slow

When is mastery the right approach?

• Foundation courses -- want knowledge• Programs in which rate of learning is not a

criterion for success• Situations in which performance will not be

timed• Professions in which high skill is expected• Why tolerate ineffective teaching?• If we don’t care or think it can not be learned

by all

Scholarship Assessed (1997)• All forms of scholarship

include:

– Clear goals– Adequate preparation– Appropriate methods– Significant results– Reflective critique– Effective presentation

Glassick, Huber, & Maeroff

inquiry into effective teaching with one hand tied behind your back

Documents

research literaturerecommend

lecture componentlive

homemade multimedia

authored multimedia

scholarship of teaching

effective teaching

jail cellverbal assessment

instructional modetime