the student’s world: learning and evaluation from the learner’s perspective kevin w. eva centre...

The student’s world:The student’s world:

Learning and Evaluation Learning and Evaluation from the Learner’sfrom the Learner’s

PerspectivePerspective

Kevin W. Eva

Centre for Health Education Scholarship

University of British Columbia

Presenter DisclosurePresenter Disclosure

I have been unable to achieve I have been unable to achieve any conflicts of interest for the any conflicts of interest for the

material included in this material included in this presentationpresentation

I just want to learnsomething new!!

AnAn over-generalizationover-generalization

\ˈfēd-ˌbak\\ˈfēd-ˌbak\

Providing constructive feedback Providing constructive feedback (1991)(1991)

1.1.Discuss the basis for evaluationDiscuss the basis for evaluation

2.2.Base feedback on specific Base feedback on specific observationsobservations

3.3.Use a balance of positive and negativeUse a balance of positive and negative

4.4.Et cetera …Et cetera …

Feedback as ImprovFeedback as Improv

““The dialogue, action, story, The dialogue, action, story, and characters are created and characters are created collaboratively as the story collaboratively as the story

unfolds.”unfolds.”

•Evaluating performanceEvaluating performance

•Incorporating feedbackIncorporating feedback

•Building professional identityBuilding professional identity

The student’s worldThe student’s world

What do these things What do these things have in common?have in common?

Running a triathlonRunning a triathlon

Writing a prescriptionWriting a prescription

Driving a carDriving a car

Offering a consultationOffering a consultation

Judging one’s own abilityJudging one’s own ability

““It’s a comfortability … when I It’s a comfortability … when I don’t have to think about it don’t have to think about it

that’s when I realize, more or that’s when I realize, more or less, [that I] have achieved my less, [that I] have achieved my

goal”goal”(Kindler, et al., 2013)(Kindler, et al., 2013)

TheThe self-assessment self-assessment literatureliterature

•Hundreds of articlesHundreds of articles

•Many literature reviewsMany literature reviews

•One conclusionOne conclusion

Self-assessment abilitySelf-assessment ability

is generally pooris generally poor

•Imagine yourself drivingImagine yourself driving

•It’s easy to do …It’s easy to do …• … … but details are inevitably left outbut details are inevitably left out• … … and those details are crucial to and those details are crucial to

determining the outcome of the actual determining the outcome of the actual experienceexperience

(Vallone, et al., 1990)(Vallone, et al., 1990)

WhyWhy can’t we self-can’t we self-assess?assess?

•Furthermore …Furthermore …

•Did anyone imagine themselves Did anyone imagine themselves having difficulty driving?having difficulty driving?

•When it’s easy to imagine an event When it’s easy to imagine an event we tend to overestimate its we tend to overestimate its likelihoodlikelihood

(The availability heuristic: Tversky and Kahneman, 1973)(The availability heuristic: Tversky and Kahneman, 1973)

WhyWhy can’t we self-can’t we self-assess?assess?

Jowett, etJowett, et al. (2007)al. (2007)

Learner

Material

TheThe generalgeneral pointpoint

•When students self-assess they do so When students self-assess they do so by drawing inferences….by drawing inferences….

……just as we do when we make just as we do when we make judgments about their performance…judgments about their performance…

……and, as a result, they’re just as and, as a result, they’re just as falliblefallible

ImplicationsImplications

•External guidance is required …External guidance is required …

Shute (2008)Shute (2008)

“formative feedback has been shown … to improve students’ learning and enhance teachers’ teaching

to the extent that the learners are receptive and the feedback is valid, objective, focused, and clear.

[BUT] there is no “best” type of formative feedback for all learners and learning outcomes

Shute (2008)Shute (2008)

…to the extent that the learners are receptive…

““Physician agreement Physician agreement with colleagues’ scores with colleagues’ scores

correlated positively correlated positively with overall mean scores with overall mean scores

from colleagues from colleagues (R=.48)”(R=.48)”

Sargeant et al. (2003)Sargeant et al. (2003)

““Physician agreement Physician agreement with colleagues’ scores with colleagues’ scores

correlated positively correlated positively with overall mean scores with overall mean scores

from colleagues from colleagues (R=.48)”(R=.48)”

Sargeant et al. (2003)Sargeant et al. (2003)

Which factors, personal or external, Which factors, personal or external, most influence students’ generation most influence students’ generation

of learning goals?of learning goals?

(Eva, et al., 2010)

What predicted generation of What predicted generation of learning goals?learning goals?

Overall R2=0.45

NOTE: Absolute values illustrated for ease of comparison (Eva, et al., 2010)

““It is impossible for It is impossible for a man to learn a man to learn

what he thinks he what he thinks he already knows”already knows”

Epictetus Epictetus

(circa 100AD)(circa 100AD)

““It is impossible for It is impossible for a man to learn a man to learn

what he thinks he what he thinks he already knows”already knows”

Epictetus Epictetus

(circa 100AD)(circa 100AD)

““I mean sometimes you’ll do it and I mean sometimes you’ll do it and somebody will watch you and it didn’t somebody will watch you and it didn’t work and they’re like, ‘You did everything work and they’re like, ‘You did everything right, right, it’s just one of those things.it’s just one of those things.’ But if ’ But if it’s successful then I think, yeah, I can do it’s successful then I think, yeah, I can do that.”that.”

““Some days I’ll go home and say I was a Some days I’ll go home and say I was a really good doc today and feel good about really good doc today and feel good about what happened in the day. And then other what happened in the day. And then other days you know, days you know, if I have a headacheif I have a headache, , maybe I wasn’t so good today.”maybe I wasn’t so good today.”

(Eva, et al. 2011; AHSE) (Eva, et al. 2011; AHSE)

Discounting the Discounting the negativenegative

Seeking the positiveSeeking the positive

(Harrison, et al., 2013)(Harrison, et al., 2013)

Self EfficacySelf Efficacy

• InfluencedInfluenced by feedback, but also by feedback, but also influencesinfluences future performance future performance

• Not merely a passive reflection of Not merely a passive reflection of performance, but part of a self-performance, but part of a self-fulfilling prophecy that affects fulfilling prophecy that affects performanceperformance

(see Bandura, 1977)(see Bandura, 1977)



… … that is sensitive to the that is sensitive to the perceptions and perspectives of perceptions and perspectives of the recipient …the recipient …

The need for The need for confidence/safetyconfidence/safety

““If you feel bad and you feel If you feel bad and you feel uncomfortable, you’re gonna not uncomfortable, you’re gonna not have the confidence to talk about have the confidence to talk about mistakes …you don’t get any kind mistakes …you don’t get any kind of constructive feedback because of constructive feedback because …you’re not comfortable saying …you’re not comfortable saying

it”it”(Eva, et al., 2012)(Eva, et al., 2012)

Self-perception

Performance

Learning goals

Motivation

AwarenessBenchmarkin

g

Feedback

A common perspectiveA common perspective

Self-perception

Performance

Feedback

A common model in A common model in HPEHPE revised

““Desirable difficulties”Desirable difficulties”

Creating situations that will Creating situations that will enable learners to discover enable learners to discover the limits of their the limits of their knowledge/abilityknowledge/ability

(see Eva, 2007 (BMJ); 2009 (AHSE))(see Eva, 2007 (BMJ); 2009 (AHSE))

Test-enhanced learningTest-enhanced learning

See Larsen, Butler, and Roediger (2008)

What predicts review What predicts review time?time?

(Agrawal, et al., 2012)

““Education is Education is learning what you learning what you didn’t even know didn’t even know you didn’t know”you didn’t know”

(Boorstin, 1914)(Boorstin, 1914)

““Education is Education is learning what you learning what you didn’t even know didn’t even know you didn’t know”you didn’t know”

(Boorstin, 1914)(Boorstin, 1914)

““Studies that augmented Studies that augmented feedback with consultation feedback with consultation

produced substantially produced substantially larger differences, but larger differences, but other methodological other methodological

variations had little effect”variations had little effect”

(Marsh and Roche, 1997)(Marsh and Roche, 1997)

““Studies that augmented Studies that augmented feedback with consultation feedback with consultation

produced substantially produced substantially larger differences, but larger differences, but other methodological other methodological

variations had little effect”variations had little effect”

(Marsh and Roche, 1997)(Marsh and Roche, 1997)

CredibilityCredibility

Of relationships: Of relationships:

Engagement and awarenessEngagement and awareness

Of the process: Of the process:

Validity and authenticity Validity and authenticity

Of intent: Of intent:

Beneficence and non-maleficenceBeneficence and non-maleficence

(Sargeant, et al., 2010)(Sargeant, et al., 2010)

CredibilityCredibility

““It doesn’t mean it’s nice to hear It doesn’t mean it’s nice to hear it. It’s not so nice; you don’t feel it. It’s not so nice; you don’t feel

so good after it. But it’s a so good after it. But it’s a difference when it’s real, if what difference when it’s real, if what they’re saying is just to help you, they’re saying is just to help you,

or if it’s just to put you down”or if it’s just to put you down”

(Eva, et al., 2012)(Eva, et al., 2012)



… … that is sensitive to the that is sensitive to the perceptions and perspectives of perceptions and perspectives of the recipient …the recipient …

… … while taking into account the while taking into account the inherent risk/strain learners feelinherent risk/strain learners feel

•2 slides of summary 2 slides of summary

•2 slides of discomfort2 slides of discomfort

Promoting receptivityPromoting receptivity

1.1. Self-assessment is vitalSelf-assessment is vital

2.2. Self-confidence is importantSelf-confidence is important

3.3. Self-assurance is powerfulSelf-assurance is powerful

4.4. Self-discovery is crucialSelf-discovery is crucial

5.5. Self-reliance can be Self-reliance can be detrimentaldetrimental

Moving beyond the scriptMoving beyond the script

Things to do:

•Avoid normative comparisons

•Use praise wisely

•Different styles for different tasks

(see Shute, 2008)

•Keep the relationship/dialogue in mind

Boehler, Rogers, et al. Boehler, Rogers, et al. (2006)(2006)

If you think this is someone else’s problem, you’re not alone

…

Pronin, Lin, Ross (2002)

…

ThanksThanks

[email protected]

Further reading:Eva, Regehr, Gruppen (2012). Blinded by insight.

the student’s world: learning and evaluation from the learner’s perspective kevin w. eva centre...

Documents

evaluationbase feedback

feedback quality0

negativeet cetera feedback

learning outcomesshute

colleagues scores

mean scores

self rating0

observer rating0