learning gain and confidence gain: design, practice ... · students endowed with individual...
TRANSCRIPT
![Page 1: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/1.jpg)
Learning Gain and Confidence Gain:
design, practice, evaluation
Dr Fabio R. Aricò
@FabioArico
L&T Festival
Medway – Sep 2017
![Page 2: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/2.jpg)
YOUR PRESENTER
Fabio AricòNational Teaching Fellow Senior Lecturer in MacroeconomicsSchool of Economics – University of East Anglia, UK
Research fields• Higher Education policy and practice (widen. access, satisfaction)• Technology Enhanced Learning• Self-Assessment and Academic Self-Efficacy
Twitter: @FabioArico
2
![Page 3: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/3.jpg)
ACKNOWLEDGEMENTS
HEA – Teaching Development Grant Scheme
HEFCE Piloting and Evaluating Measures of Learning Gain
UEA Students, Alumni, and Research Assistants
3
![Page 4: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/4.jpg)
AIMS & OBJECTIVES
My talk will be all about pedagogies which I have designed, used, and evaluated. They are now being rolled across different disciplines.
• Sharing my practice take home at least one idea!
• Inspiring research from research-led teachingto teaching-led research
• Support developments in the HE sector in the UK: TEF is coming (learning gain)
• Support personaldevelopment promoting and evidencing excellence.
4
![Page 5: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/5.jpg)
OUTLINE
1. Core Concepts
Peer-instruction Self-assessment & Self-efficacy Learning Gain
2. Peer-Instruction & Self-Assessment in an Active Learning Environment
3. Evaluating the Pedagogy: Learning Gain & Confidence Gain
4. Students’ Appraisal of the Pedagogy & Closing the Feedback Loop
5
![Page 6: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/6.jpg)
ETHICAL REMARK
You will be presented with data collected during teaching sessions.
Students involved have given informed consent for me to analyse their responses and present the results of this analysis.
I can assist with ethical queries as well, please ask me.
6
![Page 7: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/7.jpg)
1. Core Concepts
Peer-instructionSelf-assessment & Self-efficacyLearning Gain
7
![Page 8: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/8.jpg)
FLIPPED CLASS and PEER-INSTRUCTION
• Flipped classroom & Peer-Instruction pre-reading + student interaction Mazur (1997) Henderson and Dancy (2009) well-developed research in Physics and STEM.
• Learning analytics for Peer-Instruction Learning gains: Mazour Group - Bates & Galloway (2012) Student satisfaction: Hernandez Nanclares & Cerezo Menendez (2014).
• There is not much literature on the links with self-assessment skillsOpen field, with many unanswered questions e.g. role of demographics, language, previous background Pedagogically: self-assessment blends with flipping and Peer-instruction.
8
![Page 9: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/9.jpg)
SELF-EFFICACY and SELF-ASSESSMENT
Academic Self-Efficacy = confidence at performing academictasks and/or attaining academic goals.
Bandura (1977) 1. Mastery of experiences2. Vicarious experiences3. Verbal persuasion4. Environment and settings
See also: Pajares (1996) and Ritchie (2015).
Idea: Students should develop their self-efficacy to master theirlearning experience. Measure learning gain along with increased self-efficacy: ‘confidence gain’.
9
![Page 10: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/10.jpg)
LEARNING GAIN
Policy-driven research to assess student learning
Arum & Roksa (2010), ‘Academically Adrift’ (US data)
OECD approach: assessment of learning outcomesAHELO Project (OECD, 2011 and 2014)
US & UK approach: learning gain (‘distance run’ over time)McGrath et al., 2015
HEFCE commissioned research on piloting measures of learning gain Outputs to feed in Teaching Excellence Framework.
10
![Page 11: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/11.jpg)
2. Peer-Instruction &Self-Assessment in anActive Learning Environment
11
![Page 12: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/12.jpg)
ACTIVE LEARNING ENVIRONMENT
Introductory Macroeconomics (from 2013 to 2017)
• year-long module (compulsory 1st year)
• 250 students (started with 140 in 2013, 250 past 2yrs)
• 22 lectures (2hrs per week)
• 8 seminars (every second week, even)
• 8 workshops (every second week, odd)
Students endowed with individual Audience Response Systems (clickers)
continuous data collection facilitated by technology; comprehensive ethical approval obtained beforehand.
12
![Page 13: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/13.jpg)
WORKSHOPS – teaching algorithm
13
Round 1- formative question- 4 choices- no information- no answer
Self-Assessment 1- confidence question- 4 level Likert-scale- information shared
Peer-Instruction- students talk- compare answers- explain each other
Round 2- formative question- Identical to R1- information shared- correct answer
Self-Assessment 2- confidence question- 4 level Likert-scale- information shared
![Page 14: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/14.jpg)
Holding everything else constant in the world economy: If inflation in a country increases, its…
A. exports decrease and imports stay the same;
B. exports decrease and imports increase;
C. exports increase and imports decrease;
D. imports increase.
14
![Page 15: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/15.jpg)
“I think I have the skills/knowledge to answer the previous question correctly”
A. strongly agree;
B. agree;
C. disagree;
D. strongly disagree.
15
![Page 16: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/16.jpg)
Holding everything else constant in the world economy: If inflation in a country increases, its…
A. exports decrease and imports stay the same;
B. exports decrease and imports increase;
C. exports increase and imports decrease;
D. imports increase.
16
![Page 17: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/17.jpg)
“NOW I think I have the skills/knowledge to answer the previous question correctly”
A. strongly agree;
B. agree;
C. disagree;
D. strongly disagree.
17
![Page 18: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/18.jpg)
![Page 19: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/19.jpg)
![Page 20: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/20.jpg)
WORKSHOPS – asking the right questions
20
Bloom’s Taxonomy
• Aim to climb up the pyramid• Do not trivialise MCQs.
See work by Simon Lancaster.
![Page 21: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/21.jpg)
3. Evaluating the Pedagogy:
Learning Gain & Confidence Gain
21
![Page 22: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/22.jpg)
DATASETS AND CODING
Student Q1 Q2 Q3 …
1 0 1 1
2 1 0 0
3 1 1 …
…
performance per questionconfidence by question
pe
rform
ance
pe
r stud
en
tco
nfid
en
ce by stu
de
nt
22
Formative questions1 = correct 0 = incorrect
Confidence questions1 = strongly/agree0 = strongly/disagree
![Page 23: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/23.jpg)
OPERATIONALISING TWO GAINS
For each 1st and 2nd response to formative assessment questions:
% correct R2 % correct R1Normalised Learning Gain (NLG) =
100% % correct R1
For each 1st and 2nd response to self-assessment questions:
% confident R2 % confident R1Normalised Confidence Gain (NCG) =
100% % confident R1
23
![Page 24: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/24.jpg)
PEDAGOGY EVALUATION STRATEGY
24
1. Is the pedagogy developing good self-assessment skills?Are students self-assessing correctly over Round 1/2?
2. Is peer-instruction able to generate learning/confidence gain?How does learning/confidence gain relate to initial knowledge/confidence levels (Round 1)?
3. Is learning gain associated to confidence gain?Does the structure of the algorithm affect this relationship?
2016 Vicarious of Experience Scenario (VES)
2017 Mastery of Experience Scenario (MES) onlycontrasted with VES (4 sessions each).
![Page 25: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/25.jpg)
WORKSHOPS – contrast 2 teaching algorithms
25
Round 1- formative question- 4 choices- no information- no answer
Self-Assessment 1- confidence question- 4 level Likert-scale- information shared
Peer-Instruction- students talk- compare answers- explain each other
Round 2- formative question- Identical to R1- information shared- correct answer
Self-Assessment 2- confidence question- 4 level Likert-scale- information shared
VES MES
![Page 26: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/26.jpg)
26
% confident Round 1
% correct Round 1
RESULT – Self-assessment Skills Round 1
2016 VES =0.429*** R²=0.56
0.33
0.44
2017 VES =0.367*** R²=0.59
2017 MES =0.09***
![Page 27: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/27.jpg)
27
% confident Round 2
% correct Round 2
RESULT – Self-assessment Skills Round 2
2016 VES =0.282*** R²=0.26
0.57
0.60
2017 VES =0.322*** R²=0.64
2017 MES =1.44***
![Page 28: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/28.jpg)
RESULT – Normalised Gains in 2016
28
NLG
% Correct/Confidentin Round 1
2016 0.37 0.48
0.85
0.52
NCG
NLG R²=0.39
NCG R²=0.19
![Page 29: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/29.jpg)
RESULT – Normalised Gains in 2017
29
NLG
% Correct/Confidentin Round 1
0.40 0.42
0.67NCG
NLG R²=0.21
NCG R²=0.42=0.132**
![Page 30: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/30.jpg)
RESULT – Learning Gain & Confidence Gain
30
NCG
NLG
2016 R²=016
MES =0.125**
-1
0.31
0.28
2017 VES =0.19*** R²=0.39
![Page 31: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/31.jpg)
4. Students’ Appraisal ofthe Pedagogies
Closing the Feedback Loop
31
![Page 32: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/32.jpg)
WHAT DO STUDENT THINK?
• The literature on evaluation of TEL and Peer-Instruction is far too focused
on whether students ‘enjoy’ their experience (student satisfaction)
typical of academic practice literature.
• I want to give more focus on the perception of learning:
1st lecture: introduced the concept of Peer-Instruction
asked the students to share what they think of it.
each workshop: asked students to share their view on the session
and whether they felt they learnt from each other.
informal end-of-module feedback: what was the most effective component
of the blended learning environment mix within the module.
32
![Page 33: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/33.jpg)
33
1st lecture: “‘Peer-instruction’ sessions (students teaching each other) are more effective than lectures (teacher teaching students)”
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
strongly agree
agree
disagree
strongly disagree
No. Respondents = 82
![Page 34: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/34.jpg)
34
0%
10%
20%
30%
40%
50%
60%
70%
80%
Wk 4 Wk 6 Wk 8 Wk14 Wk16 Wk18 Wk20
strongly agree agree disagree strongly disagree
No. 89 38 72 69 52 39 40
Workshop feedback statement: “I have learnt more Economics by discussing answers with my classmates”
![Page 35: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/35.jpg)
35
Comparing student opinion about Peer-Instruction as an effective pedagogy before and after exposure
0%
10%
20%
30%
40%
50%
60%
70%
1st Lecture (N=82) Avg Workshop (N=57)
strongly agree
agree
disagree
strongly disagree
![Page 36: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/36.jpg)
6%
20%
51%
4%
6%
13%Lectures
Seminars
Workshops
Support
VLE
NA
36
End-of-module Feedback: What is the component of the Macro module which had the strongest impact on your learning?
![Page 37: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/37.jpg)
37
FEEDBACK to STUDENTS
• During polling sessions:
show results, talk to the students, encourage them;
use comparative-links, show them their progress;
explain what you are doing and why you are doing it.
• After polling sessions:
share session reports on your VLE and comment on these;
use mail-merge to send individual reports;
• Go beyond polling:
craft your own perfect pedagogical blend;
be aware of different student needs.
![Page 38: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/38.jpg)
38
FINAL REMARKS
• It took me 4 years to develop this teaching approach & evaluation.Think big….but start small, and build from there.
• It’s not about the technology, it’s about the pedagogy and, most importantly, it’s about the students.
• Be concerned about ethics, but do not be discouraged or scared (students are not: JISC, 2016).
• Choose your demonstrators carefully: your ‘average Fabio’ might be more convincing than a pedagogy expert or a techno-hyper-enthusiast.
![Page 39: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/39.jpg)
39
@FabioArico
“Promoting Active Learning Through Peer-Instruction and Self-Assessment: A Toolkit to Design, Support and Evaluate Teaching”, Educational Developments, SEDA, 17.1, 15-18.
STAY IN TOUCH!
![Page 40: Learning Gain and Confidence Gain: design, practice ... · Students endowed with individual Audience Response Systems (clickers) ... and whether they felt they learnt from each other](https://reader033.vdocuments.us/reader033/viewer/2022050223/5f68364b465c6f5c835c456d/html5/thumbnails/40.jpg)
Learning Gain and Confidence Gain:
design, practice, evaluation
Dr Fabio R. Aricò
@FabioArico
L&T Festival
Chatham – Sep 2017