expert guided crowd sourced learning content
DESCRIPTION
Keynote presented at the Improving University Teaching Conference (http://www.iutconference.com/) Vancouver July 2014TRANSCRIPT
A pilot study in a large enrolment introductory science course
Expert-guided crowdsourced learning content
Simon Bates [email protected]
@simonpbates bit.ly/batestalks
~2010
~2006
2013
Context - flipped classrooms
CC BY-NC 2.0 https://flic.kr/p/f3ynHx
Derek Bruff: Class time reconsidered http://prezi.com/donq036eunko/class-time-reconsidered/
Weekly rhythm for our 1A class
Poh, M.Z., Swenson, N.C., Picard, R.W., "A Wearable Sensor for Unobtrusive, Long-‐term Assessment of Electrodermal AcEvity," IEEE TransacEons on Biomedical Engineering, vol.57, no.5, pp.1243-‐1252, May 2010. doi: 10.1109/TBME.2009.2038487
Weekly rhythm for our 1A class
Weekly rhythm for our 1A class
Learning gains on PI
Learning gains on PI
1
2
3
A ball initially at rest is thrown upwards, comes back down & is caught !
Which of the following is a plausible graph of the acceleration of the ball with time?
Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111
Wieman commentary PNAS http://www.pnas.org/cgi/doi/10.1073/pnas.1407304111
“This meta-analysis makes a powerful case that any college or university that is teaching its STEM courses by traditional lectures is providing an inferior education to its students”
~2010
~2006
2013
a"web&based"MCQ"repository"created"by"students"
Ins$tu$ons((signing(up(per(year:(!
2009:! ! !22(2010:! ! !66(2011:! ! !204(2012:! ! !266(2013!(Jan,Jun):! !214(
Growing(content(repository:(!Courses:! !2,500(Logins/month:! !75,000(Ques<ons:! !600,000(Answers:! !12,000,000(
Answers (20,000,000)
Questions (1,000,000)
Student'ownership'over'learning'resource'
Student'familiarity'with'social'so7ware'
Leveraging'student'energy'and'crea9vity'
Badges' Points' Leaderboards'
Typical implementation
Minimum participation requirements for each of two assessment exercises (PW1, PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves Winter Semester: 3 sections, 791 students
ScaffoldingTOCCLT'.' -
Qest\I\sct
How 1o...
q/t4osTer
Submit ond onswer questionson topics in lhe torget region,just obove lhe physicsyou hove olreody mostered.
Dislroclers
) l*"*"ò
rNrrltngIh s region corrk:ins llte
plrysics knowledgo ond colrco¡lsyou connol leorn yel
becouse the louncalionscrrc nol in plocc
PHYSICS TOPICS IN YOUR TARG T REGION
buo'5ot't1 'Den¡¡lu
B¿:o\v\vì 1 JF:rcq - YOUR CHOSEN TOPIC
à..\auC'q^ x@bôo'osro
pn ro, d logroo, \o 9tS
COMMON MISCONCEPÎIONS AND ERRORS(Sec hHÞ:/jÞhy¡.udofo¡.cdu/CJP/trêconcoÞllo¡t.pdt
tor o llil ol common mhconcepllons)
¿K N(,qht %cce Açxs no\ exi¡þ, c.rrrçr\J On oblec'-t \5 õÞYJrnr cìü?ÀI in rtuiò. ü
lrh'brot4ont eprç,e .I¡1enÀs crqden-s,\{^ c>Ç c.,þ¡ec\ , cìo+ .\*¿n:r
oÇ Çtu,8,fr- btrrqon! $crCe c¿c\s Àoc¡:'cr
fr¿lt{e- r *hcrn ú?
Trr¿e. $ììfifflsz hrrr: ¿lÕoolr-g lnnS
(**= tcpo þ/rnl {otr-* qrÇ bq.\\:. 0" Ooz
Physics knowledgeond conccpluol underslonding
you lrove olrcodyconslrucleC ¡n your heod
¡,<'t.lerJ rt)\^,¿z! Þ.rrp \r- \rr't\¡ ,l?
Chcck lhol yovr on3Íêr ¡t rêo!ànoblê ðnd potrlble
\'f<,,cc\
Photo by Seth Casteelhttp://www.littlefriendsphoto.comPermission to use agreed
Introduced in tutorials !
Extensive scaffolding exercises
!Revisited in subsequent
tutorials !
Tutorials delivered by 24 TAs
Engagement with PeerWise
Number Multiplier Number MultiplierQuestions 1105 [1.7] (1.4) 998 [1.6] (1.3)
Answers 11393 [17.2] (14.4) 11807 [18.7] (15.0)
Comments 4901 [7.4] (6.2) 5509 [8.7] (5.0)
PW 1 PW 2
Engagement with PeerWise
Engagement with PeerWise
PW1 PW2
Contributed and met minimum requirementsContributed but did not meet mininum requirementsDid not participate
0.84 0.80
Total: 791
Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare &assess various option possibilities; often conceptual
Synthesize Ideas and topics from disparate course sections combined. Significantly challenging problem.
Text
1 2 3 4 1 2 3 40
20
40
60
Num
ber o
f que
stio
ns
Assessment 1 Assessment 2
Bloom's Taxonomy: Question Quality
Textp>0.05, NS
0%
5%
10%
15%
20%25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
Per
cent
age
of S
ubm
itte
d Q
uest
ions
First semester N = 350
Second semester N = 252
\
Bates et al, PRST PER, in press 2014 http://arxiv.org/abs/1308.2202
Question/Explanation Quality
Description of explanation quality
Score Level Description
0 Missing No explanation provided or explanation incoherent/irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 MinimalCorrect answer but with insufficient explanation/justification/ Some aspects may be unclear/incorrect/confused.
3 Good Clear and detailed exposition of correct method & answer.
4 ExcellentThorough description of relevant physics and solution strategy. Plausibility of all answers considered. Beyond normal expectation for a correct solution
0 1 2 3 4 0 1 2 3 40
20
40
60
Num
ber o
f que
stio
ns
Assessment 1 Assessment 2
Explanation Quality
p=0.02
~2010
~2006
2013
PHYS101: Energy and Waves
Logistics !
!
Cohort split into 4 groups !
Each week one group tasked with creating LOs !
Each submission counts for 2.5% of final grade !
Repeat cycle twice per Semester !
Students can submit >2 LOs & receive grade for best 2 !
Short survey on submission !
Students encourage to apply CC licenses !
Results: engagement
0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Number of students
AssignedOptional
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
Grade (%)
Num
ber o
f stu
dent
sLO Grade Distribution
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
Grade (%)
Num
ber o
f stu
dent
sLO Grade Distribution
These students didn’t even try!
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
Grade (%)
Num
ber o
f stu
dent
sLO Grade Distribution
Which students aren’t submitting LOs?
Non-participants are from all walks of life!
0 10 20 30 40 50 60 70 80 90 1000
5
10
15
20
25
Final Exam (%)
Num
ber o
f stu
dent
sFinal Exam marks of students that did not participate in LOs
Results: types of Los
PPTDOCPreziTed-EdPDFImageOther
Results: time on task
0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of students
Results: self reported change in understanding
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students
Sample 2 - Standing Wave in a bowl
Sample 3 - Colour Loss Underwater
Student generated exam content
3. Successes
[email protected] @simonpbates bit.ly/batestalks
Relevant publications: !Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002 (2014) !Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education, 1-15 (2014). !Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) in press !Student-generated assessment - Education in Chemistry (2013) 13 1
Image credits
Crowd http://www.flickr.com/photos/18378655@N00/613445810/
