the 'unknown unknowns' of assessment rubrics in practice, policy and research
TRANSCRIPT
![Page 1: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/1.jpg)
Deakin University CRICOS Provider Code: 00113Bdeakin.edu.au/learning/enhancing-your-practice/cradle
The ‘unknown unknowns’ of assessment rubrics in practice, policy and research
Associate Professor Phillip DawsonAssociate Director
Centre for Research in Assessment & Digital Learning (CRADLE)Deakin University, Melbourne, Australia
@phillipdawson
![Page 2: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/2.jpg)
Deakin University CRICOS Provider Code: 00113B
![Page 3: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/3.jpg)
Deakin University CRICOS Provider Code: 00113B
A couple of decades ago, rubric began to take on a new meaning among educators. Measurement specialists who scored students’ written compositions began to use the term to describe the rules that guided their scoring. They could have easily employed a more readily comprehensible descriptor, such as scoring guide, but scoring guide lacked adequate opacity. Rubric was a decisively more opaque, hence technically attractive, descriptor. (Popham 1997, p. 72)
![Page 4: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/4.jpg)
Deakin University CRICOS Provider Code: 00113B
Hands up if rubrics make you feel:• Confident• Concerned• Relieved
• Confused• Excited• Bored• …another feeling
you’d like to share?
![Page 5: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/5.jpg)
Deakin University CRICOS Provider Code: 00113B
Pre-1997
19982000
20022004
20062008
20102012
20140
1000
2000
3000
4000
5000
6000
7000
Mentions on Google ScholarMentions in books
Cumulative frequency, 28 April, 2015.
![Page 6: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/6.jpg)
Deakin University CRICOS Provider Code: 00113B
What is a rubric?• A tool used in the process of evaluating
student work, which usually includes:– evaluative criteria– quality definitions for those criteria at
particular levels – a scoring strategy.
Per Popham (1997)
![Page 7: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/7.jpg)
![Page 8: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/8.jpg)
Deakin University CRICOS Provider Code: 00113B
This workshop• Overview of framework• Practice sharing• Scenarios
![Page 9: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/9.jpg)
Deakin University CRICOS Provider Code: 00113B
Framework overview• 14 ‘design elements’ representing choices
made in rubric design• Brief overview of each – stop me if you
want more detail
![Page 10: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/10.jpg)
Deakin University CRICOS Provider Code: 00113B
Specificity: the particular object of assessment
• Generic– Including faculty-wide
or course level• Task-type• Task-specific
![Page 11: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/11.jpg)
Deakin University CRICOS Provider Code: 00113B
Secrecy: who the rubric is shared with, and when it is shared
• Top-secret rubrics• Rubrics returned with marks/feedback• Rubrics shared with task instructions
![Page 12: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/12.jpg)
Deakin University CRICOS Provider Code: 00113B
Exemplars: work samples provided to illustrate quality
What does a good one of these look like?
Why is this one good?
![Page 13: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/13.jpg)
Deakin University CRICOS Provider Code: 00113B
Scoring strategy: procedures used to arrive at marks and grades
Analytic• Adding up• Logic, conditions,
thresholds• Combining scores from
multiple markers
Holistic• Assessor uses all
information to make a single overall quality judgement
![Page 14: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/14.jpg)
Deakin University CRICOS Provider Code: 00113B
Evaluative criteria: overall attributes required of the student
• (usually the leftmost column)• Mandatory (or not?)• Connector between indicators of quality
and task/subject outcomes
![Page 15: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/15.jpg)
Deakin University CRICOS Provider Code: 00113B
Quality levels: the number and type of levels of quality
• Grade levels (P/C/D/HD)
• Learning outcomes taxonomies
• Levels of study
• Fail?• Every criterion at
every level?
![Page 16: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/16.jpg)
Deakin University CRICOS Provider Code: 00113B
Quality definitions: explanations of attributes of different levels of quality
“Mixture of qualitative and
quantitative feedback sources”
![Page 17: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/17.jpg)
Deakin University CRICOS Provider Code: 00113B
Judgement complexity: the evaluative expertise required of users of the rubric
Pass Credit Distinction
Clarity of expression
Bad Acceptable Good
![Page 18: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/18.jpg)
Deakin University CRICOS Provider Code: 00113B
Judgement complexity: the evaluative expertise required of users of the rubric
Pass Credit Distinction
Clarity of expression
Five or fewer grammatical errors
Two or fewer errors
Zero grammatical errors
![Page 19: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/19.jpg)
Deakin University CRICOS Provider Code: 00113B
Users and uses: who makes use of the rubric, and to what end
Teachers• Marking• Providing feedback• Communicating
standards• Understanding standards
Students• Planning work• Understanding standards• Self assessment• Peer assessment• Grade grubbing• Planning improvement to
future work
![Page 20: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/20.jpg)
Deakin University CRICOS Provider Code: 00113B
Creators: the designers of the rubric
• Teachers• Textbook publishers• Internet
communities• Students
• Rubric specialists• Researchers• Professional bodies
![Page 21: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/21.jpg)
Deakin University CRICOS Provider Code: 00113B
Quality processes: approaches to ensure the reliability and validity of the rubric
• Formal reliability and validity processes• Informal review, piloting and revision• Haven’t got time for that, just see how it
goes
![Page 22: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/22.jpg)
Deakin University CRICOS Provider Code: 00113B
Accompanying feedback information: comments, annotation or other notes on student performance
• Stimulus for feedback discussion• Focal point for written feedback• A supplement to written feedback• Replacement for written feedback
![Page 23: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/23.jpg)
Deakin University CRICOS Provider Code: 00113B
Presentation: how the information in the rubric is displayed
![Page 24: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/24.jpg)
Deakin University CRICOS Provider Code: 00113B
Explanation: instructions or other additional information provided to users
• Written instructions• Discussion around rubric in class• Marking workshop• Video explanation
![Page 25: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/25.jpg)
Deakin University CRICOS Provider Code: 00113B
How do you use rubrics?• Do you feel strongly about any of these
elements?• Discuss how you use rubrics in groups of
three or four• Structure your conversation around the
framework
![Page 26: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/26.jpg)
Deakin University CRICOS Provider Code: 00113B
Scenarios – choose one, in groupsRubric advice• 1st year educational
theory course• 1000 students• Lecturer wants your
advice on rubric for 2000 word essay
Faculty rubric policy/procedures• Education faculty at a
large university have contracted you to write rubric policy/procedures
• What would you mandate?
![Page 27: The 'unknown unknowns' of assessment rubrics in practice, policy and research](https://reader036.vdocuments.us/reader036/viewer/2022062502/58aa316a1a28abbb108b59f7/html5/thumbnails/27.jpg)
Deakin University CRICOS Provider Code: 00113B
Questions/Comments?Pre-
19971998
20002002
20042006
20082010
20122014
0
1000
2000
3000
4000
5000
6000
7000
Mentions on Google ScholarMentions in books
@phillipdawson