good assessment by design international gcse and gce comparative analyses dr. rose clesham
TRANSCRIPT
![Page 1: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/1.jpg)
Good Assessment by Design
International GCSE and GCE Comparative Analyses
Dr. Rose Clesham
![Page 2: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/2.jpg)
Put into context:we have looked at education systems that use high stakes external assessments for 16 and 18 year olds, in terms of:
• Contextual national characteristics and factors underpinning these educational systems
• How their curriculum and content standards are defined and operate
• How their summative national assessments are designed and their focus
![Page 3: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/3.jpg)
77%81%83%85% World economy
![Page 4: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/4.jpg)
The Assessment Analysis To provide comparisons between UK based GCSEs and GCEs, PISA tests, and selected high performing PISA jurisdictions in terms of:
• Their assessment structures and demands in summative assessments
• The use of question types
• Subject specific areas of focus (eg. use of context in maths, source based focus, particular skills)
• Providing examples of good assessment practice
In order to increase our global understanding, and inform the design of WCQ qualifications and assessments.
![Page 5: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/5.jpg)
Cross Business Research Approach
100+ people involved
• Internal/external subject team workshops
• Workshop subject groups agreed on and standardised subject definitions of mapping categories
• 2-4 raters at item level in each subject
• Results aggregated rather than a true score judgement
![Page 6: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/6.jpg)
The Methodology
![Page 7: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/7.jpg)
Content Standards
• Based on Uniform Content
Standards (Porter, DfE) that
can be applied
internationally, eg. for
Physics
![Page 8: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/8.jpg)
Cognitive Operations
• Based on the focus and
type of question. These
are similar to, but give
more detail than
Assessment Objectives –
and can be used to
compare national and
international
assessments (using a
variation of Porter)
![Page 9: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/9.jpg)
Cognitive Demand
• Based on considerations
of how content standards,
assessment questions and
mark schemes are written
and applied – mainly to do
with complexity and
linkage (using a new
model that incorporates
Bloom, Webb, Pollitt, Biggs
and Collis)
![Page 10: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/10.jpg)
![Page 11: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/11.jpg)
![Page 12: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/12.jpg)
![Page 13: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/13.jpg)
![Page 14: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/14.jpg)
Outcomes
For example, Content Standards in Mathematics at 16
![Page 15: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/15.jpg)
For example, Content Standards in Mathematics at 16
Outcomes
![Page 16: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/16.jpg)
For example, Content Standards in Mathematics at 16
Outcomes
![Page 17: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/17.jpg)
For example, Content Standards in Mathematics at 16
Outcomes
![Page 18: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/18.jpg)
For example, Content Standards in Mathematics
Outcomes
![Page 19: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/19.jpg)
For example, Content Standards in Mathematics at 16
Outcomes
![Page 20: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/20.jpg)
Cognitive operations in Science (at 16)
Cognitive operations
0%10%20%30%40%50%60%70%80%90%
100%
Engla
nd
Hong
Kong
NSW
iGCSE
Inte
rnat
iona
l GCSE
SQAPisa
Apply concepts / make connections
Analyse information
Demonstrate understanding
Perform procedures
Memorize
![Page 21: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/21.jpg)
Proportion of Lower to Higher Order Cognitive Operations in Science
70%
30%
![Page 22: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/22.jpg)
High order science cognitive operations: % differences from the mean, by country
![Page 23: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/23.jpg)
Cognitive demand in Science
52%
44%
4%
![Page 24: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/24.jpg)
% difference from mean of low demand items in science, by country
![Page 25: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/25.jpg)
% difference from mean of medium demand items in science, by country
![Page 26: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/26.jpg)
% difference from mean of high demand items in science, by country
![Page 27: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/27.jpg)
Dendrogram showing family relationships between Maths assessments
Porter’s Alignment Indices in Maths
![Page 28: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/28.jpg)
1960 1970 1980 1990 200240
45
50
55
60
65
Routine manual
Nonroutine manual
Routine cognitive
Nonroutine analytic
Nonroutine interac-tive
(Levy and Murnane)
Mean t
ask
inp
ut
as
perc
en
tile
s of
the 1
960
task
dis
trib
uti
on
The dilemma of schools:The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource
Why does all of this matter?Economy-wide measures of routine and non-routine task input (US)
![Page 29: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/29.jpg)
Education and assessments needs to prepare students:
• To deal with more rapid change than ever before
• For jobs that have not yet been created
• Using techniques that have not yet been invented
• To solve problems that we don’t yet know will arise
![Page 30: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/30.jpg)
• Appropriate content representation is a key assessment issue.
• In general, lower order cognitive operations dominate assessments.
• Higher order cognitive operations are not well represented in assessments, for example problem solving skills in maths and applying concepts and making connections in science.
• Cognitive operations and cognitive demand often work independently of each other and therefore need to be actively designed into assessments.
• A range of different question types was evident. The analysis showed that although more closed question types can assess a range of cognitive operations and demands, they are less effective than more open responses.
Two fundamental points …• No country’s assessment system was perfect either in design or in outcome- however, some
are clearly more designed and comprehensive than others
• Good assessment doesn’t happen by chance, it has to be carefully designed and developed to assess the knowledge and skills we want and value.
Overall conclusions of our research:
![Page 31: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/31.jpg)
This work has enabled us to:
Implications
• Evaluate assessments using a set of common criteria
• Make empirical and standardised judgements
• Contrast and compare
As our qualifications now move into developmental phases, we
need to ensure that they:
• Reflect intended curriculum aims and content standards
• Assess the knowledge and skills that will be needed for our learners
• Are informed by international best practice
• And comparable with best practice
• And will be more reliable, valid and discriminating in terms of design, focus and . . content
![Page 32: Good Assessment by Design International GCSE and GCE Comparative Analyses Dr. Rose Clesham](https://reader035.vdocuments.us/reader035/viewer/2022070410/56649ea45503460f94ba8bf9/html5/thumbnails/32.jpg)
Thank [email protected]