assessment to support the development of problem-solving ... · grace grima associate professor...
TRANSCRIPT
Grace Grima
Associate Professor & Director of Research – Pearson [email protected]
Jennie Golding
Associate Professor in Mathematics Education, UCL [email protected]
1
Assessment to support the development of
problem-solving goals in mathematics curricula 5-16
Pearson: The Efficacy Agenda
evaluating the impact on improving lives through
learning
Delivering what we promise At Pearson, efficacy means making sure that our products and services work the way we say they do. It is at the very heart of what we do. Our efficacy strategy helps us create and deliver learning products, platforms, and services across global markets—all with the aim of improving learner outcomes. The Efficacy Agenda focuses on measuring the impact on improving lives through learning. Our research focuses on the value of measuring impact on learners and the value-added to schools. 3 What is efficacy at Pearson?
Image by Jose Vidal Carrasco
Defining our outcomes
We define learner outcomes for our products in these categories:
ACCESS
The learner’s access and experience
COMPLETION ACHIEVEMENT PROGRESSION
The timeliness of completion
The standard of achievement
or level of competence
Where the learner goes next: study,
training, or employment
4 Making progress
Product or service has well-defined
outcomes and measures and a
supporting theoretical evidence
base appropriate to its purpose
Usage of the product or service is designed
and achieves appropriate levels of
satisfaction
Evidence that the
product or service is shifting or influencing
leading indicators and/or evidence of a relation to outcomes
Evidence isolates the impact of the product
or service on the target outcome(s) with
educational and statistical significance
Evidence that the
impact of the product or service on the
target outcome(s) is replicated in
multiple situations
JUSTIFY ENGAGE CORRELATE CAUSE EXPLAIN
How much impact?
The Pearson Levels of Evidence1 allow us to apply rigour
to evaluating our impact on improving learner outcomes. LEVEL
EVIDENCE
5 Making progress
1Based on nesta.org.uk/sites/ default/files/standards_of_evidence.pdf
Benefits Research on learning in schools enables us to: • Improve our services by bringing in real data from the end users • Measure the impact of services and qualifications on learners • Have learner and teacher voices at the centre of our work • Evaluate and understand implementation / classroom processes • Understand the complexities of learning in the school context
6
Assessment to
support mathematical
problem-solving
8
The policy context
• Recent curriculum changes in England - renewed focus on
mathematical reasoning and problem solving.
• New curricula for 5-16 year olds rolled out from 2014: first ‘GCSE’ assessment for all 16-year-olds in 2017 first large scale A-level assessment for new curriculum- 2019
• GCSE and A-level assessment are high stakes and via four
Awarding Organisations, including Pearson.
• A coherent curriculum offer (Schmidt and Prawat 2006) depends on achieved alignment across the system: enacted curriculum, resources, assessments and teacher CPD / capacity to deliver.
• Pearson offers resources, assessment and teacher CPD.
.
9
We asked how the resources and assessments produced by Pearson impacted student and teacher experiences for the range of student outcomes. In particular: • How is mathematical problem-solving being enacted in classrooms?
• How do the issues of teacher capacity, resources, and assessments affect the teaching of
problem solving?
Research questions
10
The studies
Study When? Size
A Primary: 5-11 2016-18 (y1-2; y 5-6)
9 schools and mathematics coordinators,
18 classes and teachers
B Secondary: 11-16 2016-18
(y7-8 or y 8-9; 10-11)
15 schools and Heads of Mathematics (HoM),
32 classes and teachers
C GCSE Mathematics / progression: 15-17 2016-Nov 18
15 schools and HoMs, 30 GCSE classes and
teachers, 32 post-16 student
groups and teachers
D A level Mathematics: 16-18 2017-2020
12 schools and HoMs, 48 A level classes and
teachers
11
Methodology for Study B: an example
Schools using Pearson resources recruited. 2 classes per school.
2016 HoM and class teacher telephone interviews Student baseline assessments
Spring 2017 Spring 2018
Lesson observations: focus on new aspects of curriculum
Teacher post-lesson interviews
Student focus groups
Summer 2018
Class teacher surveys Student surveys
External assessment outcome date
Autumn 2018 Final reflections with HoMs
Data analysis using Research question themes leading to a grounded approach (Charmaz, 2014)
to developing subthemes
12
Profile of problem-solving in assessment:
Assessment Objectives
GCSE Foundation
tier
GCSE Higher tier
A Level Mathematics
AO1 Use and apply
standard mathematical techniques*
50% 40% 50%
AO2 Reason, interpret and
communicate mathematically
25% 30% 25%
AO3 Solve problems within
mathematics and in other contexts
25% 30% 25%
13
AO3: Solve problems within mathematics and other contexts Students should be able to: • Translate problems in mathematical or non-mathematical contexts into a
process or series of mathematical processes. • Make and use connections between different parts of mathematics. • Interpret results in the context of a given problem. • Evaluate methods used and results obtained. • Evaluate solutions to identify how they may have been affected by
assumptions made.
What is meant by problem solving in AO3?
14
Challenges: What is mathematical problem solving?
Mathematical ‘problem solving’ is a contested term in the literature. Problems are those tasks to which there is no familiar approach to a solution. Teaching for problem-solving is complex because it is likely to draw on deep conceptual understanding, mathematical reasoning and well-developed communication (e.g. Schoenfeld, 2007). It therefore makes substantial demands on the capacity for change of teachers coming new to it (Golding, 2017). In England, we have little historical experience of validly assessing mathematical problem solving, despite its presence in the intended curriculum*
*but note World Class Tests
A pattern is made from four identical squares. The sides of the squares are parallel to the axes. Point A has coordinates (6, 7) Point B has coordinates (38, 36) Point C is marked on the diagram. Work out the coordinates of C.
Is this problem solving?
15
ABCDEFGH is a cuboid. AB = 7.3 cm CH = 8.1 cm Angle BCA = 48° Find the size of the angle between AH and the plane ABCD. Give your answer correct to 1 decimal place.
Is this problem solving?
16
Is this problem solving?
17
Gavin, Harry and Isabel each earn the same monthly salary. Each month, • Gavin saves 28% of his salary and spends the rest of his salary.
• Harry spends 34
of his salary and saves the rest of his salary. • The amount of salary Isabel saves : the amount of salary she spends = 3 : 7
Work out who saves the most of their salary each month. You must show how you get your answer.
The principle: Teachers, students and end-users are supportive of students learning to problem-solve. They felt Pearson resources supported that well.
In-class experiences: Problem solving was initially interpreted as solving worded exercises and only for the most able.
CPD: Pearson provide paid-for and within-resource CPD. The former is rarely used (time and cost); the latter, infrequently (time, and low perception of need) – yet many lesson observations showed teachers ill-prepared for curriculum problem-solving intentions.
Assessment: National assessments at 11, 16, 18 appeared uppermost in teachers’ thinking and framed their understanding of the meaning of ‘problem-solving’.
18
Key findings from the research projects
Evidence from the classroom
(Study D, initial teacher interview 6)
19
One of the biggest issues industry has is finding people who can solve problems. It’s not even the direct application of mathematical topics. … It’s vital. So for me it’s a no-brainer in terms of tackling those kinds of things.
I think the nice thing about the challenge is that, if we continue teaching like this, …we can say to students …we believe that you can do this. I think it must be to the benefit of the students because it's just so much more interesting and useful. Obviously it's going to impact the way you teach, and it’s early days still…but I think that's a really good thing.
(Study B, HoM 13 Summer 2018)
I think the final set of mock papers probably gave us a closer indication to what the real paper is like. But even then, we still, we didn’t expect the kind of questions that we had.
(Study C, HoM19, Autumn 2017)
There were (in first live GCSE papers) lots of problems on top of problems and layered topics which … made it very difficult, so (problem solving) has to be a real focus.
(Study C, HoM 16, Autumn 2017).
Evidence from the classroom
(Study C, HoM11, Spring 2017)
20 (Study A, Y6 teacher 5, Spring 2018).
(Study A, Y2 LO 9)
(Study A, y6 teacher 1, Summer 2018);
(The teacher) did lots of modelling and breaking down of the questions for the children…. This meant that children, in fact, did not engage very much with identifying the steps/approaches/skills needed to solve the problems.
I was thinking if we're really struggling to work out what to do and to think, what chance have the students got when they're, you know, under pressure and not so experienced with that.
‘I, and the pupils, are becoming more confident as we are regularly exposed more to open, contextualised, multi-step problems’
‘Maybe I just need things a bit more highlighted for me, but that would be useful to have that as this is a problem solving and reasoning activity or this is an avenue, if you want to extend, to go off down here. …I don’t feel overly confident with problem solving’
21
• Teachers in all phases are not clear what is meant by ‘problem solving’. • They look first to assessments to clarify what is needed, but early assessments
have had variable demands. • Teachers often lack experience of, and skills and knowledge for, teaching
problem solving, yet might not realise that. • When good use is made of ‘educative’ resources (Davis and Krajcik, 2005)
including integral PD materials, teachers can become much more effective at teaching problem-solving – but this takes time and effort.
• Ofqual has now done some work about aspects of problem solving that correlate with demand (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/668587/Summer_2017_GCSE_Maths_assessments_review.pdf; see also https://www.tandfonline.com/doi/pdf/10.1080/14794802.2017.1334576
Threats to valid classroom enactment
22
Summary
• Mathematical problem solving is *hard* to teach, especially for teachers coming new to it.
• And *hard* to assess validly, fairly and reliably. • Good assessment of related learning requires shared
understanding of terms used, and of student learning. • Teachers of mathematics in England require focused
and substantial CPD. • Classroom enactment is highly framed by high-stakes
assessments: a coherent curriculum system supports the enactment of problem solving.
• Pearson / UCL research projects to be completed in 2018 and 2019 and will be reported on separately.