Bridging the Gap from Implementation to Attainment:

Utilising Results from International Comparative

Studies.

Surette van Staden PIRLS 2011 Co National Research Coordinator

University of Pretoria

Edu Lead Conference 14 April 2015

Aim of the Presentation

• Evidence of curricular attainment that has taken place at Grade 4 level across 11 languages in South Africa.

• Interrogation of benchmark results to provide evidence for possible improvement of curricular implementation at classroom level.

• Reflections of how teacher practice can benefit from evidence provided by international comparative studies.

BACKGROUND TO PIRLS 2011

What is PIRLS?

• An international comparative study in reading literacy under the auspices of the IEA

• Measures trends across years

• Monitors policy implementation over time

• Provides a global context for achievement

prePIRLS 2011

Grade 4

Afrikaans

English

isiNdebele

isiXhosa

isiZulu

Sepedi

Sesotho

Setswana

siSwati

Tshivenda

Xitsonga

PIRLS 2011

Grade 5

Afrikaans

English

PIRLS and prePIRLS 2011 Samples

Study Grade Attained sample

No of learners

prePIRLS 4 341 15 744

PIRLS 5 92 3 515

Total 433 19 259

• Implementation in 2011• Reading Literacy

Assessment• Background

Questionnaires

GRADE 4

prePIRLS 2011 RESULTS

prePIRLS 2011 Overall Results

Colombia International Centre point Botswana South Africa0

100

200

300

400

500

600

700

576

500 463 461

Ave

rag

e sc

ale

sco

re

prePIRLS 2011 OVERALL BENCHMARK

PERFORMANCE

International Benchmarks

Represented by a four-point scale:

Low International Benchmark (400 points)

Intermediate International Benchmark (475 points)

High International Benchmark (550 points)

Advanced International Benchmark (625 points)

PrePIRLS 2011 Overall Benchmark Performance

0 20 40 60 80 100 120 140 160 180

29% 71% 42% 18% 6%

Not Attained Low Intermediate High Advanced

prePIRLS 2011 Overall Benchmark Performance

% Achieving Benchmarks

Per

cen

tag

e o

f le

ar..

. 29 71

Not Attained Low

30

23

12

6

Low IntermediateHigh Advanced

Of 71%

prePIRLS 2011 Benchmark Performance by Test Language

South Africa

Afrikaans

English

isiNdebele

isiXhosa

isiZulu

Sepedi

Sesotho

Setswana

siSwati

Tshivenda

Xitsonga

29

12

10

31

38

29

57

36

34

24

53

47

6

15

19

0.2

0.4

0.8

0

0.1

0.1

0.25

0

0

Did not reach Low International benchmark Intemediate International BenchmarkHigh International Benchmark Advanced International benchmark

HOW DO THE BENCHMARKS TRANSLATE

INTO WHAT GRADE 4 LEARNERS ARE ABLE TO

DO?

LOW INTERNATIONAL BENCHMARK

400 POINTS

Literary Texts:- Locate and retrieve explicitly stated detail

Informational Texts:- Locate and reproduce 2-3 pieces of

information

Not Attained

Low

Intermediate

High

Advanced

0 10 20 30 40 50 60 70 80

29%

71%

This means learners are probably unskilled in:

• monitoring of their own memory and comprehension

• reading as a meaning-making process • detection of contradictions in text• successfully resolving inconsistencies in

understanding • exerting control of reading processes• realising that they do not understand what is

read

ADVANCED INTERNATIONAL BENCHMARK

625 POINTS

Literary Texts:- Integrate ideas and evidence across

text- Interpret story events to provide

reasons, motivations, feelings, character traits with text-based support

Informational Texts:- Distinguish and interpret complex

information from different parts of text with text-based support

- Integrate information across text to provide explanations, interpret significance and sequence activities

Not Attained

Low

Intermediate

High

Advanced

0 10 20 30 40 50 60 70 80

6%

This means learners are skilled in:

• demonstrating awareness of what they are reading

• knowing reasons why they are reading• having tentative plans or strategies in

place for handling potential problems • monitoring their comprehension of textual

information

The Value of Benchmark Information

• Unequivocally points to the problems in the foundational levels of the education system in getting learners to even master literal comprehension regardless of the language of instruction.

• The international benchmark medians shows that it is possible for Grade 4 learners to attain higher order comprehension skills at this developmental stage.

• Benchmark profiles reveal to the South African system and teachers specifically that learners are not engaged with mastery of reading comprehension or the higher order reading comprehension development needed for further academic achievement in the schooling system.

HOW COULD TEACHER PRACTICE BENEFIT FROM EVIDENCE PROVIDED BY

INTERNATIONAL COMPARATIVE STUDIES?

Implementation Challenges

• Teacher comparisons made only within their own school and neighbouring teachers.

• Uncertainty about activities that are undertaken, pace and the level which these are pitched are often only mediated by curricular expectations and the pace of work within the school as dictated by fellow teachers and school management.

The Value of prePIRLS 2011 Results to Teachers

• Making released reading materials and questions that are used for purposes of the study available to teachers.

• Familiarity with the processes of comprehension the assessment is aimed at: Focus on and retrieve explicitly stated information Making straightforward inference Interpret and integrate ideas and information Examine and evaluate language and text elements

• Making international benchmark descriptions available. • These may serve as a guide to teachers’ own development of

reading material in the classroom against an existing, international framework.

The Value of prePIRLS 2011 Results to Teachers (continued)

• The provision of released passages to teachers to serve as exemplars of levels of expectation at Grade 4 as benchmarked against international standards.

• Exemplary learner responses linked to these items in themselves may provide valuable insights into comprehension processes and developmental requirements to teachers.

• Could serve as a trigger to a more informed understanding of what learners at this developmental stage of learning are actually capable of achieving.

The Value of prePIRLS 2011 Results to Teachers (continued)

• The impact of the PIRLS cycles on macro level policy is irrefutable (e.g. Foundations for Learning, Drop All and Read).

• Yet teacher level feedback from international testing and assessment programmes is still in its infancy and not fully realised in South Africa.

• Feedback to teachers is of cardinal importance, since change at classroom level can only be implemented in the face of empirical evidence of what constitutes effective classroom practice.

• Where international assessment results remain only as grim reminders of South African learners’ poor performance, assessment as integral to learning has failed.

• Assessment results could provide evidence to inform further support or intervention for teachers and learners in need.

• Assessment should provide indicators of those systemic factors that can be changed, adapted or used as leverage against which to ensure improved curricular implementation.

• Failing to do so, assessment as agent of such improved curricular implementation, may become powerless in providing feedback of significant factors that are at work in the home, in classrooms and in schools nationwide.

Conclusions

Thank you for your attention.