Education outcomes and contextual data: the potential of international surveys like PISA

Simone BloemOECD Directorate for Education

Programme for Co-operation with Non-member economies

Outline

• Overview of international surveys:

PISA TIMMS PIRLS

• Relevance of PISA for assessing integrity of education systems

• School system characteristics that are positively associated with performance, equity and integrity

• Next steps

2

International Surveys

3

•OECD’s Programme for International Student Assessment

PISA

•IAE’s Trends in International Mathematics and Science Study

TIMMS

•IAE’s Progress in International Reading Literacy Study

PIRLS

OECD: PISA

Triennial survey of key competencies of 15-year-old students

Main subjects tested: reading (2000, 2009), mathematics (2003) and science (2006)

Tool to compare the efficiency and equity of countries’ education systems in an international perspective

A map of PISA countries and economies

5

IAE: TIMMS

Assessment of mathematics and science competencies of 4th- and 8th-grade students

Survey conducted every four years: 1995, 1999, 2003, and 2007

Participation of more than 60 countries, diverse in terms of geographical location and economic development

IAE: PIRLS

Assessment of students’ reading literacy at the end of primary school, in some countries students are assessed in fifth and sixth grade

PIRLS is conducted every five years : 2001, 2006 and 2011

55 countries from all around the world have participated in PIRLS 2011

PIRLS 2011 has been extended to include prePIRLS —less difficult and designed to test basic reading skills that are a prerequisite for PIRLS

  Reading Mathematics Science

Mean score Mean score Mean scoreMean S.E. Mean S.E. Mean S.E.

OECD          

Estonia 501 (2.6) 512 (2.6) 528 (2.7)Poland 500 (2.6) 495 (2.8) 508 (2.4)Hungary 494 (3.2) 490 (3.5) 503 (3.1)OECD average 493 (0.5) 496 (0.5) 501 (0.5)Latvia 484 (3.0) 482 (3.1) 494 (3.1)Slovenia 483 (1.0) 501 (1.2) 512 (1.1)Czech Republic 478 (2.9) 493 (2.8) 500 (3.0)Slovak Republic 477 (2.5) 497 (3.1) 490 (3.0)Croatia 476 (2.9) 460 (3.1) 486 (2.8)Lithuania 468 (2.4) 477 (2.6) 491 (2.9)Russian Federation 459 (3.3) 468 (3.3) 478 (3.3)Serbia 442 (2.4) 442 (2.9) 443 (2.4)Bulgaria 429 (6.7) 428 (5.9) 439 (5.9)Romania 424 (4.1) 427 (3.4) 428 (3.4)Montenegro 408 (1.7) 403 (2.0) 401 (2.0)Albania 385 (4.0) 377 (4.0) 391 (3.9)

Background data

Students background questionnaire• socio-economic background, learning habits, attitudes towards reading, involvement and

motivation

School principals questionnaire• demographic school characteristics, school policies, quality of learning environment

Parents questionnaire• PISA for selected countries (optional), PIRLS• home learning environment, parental support, school choice

Teachers Questionnaire• For TIMMS and PIRLS only• Professional background, teaching methods, training and professional development

Curriculum questionnaires for National Research Coordinators• TIMMS and PIRLS only

PIRLS/TIMMS 2011 Encyclopedia• provides a profile of each country’s education system

Evidence-based, effective

education policies

Analysis in the international

context

International indicators and benchmarks

Analysis in the national and

regional context

International surveys deliver…

10

Integrity and PISA

• Starting point to tackle corruptive practices is to evaluate the integrity of an educational system

• We are assuming that systems with high level of integrity are in general those systems that perform well – well in terms of student achievement and providing equal opportunities.

Integrity

• In PISA, successful school systems are defined as those that perform above the OECD average in reading and in which students’ socio-economic background has a smaller impact on reading performance than it is the case in a typical OECD country.

• The term “successful school system” refers o two distinct areas: achievement and equity

Successful school system

• Allows identifying school characteristics that are shared by successful school systems

• PISA is used as a first point of reference in order to assign education policy areas of integrity

PISA

School characteristics related to performance and/or equity

Low levels of differentiation

Grade repetition

Student transference

Ability grouping

Low differentiation of the school system

• Unequal access to schools and classrooms creates demand situations for scarce places which may lead to corruption, in particular when perceived quality of schools differs greatly

Differentiation : Access

School characteristics related to performance and/or equity

Low levels of differentiation

Grade repetition

Student transference

Ability grouping

Accountability

The existence of standards-based

external examinations

And how they are used

Monitoring of lessons by inspectors

Accountability

• Educational achievement is made observable and transparent• Facilitate the monitoring of performance of students, teachers,

and schools• Can be considered as best practice to avoid opportunistic

behaviour by teachers and principals• Teachers cannot “get away” by skipping content areas or

employ very easy tests to have a positive teacher performance evaluation

• Since teachers do not know exam questions and areas covered in the test beforehand, incentives for corruption are nonexistent, e.g. selling exam questions in advance

Standards-based external examinations

• Improve the signal effect of school leave certificates and final examinations marks which are of particularly importance for higher educational institutions or future employers and may reduce corruption, e.g. fraud at university admission or favouritism, political affiliation in employment

External final exams

School characteristics related to performance and/or equity

Low levels of differentiation

Grade repetition

Student transference

Ability grouping

Accountability

The existence of standards-based

external examinations

And how they are used

Monitoring of lessons by inspectors

School autonomy

Autonomy to make decisions on curricula and assessments

School autonomy

• School autonomy has a positive effect on student achievement, but only when appropriate accountability practices are in place

• Schools with more autonomy may have more incentives to opportunistic behaviour, in particular when there is a high degree of information asymmetry about school behaviour, e.g. parents, governing boards or central planning authorities have insufficient information on what is happing in the school

• “Without external exams, the advantage of superior local knowledge must weighed against the disadvantage of opportunistic behaviour, and the net effect of school autonomy depends on the relative size of these two partial effects.” Wößmann et al. (2008: 34)

School autonomy

School characteristics related to performance and/or equity

Low levels of differentiation

Grade repetition

Student transference

Ability grouping

Accountability

The existence of standards-based

external examinations

And how they are used

Monitoring of lessons by inspectors

School autonomy

Autonomy to make decisions on curricula and

assessments

Spending on education

Teacher salaries

Minimum resource requirements

Spending on education

• Low teacher salaries are considered as one reason of corruption• Reflect the social status of the teaching profession in a society• Suggests commitment to the rules and guidelines of teaching in a country

and satisfaction of profession• Reduced incentives for corrupt and unethical behaviour, e.g. private tutoring

or absenteeism

High teacher salaries

• The eventuality of corruptive behaviour is greater in schools that lack material, e.g. teachers might feel compelled to ask for money to provide students with learning materials, such as textbooks and laboratory equipment or principals may illegally rent out school facilities

• Great inequity between schools in terms of resources may lead to high demand for scarce places in “better” schools. This demand may lead to corruption occurrence

Availability of sufficient human and material resources

School characteristics related to performance and/or equity

Low levels of differentiation

Grade repetition

Student transference

Ability grouping

Accountability

The existence of standards-based

external examinations

And how they are used

Monitoring of lessons by inspectors

School autonomy

Autonomy to make

decisions on curricula and assessments

Spending on education

Teacher salaries

Minimum resource

requirements

Learning environment

Disciplinary climate

Student and teacher

behaviour

Teacher-student relations

Learning environment

• High quality of the learning environment, approximated by indicators such as good teacher-student relations, positive student and teacher behaviour and good disciplinary climate, are important to ensure a high level of moral and commitment in schools and classrooms.

• This can be considered as crucial elements of an intact school system that prevent various corruptive or unethical practices.

Quality of learning environment

Next steps

• Integration of corruption-related items in the national PISA students’, parents’ and/or school questionnaire, e.g. questions related to private tutoring, gifts to teachers, cheating

PISA and corruption items

• Raise awareness of the problem of corruption in the education sector on an international level

• Open up new opportunities to research corruption empirically

• Make corruption in education empirically measurable and allow to relate corruption to educational outcomes

Thank you for your attention!