the timss 2007 study in qatar: a summary of key findings ... · the timss 2007 study in qatar: a...

The TIMSS 2007 Study in Qatar:A Summary of Key Findings and

Options for Policy and Further Study

The TIMSS 2007 Study in Qatar:A Summary of Key Findings and Options for Policy and

Further Study

This report was prepared by Doug Willms under contract with DataAngel

Policy Research Inc. The author is grateful to Adel Al Sayed, Director,

Evaluation Institute, Supreme Education Council, Abdulsattar Nagi, Project

Manager for TIMSS, and Scott Murray, President and CEO of DataAngel

Policy Research Inc. for their roles in conceptualizing this report and for

editorial comments. He is also grateful to Hasnain Mirza for report design

and layout.

© Supreme Educational Council

December, 2008

1

Purpose of the StudyThis document summarises the contents of

the report, “Skills for the Knowledge Economy:

Results from TIMSS 2007 for Qatar,” published

by the Evaluation Institute of the Supreme

Education Council, in December 2008.

The TIMSS 2007 study provides baseline

measures of the mathematics and science skill

and knowledge of Grade 4 and Grade 8 students

in Qatar. These measures can be compared to

the standards achieved by students in a large

cross-section of countries of the world. The

competencies measured offer a foundation

for effectively receiving, processing and

understanding complex information, and hence

constitute the essential basis for achieving

success in the emerging global knowledge

economy. Qatar’s participation in the TIMSS

2007 study represents an important step in

benchmarking the comprehensive educational

change process the country is pursuing - a part

of which corresponds to the “Education for a

New Era” initiative, launched by Emiri decree of

November 2002.

What is the TIMSS Study?The Trends in Mathematics and Science Study

(TIMSS) was initiated by the International

Association for the Evaluation of Educational

Achievement (IEA) to provide policy-

oriented international indicators of the skills

and knowledge of students in participating

countries. It assesses student proficiency in two

domains – mathematics and science – at Grades

4 and 8. The assessment focuses on measuring

the degree to which students have mastered

key scientific and mathematics knowledge,

concepts and processes.

TIMSS builds upon a rich tradition of assessment

extending back to the First International Mathematics Study (FIMS, 1964), the First International Science Study (FISS, 1970/71), and the Second International Science Study (SISS, 1983/84). Additional TIMSS

cycles have been run in 1994/95, 2003/04 and

now 2007. This is the first TIMSS study in which

Qatar has participated.

2

Building on previous IEA studies, TIMSS 2007

uses the curriculum as the major organizing

concept in considering how educational

opportunities are afforded to students and

how these opportunities are translated into

achievement.

TIMSS is focused on three dimensions of the

curriculum in mathematics and science:

• The intended curriculum - defined as the

curriculum as specified at the national or

system level.

• The implemented curriculum – defined as the

curriculum as interpreted and delivered by

classroom teachers.

• The attained curriculum – what parts of the

curriculum are learned by students, as

demonstrated by their attitudes and

achievements.

Each of these dimensions allows one to address

a different set of questions.

For example the focus on the intended

curriculum allows one to answer the question:

What are mathematics and science students expected to learn?

In contrast, the focus on the implemented

curriculum addresses questions such as:

What opportunities do students have to learn mathematics and science?

Most importantly, TIMSS provides answers to

questions such as:

What mathematics and science concepts, processes and attitudes have students acquired?

What factors underlie differences in student achievement in mathematics and science?

70 countries1 participated in TIMSS 2007

including several countries from the Arab

region. In Qatar, 7,013 Grade 4 and 7,184 Grade

8 students from 181 schools participated.

The TIMSS 2007 study included a direct

assessment of students’ skills, a student

background questionnaire, a teacher

questionnaire and a school questionnaire

completed by principals. The student, teacher

and school questionnaires were used to collect

background and contextual information related

to student performance.

The Qatar TIMSS Study in ContextEnsuring that all children and youth in Qatar

are equipped with the knowledge and skills

they need to fully participate in society,

and the increasingly global and competitive

world of work, are among the key objectives

pursued by the Supreme Education Council,

and shared widely by all stakeholders in the

community. Qatar’s “Education for a New Era”

initiative, being implemented by the Council

at the elementary, preparatory and secondary

formal education system levels, ranks among

the world’s most encompassing and ambitious

education reform programmes.

1 The TIMSS study allows sub-national regions, referred to as benchmarking

participants, to participate but excludes these students from the international averages and from the international reports. For example 4 Canadian provinces chose to participate in TIMSS 2007.

3

To Qatar’s credit, most of the systemic elements

have already been put in place to precipitate

a rapid and sustained improvement in student

performance over the coming decade. Essential

educational inputs such as adequately equipped

school buildings, modern and demanding

standards, curricula and textbooks, teacher

qualifications, a state of the art standardised

assessment system, and the resources invested

in the education system, are generally at a par,

or even better standing than the averages,

typically observed in the most advanced

economies of the world, the Member States of

the Organisation for Economic Co-operation

and Development (OECD).

Nevertheless, achieving the objective of raising

the levels of knowledge and skills of children

and youth in Qatar to at least the OECD

average might well take a generation. But Qatar

does possess the three assets that experience

of school improvement, in particular in East

Asia, has shown may allow it to accelerate the

reforms and capture the benefits in terms of

improved performance: adequate resources,

an ambitious vision, as well as sheer political

determination to succeed.

Findings from the OECD 2006 PISA assessment

for Qatar confirmed that the performance of

Qatar’s 15 year olds is currently well below

world-class levels, a finding that provides

ample justification for the urgency with which

decision makers in the country are pursuing the

“Education for a New Era” initiative.

It is an open question, however, whether the

reforms begun in 2004 have begun to yield

the expected improvements in performance.

The TIMSS 2007 study provides data for two

cohorts of students who have been exposed to

the first four years of the reform, enough of an

exposure to expect to begin to see higher test

scores.

Lead Hypotheses InvestigatedThe “Education for a New Era” initiative

has afforded Qatar’s students a number

of advantages, including increasingly well

qualified teachers, high levels of expenditure

per student, demanding and updated standards

and curricula, a state of the art standardised

assessment system, world class levels of support

for science instruction, and in turn these factors

have triggered high levels of student interest in

science. Unfortunately, however, the evidence

suggests that the reform have not had yet the

positive intended impact upon the performance

of the cohorts of Grade 4 and Grade 8 students

tested in TIMSS 2007.

It is not clear why this is the case. One possibility,

explored in this volume, is that teachers have

been slow to implement key elements of the

reform, most importantly their approach to

teaching. This finding is not entirely unexpected.

Experience from other countries suggests that

it takes at least a decade for education reforms

of the scale being implemented in Qatar to

yield measurable results.

4

Overview of Key FindingsThe TIMSS report for Qatar is designed to

provide readers with answers to an ordered set

of issues, briefly summarised below.

The introduction sets out why the knowledge

and skills assessed by TIMSS are economically

and socially important, and how TIMSS went

about measuring proficiency levels across a

heterogeneous group of 70 countries.

Figures 1, 2, 3 and 4 highlight the performance

of Grade 4 and Grade 8 students in Qatar

in mathematics and scientific literacy in an

international context.

Hong Kong SARSingapore

Chinese TaipeiJapan

Kazakhstan Russian Federation

England Latvia Netherlands Lithuania United StatesGermany DenmarkAustralia Hungary Italy Austria Sweden Slovenia Armenia Slovak Republic

ScotlandNew Zealand Czech Republic

Norway Ukraine

Georgia Iran, Islamic Rep. of

Algeria Colombia

Morocco El Salvador Tunisia

Kuwait Qatar

Yemen

Level 1 Level 2 Level 3 Level 4 Level 5

200 300 400 500 600 700

Figure 1: Estimated average mathematics scores of Grade 4 students and 95 per cent confidence intervals, by country, TIMSS 2007

Mean score and 95% confidence interval

Average Mathematics Scores

5

Chinese TaiperKorea, Rep ofSingapore

Hong Kong SARJapan

HungaryEnglandRussian Federation

United StatesLithuaniaCzech RepublicSloveniaArmeniaAustralia

SwedenMaltaScotlandSerbia

ItalyMalaysia

NorwayCyprusBulgariaIsraelUkraineRomania

Bosnia and HerzegovinaLebanonThailand

TurkeyJordan

TunisiaGeorgia

Iran, Islamic Rep. ofBahrainIndonesiaSyrian Arab RepublicEgypt

AlgeriaMoroccoColombia

OmanPalestinian Nat’l Auth.Botswana

KuwaitEl Salvador

Saudi ArabiaGhanaQatar


200 300 400 500 600 700

Figure 2: Estimated average mathematics scores of Grade 8 students and 95 per cent confidence intervals, by country, TIMSS 2007


Average Mathematics Scores

6

SingaporeChinese TaipeiHong Kong SAR

JapanRussian Federation

LatviaEngland United StatesHungaryItalyKazakhstan

GermanyAustraliaSlovak RepublicAustriaSwedenNetherlandsSloveniaDenmarkCzech RepublicLithuania

New ZealandScotland

ArmeniaNorwayUkraine

Iran, Islamic Rep. of Georgia

ColombiaEl Salvador

AlgeriaKuwait

TunisiaMorocco

QatarYemen


200 300 400 500 600

Figure 3: Estimated average science scores of Grade 4 students and 95 per cent confidence intervals, by country, TIMSS 2007


Average Science Scores

7

SingaporeChinese Taipei

JapanKorea, Rep. ofEnglandHungaryCzech RepublicSloveniaHong Kong SARRussian Federation

United StatesLithuaniaAustraliaSweden

ScotlandItalyArmenia

NorwayUkraineJordan

MalaysiaThailandSerbiaBulgariaIsraelBahrainBosnia and HerzegovinaRomaniaIran, Islamic Rep. ofMaltaTurkeySyrian Arab RepublicCyprus

TunisiaIndonesiaOmanGeorgia

KuwaitColombiaLebanon

EgyptAlgeriaPalestinian Nat’l Auth.MoroccoSaudi Arabia

El SalvadorBotswana

QatarGhana


200 300 400 500 600 700

Figure 4: Estimated average science scores of Grade 8 students and 95 per cent confidence intervals, by country, TIMSS 2007


Average Science Scores

8

The figures reveal sobering finding - that is

average proficiencies in mathematics and

science fall in the lowest ranks of participating

countries, even below their peers in the Gulf.

The majority of Qatari students are classified at

proficiency Level 1, and only small percentages

of students have skills that are sufficiently

advanced, measured against TIMSS benchmarks,

to place them at Levels 3 or above in either of

the two skill domains.

The data analyses presented in the report

indicate that the low average proficiencies

observed in Qatar are coincident with the fact

that the entire distributions of the proficiency

scores are shifted down into the lower levels

of the corresponding scales, and that relatively

higher proportions of Qatari students have

scores below the national average. These results

are observed for both domains at both grades,

and suggest that a high proportion of students

are not yet acquiring the foundation upon which

learning in the later grades depends.

The data analyses presented in the report also

document variation in performance among

sub-groups of students in Qatar. In contrast to

what one observes in other countries with low

performance in the TIMSS 2007 assessment,

there is little variation in students’ science or

mathematics proficiency by level of parental

education, as a proxy of socio-cultural status.

This may be due to the narrow spread in socio-

cultural status in the country.

Figure 5 shows interesting variation among

students attending different types of schools in

Qatar.

Level 1

Level 2

Below

Level 1

8 9 10 11 12 13 14 15 16 17 18

150

200

250

300

350

400

450

Figure 5: Average Mathematics Proficiency of Schools by School Type, Qatar 2007

Ministry of Education

Independent

Private Arabic

9

This finding suggests that rapid improvement in

student performance is within reach – provided

that the means can be found for less well

performing teachers and schools to learn from

their better performing peers.

The report explored the complex interplay

between science and mathematics proficiency

and a set of factors that theory and previous

evidence suggest exert a positive influence upon

observed performance. The analysis of these factors, specified in the QATAR model of school success – Quality of instruction, Appropriate level of instruction, Time on task, Attitudes to learning, and Resources invested – confirms that all of the key elements needed to support the reform are in place. The analysis examined 15 separate

classroom and school factors. Five of these had

the strongest effects on mathematics science

performance, internationally and in Qatar. Figure

6 provides a comparison of Qatar’s results

(green line) with the international average (red

bar) for each factor. Qatar compares favourably

for all five factors, except ‘opportunity to learn’.

This suggests that Qatar has a slower pace of

content coverage during the primary school

years. The scores for each of classes that

participated in the study are shown with small

blue dots on the figure. It shows that there is

wide variation among classes in ‘opportunity to

learn’ and ‘school resources.

10

0 1 2 3 4 5 6 7 8 9 10

0 1 2 3 4 5 6 7 8 9 10

Teacher

Preparation

Opportunity

to Learn

Attitude to

Mathematics

School

Climate

School

Resources

International Mean: 7.50

Qatar Mean: 8.11


Qatar Mean: 5.84


Qatar Mean: 7.91


Qatar Mean: 6.43


Qatar Mean: 6.37

Figure 6: Learning resource plot for Qatar

11

The data analysis also confirmed that certain Qatari schools are already imparting significantly higher levels of knowledge and skills than other schools. These findings can help frame and focus the in-service training of teachers in a powerful way, one in which the teachers themselves are empowered to drive reform.

This first TIMSS report for Qatar is intended

to establish a baseline for future comparisons

in coming cycles. The evidence presented

above offers a snapshot of what the study has

discovered about the levels and distributions

of mathematics and science proficiency of

students at two key points during the initial

cycle of education. The data analyses have also

documented a wealth of other important, more

specific findings.

Implications for Policy and Future StudiesThe findings carry several important implications

for current policy – not only related to education

but also to policies impacting family welfare,

youth affairs, social work, culture, immigration,

and, importantly, economic policies aimed at

creating a Qatari environment amenable to

sustainable development.

First and foremost, they provide unequivocal

support for Qatar’s “Education for a New Era”

reform initiative, and the concomitant new

investments aimed at raising the quality of

education.

Second, the strikingly low levels of science and mathematics knowledge and skills of primary students in Qatar, measured against international benchmarks, suggest that the education reforms instituted over the past several years have yet to precipitate the expected improvements in performance. Somewhat disconcertingly, this fact suggests that the factors that are inhibiting the performance of the Qatari education system are less tractable than expected. This finding is not

entirely unexpected. Evidence from other jurisdictions suggests it takes a minimum of 15 years for reforms of the scope being undertaken in Qatar to begin to bear fruit.

The on-going effort to afford high-quality early childhood education to all Qatari families should evidently be vigorously pursued, and measures now being implemented that are designed to improve learning at the early stages of life deserve centre stage.

Third, although the TIMSS 2007 findings will no doubt be interpreted by some educators and other members of Qatari society as discomforting, they should nevertheless be welcomed, not least because they offer objective confirmation that the “Education for a New Era” initiative is essential to the future well being of the nation, and was adopted in anticipation of these results. They also provide strong support for continued technically sound standardised and comparative assessment of education, and related longitudinal inferential research efforts. The participation of Qatar in the 2007 Progress In Reading Literacy Study (PIRLS) and the OECD’s PISA assessment represents an important, complementary step in building the knowledge base needed to drive reform.

Finally, the results presented offer reason for optimism. Some Qatari schools and teachers manage to impart much higher levels of skill than their peers. If ways can be found to transfer these successes then the way is clear for the rapid improvement in the mathematics and science scores of future cohorts of students. This finding alone justifies the current education reform and the need to monitor performance over time, both through the Qatar Comprehensive Education Assessment system (QCEA), as well as through further participation in international studies such as TIMSS, PIRLS and PISA.

In conclusion, the study offers compelling evidence in support of the “Education for a New Era” initiative. As noted in the PISA 2006 report subjecting oneself to the harsh light of international comparison so early in the reform process took a level of political courage and openness that, it should be

the timss 2007 study in qatar: a summary of key findings ... · the timss 2007 study in qatar: a...

Documents