BOOK REVIEW

PISA under examination: Changing knowledge,changing tests, and changing schools

By Miguel A. Pereyra, Hans-Georg Kotthoff and Robert Cowen (eds).Sense Publishers, Rotterdam, 2011, 334 pp. ISBN 978-94-6091-738-7 (pbk),ISBN 978-94-6091-739-4 (hbk), ISBN 978-94-6091-740-0 (e-book)

Michael McVey

Published online: 26 February 2013

� Springer Science+Business Media Dordrecht 2013

PISA is the acronym for the Programme for International Student Assessment of the

OECD (Organisation for Economic Cooperation and Development). The pro-

gramme involves thousands of students from 62 different countries (32 OECD

countries and 30 partner countries). While PISA has been the event that brought

some national education programmes from obscurity to international stardom, it has

also been the cause for national handwringing.

The assessment aims at 15-year-olds, presumably nearing the end of their formal

education. The two-hour test on reading, mathematics and science is administered

every year and rotated among those three subject areas; thus the reading test would

be administered every three years, then the maths test in the following year, then the

science test in the year following that. Where PISA differs from many tests of

achievement is that it aims to test young people’s ability to use their knowledge and

skills in order to meet real-life challenges. For example, the assessment ascertains

whether students can apply their reading skills to make sense of information they

find in books, newspapers, forms or instructional manuals. The results extrapolate to

the national level so the assessment can show nations what is possible and then use

that information to refine national educational policies.

In this thoughtfully organised collection of essays, the development and goals of

the PISA instrument contrast with the impact on both politics and policy.

Throughout the essays, there is some deep reflection on the difficulty inherent in

trying to analyse the complicated interconnectedness of national policies related to

education, economic and social welfare. The policies and trends span many years,

political shifts and societal revolutions, making the context of such studies very

difficult to narrow.

Ulf Lundgren’s essay sets the development of the PISA programme into the

context of educational testing. His contribution is additionally significant to this

M. McVey (&)

Eastern Michigan University, Ypsilanti, MI, USA

e-mail: mmcvey@emich.edu

123

Int Rev Educ (2013) 59:141–144

DOI 10.1007/s11159-013-9338-4

collection as he was professionally involved in the development of the assessment.

After years of experimenting with educational testing, by the early 1960s, it was

possible to use similar tests to make comparative evaluations, and by the 1980s,

politicians were comfortable with basing educational goals on measured outcomes.

The OECD introduced the PISA programme against such a background. Lundgren

reminds readers of the hopes for the programme and hints at surprising results. In

some respects, as nations focus on their position in relation to other countries on a

global scale, the PISA programme seems to have naturally evolved to serve the

needs of those nations beginning to look beyond their own borders to find new ways

to educate their children.

Thomas Popkewitz’s essay reminds us that the promise of PISA may be daunting

but there is a great deal of potential for the programme. PISA, he notes, creates

categories of equivalence across language and culture. PISA itself also has the

potential for study as an historical event. The programme creates numbers that states

may use in various ways to draw conclusions about the effectiveness of their

educational policies, their economic decisions, and the relative wealth of nations

with which they interact. Popkewitz reminds the reader that nations may use

numbers as actors to standardise a universe of capabilities in order to enable

comparisons. Once capabilities become numbers or ‘‘facts’’, they may be

standardised and, eventually, those collections of facts become simplified to

something akin to football league tables for ready consumption by a public keen to

feel good about the state of their nation’s schools in the world. The same numbers

may also become a lever to effect political change.

Clara Morgan takes this idea of political change to remind us that politics is never

far from policy. As nations come under criticism for poor performance, the policies

of these state-run institutions change. Regardless of the solution proposed, from

chartering schools to reducing school size to nationalising curriculum, there is

always a political component to proposed school reforms.

Antonio Bolıvar, in his essay, examines the reaction of nations that did not fare

well in the first few rounds of PISA and the public discourse that followed. When

news media treat the results of PISA as they would a horse race with winners and

losers, such an attitude can lead to a national dissatisfaction about the current state

of schools in the home country, or the attitude can be the very thing to spur

improvement. Bolıvar’s essay does an excellent job of outlining the complexity of

comparing national education approaches.

David Berliner moves along the discussion about politics and complexity with a

perceptive piece on the fear the results can generate. He cites a general fear of

weakened global competitiveness, fear of failing schools, fear of an erosion of

national standing, and more. He goes a significant distance towards shedding light

on the reality of the complex numbers that can emerge from the PISA results. In his

analysis, he compares the results of Finland, one of the higher-scoring countries,

with those of the United States and notes that the PISA percentage of scientifically

talented youth in Finland, while higher than many, results in 1.4 million such youth

in Finland while that number in the United States is 7.4 million. Such a number,

Berliner notes, is far from representing a shortage of scientific talent suggested in

news reports following announcements of the PISA results. Berliner also uses this

142 M. McVey

123

comparison to point out other underlying factors that affect results such as

childhood poverty rates that are 3 per cent in Finland as opposed to 25 per cent in

the US. His analyses of raw scores and quintiles demonstrate how national news

media can easily overlook the more important trends and messages of PISA.

David Scott reiterates Berliner’s conclusions and provides a theoretical overview

of test constructs, the washback effect, and examination technology that can

challenge the very nature of the assessments, and exposes their weaknesses. Scott

reminds the reader that PISA took on a challenging task when it sought to create a

curriculum-free test.

Where David Scott touches briefly on cultural issues related to PISA, Donatella

Palomba and Anselmo R. Paolone focus on interculturality as an issue that shows

how PISA affects the globalisation of education. By examining the shift in national

attitudes towards Italian students engaging on exchanges abroad, the authors see that

where Italian educators once viewed exchanges with distrust, PISA has shifted the

perception of them to their being regarded as a new way of expanding international

dialogue.

Katharina Maag Merki’s detailed examination of low-stakes German exit

examinations with the United States approach of creating high-stakes examinations

showed that results seemed to favour low-stakes approaches where teachers could

tailor efforts to the students’ needs. Of course, Merki draws considerable caveats

when making her conclusions in her data-heavy essay.

The test takers themselves, largely ignored in the first half of this collection, were

the focus of Gerry Mac Ruairc’s essay. By examining a group of test takers and

transcribing their comments after taking the tests, he discovered that many of them

felt the test was too long, the content not engaging, the personal questions too

probing, and the whole process too rushed.

Marie Duru-Bellat suggests that analysts can use the results of the PISA to

examine education as a public good and, in her own analysis, she notes that there

may also be some significant downsides and limitations to using PISA for the

benchmarking of national educational progress.

When analysts consider a programme such as PISA only as an evaluation tool of

educational progress, the result is short-sighted. Javier Salinas Jimenez and Daniel

Santın Gonzalez point out that shifts in perception about a nation’s education can

have positive effects on the very curricula employed. They examine the changes to

mathematics instruction and use PISA to draw conclusions about changes in

educational efficiency.

Aileen Edele and Petra Stanat take on questions of the reality of any programme

that examines education in a longitudinal way. Not all populations are stable and the

authors consider the influence of immigrants across national borders. Immigrants

bring into play the issue of the language of school, the effects of economic stability,

and other socio-economic factors on the educational progress of immigrant students.

Edele and Stanat take pains to demonstrate the limitations of using PISA results for

such analysis. Julio Carabana’s essay also reflects on the immigrant student and asks

profound questions about the importance of the immigrant’s originating country as a

factor in analysis.

Book Review 143

123

Hannu Simola and Risto Rinne return to the Finnish school system and provide

some much-needed insight and historical background of the expansion of schooling

in Finland and other Nordic nations that may put their PISA results into context.

In the final essay, Daniel Trohler explores the impact of PISA scores at the level

of the national psyche regarding how it perceived not only its schools but also its

fundamental self-understanding. He examines the German reaction to PISA, which

he suggests was the fiercest of all the nations: a debate arose about the importance of

a student’s competence as opposed to general knowledge and the non-empirical

concept of Bildung. PISA’s goal of measuring a student’s abilities to use school

learning to meet real-life challenges seems to rankle at the heart of the goal of

schooling embraced by Germans. As Trohler sees it, and he carefully argues his

case, PISA places the goal of education on the outer world and not the inner world.

Comparing national education systems is difficult. Assessing the effects of ten

years of schooling on 15-year-olds is even more complex. Analysing the results of

the analysis and then creating a plan for subsequent action is fraught with

challenges. Add to all those inherent difficulties a public digestion of the

disseminated results through a medium that steadfastly refuses to scratch the

surface of the numbers, and you have a programme that merits the scrutiny these

essays suggest it requires.

144 M. McVey

123