measuring up: the performance of canada’s youth in reading, mathematics and science · 2004. 12....

Measuring up:The performance of Canada’s youth inreading, mathematics and science

OECD PISA Study—First Results for Canadians aged 15

Council of Ministers of Education, CanadaConseil des ministres de l’Éducation (Canada)

Statistics StatistiqueCanada Canada

Human Resources Développement des ressourcesDevelopment Canada humaines Canada

How to obtain more information

Specific inquiries about this product and related statistics or services should be directedto: Client Services, Culture, Tourism and the Centre for Education Statistics, StatisticsCanada, Ottawa, Ontario, K1A 0T6 (telephone: (613) 951-7608; toll free at 1 800 307-3382;by fax at (613) 951-9040; or e-mail: [email protected]).

For information on the wide range of data available from Statistics Canada, you can contactus by calling one of our toll-free numbers. You can also contact us by e-mail or by visitingour Web site.

National inquiries line 1 800 263-1136National telecommunications device for the hearing impaired 1 800 363-7629Depository Services Program inquiries 1 800 700-1033Fax line for Depository Services Program 1 800 889-9734E-mail inquiries [email protected] site www.statcan.ca

Ordering and subscription information

This product, Catalogue no. 81-590-XPE, is published irregularly as a standard printedpublication at a price of CDN $10.00 per issue. The following additional shipping chargesapply for delivery outside Canada:

Single issue

United States CDN $6.00

Other countries CDN $10.00

All prices exclude sales taxes.

This publication is available electronically without charge through the internet at:

• www.pisa.gc.ca• www.statcan.ca• www.cmec.ca• www.hrdc-drhc.gc.ca/arb

This product can be ordered by

• Phone (Canada and the United States) 1 800 267-6677• Fax (Canada and the United States) 1 877 287-4369• E-mail [email protected]• Mail Statistics Canada

Dissemination DivisionCirculation Management120 Parkdale AvenueOttawa, Ontario K1A 0T6

• And in person at the Statistics Canada Reference Centre nearest you.

The paper used in this publication meets the minimum requirements of American National Standardfor Information Sciences - Permanence of Paper for Printed Library Materials, ANSI Z39.48 - 1984.

∞

Measuring up:The Performance of Canada’s Youthin Reading, Mathematics and ScienceOECD PISA Study—First Results for Canadians aged 15

Patrick Bussière, Human Resources Development CanadaFernando Cartwright, Statistics CanadaRobert Crocker, Memorial University of NewfoundlandXin Ma, University of AlbertaJillian Oderkirk, Statistics CanadaYanhong Zhang, Human Resources Development Canada

Published by authority of the Minister responsible for Statistics Canada

© Minister of Industry, 2001

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means,electronic, mechanical, photocopying, recording or otherwise, without prior written permission from License Services, Marketing Division,Statistics Canada, Ottawa, Ontario, Canada K1A 0T6.

December 2001

Catalogue no. 81-590-XPE

ISBN 0-660-18668-3

Catalogue no. 81-590-XIE

ISBN 0-662-31284-8

Frequency: Irregular

Ottawa

Human Resources Development Canada, Council of Ministers of Education, Canada and Statistics Canada

The data interpretations presented in this report are those of the authors and do not necessarily reflect those of the granting agencies or reviewers.

National Library of Canada Canadian Cataloguing in Publication Data

Measuring up : the performance of Canada’s youth in reading,mathematics and science : OECD PISA study : first result forCanadians aged 15

Issued also in French under title: À la hauteur : la performance des jeunes duCanada en lecture, en mathématiques et en sciences : études PISA del’OCDE : premiers résultats pour les canadiens de 15 ans.ISBN 0-660-18668-3 (paper)ISBN 0-662-31284-8 (Internet)CS81-590-XPECS81-590-XIE

1. High school students – Rating of – Canada.2. Academic achievement – Canada – Statistics.3. Educational evaluation – Canada – Statistics.I. Bussière, Patrick. II. Statistics Canada. II. Canada. Human ResourcesDevelopment Canada. III. Council of Ministers of Education (Canada).

LB3054.C3 M42 2001 373.26’2’0971C2001-988012-X

3

Measuring up: The Performance of Canada’s Youth in Reading, Mathematics and Science

Acknowledgements

We would like to thank the students, parents, teachersand principals who gave of their time to participate in the2000 OECD PISA study and the Youth in TransitionSurvey. The support for this Federal-Provincialcollaborative project provided by members of the PISA-YITS Steering Committee and by the coordinators in eachparticipating Ministry or Department of Education duringall steps of the study is gratefully acknowledged. Thededication of the survey development, implementation,processing and methodology teams was essential to theproject’s success and is appreciated.

This publication was prepared jointly by StatisticsCanada, Human Resources Development Canada andthe Council of Ministers of Education, Canada and wassupported financially by Human Resources DevelopmentCanada. The report has benefited from the input andcomments of reviewers in provincial Ministries andDepartments of Education, Human ResourcesDevelopment Canada and Statistics Canada. Thecontribution of members of the joint Working Group onPISA-YITS Dissemination, Satya Brink, PatrickBussière, Patrice de Broucker, Jeanine Bustros, Louis-Philippe Gaudreault, Dean Goodman, DouglasHodgkinson, Marc Lachance, Michael Lerner, Scott

Murray, Jillian Oderkirk, Monica Paabo and DiannePennock, is appreciated. The authors, Patrick Bussière,Fernando Cartwright, Robert Crocker, Xin Ma, JillianOderkirk and Yanhong Zhang, are thanked for their valuedcontribution to the dissemination of these first Canadianresults. The staff of the Centre for Education Statisticsat Statistics Canada, whose tireless efforts ensured a highstandard of quality, are appreciated, particularly MaryAllen, Greg Anderson, Rosemarie Andrews, MarcLachance and Sylvie Ouellette. A very special thank youis extended to Danielle Baum for her indispensable helpin preparing the manuscript for publication. Thecontribution of editorial, communications, translation anddissemination services staff of Statistics Canada, HumanResources Development Canada and the Council ofMinisters of Education, Canada was essential to theproject’s success and is appreciated.

Note of Appreciation

Canada owes the success of its statistical system to thelong-standing co-operation of Statistics Canada, thecitizens of Canada, its businesses, governments and otherinstitutions. Accurate and timely statistical informationcould not be produced without their continued co-operation and goodwill.

5


Acknowledgements 3

Introduction 9The Canadian Context 9What is PISA? 10PISA 2000 10What is YITS? 11Why are YITS and PISA integrated in Canada? 11Objectives and organization of the report 11Box 1: Overview of PISA 2000 12Notes 12

Chapter 1

The Achievement of Canadian Students within an International Context 13The performance of Canadian students in a global context 13Countries performing better than or about the same as Canada 13Provincial results in an international context 16The distribution of scores 18Variation in proficiency 19Distribution of high performers 21Reading skill levels 21How do boys and girls compare? 24Achievement of Canadian students by language of the school system 25Conclusion 25Notes 26

Chapter 2

The Impact of Individual Characteristics on Achievement 27Reading behaviours 28Attitudes toward school 28Student career expectations 28Student education expectations 28Working while studying 29The relative importance of individual factors in explaining achievement 29Conclusion 30Notes 30

Chapter 3

The Impact of Family Characteristics and Home Environment on Achievement 31Family background 31Home environment 33Family educational support 33Parental involvement 34Parental expectations 34The relative importance of family factors in explaining achievement 35Conclusion 36Notes 36

Table of Contents

6

Table of Contents

Chapter 4

The Impact of School Characteristics on Achievement 37School characteristics and student academic performance 37Public and private schools 38Composition of school population 38School climate 39Teacher-student interactions 40School resources 41The relative importance of school factors in explaining reading performance 42Conclusion 43Note 43

Conclusion 45

Annex A: Tables

Chapter 1Table 1.1 Averages and Confidence Intervals: Reading 50Table 1.2 Averages and Confidence Intervals: Mathematics 50Table 1.3 Averages and Confidence Intervals: Science 51Table 1.4 Averages and Confidence Intervals: Reading Retrieving 51Table 1.5 Averages and Confidence Intervals: Reading Interpreting 52Table 1.6 Averages and Confidence Intervals: Reading Reflecting 52Table 1.7 Reading Scores at 5th, 10th, 25th, 50th, 75th, 90th and 95th Percentiles 53Table 1.8 Mathematics Scores at 5th, 10th, 25th, 50th, 75th, 90th and 95th Percentiles 54Table 1.9 Science Scores at 5th, 10th, 25th, 50th, 75th, 90th and 95th Percentiles 54Table 1.10 Reading Inequality Index (90th percentile/10th percentile) 55Table 1.11 Mathematics Inequality Index (90th percentile/10th percentile) 55Table 1.12 Science Inequality Index (90th percentile/10th percentile) 55Table 1.13 Percent of Students Above 50th, 75th and 90th International Percentile: Reading 56Table 1.14 Percent of Students Above 50th, 75th and 90th International Percentile: Mathematics 56Table 1.15 Percent of Students Above 50th, 75th and 90th International Percentile: Science 56Table 1.16 Reading Proficiency Scales: Percent of Students at Each Level 57Table 1.17 Average Reading Scores by Gender 58Table 1.18 Average Mathematics Scores by Gender 59Table 1.19 Average Science Scores by Gender 60Table 1.20 Average Scores by Province and Language of the School System 60

Chapter 2Table 2.1 Effects of Reading Enjoyment and Reading Diversity on Achievement

Scores in Reading 61Table 2.2 Average Achievement Scores in Reading by Time Spent Reading for Enjoyment 61Table 2.3 Average Achievement Scores in Reading, Mathematics and Science by

Use of Public and School Libraries 62Table 2.4 Effects of Homework Time and Sense of Belonging to School on Achievement

Scores in Reading, Mathematics and Science 64Table 2.5 Effects of Student Career Expectations on Achievement Scores in Reading,

Mathematics and Science 65Table 2.6 Average Achievement Scores in Reading, Mathematics and Science by Student

Expectations of the Highest Level of Education 65Table 2.7 Average Achievement Scores in Reading, Mathematics and Science by School

Year Job Status 66Table 2.8 Effects of Weekday and Weekend Work Hours During the School Year on

Achievement Scores in Reading, Mathematics and Science 67Table 2.9 The Relative Impact of Individual Factors on Achievement Scores in Reading,

Mathematics and Science 68

Table of Contents

7


Chapter 3Table 3.1 Average Achievement Scores in Reading, Mathematics and Science by

Family Structure 69Table 3.2 Effects of Number of Siblings on Achievement Scores in Reading, Mathematics

and Science 70Table 3.3 Effects of Family Socio-economic Status on Achievement Scores in Reading,

Mathematics and Science 70Table 3.4 Average Scores in Reading, Mathematics and Science, within National and

Provincial Quarters of Family Socio-economic Status 71Table 3.5 Effects of Family Possessions, Home Educational Resources, Home Cultural

Possessions and Student’s Cultural Activities on Achievement Scores inReading, Mathematics and Science 73

Table 3.6 Average Achievement Scores in Reading, Mathematics and Science by Number ofBooks at Home 74

Table 3.7 Effects of Family Educational Support on Achievement Scores in Reading,Mathematics and Science 75

Table 3.8 Effects of Parental Academic Interest and Parental Social Interest on AchievementScores in Reading, Mathematics and Science 76

Table 3.9 Average Achievement Scores (in Reading, Mathematics and Science) of Students byTheir Parents’ Education Expectations 77

Table 3.10 The Relative Impact of Family Factors on Achievement Scores in Reading,Mathematics and Science 78

Chapter 4Table 4.1 Public School Enrolment, School Average Socio-economic Status and Family

Possessions Effects on Individual Reading Achievement 79Table 4.2 Average Scores of School Climate Indicators and Effects on Individual Reading

Achievement 80Table 4.3 Average Scores of Teacher-student Interaction Indicators and Effects on Individual

Reading Achievement 81Table 4.4 Average Scores of School Resource Indicators and Effects on Individual Reading

Achievement 82Table 4.5 The Relative Impact of School Factors on Reading Achievement Scores 83

AnnexB: Definitions of Key Variables and Constructs 85

Table of Contents

9


Introduction

T he skills and knowledge that Canadians bring to their jobs, and to our society, play an important role in determining our economic success andour overall quality of life. Evidence is mounting that theimportance of skills and knowledge will continue to growin the future. The shift from manufacturing to knowledge-and information-intensive service industries, advances incommunication and production technologies, the widediffusion of information technologies, falling trade barriers,and the globalization of financial markets and marketsfor products and services, have precipitated changes inthe skills our economy requires. These include a risingdemand for a strong set of foundation skills upon whichfurther learning rests.

Elementary and secondary education systems playa central role in laying a solid base upon which subsequentknowledge and skills can be developed. Those studentsleaving secondary education without a strong foundationmay experience difficulty accessing the postsecondaryeducation system and the labour market and may be lessprepared to succeed when learning opportunities arepresented later in life. Those individuals with limited skillsand without the tools needed to be effective learnersthroughout life risk economic marginalization.

Having invested huge sums in providing high qualityuniversal elementary and secondary schooling,governments in industrialised countries, concerned aboutthe relative effectiveness of these education systems,wanted to address these issues. Therefore, membergovernments of the Organisation for Economic Co-operation and Development (OECD) developed acommon tool to improve their understanding of whatmakes young people—and education systems as a

whole—successful. This tool is the Programme forInternational Student Assessment (PISA).

Information gathered through PISA enables athorough comparative analysis of the skill level of studentsnear the end of their compulsory education. PISA alsopermits exploration of the ways that skills vary acrossdifferent social and economic groups and the factors thatinfluence the level and distribution of skills within andbetween countries.

The Canadian context

Canada’s participation in the PISA study stems frommany of the same concerns as have been expressed byother participating countries.

Canada invests significant public resources in theprovision of elementary and secondary education. AmongOECD countries, Canada ranks sixth in expenditure onelementary and secondary education as a proportion ofGDP.1 Canadians are concerned about the quality ofeducation provided by elementary and secondary schools.How can expenditures be directed to achieve high levelsof foundation skills and to potentially reduce social inequality?

Canada’s economy is also evolving rapidly. For thepast two decades the growth rate of knowledge-intensiveoccupations has been twice that of other occupations.2

Even employees in traditional occupations have beenasked to upgrade their skills to meet the rising skilldemands of new organisational structures and productiontechnologies. Primary and secondary education systemsplay a key role in generating the new supply of skills tomeet this demand. The skills acquired by the end ofcompulsory schooling provide the essential foundation

10

Introduction

upon which we will develop the human capital needed tomeet the economic and social challenges of the future.

Questions about educational effectiveness can bepartly answered with data on the average performanceof Canada’s youth. However, two other questions canonly be answered by examining the social distribution ofskills: Who are the students whose performance placesthem at the lowest levels? Do particular groups or regionsappear to be at greater risk? These are importantquestions because, among other things, skill acquisitionduring compulsory schooling influences access topostsecondary education and eventual success in thelabour market.

Furthermore, understanding how the knowledge andskills acquired by the end of compulsory education affectfuture life transitions, including participation in highereducation and the labour market, can only be achievedby examining the life paths of students as they progressfrom youth to adulthood. To answer these questions,Human Resources Development Canada, the Council ofMinisters of Education Canada and Statistics Canadadecided to integrate the PISA skill assessment with theCanadian Youth in Transition Survey (YITS).

What is PISA?

The OECD initiated the Programme for InternationalStudent Assessment (PISA) to provide policy-orientedinternational indicators of the skills and knowledge of 15-year-old students. 3 PISA is a collaborative effort amongOECD member countries to regularly assess youthoutcomes in three domains—reading literacy,mathematical literacy and scientific literacy—throughcommon international tests. International experts fromOECD member countries have agreed on the followingdefinitions for each domain:

Reading literacy: Understanding, using andreflecting on written texts, in order to achieve one’sgoals, to develop one’s knowledge and potential,and to participate in society.

Mathematical literacy: The capacity to identify,to understand, and to engage in mathematics andmake well-founded judgements about the role thatmathematics plays, as needed for individuals’current and future private life, occupational life,social life with peers and relatives and as aconstructive, concerned and reflective citizen.

Scientific literacy: The capacity to use scientificknowledge, to identify questions and to draw

evidence-based conclusions in order to understandand help make decisions about the natural worldand the changes made to it through human activity.

PISA assessed the degree to which studentsapproaching the end of their compulsory education haveacquired some of the knowledge and skills that areessential for full participation in society. PISA hopes toanswer the following questions:

• How well are young adults prepared to meet thechallenges of the future?

• Are they able to analyse, reason andcommunicate their ideas effectively?

• Do they have the capacity to continue learningthroughout life?

• Are some kinds of teaching and schoolorganization more effective than others?

Three PISA cycles have been planned, each onefocussing on a different literacy domain. In 2000 the majorfocus was reading literacy, with mathematical andscientific literacy as minor domains. Mathematical andscientific literacy will be focused on in 2003 and 2006,respectively.

PISA 2000

Thirty-two countries4 participated in PISA 2000. Thesurvey instruments were translated and adapted from twosource languages, English and French, into 17 differentlanguages. In most countries, between 4,500 and 10,00015-year-olds participated in PISA.

In Canada, approximately 30,000 15-year-oldstudents from more than 1,000 schools participated. Thelarge Canadian sample was needed to produce reliableestimates for each province5 , and for both English andFrench language school systems in Manitoba, Ontario,Quebec, New Brunswick and Nova Scotia. Theassessment was administered in schools, during regularschool hours, in April and May 2000.

The PISA 2000 survey included a direct assessmentof students’ skills through reading, mathematics andscience tests. A total of about seven hours of test itemswere administered, with each student taking a two-hour-long assessment consisting of different combinations oftest items. The assessment focused mainly on reading,with the reading test giving three sub-test scores labelledretrieving information, interpreting and reflecting.Mathematics and science each had only a single score.In addition, as minor domains, there were fewermathematics and science items included and these itemswere administered to a sub-sample of PISA participants.

11


Students also completed a 20-minute questionnairefocussing on factors contributing to student achievementand a three-minute questionnaire focussing on informationtechnology. In addition, PISA 2000 included aquestionnaire, which was administered to schoolprincipals, to collect information about the characteristicsof participating schools.

A 30-minute self-completed contextualquestionnaire from the Youth in Transition Survey wasadministered simultaneously to students in order to collectmore information on their school experiences, their workactivities and their relationships with others. A 30-minuteinterview was also conducted with parents.

More information on the PISA and YITS can be foundon the website www.pisa.gc.ca. Included aredocumentation on the PISA framework from theOECD, the PISA and YITS project in Canada, PISA andYITS questionnaires, example PISA test items and otherhelpful references. A detailed technical appendix isalso provided in the international OECD report,Knowledge and Skills for Life: First results fromthe OECD Programme for International StudentAssessment.6 Consult the OECD websitewww.pisa.oecd.org.

What is YITS?

The Youth in Transition Survey (YITS)7 is a newCanadian longitudinal survey designed to examine thepatterns of, and influences on, major transitions in youngpeople’s lives, particularly with respect to education,training and work. Survey results will help provide a deeperunderstanding of the nature and causes of challenges youngpeople face as they manage their transitions. Informationobtained from the survey will help to support policyplanning and decision making that addresses problems.

YITS will examine key transitions in the lives ofyouth, such as the transition from high school topostsecondary education, from schooling to the labourmarket, and from the labour market to schooling. Thefactors that affect leaving school without graduating willbe a focus, as will the effects of school experiences oneducational and occupational outcomes, and thecontribution of work experience programs, part-time jobsand volunteer activities. To collect this information, currentplans for YITS are to survey youth every two years,over a period of several years. Accordingly, the secondsurvey cycle of YITS is scheduled to take place in 2002.

Two different age groups are participating in YITS,a 15-year-old cohort and an 18- to 20-year-old cohort.The youth aged 15 who participated in YITS also

participated in PISA 2000. The youth aged 18 to 20, whowere surveyed in 2000 as part of the YITS project, didnot participate in PISA. Results for the 18- to 20-year-old YITS cohort will be released in a separate report inearly 2002.

Why are YITS and PISAintegrated in Canada?As with most surveys, PISA 2000 provides a‘snapshot’—a picture at a specific point in time—of thegroup being surveyed. A longitudinal survey like YITS,on the other hand, involves surveying the same group ofpeople over a period of time.

Collecting information on the same respondentsover time makes it possible for YITS to study relationshipsbetween factors measured in one period (e.g., aspirations,attitudes, behaviours and achievement) with outcomesmeasured in future time periods (e.g., educational attainment,occupational outcomes and earnings). Moreover, theintegration of YITS and PISA will enable the examinationof the relationship between tested skills and knowledgeand education and labour market outcomes of youth.

Objectives and organizationof the reportThis report provides results of the PISA assessment ofstudent performance in reading, science and mathematicsat the provincial level that complement the informationon national performance presented in the OECDinternational report, Knowledge and Skills for Life –First results from the OECD Programme forInternational Student Assessment. Wherever possible,an attempt has been made to put Canadian and provincialresults into context by comparing and contrasting themwith those of other countries.

Emphasis is placed on the average level ofperformance and on the distribution of achievementscores among specific social groups. This information ispresented in Chapter 1 of this report. Chapter 2 focuseson how achievement is influenced by a student’s personalcharacteristics and Chapter 3 explores the relationshipbetween family characteristics and achievement.Chapter 4 presents an analysis of the relationship betweenschool characteristics and achievement. Finally, the majorfindings and opportunities for further study are discussedin the conclusion.

This report is the first of a series of national reportsconceived to capitalise on the wealth of informationoffered by the PISA study and the YITS.

12

Introduction

BOX 1

Overview of PISA 2000

INTERNATIONAL CANADA

Participating countries/provinces • 32 countries • 10 provinces

Population • Youth aged 15 • Same

Number of participating students • In general, 4,500 to 10,000 per country, with • 30,000 students8some exceptions for a total of over 250,000students

Domains • Major: reading • Same• Minor: mathematics and science

Languages in which the test was • 17 languages • English and Frenchadministered

International assessment • Two hours of direct skills assessment through • Samereading, mathematics and science tests

• 20-minute self-completed contextual questionnaireadministered to youth

• A school questionnaire administered to school principals

International options • 3-minute optional self-completed • 3-minute optional self-completedquestionnaire on information questionnaire on informationtechnology administered to students technology administered to students

• An optional self-completed questionnaire onself-regulated learning administered to students

National options • Grade-based assessment • A 30-minute YITS self-completedquestionnaire administered to youth

• Other options were undertaken in a limited • A 30-minute phone interview with a parentnumber of countries of the youth

• Items added to the school questionnaire

Notes1. OECD (2000), Education at a Glance, Paris.2. Lavoie, Marie, and Richard Roy (1998), Employment in the

Knowledge-Based Economy: A Growth Accounting Exercise forCanada, Applied Research Branch Research Papers Series,Human Resources Development Canada catalogue no. R-98-8E,Ottawa.

3. The framework of PISA is presented in OECD (1999), MeasuringStudent Knowledge and Skills: A New Framework forAssessment, Paris.

4. Australia, Austria, Belgium, Brazil, Canada, Czech Republic,Denmark, Finland, France, Germany, Greece, Hungary, Iceland,Ireland, Italy, Japan, Korea, Latvia, Liechtenstein, Luxembourg,Mexico, The Netherlands, New Zealand, Norway, Poland,Portugal, Russian Federation, Spain, Sweden, Switzerland,United Kingdom, United States.

5. No data were collected in the three territories and on IndianReserves.

6. OECD (2001), Knowledge and Skills for Life: First Results fromthe OECD Programme for International Student Assessment,Paris.

7. More information on the Youth in Transition Survey is providedin Human Resources Development Canada and Statistics Canada(2000), Youth in Transition Survey: Project Overview, AppliedResearch Branch Technical Paper, Human ResourcesDevelopment Canada catalogue no. T-00-5E and StatisticsCanada catalogue no. 81-588-X1E, Ottawa.

8. The number of participating students in each province was asfollows: Newfoundland (2,281), Prince Edward Island (1,632),Nova Scotia (2,930), New Brunswick (2,963), Quebec (4,497),Ontario (4,290), Manitoba (2,599), Saskatchewan (2,716),Alberta (2,742), and British Columbia (3,037).

13


Chapter 1

The Achievement ofCanadian Students withinan International Context

T his chapter presents results of the PISA assessment in reading, mathematics and science. It begins by comparing the achievement of 15-year-old students in Canada and the provinces with thatof students in all participating countries. These initialcomparisons focus on differences in average scores.Further analyses reveal the proportions of students atvarious levels of the scoring scale. The chapter alsocompares the performance of girls and boys. Finally, theperformance of students enrolled in anglophone andfrancophone school systems, in the five provinces thatsampled the two groups separately, is discussed.

The performance of Canadianstudents in a global contextOverall, Canadian students performed well compared withstudents in most other countries, ranking second in reading,sixth in mathematics and fifth in science among31 countries1 (Figures 1.1 to 1.3). Canada is part of acluster of countries that scored near the top in all areas.Only Finland performed significantly better than Canadain reading, only Korea and Japan performed significantlybetter than Canada in mathematics and only Korea, Japanand Finland performed significantly better in science.Differences between Canada and the top countries onthe overall scale range from about 12 to 24 points.2

Table 1.0 shows the countries that performed significantlybetter than or about the same as Canada on the threetests. The average performance of students in all othercountries was significantly below that of Canada.

TABLE 1.0

Countries performing better than or aboutthe same as Canada

Countries performing Countries performingsignificantly better about the samethan Canada as Canada

Reading Finland New ZealandAustraliaIrelandJapan

Mathematics Japan FinlandKorea New Zealand

AustraliaSwitzerlandUnited Kingdom

Science Japan New ZealandKorea AustraliaFinland United Kingdom

Note: Differences in average scores between two countries are notstatistically significant when the confidence interval for theaverage score for each country overlaps. Countries performingabout the same as Canada have a confidence interval for theaverage score that overlaps with that of Canada.

14

Chapter 1 / The Achievement of Canadian Students within an International Context

FIGURE 1.2

Average Scores and Confidence Intervals by Province and Country: MATHEMATICS

FIGURE 1.1

Average Scores and Confidence Intervals by Province and Country: READING

525475425375Average score

575

95% Confidence interval

Average score

Finland

AustraliaNew Zealand

OntarioManitoba

Ireland

Alberta

Quebec

Korea

JapanNova Scotia

France

Saskatchewan

Austria

Denmark

Liechtenstein

Spain

PolandGreece

Mexico

IcelandNorway

Italy

Hungary

Portugal

Newfoundland

United Kingdom

Brazil

SwitzerlandCzech Republic

Germany

Prince Edward Island

Sweden

Belgium

Latvia

United States

British Columbia

CANADA

Luxembourg

New Brunswick


Average score

Finland

Australia

Ontario

Ireland

AlbertaQuebec

Korea

CANADA

Japan

Nova Scotia

Saskatchewan

Sweden

Belgium

Austria

Liechtenstein

SpainPoland

Russian Federation

Greece

Mexico

Iceland

Norway

Switzerland

New Brunswick

Italy

Germany

Portugal

Latvia

United Kingdom

Brazil

New Zealand

British Columbia

Manitoba

France

Denmark


Newfoundland

Czech Republic

HungaryUnited States

Luxembourg

475325 425375 525Average score

575

Russian Federation

15


FIGURE 1.3

Average Scores and Confidence Intervals by Province and Country: SCIENCE

While this was the first time that the PISA testswere administered, this is not the first program to testinternational achievement. The International Associationfor the Evaluation of Educational Achievement hasconducted a number of such studies over the past twentyyears. The most recent of these, the Third InternationalMathematics and Science Study (TIMSS), administeredmathematics and science tests in 1995 and 1999.3 Whilethe TIMSS and PISA studies are not directly comparablebecause of differences in frameworks, age differencesand differences in some participating countries, it is fairto say that Canada’s achievement ranking relative toother countries has improved substantially over time.Canada has risen from a mid-ranked country to one ofthe top-ranked countries in the 1999 TIMSS, and now inthe PISA assessments.

This does not tell us how much achievement hasactually improved, however, or indeed if it has improvedat all. It simply indicates that Canada has advanced inrelation to other countries. Successive assessments usingthe same test are needed to examine improvement inabsolute terms. Both PISA and the OECD and StatisticsCanada adult literacy studies4 are internationalassessments that explicitly provide for tracking change

in achievement over time. These studies should enhanceour understanding of the evolution of achievement andthe factors underlying observed change.

A note on statistical comparisons

The performance of students in different countries(and within Canada, in different provinces) wascompared by looking at the average scores for allstudents in each country and at the distribution ofthese scores. For example, the score achieved by thetop 10% of students tells us something about how thebest students in each country are performing.

Because the available scores were based on samples ofstudents from each country, we cannot say withcertainty that these scores are the same as those thatwould have been obtained had all 15-year-old studentsbeen tested. We use a statistic called the standard errorto express the degree of uncertainty in the scores forthe sample compared with the population. Using thestandard error, we can construct a confidence interval,which is a range of scores within which we can say,with a known probability (such as 95%), that the scorefor the full population is likely to fall. The 95%confidence interval used in this report represents arange of plus or minus about two standard errorsaround the average.


Average score

Finland

AustraliaNew Zealand

OntarioManitoba

Alberta

Luxembourg

Korea

CANADA

Japan

Nova Scotia

France

Sweden

Belgium

Austria

United States

Spain

Poland

Russian Federation

Mexico

Iceland

Norway

Switzerland

New Brunswick

Italy

Hungary

PortugalLatvia

United Kingdom


Brazil

Saskatchewan

Newfoundland

Ireland

Czech Republic

Quebec

British Columbia

Germany

Denmark

Liechtenstein

500 550450400350Average score

600

Greece

16


When comparing scores among countries or provinces,we must consider the degree of error in each scorebefore we can say that two scores are significantlydifferent from each other. Standard errors andconfidence intervals may be used as the basis forperforming these comparative statistical tests. Suchtests allow us to say, with a known probability, whetherthere are actual differences in the populations beingcompared. For example, when we report that anobserved difference is significant at the .05 level, weare saying that the probability is less than .05 that theobserved difference could have occurred because ofsampling error. When comparing countries andprovinces, extensive use is made of this type of test toreduce the likelihood that differences due to samplingerrors will be overstated.

Only statistically significant differences are noted assignificant in this report.

FIGURE 1.4

Average Scores and Confidence Intervals for Provinces and Countries: READING RETRIEVING

Provincial results in aninternational context

Most provinces performed well in reading, science andmathematics. In fact, the majority of provinces performedas well as the top ranked countries in the world(Figures 1.1 to 1.3). The performance of students inAlberta was significantly above the Canadian average inall three domains, as was the performance of Quebecstudents in mathematics and science. In Ontario,Manitoba, Saskatchewan and British Columbia, theperformance of students was about the same as theCanadian average in all three domains while theperformance of students in Newfoundland, PrinceEdward Island, Nova Scotia and New Brunswick wassignificantly lower. The performance of students in thefour Atlantic Provinces was, however, at or above themiddle of the international range.

Australia

New Zealand

Ontario

Manitoba

CANADA95% Confidence interval

Average score


600

Alberta

Korea

British Columbia

Finland

Quebec

Ireland

Nova ScotiaFrance

BelgiumNewfoundland

Japan

Sweden

LuxembourgGreece

Mexico

Hungary

PortugalLatvia

Brazil

Austria

United States

Denmark

Liechtenstein

Spain

Poland

Russian Federation

Iceland

Norway

SwitzerlandNew Brunswick

Italy

Germany

Saskatchewan

United Kingdom

Czech Republic


17


FIGURE 1.5

Average Scores and Confidence Intervals for Provinces and Countries: READING REFLECTING

FIGURE 1.6

Average Scores and Confidence Intervals for Provinces and Countries: READING INTERPRETING



Average score

Australia

New Zealand

Ontario

Alberta

Luxembourg

Quebec

Korea

CANADA

Japan

Nova Scotia

France

Sweden

BelgiumDenmark

Liechtenstein

Spain

Czech Republic

Poland

Greece

Mexico

Iceland

Norway

Germany

HungaryPortugal

Latvia

Newfoundland

United Kingdom


Brazil

Saskatchewan

Manitoba

Ireland

Finland

British Columbia

Austria

New Brunswick

SwitzerlandItaly

United States

Russian Federation

525375 475425 575Average score


Average score

Finland

Australia

New Zealand

Ontario

Manitoba

Ireland

Alberta

Korea

Japan

British Columbia

Nova Scotia

France

Saskatchewan

Sweden

Belgium

Austria

Denmark

LiechtensteinPoland

Russian Federation

Greece

Mexico

Norway

Italy

Hungary

Portugal

United Kingdom

Brazil

Iceland


Newfoundland

Quebec

CANADA

United States

Czech RepublicSwitzerland

Germany

LatviaLuxembourg

Spain

New Brunswick

18


Results for reading achievement are also presentedfor three sub-scales measuring different readingcompetencies: retrieving, interpreting and reflecting(Figures 1.4 to 1.6). Interestingly, Canada’s performanceon the reading reflecting sub-scale was significantly abovethat of Finland. Definitions of the reading sub-scales arepresented in Annex B: Definitions of Key Variables andConstructs.

The rankings observed in the PISA data agree,generally, with those drawn from other sources. WithinCanada, the School Achievement Indicators Program(SAIP) has administered reading and writing, mathematicsand science tests to 13-year-olds and 16-year-olds in allprovinces and territories on a rotating basis since 1993.5

These assessments reveal small but persistent differencesbetween jurisdictions, with students in British Columbia,the Prairie provinces and Quebec tending to do betterthan those in Ontario and the Atlantic provinces. A similarpattern has emerged in TIMSS (although not all provinceswere sampled adequately in TIMSS to give a full set ofprovincial comparisons).

Comparing Canadian results to thoseof selected countries

For all Canadian provinces, the samples were largeenough to allow the results to be compared with thoseof other provinces and countries. To simplify thepresentation of Canadian results, all of the analysis inthis chapter that extends beyond an examination ofaverage performance, and all of the analyses in theremaining chapters, will be based on a subset ofcountries. Thirteen countries were selected forcomparison with Canada and the provinces. In additionto other G8 countries (France, Germany, Italy, Japan,United Kingdom, United States and the RussianFederation), Australia, Belgium, Finland, Mexico,Sweden and Switzerland were selected because of theirsimilarities to Canada, their record of high achievementor their relevance to Canada. An analysis of theperformance of all countries is presented in theinternational OECD report, Knowledge and Skills forLife – First results from the OECD Programme forInternational Student Assessment.

In addition to the tables and figures presented withinthe body of this and subsequent chapters, results arealso presented in a series of detailed tables in Annex A:Tables.

The distribution of scores

We can learn more about how students perform bylooking at how the scores are distributed within eachcountry or province. Two countries with the same averagemay have quite different numbers of especially high- orlow-performing students. Differences in how the scoresare distributed tell us something about the degree ofequality in proficiency among students within a countryand across countries.

For example, the distribution of reading scores inCanada, Finland, and the United States is presented inFigure 1.7. Although there are significant differences inthe average scores of these countries, the largest peak,or most frequent score, in each country is fairly similar.When examining the distribution of scores to the right ofthe peaks, it is clear that the distribution for Canada issimilar to that of Finland across a wide range. Thisindicates that Canada’s highest performing studentsperformed as well as the highest performing Finnishstudents. In contrast, only the very highest performingstudents in the United States appear to have performedas well as those in either Canada or Finland. When lookingat the distribution to the left of the peak, Finland had fewerlow-performing students than did Canada. The UnitedStates, however, had more lower performing studentsthan either Canada or Finland. In the distribution of scoresfor the United States there are two peaks. The first,smaller peak represents a large sub-population of studentsthat performed less well than those represented by thesecond, larger peak. It is the influence of this sub-population that caused the average score for the UnitedStates to fall well below that of Canada and Finland.

A simpler way to examine distributions is to calculatethe percentile scores—the scores below which a specifiedpercentage of students are found. This gives us percentileranks, or just percentiles. Thus, the 10th percentile is theone below which we find 10% of students. The50th percentile is called the median and is the score belowwhich we find half the students. By comparing scores atspecific percentiles, we are able to examine the distributionof scores within a population. Tables 1.7 to 1.9, presentedin Annex A, show the scores that correspond to the 5th,10th, 25th, 50th, 75th, 90th and 95th percentiles forreading, mathematics and science for Canada as a whole,the provinces and selected countries.

19


Returning to the examination of readingperformance in Canada, the United States and Finland,the score for the 95th percentile was identical in Finlandand Canada. In the United States, the score, whileappearing to be lower, was not significantly different fromthat of Canada or Finland. Canada’s score for the 5thpercentile, however, was significantly below that ofFinland and the score for the United States wassignificantly below that of both Canada and Finland.

Belgium is perhaps the best example of how spreadout scores can be toward the low end of the distribution.In all three sets of scores, the 5th percentile score inBelgium is among the lowest of all countries even thoughthe average score places the country at a fairly high rank.As a final example, although Canada and Australia havesimilar average scores, Australia’s distribution issomewhat broader than Canada’s, indicating a morediverse population. Most provinces have fairly narrowdistributions compared with other countries.

Variation in proficiency

A measure of the distribution of scores within and acrosscountries is obtained by examining the ratio of scores atthe 90th percentile to that at the 10th percentile. A ratioclose to 1 indicates that all students in a country achievenearly the same level. Higher ratios indicate relativelygreater variation.

Jurisdictions with high average scores tend to haveless variation in achievement than do those with lowaverage scores (Figures 1.8 to 1.10). However, whenwe rank countries according to this index, Canada’s rankshifts down slightly. This indicates that, despite high overallperformance, relatively more students in Canada are nearboth the top and the bottom of the distribution than in someother highly-ranked countries, such as Finland and Japan.

At the same time, there are differences amongprovinces in the inequality index. Generally speaking,provinces that performed better also tended to have lessvariation between the top and bottom of the distribution.

FIGURE 1.7

The Distribution of Reading Scores in Canada, Finland and the United States

200 300 400 500 600 700 800 900

Reading score

CANADA

United States

Finland

20


FIGURE 1.8

Inequality Index of Reading Scores (90th percentile/10th percentile)

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

FIGURE 1.9

Inequality Index of Mathematics Scores (90th percentile/10th percentile)

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

Finl

and

Japa

n

Sask

atch

ewan

Que

bec

Albe

rta

CAN

ADA

Briti

sh C

olum

bia

Swed

en

Ont

ario

Man

itoba

Fran

ce

Italy

Nov

a Sc

otia

Prin

ce E

dwar

d Is

land

Aust

ralia

Uni

ted

King

dom

New

foun

dlan

d

New

Bru

nsw

ick

Rus

sian

Fed

erat

ion

Mex

ico

Switz

erla

nd

Uni

ted

Stat

es

Belg

ium

Ger

man

y

Finl

and

Sask

atch

ewan

Que

bec

Albe

rta

Japa

n

New

foun

dlan

d

CAN

ADA

New

Bru

nsw

ick

Ont

ario

Man

itoba

Prin

ce E

dwar

d Is

land

Briti

sh C

olum

bia

Nov

a Sc

otia

Aust

ralia

Uni

ted

King

dom

Fran

ce

Swed

en

Switz

erla

nd

Italy

Uni

ted

Stat

es

Belg

ium

Mex

ico

Ger

man

y

Rus

sian

Fed

erat

ion

21


FIGURE 1.10

Inequality Index of Science Scores (90th percentile/10th percentile)

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

1.2

2.0

1.7

1.6

1.5

1.4

1.3

1.9

1.8

Index

Distribution of high performers

An argument can be made that individuals who canachieve at the highest possible levels make an importantcontribution to the well being of a society in an increasinglycompetitive global economy. For example, they may bemost likely to contribute to our ability to undertake world-class research and development in the future.

Another way to compare jurisdictions is to comparethe proportion of students performing at the highest levels.This can be done by examining the performance of all ofthe students in the 13 selected countries and in Canada,and finding the scores for this international group thatcorrespond to selected percentile ranks, such as the90th percentile (top 10% of students), the 75% percentile(top 25% of students) and the 50th percentile (top 50%of students). The percentage of students in eachjurisdiction with scores equal to or above theseinternational ranks is presented in Figures 1.11 to 1.13.

Again, this perspective reveals a somewhat differentpattern than the others. In particular, the differencesbetween jurisdictions in the percentage of students at thevery top of the score distribution (top 10%) are much

larger than the differences in overall averages. Formathematics and science, many provinces have shiftedlower in the international rankings, and the differencesamong provinces are more pronounced.

Reading skill levels

Rankings can tell us how countries and provinces comparewith each other overall. Rankings tell us nothing, however,about what students can actually do. We can elicit moreinformation from the data if we are able to describe whatcan be done at specific score levels. For this reason,reading achievement was divided into five levels.

As expected, the highest-ranking jurisdictionsoverall also tend to have the highest proportion of studentsat Level 5 (Figure 1.14). However, a few substantial shiftsare apparent. For example, Japan has relatively few Level5 students, although with a large proportion at Level 4 ithas a high ranking on overall performance. On the otherhand, Australia ranks higher on this scale than on theoverall performance scale, because of a relatively highproportion of students at Level 5. Generally speaking, thepositions of Canada and the provinces do not shift much.

Finl

and

Sask

atch

ewan

Albe

rta

Japa

n

Briti

sh C

olum

bia

Prin

ce E

dwar

d Is

land

Man

itoba

Nov

a Sc

otia

Ont

ario

CAN

ADA

Que

bec

New

foun

dlan

d

New

Bru

nsw

ick

Aust

ralia

Mex

ico

Swed

en

Uni

ted

King

dom

Uni

ted

Stat

es

Switz

erla

nd

Italy

Fran

ce

Ger

man

y

Rus

sian

Fed

erat

ion

Belg

ium

22


FIGURE 1.12

Percent of Students Above 90th, 75th and 50th International Percentiles: MATHEMATICS

0 2 0 4 0 6 0 1008 0Percent

90th percentile(top 10%)



FIGURE 1.11

Percent of Students Above 90th, 75th and 50th International Percentiles: READING

0 2 0 4 0 6 0 100Percent

8 0




AlbertaFinland

British ColumbiaAustraliaCANADAOntario

ManitobaQuebec

United KingdomSaskatchewan

Nova ScotiaNewfoundland

Prince Edward IslandBelgium

United StatesSweden

JapanNew Brunswick

SwitzerlandGermany

FranceItaly

Russian FederationMexico

JapanAlberta

QuebecSwitzerland

BelgiumAustralia

United KingdomBritish Columbia

ManitobaCANADA

FinlandFrance

OntarioSaskatchewan

SwedenUnited States

Nova ScotiaGermany

Russian FederationPrince Edward Island

NewfoundlandNew Brunswick

ItalyMexico

23


FIGURE 1.14

Reading Proficiency Scales: Percent of Students at Each Level

0 2 0 4 0 6 0 8 0 100

Below level 1

Level 1

Level 2

Level 3

Level 4

Level 5

Percent

0 2 0 4 0 6 0 100Percent

8 0




FIGURE 1.13

Percent of Students Above 90th, 75th and 50th International Percentiles: Science

JapanAlberta

United KingdomQuebecFinland

AustraliaBritish Columbia

CANADAManitoba

OntarioBelgium

FranceNewfoundland

SwedenUnited States

SaskatchewanSwitzerland

Nova ScotiaPrince Edward Island

GermanyNew Brunswick

ItalyRussian Federation

Mexico

AlbertaFinland

British ColumbiaAustraliaCANADAOntario

ManitobaQuebec

United KingdomSaskatchewan

Nova ScotiaNewfoundland

Prince Edward IslandUnited States

BelgiumSweden

JapanNew Brunswick

SwitzerlandGermany

FranceItaly

Russian FederationMexico

24


Five levels of reading literacy

Reading achievement was divided into five levels.Essentially, these levels represent the most difficult testitems that a student could answer. Therefore, a studentat one level could be assumed to be able to answerquestions at all lower levels. To help in interpretation,these levels were linked to specific score ranges on theoriginal scale. Because the five levels are complex todescribe, an example from each level is given for thereading retrieving scale. Tasks of similar complexitywere required for each level of the other reading scales.

Level 1 (score from 335 to 407)

Students were shown a notice from a personneldepartment about a service that would help with jobmobility. They were asked to find a single explicitlystated piece of information—how to find out moreabout the service—which was signalled by a headingin the text that matched the term used in the question.


Students were required to state how to check that abicycle seat was in the right position, by finding twopieces of connected information in an assemblymanual. The placement of the relevant informationwas clearly stated in the question.


Looking at a complex international airline timetable,with prominent competing information, students hadto find a single piece of information that satisfied threeconditions—time, destination and connecting city. Forinformation about one of the conditions, the readerhad to refer to a separate list of abbreviations.


Presented with a relatively long, dense extract from aplay, students had to use information embedded in astage direction in order to mark the positions of twoactors on a diagram of the stage.

Level 5 (score above 626)

Students were given a complex and unfamiliar set ofinstructions about how to make telephone calls from ahotel room, and a letter with the phone number of afriend in a different country. They were required tofind and organise in correct sequence four pieces ofinformation and to draw inferences to work out exactlyhow to dial the number.

Performance below level 1

Students performing below Level 1 (total reading scorebelow 335) are not able to routinely show the mostbasic type of knowledge and skills that PISA seeks tomeasure. Such students have serious difficulties in usingreading literacy as a tool to advance their knowledgeand skills in other areas. Placement at this level doesnot mean that these students have no literacy skills.Most of these students are able to correctly completesome of the PISA items. Their pattern of responses tothe assessment is such that they would be expected tosolve less than half of the tasks from a test composedof only level 1 items.

Interpreting differences in PISAreading scores

A difference of 73 points between two average scorescould be thought of as representing about oneproficiency level in reading literacy. A difference ofone proficiency level can be considered acomparatively large difference in student performancein substantive terms. For example, on the interpretingscale, Level 3 distinguishes students who can typicallyintegrate several parts of a text, understand arelationship or construe the meaning of a word orphrase, and can compare, contrast and categorisecompeting information according to a range of criteria.At Level 2, students can be expected only to identifythe main idea in a text, to understand relationships,make and apply simple categories, and construemeaning within a limited part of a text whereinformation is not prominent but only low-levelinferences are required.

How do boys and girls compare?

All jurisdictions have an interest in reducing genderdisparities in educational performance. Previous studieshave revealed a pattern of higher performance for girlsin reading and writing and for boys in mathematics andscience. There has also been a tendency for thesedisparities to widen as students progress through theeducation system. In Canada, the SAIP studies haverevealed small but significant differences favouring girlsin reading and writing but few gender differences inmathematics and science achievement.

25


In PISA, girls performed significantly better thanboys on the reading test in all countries and in all provinces(Table 1.17). In contrast, for mathematics and science,few significant differences between girls and boys wereobserved (Tables 1.18 and 1.19). This does not indicatethat there are, necessarily, few differences between thegenders in mathematics and science, only that as minordomains in PISA 2000 differences were not observable.

Canada, France and Germany were the onlycountries where small gender differences in mathematicsachievement were significant. In these countries, resultsfavoured boys, however, the difference in average scoresbetween the genders was much less than that observedfor reading. With the smaller sample sizes in individualprovinces, and consequent higher standard errors, thesmall differences between the genders in mathematicswere not significant in any individual province. For scienceachievement, there were no significant differencesbetween girls and boys in any country or province.

Achievement of Canadian studentsby language of the school system

This section examines the performance of students inEnglish and French school systems for the five Canadianprovinces that sampled these population groupsseparately.6 The focus is on the performance of theminority group (students in francophone school systemsin Nova Scotia, New Brunswick, Ontario and Manitoba,and students in the anglophone school system in Quebec)relative to the majority.7

The SAIP assessment gives the only availablehistorical data on the performance of the two linguisticgroups. In general, students from francophone schoolsystems outside of Quebec performed at levels belowthe Canadian average, below their counterparts inQuebec, and below the levels of the anglophone majoritiesin the same provinces. There are a few exceptions tothis, including the relatively high performance of thefrancophone education system in Nova Scotia on the 1997mathematics assessment. Within Quebec, theperformance of the two linguistic groups has historicallybeen similar and relatively high.

A comparison of PISA results within each provinceis given in Table 1.20. In all provinces except Quebec,students enrolled in minority language education systemsperformed at a significantly lower level in reading thandid students in the majority systems in the same provinces.In mathematics, only Ontario had significant differences

among the two school systems, with results favouringthe anglophone system. For science, there weresignificant differences favouring the anglophone systemin Nova Scotia, New Brunswick, Ontario and Manitoba.

In Quebec, the differences between the two schoolsystems were not significant for any of the subjects. Inaddition, Quebec students performed well relative to thosein other provinces with significant linguistic minorities.

Further analysis of the information collected throughPISA and YITS will clarify the extent to which importantbackground variables, such as the main language spokenat home, contribute to these differences and will explorethe role schools play in amplifying or attenuating thesedifferences.

Conclusion

Reading is key to many other areas of activity, both insideand outside school. While the cumulative effect of poorreading performance is not fully understood, it is clearfrom the International Adult Literacy Survey (IALS) thatadults with low levels of literacy skills are at a significantdisadvantage in Canada’s labour market.8 As a result,while Canada’s overall performance in PISA was verygood, the existence of disparities among provinces, andbetween official language groups within some provinces,is a matter of concern and merits further analysis.

The lower performance levels for boys in readingwill also concern policy makers both in Canada and inother countries. Further analysis, presented in the nextchapter, points to differences in key individualcharacteristics, such as enjoyment of reading. A smallerdifference, favouring boys, in mathematics achievementwas also observed at the Canada level. The results ofPISA 2003, where mathematics will be the major domain,should clarify the extent to which gender differences inmathematics occur among jurisdictions.

While the comparative approach taken in thischapter does not lend itself to developing explanationsfor these disparities, the overall PISA/YITS study, alongwith data available from SAIP, IALS and other large-scale assessments, provides, for the first time, a seriesof rich databases that should allow researchers andeducators to explore how resources, schools andclassroom conditions—as well as individual and familycircumstances—affect variation in achievement.Subsequent chapters of this report give a preliminaryanalysis of some of these factors and provide an indicationof the potential for more comprehensive analysis.

26


Notes1. Results for the Netherlands are not presented due to a low

response rate. As a result, only 31 countries are included inFigures 1.1 to 1.6.

2. The performance of students was expressed as a number ofpoints on a scale constructed so that the average score forstudents in all participating OECD countries was 500 and itsstandard deviation was 100. This means that about two-thirdsof students internationally scored between 400 and 600 on thescale.

3. For more information on TIMSS consult the websitewww.timss.bc.edu.

4. The OECD and Statistics Canada International Adult LiteracySurvey (IALS) and the forthcoming International Adult Literacyand Life Skills Survey (ALL).

5. For more information on SAIP consult the websitewww.cmec.ca.

6. The number of participating students in the francophone andanglophone school systems were, respectively: Nova Scotia(216; 2714), New Brunswick (1150;1813), Quebec (3150;1347),Ontario (1003;3287), and Manitoba (241;2358).

7. Within all anglophone school systems, both students in FrenchImmersion programs and those in regular programs completedthe reading test in English. Some French Immersion studentscompleted the mathematics and science tests in French.

8. OECD and Statistics Canada (2000), Literacy in the InformationAge: Final Report of the OECD Literacy Study, OECD andMinister of Industry, Paris and Ottawa.

27


T his chapter presents an analysis of how thepersonal characteristics of individual studentsmeasured in the PISA study influence theirachievement in reading, mathematics and science. Child-centred learning theory holds that since individuallearners have the most direct responsibility for theiracademic outcomes, their unique characteristics play animportant role in the educational process.1

This chapter examines the effects on studentachievement of a number of individual characteristicsof youth including reading behaviours, attitudes towardschool, career and education expectations, andemployment experiences. The first part of the chapterexamines the absolute effects of individual characteristicson achievement. The second part analyses theseindividual characteristics within a multifacetedframework that accounts for interrelationships amongthe variables. This analysis of relative effects allows usto discern which variables have the strongestrelationships with achievement.

Estimating absolute and relative effects

Chapters 2, 3 and 4 present the absolute and relativeeffects of individual, family and school characteristicson student achievement. The absolute effect refers tothe variable’s effect in the absence of other variables—it measures the independent contribution of thevariable to student achievement.2 The relative effectrefers to the variable’s effect in the presence of othervariables—it measures the residual contribution of thevariable to student achievement. A variable may beimportant by itself but unimportant when othervariables are also taken into consideration. To estimaterelative effects, variables are entered together into onemultiple regression model for each country andprovince.

The following thresholds in the absolute values of theeffect size statistic were used to judge the magnitudeof the effect:

Trivial: Less than |0.10|

Small: Between |0.10| and |0.30|

Moderate: Between |0.30| and |0.50|

Large: Greater than |0.50|

An effect size less than |0.10| indicates that less than1% of the variance in achievement scores is explainedby the variable and, as such, reflects a trivial impact.

For more information on standardised effects see thenotes accompanying the tables in Annex A.

Chapter 2

The Impact of IndividualCharacteristics on Achievement

28

Chapter 2 / The Impact of Individual Characteristics on Achievement

Reading behaviours

Reading enjoyment had a positive effect on readingachievement in all countries, with a higher level ofreading enjoyment associated with a higher level ofachievement (Table 2.1). In almost all countries, theeffects of reading enjoyment were moderate. Canada wasamong the countries where the effect was the greatest.Within Canada, a moderate positive effect of readingenjoyment on reading achievement was evident in allprovinces.

As with many of the individual characteristicsexamined in this chapter, the relationship between thecharacteristic and performance may be interpreted in twoways. In this case, reading enjoyment may improvereading skills, while better readers may enjoy readingmore.

Reading diversity, that is reading a variety ofdifferent types of material for enjoyment, was positivelyassociated with reading achievement (Table 2.1). Theeffects were small or moderate in almost all countries.Within Canada, small positive effects of reading diversityon reading achievement were evident in all provincesexcept Nova Scotia and New Brunswick, where moderatepositive effects were found.

Students were asked to report their daily time spentreading for enjoyment. Overall, in all countries andprovinces, reading performance increased as the timespent reading for enjoyment grew (Table 2.2). Therewas, however, no added benefit of reading more thanone or two hours daily in most countries and provinces.In Canada and Germany, (and also in New Brunswick,Saskatchewan and British Columbia), students readingtwo or more hours daily had significantly lower averageachievement scores than those reading one to two hoursdaily. This phenomenon may result because thesestudents read more slowly and thus take more time toread. Further analysis is required to understand thisrelationship.

The use of public and school libraries measuresthe frequency with which students borrow books fromlibraries to read for enjoyment. In all countries, exceptMexico, and in all provinces there was a significantdifference in reading achievement between those whoborrowed books once per month and/or several timesper month and those who never borrowed books(Table 2.3).3 The same pattern was observed in countriesand provinces for science achievement, with onlystudents in France, Mexico and the Russian Federationnot showing benefits of library use. In Mathematics,

differences in average achievement between those usinglibraries and those never using libraries were significantin all countries except France, the United States, Italy,the Russian Federation and Mexico. Among provinces,these differences were significant for mathematics in allprovinces except Newfoundland, New Brunswick andAlberta.

Attitudes toward school

Student attitudes toward school were measured throughtime spent on homework and a measure of the sense ofbelonging to school.

Time spent on homework had a small or moderatepositive effect on achievement in almost all countriesand in all provinces (Table 2.4). Among provinces, theonly exception was Newfoundland where effects formathematics and science were trivial. Sense of belongingto school did not appear to be related to achievement inmost countries and in all provinces (Table 2.4). Whereit did have an impact, the effect was small.

Student career expectations

Student career expectations, based on the occupationalstatus of the job students expect to have when they areabout thirty, were positively linked to achievement inall countries (Table 2.5). While career aspirations maymotivate students to perform better, aspirations may alsobe influenced by role models, particularly parents, byprevious academic performance and by the orientationof students’ educational programs.

The effects of student career expectations weresmall to moderate. Effects were small in six countries,including Canada, in reading; in seven countries,including Canada, in mathematics; and in five countries,including Canada, in science. Other countries hadmoderate effects. Among provinces, effects were smallfor all three domains with the exception of Saskatchewanwhere the effect for reading achievement was moderate.

Student education expectations

The Youth in Transition Survey (YITS) asked studentsto specify the highest level of education they expectedto achieve. As with career expectations, studentexpectations of the highest level of education had apositive relationship with achievement across allprovinces (Table 2.6). The interpretation of this

29


relationship, however, is complex. While the intentionto pursue a postsecondary education may motivatestudents to perform better, students with a record ofprevious academic success will be the most likely toaspire to higher levels of education. Also, students’educational aspirations will be linked to the orientationof their educational programs and to their parents’educational attainment and aspirations for them.

Working while studying

Using YITS data, a preliminary analysis was undertakento investigate how working while studying relates toachievement. Table 2.7 presents the average reading,mathematics and science achievement of students withand without a job in the school year. In all provinces,students without jobs during the school year hadsignificantly higher average reading performance thandid working students. Many provinces did not, however,have significant differences favouring non-workingstudents in mathematics achievement or scienceachievement.

Nonetheless, for all three domains, as the hoursworked per week increased, performance tended todecline. Small negative effects related to the hoursworked during weekends and during the school weekwere found in all provinces for reading, mathematics andscience achievement (Table 2.8).

Previous studies have shown that working a limitednumber of hours while in school does not increase theprobability of dropping out of high school.4 The longer-term impact of combining work and study at age 15 oneducation and labour-market outcomes will be clarifiedthrough future research that takes advantage of thelongitudinal nature of YITS.

The relative importance of individualfactors in explaining achievement

Most of the individual characteristics of youth, whenconsidered alone, are significant predictors of studentachievement across countries and provinces. Many ofthese factors, however, are inter-related. Altogether,which of these factors are the most influential, and arethere jurisdictional differences? To answer thesequestions, most of the individual factors examined inthis chapter were considered together in one multipleregression model for each country and province.

In addition to the variables previously analysed inthis chapter, this analysis includes Gender, presented inChapter 1. Variables from YITS that were included earlierin this chapter, employment experiences and educationexpectations, are not included in this analysis as theyare not available for international comparison.

Table 2.9 presents results of this multiple regressionanalysis. It shows the magnitude of the relative impactof the variables as small (s), medium (m), or large (l),using the criteria outlined earlier in this chapter. Variableswith a negative effect are indicated with a “-” sign.

When all individual characteristics are considered,reading enjoyment and student career expectationsremain important individual characteristics in almost allcountries and in all provinces. Reading enjoyment is apredictor of achievement in all countries except Mexico(for all three domains) and France and Japan (formathematics). Similarly, student career expectations isa significant predictor in all jurisdictions except in Japanfor science.

Other reading behaviours, reading diversity andtime spent reading for enjoyment, were correlated withreading enjoyment and thus did not emerge as predictorsin this analysis in many countries and in most provinces.

In this analysis, gender differences no longer hadan effect on reading achievement in all countries exceptAustralia, Finland and Mexico, and in all provinces. Thisis because the variation in reading achievement betweenboys and girls is related to differences in readingbehaviours between the genders, particularly readingenjoyment, which were included in this analysis. Smalleffects favouring males in mathematics were apparentin all countries, except Australia and Mexico, and in allprovinces. Similarly, in eight countries, includingCanada, and in all provinces except New Brunswick,there were small effects favouring males in science.

The PISA 2000 assessment focused on readingachievement and did not gather information that wouldhelp us to understand whether or not gender differencesare related to differences in enjoyment of mathematicsand science or to differences in behaviours that mayenhance mathematical or scientific literacy. A morecomprehensive analysis of gender differences inmathematics and science will be possible following theadministration of PISA in 2003 and 2006.

30

Chapter 2 / The Impact of Individual Characteristics on Achievement

In a number of provinces and countries (includingCanada for reading) the frequency of borrowing booksto read for enjoyment from school and public librariesshowed a small negative relationship to studentachievement. This counter intuitive outcome likelyresults because this analysis excludes variablesmeasuring family socio-economic background andcontrols for the effect of other reading behaviours.Among students who enjoy reading for pleasure, thoseusing libraries may have fewer resources with which topurchase reading material. The impact of family socio-economic background on achievement is explored indetail in the next chapter.

Other individual variables showed effects lessconsistently across countries and provinces. When effectswere present, they were small.

Conclusion

Analysed individually, most factors included in thischapter showed a measurable relationship toachievement. In particular, reading behaviours such asreading enjoyment, reading diversity and time spentreading for enjoyment had strong effects on reading,mathematics and science results. Similarly, time spenton homework showed a small effect on performance.Students’ education and career expectations werecorrelated with performance. For many variables, andmost particularly for career and education expectations,the relationship between the variable and achievementmay result from complex causes. While individualbehaviours may lead to improvements in academicperformance, in many cases, the behaviours themselves,

such as career aspirations, may result from a student’sown evaluation of his or her academic abilities.

Moreover, many of the individual factors analysedin this chapter are correlated. Once these interrela-tionships were taken into account, only readingenjoyment and career expectations stood out as stronglyrelated to achievement in all three domains. Genderdifferences in mathematics and science, which are alsoimportant factors, merit more comprehensive analysisfollowing future cycles of PISA.

Notes1. Miller, John P. and Wayne Seller (1990), Curriculum:

Perspectives and Practices, Copp Clark Pitman, Toronto.2. If the variable examined was categorical, average achievement

scores were calculated for each category and then compared.If the variable was continuous, regression analysis wasperformed and an effect size was used to examine the effectsof the variable on achievement.

3. In the relative effects analysis, the relationship between theuse of libraries and achievement is negative in manyjurisdictions. This outcome is likely due to the close relationshipbetween this behaviour and family socio-economic status whichbecomes evident once effects of other reading behaviours havebeen accounted for.

4. Dagenais, Marcel, Claude Montmarquette, Daniel Parent,Benoit Durocher and François Raymond (1999), Working WhileStudying and School Leavers: Causes, Consequences andPolicy Interventions. Applied Research Branch ResearchPapers Series, Human Resources Development Canadacatalogue R-99-5E, Ottawa.Sunter, Deborah (1993), “School, Work and Dropping Out”,Perspectives on Labour and Income, Statistics Canadacatalogue no. 75-001, Summer, pp. 44-52, Ottawa.

31


A student’s family and home environment influences achievement. This chapter identifies a set of variables that describe thesecharacteristics and examines their relationship to studentachievement in reading, mathematics and science.

This chapter examines the effects on studentachievement of a variety of family factors, includingfamily background, home environment, familyeducational support, parental involvement and parentalexpectations. The first part of the chapter examines theabsolute effects of family characteristics on achievement.The second part analyses these characteristics within amultifaceted framework that accounts forinterrelationships among the variables. This analysis ofrelative effects allows us to discern which variables havethe strongest relationships with achievement.1

Family background

Family background characteristics which were examinedinclude family structure, the number of siblings in thefamily and family socio-economic status. Familystructure divides students into two categories: those insingle-parent families and those in two-parent families.In half of the 14 countries examined, including Canada,students from two-parent families had significantlyhigher levels of achievement than did students fromsingle-parent families (Table 3.1). Differences in student

performance resulting from family structure were,however, linked to differences in other importantcharacteristics, particularly family socio-economicstatus. The interrelationship among family characteristicsis explored later in this chapter.

Among provinces, the average readingperformance of students in two-parent families wassignificantly above that of students in one-parent familiesin New Brunswick, Manitoba, Saskatchewan, Albertaand British Columbia. In mathematics, there was asignificant difference in New Brunswick, Quebec,Manitoba, Saskatchewan, and Alberta and in science,the advantage was significant in New Brunswick,Manitoba and Alberta.

In many countries, a larger number of siblings inthe family had a small negative relationship with studentachievement in all three domains (Table 3.2). In Canada,there was a small negative effect of the number of siblingson mathematics and science achievement. For readingachievement, effects in Canada were trivial. In Japanand Finland, effects were trivial across all three domains.

The number of siblings in the family had a negativerelationship with achievement in some provinces. InManitoba, Saskatchewan and Alberta, there was a smallnegative effect present in all three domains. There wasalso a small negative effect on mathematics achievementin Quebec, Manitoba, Saskatchewan, Alberta and British

Chapter 3

The Impact of FamilyCharacteristics and HomeEnvironment on Achievement

32

Chapter 3 / The Impact of Family Characteristics and Home Environment on Achievement

Columbia, and on science achievement in all provincesexcept Prince Edward Island, Nova Scotia and NewBrunswick.

Socio-economic differences in academicachievement have been abundantly documented in majornational and international studies. In this chapter, socio-economic status (SES) is derived from student responsesregarding parental occupations. In all countriesexamined, students with higher family socio-economicstatuses had higher achievement than did students withlower family socio-economic statuses. Almost allcountries and provinces exhibited small or moderateeffects of family socio-economic status on studentperformance in all three domains (Table 3.3).

Socio-economic impacts on academic achievementare often expressed in the research literature as socio-economic gradients. These gradients are measures of theextent to which inequalities in academic achievementexist within a population (e.g., among students in acountry or in a province) as a result of socio-economicstatus. The slope of the gradient is an indication of theextent of inequality attributable to socio-economicfactors. Steeper gradients indicate a greater relationshipbetween socio-economic status and student performance,

or more inequality; shallower gradients indicate a smallerrelationship between socio-economic background andstudent performance, or less inequality.

Figure 3.1 displays this relationship for readingachievement among G8 countries and Finland.2

Countries with high average reading achievement alsotended to have less variability in scores across socio-economic groups. Canada had both a shallow socio-economic gradient and high scores across socio-economic groups. Germany, on the other hand, had thesteepest gradient, indicating the greatest variation instudent reading performance across socio-economicgroups.

The relationship between socio-economic statusand achievement can also be examined by comparingaverage scores of students from families with the highestsocio-economic statuses with the average scores ofstudents from families with the lowest socio-economicstatuses (Table 3.4). Among the fourteen countriesincluded in this analysis, Canada, along with Finlandand Japan, exhibited far less variation in reading scoresbetween these two groups than did most other countries.Results were similar for mathematics and scienceachievement. This suggests that achievement scores are

FIGURE 3.1

Socio-economic Gradients of G8 Countries and Finland, READING

Finland

CANADA

Japan

United Kingdom

France Russian FederationGermany

Italy

United States

600

580

560

540

520

500

480

460

440

420

40020 30 40 50 60 70 80

Socio-economic status

Reading achievement

33


more equivalent among students with different socio-economic backgrounds in Canada, than they are in manyother countries.

Within Canada, students from Saskatchewanexhibited less variation in reading, mathematics andscience achievement by family socio-economicbackground than did students in other provinces.Performance of students with mid-to-high socio-economic backgrounds in this province, however, fellbelow that of several other provinces.

Alberta, on the other hand, had the highestachievement scores across all levels of fami

measuring up: the performance of canada’s youth in reading, mathematics and science · 2004. 12....

Documents