How Teaching Matters Bringing the Classroom Back Into Discussions of Teacher Quality

APOLICYINFORMATIONCENTERREPORT

1

CONTENTS

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Chapter One: Proposals to Improve Teacher Quality and WhatWe Know About Their Effectiveness . . . . . . . . . . . . . . . . . . . . . . 10

Chapter Two: A Portrait of America’s Teachers and Their ClassroomPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter Three: Linking Aspects of Teacher Quality to StudentTest Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Chapter Four: Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Appendix: How the Study Was Conducted . . . . . . . . . . . . . . . . . . . 32

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

This report was written by:

Harold WenglinskyEducational Testing Service

The views expressed in this reportare those of the author and do notnecessarily reflect the views of theofficers and trustees of EducationalTesting Service.

Additional copies of this report can beordered for $10.50 (prepaid) from:

Policy Information CenterMail Stop 04-REducational Testing ServiceRosedale RoadPrinceton, NJ 08541-0001(609) 734-5694Internet – pic@ets.org

Copies can also be downloaded from:www.ets.org/research/pic

Copyright © 2000 by the Milken FamilyFoundation and Educational TestingService. All rights reserved. EducationalTesting Service is an Affirmative Action/Equal Opportunity Employer. EducationalTesting Service, ETS, and the ETS logo areregistered trademarks of EducationalTesting Service. The modernized ETSlogo is a trademark of EducationalTesting Service.

October 2000

2

FOREWORD

Improving the quality of teachingin America’s schools continues tobe a central focus of educationalreform. Increasing salaries, chang-ing licensing requirements, andimproving pre-service instructionare just some of the major initia-tives targeted to bringing betterprepared teachers into the class-room. Certainly, there is ampleevidence, including the findingsfrom this report, that well-pre-pared teachers produce moresuccessful learners. Recruitingand preparing high quality teachersmust remain a priority forpolicymakers.

Yet, Harold Wenglinsky, inthis significant report, reminds usthat attention only at the front endof the teacher pipeline is insuffi-cient. Simply put, even if recruit-ing and preparation efforts werewildly successful, today’s studentsdo not have the luxury of waitingfor a new generation of highlyqualified teachers to staff ourschools. For these students, it isimperative that their classroomteachers, today, are as effectiveas possible.

Wenglinsky’s report is anoptimistic one, giving us reason tobelieve that effective professionaldevelopment does make a differ-ence in student achievement. Ofcourse, much of the professionaldevelopment in our schools hasbeen criticized as being superficialand irrelevant. What makes thisreport so important is that it

begins to tease out the quality ofprofessional development and itsrelationship to student learning.Professional development thatattends to student learning ofimportant skills and conceptsappears to matter, and matter forall students in mathematics andscience. Other kinds of profes-sional development do not seem tomatter. Wenglinsky has helped usmove from the question “Shouldwe support in-service professionaldevelopment” to “What kinds ofin-service professional develop-ment should be supported?”

As with many of his pastreports, Wenglinsky capitalizes onthe large-scale survey informationfrom NAEP to give us insight intoclassroom practice and studentlearning. However, such data arelimited in that they don’t provideus a depth of understanding of thephenomena under investigation, inthis case the detailed characteristicsof professional development. But,these data certainly do give usreason to explore these issues withmore intensive methodologies andsmaller samples.

If we are to improve educa-tion, we must avoid the tendencyto rely on simple generalizationsand dichotomies. We need toattend to pre-service and in-serviceissues in improving teacher quality.We need to be discerning in thekinds of professional developmentthat we support. And, we need tocontinue to do research like this

that will illuminate our under-standing of teacher practice andstudent learning.

On behalf of the EducationalTesting Service, I would like toexpress my sincere appreciation tothe Milken Family Foundation forsupporting this work.

Drew GitomerVice President - Research

3

PREFACE

The Milken Family Foundationhas been studying school reformfor over a decade. As we haveconsidered specific reforms asdiverse as early childhood educa-tion, smaller class sizes, parentalinvolvement, and learning technol-ogy, one point became abundantlyand unequivocally clear. Unless achild is taught by competentteachers, the impact of othereducation reforms will be dimin-ished. Simply put, students learnmore from “good” teachers thanfrom “bad” teachers under virtu-ally any set of circumstances. Thatis why the Foundation has devel-oped the Teacher AdvancementProgram (TAP), the goal of whichis to attract, motivate and retainthe high quality educators we needto provide all children with theeducation they need and deserve.

The challenge to those recom-mending policy changes has beento define “good” teachers. In thesimplest sense, good teachers arethose who make students learn,and a better teacher will enable thesame or similar students to learnmore. Professor William Sanders,formerly of the University ofTennessee at Knoxville and nowat the SAS Institute in NorthCarolina, along with a number ofcolleagues, has demonstrated theimpact of good teachers. He sortsteachers according to the achieve-ment of their students in a particu-lar year. He then employs sophisti-cated statistical techniques todemonstrate that, year after year,students taught by the “effective”

teachers (those whose initialstudents demonstrated highachievement) achieved more thanstudents taught by other teachers.

Sanders has shown that the top20% of teachers boosted the scoresof low-achieving students over aone-year period by an average of53 percentile points—39 percen-tile points higher than the 14percentile point gain experiencedby students assigned to the bottom20% of teachers (Sanders &Rivers, 1998). Moreover, studentswho performed equally well insecond grade showed a significantperformance gap three years laterdepending on whether they hadbeen assigned to the most effectiveor ineffective teachers. Fifthgraders assigned to effectiveteachers in third, fourth, and fifthgrades placed on average in the83rd percentile. Those assigned tothe least effective teachers over thesame grades scored in the 29th

percentile—a 54 point differenceby the end of fifth grade (Sandersand Rivers, 1998:12). The Sanderswork and similar research aroundthe country is regularly cited todemonstrate the powerful impactof “good” teachers.

But Sanders leaves us with a“black box” in regard to themeaning of a good teacher. Hiswork tells us that good teachers getgood results for students. But aregood teachers prepared differentlythan others? Are they inherentlysmarter? Do good teachers teachdifferently or use different teachingmethods in their classes? Do

teachers get higher studentachievement when they arerewarded for doing so—financiallyor otherwise?

As Dr. Harold Wenglinskydiscusses in this monograph,researchers have tried to answermost of these questions fordecades. Their findings have beeninconclusive or contradictory,mostly because of deficiencies intheir data or statistical methods.Perhaps the most enduring find-ings in this regard have been thatstudent achievement is higher,ceteris paribus, when teachershave high verbal skills (Ehrenberg& Brewer, 1995; Ballou &Podgursky, 1997) and when theyhave strong knowledge of thediscipline they teach (usuallyindicated by a major or minor inthat field) (Goldhaber & Brewer,1999; Fetler, 1999; Monk, 1994).

What we have not known,until now, is whether good andeffective teachers do things in theirclassrooms differently than lesseffective teachers. Some earliersmall scale qualitative or quantita-tive studies have been suggestive,but Dr. Wenglinsky’s work pre-sented here is the first attempt(using a national database andsophisticated analytical techniques)to answer the question of whethereffective teachers do things differ-ently in their classrooms.

By controlling for someteacher inputs, student characteris-tics, and school measures, Dr.Wenglinsky demonstrates thatcertain types of professional

4

peer experts from both within andoutside of the school. The ultimategoal is for teachers to sign three-yearrenewable contracts. In the shortterm, tenure reviews will be morethorough, and tenure will beawarded after a longer period oftime in the classroom. Initial andcontinuing certification will becomeprimarily performance based.

Ongoing, applied professionalgrowth requires a school-widecommitment and includes allteachers. Outcomes are tied tostate teaching and learning stan-dards, school improvement efforts,and a data-driven analysis ofstudent learning outcomes. Activi-ties occur at the school site duringfrequent professional growthblocks led by the principal andmaster teachers and guided bymentor teachers; they are designedto encourage more collaborationamong professional staff. A man-dated salaried, mentored inductionyear gives new teachers classroomresponsibility with intensivesupport. Ongoing, applied profes-sional growth ensures adequatetime for teachers to meet, reflect,learn, and grow professionally.

In addition, TAP has a fifthprinciple, expanding the supply ofhigh quality teachers. This isachieved by making the initialacademic degree and teachingcertification attainable in fouryears; providing alternative certifi-cation to give beginning teachersas well as mid-career professionalsthe ability to enter teaching asadjuncts or full-time throughassessments and classroom

development prepare teachers touse specific techniques in theirclassrooms that result in higherstudent achievement.

Dr. Wenglinsky finds that onlyone teacher input, majoring orminoring in the subject taught, isassociated with improved academicperformance. This is consistentwith earlier findings. Althoughteachers generally give their profes-sional development activitiesmixed reviews, this study suggeststhat certain components of suchtraining are useful. In particular,professional development inworking with special populations,in higher-order thinking skills formath, and in laboratory skills forscience are associated with betterstudent performance. Thesefindings must be kept in mind asthe professional development plansfor TAP are designed.

The findings regarding profes-sional development are supportedby what works in the classroom.Students whose teachers emphasizehigher-order thinking skills (math)and hands-on learning activities(e.g., lab work in science) outper-form their peers significantly.Also, students who frequently taketests outperform those frequentlyusing on-going forms of assess-ment such as portfolios. Thus Dr.Wenglinsky’s study confirms theFoundation’s view that what goeson in classrooms matters—effec-tive teachers do things differently.

There are four school-relatedprinciples that guide TAP’s effortsto attract, motivate, and retain highquality teachers to K-12 education.

With multiple career paths,TAP offers all teachers the oppor-tunity to advance in the professionwithout having to leave the class-room. Teachers are able to movealong a continuum ranging frominductee to master teacher whereincreased responsibilities, qualifica-tions, professional development,and performance-based account-ability requirements are commen-surate with compensation. Mul-tiple career paths provide forexpanded roles for talented teach-ers as leaders, decision-makers, andmentors at the school site, andopportunities to work in thebroader community.

Market-driven compensationreplaces lock-step salary structuresand provides flexibility to establishsalaries. Pay differentiation is basedon demand (more to those in hard-to-staff fields and schools), demon-strated teacher knowledge andskills, actual teacher performance,increased responsibilities, andstudent performance. This systemprovides increased pay for thosewho do more work and are judgedto be the best.

Performance-based accountabilityis rigorous, tied to compensation,and includes differentiated require-ments based on the teacher’sposition. Teachers are assessedagainst high standards that measuretheir performance in contentknowledge, planning, instruction,assessment, and in producingstudent learning gains. Hiring,advancement, and pay increases arebased on performance reviewsconducted by the principal and

5

demonstration; and allowingoutstanding retired teachers tocontinue working on a part-timebasis as faculty fellows. Expandedteacher job mobility—and, there-fore, increased competition andopportunity for teachers—isachieved through multi-statecredentialing, private pensionplans that make benefits moreportable, and the opportunityfor all teachers to become nation-ally certified at the beginning,middle, and advanced levels ofprofessional practice.

Dr. Wenglinsky’s study helpsus implement a number of theseprinciples. It suggests particulartypes of professional developmentthat can help teachers enhancestudent achievement. It alsoinforms TAP’s assessment modelby suggesting specific classroomactivities that should be evaluated.And it gives weight to our perfor-mance-based compensation systemin that it shows that we can rewardparticular behaviors identified asbeing associated with greaterstudent achievement.

Dr. Wenglisky’s study is thefirst of its kind, and it may not bethe final word. But the MilkenFamily Foundation is pleased tohave sponsored it because it shouldbe a catalyst for further researchand discussion about what makesgood, or even great teachers.

Lewis C. SolmonSenior Vice President andSenior ScholarMilken Family Foundation

This report and the study onwhich it is based received assis-tance from many quarters. Thestudy was funded by a generousgrant from the Milken, FamilyFoundation. Lewis C. Solmon,Senior Vice President and SeniorScholar at Milken, providedinvaluable intellectual guidancethrough all stages of the project,from its initial conception to thecompletion of the report. The datafor the statistical analyses wereprepared by Alfred Rogers. Thereport was reviewed by PaulBarton, Robin Henke, andJacqueline Jones. Richard Coleyprovided the graphics. CarlaCooper provided desktop publish-ing services. Marie Collins was theeditor. Marita Gray designed thecover. Kathleen Benischeck coordi-nated production. This report doesnot, however, necessarily representthe views of those who contributedto its production. Any errors offact or interpretation are theauthor’s responsibility.

ACKNOWLEDGMENTS

6

EXECUTIVE SUMMARY

Improving teacher quality hasbecome the subject of numerouspolicy proposals at the federal andstate levels. In the wake of effortsto raise academic standards acrossthe country, policymakers haveexpressed concern that manyteachers are not prepared to helpstudents meet the new standards.Proposals to remedy this situationhave included increasing teachersalaries to attract better qualifiedteachers, requiring more education,such as a master’s degree, or requir-ing a major in the subject a pro-spective teacher plans to teach.

These proposals tend to focuson improving nonclassroomaspects of teacher quality, oftenneglecting the nature of theteaching and learning that occurswithin the classroom. Most pro-posals either make teaching morelucrative, in the hope of recruitingbetter teachers, or raise the bar forthose going into teaching throughmore stringent licensing require-ments. The assumption theseproposals make is that such effortswill improve teacher quality,resulting in improved studentacademic performance. Unfortu-nately, the empirical evidence onthis contention is extremely mixed.Research has not consistentlydemonstrated a link betweenteacher inputs, such as salaries andeducation levels, and studentoutcomes, such as scores onstandardized tests.

The study presented in thisreport explores another route toimproving teacher quality—namely, improving teachers’classroom practices. Teachers

employ a variety of strategies—from problem sets to small-groupactivities—to encourage studentsto learn. Some of these strategiesare presumably more effective thanothers. Yet little is known aboutwhich strategies are most effective,because large-scale studies relatingclassroom practices to student testscores or other outcomes havealmost never been conducted.Without a stronger research basefor identifying effective classroompractices, policymakers will find itdifficult to develop policies thatencourage these practices.

The current study represents afirst step toward linking classroompractices to student academicperformance. It does so by analyz-ing data from the National Assess-ment of Educational Progress(NAEP). Known as “the Nation’sReport Card,” NAEP is adminis-tered to students across the nationevery year or two in a variety ofsubjects. In addition to standard-ized tests consistent with highacademic standards, the NAEPdatabase includes questionnairessent to students, their teachers, andtheir principals. From these data itis possible to relate various aspectsof teacher quality to student testscores, while taking into accountother potential influences on thesescores, such as class size andstudent social background. For thisstudy, data are examined for twonational samples of students:

� 7,146 eighth graders who tookthe NAEP mathematics assess-ment in 1996

� 7,776 eighth graders who tookthe NAEP science assessmentin 1996

Three types of teacher qualityare measured:

� teacher inputs, such as teachereducation levels and yearsof experience

� classroom practices, such as theuse of small-group instructionor hands-on learning

� professional development,meaning training to supportcertain classroom practices

The study begins by describingthe national picture for teacherinputs, professional development,and classroom practices. Some keyfindings are:

� Among teacher inputs, in bothmath and science, one out ofthree teachers has at least amaster’s degree; three out of fourmajored or minored in thesubject they are teaching; andsix out of 10 have at least 10years of teaching experience.

� In both math and science,approximately half of all teach-ers have received more than twodays of professional develop-ment in the last year.

� In both math and science, themost common topic for profes-sional development is coopera-tive learning.

7

� In math, the least commontopics for professional develop-ment are those dealing withspecial student populations,such as those with limitedEnglish proficiency or specialneeds.

� In science, the least commontopics for professional develop-ment are those dealing withspecial student populationsand those dealing with labora-tory skills.

� In math, teachers are more likelyto prepare students to answerroutine problems than to answerproblems involving new orunique situations.

� Less than one out of four mathteachers engages in hands-onlearning activities with theirstudents, whereas two out ofthree science teachers reportengaging in such activities.

� While nearly all math andscience teachers report testingstudents at least once a month,these tests are more likely toinvolve extended writtenanswers than multiple-choiceresponses.

The study then links theseclassroom practices, professionaldevelopment experiences, andteacher inputs to student academicperformance. It finds:

� Of the teacher inputs, onlymajoring or minoring in therelevant subject is associated

with improved student academicperformance. Students whoseteachers majored or minored inthe subject they are teachingoutperform their peers by about40% of a grade level in bothmath and science.

� In math, students whose teach-ers have received professionaldevelopment in working withspecial populations outperformtheir peers by more than a fullgrade level, and students whoseteachers have received profes-sional development in higher-order thinking skills outperformtheir peers by 40% of a gradelevel.

� In science, students whoseteachers have received profes-sional development in labora-tory skills outperform theirpeers by more than 40% of agrade level.

� In math, students whose teachersemphasize higher-order thinkingskills outperform their peers byabout 40% of a grade level.

� Students whose teachers conducthands-on learning activities out-perform their peers by more than70% of a grade level in math and40% of a grade level in science.

� Students who frequently takepoint-in-time tests outperformthose frequently using on-goingforms of assessment, such asportfolios, by 46% of a gradelevel in math and 92% of agrade level in science.

The last of these findingsshould not be taken to mean thatportfolio assessments should beentirely supplanted by multiple-choice tests. For one, portfoliosmay have important functionsother than monitoring the progressof individual students, such asproviding professional develop-ment for teachers and informationon the progress of entire classes.For another, the tests that benefitstudents are not necessarily of amultiple-choice format. Rather,both tests involving multiple-choice responses and those involv-ing extended written responsesproved effective.

Overall, these findings suggestthat policymakers are correct inemphasizing the importance ofimproving teacher quality as amechanism for improving studentacademic performance. However,these findings indicate that greaterattention needs to be paid toimproving classroom aspects ofteacher quality. In particular,teachers should be encouraged toconvey higher-order thinkingskills, conduct hands-on learningactivities, and rely primarily upontests to monitor student progress.Policymakers can encourage thesepractices by providing rich andsustained professional developmentthat is supportive of these practices,and provided that teachers haveaccess to such professional devel-opment, perhaps by rewardingthem either financially or throughadvanced forms of certification forengaging in these practices.

8

INTRODUCTION

Teacher quality has come to thefore in discussions of educationalreform. During the 1980s andmuch of the 1990s, policymakersfocused on setting high academicstandards for all students inelementary and secondary schools.The focus started changing withthe release of the Hunt Commis-sion report, What Matters Most:Teaching for America’s Future, in1996 (National Commission onTeaching and America’s Future,1996). The report contended that,for students to meet high academicstandards, the quality of theteaching force needed to beimproved. A flurry of proposals,pronouncements, and legislationon improving teacher qualityfollowed. While the approachesvaried somewhat, they had incommon the belief that improvingteacher quality was the nextstep that needed to be taken toimprove education.

The approaches had anotherthing in common: They tendedto stress aspects of teaching thatoccurred outside of the classroom,known as teacher inputs, as theinstrument for improving studentperformance. The nonclassroomfactors manipulated by theseproposals included financialaspects of teaching (such as sala-ries, tax abatements, or bonuses forhigh performance) and the qualifi-cations of teachers (such as requir-ing teachers to pass a licensureexamination or encouragingteachers to obtain a master’sdegree). In manipulating teacherinputs, policymakers assumed that

there was an empirical link betweenthese inputs and the outputs ofeducation, in particular, studenttest scores and other measures ofacademic performance. Unfortu-nately, this assumption is notnecessarily tenable; research on thelink between teacher inputs andstudent outputs has been inconclu-sive. In addition, the focus onnonclassroom factors neglects theimportant role that the practicesactually occurring in the classroommay play in student learning. Inaddition to being the mechanismthrough which improvements inteacher inputs might translate intohigher student achievement,classroom practice may influencestudent achievement irrespective ofthe finances and qualifications ofthe teachers entering the class-room, and may even influencestudent achievement more stronglythan teacher inputs.

The purpose of the currentstudy is to explore the possibleinfluence of classroom practices onstudent achievement. The studyuses data from the NationalAssessment of Educational Progress(NAEP), a national sample ofstudents and their schools. Fromthis database, it is possible tomeasure student scores on assess-ments of mathematics and science,a comprehensive set of classroompractices, the training teachersreceive that is specifically tailoredto these practices (known asprofessional development), andvarious teacher inputs. In addition,information is available from theNAEP database on other

characteristics of schools andstudents that may potentiallyinfluence test scores, such as classsize. By analyzing these data, thestudy can measure the impact ofclassroom practices, professionaldevelopment, and teacher inputsrelative to one another. It can alsoplace classroom practices in theprocess through which otheraspects of teacher quality affectstudent learning; a flow chart canbe developed that measures theextent to which teacher inputs andprofessional development mayinfluence classroom practices thatmay in turn influence studentacademic performance. The NAEPdata thus make it possible to drawtogether more information aboutthe relationship between classroompractices and student academicperformance than has ever existedbefore for a national sampleof students.

The study finds that whileteacher inputs, professional devel-opment, and classroom practicesall influence student achievement,the greatest role is played byclassroom practices, followed byprofessional development that isspecifically tailored to thoseclassroom practices most condu-cive to the high academic perfor-mance of students. In particular,when teachers make use of hands-on activities to illustrate conceptsin mathematics and science,students perform better on assess-ments in these subjects. Also, whenteachers focus on conveyinghigher-order thinking skills,particularly those involving the

9

development of strategies to solvedifferent types of problems, stu-dents perform better on math-ematics assessments (though nobetter on science assessments). Thestudy also finds that the methodsteachers use to assess studentprogress have a similar impact onachievement in mathematics andscience. Tests given at a particularpoint in time are associated withhigher student performance thanon-going techniques, such asportfolio or project-based assess-ment. Professional developmentactivities in hands-on learning andhigher-order thinking skills are alsoassociated with improved studentperformance. Of the teacher inputsstudied, the only one that makes adifference is the teacher’s major orminor in college. Students dobetter when their teachers major orminor in the subject areas theyteach. Overall, professional devel-opment is three times as importantas teacher inputs in mathematicsand one and a half times as impor-tant in science. Classroom prac-tices are five times as important asteacher inputs in mathematics andfour times as important in science.

Before discussing these find-ings in more detail, however, it isnecessary to provide some back-ground. Chapter One presentssome of the major policies toimprove teacher quality that havebeen proposed by federal and stateofficials. It then summarizes whatis known from prior research aboutteacher inputs, professional devel-opment, and classroom practices,

as well as their impact on studentacademic performance. ChapterTwo presents descriptive informa-tion from NAEP on the prevalenceof various teacher inputs, profes-sional development, and classroompractices in U.S. schools. ChapterThree then uses statistical tech-niques to relate these aspects ofteacher quality to student perfor-mance in mathematics andscience. Chapter Four teasesout some implications of thisstudy for policies on teacherquality, and suggest directionsfor further research.

10

CHAPTER ONE:PROPOSALS TO

IMPROVE TEACHER

QUALITY AND WHAT

WE KNOW ABOUT

THEIR EFFECTIVENESS

Proposals and some actual legisla-tion to improve teacher qualityhave come fast and furious overthe last few years. At the federallevel, President Clinton has givenvisibility to the issue in State of theUnion addresses. In February1999, Secretary of EducationRichard Riley unveiled the ClintonAdministration’s program forimproving teacher quality. It calledfor states to institute three levels oflicensure for teachers. Under thissystem, an initial license to teachwould be granted to prospectiveteachers who could pass writtenexaminations in the subject theyplanned to teach, knowledge ofpedagogy, and basic skills. Withina few years, teachers would beexpected to obtain professionallicenses. These licenses would begranted based upon observationsof classroom performance byfellow teachers. Still later in theircareers, teachers could be evaluatedfor advanced licenses along thelines of the National Board forProfessional Teaching Standards(NBPTS) system, which involves aprotocol for collecting and scoringclassroom observations, portfoliosof student work, and other kindsof information about teachers.In addition to proposing the

three-step licensure system, Rileyproposed a national job bankthrough which school districtscould recruit teachers, a nationalconference on teacher quality, anda national commission on math-ematics and science teaching(Robelen, 1999).

Teacher quality was also thecentral topic of discussion at thethird educational summit inOctober 1999. Prior summits hadfocused on national academicstandards and how to measurestudent progress toward thosestandards. The third summitidentified improving teacherquality as a crucial mechanism forenabling students to meet highstandards, and pointed to somepossible tools for improvingteacher quality. These toolsincluded providing bonuses toteachers in exchange for highacademic performance amongtheir students; offering teachersprofessional development closelylinked to high academic standards;and permitting pathways tolicensure other than through thetraditional undergraduate-leveleducation programs based incolleges and universities (Olson &Hoff, 1999).

This year, the candidates forthe 2000 presidential election haveoffered their own proposals forimproving teacher quality. Al Gorehas called for 60,000 scholarshipsfor students who commit toteaching in schools in particularneed of teachers for at least fouryears; bonuses and training for15,000 mid-career professionalswho want to become teachers; and

“Higher Standards, Higher Pay”grants to help urban and ruraldistricts attract and retain highquality teachers by offering highersalaries and other benefits. GeorgeW. Bush has proposed $2.9 billionin new spending to recruit, hire,and train teachers; tax incentivesto help teachers pay for schoolsupplies and other school-relatedout-of-pocket expenses; andincreased funding for Troops toTeachers, a program to movemembers of the Armed Servicesinto teaching.

Proposals have been proliferat-ing among state officials as well.Governor Gray Davis of Californiahas signed legislation providing taxbenefits to teachers as well asraising their salaries. San Franciscoplans to offer its teachers subsi-dized housing. In Massachusetts,mathematics teachers in schoolswhere more than 30% of thestudents fail the state mathematicsassessment will be required to takea mathematics proficiency test tobe recertified as teachers. In NewYork, Governor George Pataki isoffering financial rewards tostudents in the state college anduniversity system who agree toteach in districts or subjects inwhich teachers are in short supply.Washington State has recentlybecome the 40th state to requirenew teachers to take tests in theirsubject area and in knowledge ofpedagogy; the tests are to beoverseen by a new professionalstandards board. And in Kentucky,legislation was defeated that alsowould have created a state profes-sional standards board, as well as

11

required middle school teachersto demonstrate proficiency in theirsubject area. Instead, the legislationwhich passed expanded summerschool training of middle schoolteachers and established a newsystem for evaluating teachertraining programs (e.g., Bradley,2000; Hoff, 2000).

Despite all of the attentionthat improving teacher quality hasgarnered, the policies that havebeen proposed generally sufferfrom two problems. The proposalseach involve manipulating aparticular teacher input in thehope that changes in the input willautomatically lead to improve-ments in an output—namelystudent academic performance. Oneproblem with this approach is thatthere is little empirical evidence ofa link between improvements inteacher inputs and improvements instudent outputs. A second problemis that, in emphasizing aspects ofteacher quality that occur outsideof the classroom, these proposalsoften ignore practices in theclassroom. Yet changing the natureof teaching and learning in theclassroom may be the most directway to improve student outcomes.

Research on the links betweenteacher inputs and student outputsbegan nearly 35 years ago with thepublication of the Equality ofEducational Opportunity Study of1966, also known as the ColemanReport (Coleman et al., 1966).This study collected informationon a series of teacher inputs, suchas teacher education levels andyears of teaching experience, and aseries of student outputs, such as

scores on standardized tests, for anational sample of students. Thestudy then related the inputs to theoutputs, and found that, in mostinstances, the link between the twowas weak. In the wake of theColeman Report, about 400studies have been conductedrelating teacher inputs to studentoutputs. Their results have beenextremely mixed. Of those studieslinking the number of years ofteaching experience to studentoutcomes, 30% showed a benefi-cial effect. Of those studies linkingthe salaries of teachers to studentoutcomes, 20% showed a benefi-cial effect. And of those studieslinking the education levels ofteachers (master’s degrees asopposed to bachelor’s degrees),10% showed a beneficial effect.Thus, while a substantial numberof studies did find evidence of alink between a given input andstudent outputs, most did not.Not only do the studies disagreeon the efficacy of teacher inputs,but studies that attempt to tabulateand interpret the results of thesestudies come to different conclu-sions. Some suggest that thesestudies provide sufficient evidencethat teacher inputs can be impor-tant; others contend that strongerevidence would be needed tomake that claim (Hanushek,1997; Greenwald, Hedges &Laine, 1996).

One exception to the mixedfindings of input-output researchhas been studies of teacher skills asmeasured by standardized tests.The original Coleman Reportfound a strong link between

teacher scores on a vocabularytest and student academicperformance. More recently,Ferguson (1991) found that whenteachers perform well on basicskills tests, their students do betteras well. And Wright, Horn, andSanders (1997) found that teachertest scores are strongly related toimprovements in student testscores over the course of a year(referred to as the “value added”by the teacher to the student).

The mixed findings of theinput-output studies can beaccounted for in two ways. It maybe that the link between teacherinputs and student test scores issimply weak and mixed; it wouldthus be expected that some studiesmight pick up a little bit of a linkwhile others do not. Or, the mixedfindings may reflect problems instudy design. Other than theColeman Report, most of thesestudies were small in scale, consist-ing of a single state or even a singleschool district. Thus, the mixedfindings may reflect the fact thatcertain inputs may be important incertain places and not others.Another design problem is that thestudies often lack rich measures ofthe output. They may measurestudent outcomes based upon aminimum skills test, for instance,rather than through a standards-based test that requires students todemonstrate mastery of criticalthinking skills. These studies alsooften lack measures of inputs,other than teacher inputs, thatmay have an independent effect onstudent test scores, such as thesocial background of students.

12

And, many of these studies fail totake into account what is referredto as the “multilevel” nature ofinput-output studies. That is, suchstudies relate an input that is atone level of aggregation—theclassroom or its teacher—to anoutput that is at a lower level ofaggregation—each student in eachclassroom. To the extent that eachof these problems occurs in thestudy design, results may vary.

Thus, current policy proposalsseek to improve student academicperformance by tapping aspects ofteacher quality, namely aspectsoutside of the classroom, whenthere is little research support fortheir efficacy. In addition, theseproposals ignore other aspects ofteacher quality, namely aspectswithin the classroom which maybe more promising.

For the most part, researchersof the links between teacherquality and student performancetend to ignore classroom practicesas much as policymakers do. Todate, there is little quantitativeresearch that relates a comprehen-sive list of classroom practices tostudent achievement, let alone alsotaking into account professionaldevelopment and teacher inputs.Nonetheless, prior research doesprovide some useful informationon the issue. There have been ahost of qualitative studies ofclassroom practices that, largelythrough the technique of class-room observation, identify differ-ent sets of practices common toteachers and suggest the effective-ness of the different sets. Therehave also been quantitative

studies of classroom practiceswhich—while they do not relatethese practices to student out-comes—do provide useful infor-mation on their prevalence.Finally, there has been one quanti-tative study that related a set ofclassroom practices to studentperformance in mathematics andscience. While the set of practiceswas rather limited, and otheraspects of teacher quality, such asprofessional development, werenot taken into account, this studydoes provide some suggestiveinformation upon which thecurrent study can build.

Qualitative research on class-room practice has put forwardcertain propositions regardingwhich practices are most effec-tive—propositions that can betested through quantitativeresearch. First, it asserts the impor-tance of teaching higher-orderthinking skills. Very often, teach-ing involves conveying informationto students. Students are expectedto retain this information, andlearn to do so either throughmemorization or by solving prob-lems that are similar to one an-other. Such techniques for knowl-edge acquisition are often referredto as drill and practice, learning byrote, or lower-order thinking skills.In other cases, teaching involvesnot so much conveying informa-tion as conveying understanding.Students learn concepts and thenattempt to apply them to variousproblems, or they solve problemsand then learn the concepts thatunderlie the solutions. This move-ment between the abstract and the

concrete is referred to as “higher-order thinking skills,” or alter-nately as “critical thinking skills.”These skills tend to be conveyed inone of two ways: through applyingconcepts to problems (applica-tions) or by providing examples orconcrete versions of the concept(simulations). In either case,students learn to understand theconcept by putting it in anothercontext. In the case of an applica-tion, this might mean solving aunique problem with which thestudent is unfamiliar. In the case ofa simulation, this might meanexamining a physical representa-tion of a theorem from geometryor engaging in a laboratory exercisethat exemplifies a law from chem-istry. While both lower-order andhigher-order thinking skills un-doubtedly have a role to play inany classroom, much of the quali-tative research asserts that thestudents of teachers who canconvey higher-order thinking skillsas well as lower-order thinkingskills outperform students whoseteachers are only capable of con-veying lower-order thinking skills(McLaughlin & Talbert, 1993).

Second, the qualitative researchsuggests a few teaching methodsthat are conducive to the teachingof higher-order thinking skills.Individualization of instruction isimportant. Students bring differentbodies of knowledge with theminto the classroom. Rather thantreat all students as blank slates,teachers should instruct eachstudent by drawing upon theknowledge and experience thatparticular student already has.

13

Collaborative learning is alsoimportant. Teachers should allowstudents to work together ingroups, and teachers should workcollaboratively with students at anindividual, small group, and wholeclass level to bring out key con-cepts and problem-solvingtechniques, rather than have allideas originate from the teacher.Finally, student progress should beassessed in an on-going fashionrather than at discrete points oftime. Assessment should be nodifferent than the student’s ordi-nary activities for learning. It isimportant not to disrupt thelearning environment with artifi-cial activities, such as tests.

The prevalence of many ofthese practices as well as theprevalence of many nonclassroomaspects of teacher quality has beenmeasured in national surveys ofAmerica’s teachers. A recent surveyof teachers undertaken by the U.S.Department of Education providesa snapshot of the distribution ofteacher inputs across the nation(National Center for EducationStatistics, 1999a). One teacherinput frequently discussed is theeducation level of teachers; manyreform proposals call for greaternumbers of teachers or prospectiveteachers to obtain master’s degrees.The Department of Educationstudy indicates that, nationwide,45% of teachers hold master’sdegrees. Teachers with master’sdegrees are disproportionatelylocated in the Northeast (60% inthis region have master’s degrees),teach high school (55%), have

20 or more years of teachingexperience (62%), and havepredominantly affluent students(57% teach students in schoolswhere less than one in six studentsqualifies for a free or reduced pricelunch). Another frequently dis-cussed input is the undergraduatemajor of the teacher. The Depart-ment of Education study indicatesthat 38% of teachers major in anacademic field, 24% in a specificsubject area in education (such asmathematics education), and theremainder in education not tiedto a specific subject area, whichmeans that a substantial percentageof teachers do not major (orminor) in the subject area theyteach. Ingersoll (1999) estimatedthat one-third of mathematicsteachers and one-fourth of Englishteachers do not major or minorin the subject area they teach.Another input often seen as impor-tant is experience. On average,according to the Department ofEducation study, teachers have 15years of experience.

National surveys also providesome information on the profes-sional development teachersreceive. According to the Depart-ment of Education study (NationalCenter for Education Statistics,1999a), nearly all teachers (99%)receive some professional develop-ment each year. The most com-mon topics of this professionaldevelopment are curriculum andperformance standards (81% ofteachers), educational technology(78%), new teaching methods(77%), and in-depth learning in

the subject area (73%). The leastcommon topic is dealing withlimited-English-proficient andculturally different students (31%).Most professional developmentappears not to be very intensive,typically involving a one-dayworkshop. The only topic forwhich a majority of the teachersbeing trained receive more thaneight hours of professional devel-opment is in-depth learning in therelevant subject area (56%).

Finally, another Department ofEducation study tabulated class-room practices for the nation as awhole (National Center for Educa-tion Statistics, 1999b). It describedfour types of classroom practice:interactions between teachers andstudents, materials and resourcesused in the classroom, the natureof the learning tasks studentsperform in school and at home,and methods for assessing studentprogress. In terms of interactions,the study found that most teachersuse a combination of interactionswith the class as a whole, interac-tions with small groups of stu-dents, and one-on-one interactionswith individual students. Mostteachers also use a combination oflecturing, talking with students,and having students talk amongthemselves. Materials and resourcescommonly used in classroomsconsist of printed materials,concrete materials, and educationaltechnology. Of printed materials,teachers are equally likely to usetextbooks or supplementarymaterials in class. The use oftextbooks predominates in

14

homework, however. Two out ofthree teachers also make use ofworksheets. The use of variousconcrete materials is also quitecommon. Three out of four teach-ers use physical models and otherobjects; nine out of 10 use black-boards or overhead projectors; andmore than half use computers. Interms of learning tasks, the studyfound a mixed picture. Inclasswork, about three out of fiveteachers engage in activities involv-ing higher-order thinking skills.Homework, however, only involveshigher-order activities 13% of thetime; two-thirds of homeworkactivities are drills. Finally, infor-mation on assessing studentprogress reveals that slightlymore than half of all teachers areengaged in some kind of on-goingportfolio assessment (57%).Interestingly, these portfolios,which involve collections ofstudent work over time, ofteninclude point-in-time assess-ments—namely, tests (62%).

While the information fromthese surveys does provide a sketchof classroom practices, professionaldevelopment, and teacher inputs,it does not relate them to studentoutcomes, such as scores onstandardized tests. One exceptionis an analysis of the NationalEducational Longitudinal Study of1988 (NELS:88; National Centerfor Education Statistics, 1996). Toproduce the NELS:88 database, anational sample of students wassurveyed when they were in eighthgrade, then resurveyed in the 10thand 12th grades. The surveys

included assessments in mathemat-ics, science, history, and reading.The students’ teachers, principals,and parents were also surveyed.The analysis related the informa-tion on classroom practicesreported by mathematics andscience teachers to student testscores in those two subjects fornearly 10,000 students, takinginto account various backgroundcharacteristics of the students andother factors that might potentiallyinfluence test scores. The studyfound that students performedbetter in mathematics when theirteachers emphasized higher-orderthinking skills. In science, empha-sizing higher-order thinking skillsdid not make a difference.

While the NELS:88 analysis isthus suggestive of a link betweenclassroom practices and studentacademic performance, it is ham-pered by various shortcomings inthe database. The list of classroompractices in the database isextremely limited; for instanceNELS:88 does not include infor-mation on hands-on learning orthe prevalence of tests in class-rooms. The database also includesvery little professional develop-ment information, and as a result,professional development is notrelated to student test scores in theanalysis. Also, like the input-output studies mentioned earlier,the analysis does not take intoaccount the multilevel nature ofthe data.

The current study seeks toremedy these problems in relatingclassroom practices to student

achievement by applying multi-level techniques to a more recentdatabase with more comprehensiveinformation on classroom prac-tices, the National Assessment ofEducational Progress (NAEP).Before presenting the results ofthe current study, however, it isworthwhile to investigate whatNAEP can add to the othernational surveys of the prevalence ofteacher inputs, professional develop-ment, and classroom practice.

15

CHAPTER TWO:A PORTRAIT OF

AMERICA’S TEACHERS

AND THEIR

CLASSROOM

PRACTICES

The data used in the current studyfor the description of teacherinputs, professional development,and classroom practices, as well asfor linking them to studentachievement in mathematics andscience, come from the NationalAssessment of Educational Progress(NAEP). Known as “the Nation’sReport Card,” NAEP is adminis-tered every year or two to nationalsamples of fourth, eighth, andtwelfth graders in various subjects,including mathematics, reading,science, history, and geography. Inaddition to administering anassessment in the subject area tostudents, NAEP sends question-naires to students, teachers, andschool administrators.1 All of thisinformation is used to measuretrends in student performance overtime and to compare performanceamong subgroups of students, suchas males and females.

For this study, the eighthgraders who took the 1996 assess-ments in mathematics and sciencewere selected for analysis. In total,7,146 eighth graders took the

mathematics assessment and adifferent group of 7,776 eighthgraders took the science assess-ment. The mathematics andscience databases includedstudents’ scores on the assessments,background information about thestudents (such as their socioeco-nomic status) drawn from thestudent questionnaires, andinformation on teacher inputs,professional development, andclassroom practices, as well asother school information (such asclass size) drawn from the teacherquestionnaires. Because it comesfrom a national sample, thisinformation makes it possible to

describe teacher inputs, profes-sional development, and classroompractices, as well as their impact ontest scores in mathematics andscience, for the nation as a whole.It should be kept in mind, though,that because the sample is of eighthgraders, these results pertain to themiddle school level only.

NAEP reveals that manyeighth-grade math and scienceteachers are lacking in certaininputs (Figure 1). One out of threeof these teachers has at least amaster’s degree, which means thatthe remaining two out of three donot.2 Three out of four of theseteachers have majors or minors in

Figure 1Characteristics of Eighth-Grade Mathematics andScience Teachers

Mathematics Teachers

30 40 50 60 70

37

74

61

Science Teachers

30 40 50 60 70

38

73

59

Master’s degreeor higher

Major or minorin subject area

More than 10 yearsteaching experience

Percentage Percentage

1 NAEP surveys students’ teachers in the relevant subject area.

2 NAEP is nationally representative of students, but not necessarily of teachers. Consequently, the percentages reported here should not be under-stood to mean the percentages of teachers with a certain characteristic, but the percentage of students whose teachers have that characteristic.

Source: 1996 National Assessment of Educational Progress in Mathematicsand Science.

16

their subject area (mathematics ormathematics education for math-ematics teachers and science orscience education for scienceteachers), which means that oneout of four does not. Finally, sixout of 10 teachers have more than10 years of experience, whichmeans that four out of 10 lacksuch experience.

NAEP shows that mathematicsand science teachers receive someprofessional development in avariety of topics (Figure 2). NAEPasks mathematics teachers whetherthey have received any professionaldevelopment at all in the last fiveyears in nine topics, and asksscience teachers the same questionfor eleven topics. For mathematicsteachers, the most common formof professional developmentconcerns cooperative learning;seven out of 10 received suchtraining in the last five years. Halfreceived professional developmentin interdisciplinary instruction.Only a minority of math teacherswas exposed to the remainingtopics. Just under one-half receivedinstruction in higher-order think-ing skills; four out of 10 receivedinstruction in classroom manage-ment, performance-based assess-ment, and portfolio assessment;and less than one-third receivedprofessional development in topicsinvolving responsiveness to differ-ences in the student population,namely dealing with culturaldiversity, limited English profi-ciency, and special needs.

By and large, a similar patternis revealed among science teachers.A majority of teachers received

professional development incooperative learning and interdisci-plinary instruction. Less than one-half but more than one-thirdreceived professional developmentin higher-order thinking skills,classroom management, andportfolio assessment. And one-third or less of teachers receivedprofessional development indealing with different studentpopulations. One topic in whichthe experiences of math andscience teachers differ is perfor-mance-based assessment: 38% ofmath teachers received professionaldevelopment on this topic, asopposed to 53% of science teach-ers. Science teachers were alsoasked about other professionaldevelopment topics. NAEP indi-cates that 27% of science teachersreceived professional developmentin laboratory skills and 45% inintegrating science instruction.

These patterns indicate signifi-cant gaps in professional develop-ment among the teaching force.Because teachers are only askedwhether they have received anyprofessional development at all foreach topic, a “yes” answer couldindicate anything from a two-hourseminar to a semester-long course.For most topics, a majority ofteachers did not even reach thisminimum threshold. And for thosethat did, the professional develop-ment they received may indicateonly a superficial acquaintancewith the topic. This possibility isfurther supported by the resultsfrom another question asked ofteachers: How much time did theyspend in all forms of professional

development over the last year?About one-half of mathematicsand science teachers reportedspending more than 15 hours inprofessional development. Thismeans that half of the teachersreceive no more than two daysworth of professional developmenta year.

NAEP asked teachers aboutthree aspects of classroom prac-tices: the knowledge and skills theysought to convey, the activities andmethods they used in class toaccomplish this, and the types ofassessments they used to monitorstudent progress. In mathematics,the most advanced knowledgecovered in eighth grade tends to bealgebra and geometry (Figure 3).Among math teachers, 57% reportcovering some algebra and 24%some geometry. Moreover, teachersare much more likely to conveylower-order skills than higher-orderskills. Four out of five teachersreport spending time solving aseries of routine problems, whereasjust one out of two reports spend-ing time applying concepts to solveunfamiliar or unique problems.

The activities and methods ofmathematics teachers includedassigning homework, working ingroups, using written materialssuch as textbooks or worksheets,having students write aboutmathematics, solving problemsinvolving real world situations,engaging in hands-on activities,and discussing math. Eighthgraders are generally assignedbetween one and 30 minutes ofhomework a night; only 14% oftheir teachers report assigning

17

Figure 2Professional Development Experience of Eighth-Grade Mathematics and ScienceTeachers in the Last Five Years

Science Teachers

10 20 30 40 50 60 70

64

56

47

48

35

53

33

30

14

45

27

Mathematics Teachers

10 20 30 40 50 60 70

71

50

47

43

39

38

30

26

13

NA

NA

Percentage Percentage

Cooperative learning

Interdisciplinary instruction

Higher-orderthinking skills

Classroom management

Portfolio assessment

Performance-basedassessment

Cultural diversity

Special-needsstudents

Limited-English-proficient students

Integratingscience instruction

Laboratories

Type ofExperience

5545>15 hours professionaldevelopment in last year

Source: 1996 National Assessment of Educational Progress in Mathematics and Science.

18

more than a half hour. Groupactivities vary in their frequency.Most teachers report havingstudents discuss mathematics ingroups (86%) and assigning grouppapers (66%). But just 33% reporthaving students solve problemswith partners. Of the writtenmaterials, textbooks are the mostcommonly used (92% of teachers).Two out of three teachers useworksheets. Writing about math-ematics is fairly uncommon. Just36% of teachers report assigningwritten reports about mathematics,and only 30% report assigningproblems that involve writing.Solving real world problems isfairly common; three out of fourteachers report assigning suchproblems at least once a week.Hands-on activities are fairlyuncommon; just 8% of teachersreport working with blocks and26% with objects such as models.Class discussion of math occursat least once a week for 58% ofthe teachers.

Assessments of studentprogress can take a variety of formsin a math class. Students can begiven tests or quizzes at particularpoints in time. The tests thusmeasure retrospectively what thestudents have learned since the lasttest. Such tests can take the formof multiple-choice questions orquestions requiring more extendedwritten answers (referred to as“constructed responses”). Studentscan also be assessed throughon-going work. One approachis to collect a portfolio of studentwork, from worksheets to writtenreports. Another approach is to

Figure 3Classroom Practices of Eighth-Grade MathematicsTeachers

0 20 40 60 80 100

100

92

86

79

74

66

65

58

58

57

52

36

35

34

33

30

29

26

24

14

8

Assess from testsat least once a month

Use textbook atleast once a week

Hold group discussionsat least once a week

Address routine problems

Solve real-world problemsat least once a week

Write a group paperat least once a week

Use worksheet atleast once a week

Talk about math atleast once a week

Assess from written responsesat least once a month

Address algebra

Address unique problems

Write report atleast once a month

Assess from multiple-choicetests at least once a month

Assess from projectsat least once a month

Partner students atleast once a week

Solve writing problemsat least once a week

Assess from portfoliosat least once a month

Work with objects atleast once a week

Address geometry

Assign more than a half hourof homework per day

Work with blocksat least once a week

Percentage

Source: 1996 National Assessment of Educational Progress in Mathematics.

19

assign a project which brings intoplay all of the skills the teacherwishes the student to acquire. Ofall of these forms of assessment,tests are the most common. Nearly100% of teachers administer somesort of test at least once a month.However, most tests do not con-form to the pattern of beingsimply multiple-choice. Only 35%of teachers report using a multiple-choice format at least once amonth. More common are teststhat call for constructed responses,with 58% of teachers administer-ing this type of assessment on atleast a monthly basis. On-goingassessments are the least common,with approximately one-third ofteachers using these to monitorstudent progress.

In science, the knowledge andskills measured by NAEP arephysical science, earth science, lifescience, conveying facts, conveyingconcepts, and teaching problem-solving skills (Figure 4). Of thethree subjects, physical and earthscience are most commonly taught(49% and 41% of teachers, respec-tively, report addressing thesesubjects). Just 19% of teachersaddress life science. Of the threeskills, concepts are most frequentlyconveyed (88% of students’ teach-ers) followed by problem solving(67%) and facts (40%).

Of the activities and methods,homework appears to be the mostcommon, and reading books theleast common. Nearly nine out of10 teachers report assigning morethan a half hour of homework eachweek. Students also frequentlywork in small groups; 68% of

Figure 4Classroom Practices of Eighth-Grade ScienceTeachers

0 20 40 60 80 100

97

88

87

86

81

77

77

68

68

67

65

59

49

48

48

41

40

29

19

18

16

Assess from testsat least once a month

Address concepts in science

Assess from writtenresponses at least once a month

Perform hands-on activitiesat least once a week

Assess from multiple-choicetests at least once a month

Use textbook atleast once a week

Work in groups atleast once a week

Write report atleast once a month

Address problem-solvingin science

Talk about hands-on activitiesat least once a week

Do science demonstrationat least once a week

Address physical science

Assign more than one-half hourof homework per week

Give oral report atleast once a month

Talk about sciencein the news

Address earth science

Address facts in science

Assess from projectsat least once a month

Address life science

Assess from portfoliosat least once a month

Read books atleast once a week

Percentage

Source: 1996 National Assessment of Educational Progress in Science.

20

teachers report this practice. Ofthe written materials, textbooks areused most frequently (77% ofteachers) whereas reading books orother supplementary materialsoccurs less frequently (16% ofteachers report this). Students alsooften give reports—more typicallywritten reports (68% of teachers),but also oral reports (48%). Sixout of 10 teachers report demon-strating a science concept at leastonce a week. Hands-on activitiesoccur with similar frequency, with65% of teachers reporting discuss-ing hands-on activities, and 81%reporting doing hands-on activities.About one-half of teachers reportdiscussing science in the news withtheir students.

In monitoring studentprogress, science teachers placeheavy reliance on tests at discretetime points (97% at least once amonth). These tests most ofteninvolve constructed responses(87% at least once a month), butalso frequently involve multiple-choice questions (77% at least oncea month). Portfolios and projectsare used infrequently. Three out of10 teachers assess students usingprojects, and two out of 10 assessstudents using portfolios.

The classroom practices ofmathematics and science teachersseem to differ primarily in threerespects: Science teachers engagein more hands-on learning, assignmore writing, and make more useof traditional forms of assessmentthan mathematics teachers. Amongmath teachers, 26% report work-ing with objects at least once a

week and only 8% report workingwith blocks at least once a week. Bycontrast, 81% of science teachersreport conducting hands-on activi-ties. Writing is relatively rare inmath class, with 36% of teachersassigning written reports, whereas68% of science teachers assign suchwork. And while both mathematicsand science teachers seem to relymore heavily on tests than onportfolios or projects to assessstudent progress, math teachers aremore likely to use on-going formsof assessment. Assessment fromportfolios is reported by 29% ofmath teachers, as opposed to 18%of science teachers, and assessmentfrom projects is reported by 34% ofmath teachers, as opposed to 29%of science teachers.

These descriptions provide amixed picture of the extent to whichthe kinds of effective classroompractices that the qualitative litera-ture suggests are important areactually being practiced. First, thepractice of higher-order thinkingskills occurs differently in math-ematics and science. In mathemat-ics, higher-order thinking skills seemto be taking a back seat to rotelearning. Teachers are more likely toassign routine problems than toteach students to apply concepts tonew problems. Teachers also spendlittle time making concepts concretethrough simulations; little timeis spent on hands-on activities,although real world problems areoften assigned. Higher-orderthinking skills seem to play a greaterrole in science. Science teachers aremuch more likely to convey

concepts or problem-solvingtechniques than facts. It is alsoquite common for them to makeconcepts concrete through hands-on activities and demonstrations.Second, collaborative learningseems to be prevalent in bothmath and science; students oftenseem to work in groups. Third,learning seems not to be veryindividualized, if professionaldevelopment is any guide. It isvery rare for teachers to receiveprofessional development to helpthem deal with the diverse back-grounds of their student bodies, beit training in issues regardingspecial-needs children or trainingin issues regarding limited-English-proficient students.Finally, assessments tend to bemore of the point-in-time typethan the on-going type. Thus,some effective practices are beingimplemented and others are not.

The foregoing discussion raisesthe question of whether thepractices understood to be effec-tive indeed are so. “Effectiveness”implies that these practices wouldimprove student academic perfor-mance. Are applications, simula-tions, collaboration, individualiza-tion, and on-going assessmentindeed associated with highstudent performance? This is thequestion the next chapter seeks toanswer, by linking teacher inputs,professional development, andclassroom practices to students’scores on assessments of math-ematics and science.

21

CHAPTER THREE:LINKING ASPECTS OF

TEACHER QUALITY TO

STUDENT TEST

SCORES

The purpose of this study is tomap out the ways in which threeaspects of teacher quality—teacherinputs, professional development,and classroom practices—influencestudent academic performance, aswell as one another. This is accom-plished through the statisticaltechnique of multilevel structuralequation modeling (MSEM). Likemost techniques that are referredto as “multivariate,” MSEM makesit possible to isolate the influenceof any given factor on an outcome,taking into account the otherpotential influences. For instance,classroom practices can be relatedto student test scores, taking intoaccount the various teacher inputsand types of professional develop-ment that might also influencethese scores. In addition, MSEMmakes it possible to relate a set offactors to one another, telling astory about how the outcome ofinterest is influenced. For instance,professional development on agiven topic may not simply influ-ence student test performance; itmay also encourage teachers toengage in certain classroom

practices that then translate intoimproved student performance. Insum, MSEM can be used to gener-ate flow charts that indicate howvarious aspects of teacher qualityinfluence one another, andhow these myriad influencesculminate in improved studentacademic performance.3

For this study, five sets ofpotential influences on studentachievement are taken intoaccount: teacher inputs, teacherprofessional development, class-room practices, student socioeco-nomic status, and class size. FromNAEP it is possible to measurethree teacher inputs: the numberof years of teaching experience,whether the teacher has obtaineda master’s degree or higher, andwhether the teacher majored orminored in the relevant subjectarea (math or math education formath and science or scienceeducation for science).

The second set of influencesconsists of aspects of professionaldevelopment. In addition to ameasure of the amount of timespent in any type of professionaldevelopment over the last year, thisset of influences includes six sets ofprofessional development topicsdrawn from nine measures inmathematics, and eight sets ofprofessional development topicsdrawn from eleven measures inscience. The mathematics topics

are classroom management,cooperative learning, working withdifferent student populations(measured from professionaldevelopment in cultural diversity,limited-English- proficient stu-dents, and special- needs students),on-going forms of assessment(measured from professionaldevelopment on performance-based and portfolio assessments),higher-order thinking skills, andinterdisciplinary instruction. Thescience topics include these sixtopics, with the addition oflaboratory skills and integratingscience instruction.

The third set of influencesconsists of classroom practices. Inmathematics, a total of 12 types ofpractice based on the 21 measuresshown in Figures 2 and 3 areincluded. The amount of workconducted in groups is measuredby asking teachers how oftenstudents engage in group discus-sions, write group papers, andwork on problems with partners.The extent to which writtenmaterials are used is measured byasking teachers how often studentsuse textbooks and worksheets. Theextent to which students writeabout math is measured by askingteachers how often students writereports and solve problems thatinvolve writing. The extent towhich students are immersed inconcrete activities is measured by

3 For a more extended discussion of MSEM, see Appendix.

22

asking teachers how often studentssolve real-world problems, workwith blocks, and work with otherphysical objects. The extent towhich students take point-in-timeassessments is measured by askingteachers how often students takeany type of test, as well as howoften they take multiple-choicetests and constructed-responsetests. The extent to which studentsparticipate in on-going assessmentsis measured by asking teachers howoften they utilize portfolios orprojects for assessment purposes.Assigning homework, coveringalgebra and geometry, drillingstudents on routine problems,solving unique problems, andtalking about math in class are allmeasured by asking teachers abouteach of these practices directly. Inscience, 14 kinds of practices basedon 21 measures are included. Theextent to which students usewritten materials is measured byasking teachers how often studentsread textbooks or other books. Theextent to which students givereports is measured by askingteachers how often they assignwritten or oral reports. The extentto which students are immersed inconcrete activities is measured byasking teachers how often they dodemonstrations, talk about hands-on activities, and have students dohands-on activities. The extent towhich students take point-in-timetests is measured by asking teachers

how often students take any typeof tests, as well as how often theytake multiple-choice tests andconstructed-response tests. Theextent to which students partici-pate in on-going assessments ismeasured by asking teachers howoften they utilize portfolios andprojects for assessment purposes.Assigning homework, working ingroups, covering life, earth, andphysical science topics, dealingwith facts, concepts, and problemsolving, and talking about sciencein the news are all measured byasking teachers about each of thesepractices directly.

The fourth and fifth sets ofinfluences do not involve teacherquality. The fourth set—therelative affluence of students, orsocioeconomic status—is measuredby asking students about theeducation levels of their parents, aswell as whether their familiespossess certain resources: newspa-pers, magazines, more than 25books, and an encyclopedia. Thefifth set—one aspect of the level ofresources in the school—is mea-sured by asking teachers how manystudents are in each class.

These five sets of factors arerelated to student academic perfor-mance in three steps. First, theimpact of teacher inputs aboveand beyond class size and studentsocioeconomic status is measured.Second, professional developmentis added to the mix; teacher inputs,

class size, and socioeconomic statusare related to professional develop-ment, which, in turn, is related tostudent academic performance.Finally, classroom practices areincluded. Teacher inputs, class size,and socioeconomic status arerelated to professional develop-ment and classroom practices.Professional development is thenrelated to classroom practices,which are related to studentacademic performance. The storyof teacher quality’s influence onstudent academic performancethus consists of three parts:Teacher inputs influence profes-sional development, professionaldevelopment influences classroompractices, and classroom practicesinfluence student achievement.And all of these influences takeinto account socioeconomic statusand class size, meaning that theimpact of teacher quality is mea-sured above and beyond these non-teacher-quality factors. Theremainder of this chapter presentsthe results of these analyses, first byshowing flow charts of the path-ways through which aspects ofteacher quality influence oneanother and student achievement,excluding from the charts thoseaspects that do not have anyinfluence. Then the size of theinfluence of these aspects ofteacher quality is measured usinggrade levels to indicate whether theinfluence is substantial enough to

23

be of policy interest, and usinganother number to indicate theinfluence of each factor relative tothe others.

The flow chart for mathemat-ics indicates that one teacherinput, two aspects of professionaldevelopment, and two classroompractices have a positive impact onstudent achievement, whereas oneclassroom practice has a negativeimpact (Figure 5). Both non-teacher-quality factors influencemath achievement. More affluentstudents score higher than lessaffluent students. Students insmaller classes outperform studentsin larger classes. Of the threeteacher inputs, only the teacher’smajor plays a role. Studentsperform better when their teachershave majored or minored in thesubject they are teaching. Of thesix professional developmenttopics, two prove influential.Students whose teachers receiveprofessional development inworking with different studentpopulations outperform studentswhose teachers lack professionaldevelopment on this topic. Also,students whose teachers receiveprofessional development inhigher-order thinking skills outper-form students whose teachers lacksuch professional development. Ofthe 12 classroom practices, threehave an influence on mathematicsachievement. Students who fre-quently engage in hands-onlearning seem to outperform thosewho spend less time in this man-

ner. Students who are frequentlyexposed to higher-order thinkingskills outperform those who lacksuch exposure. And students whoparticipate in on-going assess-ment activities actually performworse than other students.

These factors also seem tohave complex interrelationshipswith one another. The amountof time spent in professionaldevelopment in the last year isincluded in the chart because,although it does not directlyinfluence mathematics achieve-ment, it does influence a class-room practice that itself positivelyinfluences student achievement:Teachers with more professionaldevelopment seem to be morelikely to engage in hands-onlearning activities. Socioeconomicstatus and class size influencefactors other than studentachievement. More affluentstudents are less likely to engagein hands-on learning activities,less likely to have teachers whohave received professional devel-opment in working with differentstudent populations, and morelikely to have teachers who havespent less time in professionaldevelopment overall. Studentsin smaller classes are also lesslikely to have teachers who havereceived professional developmentin working with different studentpopulations and more likely tohave teachers who have spentless time in professional develop-ment. On the other hand, teach-

ers who have majored in therelevant subject area tend to spendmore time in professional develop-ment. They are also more likely toconvey higher-order thinkingskills. Professional development insuch skills also seems to influenceclassroom practice; teachers receiv-ing such professional developmentare more likely to engage in hands-on learning and to use on-goingforms of assessment.

In science, one teacher input,one aspect of professional develop-ment, and two classroom practiceshave a positive influence on scienceachievement, whereas one aspectof professional development hasa negative influence (Figure 6).Socioeconomic status and class sizeagain influence science achieve-ment. More affluent students andthose in smaller classes have anedge over other students. As withmathematics, the only teacherinput to make a difference is theteacher’s major. The students ofteachers who majored in science(or science education) outperformthose whose teachers had othermajors. Of the eight professionaldevelopment topics, the only one topositively influence science achieve-ment is laboratory skills; studentsdo better when their teachers havereceived this form of training. Onthe other hand, students do worsewhen their teachers have receivedprofessional development inclassroom management.4

Of the 14 classroom practices,two prove important: students

4 This negative relationship may be due to selection effects. For instance, teachers who are unskilled in managing their classes might be more likelyto be steered toward this type of professional development.

24 Figure 5Links Among Teacher Inputs, Professional Development, and Student Performance in Mathematics

Mathematicsachievement

Hands-onlearning

Higher-orderthinking skills

Assessmentwithouttesting

Professional develop-ment in working with

different studentpopulations

Professional develop-ment in higher-order

thinking skills

Time inprofessional development

Studentsocioeconomic

status

Averageclass size

Teacher majorin subject

Positive relationship

Negative relationship

25

Figure 6Links Among Teacher Inputs, Professional Development, and Student Performance in Science

Positive relationship

Negative relationship

Studentsocioeconomic

status

Averageclass size

Teacher majorin subject

Professionaldevelopment

inlaboratory

Professionaldevelopment in

classroommanagement

Time inprofessionaldevelopment

Hands-onlearning

Assessmentwith

testing

Scienceachievement

26

Table 1Impact of Teacher Inputs, Professional Development, andClassroom Practices on Mathematics Achievement:Grade Levels

the percentage of a grade level astudent’s test score will increasewhen a certain factor is present, andby the relative influence of eachfactor on student performance,measured using a common scale.

All of the factors that influencemathematics and science achieve-ment seem to have an impact thatcan be regarded as substantial interms of grade levels (Tables 1 and2). Students whose teachers majorin the relevant subject area are39% of a grade level ahead of otherstudents both in math and science.Students whose teachers receiveprofessional development in

working with different studentpopulations are 107% of a gradelevel ahead of their peers in math.Students whose teachers receiveprofessional development inhigher-order thinking skills are40% of a grade level ahead ofstudents whose teachers lack suchtraining in mathematics. Studentswhose teachers receive professionaldevelopment in laboratory skillsare 44% of a grade level ahead ofthose whose teachers lack suchtraining in science. Students whoseteachers receive professionaldevelopment in classroom manage-ment are 37% of a grade level

whose teachers conduct hands-onlearning activities and those whoseteachers utilize point-in-timeassessments outperform theirpeers. Both of these practices arethus important in both subjects;hands-on learning is important inmathematics as well, and thefinding of point-in-time assess-ments having a positive influencein science means the same thing asthe finding of on-going assessmentshaving a negative influence inmathematics.

Some of these factors are alsorelated to one another, althoughless so than in math. In science,more affluent students tend toengage in hands-on learning.Teachers majoring in the relevantsubject area tend to spend moretime in professional developmentand are more likely to engage inhands-on learning. As in math,the more time teachers spend inprofessional development, themore time they and their studentsengage in hands-on learning. Andteachers who receive professionaldevelopment in laboratory skillsare more likely to use point-in-time assessments.

Although the flow chartsindicate the important rolesvarious teacher inputs, aspects ofprofessional development, andclassroom practices play in studentlearning, they do not quantifythese roles. The size of the impactof each aspect of teacher qualitycan be measured in two ways: by

Aspect of Teacher Quality Grade Level

Major/minor in math/math education 39%

Professional development in working withdifferent student populations 107%

Professional development in higher-orderthinking skills 40%

Hands-on learning 72%

Higher-order thinking skills 39%

Assessment without testing -46%

27

Table 2Impact of Teacher Inputs, Professional Development,and Classroom Practices on Science Achievement:Grade Levels

Aspect of Teacher Quality Grade Level

Major/minor in science/science education 39%

Professional development in laboratoryskills 44%

Professional development in classroommanagement -37%

Hands-on learning 40%

Assessment with testing 92%

Factor Math Science

Student socioeconomic status .76 .75

Average class size .10 .11

Major/minor in subject area .09 .09

Professional development in working with different student populations .21 .00

Professional development in higher- order thinking skills .12 .00

Professional development in laboratory skills N/A .13

Professional development in classroom Management .00 -.13

Hands-on learning .25 .18

Higher-order thinking skills .13 .00

Assessment without testing -.18 .00

Assessment with testing .00 .21

Table 3Impact of Teacher Inputs, Professional Development,and Classroom Practices on Mathematics Achievement:Relative Scores

behind their peers in science.When students are exposed tohands-on learning on a weeklyrather than a monthly basis, theyprove to be 72% of a grade levelahead in mathematics and 40% ofa grade level ahead in science.When students are exposed tohigher-order thinking skills “a lot”rather than “some,” they prove tobe 39% of a grade level ahead inmathematics. Students exposed ona more frequent basis to on-goingtypes of assessment in mathematicslag 46% of a grade level behindthose who are exposed to thesetesting practices on a less frequentbasis. Students exposed to point-in-time assessments in science on amore frequent basis are 92% of agrade level ahead of those exposedto point-in-time assessments on aless frequent basis.

The relative importance ofvarious aspects of teacher quality ismost clearly revealed by examiningmeasures of their impact on testscores using a common scale (Table3). Consistent with prior studies asfar back as the Coleman Report,the data indicate that the mostinfluential single measure is socio-economic status, with a score of.76 in math and .75 in science.However, when added together, theaspects of teacher quality (rows 3through 11 of Table 3) are about asstrong an influence, totaling .98 inmath and .74 in science (negativeinfluences are added as positiveinfluences, meaning that they havea positive impact when the oppo-site phenomenon occurs). Theyfar overshadow class size, which

28

has a score of .10 in math and .11in science. Of the aspects ofteacher quality, classroom practicesare the most important, followedby professional development, withteacher inputs being the leastimportant. In mathematics, class-room practices (rows 8 through 11of Table 3) have a total influence of.56 , followed by the four aspects ofprofessional development whichtotal .33 and teacher inputs at .09.In science, scores for classroompractices total .39, those for profes-sional development total .26, andthose for teacher inputs total .09.

In sum, the MSEM reveals theoverwhelming importance ofteacher quality to student academicperformance. Both in mathematicsand science, various aspects ofteacher quality do have a substan-tial impact on test scores. More-over, it appears that it was impor-tant to include classroom practicesin the study, as these proved tohave a larger impact than any othermeasures of teacher quality. Beforediscussing what these findingsimply, however, they need to beinterpreted in more detail.

29

CHAPTER FOUR:

CONCLUSIONS

The findings of this study can bemost easily interpreted by placingthem in the context of priorqualitative work on classroompractices. That work noted theefficacy of teaching that conveyedhigher-order thinking skills, usedmethods that were collaborativeand individualized, and monitoredstudent progress through on-goingforms of assessment.

This study found strongsupport for the notion that con-veying higher-order thinking skillsleads to improved student perfor-mance. At first glance, the studyseems to suggest that higher-orderthinking skills are important inmath but not science. But themeasures of higher-order thinkingskills used in this study involve theprocess of generalizing throughapplications. Concepts are devel-oped from one set of problems,then applied to help solve a verydifferent set of problems. Yet,another factor included in thestudy, hands-on learning, actuallyrepresents another aspect ofhigher-order thinking skills—namely, concretizing conceptsthrough simulations. As hands-onlearning proved to influence bothmath and science achievement, itwould seem that the usefulness ofapplications of higher-order think-ing skills is particular to math, butthat the usefulness of simulations ofhigher-order thinking skills isgeneral to both subjects. Thisfinding is consistent with theanalysis of NELS:88, which

measured higher-order thinkingskills primarily in terms of general-izing from one set of problems toanother, and found an impact formath but not science. TheNELS:88 analysis did not, how-ever, include hands-on learning.

The current study also foundsome support for the effectivenessof individualizing instruction totake into account the differingknowledge and skills which differ-ent students bring into the class-room. Professional development incultural diversity, teaching studentswith limited English proficiency,and teaching students with specialneeds were all linked to higher testscores in mathematics. In science,however, these topics of profes-sional development did not seemto make a difference.

This study did not find sup-port, however, for the views of thequalitative literature on collabora-tive learning and assessmentpractices, however. With regard tocollaborative learning, the studyrevealed no benefits in eithersubject from working in smallgroups. With regard to assessmentpractices, the study actually foundthe opposite of what the qualita-tive research would expect. Whilethe qualitative work supports theidea that portfolios and projectsare more effective as assessmenttools than point-in-time tests, thisstudy found that students sufferacademically from a lack of point-in-time testing. It would be over-reaching, however, to interpret thisresult as meaning that portfoliosand projects should be taken out of

the classroom and replaced withmultiple-choice tests. The virtue ofpoint-in-time tests is that they arethe only situation in which stu-dents are on their own; their workproduct is to a large extent afunction of their own knowledgeand understanding, rather than amixture of contributions fromthem, their teachers, and theirpeers. Being able to distinguish thework of each student from that ofothers has important ramificationsin that it helps the teacher betterunderstand individual students’strengths and weaknesses andmakes it possible to hold studentsaccountable for their learning. Thisbeing the case, portfolios andprojects may not be adequate astools for the assessment of anindividual student because theycannot as easily distinguish thecontribution of the student fromthat of his or her peers and teacher.But portfolios and projects maystill be important in helpingteachers assess what the class as awhole has learned. In this sense,portfolios and projects may beexcellent tools for professionaldevelopment, helping teachersidentify their own strengths andweaknesses. They may also beexcellent tools for teacher account-ability, serving as one basis forevaluations of their work. Also,while the study found tests to beeffective assessment tools, it didnot find any format to be particu-larly effective. Multiple-choice testsare no more effective than con-structed-response tests, or indeedany other kind of format. Thus,

30

while the study supports the use ofpoint-in-time tests, it does notnecessarily support the notion thatportfolios and projects should notbe used, nor that multiple-choicetests should be used.5

Some of the findings aboutnonclassroom practices are alsoworth noting. The study finds thatprofessional development is closelylinked to classroom practice;various types of professionaldevelopment encourage effectivepractices. In addition, the studysuggests that the more extendedthe professional development, themore it encourages effective class-room practices. And teachers whoare more knowledgeable about thesubject area they teach, as mea-sured by majoring or minoring inthat subject, are also more likely toengage in effective classroompractices. All of these notions aresupported by the qualitativeliterature, which suggests thatteachers with greater mastery oftheir subject and armed with richerand more sustained professionaldevelopment, are better able toteach higher-order thinking skillsand engage in related practices,such as hands-on learning.

The methods employed toarrive at these findings did addressmany of the problems in priorresearch. The study relates a setof classroom practices to student

5 It should also be noted here that because these findings pertain to various kinds of tests, it is not clear whether they pertain primarily to teacher-written tests used for instructional purposes or also include state- or district-mandated tests used for accountability purposes.

6 Such a project is under way for reading using the 1998 NAEP reading assessment. Because of the noncomparability of some measures of teacherquality between the 1996 NAEP math and science assessments and the 1998 NAEP reading assessment, results from the reading assessment arenot included here.

area assessments for all threegrade levels.6

Second, this study is cross-sectional, not longitudinal.Cross-sectional studies collectinformation at a single point-in-time, whereas longitudinalstudies follow a group of studentsacross many years. The disadvan-tage of cross-sectional studies suchas this one is that the outcomesoccur at the same time as thefactors that apparently influencethem, raising the possibility thatthe outcomes influence the factorsrather than the other way around.Perhaps students who do well ontests tend to gravitate to classeswith more hands-on learning, forexample. Also, without a priormeasure of student test scores,eighth-grade scores may representthe cumulative impact of schooland family influences over thechild’s lifetime. If this is the case,then many practices of eighth-grade teachers that influencestudent test scores might appearnot to, because they are overshad-owed by the impact of the prac-tices of earlier teachers on students’test scores. But in this case, theclassroom practices of eighth-gradeteachers that do appear to influ-ence student test scores are all themore influential, because theyemerge on top of the influenceof the classroom practices of earlier

achievement, a more comprehen-sive set than has previously beenavailable. For the first time, thisanalysis includes professionaldevelopment and teacher inputsalong with these practices, showinghow professional development andteacher inputs can both contributeto student performance and toeffective classroom practices; itaccomplishes this with a nationalsample, rather than the smallsamples that are typical in input-output research; it includes asophisticated measure of studentoutcomes—scores on the stan-dards-based NAEP tests in mathand science; it takes into accountcharacteristics of students andschools, such as student socioeco-nomic status; and it employsmultilevel methods to handlethe multiple levels of analysisin the data.

These methodologicalstrengths of the study notwith-standing, it has several shortcom-ings that leave ample room forfurther research. First, this studycovers students at only one gradelevel and in two subjects. Eighthgraders in math and science areanalyzed. Results might differ atother grade levels and in othersubjects. Subsequent researchshould examine the math andscience assessments for fourth andtwelfth graders, and other subject

31

teachers. Subsequent researchshould address these issues byanalyzing a national longitudinalsample. Since none of the existingnational longitudinal databasesincludes a comprehensive set ofclassroom practices, such a studywould have to collect new data.But, given the importance ofclassroom practices, such an effortwould be well worth undertaking.7

Third, more detailed informa-tion on classroom practices isneeded. The 1996 NAEP databasesused in this study include informa-tion on whether professionaldevelopment was received invarious topics, but not on howmuch time has been spent oneach topic. Also, while the data-bases indicate whether there isprofessional development ondealing with special populationsof students, they do not documentwhat teachers do in the classroomto adapt to the differing needs oftheir students. Perhaps subsequentadministrations of NAEP couldinclude more questions at this levelof detail.

Finally, researchers need tomeasure the impact of classroompractices on outcomes other thantests scores. The tests used in thisstudy are a better proxy for studentunderstanding than many othertests would be. They includequestions that tap into higher-order thinking, as indicated bythe fact that test scores are relatedto the classroom practice ofconveying of higher-order thinking

skills. The NAEP tests are alsodesigned to be consistent withnational academic standards,making them useful as measures ofwhether students are meeting thosestandards. Nonetheless, it wouldbe worthwhile to examine otherimportant student outcomes, suchas scores on Advanced PlacementTests, and outcomes that do notinvolve tests at all, such as how farstudents go in school and theiroccupations 10 years out of school.

These methodological short-comings not withstanding, certainpolicy implications can be drawnfrom this study. First, the studyprovides ample evidence that theinterest of policymakers in improv-ing teacher quality is justified.Some policymakers maintain thatthere is little that school systemscan do to improve student aca-demic performance; how studentsperform depends too much onfactors that are outside the purviewof the school, such as socioeco-nomic status. Yet, this studyindicates that one aspect of schools,the quality of their teaching force,does have a major impact onstudent test scores—indeed animpact that is comparable in sizeto that of student socioeconomicstatus. Other policymakers main-tain that schools can make adifference, but primarily throughincreasing the quantity rather thanthe quality of teachers, thusreducing class sizes. Yet, this studyindicates that the potential benefitsto students of smaller class sizes,

while substantial, are far overshad-owed by the potential benefits ofimproved teacher quality. Theaspects of teacher quality measuredhere have an impact seven to 10times as great as that of class size.

But if the focus of policy-makers on teacher quality is justi-fied, their focus on non-classroomaspects of teacher quality is not.Policymakers seeking to improveteacher quality often propose toalter the qualifications or financesof teachers, hoping that suchchanges will result in improvedstudent achievement. This studyfinds, however, that two of thethree teacher inputs measured hereproved unrelated to studentacademic performance. And theone input that did make a differ-ence, whether the teacher majoredor minored in the relevant subject,had only a modest effect, farovershadowed by the effects ofclassroom practices and profes-sional development. Other inputsnot included in this study, such asthe preservice training of teachersor their proficiency in pedagogicalknowledge, as measured by stan-dardized test scores, might verywell make a difference. Nonethe-less, policymakers should stoprelying on inputs that do notmake a difference, and pay greaterattention to the classroom practicesthat do. To improve teacher qualityin ways that will improve studentachievement, then, policymakersneed to find ways to encourageeffective classroom practices.

7 Measuring student test scores longitudinally is often referred to as the “value-added” approach.

32

Another implication of thisstudy is that professional develop-ment is a useful tool for improvingclassroom practices. This studyindicates that the most effectiveclassroom practices involve convey-ing higher-order thinking skillsand engaging in hands-on learningactivities. The study also finds thatteachers who receive rich andsustained professional developmentgenerally, and professional devel-opment geared toward higher-order thinking skills and concreteactivities such as laboratoriesparticularly, are more likely toengage in effective classroompractices. Policymakers could thusimprove teacher quality by provid-ing more opportunities for teachersto receive professional develop-ment. That professional develop-ment should occur over anextended period of time ratherthan being limited to a weekendseminar, and it should cover topicsclosely tied to classroom practices.

Policymakers might also makeuse of more prescriptive mecha-nisms to encourage the teaching ofhigher-order thinking skills andrelated practices. Policymakerscould reform state math andscience standards to make sure thatthey are consistent with the call ofnational math and science stan-dards for greater emphasis onhigher-order thinking skills. Also,provided that teachers receiveadequate professional development,they might be provided withrewards for carrying this emphasisinto the classroom. Financial

bonuses could be provided toteachers, not simply for studenttest scores (a portion of which areinfluenced by factors outsideteachers’ control), but also forputting into practice a curriculumoriented toward effective classroompractices. Advanced certification,such as that of the National Boardof Professional Teaching Standards,could also be offered as a rewardfor effective teaching along thelines described here.

In sum, this study shows notonly that teachers matter most, buthow they most matter. For toolong, policymakers have sought toimprove teacher quality by chang-ing the mix of who goes intoteaching, making teaching moreattractive through higher salaries,requiring more education, orstreamlining recruitment proce-dures. Yet what really matters isnot where teachers come from,but what they do in the classroom.And it is possible to makeimprovements in classroom prac-tices with the current teachingforce, irrespective of educationallevels or other qualifications. Thefirst step is for policymakers tostop scratching the surface ofteaching and learning throughsuperficial policies that manipulateteacher inputs, and instead roll uptheir sleeves and dig into thenature of teaching and learningby influencing what occurs inthe classroom.

33

APPENDIX: HOW

THE STUDY WAS

CONDUCTED

This study makes use of thetechnique of multilevel structuralequation modeling (MSEM). Itsstructural equation modelingaspect involves two components:measurement models that relateconstructs to multiple indicatorsof those constructs, and pathmodels that relate the constructsto one another. The hypothesizedmeasurement and path modelsare tested against a covariancematrix of data on the observedvariables, and the goodness offit between the hypothesizedmodels and the observed data ismeasured through various statis-tics, including goodness-of-fitindices and the root-mean-squarederror of approximation. Themultilevel version of a structuralequation model has separatemeasurement and path models forstudent- and school-level variables,and it tests these models against abetween-school covariance matrixand a within-school covariancematrix that is orthogonal to it.Effect sizes and significance testsare thus estimated in a mannerthat takes into account both levelsof analysis (Muthen, 1994). Onesoftware package that supportsMSEMs is STREAMS. STREAMSacts as a pre- and post-processorfor structural equation modeling(SEM) software, such as AMOS orLISREL, loading in the elaboratesyntax required for an MSEM and

translating the SEM output interms of a multilevel model(Gustafsson & Stahl, 1997).

For this study, three MSEMswere developed in each subject areausing STREAMS. First, teacherinputs were related to student testperformance, taking into accountstudent socioeconomic status andclass size. For both math andscience, these teacher inputs wereyears of experience, a master’sdegree or higher, and a major orminor in the relevant subject area.Second, professional developmentactivities were related to studenttest performance, taking intoaccount student socioeconomicstatus, class size, and those teacherinputs found to be significantlyrelated to student test scores in thefirst model. Student socioeconomicstatus, class size, and teacher inputswere also related to professionaldevelopment. For both subjects,the professional developmentactivities that were measuredincluded cooperative learning,interdisciplinary instruction,higher-order thinking skills,classroom management, portfolioassessment, performance-basedassessment, cultural diversity,special needs students, and limited-English-proficiency students, andthe number of hours of profes-sional development irrespective oftopic. For science, integratingscience instruction and laboratoryskills were included as well. Third,classroom practices were related tostudent test performance, takinginto account student socioeco-nomic status, class size, and those

teacher inputs and professionaldevelopment activities that werefound to be significantly related totest scores. The amount of time inprofessional development was alsoincluded, although it had not beendirectly related to test scores in thesecond model. Student socioeco-nomic status, class size, teacherinputs, and professional develop-ment were also related to class-room practices. The practicesincluded in math were those listedin Figure 3 and the practicesincluded in science were thoselisted in Figure 4.

The results for these six modelswere combined into the pathdiagrams displayed in Figures 5and 6. Relationships to mathemat-ics achievement were drawn fromthe appropriate models (teacherinputs from model 1, professionaldevelopment from model 2, andclassroom practices from model 3).Relationships among variables weredrawn from model 3. The effectsizes expressed in terms of gradelevel increments in ChapterThree were derived from theunstandardized coefficients fromthe appropriate models. Themodels express these coefficients aspoints on the NAEP scale. As thereare roughly 50 points between theaverage fourth and eighth graderon the NAEP scale in each subject,one-fourth of this amount, or12.25 points was treated as a gradelevel. The effect sizes expressed on acommon scale in Chapter Threeare the standardized coefficientsfrom the appropriate models.

34

All models had adequate goodness-of-fit.

The current study makes use ofdata from the National Assessmentof Educational Progress (NAEP),and some words are in order aboutthe pitfalls in using such data. Thetwo most serious potential pitfallsin using NAEP data involvevariability in the measurement ofstudent academic performance andvariability in the sample. Measure-ment variability stems from thefact that only a subset of the itemsin the assessment is administeredto each student. The student’sscore is imputed through a proce-dure known as “conditioning.”Based upon information about thestudent and his or her school, fivepossible scores are developed, andare referred to as “plausible values.”Any analysis of the data needs totake into account the error underly-ing the variability in the plausiblevalues. Sampling variability stemsfrom the fact that NAEP draws astratified, clustered sample. If thedata are treated as if NAEP is asimple random sample, standarderrors will be underestimated,making nonsignificant differencesappear significant (Johnson, 1989;Johnson, Mislevy & Thomas, 1994;Johnson, Rust & Wallace, 1994).

For this study, the solution tothe first problem was to model

measurement variability explicitly.All five plausible values weretreated as manifest variables and alatent variable of student academicperformance was generated fromthe manifest variables, and theircorresponding error terms. Thelevel of error estimated in this wayclosely matches the level of errorcalculated using other methods.The solution to the second prob-lem was to estimate a designeffect and inflate the standarderrors accordingly.

It is worth noting that onecommon misconception about theanalysis of NAEP data is that it isnot possible to conduct analyses atthe individual level. One reasonthat this is a misconception is thatall measures except for studentacademic performance do not useplausible values methodology andconsequently are meaningful for anindividual respondent. And whilethe measures of student academicperformance are not meaningfulfor individuals in absolute terms,the variance in performanceamong individuals can be esti-mated, provided that the errorcaused by measurement variabilityis taken into account.

35

REFERENCES

Ballou, D. & Podgursky, M. (1997). Teacher pay and teacher quality. W. E. Upjohn Institute for EmploymentResearch.

Bradley, A. (2000). Washington state to start testing teachers. Education Week, 19 (28), 24, 32.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L.(1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office.

Ehrenberg, R. G. & Brewer, D. J. (1995). Did teachers’ verbal ability and race matter in the 1960s? Colemanrevisited. Economics of Education Review, 14 (1), 1-21.

Ferguson, R. (1991). Paying for public education: New evidence on how and why money matters. HarvardJournal of Legislation, 28, 465-491.

Fetler, M. (1999). High school staff characteristics and mathematics test results. Education Policy AnalysisArchives, 7 (9).

Goldhaber, D. D. & Brewer, D. J. (1999). Teacher licensing and student achievement. In M. Kanstoroom andC. Finn (Eds.), Better teachers, better schools, (pp. 215-238). Washington, DC: Thomas B. FordhamFoundation.

Greenwald, R., Hedges, L. V. & Laine, R. D. (1996). The effects of school resources on student achievement.Review of Educational Research, 57, 415-435.

Gustafsson, J. E., & Stahl, P. A. (1997). STREAM user’s guide: Version 1.7. Molndal, Sweden: MultivariateWare.

Hanushek, E. A. (1997). Assessing the effects of school resources on student performance: An update. Educa-tional Evaluation and Policy Analysis, 19 (2),141-164.

Hoff, D. J. (2000). Massachusetts to put math teachers to the test. Education Week, 19 (38),16,18.

Ingersoll, R. M. (1999). The problem of underqualified teachers in American secondary schools. EducationalResearcher, 28 (2), 26-37.

Johnson, E. (1989). Considerations and techniques for the analysis of NAEP data. Journal of EducationalStatistics, 14 (4), 303-334.

Johnson, E., Mislevy, R. J., & Thomas, N. (1994). Scaling procedures. In E. Johnson & J. E. Carlson (Eds.),The NAEP 1992 technical report (pp. 241-256). Princeton, NJ: Educational Testing Service.

Johnson, E., Rust, K. & Wallace, C. (1994). Weighting procedures and the estimation of sampling variance. InE. Johnson & J. E. Carlson (Eds.), The NAEP 1992 technical report (pp. 193-239). Princeton, NJ:Educational Testing Service.

36

McLaughlin, M. W. & Talbert, J. E. (1993). Introduction: New visions of teaching. In D. K. Cohen, M. W.McLaughlin, & J. E. Talbert (Eds.), Teaching for understanding (pp. 1-10). San Francisco, CA: Jossey-Bass.

Monk, D. H. (1994). Subject area preparation of secondary mathematics and science teachers and studentachievement. Economics of Education Review, 13 (2), 125-145.

Muthen, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods and Research, 22 (3), 399-420.

National Center for Education Statistics. (1996). High school seniors’ instructional experiences in science andmathematics. Washington, DC: U.S. Government Printing Office.

National Center for Education Statistics. (1999a). Teacher quality: A report on the preparation and qualificationsof public school teachers. Washington, DC: U.S. Government Printing Office.

National Center for Education Statistics. (1999b). What happens in classrooms? Instructional practices in elemen-tary and secondary schools, 1994-95. Washington, DC: U.S. Government Printing Office.

National Commission on Teaching and America’s Future. (1996). What matters most: Teaching for America’sfuture. New York: Author.

Olson, L. & Hoff, D. J. (1999). Teaching tops agenda at summit. Education Week, 19 (6), 1, 20.

Robelen, E. W. (1999). Riley outlines licensure plan for teachers. Education Week, 18 (24), 1, 24.

Sanders, W. L. & Rivers, J. C. (1998). Cumulative and residual effects of teachers on future student academicachievement. In Education Trust, Thinking K-16: Good teaching matters: How well qualified teachers canclose the gap.

Wright, S. P., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom context effects on student achieve-ment: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 11, 57-67.

88502-005803 • RR100M10 • Printed in U.S.A.I.N. 989495

®