hard factsfacing the a policy information perspective · 2016-05-09 · education reform under the...

XXXXX

Educational Testing ServicePrinceton, New Jersey 08541-0001

88502-007983 • CL51M6 • Printed in U.S.A.

I.N. 991472

A POLICY INFORMATION PERSPECTIVE

by Paul E. Barton

Facing theHard Factsin Education Reform

Research DivisionPolicy Information Center

Additional copies of this report can beordered for $10.50 (prepaid) from:

Policy Information CenterMailstop 04-REducational Testing ServiceRosedale RoadPrinceton, NJ 08541-0001(609) 734-5694Internet- [email protected]://www.ets.orgCopies can also be downloaded from www.ets.org/research/pic

Copyright© 2001 by EducationalTesting Service. All rights reserved.Educational Testing Service is anAffirmative Action/Equal OpportunityEmployer. The modernized ETS logo is atrademark of Educational Testing Service.

July 2001

FACING THE HARD FACTS • 1

TABLE OF CONTENTS

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

The Test is Not the Treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Student Behavior: Standards to Increase Achievement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Weak Signals That Academic Achievement is Important . . . . . . . . . . . . . . . . . . . . . . . . . . 15

A Learning Policy Recognizes Sources of Achievement Outside the School . . . . . . . . . . . . 20

Dumb Computers and Smart People . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2 • FACING THE HARD FACTS

PREFACE

As I write this preface, the Congress is movingtowards approval of the Elementary andSecondary Education Act, which will be theprimary mechanism for charting federal involve-ment in our nation’s schools. Among themajor issues being debated is the role ofincreased student testing as a tool for ensuringeducational quality. As an assessmentorganization, Educational Testing Service has avested interest in how these matters play out.Indeed, as an organization that has at the coreof its mission serving as the leader in educationalassessment, we try to do more than simply watchpolicies unfold — we try to provide usefulinformation to policymakers that can help themin their deliberations.

We do this by supporting the kind of policywork that is exemplified in this report by PaulBarton. We don’t undertake this work to selltests. Rather, we do it because we take seriouslyour role in opening up educational opportunityfor all. Policy research that illuminates condi-tions, challenges conventional wisdom, unpacksthe complexity of issues, and provides salientand credible information on politically chargedmatters can play a significant role in helpingpolicymakers make informed decisions.

Facing the Hard Facts is a synthesis of adecade of work and illuminates some very criti-cal issues involved in transforming Americaneducation, issues that are not at the center ofthe current educational debate and developinglegislation. If we pay attention to Barton’s claims,we will make progress on issues that open upopportunities for all students. If we only focus

on the test scores, and not on the alignment ofeducational policies and practices that influ-ence those scores, we will do nothing more thanblame those who have not benefited from thesystem. If there isn’t sufficient order in schoolsand classrooms to ensure a reasonable learningenvironment, then it matters not what curriculaare followed or what textbooks are purchased.If we don’t, as a society, begin to value academicachievement more consistently and encouragepractices that develop social capital for all, thenwe will continue to expand the divide of thehaves and have-nots. Finally, if we do not takeadvantage of the power of technology in allschools, then we will have wasted enormousfinancial resources as we erroneously equateaccess to hardware with access to ideas.

Barton raises a number of key points, buteven these are not an exhaustive list. Schoolfinance and teacher quality are but two addi-tional issues that will continue to challenge ourvalues and assumptions about the importanceof education, the nature of social responsibil-ity and accountability, and fairness.

I hope that you as readers will find thisreport as provocative as I do. I also encourageyou to visit our website at www.ets.org/research/pic, where you can download addi-tional Policy Information Center reports.

Drew H.GitomerSenior Vice President,Statistics and Research

Educational Testing Service


ACKNOWLEDGMENTS

We wish to thank several people who reviewed themanuscript, who made thoughtful, and sometimesextensive, comments. At ETS, the reviewers wereDrew Gitomer, Richard Coley, and HaroldWenglinsky. The report was also reviewed by Mar-garet Goertz, Jack Jennings, and Diane StarkRentner. Any errors of commission or omission arethe responsibility of the author.

Carla Cooper provided desktop publish-ing, Marie Collins was the editor, Marita Graywas the cover designer, and Trina Hess was theproduction manager.


INTRODUCTION

Considerable momentum has been gained ineducation reform under the umbrella of standards-based reform.

There is general agreement that this includes,at a minimum, setting standards for what con-tent should be taught and what performancestudents should be expected to demonstrate,and for creating assessments aligned to thosestandards. The approach seems inherentlylogical, and a general consensus has developedthat this is the way to proceed. I am part ofthat consensus. However, I also believe thatthe formulation is incomplete, and that imple-mentation in many places (but by no meansall) has strayed from the path.

I argue that this standards-based reformmovement is too limited an approach to relyon, and that there is a set of hard facts wemust face if we are to see any really significantimprovement in student achievement. Thenation has set ambitious goals, and the effortto achieve them must be equally ambitious.The analysis presented here does not addressways to restructure schools or schooling,though this important debate, subject as it isto polarization, should continue. It is mybelief that the matters discussed in this report,while they have yet to take center stage, aremostly within the broad center of viewpoints.My intent is to bring information and per-spectives to bear that have not yet been givenserious attention.

Of course, while standards-based reformis the driving force today, with a focus ondefining and measuring outcomes, policymakers and educators do not all view the

current reform movement as standards-basedreform and nothing else. There are efforts invarious places and levels of intensity thatreflect a broader view of education reform.But school standards, accountability, and test-ing have occupied center stage.

It is not surprising that basic change ininstitutions is difficult as societies and culturesstrive to continue to exist, and to recreatethemselves. Education is very much engagedin passing on the civilization to succeedinggenerations, and formal schooling is part ofthat process. As a result, we see a demand forteaching values we fear the younger genera-tion has lost or may lose. We see parents look-ing at what goes on in schools and comparingtheir children’s experiences to the experiencesthey had; they are often suspicious of largechanges. Due to a myriad of conflicting cul-tural attitudes, we see considerable tolerationof disorder in the schoolroom, although com-mon sense tells us it harms learning; boys andgirls will be boys and girls, won’t they? Andnew uniforms for the marching band may wellget more support than making sure every stu-dent has a textbook.

If we truly want succeeding generationsto learn significantly more than past ones, wemust, as a society, in a sense, pull ourselves upby our bootstraps. Knowing this, we examineour shoelaces and other seemingly mundanematters, such as cutting classes, gettingemployers to ask for school transcripts, and


reading to young children in the home. Andthen there are those computers that, so far,have not realized their promise. At any onetime, a few proposals and issues dominate thepolitical and professional discussion. But asthe buzz goes on, there remain those attitudes,practices, and conditions that are so embed-ded in our culture and personal experiencethat they escape our serious attention. Theseare facts that must be faced.

This report does not recommend a lot ofspecific policy prescriptions. It is intended toprovide the understanding necessary to for-mulate those prescriptions. Without anunderstanding of the many preconditions thatimpact significant improvement in academicrigor and student achievement, there is littlebasis for serious consideration of new waysand new means of raising educational success.


They also recognize the role testing shouldplay in our total approach to educationreform. It turns out, though, that in imple-menting reform, many states have somehowcome to view the test itself as the treatment,rather than as a way to monitor whetherreform measures are actually producingresults. At the same time, tests are increasinglybeing used for high-stakes purposes — themeans of holding schools, teachers, andstudents accountable. All this is creatinga backlash in many states that is drivenby students, parents, and the public, andwhich is threatening forward progress in rais-ing student achievement.

There are those who would protest mostall uses of standardized achievement testswhen important consequences are attached tothe results. But when used properly, the righttests can be an important element in raisingstudent achievement. In a recent report, theNational Research Council put it this way:

When tests are used in ways that meetrelevant psychometric, legal, and educationalstandards, students’ scores provide importantinformation that, combined with informationfrom other sources, can lead to decisions thatpromote learning and equality of opportunity.1

There are many considerations involvedin the use of standardized tests. The pointsdiscussed below have to do with two princi-pal aspects. One is that we are using tests tohold students and schools accountable forstandards before building much needed teach-ing capacity. The other is that tests that mea-sure student achievement are being misused.Of course, whole books and thick reports arewritten about tests and their uses. We willconfine this discussion to several importantways testing can go astray through inappro-priate use. We need to heed the warning inthe National Research Council report, citedabove: “When test use is inappropriate... Itcan undermine the quality of education andthe equality of opportunity.”

ALIGNING TESTS TO CURRICULUM

Students should be tested for what theyare taught; that is, tests should be aligned tocurriculum. The standards-based reformmovement that began in the 1980s contem-plated a process that would set challengingcontent standards based on broad agreementas to what students should know and be ableto do. Next, performance standards were to

THE TEST IS NOT THE TREATMENT

People who have committed themselves tothe standards-based reform approach recognize that standardized testing is a veryimportant part of that approach.

1 Heubert, J. P., & Hauser, R. M. (eds.). (1999). High Stakes: Testing for Tracking, Promotion, and Graduation, (NationalResearch Council Report). Washington, DC: National Academy Press.


be set to determine how much of this contentstudents should be held accountable for mas-tering. Finally, after content standards hadbeen translated into curriculum actually in usein the classroom, tests were to be developedto help determine whether students hadlearned what they were supposed to learn.

By now, all states have by and largeestablished the content standards, althoughreviewers have judged them to be uneven inquality with regard to such things as how spe-cific and rigorous they are. Next, most ormany states jumped quickly into administer-ing standardized tests on a large-scale basis,and established passing scores, often with fed-eral help and prodding. But the interveningsteps of changing curriculum, improving theteaching materials, and preparing teachers todeliver new curriculum were often skippedover, or at least not completed.

Despite skipping these vital steps, theresults of these tests frequently have impor-tant consequences. They are placed on indi-vidual student records; they are used to judgethe performance of schools and their teach-ers; and they are the bases for allocatingresources; and sometimes for determiningwhether students pass, fail, go to summerschool, or graduate. As far as the experts ineducational measurement are concerned, thisis an inappropriate use of tests; for those whoare not experts, it simply defies common sense.

We are beginning to get informationbased on careful observation of what ishappening in practice in some uses of large-scale testing. A recent study issued by theCouncil of Chief State School Officers,entitled Alignment of Science and Mathemat-ics Standards in Four States,2 found that “align-ment between assessments and standards var-ied across grade levels, content areas, and stateswithout any discernible pattern.” The Coun-cil also has underway an 11-state study calledthe Survey of the Enacted Curriculum, whichis being carried out in collaboration with theWisconsin Center for Education Research. Arecently issued summary of this project con-cluded that, “even where widely disseminatedand used by teachers, standards do not pro-vide curricula for teaching in classrooms.”3 Ineight of the states studied, the survey foundthat in fourth-grade mathematics and eighth-grade science, “less than half the intersectionsof content topics [being taught in the class-room]... were in common with the assessmentitems found on the state test.”

In other words, we see tests not alignedto the standards, and tests not aligned to thecurriculum. When students and parents seetests that do not reflect what is being taught,they lose confidence in the reform movement.The testing results tell as much about theincompleteness of implementing standards-based reform as they do student, teacher, andschool performance.

2 Webb, N. L. (2000).3 May 2000. Council of Chief State School Officers.


USING MULTIPLE CRITERIA FOR HIGH-STAKES DECISIONS

When making important decisions aboutstudents, teachers, and schools, good testsused appropriately may help, but standard-ized tests should never be the sole basis forsuch decisions. This is one matter that expertsin the testing business agree on, whether thoseexperts are reputable testing companies, edu-cational measurement professionals, orresearchers in the area of testing and psycho-metrics. Standards for test use issued by theAmerican Psychological Association, theAmerican Educational Research Association,and others frown on such a practice. A 1999report issued by the National Academy ofScience,4 had this to say: “No single instru-ment can serve all purposes well. Assessmentshould involve a range of strategies appropri-ate for inferences relevant to individual stu-dents, classrooms, schools, districts, andstates.” The National Research Councilreport, cited earlier, gives this admonition:

A test score, like other sources of information,is not exact. It is an estimate of the student’sunderstanding or mastery at a particulartime. Therefore, high-stakes educationaldecisions should not be made solely or auto-matically on the basis of a single test scorebut should also take other relevant informa-tion into account.

It is understandable that many wouldwant to rely exclusively on test scores. Thereis a desire for criteria external to the classroom— to show objectivity when important deci-sions are made. Test scores are relatively inex-pensive. And they provide a specific number,which gives an aura of precision. But peoplewho know that a single number is not anadequate basis for such decisions stress thathuman judgment is still required. Other cri-teria are also important, such as teacher judg-ment, grades, homework, classroom partici-pation, and tests given by teachers. If weignore the professional judgments of teach-ers, we send a message to those who teach,and those who might want to be teachers, thattheir judgments are not valued or trusted.Good judgment in the classroom will decline,not increase.

If standardized tests are to be used tojudge schools, they must measure the gain inknowledge during a school year, not the abili-ties and knowledge acquired during a lifetime.The tests we now use, with only a few excep-tions, measure what students know and cando at a point in time. If it is a math test at theend of the eighth grade, it captures what thestudent has learned in the prior seven grades,in the family, and in the community. What’smore, it reflects the student’s cognitive devel-opment — the result of nutrition and intel-lectual stimulation provided generally in theearly childhood period. How do we identify

4 Elmore, R., & Rothman, R. (eds.). (1999). Testing, Teaching, and Learning: A Guide for States and School Districts.Washington, DC: National Academy of Science.


the addition to knowledge a school achievedduring the eighth grade? And if we look attest scores of a school’s eighth graders over 10years, how do we factor in how much of thechange, up or down, is due to changingdemographics of the student body and howmuch is due to what the school did?

Data from the National Assessment ofEducational Progress (NAEP) illustrate whata test of gain in knowledge — as opposed to atest of level of knowledge — can tell us aboutschools. In l992 and 1996, of the 37 statestaking the assessment in both years, Mainehad the highest fourth- and eighth-gradeNAEP scores in mathematics, and Arkansashad the lowest. However, the gain in theseNAEP scores — from 1992 when the studentswere in the fourth grade to 1996 when thestudents were in the eighth grade — was 52scale points in both Maine and Arkansas.Arkansas, of course, is one of the poorest statesin the nation in terms of per capita income.Yet, the schools in Arkansas added as muchmath knowledge in those four years asdid Maine. In terms of state testing, onlyTennessee currently uses such a gain-score test-ing system, and William Sanders has pio-neered the development and use of thesevalue-added measures.5

It isn’t either/or in testing; we need toknow both. The test of total knowledge tellshow society as a whole is doing in youthdevelopment; the test of gain in knowledge,

from the beginning to the end of the year, getscloser to revealing how well schools do in add-ing knowledge. From a technical standpoint,measuring gain has its own set of difficulties,which need to be overcome.

The large-scale use of standardized test-ing has become a mainstay of educationreform efforts in the United States, and inholding schools accountable for results. If edu-cation reform will stand or fall on how wellwe use standardized testing, we better take aclose look at what we are doing and get it right.It’s a matter of achieving success, and of beingfair to students, parents, teachers, and the edu-cation enterprise generally.

We argue here for putting standards-basedreform back on the track it started on. Buteven if we do, the reform movement can suc-ceed only if policymakers and educators payattention to the larger societal context andenvironment in which change is to be imple-mented. This larger context is addressed in thenext sections of this report.

5 This analysis is provided in Growth in School, by Paul Barton and Richard Coley, published by Educational TestingService in 1998.


As important as this is, and as correct asparents and students are to be concerned, theseare relatively rare events; 99 percent of allhomicides involving children happen awayfrom school.

What is not rare, is in fact widespreadand is growing — not declining — is studentbehavior that is disruptive to the learningenvironment of the school and of the class-room. This situation has attracted almost nonational attention, other than the campaignof the American Federation of Teachersbegun by the late Albert Shanker, and is notaddressed in the standards-based reformmovement. But ask any teacher and you willbe told how much order and discipline mat-ter to learning.

THE BEHAVIORS

School behavior problems are not uni-form across states, and they vary as well amongchildren from different socioeconomic classes.Exposure to learning disruption also fallsunequally by race and ethnic group. The kindsof behavior that are problematic vary highly,from fighting, to verbal abuse of teachers, tocutting classes. Problem behaviors tend to fallinto two categories: those like cutting classes,that impact the achievement of the student

practicing the behavior, and those that impactthe climate of the entire classroom or school,and thus the achievement of all students.

While nationally, just 14 percent of eighthgraders said, in 1996, that physical conflictswere moderate or serious problems in theirschools, the percentage was higher in indi-vidual states: 44 percent in Hawaii, 38 per-cent in Maryland, and 35 percent in Georgia.According to data from the NAEP mathemat-ics assessment, just nine percent of studentsfelt unsafe or very unsafe in the nation’s schoolsas a whole in 1996, but 20 percent of stu-dents in the District of Columbia, and 14percent in Mississippi, South Carolina, andFlorida, felt this way. Fewer students, just fivepercent, felt unsafe in North Dakota.

As Figure 1 demonstrates, trends track-ing problem behaviors from 1991 to 1997 aregenerally unfavorable. While there was adecline in alcohol use during this period, andno statistically significant changes werereported for some discipline issues, behaviorsthat worsened6 were:

� tardiness� absenteeism or cutting classes� drug use� verbal abuse of teachers� sale of drugs on school grounds

STUDENT BEHAVIOR: STANDARDS TO INCREASE ACHIEVEMENT

For several years now, the focus of attentionon student behavior has been on instances of extreme violence.

6 That is, the change was statistically significant.


For a slightly different time period, 1989to 1995, we know that there was an increasein “violent victimization” of 12- to 19-year-old students from 3.4 to 4.2 percent.7 Thesame group of students also reported anincrease in the presence of street gangs inschools — 28 percent reported their presencein 1995, compared to 15 percent in 1989.However, for Black students the statistic rose

0 10 20 30 40 50 60 70

5067*

4048

3952*

37*27

2036*

1717

1717

1415

1320*

1313

78

57

32

210*

11

1

9

1990 - 1991

1996 - 1997

* Indicates statistically significant difference from 1991.

Student tardiness

Student tobacco use

Student absenteeism/class-cutting

Student alcohol use

Student drug use

Physical conflictsamong students

Teacher absenteeism

Robbery over $10

Verbal abuseof teachers

Vandalism ofschool property

Trespassing

Racial tensions

Student possessionof weapons

Sale of drugson school grounds

Physical abuseof teachers

Teacher alcoholor drug use

Gangs

Percentage

Not surveyed in 1991<.5

FIGURE 1: PERCENTAGE OF PUBLIC SCHOOL PRINCIPALS REPORTING THAT

VARIOUS DISCIPLINE ISSUES WERE SERIOUS OR MODERATE PROBLEMS INTHEIR HIGH SCHOOLS, 1990-91 AND 1996-97

Source: NCES, 1998

from 20 to 35 percent, and for Hispanic stu-dents, from 32 to 50 percent. Gangs at schoolare becoming a widespread phenomenon inthe United States, and their presence is beingfelt unequally.

7 The U.S. Department of Justice’s School Crime Supplement, 1989 and 1995.


IT MATTERS

Some of these behaviors have obviousimpacts on teaching and learning. Class cut-ting, absenteeism, and tardiness reduce learn-ing time. Drugs and alcohol are a drag onhealth, on studying, on attention, and onattendance. Physical conflict and the presenceof gangs cause fear in going to and from schooland at the school itself. And that fear is ris-ing. In 1989, six percent of all 12- to 19-year-old students feared attack or harm at school,and four percent were afraid while going toand from school. By 1995, those figures hadrisen to nine and seven percent, respectively.The proportion who fear harm at school ishigher for Black students: one in six Blackstudents felt fear in suburban schools.8 Fearand learning are not good companions.

As a matter of science, though, little hasbeen done to enable us to say precisely whichbehaviors harm achievement, or how much.An ambitious 1978 study measured problembehaviors, but not achievement. Recently,Harold Wenglinsky of Educational TestingService (ETS�) analyzed the results of a studybegun in 1988 called the National EducationLongitudinal Study. Using that study’s

achievement test in combination with infor-mation regarding student behaviors elicitedby questionnaires answered by students, teach-ers, and parents, Wenglinsky found arelationship between student behaviors andachievement. He summarized the resultsthis way:

The frequency of serious and nonseriousoffenses is negatively related to academicachievement in all four subject areas studied— mathematics, reading, science, and socialstudies... The frequency of drug offenses isnegatively related to academic achievementin mathematics and science, but not in socialstudies and reading.9

Another likely impact of growing disrup-tion in the school environment is its effect onthe supply of people willing to enter and con-tinue in the teaching profession. Many prob-lem behaviors are directed toward teachers. Areview of what research exists on this relation-ship led this author to conclude, in a bookrecently published by The Century Fund,that discipline does make a difference in teach-ers leaving.10

8 Kaufman, P., Chen, X., Choy, S. P., Chandler, K. A., Chapman, C. D., Rand, M. R. & Ringel, C. (1998). Indicators ofSchool Crime and Safety, 1998 (NCES 98-251/NCJ-172215). Washington, DC: U.S. Department of Education and Justice

9 Barton, P., Coley, R. & Wenglinsky, H. (1998). Order in the Classroom: Violence, Discipline, and Student Achievement(ETS Policy Information Report). Princeton: NJ: Educational Testing Service.

10 A Notion at Risk: Preserving Public Education as an Engine for Social Mobility, in a chapter by this author titled “UnequalLearning Environments: Discipline That Works,” edited by Richard Kahlenberg and commissioned and published by TheCentury Fund (formerly the 20th Century Fund) in September of 2000.


A VARIETY OF APPROACHES

In the last couple of decades, a variety ofserious attempts to improve student behaviorshave been directed at the more serious prob-lems of bringing weapons to school, at aggres-sive behaviors more generally, and at more gar-den variety forms of disorder and disruption.Some of these efforts involved evaluations oftheir effectiveness, conducted with varyingdegrees of rigor, and with generally positiveresults. Others used “before and after” com-parisons, or anecdotal evidence of improve-ment, to measure their success. A review ofthese practices, as well as the results of thoseevaluations that exist, are provided in the Cen-tury Fund book, referenced above. Order inthe Classroom, also previously cited, presents astudy of trends in behaviors and differentapproaches to modifying them.

Student behaviors that reduce the effec-tiveness of teaching and impact student learn-ing do not have to be accepted as inevitable.However, to reduce the incidence of thesebehaviors, choices must be made among a widevariety of approaches for which solid scientificevidence of effectiveness is not always avail-able (much like the case in the education fieldgenerally). Experience must be combined withgood judgment. Likely, the best solution is nota simple matter of using a single approach, butthe combination that works best and fits bestwith the culture of the school and community.Examples of these approaches include:

� Statewide “zero tolerance” legislation, as inWest Virginia and Texas.

� Alternative schools. Expelling students is amore frequent means of enforcing zerotolerance policies. However, there are notenough of these “alternative schools” toaccommodate the students who needthem.

� New district level “code of behaviors,” as inCincinnati, Ohio.

� New disciplinary authority for teachers incollective bargaining contracts, as in Min-neapolis, Minnesota.

� Character education, now supported withFederal grants and undergoing evaluationin each state receiving those grants.

� Teaching conflict resolution, as in the Day-ton, Ohio program called the “PositiveAdolescent Choices Training Program.”

� Strict school security measures, such as metaldetectors, restricted school entry, “closedcampuses” that restrict students from leav-ing school, and the presence of lawenforcement personnel in schools. (Suchapproaches were surveyed in 1997 by theNational Center for Education Statistics.)

� Classroom management systems, such as onecalled “Consistency Management andCooperative Discipline.”

There is, of course, an interactionbetween student behavior and the academicenvironment. The brief review providedabove considers how improving studentbehavior can improve teaching and learning.It is likewise true that good teaching and


more challenging subject matter can improvestudent behavior and classroom discipline. Itis a two-way street, and we must go down bothsides of it. Standards of behavior are a vitalpart of standards-based reform, and reportsof behaviors must be viewed against thosestandards. In some states, student behaviorsare factored into accountability systems, andmeasures of school climate are presented onschool report cards. It may be such account-ability that forces us to find ways to improvestudent behaviors and the learning climate inthe school and classroom.


But when it comes down to the signals theindividual student receives from his or herenvironment, how high is the country’s goalfor student learning? That is the question.True, teachers and schools share a responsi-bility to create the desire to learn bypresenting challenging subject matter in theclassroom and by using teaching skills tostimulate interest. But this is not the onlysource of student motivation by far. Motiva-tion comes from a myriad of messages a stu-dent receives — in the family, in the commu-nity, on the job, and in the culture — fromthe portrayal of learning in sitcoms to the waysstudents gain social acceptance from theirpeers. The hard fact is that American cultureis not really a learning culture, at least not inthe K-12 period of life.

SIGNALS FROM PARENTS

Recognizing, of course, that there is a lotof variation among individuals from the samelife circumstances, the shaping of the motiva-tion to excel in school comes from the imme-diate family and environment during child-hood. Generally, striving in school has a lotto do with the signals young children receive.I would argue that the “typical” Americanfamily wants to raise a fairly “typical” Ameri-can child — one who is popular with play-mates and has a lot of interests and activities.

WEAK SIGNALS THAT ACADEMIC ACHIEVEMENT IS IMPORTANT

The quality of education is at the top ofthe list of what concerns Americans these days, particularly when it comes to electingpublic officials.

Yes, do your homework, but concern willdevelop if the parent thinks that the teacher isloading too much on the child. If the mathgrades are a little low, it may well prompt aremark that math was hard for me also. Par-ents have a larger agenda for their children thanstudying: In a pubic opinion poll conductedby Gallup for Phi Delta Kappa Internationalin June of 2000 (The Washington Times,August 23, 2000), 42 percent of parents saidthat school extracurricular activities are asimportant as academic studies. Parents, though,put great emphasis on getting a high schooldiploma; the diploma itself is often valued morethan academic achievement.

Developing student interest in learning iscritical. In the Third International Mathemat-ics and Science Study (TIMSS), 51 percent ofAmerican eighth-grade math teachers said thathaving “uninterested students” limited theirteaching “quite a lot” or “a great deal.” That’sjust the teacher’s perspective, of course, but thepercentages were lower in Germany (43 per-cent) and Japan (37 percent).

Yes, improving education seems to be a hotpolitical issue. Real estate agents will tell youthat schools are a key consideration whenpeople pick out their houses. But the Galluppoll director speaking in Phi Delta KappaInternational said:


The notion the public is dissatisfied with itspublic schools is based on myth instead of fact...The data indicate that public approval of thepublic schools has trended consistently upwardand is near its all-time high... And as demon-strated in every previous poll, the closer peopleget to the public schools [in their neighbor-hoods], the better they like them.

While there is a demand for better schoolquality nationally, parents are likely to get upin arms if they see too much changing in theirschools — especially if those changes upset thepattern of education they experienced and areused to. Thus, if we are to see much move-ment upward in academic achievement, it iscritical that families come to believe that theirschools must raise their academic standardsand that their children must do muchbetter academically.

SIGNALS FROM PEERS

Students feel as though they are beingasked to commit an unnatural act if they areexpected to strive for a goal not appreciatedby their peers. Striving too much, or tooobviously, may actually cause them to be lesspopular with their peers. They are often lessthan admired if they study too much. If theyhave a strong bent in a particular academicarea, they may be thought of as geeks, par-ticularly if they want to talk about thisinstead of the current movie in town. If theyare both academic achievers and not terriblyapt socially, they may receive a worselabel, nerds.

Students who do well in school, but with-out obvious effort, can be socially accepted; it’sOK to be smart. On the other hand,parents can sometimes be heard apologizing fora daughter’s success: “Well, yes, but she has towork hard for it.” The value placed on hardwork varies with individuals and families, andwith the cultures that frequently accompanyrace and ethnicity. For example, Asian Ameri-cans have a reputation for placing a high valueon academic achievement. And according tothe scholarly literature, Black youth are oftenaccused of “acting White” when they workhard.

SIGNALS FROM COLLEGES

Given that there is a high value placed ongoing to college in the United States and col-leges have entry requirements, college must pro-vide some incentive to do well in academics, atleast during high school when students andfamilies realize the time for college is looming.But, looking at the nation as a whole, theincentive to excel in school is not deliveredstrongly from the admissions requirements ofthe postsecondary system. True, for selectivecolleges, doing well is necessary for admission.But it is also true that only one in four 25- to29-year-olds has earned a bachelor’s degree, aproportion that has been a constant over mostof the last quarter of a century, moving up afew percentage points only over the last two orthree years. And it is also true that the vastmajority of postsecondary institutions are notselective, or not very selective. How well youdo in high school may determine where youcan go to college, but it seldom determines


whether you can go. Didn’t learn the basics ofgood English and math in school? Well, threeout of 10 college freshmen take remedialcourses to prepare for real college courses.

SIGNALS FROM EMPLOYERS

How about the incentive to excel inschool in order to get a better paying job? It isa frequently heard message in the UnitedStates that how far you go in education deter-mines how well you do in the employmentworld. The facts clearly support such state-ments. In the workforce, each advance in levelof education is accompanied, on average, withan advance in success in the labor market(as measured by lower unemployment andhigher pay).

Unfortunately, there is a period from age17 or 18 until well into the 20s whenthis does not hold up. This finding is socounterintuitive, and counter to conventionalwisdom, it is hard for many people to acceptas true. Employers who offer what I call adult-type work, with good pay and fringe benefits,are generally loath to hire young people justout of high school, preferring to wait untilthey are well into their 20s — their percep-tion of the age of maturity. Employers whodo hire young people at age 17, 18, or 19,hire them for “youth jobs,” all of which payabout the same and do not require much edu-cation. In actuality, the employment prospectsof high school graduates with above averagegrades and test scores are no better, at these

ages, than those with lower grades and testscores; high school dropouts do as well as highschool graduates. But as these young peopleadvance into their late 20s, the situationbegins to change, and those who excel inschool begin doing better. Employers do valuea high school diploma.

The summer 1999 issue of ETS PolicyNotes11 examines the labor market experienceof youths leaving high school. Figure 2 dem-onstrates the difference grades make two yearsout of high school in terms of employmentand earnings, and the data show very littledifference in terms of the percent of youngpeople employed, and no statistically signifi-cant differences in terms of average monthlyearnings. A national database linking testscores and labor market success from age 19to age 31 also shows little difference in theseearly years, but it shows that the higher per-formers begin to pull ahead at about age 23.

The message for young people is thatperformance in school does make a difference,but they may not see it for a while. Their aca-demic performance will also determine theircollege chances, and college will give thembetter opportunities. But it is likely a hardmessage to get across when high school stu-dents see what happens in the short term tothose who graduate. They know that, by andlarge, employers do not ask to see school tran-scripts.12 The incentive to achieve in highschool based on employment success is notapparent enough to young people with shorttime perspectives.

11 ETS Policy Notes. (1999). “Learn More, Earn More?” Princeton, NJ: Educational Testing Service.12 More are, though, through recent efforts of a business coalition led by the National Alliance of Business.


600 800 1,000 1,200 1,400

$745

$795

GPA of 0 - 1.99

GPA of 3.0 & over

Dollars

GPAs, Female High School Graduates (1992)

GPA of 0 - 1.99

GPA of 2.0 - 2.99

GPA of 3.0 & over

GPA of 0 - 1.99

GPA of 2.0 - 2.99

GPA of 3.0 & over

50 60 70 80 90 100

89

92

94

78

88

78

GPA of 0 - 1.99

GPA of 3.0 & over

$1,252

$1,062

Percentage

GPAs, Male High School Graduates (1992)

GPAs, Female High School Graduates (1992)

GPAs, Male High School Graduates (1992)

Percentage Employed After Two Years (1994)

Average Earnings per Month After Two Years (1994)

FIGURE 2: THE DIFFERENCE GRADES MAKE:EMPLOYMENT AND EARNINGS OF MALE AND FEMALE 1992 HIGH SCHOOL

GRADUATES (NOT ENROLLED IN SCHOOL) TWO YEARS AFTER HIGH SCHOOL

(1994), BY GRADE POINT AVERAGE

Source: NELS:88 Data (National Center for Education Statistics), calculated by theETS Policy Information Center.


Strong signals to achieve are necessary ifwe are to raise academic achievement in pub-lic schools. The signals young people get andrespond to come from parents, peers, teach-ers, counselors, colleges, and employers.Teachers, who want the best for their studentsand are being pushed by standards-basedreform and accountability efforts to raise stu-dent achievement, deliver strong messages totry hard. But schools are not getting a lot ofhelp from the other players in students’ lives.They will not get far if they have to go it alone.

It is true that opinion polls put educa-tion at the top of the list of public concerns,at least in terms of what people want frompolitical leaders. But little of this demandtranslates effectively into improving the qual-ity of classroom teaching, paying teachers pro-fessional wages, or advancing the professionaldevelopment of the existing teaching force. Astrong signal from the public that educationreally does matter would be to take the stepsnecessary to improve teaching and learning.


I am in total agreement with both opinions.But what has emerged along the road toachieving reform is a reluctance to examineall of those factors that determine studentachievement. Our focus is on academic stan-dards in schools. And although this is, in part,as it should be, we must also work on a broaderfront if we are to raise educational achieve-ment significantly. This includes importantfactors beyond school doors.

To a considerable extent, our reluctanceto address important nonacademic factorsstems from a fear that to consider such fac-tors may cause us to lose focus, and that rec-ognition of these factors may provide excusesfor not raising standards and achievement. Infact, it is becoming popular to say “no excuses”— heading teachers, schools, and others whoexamine such things off at the pass. We tendto put considerations of family, community,and economy off-limits in education-reformpolicy discussions. However, we do so at ourperil. The seriousness of our purpose requiresthat we learn to rub our bellies and pat ourheads at the same time.

The research on the role these factorsoutside the school play is clear enough,although it tends to be expressed in research-ese. No matter: Plain common sense tells usthat if parents are important, two are betterthan one; if young children are not exposedto reading, they will not learn to read; ifchildren are not well nourished and kept ingood health, the development of their minds

will be held back; rich parents have moreresources than poor parents; caring neighborsand a sense of community create a positiveenvironment for children to grow up in; andparents who involve themselves in theirchildren’s schools and education contribute tostudent learning.

One way we have examined the role ofnonschool factors in student achievement hasbeen through the relationship between socio-economic status (family income and parentoccupations) and school success. The firstlarge-scale study of this relationship — thenow famous Coleman Report, commissionedby the Federal Government and published in1966 — showed that variations in studentsocioeconomic status was by far the largestexplanation for variations in student testscores. This nonschool factor explained muchmore of these differences than did school fac-tors. The Coleman data has been analyzed andre-analyzed. Subsequently, many other stud-ies have confirmed the strength of such fac-tors in correlations with student test scores.In other writing and research, James Colemanused the concept of “social capital” (definedby David Grissmer, below) to describe the roleof socioeconomic status in student achieve-ment, independent of what happens inthe schoolroom.

It is, of course, not family income or whatthe parent does at work all day that affectsstudent learning. This gross construct, socio-economic status, has not been decomposed

A LEARNING POLICY RECOGNIZES SOURCES

OF LEARNING OUTSIDE THE SCHOOL

There is nationwide consensus about the needto raise educational achievement, and there is widespread agreement that we are gener-ally on the right track in the standard-based reform movement.


into those specific things that do affectstudent learning, the totality of which is alsoevidently correlated with student achieve-ment. We could, of course, pursue a policy ofreducing this wide spread in family incomein the United States, and in doing so we couldexpect to narrow the difference in studentachievement. But while we are a nation com-mitted to equality of opportunity, we are notcommitted to equalizing family income. Infact, income has become more unequal in thepast few decades (and is also more unequalthan in other developed countries). To betterinform a learning policy, we need to identifymore specifically those factors that affect stu-dent development, and separate those that wecan do something about in specific effortsfrom those that are more dependent on largersocietal change.

Experience in one area reassures us that wecan improve learning outcomes with preschoolinterventions. The Head Start program of the1960s has been found effective in evaluationsover several decades. There is strong circumstan-tial evidence that it was the investment in HeadStart and other preschool programs in the 1960sthat reduced the gap in scores between Whiteand minority students in the 1980s.13

The matter of “family factors” in studentachievement was explored in 1992 in a reportissued by the ETS Policy Information Cen-ter.14 The report identified family factors fromthe 1990 NAEP assessments and related them

to achievement scores. For example, Figure 3shows, state by state, the percentage of eighthgraders with three or more types of readingmaterials in the home, which varies from 90percent in North Dakota to 68 percent inCalifornia. The figure also shows students’average NAEP score in mathematics for eachstate. North Dakota is at the top in mathachievement and Louisiana is at the bottom.15

In the Summer 1993 issue of ETS PolicyNotes,16 the five family factors most stronglyrelated to achievement were identified. Thepositive factors were having two parents in thehome, reading more than two pages a day forschool and homework, and having at leastthree types of reading materials in the home.The negative factors were student absentee-ism and excessive television watching. Thisreport showed that 91 percent of the differ-ences among the states in test scores wereassociated with these five factors, consideredtogether. Figure 4 shows how actual stateNAEP scores compare with scores that wouldbe expected based on the five factors listedabove. We see that some higher-scoring states,like New Hampshire and Wyoming, did lesswell than expected based on these familyfactors, and that some lower-scoring south-ern states, such as Texas and Florida, did bet-ter than expected.

In 2000, David Grissmer developed mea-sures of family and social capital and appliedthem to state NAEP achievement scores.17 By

13 An early analysis of this can be found in The Reading Report Card, 1985, Educational Testing Service.14 Barton, P. & Coley, R. (1992). America’s Smallest School: The Family (ETS Policy Information Report). Princeton, NJ:

Educational Testing Service.15 The line (on the right hand side) that runs down the middle of the score chart shows how close the relationship is

between the number of reading materials in the home and NAEP scores.16 ETS Policy Notes. (1993). “Angles on Math Achievement.” Princeton, NJ: Educational Testing Service.

17 See Improving Student Achievement, What State NAEP Scores Tell Us, Rand.


60 65 70 75 80 85 90 220 230 240 250 260 270 280

Average NAEP Math ScorePercent of Students with Three or MoreTypes of Reading Materials in Home

North Dakota

IowaNew Hampshire

NebraskaMontana

MinnesotaPennsylvania

WyomingWisconsin

ConnecticutColorado

IndianaMichigan

New JerseyIdahoOhio

MarylandDelaware

Virginia

IllinoisOregon

West VirginiaGeorgia

Rhode IslandNew YorkOklahoma

AlabamaKentucky

North CarolinaArkansas

District of ColumbiaLouisiana

Virgin IslandsArizona

FloridaNew Mexico

Texas

HawaiiCalifornia

Guam

FIGURE 3: THE PERCENTAGE OF EIGHTH GRADERS WITH THREE OR MORE TYPES OF

READING MATERIALS IN THE HOME AND NAEP MATH PROFICIENCY, 1990

Source: America’s Smallest School: The Family. (1992). (ETS Policy Information Report). Princeton,NJ: Educational Testing Service.


Source: Data are drawn from Ina V.S. Mullis et al. The State of Mathematics Achievement, National Centerfor Education Statistics, Educational Testing Service, June 1991.

160 180 200 220 240 260 280

North DakotaMontana

IowaMinnesotaNebraskaWisconsin

New Hampshire

IdahoOregon

ConnecticutNew Jersey

ColoradoIndiana

ArizonaGeorgia

TexasCaliforniaKentucky

New MexicoWest Virginia

ArkansasFlorida

AlabamaHawaii

North CarolinaLouisiana

District of Columbia

PennsylvaniaMichigan

VirginiaOhio

OklahomaDelawareNew York

IllinoisRhode Island

Maryland

NAEP Math Score (0-500)

Actual Predicted

Wyoming

FIGURE 4: ACTUAL AND PREDICTED NAEP MATHEMATICS SCORE BY STATE, 1990(PREDICTION BASED ON SCHOOL ABSENCE, TV WATCHING, READING AT HOME, READING

MATERIALS IN HOME, AND PRESENCE OF TWO PARENTS IN THE HOME)


family capital, Grissmer means “the charac-teristics within families that create higherachievement;” social capital refers to influenceson achievement “through such things as peereffects, quality of communication and trustamong families in communities, the safety ofneighborhoods, and the presence of commu-nity institutions that support achievement.”Using raw NAEP scores, Grissmer asks andanswers two questions: How much do thesescores differ on an absolute basis, and then,how much do they differ among students fromfamilies that are similar in terms of family andsocial capital. The actual variables used byGrissmer were parental educational levels, fam-ily income, race and ethnicity, family type,family mobility, and two measures of socio-economic status. Using the raw NAEP scores,there was considerable variation among thestates: “The average scores in the highestranked state would be approximately betweenthe 62nd and 67th percentile nationally, whileaverage scores from the lowest state would bearound the 33rd to 38th percentile. This rep-resents a significant variation in test scoresamong students from different states.”

However, when he controlled for familyand social capital characteristics, this differ-ence in test scores shrunk tremendously. Inthe estimates of score differences for studentsfrom similar families, there is a difference fromtop to bottom of only 11 or 12 national per-centile points. Who is near the top and whois near the bottom shifts considerably, as wasthe case in the ETS comparison of actual andpredicted NAEP scores based on familyresources and practices (Figure 4). Some

well-off northern states near the top in theraw scores ended up much lower down in theadjusted ranking, and some southern statesnear the bottom ended up nearer the top. Interms of average state scores, the differencesthat various school approaches made weremuch less significant than differences due tofamily and social capital. These latter variablesaccounted for 75 percent of the differences intest scores. However, the Grissmer study alsoidentifies education policies and practices thatdo matter.

In the ETS Policy Information Report,Growth in School, cited earlier, NAEP data arepresented based on growth in scores ofcohorts of students from the fourth to theeighth grade, and trends in these cohort scoresare compared to trends based on levels ofknowledge of fourth and eighth graders.Across about two and a half decades, level ofknowledge rose in three out of four subjects,but did not rise in any of the four in terms ofthe cohorts, or in the growth in knowledgeover four years of schooling. Therefore, theimprovement stemmed from what was hap-pening before the students took the NAEPassessment in the fourth grade, and likely fromchanges happening outside of school beforethey entered the first grade.

Another example of this differencebetween level and growth is shown in Figure5. As noted earlier, of the 37 states participat-ing in NAEP in both 1992 and 1996, Mainehad the highest math scores and Arkansas hadthe lowest. But in terms of growth, the stu-dents in both states improved by 52 scalepoints. Arkansas is a poor state, but its


students grew as much, over their earlierscores, as students in Maine. We ask, doschools in Maine do a better job in teachingmathematics from fourth grade to eighth gradethan do schools in Arkansas? Further, can weraise student achievement significantly by deal-ing with the nonschool factors involved?

We need a better means of tracking wherechanges in educational achievement are com-ing from. We need to face the hard fact thatimproving educational achievement means at-tending to what happens in American familiesand the effects of our social capital. Withoutdoubt, in the last several decades since NAEPstarted providing measures of educationalprogress, schools have faced an uphill battle interms of trends in social capital. This periodcoincides with what Francis Fukuyama called

The Great Disruption in a book by that name.He concluded that “beginning in about 1965,a large number of indicators that can serve asnegative measures of social capital all startedmoving upward rapidly and at the same time.”18

The purpose of measurement is to helpus find those pressure points at which we canfashion efforts to improve student achieve-ment — including those that occur early inlife and in spheres outside the school. On thebright side, significant efforts are being madein some areas, and federal and state supportfor early childhood and early literacy programshas grown. But education achievement indi-cators have hardly begun to budge. Thesocioeconomic status of young childrenremains as strong as ever a predictor of theirrelative achievement in American society.

Arkansas

Maine

Arkansas

Maine

Arkansas

Maine

0 50 100 150 200 250 300

210

232

262

284

+52

+52

Mathematics Scale Score

Average Score, Fourth Grade, 1992

Average Score, Eighth Grade, 1996

Cohort Gain, Fourth to Eighth Grade

FIGURE 5: AVERAGE NAEP MATHEMATICS SCORES AND COHORT GROWTH,ARKANSAS AND MAINE

Source: NELS:88 Data (National Center for Education Statistics),calculated by the ETS Policy Information Center.

18 Fukuyama, F. (2000). The Great Disruption. New York, NY: Simon & Shuster.


Schools have not ignored technology. Com-puter use is becoming pervasive, and manylook to this advancement as a magic bullet.But what has not been grasped is how muchmore we have to do. The hard fact here is howmuch more we must invest in people and inthe educational content that gets put into themachines.

This story begins in the 1930s, and itwould be fascinating to know where it willend. Konrad Zuse is credited with being thefirst to use the binary system in the creationof a computer in Nazi Germany in 1935 and1936; he was perhaps the first to have the ideaof doing so. Yet, even as computers do evermore amazing things, it remains a fact that allthey can really do is distinguish between 0and 1 — albeit with ever increasing speed. Inthis sense, the computer is a dumb machine,one that can be made by human intelligenceto perform remarkable tasks. That said, dumbmachine or not, the ability of Colossus tobreak the German codes created by ENIGMAreceived a lot of credit in the winning of WorldWar II, and computers continue to be madeto do things that amaze us.

The real possibilities of computers in theclassroom may have been imagined by a num-ber of the early creators of computers, but theywere put in writing in a prophetic way byChristopher Evans in 1979.19 He said, “In

schools, computer teaching — initially for‘drill and practice’ — will begin to spread ascosts spiral downwards.” Schools were indeedearly users of computers, and at some pointthey began arriving at schools in truckloads assome combination of lower costs, obsolescence,and tax laws resulted in massive donations fromcomputer makers. To a considerable extent theywere still pretty dumb machines, lacking thesoftware brains that would enable them to de-liver useful instruction.

And used for drill and practice they were,just as had been delivered earlier by variousversions of programmed learning. Drill andpractice is still a principal use of computers interms of instruction, as is using word process-ing programs to write. According to a recentassessment by Larry Cuban, professor of edu-cation at Stanford University, “When the typeof classroom use is examined, we find thatthese powerful technologies end up being usedmost often for word processing and low-endapplications. And this is after a decade ofincreases in access to computers, Internetcapability, and purchases of software.” Thereare, to be sure, a variety of creative instruc-tional uses now employed in many classrooms,and schools are making investments in soft-ware. One step forward is that we can nowtrack these developments in Education Week’syearly issue of Technology Counts.

DUMB COMPUTERS AND SMART PEOPLE

In a somewhat different category is the mat-ter of the use of computers in the classroom.

19 Evans, C. (1979). The Micro Millennium. New York, NY: First Washington Square Press.


There is little doubt that the computercan be a useful tool for education. The ques-tions before us now are how useful, forexactly what purposes, and what is their rolein relation to human teachers. Often, thedebate about appropriate roles and uses is justpart of the general debate about effectiveinstructional strategies and pedagogies. Howmuch “drill and practice” should there be —whether delivered through traditionalmeans or through a computer? How much“constructivism” should there be in instruc-tional approaches — whether instruction isfrom the teacher or the computer? Increas-ingly, it seems computers can do most any-thing software designers want them to do,except supply the human element that is rec-ognizably important for the motivation tolearn. Each remarkable use, such as Big Bluebeating Karpov, reinforces the view that theycan one day serve well in the classroom.

The extent of the belief that computerscan be made to deliver instruction is appar-ent in the conclusion that the highly regardedHoward Gardner has come to: The ultimatein approaches is one tutor to one student, andthe computer can be used to deliver individu-alized instruction. He puts it this way:

For the first time in human history, it is easyto envision a mass educational environmentwhere instruction can be truly individual.Computers can be programmed so that theypresent information and interactions that areappropriate for each student; moreover, as‘intelligent systems,’ they can recordapproaches that have worked more or less well

for a particular student, and readjust the cur-riculum and assessments accordingly.

Gardner is quick to point out, however,that the human teacher is not replaceable.“Human beings learn, and want to learn,because they want to be like individuals theyadmire, who care for them, and for whom theycare,” he states. The computer does notreplace the teacher on the other end of thelog, but a computer can be a huge help byproviding access to a tutorial.

The problem is not how smart the com-puter can be made to behave, but whether thehuman beings that form the system of school-ing can be smart enough, and motivatedenough, to figure out how to get students touse computers effectively, and how to createaccess to this resource on as routine a basis aswe now have teachers in classrooms. How doessuch a problem get resolved within the pro-fessional, sociocultural, bureaucratic environ-ment that exists within the larger governancesystem? How we will make decisions aboutcomputers is related to how we make otherdifficult decisions — such as how to antici-pate, or not anticipate, a teacher shortage dueto the baby boom echo; how to make salariescompetitive enough to attract capable people;how to get good textbooks; or how to createsafe and orderly learning environments. Theseare all things we don’t do very well. If we cancreate the demand, and provide the resources,computer hardware and software systems seemto be in ready supply. But the hard fact is thatwe must achieve computer use in an institu-tional arrangement that has many difficultiesin making such large transformations.


We have little scientific proof of whatcomputers can accomplish in the classroom.Here again, judgments and deductions ofvarious kinds are necessary. One large-scalestudy of mathematics achievement using theNAEP database found that using computersto teach higher order thinking skills, and pro-viding professional development in classroomcomputer uses to teachers, results in higherstudent achievement scores.20 Such evaluationwill have an important role in the choices tobe made.

PREPARING TEACHERS TO USE

COMPUTERS IN THE CLASSROOM

There is a huge challenge in preparingteachers to use computers in the classroom.Several different measurement efforts discloseinadequate preparation of teachers now inthe workforce.

Schools of education now requireinstruction in technology. But EducationWeek’s 1999 Technology Counts issue cites anumber of people who tell us exactly what isgoing on in these schools. For example, Rob-ert F. McNergney, president of the AmericanAssociation of Colleges of Education, tells usthat colleges are still failing to blend trainingon integrating technology into the curricu-lum. He says, “It’s not happening. It’sextremely difficult to wedge technology train-ing into programs that are already packed.”And, of course, these colleges are under pres-sure to improve instruction across the board.

New competency-based standards for schoolcertification by the National Council forAccreditation of Teacher Education addresscurriculum as a whole as well as technology.

The lack of preparation of teachers hasramifications beyond using technologies nowin the classroom. The software teachers askfor, and the software they purchase, influencewhat the market supplies. While a lot of schoolbuying is centralized, a lot is not. An Educa-tion Week survey found that while 23 stateshave group-purchasing programs, the generalpicture is one of teachers searching for soft-ware they think will help them teach; the ini-tiative rests with individual teachers, and tech-nology providers look to them to find outwhat they want and what will sell. Teacherdevelopment is not just preparing teachers forthe technology now available; what becomesavailable depends on how much capabilityteachers possess.

As standards-based reform becomes ubiq-uitous, teachers increasingly find themselvesup against the blackboard to deliver what isbeing measured on standardized tests. If thecurriculum has actually been changed toreflect new content standards, teachers willneed technology to help them deliver that cur-riculum; and if tests are aligned to the revisedcurriculum, then the technology — if teach-ers find it and use it — will help their stu-dents do well on tests. But in many states,content standards become little more thanspecifications for new tests; there is no sys-tematic effort to change curriculum and

20 Wenglinsky, H. (1998). Does it Compute? The Relationship between Educational Technology and Student Achievement inMathematics (ETS Policy Information Report). Princeton: NJ: Educational Testing Service.


prepare teachers in its use. In such cases, teach-ers may well look for technology that is alignedwith standardized tests, for that is where theywill be judged.

We tend to discuss technology as a sepa-rate matter, not as one piece of the educationpuzzle. But what the discussion above showsis that there can be a disconnect between con-tent standards, curriculum, technology, andtests. When this happens, technology may notbe effective. Technology cannot make up fora standards-based reform effort that has fallenoff track. Teachers might as well go back tothe blackboard, an enduring technology.

More and more information is being col-lected about the availability and use of tech-nology in the classroom — through surveysof the National Center for Education Statis-tics and private efforts, such as those ofEducation Week and the Milken Exchange onEducation Technology. But a lot of key infor-mation is not available. We tend to count howmany machines have been delivered toschools, and how many schools have beenhooked up to the Internet. But we need toknow more: How many of those computersare broken? How many are locked behindlaboratory doors and seldom used? How arethe Internet hook-ups being used in serviceof the curriculum? Perhaps we could devisesome measure of computer-delivered instruc-tional hours, and then move on to identify-ing what kind of instruction these hours areused for. Perhaps it will take NAEP to leadthe way, as it has over the years in other areas.

There seem to be huge possibilities inharnessing technology to serve teaching andlearning. Even with the nation’s attention riv-eted on education, we seem to have extraor-dinary difficulty taking advantage of it. Whileclear and steady progress is being made, therealization of the power of computers remains,in the words of a Charles Shultz character, an“insurmountable opportunity.”


CONCLUSION

This report summarizes a broad spectrum of

research on forces that affect student achievement.

It draws on eight publications of the ETSPolicy Information Center, as well as numer-ous other reports and sources. My concernhas been to illuminate problems that are notbeing addressed, or have been addressedinsufficiently, in the current education debate.I have attempted to glean from these manyefforts and information sources those avenuesto improving student achievement that are notbeing traveled — or at least not being giventhe emphasis they deserve — in the currenteducation reform movement. In conclusion,I leave the following thoughts.

Standards-based reform is in danger ofbecoming simply a testing movement; test-ing itself is not the treatment, but a way offinding out whether new content standards,rigorous curriculum, and teacher preparationare producing results. In a full standards-basedreform effort, testing is just one importantcomponent.

If there is no order in the classroom therewill be no learning. Student behavior prob-lems are getting worse, not better; in the1990s, absenteeism, cutting classes, drug use,verbal abuse of teachers, and the presence ofstreet gangs increased. While widespreadattention has been given to murders at school,disruptive student behaviors have not beenthe focus of attention. Research shows thatsuch behaviors do, in fact, adversely affect

student achievement. There are, however, ef-forts in some places that can be emulated, andthey are described in this report.

While educational improvement sits atthe top of opinion polls, family, student peers,and society send weak signals to students thathigh achievement is important. More thantwo out of five parents think extracurricularactivities are as important as academic stud-ies; students fear that they will be less sociallypopular if they work hard in school; employ-ers of high school graduates do not ask to seeschool transcripts; and many colleges havesuch low standards for admission that theyroutinely provide remedial courses.

The nation’s focus has been largely oneducation policy and schools. But a learningpolicy that encompasses preschool develop-ment and draws on the resources of familyand community is necessary to raise achieve-ment. Research clearly shows that, in addi-tion to school factors, family factors areimportant to learning. We need to better tracksources of incentive for educational achieve-ment and learning, whether in school or out.While significant efforts are being made inmany places, we must elevate such efforts tothe level of attention given high content andperformance standards if we are to enlargeyoung peoples’ capacities and advanceachievement to the high level the publicis demanding.


The great promise of computers is notnear to being realized. While hardware isbecoming widely available, we are not yet put-ting education content into these machinesand we must invest in the preparation of teach-ers to perform the miracles these machinesmay well be capable of.

In identifying these areas where too littleattention has been paid, I do not suggest thatwe lessen our efforts in implementing the stan-dards-based reform agenda; it makes good andcommon sense to make instruction rigorous,set high standards, and develop quality stan-dardized tests, the validity of which has beenestablished for the purposes they are used. Thequality of teacher preparation has drawnnational attention, and important policyapproaches are still being debated. However,drastic changes and considerable resources areneeded, and there is little indication thatthe nation is prepared to do what needs tobe done.

XXXXX

Educational Testing ServicePrinceton, New Jersey 08541-0001

88502-007983 • CL51M6 • Printed in U.S.A.

I.N. 991472

A POLICY INFORMATION PERSPECTIVE

by Paul E. Barton

Facing theHard Factsin Education Reform

Research DivisionPolicy Information Center

hard factsfacing the a policy information perspective · 2016-05-09 · education reform under the...

Documents