do textbooks and tests define a national curriculum in...

Do Textbooks and Tests Define a National Curriculum in Elementary School Mathematics?Author(s): Donald J. Freeman, Therese M. Kuhs, Andrew C. Porter, Robert E. Floden, WilliamH. Schmidt, John R. SchwilleSource: The Elementary School Journal, Vol. 83, No. 5 (May, 1983), pp. 501-513Published by: The University of Chicago PressStable URL: http://www.jstor.org/stable/1001074Accessed: 27/07/2009 10:47

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available athttp://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unlessyou have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and youmay use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained athttp://www.jstor.org/action/showPublisher?publisherCode=ucpress.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printedpage of such transmission.

JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with thescholarly community to preserve their work and the materials they rely upon, and to build a common research platform thatpromotes the discovery and use of these resources. For more information about JSTOR, please contact [email protected].

The University of Chicago Press is collaborating with JSTOR to digitize, preserve and extend access to TheElementary School Journal.

http://www.jstor.org

http://www.jstor.org/stable/1001074?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp

http://www.jstor.org/action/showPublisher?publisherCode=ucpress

Do Textbooks and Tests Define a National Curriculum in

Elementary School Mathematics?

Donald J. Freeman Therese M. Kuhs Andrew C. Porter Robert E. Floden William H. Schmidt John R. Schwille Institutefor Research on Teaching, Michigan State University

The Elementary School Journal Volume 83, Number 5 ? 1983 by The University of Chicago. All rights reserved. 0013-5984/83/8305-0006$0 1.00

In the folklore of education, popular claims often survive with little or no empirical verification. Some claim that a national consensus determines the content covered in

elementary school mathematics textbooks and tests. If so, when teachers "teach to the test" or look to textbooks for guidance in

deciding what to teach, topics included in the national curriculum will constitute the content of classroom instruction. In contrast, substantive differences in content covered in textbooks and tests would call the concept of a national curriculum into

question. These differences would pro- mote diversity in what is taught, rather than consensus, and would give rise to conditions in which standardized tests con-

sistently underestimate student achievement (Porter, Schmidt, Floden, & Freeman 1978).

These concerns highlight the need to examine the claim that a national curriculum determines the content of textbooks and tests in elementary school mathematics. To what extent are the same

topics emphasized in all books and all tests at a given grade level? Is the number of common topics consistent with the claim that there is a national curriculum in

elementary school mathematics? To test the

concept of a national curriculum, content

analyses were conducted of the most fre-

quently used textbooks and standardized tests of fourth-grade mathematics.

Selection of textbooks and tests It is generally recognized that the range of

topics that might be covered in elementary school mathematics is most restricted at the

Kindergarten level and least restricted at

:I, ::

ii:sl:::: rl :ii-&i-i s;a

d "'*;:: bli a

ii ?-

l?i% 5

i 't.

,-""iil - j

::i::-----":' ':'"'"," ;: -? - , ? ??-:- s:f' i;.-:::: :: .. i i

"i:I- L_:_ 8. :i .iI -i?-i-: ::g: : I: -:::::-

;n ::.:_:I:::: - :_:l.::::i: . ::: ::II; ?:\ : -:_ -:::-i : :I::: :-: ?:::;::::::?:- '':- ::?: : :

_-:--,:-j_:i-_ ::iil:i:a:::_ -::-_ :--:--?- :I-:i ::- :i_ : : : : :- :: : :: li:*::: :i:: -::-:::::? -::-: ::: ::: ::?-: :::: ;-;: -::I::- ::

: ::----: ::::: ::-:-:-i??:::-i;_:-:: :::::::;i

: ::::: :: -::::::-- :: r-9? .. Ha -: :::::-.i:ia

i-n r::::l:;: ; C:

-:-- I

?:?: ? ::;::

::::--,

::-:::::;-:

I

THE ELEMENTARY SCHOOL JOURNAL

the upper grades. Within this sequence, fourth grade should be early enough to reflect commonalities in content covered in textbooks and tests if there is a national curriculum, yet it would be late enough for substantive content variations to occur if there is not a consensus on what should be taught.

The commonly administered standardized tests of fourth-grade mathematics selected for analysis were: Metropolitan Achievement Tests (Elementary Level/ Grades 3.5-4.9), Harcourt Brace Jovanovich, Inc., 1978; Stanford Achieve- ment Test (Intermediate Level/Grades 4.5-5.6), Harcourt Brace Jovanovich, Inc., 1973; Comprehensive Tests of Basic Skills (Level I/Grades 2.5-4.9 and Level II/ Grades 4.5-6.9), McGraw-Hill, 1976; and Iowa Test of Basic Skills (Level 1 0/Grade 4), Houghton Mifflin, 1978.

According to the publishers, each of these tests provides a measure of mathematics achievement at the end of fourth grade. However, those schools that use the Comprehensive Tests of Basic Skills (CTBS) for this purpose must decide between Level I or Level II. Thus, both levels of the CTBS were considered, resulting in a total of five tests that were analyzed. The number of items classified ranged from a low of 50 items on the Metropolitan (MAT) to a high of 112 items on the Stanford.

Selection of textbooks was guided by the results of a recent national survey that indicated that the following mathematics textbook series are among the most widely used in Grades 4-6 (Weiss 1978). The fourth-grade editions analyzed were: Mathematics in Our World, Addison-Wesley Publishing Co., 1978; Holt School Mathemat- ics, Holt, Rinehart & Winston, 1978; Mathematics, Houghton Mifflin Co., 1978; and Mathematics Around Us, Scott, Foresman and Co., 1978.

Lessons in each book contain two distinct components: instructional activities directed by the teacher and practice exercises assigned to students. Our analyses of

textbooks were limited to items in the student exercise portions of each lesson. The decision to limit textbook analyses in this way was based on practical considerations as well as the results of a survey conducted by the National Advisory Committee on Mathematics Education (1975), which indicated that a majority of teachers rely primarily on the student exercise portion of each lesson. The total number of exercise problems classified ranged from a low of 4,288 in the Addison-Wesley text to a high of 6,986 in the Houghton Mifflin text.

The taxonomy for content analyses The content analyses of textbooks and tests were guided by a classification manual describing the rules for using a three- dimensional taxonomy of elementary school mathematics developed for this purpose (Kuhs, Schmidt, Porter, Floden, Freeman, & Schwille 1979). The three dimensions of the taxonomy describe the general intent of the item (e.g., conceptual understanding or application), the nature of material presented to students (e.g., fractions or decimals), and the operation the student must perform (e.g., estimate or multiply).

Figure 1 illustrates the flexibility of the taxonomy in describing content at different levels of detail. Specific topics covered on the Stanford Achievement Test are represented by the cells of the classification ma- trix (e.g., three of the 112 Stanford items focus on column addition of multiple-digit numbers). More general topics can also be addressed by summing across cells to obtain margin totals (e.g., seven of the 112 items deal with column addition).

Interrater agreement Each textbook and test was independently analyzed by two raters. All items in each of these sources were described along all three dimensions of the taxonomy. The results for a given pair of raters were then compared, and discrepancies were resolved by a third rater. Although there was some vari-

MAY 1983

502

Operations Conceptual Understanding 1 2 3 4 5 6 7 8 9 1011121314

Identify Equiv. \ \1 Order

Add W/o Carrying

Add with Carrying

Add Columns

Sub.W/o Borrowing

Sub. w/ Borrowing\ \\\\

Multiply

Divide W/o Rein.\ \ Divide w/ Rem.

Combination

Concepts (terms) 2\ \\\ \ \

Properties \xx

Place Value 41

Estimate [ \ J\

Skills

1 2 3 4 5 6 7 8 9 1011 121314

1'

X \ u \\\f \\X\\\

\ u\ \\\\\\\\ \\\^^\\\\\\\\\

\\\\\\\\\\v\\\~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I\\\\\\\

Applications 1 2 3 4 5 6 7 8 9 1011121314

^\\\\\^\\1\

1\ < ^\ I

I N

I~~

^\^\\^Y\\\1^\V^~~L

Nature of Material 1. sing. dig./basic facts 2. sing. & Iult. digit 3. multiple digit

4. no. sen./phrase 5. alg. sen./phrase 6. sing./like frac.

7. unlike frac. 8. mixed no. 9. decimals

w/out m with pictures

10. percents 13. geometry 11. measurement 14. other 12. essn. units of measurement

FIG. 1.-Content analysis of 1978 Stanford Achievement Test, Intermediate Level I. Entries represent the number of items out of the total of 112 on the test for each cell-level topic.


ance among the five raters who participated in the study, interrater agreements were generally high. Interrater correlation coefficients at the cell level of the taxonomy were consistently above .98 for the five standardized tests. The corresponding figures for the Houghton Mifflin and Holt textbooks were .97 and .94, respectively.1

Degree of consensus on what should be taught If the claim that there is a national curriculum is valid, it should be possible to identify a relatively large number of specific topics that are emphasized in all textbooks and standardized tests for a given grade level. As a first step in testing this claim, topics emphasized in textbooks were identified. A cell-level topic was included in the core curriculum for textbooks if it was covered by at least 20 items in all four books.2 Next, the core curriculum defined by tests was identified. A specific topic was included in this set if there was at least one item that dealt with the topic on the MAT, Stanford, Iowa, and either the CTBS-I or CTBS-II. The fact that the two forms of the CTBS are designed for use at different age levels, yet overlap at the end of the fourth grade, prompted the either/or condition in this decision rule.

As portrayed in table 1, only 22 specific content areas satisfied one or both def- initions for core topics. Of these 22 topics, only six were emphasized in all textbooks and tests analyzed. Three topics were emphasized in all books but in no tests. Three other topics were covered in all tests, but they received limited attention (i.e., did not meet the criterion for emphasis) in the books. The other 10 topics were emphasized in all four books, but they appeared in only some of the tests.

The diversity of content covered in the various sources that were analyzed is further illustrated by a consideration of the number of specific content areas covered by a single textbook or test. The total number of cell-level topics covered in at least one source was 385. This number provides

some sense of the total domain of specific topics that might be taught in fourth-grade mathematics. The number of topics within this domain that were covered by a particular test ranged from a low of 38 for the MAT (a test with only 50 items) to a high of 72 for the Stanford. Only nine of the topics tested on a given exam were common to all standardized tests of fourth-grade mathematics (sections 4 and 5 of table 1). The number of specific content areas that were covered by 20 or more items in a book ranged from a low of 42 topics in the Addison-Wesley text to a high of 50 topics in the Scott, Foresman text. Of these, only 19 were represented in all four books (sections 1-4 of table 1). It is important to note, however, that this set of common topics did receive a great deal of attention. Approxi- mately 50%-60% of the more than 4,000 problems in each book focused on these 19 cell-level topics.

As described earlier, only six specific topics were emphasized in all four books and all five tests. Three of these topics in- volved computational skills: addition of multiple-digit numbers with carrying, subtraction of multiple-digit numbers with borrowing, and multiplication of one-digit multipliers and multiple-digit multi- plicands. All other computational skills commonly associated with fourth-grade mathematics were emphasized in some books or tests but not in others. Even though basic multiplication facts accounted for nearly 12% of all problems in the student exercise portions of the Addison- Wesley book, for example, this topic did not serve as the focus of any items on either the CTBS-II or Stanford tests.

The other three topics in the core curriculum defined by both textbooks and tests concerned conceptual understanding: geometric terms, place value concepts, and the concept of a fraction represented by pictorial models. Other topics involving conceptual understanding that one might attribute to the fourth-grade curriculum were emphasized in some sources but not in

MAY 1983

504

TABLE 1. A Description of the Core Curriculum of Fourth-Grade Mathematics Textbooks and Tests

Textbooks Tests

A.W. Holt H.M. S.F. CTBS-I CTBS-II Iowa MAT Stanford (4,288) (4,479) (6,986) (4,518) (98) (98) (104) (50) (112)

1. Topics emphasized in all four books but none of the tests:

Basic addition facts 2.1 3.1 Basic subtraction facts 2.2 3.6 Find equivalent fractions 1.8 1.1

2. Topics emphasized in all four books but only one test:

Column addition (single-digit no.) 1.2 .6 Division w/ remainder (multiple-digit no.) 3.4 1.1 Estimating or rounding off 1.8 2.5

3. Topics emphasized in all four books and on two or three tests:

Basic multiplication facts 11.9 4.3 Basic division facts 7.7 3.0 Division w/ remainder

(single-digit divisor) 3.3 1.2 Ordering multiple-digit no. .5 1.7 Column addition (multiple-digit no.) 1.4 1.9 Multiplication (multiple-digit no.) 5.9 2.4 Read measurement instruments 1.1 .8

4. Core topics in all textbooks and all tests: Pictorial models of a fraction .7 1.5 Geometric terms (w/ pictures) 1.5 8.8 Place value (multiple-digit no.) 2.1 3.1 Addition w/ carrying (multiple-digit no.) 1.8 2.4 Subtraction w/ borrowing

(multiple-digit no.) 4.5 4.1 Multiplication

(single- x multiple-digit no.) 4.8 3.6 5. Topics included on all tests but not emphasized in all books:

Reading large numbers .8 1.4 Number sentences involving properties .0 1.3 Story problems

(basic multiplication facts) .2 .1

3.6 2.0 2.3 2.3 1.9 .6

1.2 .9 2.3 1.7 1.7 1.6

11.6 7.5 8.4 6.0

.0

.0

.0

2.0 .0 .0

.0

.0

.0

.0 1.0 .0

.0 .0

.0 .0

.0 .0

.0 .0

.0 .0 1.0 .0

3.1 .0 2.9 4.0 9.2 1.0 1.9 4.0

.0 1.0 4.1

.0 1.0

1.0 5.1 2.0 4.1

5.0 2.0 1.6 .7 .5 .8

4.4 3.2 .9 1.4

.6 1.1 2.3 4.8 1.5 2.8 2.4 1.8

2.2 2.6

8.3 4.7

.2 1.2

.0 1.8

.4 .2

1.0 .0

4.1 2.0 1.0

2.0 4.1

.0 1.0

1.0 .0 .0 6.0

3.8 .0 2.9 .0

.0 2.0

1.0 2.0 1.9 2.0 2.9 4.0 4.8 2.0

.0

.0

.0

.0

.0

.0

.0

.0

.0

.0 2.7 2.7

.0

.9 1.8 3.6

.9

2.0 4.1 6.7 4.0 3.6

9.2 4.1 5.8 4.0 2.7

2.0 1.0 1.9 2.0 2.0 .0 1.9 2.0

1.0 .0 1.0 2.0

.9 2.7

.9

NOTE.-Entries represent the percentage of items in a textbook or test that deal with each specific topic. Figures in parentheses are the number of exercise problems (in the texts) or items (in the tests). A.W. = Addison-Wesley; H.M. = Houghton Mifflin; S. F. = Scott, Foresman; CTBS-I = Comprehensive Tests of Basic Skills, Level I; CTBS-II = Comprehensive Tests of Basic Skills, Level II; MAT = Metropolitan Achievement Tests.


others (e.g., understanding the concepts of multiplication and division through pictorial models).

Degree of consensus at different levels of detail

By changing the definition of the core curriculum for textbooks or tests, it is possible to increase the list of topics common to all sources. Reducing the criteria for core textbook topics from 20 items to one item in each book increases the number of common core topics from six to eight (sections 4 and 5 of table 1). Changing the criterion for core test topics to a smaller proportion of tests (e.g., two or three of five) increases the number of common core topics to 16 (sections 3 and 4 of table 1).

It is also possible to increase the list of common textbook and test topics by describing content at a more general level. Aggregating across cells identifies margin-level topics suggested by each of the three dimensions of the taxonomy (fig. 1). Most of these are common to all books and tests. For the first dimension, general intent, all books and tests included problems involving conceptual understanding with and without pictures, skills with and without pictures, and applications without pictures. However, even at this level of analysis, there were differences in content covered in textbooks and tests. As described in table 2, all books, but not all tests, included items in which the information needed to solve the problem was conveyed in charts, graphs, or other pictorial forms. Moreover, diversity in content treatment was not limited to differences in topics that were covered or not covered. There were important differences in the extent to which topics were emphasized. For example, the proportion of problems in the Scott, Foresman and Holt textbooks that focused on conceptual understanding (approximately 25%) was roughly equal to that on tests, but it was more than double that for the Houghton Mifflin and Addison- Wesley books (approximately 10%).

Most of the margin-level topics suggested by the second dimension- nature of material-were considered in all books and tests. The set of common topics included: three categories of whole numbers, fractions with like denominators, two categories of measurement, and geometry. However, algebraic phrases, fractions with unlike denominators, mixed numbers, and percentages were not covered in all tests and textbooks (table 2). For the operations dimension of the taxonomy, there were 11 margin-level topics common to all books and tests. The three topics considered in some, but not all, sources were division with remainders, applying properties, and estimation (table 2).

Clearly, the more general the topics considered, the more commonality there will be across textbooks and tests. It is therefore important to recognize that teachers are likely to emphasize distinctions at the cell level, as well as at the margin level, of the taxonomy when planning instruction. In fact, preliminary analyses of daily logs prepared by seven elementary school teachers indicate that this group typically emphasized even finer distinctions when planning lessons in the skills area.3 Nevertheless, for the reader who finds the cell level of the taxonomy too detailed, it is important to note that the lack of consensus in coverage across textbooks and tests per- sists even at the level of the marginals of the taxonomy.

Textbook/test comparisons What is taught and not tested

Imagine that a teacher looks to a textbook for guidance in deciding what to teach in fourth-grade mathematics. If that teacher decides to teach all of the lessons in the book and nothing else, what topics will be taught that will not be covered on a standardized test administered at the end of fourth grade? Will the match between what is taught and what is tested be better for some combinations of textbooks and tests than for others?

MAY 1983

506

TABLE 2. Margin-level Topics Covered in Some But Not All Books and Tests

Textbooks Tests

A.W. Holt H.M. S.F. CTBS-I CTBS-II Iowa MAT Stanford (4,288) (4,479) (6,968) (4,518) (98) (98) (104) (50) (112)

1. General intent: Applications with pictures 4.6 3.5 1.1 5.9 .0 .0 10.6 2.0 6.3

2. Nature of material: Algebraic phrases 1.4 2.0 .0 1.7 4.1 5.1 1.9 4.0 4.5 Fractions with unlike denominators 3.5 4.2 4.0 4.1 .0 4.1 .0 .0 .9 Mixed numbers 4.4 2.2 .5 2.4 .0 4.1 1.0 .0 .0 Percents .0 .0 .0 .0 .0 1.0 .0 .0 .9

3. Operations: Division with remainders 7.2 2.5 7.7 4.7 .0 2.0 1.9 .0 .9 Apply properties .0 1.7 .1 1.8 2.0 .0 2.9 6.0 5.4 Estimate 6.2 5.4 3.2 3.5 .0 1.0 2.9 .0 1.8

NOTE.-Entries represent the percentage of items in a textbook or test that deal with each general topic. Figures in parentheses are the number of exercise problems (in the texts) or items (in the tests). A.W. = Addison-Wesley; H.M. = Houghton Mifflin; S.F. = Scott, Foresman; CTBS-I = Comprehensive Tests of Basic Skills, Level I; CTBS-II = Comprehensive Tests of Basic Skills, Level II; MAT = Metropolitan Achieve- ment Tests.


Table 3 describes the percentage of textbook topics covered by each test. The rows labeled "T" represent the percentages of all book topics covered in each test. These values ranged from a low of 13.6% for the Holt book and MAT test to a high of 28.7% for the Houghton Mifflin text and Iowa test. The rows labeled "T1" describe the percentage of topics covered by 20 or more items in each book and also considered in a given exam. These figures ranged from a low of 28.6% for the Addison- Wesley text and MAT test to a high of 46.9% for two of the books (the Holt and Houghton Mifflin texts) and the CTBS-II test.

At first glance, these figures seem sensi- ble. Many factors implicit in the design of standardized tests make it unreasonable to expect a one-to-one correspondence between content covered in textbooks and tests. Standardized tests are designed to measure achievement at more than one grade level and are typically designed to measure transfer as well as achievement. Thus, they are intended to represent sam- ples, not the entire domain, of content that might be tested. As the above figures indicate, at least 28% of the topics emphasized in a given book were also covered in each of the tests.

However, reexamination of the data in

table 1 suggests that the topics tested were not always those most emphasized in the textbooks. At least three of the topics covered by 20 or more items in all four books were not covered by a single item on any of the five tests (e.g., finding equivalent fractions). Ten other topics emphasized in all books were covered on some but not all tests. This set of topics included basic multiplication facts, basic division facts, multiplication involving multiple-digit numbers, and division involving one-digit divisors and remainders. These four topics alone accounted for more than one-fourth of all items in the student exercise portions of the Addison-Wesley text. A teacher who uses this book might therefore be surprised to learn that only one of the four topics, multiplication involving multiple-digit numbers, is considered in the Stanford test. Mismatches of this type suggest that, even if the sampling fraction is reasonably high, considerable instructional time and energy may be devoted to topics that are not tested.4

What is tested that has not been taught? A second way of thinking about the

consistency of content covered in a specific textbook and test is to ask what percentage of the topics on a test are covered in a given textbook. The four columns labeled

TABLE 3. Percentages of Textbook Topics Covered by Each Test

MAT Stanford Iowa CTBS-I CTBS-II Text (38) (72) (66) (53) (61)

Addison-Wesley: T (148 topics) 16.2 26.4 24.3 20.3 25.0 T' (42 topics) 28.6 38.1 40.5 40.5 40.5

Holt: ' (242 topics) 13.6 21.9 20.2 17.4 21.5

rT (49 topics) 38.8 32.7 38.8 34.7 46.9 Houghton Mifflin:

T (167 topics) 16.8 22.8 28.7 20.4 21.6 'F (49 topics) 30.6 30.6 42.9 40.8 46.9

Scott, Foresman: T (197 topics) 14.2 22.8 23.9 27.3 20.8 rT (50 topics) 32.0 32.0 34.0 38.0 42.0

NOTE.-T = Topics covered by at least one item in the book; T1 = Topics covered by at least 20 items in the book. MAT = Metropolitan Achievement Tests; CTBS-I = Comprehensive Tests of Basic Skills, Level I; CTBS-II = Comprehensive Tests of Basic Skills, Level II.

MAY 1983

508

A NATIONAL CURRICULUM

"T" in table 4 describe the percentages of topics in each test that served as the focus of at least one item in the student exercises in each book. In interpreting these figures, it is important to remember that at least 4,000 items were classified for each book. The percentage of tested topics covered in a given book ranged from a low of 52.8% for the Stanford test and Houghton Mifflin text to a high of 86.8% for the MAT test and Holt textbook. Thus, only about half of the topics represented in the Stanford test were covered by one or more of the 6,986 items in the student exercise portions of the Houghton Mifflin text.

The columns labeled "TT1 in table 4 describe the percentages of test topics that served as the focus of at least 20 items in each book. If one assumes that this subset of book topics represents content students will have had an adequate opportunity to learn or to practice during the academic year preceding the test, these figures should provide reasonable estimates of the relation between test content and the content of instruction suggested by the book. These values ranged from a low of 20.8% for the Stanford test and Houghton Mifflin text to a high of 50.0% for the MAT test and Holt text. In other words, the proportion of topics presented on a standardized test that received more than cursory treatment in each textbook was never higher than 50%.

The figures in tables 3 and 4 present an interesting contrast. Whereas the Holt

book covered the highest percentage of topics tested for three of the five tests (table 4), it did so primarily because it covered the largest number of topics. For this reason, it also had the lowest percentage of its topics tested (table 3). It is therefore difficult to determine which book provides the best match with the tests. Overall, it is clear from both tables that no single test is equally well suited for all textbooks and no single book is equally well suited for all tests.

The lack of consensus on what should be taught The primary purpose of this investigation was to examine critically the claim that in elementary school mathematics there is a national curriculum defined by textbooks and tests. The results indicate that this as- sertion is valid only when content is described at a relatively high level of gener- ality. All of the textbooks and tests examined did contain material on addition, subtraction, multiplication, division, and geometry. All sources were also concerned with conceptual understanding, computational skills, and applications. Furthermore, most of the margin-level topics described by the taxonomy were considered in all books and tests.

However, when the analysis shifted to descriptions of content at the cell level of the taxonomy, the appearance of a national curriculum quickly vanished. Con- sidering the emphasis on multiplication at

'ABLE 4. Percentages of Tested Topics Covered in Each Textbook

Addison-Wesley Holt Houghton Mifflin Scott, Foresman

Test T (148) T1(42) T (242) Tr(49) T (167) T1(49) T (197) Tl(50)

MAT (38 topics) 63.2 31.6 86.8 50.0 73.7 39.5 73.7 42.1 Stanford (72 topics) 54.1 22.2 73.6 22.2 52.8 20.8 62.5 22.2 Iowa (66 topics) 54.5 25.8 74.2 28.8 72.7 31.8 71.2 25.8 CTBS-I (53 topics) 56.6 32.1 79.2 32.1 64.2 37.7 64.2 35.8 CTBS-II (61 topics) 60.7 27.9 85.2 37.7 59.0 37.7 67.2 34.4

NOTE.-T = Topics covered by at least one item in the book; T' = Topics covered by at least 20 items in the book. MAT = Metropolitan Achievement Tests; CTBS-I = Comprehensive Tests of Basic Skills, Level I; CT BS-II = Comprehensive 'ests of Basic Skills, Level II.

509


the fourth-grade level, it was not surpris- ing that all of the books and tests contained story problems involving multiplication of whole numbers. However, even in this focal area of the curriculum, consensus did not prevail at more specific levels of the taxonomy. Some books and tests, for example, did not include story problems involving multiplication facts or multiplication of single-digit by multiple-digit numbers. In fact, in our search for specific topics that were covered in all sources, we found that only six cell-level topics were emphasized in all four books and all five tests. This inability to confirm the presence of a national curriculum has critical implications for classroom practice.

Implicationsfor student opportunity to learn National surveys report that many classroom teachers look to textbooks for guidance in deciding what to teach in mathematics (National Advisory Committee on Mathematics Education 1975). This con- clusion is also supported by yearlong case studies conducted by the authors of seven elementary school classrooms. Although there were substantial differences in their style of textbook use, books had an important influence on the content decisions of all seven teachers. One teacher com- mented, "Ninety percent of what you do comes out of the textbook."

Because teachers are influenced by the implied curricula of textbooks, it is important to recognize that the four books examined did not reflect a consensus on what should be taught in fourth-grade mathematics. The content of mathematics instruction is therefore likely to vary in substantive ways in any two fourth-grade classrooms in which different textbooks are used.

Even though this study did not consider more than one grade level, the results suggest that the continuity of the mathematics curriculum is likely to be disrupted whenever students change textbooks as a result of moving from one grade level to

the next. In one of the schools in our case study, for example, four different mathematics textbook or workbook series were used in Kindergarten through Grade 5. Even though the staff recognized that this practice might influence student achievement, a feeling of confidence and comfort with the materials they were using made individual teachers reluctant to press for a more uniform set of instructional materials for the building.

Standardized tests may also influence a teacher's choice of content for classroom instruction. In fact, an earlier study of teachers' responses to hypothetical schools indicated that when a school district re- ports test results by grade level in the local newspaper, standardized tests may function, as one of the strongest sources of curriculum influence (Floden, Porter, Freeman, Schmidt, & Schwille 1981). But what happens when textbooks and tests do not agree? Consider a school district that has mandated the Houghton Mifflin text for all teachers and has elected to report the results of the Stanford Achievement Test in the local newspaper soon after it is administered at the end of the school year. When a fourth-grade teacher in this district carefully examines the Houghton Mifflin text, the message will be to teach 167 specific topics. When the Stanford test is analyzed, it will suggest that 72 distinct topics should be taught, only 38 of which are included in the content message communicated through the book. Because this teacher might also receive content messages from other sources, such as district objectives or comments made by parents, the principal, or other teachers, it is appar- ent that some of these messages must be ignored. Within the time available for mathematics instruction, it is not possible for teachers to cover adequately all the topics they may be asked to teach.

What topics should be ignored? There are no clear directives to guide these decisions. Therefore, as our case studies indicate, teachers develop unique patterns of

MAY 1983

510


responding to content messages and may choose to ignore even those messages that are consistently communicated across various sources of influence (e.g., geometry terms). As a result, significant differences in the content of mathematics instruction are almost certain to exist among elementary school classrooms.

Overall, the results of this investigation suggest that diversity rather than consensus is likely to characterize the mathematics curricula of elementary schools. If so, how much variation in the content of classroom instruction is desirable? Although we are not unanimous in our stance toward this issue, we are convinced that the question must be addressed by educational policymakers. For example, policies that allow buildings or individual teachers to select textbooks cannot be justified on the basis of an assumed national curriculum. Advocates must show that the discontinuity of instruction brought about by such policies is trivial compared with the benefits achieved through more autonomous decision mak-

ing. In general, the results of our content analyses underscore the need to examine any educational policy that is grounded in the assumption that there is a national curriculum.

Implications for interpreting standardized test scores Most educators assume there is a reasonable match between content taught in textbooks and content tested in standardized tests of achievement. This investigation challenges this assumption for fourth-grade mathematics. The proportion of topics covered on a standardized test that received more than cursory treatment in a textbook was never more than 50%. In other words, if a fourth- grade teacher limits instruction to one of the four books analyzed, students will have an adequate opportunity to learn or to re- view less than half of all topics that will be tested at year-end. Therefore, it would ap-

pear that the five standardized achievement tests measure mathematical aptitude as well as achievement of some of what is taught.

When there are mismatches between content taught and content tested, standardized tests underestimate student achievement (Porter et al. 1978). This phenomenon is perhaps most clearly illustrated by an analysis of content that is taught but not tested. Our yearlong case studies indicated that fourth-grade teachers devoted considerable instructional time to multiplication facts and to multiplication and division algorithms. These three topics were also heavily emphasized in all four books we examined. Yet one or more of these topics was totally ignored on all but the Iowa test (see table 1). In other words, fourth-grade students who worked long and hard to attain mas- tery in all three content areas had an opportunity to demonstrate the full range of their achievement on only one of the five tests.

It should also be noted that our analyses provide liberal estimates of the match between content taught and content tested. The figures that have been re- ported overlook the fact that teachers are almost never able to cover the entire book (see Freeman, Belli, Porter, Floden, Schmidt, & Schwille 1981). They also overlook distinctions that may be important to student performance on tests such as the match in terminology used in the book and test. Our results, therefore, provide conservative estimates of the extent to which standardized tests of fourth-grade mathematics are likely to underestimate student achievement.

Another common assumption among educators is that standardized achievement tests may be used interchangeably. Once again, our findings challenge this assumption for fourth-grade mathematics. The match in content tested was clearly better for some textbook-test pairings than for others. Therefore, it is likely that overall

511


levels of student achievement will vary as a function of the particular combination of textbook and standardized test a school district has elected to use. The greater the match in content covered in these two sources, the higher the overall level of student performance.

This relationship is particularly significant during this era in which schools are being pressed toward greater account-

ability. When achievement test scores in district A exceed those in district B, parents may be tempted to conclude that the

quality of instruction in district A exceeds that provided in district B. However, it is

possible that this difference in scores results solely from a mismatch between content taught and content tested. In the ab- sence of a fully standardized curriculum, any comparison or simple interpretation of student performance on standardized achievement tests must consider the match between content taught and content tested.

Notes

This work was sponsored in part by the In- stitute for Research on Teaching, College of Education, Michigan State University. The In- stitute for Research on Teaching is funded primarily by the Teaching Division of the Na- tional Institute of Education, Department of Education. The opinions expressed in this arti- cle do not necessarily reflect the position, policy, or endorsement of the National Institute of Education (Contract No. 400-76-0073).

1. Measures of interrater agreement were based on the domain of cell-level topics that were identified for a given book or test by either of the two raters. The number of items one rater attributed to each of the topics within this domain was correlated with the corresponding set of figures for the second rater.

2. The number of items in a particular textbook lesson varies as a function of the type of content covered. Twenty items may represent as many as 1 /2 lessons for content areas such as story problems or as little as one-third of a lesson for other areas such as basic number facts.

3. The results of yearlong case studies of

seven elementary school classrooms conducted by the authors will be summarized in a mono- graph to be published by the Institute for Re- search on Teaching, Michigan State University.

4. For a more comprehensive discussion of the relation between content taught and content tested as it affects content validity, see Schmidt, Porter, Schwille, Floden, & Freeman (1981).

References

Bolster, L. C., et al. Mathematics around us. Glen- view, Ill.: Scott, Foresman, 1978.

CTB/McGraw-Hill. Comprehensive tests of basic skills. Monterey, Calif.: McGraw-Hill, 1976.

Duncan, E. R.; Quast, W. G.; Allen, C. E.; Coops, L. R.; Ebos, F.; & Hater Haubner, M. A. Mathematics. Boston: Houghton Mifflin, 1978.

Eicholz, R. E.; O'Daffer, P. G.; & Fleenor, C. R. Mathematics in our world. Menlo Park, Calif.: Addison-Wesley, 1978.

Floden, R. E.; Porter, A. C.; Freeman, D. J.; Schmidt, W. H.; & Schwille, J. R. Responses to curriculum pressures: a policy-capturing study of teacher decisions about content.

Journal of Educational Psychology, 1981 73, 129-141.

Freeman, D.; Belli, G.; Porter, A.; Floden, R.; Schmidt, W.; & Schwille, J. The conse- quences of different styles of textbook use in preparing students for standardized tests (Research Series No. 107). East Lansing: In- stitute for Research on Teaching, Michigan State University, 1981.

Hieronymus, A. N., et al. Iowa testsfor basic skills. Boston: Houghton Mifflin, 1978.

Kuhs, T.; Schmidt, W.; Porter, A.; Floden, R.; Freeman, D.; & Schwille, J. A taxonomy for classifying elementary school mathematics content (Research Series No. 4). East Lan- sing: Institute for Research on Teaching, Michigan State University, 1979.

Madden, R.; Gardner, E.; Rudman, H.; Karlsen, B.; & Merwin, J. Stanford achievement tests. New York: Harcourt Brace Jovanovich, 1973.

National Advisory Committee on Mathematics Education. Overview and analysis of school mathematics: grades K-12. Washington, D.C.: Conference Board on the Mathematical Sci- ences, 1975.

Nichols, E. D., et al. Holt school mathematics. New York: Holt, Rinehart & Winston, 1978.

MAY 1983

512


Porter, A. C.; Schmidt, W. H.; Floden, R. E.; & Freeman, D. J. Practical significance in pro- gram evaluation. American Educational Re- search Journal, 1978, 15, 529-539.

Prescott, G. A.; Balow, I. H.; Hogan, T. P.; & Farr, R. C. Metropolitan achievement tests. New York: Harcourt Brace Jovanovich, 1978.

Schmidt, W.; Porter, A.; Schwille, J.; Floden, R.; & Freeman, D. Validity as a variable: can

the same certification test be valid for all students? In G. F. Madaus (Ed.), The courts, validity, and minimum competency testing. Bos- ton: Kluwer-Nijhoff, 1983.

Weiss, I. R. Report of the 1977 national survey of science, mathematics and social studies (RTI/1266/06-01F). Research Triangle Park, N.C.: Research Triangle Institute, 1978.

513

do textbooks and tests define a national curriculum in...

Documents