impact of student calculator use on the 2013 naep twelfth ... · calculator use twelfth- grade naep...

Running head: CALCULATOR USE TWELFTH- GRADE NAEP MATHEMATICS 1

Impact of Student Calculator Use on the 2013 NAEP Twelfth-Grade Mathematics Assessment

Beverly M. Klecker

Morehead State University

[email protected]

Richard L. Klecker

University of Kentucky

Paper presented at the annual meeting of

Mid-South Educational Research Association

Knoxville, TN

November 5, 2014

CALCULATOR USE TWELFTH-GRADE NAEP MATHEMATICS 2

Abstract

This descriptive research study examined 2013 NAEP 12th-grade mathematics scores by

students’ use of graphing calculators in math classes and the kind of calculator students used

during NAEP assessment. NAEP Data Explorer analysis included two questions from Student

Factors: How often do you use these different kinds of calculators in math class? Graphing

[M817601], and Kind of calculator used for test? [M830201]. The literature review included

international experimental and quasi-experimental research on student use of calculators during

learning and assessment. The overall (100%) M=152, SD=34 (scale-range 0-300). NAEP scores

by use of graphing calculator in math class were: never use (18%) M=137, SD=29; sometimes

not often use (21%) M=145, SD=34; usually use (61%) M=165, SD=32. NAEP scores by

calculator use for the test: None: (35%) M=153, SD=33; NAEP provided: (39%) M=147,

SD=30; Scientific non-graphing: (8%) M=146, SD=33, Graphing: (17%) M=171, SD=33. Effect

sizes (Cohen, 1988) ranged from d=0.26 to d=0.90.

CALCULATOR USE TWELFTH- GRADE NAEP MATHEMATICS 3

Impact of Student Calculator Use on the 2013 NAEP Twelfth-Grade Mathematics

Assessment

The use of calculators in teaching mathematics in elementary and secondary schools in

the United States has been a variable for instruction and assessment across the years. Robelen

(2013), citing Horizon Research, Inc., stated that 64% of high school students use graphing

calculators at least once a week (p. 12). Robelen suggested that the use of graphing calculators in

teaching and testing in the United States would change with the coming Core-aligned exams.

The use of calculators will continue to be permitted in the international assessments

Program for International Student Assessment (PISA) (Organization for Economic Co-Operation

and Development (OECD), March, 2013), and Trends in International Mathematics and Science

Study (TIMSS). TIMMS calculator policy for the eighth-grade level was described by Mullis and

Martin (2013):

Calculators and policies for their use differ across the TIMSS countries. Also, the

availability of calculators varies widely. It would not be equitable to require

calculator use when students in some countries may never have used them.

Similarly, however, it is not equitable to deprive students of the use of a familiar

tool. In order to give students the best opportunity to operate in settings that

mirror their classroom experience, TIMSS has permitted calculator use at the

eighth grade since 2003. Thus, if eighth grade students are accustomed to having

calculators for their classroom activities, then the country should encourage

students to use them during the assessment. On the other hand, if students are not

accustomed to having calculators or are not permitted to use them in

their daily mathematics lessons, then the country need not permit their use. In

developing the new assessment materials, every effort will be made to ensure


that the test questions do not advantage or disadvantage students either way,

with or without calculators. (p. 24)

Calculator use permitted on NAEP fourth-, eithth-, and twelfth-grade mathematics

assessments varies by grade level. National Center for Educational Statistics (NCES)

(November, 2013) stated:

NAEP Calculator Policy

The mathematics assessment contains some sections for which calculators are not

allowed, and other sections that contain some questions that would be difficult to

solve without a calculator. At each grade level, approximately two-thirds of the

assessment measures students’ mathematical knowledge and skills without access

to a calculator; the other third allow a calculator’s use. The type of calculator

students may use varies by grade level, as follows:

At grade 4, a four-function calculator is supplied to students, with

training at the time of administration.

At grades 8 and 12, students are allowed to bring whatever

calculator, graphing or otherwise, they are accustomed to using in

the classroom with some restrictions for test security purposes. For

students who do not bring a calculator to use on the assessment,

NAEP will provide a scientific calculator.

No questions in the test are designed to provide an advantage to students

with a graphing calculator. Questions are categorized according to the

degree to which a calculator is useful in responding to the item:

A calculator inactive question is one whose solution neither

requires nor suggests the use of a calculator.


A calculator is not necessary for solving a calculator neutral

question; however, given the option, some students might choose

to use one.

A calculator is necessary or very helpful in solving a calculator

active question; a student would find it very difficult to solve the

problem without the aid of a calculator. (para 10-11)

The Purpose of the Study

The purpose of this quantitative descriptive research study was to examine whether or

not there were differences in 2013 NAEP twelfth-grade mathematics average scale scores by: (1)

students’ use of a graphing calculator in math classes, and (2) kind of calculator the students

used for the NAEP assessment.

This secondary analysis of the NAEP data used the national public data composite

mathematics scale scores from the 2013 NAEP twelfth-grade mathematics assessment. NAEP

Data Explorer was used for analyses. Two questions were selected using: Student Factors

>Instructional Content and Practice> Modes of instruction/classroom activities from the Select

Variables option in the NAEP Data Explorer (NCES, 2014a):

(1) How often do you use these different kinds of calculators in math

class? Graphing. Options: never use; sometimes not often; usually use

(answered by students). [M817601]

(2) Kind of calculator used for test? Options: no calculator use; scientific non-

graphing calculator; NAEP provided calculator; graphing calculator (answered by

students). [M830201]


Review of Literature

The review of literature begins with studies that examined the impact of the use of

graphing calculators during learning followed by research that examined the impact of calculator

use during assessments.

Graphing/Graphic Calculator Use During Learning

Lukens and Feinstein (2000) studied the impact of the use of graphing calculators in

teaching on Advanced Placement Biology exam scores:

Students enrolled in the integrated Advanced Placement Biology/Advanced

Placement Calculus block course where graphing calculators were a meaningful

part of the curriculum performed significantly better on the Advanced Placement

Biology exam when compared to students enrolled in the traditional Advanced

Placement Biology where graphing calculators were not heavily integrated into

the course. There appears to be a positive correlation between integrating

graphing calculators in the higher level biology classes and the understanding and

achievement of the students enrolled in the integrated course. (p. 1)

Hasan, Azizan, & Kassim (December, 2005) described the introduction of graphic

calculators into the Malaysian New Curriculum for secondary schools Form 4 (16 years old).

The students came from low-income families and had little access to technology. The researchers

used an experimental control-group design with use of graphic calculators (TI-83 Plus) as the

manipulated independent variable. They found that the mean post-test of the group using graphic

calculators (M=9.83, SD 2.678) was higher than the post-test mean of the control group

(M=3.86, SD=2.248) (T3 p-value =0.000).

In a similar study Nor’ain Mohd. Tajudin & Noraini Idris (2013) used a quasi-

experimental non-nonequivalent control group design to compare the effect between the TI-


Nspire CX graphing calculator (CG) and conventional instruction (CI) on learning Straight Lines

and Statistics. Two intact classrooms of students were selected from four schools in Perak and

Selangor. One class in each school was assigned the experimental treatment; the other served as

the control. They found that students in the experimental group from all schools had significantly

higher scores on the outcome measures than the students from the control group. They concluded

that TI-Nspire has the potential to promote higher-level mathematical thinking.

Graphing/Graphic Calculator Use During Assessment

Haimer (1999) examined calculator use and equity on the 1998 Tertiartiary Entrance

Exam (TEE) in Australia. This exam required the use of graphic calculators by all students.

Haimer examined gender equity and hypothesized that females would have more difficulty than

males using graphing calculators. The hypothesis was not supported. A later secondary analysis

of the same data by Haimer and Webster (2001) compared the mathematics scores of rural and

urban Western Australian students. Urban students scored higher than rural students in the three

years studied; however, the gap was inconsistent across items. Haimer and Webster (2001)

concluded that rural students were not further disadvantaged by the requirement of graphic

calculator use for the test.

Schwarz, Rich, Arenson, Podrabsky & Cook (2002) explored the question of differences

by calculator use in evaluation with data from the Tennessee Gateway assessment given as an

end-of course test in Algebra 1. Data were from approximately 7,000 students who answered

questions about their use of calculators. Response options were “use” or “no use” and type of

calculator. Schwartz et al. (2002) found:

Calculator type, usage, and familiarity were associated with differences in the

univariate comparison of test scores. For instance, students who responded that a

graphing calculator was used performed higher than the other groups. The use of a


graphing calculator could indicate that higher-level mathematics courses had been

taken. However, this may not be the case in this instance since all students take

the Gateway examination at the conclusion of their first Algebra I class. (p. 16)

DeLoach (2013) used a quantitative, quasi-experimental study to compare the

performance of Algebra II students. Students in the experimental group used a graphing

calculator to complete a standardized mathematics test. Students in the control group did not use

a graphing calculator to complete the same test. The results of the independent t-test found that

the mean of the experimental group was higher than the mean of the control group (t (51) = 2.69,

p = .01).

Ellington (2006) conducted a meta-analysis of research that had investigated the

achievement of students using Non-CAS Graphing Calculators. She summarized in the Abstract:

Forty-two studies comparing students with access to graphing calculators during

instruction to students who did not have access to graphing calculators during

instruction are the subject of this meta-analysis. The results on the achievement

and attitude levels of students are presented. The studies evaluated cover middle

and high school mathematics courses, as well as college courses through first

semester calculus. When calculators were part of instruction but not testing,

students' benefited from using calculators while developing the skills necessary to

understand mathematics concepts. When calculators were included in testing and

instruction, the procedural conceptual and overall achievement skills of students

improved. (p. 2)


Large-Scale Study with Scientific Non-Graphing Calculators

Close, Oldham, Shiel, Dooley, & O’Leary (2012) described an extended study with 1,469

Grade nine students; however, the type of calculator was “scientific.” The authors described the

study:

Three calculator tests were administered to a national sample of 1,469 Irish

students in Grade 9-the last cohort to study mathematics without calculators

(Phase 1). Three years later, the same tests were administered to a similar sample

with calculators (Phase 2). Scores on a test of calculator-inappropriate items

showed no significant change over the 3 years. For a test of calculator optional

items, students were divided randomly into 2 groups, 1 with calculator access and

the other without. In both phases, the students with calculators achieved

significantly better than the students without calculators. Achievement on a test of

calculator appropriate items showed significant improvement over the 3 years.

Students’ attitudes toward calculators also improved over the time. (p. 377)

However, the types of calculators used by the Grade 9 students in the study were not well-

defined. The authors stated:

Students’ use of calculators. [italicized heading in the original] In Phase 1, 55.8%

of students reported that they had access to a calculator at school, whereas

practically all students in Phase 2 had access to one. By Phase 2, scientific

calculators were most frequently owned, and fewer than 1% of students used or

had access to a graphing calculator. (p. 385)

Summary

The literature reviewed included correlational and quantitative experimental and quasi-

experimental designs that investigated the effects of graphing (graphic) calculators both in


teaching and assessment. The large-scale Irish study that was conducted across time included

mostly “scientific” calculators and very few (about 1%) graphing calculators. In the literature

reviewed higher assessment scores were obtained by students who used graphing (graphic)

calculators for learning and/or during assessments.

Method

This secondary analysis of NAEP data used a quantitative descriptive research design that

examined differences in the average scale scores (composite) by the two research questions.

Participants and Sampling

Because state NAEP assessments do not include 12th

-grade students, a grade twelve

sample of schools was selected (NCES, 2014b). The sample was designed to provide national

estimates of 12th

-grade achievement. The 2009 and 2013 samples were from selected students

within selected schools from eleven volunteer states: Arkansas, Connecticut, Florida, Idaho,

Illinois, Iowa, Massachusetts, New Hampshire, New Jersey, South Dakota, and West Virginia

(NCES, 2014b). The target population (rounded to nearest thousand) for the national public data

was 2,986,000. The sample for the national public data (rounded to nearest hundred) was 46,000

(U.S. Department of Education, 2013, p.1).

Data Analysis

The NAEP Data Explorer (NCES 2014a) was used for the data analysis. The 2013

NAEP twelfth-grade mathematics assessment composite average scale scores and standard

deviations were selected for the analyses. The three coded questions selected through Data

Explorer (NCES, 2014a) were:

(1) How often do you use these different types of calculators in your math

class? Graphing. Options: never use; sometimes not often; usually use

(answered by students). [M817601]


(3) Kind of calculator used for test? Options: no calculator use; scientific non-

graphing calculator; NAEP provided calculator; graphing calculator (answered by

students). [M830201]

Descriptive tables and tests of statistically significant of differences (alpha set a priori at

0.001 because of the large n of the sample) were calculated and presented by Data Explorer

(NCES, 2014a). In several instances, the tables were re-formatted without editing the data in the

tables. Cohen’s d effect sizes (Cohen, 1988) were hand calculated.

Results

Missing Frequency (N) in Tables

The NAEP Data Explorer tool (NCES, 2014a) does not include the number of students

either in the overall data (100%) or within sub-strata of the data. Consequently, tables in this

Results section include the percentages of the sample that are in the sub-strata reported without

the expected frequency (N).

The average scale score mean for 100% of the twelfth-grade students on the 2013

NAEP Mathematics assessment was 152 (scale-range 0-300) with a standard deviation of 34.

Differences in scores by questions are presented in tables throughout the results section.

[place Table 1 about here]

The average scale score of the twelfth-grade students who reported that they never

used graphing calculators (18%) was 137 (29). The mean of the average scale score of the

students who “sometimes not often” used graphing calculators in math class (21%) was 145 (34).

The average scale score of the students who “usually use” graphing calculators in math class

(61%) was 165 (32) (Table 1).



Table 2, created by NAEP Data Explorer, presents mean differences and the results of

multiple independent t-tests. Note that NAEP Data Explorer analyses have alpha set at 0.05

rather than the 0.001 set a priori by the researchers. The average scale scores of students (21%)

who “sometimes not often” used graphing calculators in math class (M=145; SD=34) was

significantly (p <0.001) higher than the average scale scores of students (18%) who “never

used” graphing calculators in math class (M=137; SD=29). The average scale scores students

(61%) who “usually use” graphing calculators in math class (M=165, SD=32) was 20 points

higher than students who “sometimes not often” used graphing calculators in math class and 28

points higher than students who “never used” graphing calculators in mathematics classes.


The effect size of the difference between the mean average scale score of students who

reportedly “usually use” graphing calculators in math class and the mean for students who “never

use” graphing calculators in math class was d=0.90 (Table 3). The effect size of the difference

between mean average scale scores of students who “usually use” and students who “sometimes

not often” use graphing calculators in math class was d=0.60. The effect size of the difference

between mean average scale scores of students who “sometimes not often” and “never use”

graphing calculators in math class was d=0.26. The effect sizes were small (0.26), medium

(0.60), and large (0.90).

The large effect size (d = 0.90) was between the mean scale scores of students (18%) who

“never use” used graphing calculators in math class and the mean average scale scores of

students (61%) who “usually use” graphing calculators in math class. This finding is consistent

with the findings of studies reviewed for this study (e.g., Hasam, Azizan, &Kassim, December,

2005; Lukens & Feinstein, 2000; Nor’am Mohd, Tajudi, & Noraini Idris, 2013).


There is a dissonance between the data in Table 1 and Table 4 (below). From the data in

Table 1, 61% of the students reported that they “usually use” graphing calculators in

mathematics class. However, only 17% of the students indicated that they used graphing

calculators (Table 4) to take NAEP 2013 twelfth-grade mathematics assessment.


The average scale score for the 35% of the twelfth-grade students who used no calculator

for the test was 153 (33) (Table 4). The average scale score for the 39% of the students who used

the NAEP-Provided calculator was 147 (32). The average scale score for the 8% of the students

who used a Scientific Not Graphing calculator was 146 (31). The average scale score for the

17% of the students who used a Graphing calculator was 171 (33) (Table 4).


Table 5 presents results of the NAEP Data Explorer test for statistical significance

between the average score means by calculator-use category. There was no statistically

significant difference between the average scale scores of the students who used the NAEP-

provided calculator (M=146) and the students who used the Scientific Non-Graphing calculator

(M=147), (p = 0.4592). The NAEP-provided calculator was described as a scientific-non-

graphing calculator, thus, this result is not surprising. Means for each of these categories were

significantly (p <0.001) lower than the mean for students reporting that they used no calculator

(M=153).

The average scale score of the students who used a graphing calculator for the NAEP

assessment was significantly different from the average scale scores of students who selected

other categories (Table 5). The scores of students using graphing calculators was 18 point higher

than average scale scores of students using no calculator, 24 points higher than the average scale


scores of students using NAEP-provided calculators, and 25 points higher than the average scale

scores of students using scientific non-graphing calculators.


The effect size for the difference in the average scale score of students using graphing

calculators and the average scale score of students using no calculator was d=0.55 (Table 6).

The effect size of the difference between the average scale score of students using the graphing

calculator and the average scale score of students using the NAEP-provided calculator was

d=0.74. The effect size for the difference in the average scale score of students who used

graphing calculators and the average scale score of students who used scientific, non-graphing

calculators was d= 0.78. The effect sizes were medium (d=0.55) to large (d=0.74 and d=0.78)

(Cohen, 1988).


Table 7 presents the means and standard deviations of the NAEP 2013 twelfth-grade

mathematics scores by Kind of Calculator Used for the NAEP Test by Graphic Calculator Used

in Math Class. The highest average scale score on Table 7 (M=177, SD=29) is for the students

who “usually used” a graphing calculator for math class and who also used a graphing calculator

while taking the NAEP test. This score was 11 points higher (effect size d=0.37) than the next

highest average scale score (M=166, SD=31) obtained by students who “usually use” a graphing

calculator in math class and used no calculator on the NAEP test.

The group of students who “usually used” a graphing calculator in math class and used a

graphing calculator while taking the NAEP twelfth-grade 2013 mathematics exam exceeded the

overall average score (M=152, SD=34) by 25 points with a large effect size of d=0.79.


Discussion

This paper was inspired by a conversation with colleagues about equity in student

technology use in public schools. A question was raised, “Is there research evidence that the use

of technology causes increased learning?” With “technology” more narrowly defined as

calculator use in mathematics learning and assessment, a review of experimental and quasi-

experimental research studies found that the use of graphing calculators in mathematics teaching

and assessment in secondary schools caused higher student achievement.

The NAEP 2013 twelfth-grade mathematics public school data were selected for the

study for three reasons: (1) the data were from a large representative sample of twelfth-grade

students from the United States, (2) the database included information about secondary students’

calculator use in the classroom and during the NAEP test, and (3) the researchers’ familiarity

with NAEP data, the NAEP Data Explorer, and the use of graphing calculators in math class and

assessments.

Summary of Findings

1. The average scale score of students (61% ) who “usually used” graphing

calculators in math class was 20 point higher than students who “sometimes not

often used” graphing calculators in math class (effect size d=0.60) and 28 points

higher than students who “never used” graphing calculators in math class (effect

size d=0.90).

2. A noted dissonance in the data, overall 61% of the students reported that they

“usually used” graphing calculators for math class; 17% of the students reported

that they used graphing calculators for the test.


3. The average scale score of students (17%) who used graphing calculators for the

test was:

18 points higher the than average scale score of students (35%) who did

not use a calculator for the test (effect size d=0.55 )

24 points higher than the average scale score of students (39%) who used

the NAEP-provided calculator (effect size d=0.74)

25 points higher than the average scale score of students (8%) who used a

scientific non-graphing calculator (effect size d=0.78)

4. The average scale score of students who “usually used” a graphing calculator in

math class and used a graphing calculator while taking the NAEP twelfth-grade

2013 mathematics assessment (M=177, SD=29) (Table 5) exceeded the overall

average score (M=152, SD=34) by 25 points with a large effect size of d=0.79.

Conclusions

Results of this quantitative descriptive research study indicate that the average scale

scores of 12th

-grade students who “usually use” graphing calculators in math class (61%) are

higher than students who do not use graphing calculators in math class. Additionally, average

scale scores of students who used a graphing calculator while taking the 2013 NAEP 12th

-grade

Mathematics assessment had higher scores than students who used no calculator or another type

of non-graphing calculator. Further, the smaller group of students who both “usually use” a

graphing calculator in math class and used a graphing calculator for the test had an average scale

score that was higher than that of other students who “usually use” a graphing calculator in math

class but used either no calculator of another kind of non-graphing calculator for the test. The

effect size of the difference between the scores of the students who both usually used a graphing

calculator in mathematics class and also used a graphing calculator for the test (Table


These findings are consistent with the results of the experimental and quasi-experimental

studies in the review of literature (DeLoach , 2013; Hasan, Azizan, & Kassim, December, 2005;

Nor’ain Mohd. Tajudin & Noraini Idris, 2013) that found that the use of graphing calculators

either during teaching, during assessment, or both caused an increase in achievement. Ellington

(2006) conducted a meta-analysis of research that had investigated the achievement of students

using Non-CAS Graphing Calculators and summarized, “…When calculators were included in

testing and instruction, the procedural conceptual and overall achievement skills of students

improved” (p. 2).

The large effect sizes in this descriptive study suggest that the use of graphing calculators

in math class and/or for the assessment explains some of the variance in the average scale scores.

However, it is always unclear from this type of research whether the “use of graphing calculator”

in class and during assessment is a proxy for characteristics of the students, the students’ home

life, the socioeconomic status (SES) of the students’ parents/guardians, the quality of the school

that the students attend, or the teachers who teach them.

Suggestions for Further Research

The use of graphing calculators on large-scale assessments such as NAEP, PISA,

TIMMS, and the SAT could be manipulated in a quasi-experimental design. The international,

cross-cultural studies using quasi-experimental and experimental designs—cited in the review of

literature for this study—found that the secondary school student use of graphing calculators

increased both learning and assessment scores.

This descriptive study found that the combination of using a graphing calculator in math

class and using a graphing calculator while taking the NAEP assessment resulted in higher

achievement (consistent with experimental/quasi-experimental research reviewed).


Future Research Question:

“If we increase the number of 12th

-grade US students who use graphing calculators

both in math class and during the NAEP assessment, will the NAEP average scale

score increase?”

END NOTE: The authors suggest that the two questions used in this study be continued in

future NAEP twelfth-grade mathematics assessments. It would be helpful if these questions

could be added to the NAEP 8th

-grade mathematics assessments. Further, the responses to these

two questions would also be valuable for TIMMS and PISA for use by cross-cultural researchers

and policy-makers.

.


References

Close, S., Oldham, E., Shiel, G., Dooley, T., & O’Leary, M. (2012). Effects of calculators on

mathematics achievement and attitudes of ninth-grade students. The Journal of

Educational Research, 105(6), 377-390. DOI: 10.1080/00220671.2011.629857

DeLoach, M. (2013). The impact of graphing calculators on high-school students’ performance

on a standardized test (Unpublished doctoral dissertation). University of Phoenix.

ProQuest Dissertations & Theses. Publication Number: 3585951

Ellington, A. J. (2006). The Effects of Non-CAS Graphing Calculators on Student Achievement

and Attitude Levels in Mathematics: A Meta-Analysis. School Science & Mathematics,

106(1), 16-26. Retrieved from ERIC Database (EJ751981)

Haimes, D. (1999). Graphics calculators in examinations: A question of equity?

Paper presented at the combined Annual Meeting of the Australian Association for

Research in Education and the New Zealand-Association for Research in Education

Melbourne, Australia, November 29-December 2, 1999. Retrieved from ERIC Database

(ED453068)

Haimes, D., & Webster, B. (2001). Rural students and graphics calculators in examinations.

Education in Rural Australia, 11,(1). 54-61 Retrieved from ERIC Database (EJ631525)

Hasan, H., Azizan, M., & Kassim, S. (2005, December). The use of graphic calculators in

Malaysian secondary schools: Students’ perception and teachers’ constraints. In

Proceedings of 10th Asian Technology Conference in Mathematics. Blacksburg, VA:

ATCM Inc. Retrieved from http://epatcm.any2any.us/EP/EP2005/2005P146/fullpaper.pdf

Mullis, I.V.S. & Martin, M.O. (Eds.). (2013). TIMMS 2015 assessment frameworks. Chestnut

Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Retrieved from

http://timssandpirls.bc.edu/timss2015/frameworks.html


National Center for Educational Statistics (NCES). (November, 2013). NAEP Calculator Policy

Retrieved from http://nces.ed.gov/nationsreportcard/mathematics/whatmeasure.aspx

NCES. (2014a). NAEP Data Explorer . Retrieved from http://nces.ed.gov/nationsreportcard/naepdata/dataset.aspx

NCES (2014b). NAEP assessment sample design 12th

grade. Retrieved from

http://www.nationsreportcard.gov/reading_math_g12_2013/#/about#naep_samples

Nor’ain Mohd. Tajudin & Noraini Idris (2013). TI-NSPIRE CX graphing calculator: Enhancing

students’ performance in mathematics learning. Asia Pacific Journal of Multidisciplinary

Research, 1, (1), 105-115.

Organization for Economic Co-Operation and Development (OECD). (March 2013). PISA 2015

draft mathematics framework. Retrieved from

http://www.oecd.org/callsfortenders/Annex%20IC_PISA%202015%20Math%20Framew

ork%20.pdf

Robelen, E.W. (2013, August 21). Use of calculators to shift with Core-aligned exams.

Education Week, 33, (1), pp. 1, 12.

Schwarz, R.; Rich, C.; Arenson, E.; Podrabsky, T.; & Cook, G. (2002). An analysis of

Differential Item Functioning based on calculator type. Paper presented at the annual

meeting of the National Council on Measurement in Education, New Orleans, LA.

U.S. Department of Education. (2013). Institute of Education Sciences, National Center for

Education Statistics, National Assessment of Educational Progress (NAEP), 2013

Mathematics Assessment. Retrieved from

http://www.nationsreportcard.gov/reading_math_g12_2013/files/Tech_Appendix_Math_

G12.pdf


Table 1. How Often Do You Use These Different Kinds of Calculators in Math

Class? Graphing. [M817601]

Never Use Sometimes Not Often Usually Use

18%

21%

61%

M SD M SD M SD

137

29

145

34

165

32

Scale Range 0=300

Table 2. Difference In Average Scale Scores Between Variables, For Use Graphing

Calculator In Math Class [M817601] National Public, 2013

Never use

(137)

Sometimes but

not often

(145)

Usually use

(165)

Never use

(137)

<

Diff = -8

P-value = 0.0000

Family size = 3

<

Diff = -28

P-value =

0.0000

Family size = 3

Sometimes but not

often

(145)

>

Diff = 8

P-value =

0.0000

Family size = 3

<

Diff = -20

P-value =

0.0000

Family size = 3

Usually use

(165)

>

Diff = 28

P-value =

0.0000

Family size = 3

>

Diff = 20

P-value = 0.0000

Family size = 3

LEGEND:

< Significantly lower.

> Significantly higher.

x No significant difference.

NOTE: All comparisons are independent tests with an alpha level of 0.05 adjusted for

multiple pairwise comparisons according to the False Discovery Rate procedure. For

comparisons between two jurisdictions, a dependent test is performed for cases where

one jurisdiction is contained in the other. For more detailed information about the

procedures and family sizes please see the Help document.

Table created by NAEP Data Explorer (NCES, 2014a)


Table 3. Effect Sizes of Differences in Scores by Calculator Use During Math Class

Type of Calculator Cohen’s d Effect Size

Usually Use Never Use d = 0.90

Usually Use Sometimes Not Often d = 0.60

Sometimes Not Often Never Use d = 0.26

Table 4. Kind of Calculator Used for Test [M830201] by Average Scale Score

None

35%*

NAEP Provided

39%*

Scientific Not

Graphing

8%*

Graphing

17%*

M SD M SD M SD M SD

153

33

147

32

146

31

171

33

Scale: 0-300; Percentages do not sum to 100% due to rounding.


Table 5. Mathematics, Grade 12. Difference in Average Scale Scores Between Variables, for

Kind of Calculator Used for Test [M830201] National public, 2013

None

(153)

NAEP-

provided

(147)

Scientific not

graph

(146)

Graphing

(171)

None

(153)

>

Diff = 6

P-value =

0.0000

Family size = 6

>

Diff = 7

P-value = 0.0000

Family size = 6

<

Diff = -18

P-value = 0.0000

Family size = 6

NAEP-provided

(147)

<

Diff = -6

P-value =

0.0000

Family size = 6

x

Diff = 1

P-value = 0.4592

Family size = 6

<

Diff = -24

P-value = 0.0000

Family size = 6

Scientific not

graph

(146)

<

Diff = -7

P-value =

0.0000

Family size = 6

x

Diff = -1

P-value =

0.4592

Family size = 6

<

Diff = -25

P-value = 0.0000

Family size = 6

Graphing

(171)

>

Diff = 18

P-value =

0.0000

Family size = 6

>

Diff = 24

P-value =

0.0000

Family size = 6

>

Diff = 25

P-value = 0.0000

Family size = 6

LEGEND:

< Significantly lower.

> Significantly higher.

x No significant difference.

NOTE: All comparisons are independent tests with an alpha level of 0.05 adjusted for multiple pairwise

comparisons according to the False Discovery Rate procedure. For comparisons between two jurisdictions, a

dependent test is performed for cases where one jurisdiction is contained in the other. For more detailed

information about the procedures and family sizes please see the Help document.

Table created by NAEP Data Explorer (NCES, 2014a)


Table 6. Effect Sizes of Differences in Scores by Calculator Use During NAEP Test

Type of Calculator Cohen’s d Effect Size

Graphing Calculator No Calculator d = 0.55

Graphing Calculator NAEP-Provided d = 0.74

Graphing Calculator Scientific Non-

Graphing

d = 0. 78

Table 7. Average Mathematics Scale Score by Kind of Calculator Used for NAEP Test

[M830201] by Use Graphing Calculator in Math Class [M817601]

Use Graphing

Calculator in

Math Class

None NAEP-provided Scientific Not

Graph Graphing

Average

scale

score

Standard

deviation

Average

scale

score

Standard

deviation

Average

scale

score

Standard

deviation

Average

scale

score

Standard

deviation

Never use

137

29

135

29

142

27

139

38

37%

42%

17%

5%

Sometimes

but not often

147

34

140

33

146

33

159

37

35%

39%

14%

11%

Usually use

166

31

159

32

153

33

177

29

34%

33%

5%

28%

Table generated by NAEP Data Tool (NCES, 2014a). Format modified to show percentages.

Row percentages may not sum to 100% due to rounding.

impact of student calculator use on the 2013 naep twelfth ... · calculator use twelfth- grade naep...

Documents