1

How to Model and Test for the Mechanisms that make Measurement Systems Tick

IMEKOJena, Germany

Wednesday, August 31, 2011

A. Jackson Stenner

Chairman & CEO, MetaMetricsjstenner@lexile.com

2

Reader Ability

Temperature

3

Four well researched constructs

Reader ability

Text Complexity

Task Difficulty

Comprehension

4

Reading is a process in which information from the text and the

knowledge possessed by the reader act together to

produce meaning.

Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation of readers: The report of the Commission on ReadingUrbana, IL: University of Illinois

5

Reading is a process in which information from the text [Complexity]

and the knowledge possessed by the reader [Ability]act together to

produce meaning [Comprehension] as indexed by a specific task type

[Difficulty].

6

An Equation

=Reader Ability

Text ComplexityComprehension -

Conceptual

Statistical

RawScore

=i

e (RA – TC - TD)i

1 + e (RA – TC i - TD)

RA = Reading Ability

TC = Text Calibrations

TD = Task Difficulty

- TaskDifficulty

7

A causal model relating reader ability, text complexity, task difficulty and comprehension

Measures reader ability, text complexity, and task difficulty on a common scale—the Lexile scale

Allows educators to forecast the level of success a reader is likely to experience with a particular text when responding to a specific task requirement

The Lexile Framework for Reading

8

Eight Features of the Causal Model Relating Text Complexity, Reader Ability, Task Difficulty and Comprehension1. The model is individual centered. The focus is on explaining

variation within person over time.

2. In this framework the measurement mechanism is well specified and can be manipulated to produce predictable changes in measurement outcomes (e.g. percent correct).

3. Item parameters are supplied by substantive theory and, thus, person parameter estimates are generated without reference to or use of any data on other persons or populations. Therefore, effects of the examinee population have been completely eliminated from consideration in the estimation of person parameters for reader ability.

9

4. The quantitivity hypothesis can be experimentally tested by evaluating the trade-off property for the individual case. A change in the person parameter can be off-set or traded-off for a compensating change in text complexity to hold comprehension constant. The trade-off is not just about the algebra.

5. When uncertainty in item difficulties is too large to ignore, individual item difficulties may be a poor choice to use as calibration parameters in causal models. As an alternative we recommend, when feasible, averaging over individual item difficulties to produce “ensemble” means. For example empirical text complexities can be excellent dependent variables for testing causal theories.

Eight Features of the Causal Model cont’d.

10

6. Causal Rasch models are individual centered and are explanatory at both within-subject and between-subject levels. The attribute on which I differ from myself a decade ago is the same attribute on which I differ from my brother today.

8. When data fit a Rasch model, differences between person measures are objective. When data fit a causal Rasch model absolute person measures (reader abilities) are objective (i.e. independent of instrument).

9. Causal Rasch models make possible the construction of generally objective growth trajectories. Each trajectory can be completely separated from the instruments used in its construction and from the performance of any other persons, whatsoever.

Eight Features of the Causal Model cont’d.

11

Let’s examine in detail each of the four constructs that make up the Lexile Framework for Reading

Text Complexity Reader Ability Task Difficulty Comprehension

12

May 2016(12th Grade)

1200

1000

1400

1600

Text Demands forCollege and Career

May 2007 – April 2011

347 Encounters138,695 Words3,342 Items983 Minutes

Student 1528

7th GradeMaleHispanicPaid Lunch

Expected: 73.5%Observed: 71.7%

13

14

Mythology Text Complexity Theoretical: 1300L Empirical:

1357L Adapted from Oasis Article courtesy of EBSCO Publishing

The study and interpretation of myth and the body of myths of a particular culture. Myth is a complex cultural phenomenon that can be approached from a number of viewpoints. As generally understood, a myth is a story or narrative that is traditional in a certain culture, having been passed down from early times and regarded as true. It may be said to 1 symbolically the origin of the basic elements and assumptions of a culture. Mythic narratives frequently revolve around the doings of gods or heroes, and may relate, for example, how the world began, how humans and animals came into being, or how certain customs, gestures, or forms of human activities 2. Almost all cultures possess or at one time possessed and lived in terms of myths.

1 immerse belittle portray contradict

2 originated adorned handicapped entwined

15

r = 0.952

r” = 0.960

R2” = 0.921

RMSE” = 99.8L

Figure 1: Plot of Theoretical Text Complexity versus Empirical Text Complexity for 475 articles

“Mythology”

Reliability = .996SEM = 12L

16

What could account for the 8% unexplained variance?

Missing Variables or Theory misspecification Better Criterion Variable – task specificity

hypothesis Improved Proxies/Operationalizations Expanded Error Model – Treat Item Type as

Random Rounding Error Imperfections in Theory Implementation

17

18

Connect reading to postsecondary demands—

www.lexile.com/toefl

19

Examine growth overtime—

20

Task Plane

Different task types may vary in average difficulty and in unit size. Adjustments for added easiness or hardness and adjustments for unit size (Humphry, 2011) can be made to bring new task types into the Lexile Frame of Reference.

21

Task Plane

0 Added HardnessAdded Easiness

Reference Value(Native Item)

Unit Size

22

How Temperature and Pressure Relate Under Constant Volume

Temperature Volume Pressure

2000°K 20 Liters 20.0 atm

1000°K 20 Liters 10.0 atm

500°K 20 Liters 5.0 atm

250°K 20 Liters 2.5 atm

125°K 20 Liters 1.25 atm

23

Comprehension Rates for Readers of the Same Ability with Texts of Different Complexity or How Reader Ability and Comprehension Rate Relate Under Varying Text Complexity

ReaderAbility

TextComplexit

yText Titles

Comprehension Rates

1000L

1000L

1000L

1000L

1000L

500L

750L

1000L

1250L

1500L

The Magic School Bus, Inside the

Earth (Cole)

The Martian Chronicles (Bradbury)

The Reader’s Digest

The Call of the Wild (London)

On Equality Among Mankind

(Rousseau)

96%

90%

75%

50%

25%

Comprehension Rates for Fixed Reader Ability

24

Comprehension Rates for Readers of Different Ability with Texts of the Same Complexity or How Reader Ability and Comprehension Rate Relate Under Constant Text Complexity

Reader Ability Classroom Textbook Comprehension Rates

500L750L

1000L1250L1500L

1000L1000L1000L1000L1000L

25%50%75%90%96%

Comprehension Rates for Fixed Text Complexity

25

Testing the Lexile Theory

How closely does Observed Comprehension (success rate) correspond to what the Lexile theory predicts?

26

27

Oasis: Usage Report by Reader Lexile

28

29

Oasis: Usage Report By Category of Article

30

To causally explain a phenomenon [a measurement outcome] is to provide information about the factors [person processes and instrument mechanisms] on which it depends and to exhibit how it depends on those factors. This is exactly what the provision of counterfactual information…accomplishes: we see what factors some explanandum M [measurement outcome, raw score] depends on (and how it depends on those factors) when we have identified one or more variables such that changes in these (when produced by interventions) are associated with changes in M (Woodward, 2003, p.204).

31

How Many Ways Can We Say X Causes Y?

X “elicited a greater” Y

X “impacts” Y

X “accounts for” Y X “has been linked to” Y

Y “is the result of” X X “didn’t diminish” Y

Y “because of” X Y “depends on” X

X “has led to” Y X “largely motivates” Y

Y “stemmed from” X X “proved critical to” Y

X “fosters” Y X “changes” Y

X “triggers” Y X “affects” Y

32

A. Jackson Stenner CEO, MetaMetricsjstenner@Lexile.com

Contact Info: