compass math test validity april 2011

Compass Math Test Validity April 2011

Mike Orkin, Jo Ann Phillips, Hui ZhangOffice of Institutional Research

Peralta Community College District

1

Compass Math Test• Peralta students take assessment tests for

placement into Math, ESL, and English courses. • Course placement based on test scores is

recommended not mandatory. • A student may take courses without taking

placement tests.

2

Compass Math Test • Peralta uses the ACT Compass test for math

placement.• When students take Compass, they advance

through “subtests” until they reach a “placement domain.”

• The higher the placement domain, the higher the difficulty level of the recommended courses.

3

Data• Students taking Compass-designated math courses

in F08, S09, F09, S10. • Took Compass test immediately preceding

semester when they took math course, or• Never took Compass test prior to semester when

they took math course.

4

Data Totals• 12,657 student enrollments. • 4,915 (39%) took Compass test.• 7,742 (61%) did not.

5

Data by Placement Domain Courses

Placement Domain Courses Compass

No Compass Total

% Take Compass

MPRAL 3,749 2,644 6,393 59%

MALGE 755 5,268 6,023 13%

MTRIG 179 643 822 22%

MCOAL 232 3,590 3,822 6%

6

Placement Domains and Key Courses• MPRAL – Prealgebra

Arithmetic – Math 250 (0-32)Pre-Algebra – Math 253 (33-39)Elementary Algebra – Math 201 (40-100)

7

Placement Domains and Key Courses• MALGE – Algebra.

Elementary Algebra – Math 201 (0-34)Intermediate Algebra – Math 203 (35-49)Statistics – Math 13 (50-100)

8

Placement Domains and Key Courses• MTRIG – Trigonometry.

Trigonometry – Math 50 (0-45)Pre Calculus – Math 1 (46-100)

9

Placement Domains and Key Courses• MCOAL – College Algebra.

Statistics – Math 13 (0-34)Pre Calculus – Math 1 (35-49)Calculus – Math 3A (50-100)

10

Compliance Within Placement Domain• Most complied with cutoff score assignment

within their placement domain.

11

Compliance Within Placement Domain

Placement Domain Compliance Total %MPRAL COMPLY 3,264 87% NONCOMPLY 485 13%MPRAL Total 3,749MALGE COMPLY 682 90% NONCOMPLY 73 10%MALGE Total 755MTRIG COMPLY 125 70% NONCOMPLY 54 30%MTRIG Total 179MCOAL COMPLY 169 73% NONCOMPLY 63 27%MCOAL Total 232

Grand Total 4,915

12

Validity• A test is “valid” if it measures what it’s supposed

to measure. • We looked at this in two ways:

I. Are grades and success positively related to Compass scores?

II. Do students who take Compass do “better” than students who don’t take Compass?

13

Validity• Are grades positively related to Compass scores?

Analysis of Variance (ANOVA) determines if there are significant differences in average Compass scores across course grades.

On average, as grades increase, so do scores.

14

Validity ANOVA - Assessment test averages by grade.

Placement Domain A - Test ave B - Test ave C - Test ave D - Test ave F/W- Test ave Sig.

MPRAL 38.49 35.77 34.41 34.97 32.58 0.000

MALGE 48.77 44.73 43.33 41.36 44.51 0.014

MTRIG 39.55 39.03 38.62 37.45 36.56 0.401

MCOAL 71.87 69.6 65.23 64.43 64.77 0.037

ALL 44.12 39.35 37.25 36.54 34.53 0.000

15

Validity Grade Counts

Placement Domain A B C D F W Total

MPRAL 767 569 471 275 504 1,107 3,693

MALGE 239 139 113 47 70 144 752

MTRIG 69 39 26 11 9 25 179

MCOAL 127 43 22 7 10 21 230

ALL 1,202 790 632 340 593 1,297 4,854

16

Validity• Do assessment test scores predict a student’s

chance of success (C or better) in a course?

Logistic regression estimates the probability of success for a given assessment test score.

For all subtests, as scores get higher, the predicted chance of success increases, indicating predictive validity.

17

Validity

18

Validity

19

Validity

20

Validity

21

Validity

22

Perfect Success• We have shown that the chance of success

increases as Compass scores increase.• However, if the assessment process works

perfectly, there will be an appropriate course for every student, even those who get low Compass scores, and so everyone (or almost everyone) will succeed, regardless of their Compass score.

23

Perfect Success• If everyone succeeds, the chance of success will be

100% no matter what the Compass score, and so the regression analysis will not be meaningful.

• In the same way, if there is an appropriate course for everyone, and if everyone succeeds, there will be no low grades, and, more generally, there won’t be a relationship between Compass scores and grades.

24

Perfect Success• In other words, looking for a relationship between

Compass scores and success can be clouded by too much success.

• It thus becomes useful to compare those who take the Compass test and those who don’t.

25

Compass v. no Compass• Do students who take the Compass test do better

than students who don’t take the test?

Assume that those who don’t take the Compass test essentially select a class at random.

26

Compass v. no Compass• For non-Compass students, just by chance, some

strong students will take lowest-level courses (too easy) and some weak students will take higher level courses (too difficult).

• If Compass is properly assessing students, only weaker students will take lowest level courses and stronger students will take higher level courses.

27

Compass v. no Compass• Thus, students who take Compass should do worse

than those who don’t at lowest subtest and better than those who don’t at higher levels.

• This is exactly what we observe in the data.

28

Compass v. no CompassChi Square - Success – Compass Test v. No Test

Placement Domain Courses

Take Test % Success

No Test % Success Sig.

MPRAL 48.2% 50.9% 0.020

MALGE 65.0% 57.5% 0.000

MTRIG 74.9% 56.9% 0.000

MCOAL 82.8% 60.8% 0.000

29

Compass v. no Compass• Similar result when comparing grades of students

who take Compass with those who don’t.

30

Compass v. no Compass

ANOVA - Test v. No Test Course GPA

Placement Domain Courses Test GPA No Test GPA Sig.

MPRAL 2.3 2.6 0.000

MALGE 2.7 2.6 0.245

MTRIG 3.0 2.8 0.083

MCOAL 3.3 2.8 0.000

31

Summary• All four methods of analysis yield statistically

significant results supporting Compass validity.

32

Summary• On average, as grades increase, so do Compass scores.• As Compass scores increase, the predicted chance of

success increases. • Students who take Compass do better than those

who don’t in the higher subtests and worse in the lowest subtest, in both course success and grades.

33