results you can rely on - questionmark · results you can rely on: what we’ve learned from 25...
TRANSCRIPT
Results You Can Rely On:
What We’ve Learned from 25 Years of
Criterion-Referenced Measurement
Sharon Shrock & Bill Coscarelli
CRT Time Line
1930 1940 1960 1970 19801950
1935
Karl Pearson & R.A. Fisher
L.L. Thurstone
Ralph Tyler
The Love Test (1935)
-Louis Hayward
CRT Time Line
1930 1940 1960 1970 19801950
1935 1963
CRT Time Line
1930 1940 1960 1970 19801950
Robert Glaser
Norm-Referenced Test
Criterion-Referenced Test
CRT Time Line
1930 1940 1960 1970 19801950
1963
Before Glaser Era
StartTeacher
Talks
Teacher
Tests
Teacher
Assigns
Grades
Stop
CRT Time Line
1930 1940 1960 1970 19801950
1963
StartWrite
Objectives
Write
Matching
Test ItemsStopGive Test
Glaser Model
President Johnson
CRT Time Line
1963
1960 1970 1990 2000 20091980
1966
CRT Time Line
1966
1960 1970 1990 2000 20091980
1981
CRT Time Line
1981
1960 1970 1990 2000 20091980
1982
William Angoff
1. Identify judges.
2. Have judges estimate the probability of minimally competent test taker getting each item correct.
3. Choose cut-off score by summing the probabilities.
4. Adjust the cut-off as necessary in light of the consequences of misclassification.
The Angoff Method
Item Probability
1
2
3
4
5
.50
1.00
.50
.75
.25
3.00
Angoff Example
Using Multiple Judges
Item Judge 1 Judge 2 Judge 3
1 .33 .50 .402 .80 .90 1.003 .20 .33 .204 .20 .90 .335 .50 .75 .50
Total 2.03 3.38 2.43
Average the Totals for Each Judge
2.03 + 3.38 + 2.43 = 7.84
7.84 / 3 = 2.61
Cut Score = 2.6
1. D) Wanted to play football
2. B) Gridiron
3. A) Fort Caffee, Arkansas
4. E) Flaming Star
5. C) Fried peanut butter and banana
Graceland Guide Test
Answers
CRT Time Line
1982
1960 1970 1990 2000 20091980
1984
First Mac
ADDIE Model
Start Write Items StopGive Test
1980’s Corporate Model
Star
t
Set
Passing at
80%
Give Test
Did
80%
Score
80%?
Stop
Y
Remediate
N
Mager 80-80 Rule
Another View of Items…
• Intensional Meaning—A precise
definition of a concept in terms of
its characteristics
• Extensional Meaning—The
examples of a concept
Intensional
An endowment insurance policy:
A) is death protection for a specific period of one or more years.
B) gives death protection for a long as the policyholder is alive.
C) pays a sum or income to the policyholder if they live to a certain age.
D) allows the amount of insurance (and the related premiums) to be increased or decreased as needs change, subject to evidence of insurability.
Extensional
Jay is a 28 year old graduate student just beginning
studies for a Ph.D. in philosophy. He has four children.
His wife stays home to care for and school the children.
His assistantship pays $920/month. The best type of
policy for him would be:
A) Term
B) Whole Life
C) Endowment
D) Universal Life
ExtensionalMichael was flying his plane when he encountered a fierce
windstorm just as he was running out of gas on an approach to
landing. Unable to make it to the airport due to the headwind he
landed on South Illinois Avenue just as the student bars were
letting out.
As the students saw the plane and ran for safety they trampled
Dr. Post’s four Bradford Pear trees valued at $1200 each. (Dr.
Post has a $250 deductible on her homeowner’s policy.) Dr. Post
files a claim and:
A) She is compensated at $4800.
B) She is compensated at $2400.
C) Her claim is denied.
Southern Illinois University at Carbondale
TRAVEL EXPENSE VOUCHER
ACCT. NO. AMOUNT ACCT. TITLE FISCAL OFFICER SIGNATURE/DATE
Social Security Number Zip Code Type Code Voucher No.
Voucher Date
Traveler Name and Address - Payee Purpose of Trip (State Briefly)
LAST NAME FIRST NAME MI
Traveler Signature Date
Departed From Arrived At Auto Auto Other Expenses Line
Date Place Time Place Time Mileage Reimb. Trans. Lodging Meals Item Amount Totals
TOTALS
Mode of Travel: University
Plane Vehicle
Personal Vehicle Other
Date of meeting to
Was a registration fee paid?
Yes No
If yes, attach material which gives details of the fee paid.
LESS: AMOUNT NOT ALLOWED
NET AMOUNT OF REQUEST
LESS: TRAVEL ADVANCE
AMOUNT DUE TRAVELER
AMT. DUE UNIV. (ATTACH CHECK)
Date Check No. And/Or Voucher No.
I CERTIFY THAT THE ABOVE STATEMENT IS CORRECT AND IS PAYABLE FROM
THE UNIVERSITY FUND OR ACCOUNT INDICATED ABOVE.
Chairman, Board of Trustees of SIU
Secretary, Board of Trustees of SIU
Some Travel Voucher Rules
• Vouchers must be submitted within two weeks.
• Up to four accounts may be used.
• Mileage rate is 50.5 cents per mile.
• Receipts are required for any expense over $10.00.
• Etc.
What Should The Test Be?
Southern Illinois University at Carbondale
TRAVEL EXPENSE VOUCHER
ACCT. NO. AMOUNT ACCT. TITLE FISCAL OFFICER SIGNATURE/DATE
Social Security Number Zip Code Type Code Voucher No.
123456789 62901 30 Voucher Date
Traveler Name and Address - Payee Purpose of Trip (State Briefly)
LAST NAME FIRST NAME MI ISPI Conference
Shrock Sharon A
108 S. Maple St.
Carbondale, Illinois 62901
Traveler Signature Date
Departed From Arrived At Auto Auto Other Expenses Line
Date Place Time Place Time Mileage Reimb. Trans. Lodging Meals Item Amount Totals
TOTALS
Mode of Travel: University
Plane Vehicle
Personal Vehicle Other
Date of meeting 2-5-96 to 2-9-96
Was a registration fee paid?
Yes No
If yes, attach material which gives details of the fee paid.
LESS: AMOUNT NOT ALLOWED
NET AMOUNT OF REQUEST
LESS: TRAVEL ADVANCE
AMOUNT DUE TRAVELER
AMT. DUE UNIV. (ATTACH CHECK)
Date Check No. And/Or Voucher No.
I CERTIFY THAT THE ABOVE STATEMENT IS CORRECT AND IS PAYABLE FROM
THE UNIVERSITY FUND OR ACCOUNT INDICATED ABOVE.
Chairman, Board of Trustees of SIU
Secretary, Board of Trustees of SIU
XX X
2/5 Carbondale 1030 Washington 1530 119 00 35 38 18 94 Taxi 7 75 181 07
2/6 64 80 36 40 Taxi 4 00 105 20
2/7 64 80 13 00 77 80
2/8 Friends 29 20 29 20
2/9 Washington 1255 Carbondale 1800 119 00 8 30 127 30
Regis. 180 00 180 00
238 00 164 98 105 84 1 91 75 700 57
700 57
700 57
Certification Suite
• Level A – Full model, real world
• Level B – High fidelity simulation
• Level C – Scenario based
• Level D – Memory level
• Level E – Attendance
• Level F – Affiliation
CRT Time Line
1984
1960 1970 1990 2000 20091980
1988
Founding Father
Founding Fathers
Dan Fogelberg
CRT Time Line
1988
1960 1970 1990 2000 20091980
1989
CRT Time Line
1989
1960 1970 1990 2000 20091980
1998
StartCan we
test?
Stop
N
Can we
make
money
testing?
YTest
Y
Stop
N
1990’s Corporate Model
Stop
Start
Is
there
a
test?
StopStop
N
Do I
want to
take the
test?
Y
N
Make
Fuss
Employee Model
StopAre you
joking?
Y
Evaluating Training Programs
Start Pass?
Trainer’s Model
CRTJob Task
Analysis Reward
Y
Remediate/Terminate
N
• Maintain security of test items before the
test
• Discourage conversation unless it is
allowed as part of the test
• Encourage on-site support from
supervisors when tests are sent out for
completion at the work site
Honesty and Integrity in Testing
Signs of the Times310514,158
The Cheating Model Start
Do I
want to
study?
NSteal
the
answer
?
Buy the
answers
?N
Get
Answer
s From
Smarter
Friend?N
Good guesser?
NN
Fail TestStop
Stop
Y
Stop
Y
Stop
Y
Stop
Y
StopY
Are
you
joking?
Y
Look
off
some
one's
test?
N
Item Intimations
Start
Contact
Legal
Test
Purpose?
Make
Money
Certify
Employee
Competence
Established
Industry
Standards?
Industry
Wide
Accountability
Y
Sell Testing
Along w/
Product
N
Do Not
Test
Legal Model
This image and comments are not to be construed as legal advice nor advocating any particular course of action. Please consult legal counsel for further guidance.
N
Test
Individuals
on Paper?
Bury Results
in
Landfill?
Y
Y
Are you
joking?
Test Individuals
in a group
on Paper?
N
N
Make Reasonable
Accommodation?
Y
Are you
joking?
Y
Sued for
ADA
Violation
N
Bury Results
in
Landfill?
Y
Can you insure
Safety of all
Equipment? Give Performance
Test Using
Equipment
N
Sued for Working
with Unsafe
Equipment
N
Y
CRT Time Line
1960 1970 1990 2000 20091980
20001996
Shrock & Coscarelli Model
Content
Validity
Analyze
Job Content
Validity
Cognitive
Items
RatingInstruments
Initial Test
Pilot
Statistical
Pilot
Parallel
Forms
Monitor &
Revise
Cut-off
Scores
Reliability
Inter-Rater
Reliability
Report
Scores
CRT Time Line
2000
1960 1970 1990 2000 20091980
2001
CRT Time Line
2001
1960 1970 1990 2000 20091980
2007
Analyze Job Content
Content Validity of Objectives
Paper and
Pencil
Rating
Instrument
Content Validity of Test Items
Test Pilot
Set Cut-off Score
Reliability
Report
What Have We Learned from
25 Years of
Criterion-Referenced Measurement?
We Have Learned That…
• Test questions that measure jobs have to be above the memory level.
• The conditions component of an instructional objective is the one that is most likely to be misunderstood—and it is more important than the verb itself.
• Using a hierarchical arrangement of the content is your best map for test efficiency and validity.
• Most people require practice and feedback in writing questions above the memory level.
• Traditional NRT test statistics work reasonably well in a CRT world for low and medium stakes tests.
• Organizations want to skip Level 2 evaluations and move directly to Levels 3 and 4.
• The Achilles’ Heel of the Angoff technique is getting SMEs to understand what is meant by the minimally competent master.
We Have Learned That…
• Establishing CRT validity rather than reliability is the most important thing.
• Computers are blurring the boundary between cognitive and performance tests.
• Certification testing can be marketed as part of a product that requires competence to use or maintain.
• A very conservative legal department will paralyze a testing program.
• People’s questions about testing are much more sophisticated now than when began our work.
• HR and legal departments probably have not adjusted their attitudes and policies to the increased sophistication of CRT development
• Quality CRT implementation--with consequences—is best viewed as an Organizational Development intervention.
CRT Time Line
1960 1970 1990 2000 20091980
?
What Do You See?
Credits
Bill Coscarelli
Sharon Shrock
David Hallmon
Nikki Schutte
Jayme Frey
Tonya Shride
Results You Can Rely On:
