an overview of skills diagnosis with an application to a standardized test
Post on 23-Dec-2015
215 Views
Preview:
TRANSCRIPT
An Overview of Skills Diagnosis An Overview of Skills Diagnosis with an application to a Standardized Testwith an application to a Standardized Test
An Overview of Skills Diagnosis
• The objective of “Skills diagnosis” is to use the results of an assessment instrument to classify students in terms of how well they have mastered a set of skills or knowledge domains in regard to a specific course of study or content area.
Skills Diagnosis Skills Diagnosis TerminologyTerminology
• By “Skills”, we mean a set of unobservable discrete (categorical) attributes for each individual.
Examples: a specific set of knowledge domains (geography: capitals, products, ethnic groups, etc.), or a specific set of skills (reading: understanding word meaning, locating information, understanding specific information, connecting information)
Skills Diagnosis Skills Diagnosis TerminologyTerminology
• By “Mastery”, we mean the level of achievement on a skill or knowledge domain in terms of ordered categories, such as “below, meets, and exceeds” standards, or “below average, average, and above average”.
• By “Diagnosis”, we mean using an assessment instrument to produce a skills profile: lists each skill or knowledge domain and the student’s level of mastery on each.
Implementation Framework
• Define purpose of the assessment
• Define the skills
• Develop and Analyze the items
• Select an appropriate model
• Fit model to data & evaluate results
• Develop score reports
Define Assessment Purpose
• Discrete classification vs. Continuous scaling– Impacts test design and item development
• Formative vs. Summative vs. Benchmark– Kinds of feedback from the assessment (skill
granularity; relationship to standards)– Timing of feedback– Users and uses of the feedback
Define the Skills
• Purpose of the assessment
• Number of skills– Level of granularity– Relationship to standards
• Number of categories of “mastery”
• Skill correlations and ordered relationships
• Communication w/ teachers, students, & parents
Item Development & Analysis
• Purpose of the assessment: Purely diagnostic?
• Diagnostically developed items?
• Item analysis– No. of skills per item– Skill interaction: Compensatory vs. Conjunctive– Skill difficulty vs. Item difficulty
• Q matrix coding
Q Matrix
• Relates items to attributes• User-specified Q matrix is merely an initial
estimate
Conjunctive vs. Compensatory
• Conjunctive: Successful application of all skills required for successful item performance
• Compensatory: Successful application of some skills makes up for unsuccessful application of others, resulting in successful item performance.– Disjunctive: Success application of any one skill
yields successful performance on item
Model Selection
• Conjunctive vs. Compensatory
• Number and type of item parameters– Difficulty level of the item
– How well an item discriminates between masters and non-masters of the skills measured by the item
Fit model to data & evaluate results• Interpret estimated model parameters.
– Item parameters– Skill mastery population proportions– Re-evaluate skill definitions
• Compare model-predicted statistics to observed statistics (model fit)– Individual scores– Item-pair correlations
• Estimate classification accuracy
Example: ETS LanguEdge RC Test(By Prof. Eunice Jang, Univ. of Toronto)
• Two forms: 37 & 39 multiple-choice items
• ETS prototype for New TOEFL
• Field test data -- 1350 examinees per form
• Applied as diagnostic Pretest & Posttest in a summer English program.
Purpose of the Assessment
• To aid instruction & learning with estimated skill mastery profiles on agreed upon skills
– Simple Mastery vs. Non-mastery low-stakes decisions desired
– Needed understandable skills & easy to interpret feedback on skills and scores
Examples of LanguEdge RC Skills:
Analyze and evaluate relative importance of information in the text by distinguishing major ideas from supporting details.
Deduce word meaning from context (CDV)Deducing the meaning of a word or a phrase by searching and analyzing text and by using contextual clues appearing in the text.
Determine word meaning out of context (CIV)Determine word meaning out of context with recourse to background knowledge
Summarize major ideas from minor details (SUM)
Education Example
Plots of Performance Difference Between Masters and Non-masters
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15 20 25 30 35 40
Items
Prop
ortio
n co
rrect
Masters
Non-masters
Form 1
Model Fit: Score Distribution
0
0.2
0.4
0.6
0.8
1
1.2
1 4 7 10 13 16 19 22 25 28 31 34 37 40
Score
Cum
ulat
ive
prob
abili
ty
Observed
Model estimated
Comparing Students’ Skill Mastery over Time
Dongin
0
0.2
0.4
0.6
0.8
1
1.2
CDV CIV SSL TEI TIM INF NEG SUM MCF
Skills
Pro
bab
ility
Pre-test
Post-test
Comparing Students’ Skill Mastery over Time
Hyung
0
0.2
0.4
0.6
0.8
1
1.2
CDV CIV SSL TEI TIM INF NEG SUM MCF
Skills
Pro
bab
ility
Pre-test
Post-test
DiagnOsis scoring report Student Name: Yoshi LanguEdge Reading Comprehension Test 1
Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
Your Answer √ √ √ 2 1 √ 2 √ 3 1 √ 1 √ √ 1 1 √ 1 √ 2 √ √ 2 √ +1 3 √ 1 2 √ √ 1 3 √ 2 3 +2
Correct Answer 2 3 2 3 3 3 1 1 1 4 4 3 2,4,6 2 3 2 1 3 2 3 4 2 4 1 1,5,6 4 2 2 3 3 1 2 2 2 4 11,5,6
3,7
Difficulty e m e h m m h h m m h m m e m m m e e e e h m m m m e h m e e e e m h m h
Needs improvement Not determined Mastered
Key√ Correcto Omitted+ Plus partial points e = Easy, m = Medium, h = Hard(Difficulty is based on 1372 students’ performanceon this test)
Review Your Answers
Improve Your Skills
Score
You earned 21 out of maximum 41 points.
8 points from 12 easy questions
8 points from 17 medium questions
5 points from 8 hard questions
Skill mastery standing
0 0.5 1
Skill 1
Skill 2
Skill 3
Skill 4
Skill 5
Skill 6
Skill 7
Skill 8
Skill 9
Reading Skill
Probability
ScoringCorrect answer to questions with 4 choices = Plus 1 pointWrong or omitted answer = No pointQ13 & 25: 3 correct = 2 points, 2 correct=1 pointQ37: 5 correct=3 points, 4 correct=2 points, 3 correct = 1 point
• Nine primary reading skills are assessed in thisreading comprehension test. Please review skill descriptions and example questions attached to this scoring report.
• The graph on the left side shows your probablemastery standing of each skill.
• The grey region indicates that your probable mastery standing cannot be determined.
• There may be some measurement error associated with the classification.
• This diagnostic information can be more usefulwhen used in combination with your teacher’s and your own evaluation of your reading skills.
How to Interpret Skill Mastery
1
2
Question 1 2 3 4 5 6 7 8 9 101112 13 1415161718192021222324 25 2627282930313233343536 37
Your Answer √
√ √
2 1 √ 2 √ 3 1 √ 1 √ √ 1 1 √ 1 √ 2 √ √ 2 √ +1 3 √ 1 2 √ √ 1 3 √ 2 3 +2
Correct Answer
2 3 2 3 3 3 1 1 1 4 4 32,4,6
2 3 2 1 3 2 3 4 2 4 11,5,
64 2 2 3 3 1 2 2 2 4 1
1,5,6
3,7
Difficulty e m e h m m h h m m h m m e m m m e e e e h m m m m e h m e e e e m h m h
Review Your Answers
ScoreYou earned 21 out of maximum 41 points.
8 points from 12 easy questions 8 points from 17 medium questions 5 points from 8 hard questions
Skill mastery standing
0 0.5 1
Skill 1
Skill 2
Skill 3
Skill 4
Skill 5
Skill 6
Skill 7
Skill 8
Skill 9
Reading Skill
Probability
DiagnOsis scoring report
Primary Skill Descriptions and Example Questions
Skill Skill DescriptionsExample Questions
1Understand the meanings of words or phrases by searching and analyzing surrounding text and using contextual clues appearing in the text. With this skill, you can determine which option word or phrase has the closest meaning to the author’s intended meaning by making use of clues appearing in the text.
33, 14, 32, 4, 3, 11
2Determine word meanings by identifying which option word or phrase has the closest meaning to the referenced word or phrase in the text. Textual clues often do not appear explicitly in the text. With this skill, you can comprehend the text using your prior knowledge of vocabulary.
9, 27, 10, 29, 19, 21, 7
3Comprehend grammatical relationships of words or phrases across successive sentences. With this skill, you can identify words or phrases that particular pronouns refer to OR determine where a new sentence can be inserted without logical or grammatical problems in the text.
3, 26, 12, 36, 4, 2, 22, 33, 24
4Search across sentences within a paragraph and locate relevant information that is clearly stated. Words or phrases in the options often match literally with words or phrases appearing in the relevant section of the text.
22, 18, 30, 17, 8, 24, 36, 20, 12, 25, 14
5
Search across paragraphs and locate relevant information that is not clearly stated in the text. Words or phrases in the options do not match literally with those in the text, but they are paraphrased in different words or phrases. With this skill, you can determine which option most accurately preserves the author’s intended meaning.
6, 34, 26, 4, 5, 35
6Comprehend an argument or an idea that is implied but not explicitly stated in the text OR the author’s rhetorical purpose of mentioning particular phrases in text. With this skill, you can infer information that is implied from the text or can determine the author’s underlying purpose of using particular expressions.
31, 16, 23, 15, 28, 2, 11, 7, 32
7Search and locate relevant information across the text and determine what information is true or not true. With this skill, you can verify which option is true or false based on information stated in the passage or identify key information from negatively stated options (or questions).
22, 7, 28, 5
? 8Identify major ideas by distinguishing them from nonessential information across paragraphs. With this skill, you can distinguish major ideas from ideas that are not mentioned or less important in the text.
13, 5, 17, 25, 20
? 9Recognize major contrasts and arguments presented across paragraphs. With this skill, you can comprehend the organization of the text which often contains the relationships such as compare/contrast, cause/effect, or alternative arguments) and can determine major contrasting ideas or arguments.
37, 23, 35
• Not all example questions are equally informative in assessing related skills. Questions are listed in the order from most informative to least informative for your review. • indicates that these skills are weak areas you need to improve. ‘?’ indicates that your mastery is not determined.
Yoshi4 5
DiagnOsis scoring report Student Name: Yoshi LanguEdge Reading Comprehension Test 2
Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Correct Answer 2 2 1 1 4 3 4 4 3 1 3 21,3,
43 2 3 2 3 4 1 4 2 1 1 3
3,5,6
2 1 3 3 1 2 4 4 2 3 4 21,2,
5
Your Answer √ √ √ 2 √ o 1 √ 1 √ 2 √ +1 √ 1 √ 3 √ √ 4 2 √ 3 √ 4 +1 3 √ √ √ √ 4 1 3 √ √ 2 √ 0
Difficulty m m m m e m m m h m m m m m h m m e m h h h m e m h h m m m e h m m m m h m m
Review Your Answers
Your Scores
Difficulty Test 1 (06/21/04)
Test 2 (08/02/04)
Easy 8 / total 12 4 / total 4
Medium 8 / total 17 16 / total 26
Hard 5 / total 8 2 / total 9
TOTAL 21 / total 41 22 / total 42
Hi! Find your scores on the left side. Note that the two tests differ at difficulty levels. Minor differences in your scores may be associated with some measurement error.Pay close attention to the summary of reading skills masteryon the next page. Ask yourself the following questions:
Which skill is consistently below 0.4? Which skill is consistently above 0.6? Which skill shows a significant improvement or remained stable?
With two tests, now you can have more confidence about your strengths and weaknesses in reading ability. Use this information wisely. Estimation of your skill mastery standing is not perfectly accurate. Andnot all example questions provided in the skill descriptions are equallyinformative in assessing intended skills.
Making Progress
• Skill 1: Word meanings in context
You can determine which option word or phrase has the closest meaning to the author’s intended meaning by searching and analyzing surrounding text and making use of clues appearing in the text. (1, 7, 24, 5, 29, 18)
• Skill 2: Vocabulary knowledge
You can determine word meanings using your knowledge of vocabulary instead of heavily relying on textual clues.
(3, 17, 37, 32, 27)
• Skill 3: Grammatical relationships of words/phrases
You can comprehend grammatical relationships of words/phrases across sentences. Typical questions assessing this skill involves identifying words or phrases that particular pronouns refer to OR determining where a new sentence can be inserted without logical or grammatical problems. (38, 25, 31, 12, 17, 24, 22)
• Skill 4: Information explicitly stated in the text
You can comprehend information that is explicitly stated in the text by searching across sentences within a paragraph and matching words/phrases with information that is clearly stated in the text. (23, 25, 6, 9, 2, 30, 12)
0
0.2
0.4
0.6
0.8
1
Skill 1 Skill 2
test1
test2
0
0.2
0.4
0.6
0.8
1
Skill 3 Skill 4
test1
test2
Yoshi
• Skill 5: Information NOT explicitly stated in the text
You can comprehend information that is not clearly stated in the text and determine which paraphrased option most accurately preserves the author’s intended meaning.
(36, 16, 10, 4, 19, 7, 8, 34, 20, 14, 21, 22, 15)
• Skill 6: Inference about arguments and author’s purpose
You can make inferences about arguments that are not explicitly stated in the text or determine the author’s underlying purpose of mentioning particular phrases in text. (33, 34, 11, 35, 28, 15, 37 )
• Skill 7: Negation
You can determine what information is true or not true by searching across paragraphs and identifying key information. (19, 9, 33, 14)
• Skill 8: Summarizing major ideas
You can identify major ideas by distinguishing them from nonessential information across paragraphs.
(8, 13, 2, 6, 28, 14, 26)
• Skill 9: Organization of the text and major contrasts
You can determine major contrasts and arguments presented across paragraphs by recognizing the organization of the text which often contains the relationships such as compare/contrast, cause/effect, or alternative arguments. (19, 21, 39, 23, 30)
0
0.2
0.4
0.6
0.8
1
Skill 5 Skill 6
test1
test2
0
0.2
0.4
0.6
0.8
1
Skill 7 Skill 8 Skill 9
test1
test2
Yoshi
Implications
• Diagnostic results welcomed by teachers & students
• Teachers need instruction on how to integrate diagnosis results with lesson plans
• Model seems to be working
• Need to explore compensatory model
Summary
• Skills diagnosis is a Team Effort. – Cognitive psychologists– Psychometricians– Content-matter experts
• Need more validity studies• How to define “mastery”?• How to align tests with classroom teaching?• How to design diagnostic tests?• How to report “scores”?
Skill granularity & refinement:• 32 16 9
• Theoretical defensibility
• Sufficient number of tasks per skill
• Compatibility with modeling assumptions
• Inter-rater agreement on skill coding
• Avg. of 2 skills/item (8 items/skill)
• Unidimensional IRT analysis verified assumed skill difficulties, except for one skill
Model Selection• Conjunctive model: Correct application of all
required skills is presumed necessary for Correct item response.
• Item parameter for each item/skill combination– Ratio of performance of skill non-master to performance
of skill master (desired to be small)
• Ability model– Dichotomous, mastery/non-mastery on each skill
• Estimate is posterior probability of mastery (ppm)
• ppm < .4 non-master ppm > .6 mastery
Fit model to data & Evaluate results
• Joint calibration of two forms– Mastery proportions range: .38-.56; .10 indeterminate
• Interpret item parameters: deleted 13 item/skill combinations (about 9%)– Items did not discriminate for some assigned skills.
• Mastery estimation: Skill mastery consistently improved or remained same pretest to posttest.
Classification Accuracy RateForm 1 Form 2
SkillOverall M NM Overall M NM
CDV 0.87 0.88 0.85 0.92 0.94 0.88
CIV 0.93 0.94 0.92 0.84 0.86 0.80
SSL 0.89 0.89 0.89 0.89 0.90 0.88
TEI 0.94 0.94 0.94 0.88 0.88 0.87
TIM 0.91 0.90 0.91 0.95 0.96 0.94
INF 0.91 0.91 0.91 0.91 0.91 0.90
NEG 0.84 0.83 0.84 0.81 0.83 0.78
SUM 0.90 0.90 0.88 0.89 0.90 0.87
MCF 0.95 0.94 0.95 0.85 0.87 0.83
DiagnOsis Summary report for teachers Teacher: Nancy, B Language Reading Comprehension Test 1
Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
Correct Answer 2 3 2 3 3 3 1 1 1 4 4 3 2,4,6 2 3 2 1 3 2 3 4 2 4 1 1,5,6 4 2 2 3 3 1 2 2 2 4 11,5,6
3,7
No of students who correctly answered Q’s
14 9 8 5 10 9 5 9 8 5 6 86=+2
7=+110 6 9 10 11 10 8 12 6 8 7
5=+2
6=+15 10 5 7 10 11 10 8 9 5 9
2=+3
6=+2
1=+1Average difficulty
in class1.0 .6 .6 .4 .7 .6 .4 .6 .6 .4 .4 .6 .7 .7 .4 .6 .7 .8 .7 .6 .9 .4 .6 .5 .6 .4 .7 .4 .5 .7 .8 .7 .6 .6 .4 .6 .5
Average difficulty in population
.9 .6 .8 .4 .7 .6 .4 .4 .6 .7 .4 .6 .6 .7 .7 .7 .7 .7 .7 .9 .5 .5 .7 .5 .6 .6 .7 .5 .6 .7 .7 .7 .8 .7 .3 .7 .4
Review Your Students’ Answers
Hail 27
Chris 20
Gao 30
Shu 22
Gkyung 10
Dongin 32
Hyung 18
Lee 31
Yoshi 21
Ohmi 16
Take 19
Karen 21
Heather 32
Kyung 37
Individual Students’ Scores Review Students’ Skill Mastery
SkillNo of Non-
MastersNot
determinedNo of
Masters
CDV 7 0 7
CIV 7 0 7
SSL 7 1 6
TEI 6 1 7
TIM 9 0 5
INF 8 1 5
NEG 8 1 5
SUM 4 2 8
MCF 5 1 8
See individual students’ report cards to review raw responses and scoring guideline
Reading Comprehension Skills
• Skills-based approach to RC is prominent in the literature.
• No general agreement on a particular set of RC skills – depends on the setting.
Skills Set Construction for LanguEdge RC
• Preliminary Analyses– Task analysis and Textual features– ETS content codes– Nonparametric IRT dimensionality analyses
with field test data (3 skill clusters)
• Think-aloud Protocols (18 processes)
top related