An Overview of Skills Diagnosis An Overview of Skills Diagnosis with an application to a Standardized Testwith an application to a Standardized Test

An Overview of Skills Diagnosis

• The objective of “Skills diagnosis” is to use the results of an assessment instrument to classify students in terms of how well they have mastered a set of skills or knowledge domains in regard to a specific course of study or content area.

Skills Diagnosis Skills Diagnosis TerminologyTerminology

• By “Skills”, we mean a set of unobservable discrete (categorical) attributes for each individual.

Examples: a specific set of knowledge domains (geography: capitals, products, ethnic groups, etc.), or a specific set of skills (reading: understanding word meaning, locating information, understanding specific information, connecting information)

Skills Diagnosis Skills Diagnosis TerminologyTerminology

• By “Mastery”, we mean the level of achievement on a skill or knowledge domain in terms of ordered categories, such as “below, meets, and exceeds” standards, or “below average, average, and above average”.

• By “Diagnosis”, we mean using an assessment instrument to produce a skills profile: lists each skill or knowledge domain and the student’s level of mastery on each.

Implementation Framework

• Define purpose of the assessment

• Define the skills

• Develop and Analyze the items

• Select an appropriate model

• Fit model to data & evaluate results

• Develop score reports

Define Assessment Purpose

• Discrete classification vs. Continuous scaling– Impacts test design and item development

• Formative vs. Summative vs. Benchmark– Kinds of feedback from the assessment (skill

granularity; relationship to standards)– Timing of feedback– Users and uses of the feedback

Define the Skills

• Purpose of the assessment

• Number of skills– Level of granularity– Relationship to standards

• Number of categories of “mastery”

• Skill correlations and ordered relationships

• Communication w/ teachers, students, & parents

Item Development & Analysis

• Purpose of the assessment: Purely diagnostic?

• Diagnostically developed items?

• Item analysis– No. of skills per item– Skill interaction: Compensatory vs. Conjunctive– Skill difficulty vs. Item difficulty

• Q matrix coding

Q Matrix

• Relates items to attributes• User-specified Q matrix is merely an initial

estimate

Conjunctive vs. Compensatory

• Conjunctive: Successful application of all skills required for successful item performance

• Compensatory: Successful application of some skills makes up for unsuccessful application of others, resulting in successful item performance.– Disjunctive: Success application of any one skill

yields successful performance on item

Model Selection

• Conjunctive vs. Compensatory

• Number and type of item parameters– Difficulty level of the item

– How well an item discriminates between masters and non-masters of the skills measured by the item

Fit model to data & evaluate results• Interpret estimated model parameters.

– Item parameters– Skill mastery population proportions– Re-evaluate skill definitions

• Compare model-predicted statistics to observed statistics (model fit)– Individual scores– Item-pair correlations

• Estimate classification accuracy

Example: ETS LanguEdge RC Test(By Prof. Eunice Jang, Univ. of Toronto)

• Two forms: 37 & 39 multiple-choice items

• ETS prototype for New TOEFL

• Field test data -- 1350 examinees per form

• Applied as diagnostic Pretest & Posttest in a summer English program.

Purpose of the Assessment

• To aid instruction & learning with estimated skill mastery profiles on agreed upon skills

– Simple Mastery vs. Non-mastery low-stakes decisions desired

– Needed understandable skills & easy to interpret feedback on skills and scores

Examples of LanguEdge RC Skills:

Analyze and evaluate relative importance of information in the text by distinguishing major ideas from supporting details.

Deduce word meaning from context (CDV)Deducing the meaning of a word or a phrase by searching and analyzing text and by using contextual clues appearing in the text.

Determine word meaning out of context (CIV)Determine word meaning out of context with recourse to background knowledge

Summarize major ideas from minor details (SUM)

Education Example

Plots of Performance Difference Between Masters and Non-masters

0

0.2

0.4

0.6

0.8

1

1.2

0 5 10 15 20 25 30 35 40

Items

Prop

ortio

n co

rrect

Masters

Non-masters

Form 1

Model Fit: Score Distribution

0

0.2

0.4

0.6

0.8

1

1.2

1 4 7 10 13 16 19 22 25 28 31 34 37 40

Score

Cum

ulat

ive

prob

abili

ty

Observed

Model estimated

Comparing Students’ Skill Mastery over Time

Dongin

0

0.2

0.4

0.6

0.8

1

1.2

CDV CIV SSL TEI TIM INF NEG SUM MCF

Skills

Pro

bab

ility

Pre-test

Post-test

Comparing Students’ Skill Mastery over Time

Hyung

0

0.2

0.4

0.6

0.8

1

1.2

CDV CIV SSL TEI TIM INF NEG SUM MCF

Skills

Pro

bab

ility

Pre-test

Post-test

DiagnOsis scoring report Student Name: Yoshi LanguEdge Reading Comprehension Test 1

Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

Your Answer √ √ √ 2 1 √ 2 √ 3 1 √ 1 √ √ 1 1 √ 1 √ 2 √ √ 2 √ +1 3 √ 1 2 √ √ 1 3 √ 2 3 +2

Correct Answer 2 3 2 3 3 3 1 1 1 4 4 3 2,4,6 2 3 2 1 3 2 3 4 2 4 1 1,5,6 4 2 2 3 3 1 2 2 2 4 11,5,6

3,7

Difficulty e m e h m m h h m m h m m e m m m e e e e h m m m m e h m e e e e m h m h

Needs improvement Not determined Mastered

Key√ Correcto Omitted+ Plus partial points e = Easy, m = Medium, h = Hard(Difficulty is based on 1372 students’ performanceon this test)

Review Your Answers

Improve Your Skills

Score

You earned 21 out of maximum 41 points.

8 points from 12 easy questions

8 points from 17 medium questions

5 points from 8 hard questions

Skill mastery standing

0 0.5 1

Skill 1

Skill 2

Skill 3

Skill 4

Skill 5

Skill 6

Skill 7

Skill 8

Skill 9

Reading Skill

Probability

ScoringCorrect answer to questions with 4 choices = Plus 1 pointWrong or omitted answer = No pointQ13 & 25: 3 correct = 2 points, 2 correct=1 pointQ37: 5 correct=3 points, 4 correct=2 points, 3 correct = 1 point

• Nine primary reading skills are assessed in thisreading comprehension test. Please review skill descriptions and example questions attached to this scoring report.

• The graph on the left side shows your probablemastery standing of each skill.

• The grey region indicates that your probable mastery standing cannot be determined.

• There may be some measurement error associated with the classification.

• This diagnostic information can be more usefulwhen used in combination with your teacher’s and your own evaluation of your reading skills.

How to Interpret Skill Mastery

1

2

Question 1 2 3 4 5 6 7 8 9 101112 13 1415161718192021222324 25 2627282930313233343536 37

Your Answer √

√ √

2 1 √ 2 √ 3 1 √ 1 √ √ 1 1 √ 1 √ 2 √ √ 2 √ +1 3 √ 1 2 √ √ 1 3 √ 2 3 +2

Correct Answer

2 3 2 3 3 3 1 1 1 4 4 32,4,6

2 3 2 1 3 2 3 4 2 4 11,5,

64 2 2 3 3 1 2 2 2 4 1

1,5,6

3,7

Difficulty e m e h m m h h m m h m m e m m m e e e e h m m m m e h m e e e e m h m h

Review Your Answers

ScoreYou earned 21 out of maximum 41 points.

8 points from 12 easy questions 8 points from 17 medium questions 5 points from 8 hard questions

Skill mastery standing

0 0.5 1

Skill 1

Skill 2

Skill 3

Skill 4

Skill 5

Skill 6

Skill 7

Skill 8

Skill 9

Reading Skill

Probability

DiagnOsis scoring report

Primary Skill Descriptions and Example Questions

Skill Skill DescriptionsExample Questions

1Understand the meanings of words or phrases by searching and analyzing surrounding text and using contextual clues appearing in the text. With this skill, you can determine which option word or phrase has the closest meaning to the author’s intended meaning by making use of clues appearing in the text.

33, 14, 32, 4, 3, 11

2Determine word meanings by identifying which option word or phrase has the closest meaning to the referenced word or phrase in the text. Textual clues often do not appear explicitly in the text. With this skill, you can comprehend the text using your prior knowledge of vocabulary.

9, 27, 10, 29, 19, 21, 7

3Comprehend grammatical relationships of words or phrases across successive sentences. With this skill, you can identify words or phrases that particular pronouns refer to OR determine where a new sentence can be inserted without logical or grammatical problems in the text.

3, 26, 12, 36, 4, 2, 22, 33, 24

4Search across sentences within a paragraph and locate relevant information that is clearly stated. Words or phrases in the options often match literally with words or phrases appearing in the relevant section of the text.

22, 18, 30, 17, 8, 24, 36, 20, 12, 25, 14

5

Search across paragraphs and locate relevant information that is not clearly stated in the text. Words or phrases in the options do not match literally with those in the text, but they are paraphrased in different words or phrases. With this skill, you can determine which option most accurately preserves the author’s intended meaning.

6, 34, 26, 4, 5, 35

6Comprehend an argument or an idea that is implied but not explicitly stated in the text OR the author’s rhetorical purpose of mentioning particular phrases in text. With this skill, you can infer information that is implied from the text or can determine the author’s underlying purpose of using particular expressions.

31, 16, 23, 15, 28, 2, 11, 7, 32

7Search and locate relevant information across the text and determine what information is true or not true. With this skill, you can verify which option is true or false based on information stated in the passage or identify key information from negatively stated options (or questions).

22, 7, 28, 5

? 8Identify major ideas by distinguishing them from nonessential information across paragraphs. With this skill, you can distinguish major ideas from ideas that are not mentioned or less important in the text.

13, 5, 17, 25, 20

? 9Recognize major contrasts and arguments presented across paragraphs. With this skill, you can comprehend the organization of the text which often contains the relationships such as compare/contrast, cause/effect, or alternative arguments) and can determine major contrasting ideas or arguments.

37, 23, 35

• Not all example questions are equally informative in assessing related skills. Questions are listed in the order from most informative to least informative for your review. • indicates that these skills are weak areas you need to improve. ‘?’ indicates that your mastery is not determined.

Yoshi4 5

DiagnOsis scoring report Student Name: Yoshi LanguEdge Reading Comprehension Test 2

Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

Correct Answer 2 2 1 1 4 3 4 4 3 1 3 21,3,

43 2 3 2 3 4 1 4 2 1 1 3

3,5,6

2 1 3 3 1 2 4 4 2 3 4 21,2,

5

Your Answer √ √ √ 2 √ o 1 √ 1 √ 2 √ +1 √ 1 √ 3 √ √ 4 2 √ 3 √ 4 +1 3 √ √ √ √ 4 1 3 √ √ 2 √ 0

Difficulty m m m m e m m m h m m m m m h m m e m h h h m e m h h m m m e h m m m m h m m

Review Your Answers

Your Scores

Difficulty Test 1 (06/21/04)

Test 2 (08/02/04)

Easy 8 / total 12 4 / total 4

Medium 8 / total 17 16 / total 26

Hard 5 / total 8 2 / total 9

TOTAL 21 / total 41 22 / total 42

Hi! Find your scores on the left side. Note that the two tests differ at difficulty levels. Minor differences in your scores may be associated with some measurement error.Pay close attention to the summary of reading skills masteryon the next page. Ask yourself the following questions:

Which skill is consistently below 0.4? Which skill is consistently above 0.6? Which skill shows a significant improvement or remained stable?

With two tests, now you can have more confidence about your strengths and weaknesses in reading ability. Use this information wisely. Estimation of your skill mastery standing is not perfectly accurate. Andnot all example questions provided in the skill descriptions are equallyinformative in assessing intended skills.

Making Progress

• Skill 1: Word meanings in context

You can determine which option word or phrase has the closest meaning to the author’s intended meaning by searching and analyzing surrounding text and making use of clues appearing in the text. (1, 7, 24, 5, 29, 18)

• Skill 2: Vocabulary knowledge

You can determine word meanings using your knowledge of vocabulary instead of heavily relying on textual clues.

(3, 17, 37, 32, 27)

• Skill 3: Grammatical relationships of words/phrases

You can comprehend grammatical relationships of words/phrases across sentences. Typical questions assessing this skill involves identifying words or phrases that particular pronouns refer to OR determining where a new sentence can be inserted without logical or grammatical problems. (38, 25, 31, 12, 17, 24, 22)

• Skill 4: Information explicitly stated in the text

You can comprehend information that is explicitly stated in the text by searching across sentences within a paragraph and matching words/phrases with information that is clearly stated in the text. (23, 25, 6, 9, 2, 30, 12)

0

0.2

0.4

0.6

0.8

1

Skill 1 Skill 2

test1

test2

0

0.2

0.4

0.6

0.8

1

Skill 3 Skill 4

test1

test2

Yoshi

• Skill 5: Information NOT explicitly stated in the text

You can comprehend information that is not clearly stated in the text and determine which paraphrased option most accurately preserves the author’s intended meaning.

(36, 16, 10, 4, 19, 7, 8, 34, 20, 14, 21, 22, 15)

• Skill 6: Inference about arguments and author’s purpose

You can make inferences about arguments that are not explicitly stated in the text or determine the author’s underlying purpose of mentioning particular phrases in text. (33, 34, 11, 35, 28, 15, 37 )

• Skill 7: Negation

You can determine what information is true or not true by searching across paragraphs and identifying key information. (19, 9, 33, 14)

• Skill 8: Summarizing major ideas

You can identify major ideas by distinguishing them from nonessential information across paragraphs.

(8, 13, 2, 6, 28, 14, 26)

• Skill 9: Organization of the text and major contrasts

You can determine major contrasts and arguments presented across paragraphs by recognizing the organization of the text which often contains the relationships such as compare/contrast, cause/effect, or alternative arguments. (19, 21, 39, 23, 30)

0

0.2

0.4

0.6

0.8

1

Skill 5 Skill 6

test1

test2

0

0.2

0.4

0.6

0.8

1

Skill 7 Skill 8 Skill 9

test1

test2

Yoshi

Implications

• Diagnostic results welcomed by teachers & students

• Teachers need instruction on how to integrate diagnosis results with lesson plans

• Model seems to be working

• Need to explore compensatory model

Summary

• Skills diagnosis is a Team Effort. – Cognitive psychologists– Psychometricians– Content-matter experts

• Need more validity studies• How to define “mastery”?• How to align tests with classroom teaching?• How to design diagnostic tests?• How to report “scores”?

Skill granularity & refinement:• 32 16 9

• Theoretical defensibility

• Sufficient number of tasks per skill

• Compatibility with modeling assumptions

• Inter-rater agreement on skill coding

• Avg. of 2 skills/item (8 items/skill)

• Unidimensional IRT analysis verified assumed skill difficulties, except for one skill

Model Selection• Conjunctive model: Correct application of all

required skills is presumed necessary for Correct item response.

• Item parameter for each item/skill combination– Ratio of performance of skill non-master to performance

of skill master (desired to be small)

• Ability model– Dichotomous, mastery/non-mastery on each skill

• Estimate is posterior probability of mastery (ppm)

• ppm < .4 non-master ppm > .6 mastery

Fit model to data & Evaluate results

• Joint calibration of two forms– Mastery proportions range: .38-.56; .10 indeterminate

• Interpret item parameters: deleted 13 item/skill combinations (about 9%)– Items did not discriminate for some assigned skills.

• Mastery estimation: Skill mastery consistently improved or remained same pretest to posttest.

Classification Accuracy RateForm 1 Form 2

SkillOverall M NM Overall M NM

CDV 0.87 0.88 0.85 0.92 0.94 0.88

CIV 0.93 0.94 0.92 0.84 0.86 0.80

SSL 0.89 0.89 0.89 0.89 0.90 0.88

TEI 0.94 0.94 0.94 0.88 0.88 0.87

TIM 0.91 0.90 0.91 0.95 0.96 0.94

INF 0.91 0.91 0.91 0.91 0.91 0.90

NEG 0.84 0.83 0.84 0.81 0.83 0.78

SUM 0.90 0.90 0.88 0.89 0.90 0.87

MCF 0.95 0.94 0.95 0.85 0.87 0.83

DiagnOsis Summary report for teachers Teacher: Nancy, B Language Reading Comprehension Test 1

Question 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

Correct Answer 2 3 2 3 3 3 1 1 1 4 4 3 2,4,6 2 3 2 1 3 2 3 4 2 4 1 1,5,6 4 2 2 3 3 1 2 2 2 4 11,5,6

3,7

No of students who correctly answered Q’s

14 9 8 5 10 9 5 9 8 5 6 86=+2

7=+110 6 9 10 11 10 8 12 6 8 7

5=+2

6=+15 10 5 7 10 11 10 8 9 5 9

2=+3

6=+2

1=+1Average difficulty

in class1.0 .6 .6 .4 .7 .6 .4 .6 .6 .4 .4 .6 .7 .7 .4 .6 .7 .8 .7 .6 .9 .4 .6 .5 .6 .4 .7 .4 .5 .7 .8 .7 .6 .6 .4 .6 .5

Average difficulty in population

.9 .6 .8 .4 .7 .6 .4 .4 .6 .7 .4 .6 .6 .7 .7 .7 .7 .7 .7 .9 .5 .5 .7 .5 .6 .6 .7 .5 .6 .7 .7 .7 .8 .7 .3 .7 .4

Review Your Students’ Answers

Hail 27

Chris 20

Gao 30

Shu 22

Gkyung 10

Dongin 32

Hyung 18

Lee 31

Yoshi 21

Ohmi 16

Take 19

Karen 21

Heather 32

Kyung 37

Individual Students’ Scores Review Students’ Skill Mastery

SkillNo of Non-

MastersNot

determinedNo of

Masters

CDV 7 0 7

CIV 7 0 7

SSL 7 1 6

TEI 6 1 7

TIM 9 0 5

INF 8 1 5

NEG 8 1 5

SUM 4 2 8

MCF 5 1 8

See individual students’ report cards to review raw responses and scoring guideline

Reading Comprehension Skills

• Skills-based approach to RC is prominent in the literature.

• No general agreement on a particular set of RC skills – depends on the setting.

Skills Set Construction for LanguEdge RC

• Preliminary Analyses– Task analysis and Textual features– ETS content codes– Nonparametric IRT dimensionality analyses

with field test data (3 skill clusters)

• Think-aloud Protocols (18 processes)