frameworks, methods, and applications in diagnostic ......rohrschach, mmpi, and more diagnostic time...

Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728

Frameworks, Methods, and Applications in Diagnostic Educational Testing

International Symposium on “Diagnostic Testing in Education”

André A. RuppEducational Testing Service (ETS)

[email protected]


Overview of Keynote

• Diagnostic educational testing

• Cross-walk of current landscape

- Cognitive models - Principled assessment design- Validation- Scoring and reporting

• Implications for R&D

2

Diagnostic Educational Testing

Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 4

We clearly want this!


Under appropriate (systemic) circumstances, diagnostic educational testing can:

- Supplement coarse-grained summaries about learners from summative assessments

- Provide fine-grained feedback that is actually useful (“interpretable”, “actionable”)

- Foster openness to new forms of learning and assessment as a result of needed diagnostic information

- Serve as a mechanism for professional development of teachers, administrators, and research teams

5


But it is not as easy as we had thought…


Implementing a successful diagnostic educational testing effort at a system level is very challenging due to:

- Design and implementation complexities for individual diagnostic assessments

- Alignment complexities across all components in educational systems

- Financial and political constraints for individual companies, funders, and governing bodies

- Cultural resistances to aspects such as additional testing time for learners to gather “richer” diagnostic information or automated scoring of performances

7


So we have learned some lessons…

8


Diagnostic testing ≠Diagnostic testing

9


Foci: Attention disorders, learning disabilities, psychiatric problems

Cases: 13-year old child with combined performance issues in mathematics such asstruggles with fluent retrieval of math facts, with the ability to retain new knowledge,and with the calculation or representation of fractions and percentages

Diagnosis: After thorough testing, it may be determined that the child hasdyscalculia and an attention disorder

Testing batteries (about 20 subtests): WISC, Woodcock Johnson Test ofAchievement and Test of Oral Language, Tower of London, Rey-O, WRAML-2, IVA-2,TOWRE-2, OWLS-2, Boston Naming, Autosequencing, CTOPP-2, Nelson-Denney,Rohrschach, MMPI, and more

Diagnostic time (about 30 hours): intake interview (2 hours), batteries (8-12hours), analysis (4-6 hours), writing (10-12 hours), interpretative conference /feedback session for parents (3 hours)

Total cost: $ 4000 - $ 6000

Professional training: Doctorate in clinical psychology or neuro-psychology, postdoc, clinical testing experience

10

Neuropsychological (Diagnostic) Testing


Be Careful to Manage Expectations!

11

Cross-walk of Current Landscape

(Fast Walk)


+ Glossary


Key Themes

• Cognitive, socio-cognitive, and non-cognitive factors • 21st-century / complex performance skills• Learning progressions• Principled assessment design • Validation argumentation• Diagnostic measurement • Learning analytics / process data• Automated scoring• Digital learning environments• Stealth assessment• Accessibility

14

Cognitive Models


New Constructs

16

Figure 1 Viral Internet image, circa 2014.

Shute & Wang (2016)


Framework for Non-cognitive Skills

17

Kyllonen (2016)


New Constructs = New Assessments

18

Oranje et al., Gobert et al., Shute et al., Jackson et al. (2016)

Principled Assessment Design


Three characteristics of principled assessment design:

• Construct-centered nature of approach• Engineering towards intended interpretation and uses• Explicit design decisions and rationales

Three criteria to evaluate the fitness of theories of learning and cognition to inform assessment design and development:

• Clarify the targets of inference• Identify relevant content features for items or tasks• Identify relevant features of learner performance

20

Ferrara et al. (2016)

Nichols et al. (2016)


Several Ways to be Principled

21

• Evidence-centered design (ECD)• Cognitive design system (CDS)• Assessment engineering (AE)• Principled design for efficacy (PDE)• BEAR assessment system (BAS)

They are not methodologically or contextually interchangeable but share certain deep-structure principles and goals


Ferrara et al. (2016)


Gierl & Lai (2016)

Validation


Construct Validity Framework

25

Embretson (2016)


Bachman (2005)

Validation Argumentation Framework


Cognitive Model Validation(Theory)

27

Keehner et al. (2016)


Cognitive Model Validation(Practice)

Keehner et al. (2016)


Nichols et al. (2016)

29

Validation is complex!


Especially for digitally-delivered tasks…

Figure 3 Screenshot of the Inq-ITS system with all phases of inquiry shown.

Gobert & Sao Pedro (2016)

Figure 2 Assessment report for teachers reported out by inquiry practice and sub-skills.

Scoring and Reporting


Alignment with Design (I)

32

Shute & Wang (2016)

Figure 6 CM of creativity and sample indicators; [R] indicates reverse coded.

Figure 1 Viral Internet image, circa 2014.


Alignment with Design (II)

Shute & Wang (2016)

Figure 8 Bayes net fragment for creativity for “Swamp People” level.Figure 7 “Swamp People” level in Physics Playground (a) with typical ramp solution (b), less common springboard solution (c), and rare

lever solution (d) .


Process Data Design and Analysis

Phase 1: Principled design of logfile structure and tasksPhase 2: Alignment of analytic goals with study designPhase 3: Feature generation and aggregationPhase 4: Initial descriptive and inferential analysesPhase 5: Formal measurement and learning analytics modelingPhase 6: Refinement of learning theories, platform design, and

research plans

All of this requires computational infrastructures and investments into professional development of staff!

34


Principled Data Capture from Logfiles

35

Kerr et al. (2016)


Modern Modeling Approaches

• Explanatory item response theory

• Diagnostic classification models (parametric)

• Diagnostic classification approaches (semi-parametric)

• Bayesian inference networks

• Models for repeated-measures data

• Data mining and learning analytics

• Related modeling families

Selected future directions:

- Integrating indicators from process and product data into parametric models with uncertainty estimation

- Advancing methodologies for dynamic predictions within digital environments

- Leveraging subject-matter expertise more smartly for model specification, criticism, and refinement to be generative

36


Principled Criticism and Refinement

37

Harring & Hauser (2016)


Automated Scoring Technologies

• Extended answers / essays• Short content-focused answers• Mathematical responses (numerical, graphical)• Spoken responses• Multimodal responses (combinations of above)• Multimodal responses (novel constructs)• Activity sequences (including chats)• Complex performance products (semantic webs,

flowcharts, designs)

This requires feature development, model training, model evaluation, and score validation!

38

Bejar et al. (2016)

Implications for R&D


Bachman (2005)

Conceptual diagrams are helpful but….


…we need more “PAD Live / PAD Illustrated”

• More “knowledge transfer” from agencies, companies, and research institutions about process, rather than product

• More “worked exemplars” that describe the design decisions, rationales, and processes

• More flexibility in design, prototype, and refine cycles of innovation for assessments

• More generalizable capabilities and architectures that facilitate principled assessment design and scoring work

• More sharable artifacts that can be tagged, annotated, adapted, and adopted

• More principled leveraging of digital technology for learning and assessment

41


This symposium seems like a great step in that direction!

42

You’re Awesome!


Therefore: my excitement about this symposium…

43


Frameworks, Methods, and Applications in Diagnostic Educational Testing

International Symposium on “Diagnostic Testing in Education”

André A. RuppEducational Testing Service (ETS)

[email protected]


+ Glossary

frameworks, methods, and applications in diagnostic ......rohrschach, mmpi, and more diagnostic time...

Documents