test quality check: selecting the right test for the … · (carol chapelle and geoff brindley) 5...
TRANSCRIPT
![Page 1: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/1.jpg)
www.britishcouncil.org 1
Dr Victoria Clark
Assessment Development Manager, East Asia
British Council
Indonesia, March 2018
Test Quality Check:
Selecting the Right Test for the Right Use
![Page 2: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/2.jpg)
2 www.britishcouncil.org
![Page 3: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/3.jpg)
PART 1
3
INTRODUCTION
www.britishcouncil.org
![Page 4: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/4.jpg)
www.britishcouncil.org 4
Assessment
In the context of language teaching and learning,
'assessment' refers to the act of collecting information
and making judgments about a language learner's
knowledge of a language and ability to use it.
(Carol Chapelle and Geoff Brindley)
![Page 5: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/5.jpg)
5
Assessment
Evaluation
Decision making
www.britishcouncil.org
![Page 6: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/6.jpg)
www.britishcouncil.org 6
Test
An educational test is a procedure designed to elicit
certain behaviour from which one can make
inferences about certain characteristics of an
individual
(J. B. Carroll, 1968, p. 46)
![Page 7: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/7.jpg)
Test as a Tool
www.britishcouncil.org 7
![Page 8: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/8.jpg)
PART 2
8
CRITERIA
www.britishcouncil.org
![Page 9: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/9.jpg)
Criteria to consider when selecting tests
1.
Test purpose
2.
Test taker characteristics
3.
Target language use domain
5.
Reliability
6.
Validity
7.
Score reporting
8.
Test administration and practicality
9.
Impact
4.
Skills/competencies
9 www.britishcouncil.org
![Page 10: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/10.jpg)
www.britishcouncil.org 10
![Page 11: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/11.jpg)
Test purpose
Placement
Admissions
Achievement
Diagnostic
Filtering
Proficiency
www.britishcouncil.org 11
![Page 12: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/12.jpg)
Test taker characteristics
Age
Gender
Level of proficiency
First language background
Cultural background
Educational level
Background knowledge
www.britishcouncil.org 12
![Page 13: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/13.jpg)
Target language use domain
Characteristics of the TLU task
Characteristics of the test task
www.britishcouncil.org 13
TLU domain is a “situation or context in which
the test taker will be using the language outside
of the test itself” (Bachmann and Palmer 1996:
18).
![Page 14: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/14.jpg)
www.britishcouncil.org 14
![Page 15: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/15.jpg)
Skills and competencies
What is being measured by the test?
Construct =
theoretical definition of
what it is we want to assess
www.britishcouncil.org 15
![Page 16: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/16.jpg)
“Reliability can be defined as the extent to which the same results
would be obtained on a different occasion, in a different context, or
by a different assessor”(Bartmaan et al., 2007, p12).
Reliability
16 www.britishcouncil.org
![Page 17: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/17.jpg)
Threats to Reliability
www.britishcouncil.org 17
Administration Scoring
Test
![Page 18: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/18.jpg)
Validity
Validity refers to the appropriateness of the inferences and uses that
result from the assessment.
Is the assessment measuring what it is supposed to
measure?
Is the information you have collected the right kind of
evidence for the decisions you need to make or the
intended use?
“Validity is concerned with the clarification and justification of the
intended interpretations and uses of observed scores” (Kane, 2001,
p.339)
18 www.britishcouncil.org
![Page 19: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/19.jpg)
Validity = Evidence
www.britishcouncil.org 19
![Page 20: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/20.jpg)
20
Publications
www.britishcouncil.org
![Page 21: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/21.jpg)
21 www.britishcouncil.org
![Page 22: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/22.jpg)
22 www.britishcouncil.org
![Page 23: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/23.jpg)
www.britishcouncil.org 23
![Page 24: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/24.jpg)
Score reporting
Description/explanation
to aid interpretation?
When?
How are the scores reported?
- Numerical scores
- CEFR
- Subtest scores
- Overall score
www.britishcouncil.org 24
![Page 25: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/25.jpg)
Test administration and practicality
Practicality: best use of resources, time and costs
• Ease of administration
• Instructions
• Invigilated?
• Computer-based/pen and paper
• Number of versions
• Preparation and test practice
www.britishcouncil.org 25
![Page 26: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/26.jpg)
Impact
www.britishcouncil.org 26
![Page 27: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/27.jpg)
Criteria to consider when selecting tests
1.
Test purpose
2.
Test taker characteristics
3.
Target language use domain
5.
Reliability
6.
Validity
7.
Score reporting
8.
Test administration and practicality
9.
Impact
4.
Skills/competencies
27 www.britishcouncil.org
![Page 28: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/28.jpg)
PART 3
28
CONCLUSION
www.britishcouncil.org
![Page 29: Test Quality Check: Selecting the Right Test for the … · (Carol Chapelle and Geoff Brindley) 5 Assessment Evaluation Decision making . ... 2001, p.339) 18 . Validity = Evidence](https://reader031.vdocuments.us/reader031/viewer/2022030913/5b5d7bd77f8b9a9c398e3d8b/html5/thumbnails/29.jpg)
Fitness for purpose
www.britishcouncil.org 29