developing the aptis testing system - piap.cl · developing the aptis testing system professor...
TRANSCRIPT
![Page 1: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/1.jpg)
Developing the APTIS Testing System
Professor Barry O’SullivanSenior Advisor, English Language Assessment
![Page 2: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/2.jpg)
Overview
Localisation
Developing theories
Computerisation
Conclusions
![Page 3: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/3.jpg)
DEVELOPING THEORIESconceptualising the test
![Page 4: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/4.jpg)
System Design
![Page 5: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/5.jpg)
Thinking about Test Performance
PERFORMANCETask Language
Target Language
Task Performance Parameters
Social Factors
Personal Characteristics
Cognitive Characteristics
![Page 6: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/6.jpg)
Then There’s Scoring…
Machine or Human
Answer Key or Rating Scale
Selection & Training Consistency
Accuracy
Interpretation
![Page 7: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/7.jpg)
Validation Theory
![Page 8: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/8.jpg)
Decisions, Decisions…
Why are we testing?
![Page 9: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/9.jpg)
How Many Decisions?
TEST SCORE
Task Language•Vocabulary &
Grammar•Genre, Mode, etc.
Target Language•Vocabulary &
Grammar•Function, Mode etc.
Task Performance Parameters•Time & Support•Knowing Assessment
Criteria•Physical Conditions, etc.
Social Factors•Interlocutor [who;
number]•Relative Age, gender
etc.•Familiarity & Status•Etc.
Personal Characteristics•Physical•Psychological•Experiential
Cognitive Characteristics•Maturity•Ability•Etc.
Scoring System•Raters & Rating•Scales & Keys•Analysis & Grading•Etc.
![Page 10: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/10.jpg)
Why all the Fuss?
Curriculumexplicitimplicit
Deliveryroom design
materialsteachers
Assessmentjudgemental
developmental
Decisions
![Page 11: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/11.jpg)
FOCUS ON READINGtheories of language progression
![Page 12: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/12.jpg)
The Reading Model
![Page 13: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/13.jpg)
Word/Sentence
Paragraph
Text [within]
Text [external]
Inter Text [full text level]
Inter Text [paragraph level]
A
B
C
How it Works
![Page 14: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/14.jpg)
Introducing
CORE[Grammar & Vocabulary]
Speaking
Writing
Listening
Reading
![Page 15: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/15.jpg)
LOCALISATIONthe british council approach
![Page 16: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/16.jpg)
Theory to Practice – Test Design
Scoring System
Appropriate Content
Appropriate Task
Appropriate Language
Appropriate Scoring
Appropriate Topic
Test Taker
Test Task
LOCALISATION
![Page 17: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/17.jpg)
Defining Localisation
The process of ensuring that all construct-irrelevant
variance related to the test taking population is
controlled for in the design and delivery of a test
Testing a well defined population within a well defined context in order to make decisions that will apply only to that context
![Page 18: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/18.jpg)
Localisation – Case Study
Curriculum
Delivery Assessment
![Page 19: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/19.jpg)
Case Study - Approach
Context
Population 15 & 17 yrsSecondary SchoolMixed ability, GenderVarious L1 [multilingual]Similar CultureSimilar CurriculumSimilar Delivery
Localisation
Language Reviewed[inc. instructions]
Reading TopicsListening TopicsWriting Topics/TasksSpeaking TopicsSpeaking Images
![Page 20: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/20.jpg)
COMPUTERISATIONthe growing importance of the machine
![Page 21: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/21.jpg)
The Computer & Test Practice
Item DevelopmentSpecificationWritingTriallingBankingArchiving
Test ScoringRater TrainingTest Break UpScoringAnalysing/AwardingScore Reporting
Test TakingKnowledgeReceptive SkillsProductive Skills
ResearchLanguageData AnalysisTask PerformanceRating Performance
System ManagementBack OfficeReporting to Clients
Test DeliveryComputerTabletOther…
![Page 22: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/22.jpg)
The Computer & Test Practice
Item DevelopmentSpecificationWritingTriallingBankingArchiving
![Page 23: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/23.jpg)
CONCLUSIONSwhat’s the point?
![Page 24: Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment. Overview Localisation](https://reader030.vdocuments.us/reader030/viewer/2022040200/5e284986ebf45a33a9257a06/html5/thumbnails/24.jpg)
The Message?
Every decision impacts on ALL stakeholders, though especially on the test taker
Without a clearly defined ABILITY MODEL the test is suspect
Without a clearly defined VALIDATION MODEL the test is suspect
It is close to impossible to deliver any test without technology