nato bat testing: the first 200
DESCRIPTION
NATO BAT Testing: The First 200. BILC Professional Seminar 6 October, 2009 Copenhagen, Denmark Dr. Elvira Swender, ACTFL. This Report. History of Benchmark Advisory Tests (BAT) 2009 Administration of BAT in 4-Skills BAT Scores Comparing National Scores to Benchmark Scores Observations. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/1.jpg)
NATO BAT Testing: The First 200
BILC Professional Seminar
6 October, 2009
Copenhagen, Denmark
Dr. Elvira Swender, ACTFL
![Page 2: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/2.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT in 4-Skills
3. BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 3: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/3.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT
3. Combined BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 4: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/4.jpg)
Why Benchmark Testing?
• To provide an external measure against which nations can compare their national STANAG test results
• To promote relative parity of scale interpretation and application across national testing programs
• To standardize what is tested and how it is tested
![Page 5: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/5.jpg)
BAT History
• Launched as a volunteer, collaborative project – The BILC Test Working Group
• 13 members from 8 nations• Contributions received from many other nations
– The original goal was to develop a Reading test
• Later awarded a competitive contract by ACT– December, 2006
![Page 6: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/6.jpg)
BAT History (cont’d)
• ACTFL working with BILC Working Group– To develop tests in 4 skill modalities.
• Reading and Listening tests piloted and validated
• Speaking and Writing tests developed– Testers and raters trained and certified
• Test administration and reporting protocols developed
• 200 BAT 4-skills tests allocated under the contract
• Tests administered and rated• Scores reported to Nations
![Page 7: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/7.jpg)
BAT Reading and Listening Tests
• Internet-delivered and computer scored• Criterion-referenced tests
– Allow for direct application of the STANAG Proficiency Scale • Each proficiency level is tested separately
– Test takers take all items for Levels 1,2,3– 20 texts at each level; one item with multiple choice responses
per text• The proficiency rating is assigned based on two separate
scores– “Floor” – sustained ability across a range of tasks and contexts
specific to one level– “Ceiling” – non-sustained ability at the next higher proficiency
level
![Page 8: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/8.jpg)
BAT Speaking Test
• Telephonic Oral Proficiency Interview– Goal is to a produce a speech sample that best demonstrates
the speaker’s highest level of spoken language ability across the tasks and contexts for the level
• Interview consists of– Standardized structure of “level checks” and “probes”– NATO specific role-play situation
• Conducted and rated by one certified BAT-S Tester– Independently second rated by a separate certified tester or
rater
• Ratings must agree exactly– Level and plus level scores are assigned– Discrepancies are arbitrated
![Page 9: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/9.jpg)
BAT Writing Test
• Internet-delivered• Open constructed response• Four, multi-level, prompts
– Prompts target tasks and contexts of STANAG levels 1,2,3
– NATO specific prompt• Rated by a minimum of two certified BAT-W
Raters– Ratings must agree exactly– Level and plus level scores are assigned– Discrepancies are arbitrated
![Page 10: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/10.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT battery
3. Combined BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 11: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/11.jpg)
2009 BAT Administration
• Allocation to 11 Nations– 8 Nations have completed testing
• Testing began in May, 2009
• Tests administered by LTI, the ACTFL Testing Office
![Page 12: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/12.jpg)
2009 BAT Administration
• Each Nation has a customized client site– Request tests
– View and print test schedules
– Obtain test administration instructions, passwords, and test codes
– Retrieve Ratings
![Page 13: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/13.jpg)
]
![Page 14: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/14.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT
3. Combined BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 15: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/15.jpg)
Total Number of BAT Scores
Skill BAT
Listening 119
Speaking 115
Reading 119
Writing 115
![Page 16: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/16.jpg)
BAT Scores by Level Cumulative
3
12
19
15
21
49
0
10
22
39
29
5
1
1113
16
12
66
0
11
28
51
22
3
0
10
20
30
40
50
60
70
Listening Speaking Reading Writing
0+
1
1+
2
2+
3
![Page 17: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/17.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT
3. Combined BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 18: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/18.jpg)
Alignment of National Scores and BAT Scores
Listening Speaking Reading Writing
Black (5) 40% (7) 29% – – – –
White (11) 64% (18) 56% (13) 92% (18) 39%
Red (18) 89% (18) 83% (18) 83% (18) 50%
Blue (20) 85% (19) 47% (20) 55% (20) 60%
Maroon (16) 69% (15) 47% (14) 64% (18) 50%
Purple (12) 8% – – (13) 54% – –
Yellow (17) 24% (18) 0% (18) 33% (18) 0%
Listening Speaking Reading Writing
![Page 19: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/19.jpg)
This Report
1. History of Benchmark Advisory Tests (BAT)
2. 2009 Administration of BAT
3. Combined BAT Scores
4. Comparing National Scores to Benchmark Scores
5. Observations
![Page 20: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/20.jpg)
Observations – Listening Scores
• Exact agreement of BAT and National Scores is 58%– 69 of the 119 Listening scores agree exactly
• When the scores disagree, the National score is HIGHER 88% of the time
• In 8 cases (7%), disagreement is across two levels – 1 vs 3 and 2 vs 4
![Page 21: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/21.jpg)
Observations – Speaking Scores
• Exact agreement of BAT and National Scores is 46%– 53 of 115 Speaking scores agree exactly
• When the scores disagree, the National score is HIGHER in all cases
• In 6 cases (6%),the disagreement is across two levels– 1 vs 3 and 2 vs 4
![Page 22: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/22.jpg)
Observations – Reading Scores
• Exact agreement of BAT and National Scores is 62%– 74 of 119 Reading scores agree exactly
• When the scores disagree, the National score is HIGHER in 85% of the cases
• In 2 cases, the disagreement is across two levels– 1 vs 3
![Page 23: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/23.jpg)
Observations – Writing Scores
• Exact agreement of BAT and National Scores is 38%– 44 of 115 Writing scores agree exactly
• When there is disagreement, the National score is HIGHER in all cases
• In 15 cases, the disagreement is across two levels – 1 vs 3 and 2 vs 4
•
![Page 24: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/24.jpg)
Accounting for Strictness or Leniency
• Testing rehearsed rather than unrehearsed material– Performance vs proficiency
• Inconsistencies in interpretation of the STANAG• When “plus” ratings are not used, the tendency to
award the next higher level rating to a performance that is substantially better than a baseline performance
![Page 25: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/25.jpg)
For Receptive Skills
• Compensatory cut score setting
• Lack of alignment of author purpose, text type, and reader task at level
• Inadequate item response alternatives
![Page 26: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/26.jpg)
For Productive Skills
• Misalignment of test type and test purpose– Ex: list of discrete questions when goal is to
measure spoken language proficiency
• Inadequate tester/rater norming
![Page 27: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/27.jpg)
Plus Ratings
• Within the Level 1 Range– 60% of ratings are 1– 40% of ratings are 1+
• Within the Level 2 Range– 50% of ratings are 2– 50% of ratings are 2+
![Page 28: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/28.jpg)
Profiles
• Only 12 of 115 profiles (10%) were “flat” – 1 1 1 1 (8)– 2 2 2 2 (2)– 3 3 3 3 (2)
• All remaining profiles are mixed
![Page 29: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/29.jpg)
We are all wondering.
What will the future bring?
![Page 30: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/30.jpg)
Let’s hope it’s not
the same kind of anxiety
these early linguists
experienced.
![Page 31: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/31.jpg)
![Page 32: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/32.jpg)
Questions?Questions?
![Page 33: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/33.jpg)
Extra Slides
![Page 34: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/34.jpg)
Side-by-side BAT and National Test Scores
SkillBAT scores
onlyBAT Scores and National Scores
Reading 119 103
Listening 119 100
Speaking 115 95
Writing 115 95
![Page 35: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/35.jpg)
1
1113
1612
66
0
10
20
30
40
50
60
70
0+11+22+3
BAT Scores by Level Reading
Level BAT- R % of Total
0+ 1 1
1 11 9
1+ 13 11
2 16 14
2+ 12 10
3 66 55
Total 119
![Page 36: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/36.jpg)
3
12
19
15
21
49
0
10
20
30
40
50
0+11+22+3
BAT Scores by LevelListening
Level BAT- R % of Total
0+ 3 2
1 12 10
1+ 19 16
2 15 13
2+ 21 18
3 49 41
Total 119
![Page 37: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/37.jpg)
0
10
22
39
29
15
0
10
20
30
40
0+11+22+3
BAT Scores by LevelSpeaking
Level BAT- S % of Total
1 10 9
1+ 22 19
2 39 34
2+ 29 25
3 15 13
Total 115
![Page 38: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/38.jpg)
0
11
28
51
22
3
0
10
20
30
40
50
60
0+11+22+3
BAT Scores by LevelWriting
Level BAT- W % of Total
1 11 10
1+ 28 24
2 51 44
2+ 22 19
3 3 3
Total 115
![Page 39: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/39.jpg)
Comparing Scores by Level Reading
BAT-R National Test
Level 1 23 9
Level 2 23 35
Level 3 55 49
Level 4 - 10
BAT
L1
BAT
L 2
BAT
L3
National
L 19 -
National
L 212 17 6
National
L3 2 5 40
National
L 4 10
![Page 40: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/40.jpg)
Comparing Scores by Level Listening
BAT-L National Test
Level 1 24 12
Level 2 29 28
Level 3 44 52
Level 4 - 8
BAT
L1
BAT
L 2
BAT
L3
National
L 1 10
National
L 28 15 5
National
L36 12 33
National
L 4 2 6
![Page 41: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/41.jpg)
Comparing Scores by Level Speaking
BAT-S National Test
Level 1 28 11
Level 2 52 34
Level 3 15 44
Level 4 - 6
BAT
L1
BAT
L 2
BAT
L3
National
L 111
National
L 214 20
National
L3 4 28 12
National
L 4 4 2
![Page 42: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/42.jpg)
Comparing Scores by Level Writing
BAT-W National Test
Level 1 35 14
Level 2 57 36
Level 3 3 35
Level 4 - 10
BAT
L1
BAT
L 2
BAT
L3
National
L 114
National
L 216 20
National
L3 5 27 3
National
L 4 10
![Page 43: NATO BAT Testing: The First 200](https://reader035.vdocuments.us/reader035/viewer/2022062720/5681351c550346895d9c77b9/html5/thumbnails/43.jpg)
Listening Speaking Reading Writing
Black (5) 40% (7) 29% – – – –
White (11) 64% (18) 56% (13) 92% (18) 39%
Red (18) 89% (18) 83% (18) 83% (18) 50%
Blue (20) 85% (19) 47% (20) 55% (20) 60%
Maroon (16) 69% (15) 47% (14) 64% (18) 50%
Purple (12) 8% – – (13) 54% – –
Yellow (17) 24% (18) 0% (18) 33% (18) 0%
Alignment of National Scores and BAT Scores