ines paland riedmuller, claudia pop: assessing oral proficiency
TRANSCRIPT
![Page 1: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/1.jpg)
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Assessing Oral Proficiency
Ines Paland-Riedmüller,Deutsch-Uni Online
Claudia Pop,TestDaF-Institut,
g.a.s.t. e.V. Germany
www.eaquals.org
![Page 2: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/2.jpg)
Content of the Workshop
Specifications of speaking The basics of assessment: Purposes,
methods, formats The subjectivity of assessment Quality management tools What is viable for your institution?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 3: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/3.jpg)
articulating (phonetic)
phrasing / formulating (linguistic)
planning & organising (cognitive)
Specifications of speaking
Complexity Pronun-ciation
Time
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 4: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/4.jpg)
The basics of assessment: Purposes
• Summative assessment• used at end of a certain unit of the learning process• ”assessment of learning”
• Formative assessment• used continuously, during the learning process• ”assessment for learning”
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 5: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/5.jpg)
The basics of assessment: Purposes
• Reference group orientated assessment• comparing the test outcome with the group
outcome• ranking, e.g. course
• Criteria orientated assessment• comparing the outcome directly with the aspects
intended to be measured
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 6: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/6.jpg)
The basics of assessment: Methods
• Holistic/global scoring• assessing the performance as a whole• one general mark
• Analytic/profile scoring• decomposing the performance according to
several aspects of interest• several marks
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 7: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/7.jpg)
The basics of assessment: Formats
Direct assessment face-to-face interview one or more candidates one or more raters
Semi-direct assessment recorded tasks recorded performances
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 8: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/8.jpg)
The basics of assessment: Formats
Eaquals International Conference, Lisbon, 21 – 23 April 2016
situation face to face, media transmittedreal communication, simulated communication
partners interlocutor, assessor, co-candiate(s)real person, avatar
activities production, interaction, spontaneous, reflected
number of partners
monologue, dialogue, multilogue
input material
aural, written, visual, graphical
rating simultaneous, posteriorhuman, (semi-) automated
![Page 9: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/9.jpg)
Which aspects actually influence
our decision making?
An example
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 10: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/10.jpg)
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
X
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 11: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/11.jpg)
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 12: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/12.jpg)
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 13: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/13.jpg)
X
Is the grey surface of object
X in shape and size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 14: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/14.jpg)
X
Is the grey
surface of object X in shape and
size identical with the grey
surface of object A, object B or
object C?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 15: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/15.jpg)
We have built-in perceptions of
geometrical forms.
How much stronger might this effect be in assessing learners´ proficiency?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 16: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/16.jpg)
The subjectivity of assessment
• Rudolf Weiss, 1965• main question: How large is the variability in
teachers´ assessment?• rating of one student´s essay by 92 teachers
independently, criteria: • correctness• style• content• general mark
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 17: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/17.jpg)
The subjectivity of assessment
Eaquals International Conference, Lisbon, 21 – 23 April 2016
Mark Correctness Style Content
1 7% 24% 26%
2 28% 41% 47%
3 39% 19% 20%
4 22% 14% 7%
5 4% 2% 0%
Weiss, 1965
![Page 18: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/18.jpg)
The subjectivity of assessment
1 2 3 4 5
-10%
0%
10%
20%
30%
40%
50%
60%
10%
45%
35%
10%
0%
General mark
Weiss, 1965
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 19: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/19.jpg)
The subjectivity of assessmentReplication of the Weiss-study: Birkel & Birkel, 2002
1 2 to 3+ 3 to 4+ 4 to 5+ 50
10
20
30
40
50
60
1
18
51
16
2
Birkel & Birkel, 2002
![Page 20: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/20.jpg)
The subjectivity of assessment
Assessment “errors”:• halo effect• contamination effect• serial effect• central/ extreme tendency• personal dispositions
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 21: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/21.jpg)
Serious reliability problem – what can we do?
Standardise!
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 22: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/22.jpg)
Quality management tools
trainings calibration material monitoring system rating grids
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 23: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/23.jpg)
Tools: Trainings
• Face-to-Face-Interview • standardise the procedure• discuss appropriate behaviour• analyse recorded material• practise: Role play
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 24: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/24.jpg)
Tools: Trainings
• Assessment• recollect the goal (construct)• introduce criteria • operationalise the criteria• assess in groups and discuss the
results • raise error awareness• discuss problems
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 25: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/25.jpg)
Tools: Calibration material
• Guidelines• manual, script, interlocutor frame • best practice example of a face to face
interview• Bench marks
• typical performances for each level• calibrated assessment (expert group)• framework of expectation
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 26: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/26.jpg)
Tools: Monitoring
• two or more raters for the same performance
• (peer) observation• video analysis
constructive feedback
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 27: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/27.jpg)
Tools: Rating grids
• “dare” to modify the CEFR according to your context and aim of assessment
• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 28: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/28.jpg)
![Page 29: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/29.jpg)
Descriptors should be
• positive • short • accurate/ precise• independent of each other
![Page 30: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/30.jpg)
Tools: Rating grids
• “dare” to modify the CEFR according to your context and aim of assessment
• e.g. ALTE-Materials for the Guidance of Test Item Writers (module 4)
• evaluate and modify your grid before introducing it
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 31: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/31.jpg)
Feasible?
• What is already done in your institution?
• Which additional tools do you think might be suitable?
Eaquals International Conference, Lisbon, 21 – 23 April 2016
![Page 32: Ines Paland Riedmuller, Claudia Pop: Assessing oral proficiency](https://reader031.vdocuments.us/reader031/viewer/2022030307/58e793461a28abc52e8b53a1/html5/thumbnails/32.jpg)
Thank you
[email protected]@deutsch-uni.com
Eaquals International Conference, Lisbon, 21 – 23 April 2016