erickson: large-scale assessment of oral language proficiency - a swedish example

LukSuS project Helsinki, 30 November 2013

Gudrun Erickson University of Gothenburg, Sweden [email protected] 1

Helsinki, 30 November 2013

LukSuS project

Large-scale Assessment of Oral Language Proficiency

— A Swedish Example

Gudrun Erickson

University of Gothenburg, Sweden [email protected]

Overview

Context – “Facts about Sweden”

Why? – Purpose(s)

What? – Construct

How? – Methods & Models

When? – Timing and Sequencing

Who? – Actors

And…? – Uses, Consequences, ‘Issues’…

EALTA



One of the protagonists about oral testing

It’s always good to be able to pronounce words. If you’re in

England for example, you can’t go around and write notes to

everybody you see…

Swedish 15-year-old about the oral test of English for grade 9

The Swedish School System • 9-year compulsory school; 98 % continue to three-year upper

secondary education (c. 75 % graduate)

• National curricula and syllabuses for subjects

• Grading from year 6 (students 11-12 years of age)

• Goal/criterion referenced grading system; six grade levels (A-E + F)

• Teachers responsible for marking national tests, and for final grading

• Grades high stakes – used for selection for higher education

• Extensive national testing programme – advisory function (however, quantitatively undefined) – No exams

• 4 mandatory tests of EFL: ages 12, 16 // 17, 18 (approx. A2, B1, B1+, B2)



[Current] Aims of national tests

• To enhance fairness and equity in assessment and grading

• To provide data for local, regional and national quality control

“The system may also contribute to”

• clarifying and operationalizing the national syllabus

• enhancing students’ learning and attainment of national objectives

A collaborative approach to test development

• Analyses of relevant literature and research

• Continuous work in broad groups of experts (teachers, teacher educators, special needs teachers, researchers, L1 speakers…)

• Small-scale piloting > Adjustments (an iterative process)

• Large-scale pre-testing in randomly selected groups of students in the country (n ≈ 400)

• Analyses of results and of students’ and teachers’ views and suggestions

• Compilation of tests (reference groups)

• Standard setting and benchmarking in broad groups

• Analyses / Research / Reporting



Focal areas in the national syllabuses of FL A functional, action oriented approach (cf CEFR)

• Receptive competence (listening and reading)

• Productive competence (speaking and writing)

• Interactive competence (speaking and writing)

• Intercultural communicative competence

• Strategies; Adaptation (to purpose, context & receiver)

The typical national FL test Four subtest (LC & RC aggregated into one score)

•  Oral production and interaction

•  Reception: Listening comprehension

• Reception : Reading comprehension

•  Written production (and interaction) — — — — — — — — — — — — — — — —

• Strategies and Adaptation (to purpose, content and receiver) integrated • “Culture”: an integrated aspect of test content and design

• Reflection, e.g. self assessment, focused upon in separate sections/materials and/or in student profiles

• Results / Sub-scores presented in profiles • Model provided for aggregation into total score



Oral proficiency in the national syllabuses (upper secondary level)

• Clear distinction between production and interaction

• Speaking and Writing combined

• “CENTRAL CONTENT”: …Oral and written production and interaction of different kinds, including more formal contexts, where students instruct, tell about things, summarise, explain, comment, evaluate, justify their views, discuss and argue. – Strategies for contributing to and actively participating in discussions related to societal and working life. – Revision and modification of their own and others' oral and written communication in order to vary, clarify and specify, as well as to create structure and adapt it to purpose and situation.

• PERFORMANCE STANDARDS: …In oral and written communication of different genres, students can express themselves in relatively varied ways, relatively clearly and relatively coherently. Students can express themselves with some fluency and in a way that is to some degree adapted to purpose, recipient and situation. Students revise and make improvements to their communications. In oral and written interaction in different, including more formal, contexts, students can express themselves clearly and with some fluency and some adaptation to purpose, recipient and situation. In addition, students can choose and use broadly functional strategies that will, to some extent, solve communication problems and improve interaction.

The Development of oral tests of EFL in Sweden

• Offered as a complement to the national tests since the late 1980s

• Mandatory in grade 9 since 1998, in upper sec. since 2002

* * * • One to one (teacher + student) 1991-1993

• Student interactive (pairs) + one to one 1994-1997

• Student interactive (paired orals) 1998- Production + Interaction; Often three parts: Warm up > Production – Argue – Discuss (10-20 mins per pair, depending on age, level etc.)

* * *

Interrogation… – Interview… – Interaction…



Typical themes of oral tests

Year 5/6 (≈ A2) • What Do You Think?

• The International Talk Show

Year 9 (≈ B1) • The World Around Us

• Quality of Life

English 5 (≈ B1+) • Doing the Right Thing

• Skills for Life

English 6 (≈ B2 )

• Integrity and Technology • Life Balance



Holistic rating supported by analytical factors

• CONTENT Clarity – Complexity and variation (different examples and perspectives) –

Coherence and structure – Adaptation to purpose, recipient, situation

and genre

• LANGUAGE AND /SELF/ EXPRESSION

Communicative strategies (to develop and advance a conversation; to solve

linguistic problems, e.g. by rephrasing, explaining and clarifying) – Fluency and

ease – Range, variation, complexity, clarity and accuracy (vocabulary,

phraseology and idiomatic expression; pronunciation and intonation; grammatical

structures) – Adaptation to purpose, recipient, situation and genre

Materials to support teachers

• General guidelines

• Performance standards

• Analytical factors

• Authentic, graded benchmarks with extensive

comments related to the syllabus, standards and

factors

[Year 6: Extensive training material]



Some results ( % )

Grade level Year 6 Year 9 En 5 En 6

F 5 3 4 1

>E 78 81 79 86

A 18 18 16 18

Teacher Questionnaire (oral test)

Good/Ok test 98 95 95 99

Recording 46 46 49 50

Correlations between raters

NB: Based on benchmarking data

Average Lowest Highest

Year 6 (11) .75 .53 .95

Year 9 (12) .90 .82 .97

E 5 (10) .85 .59 .98

E 6 (8) .80 .53 .93



Co-Rating ?

The teacher as the only rater

Subtest Year 6 Year 9 En 5 En 6

A (oral) 58 52 55 57

B (rec) 44 42 57 57

C (writ) 17 10 13 11

Opinions about oral tests & testing

• Topics good and relevant / Nice things to talk about

• Positive reflection of the syllabus / Important to know/learn

• Students really enjoy doing it / Fun to do!

* * * *

• Practical difficulties; time consuming

• No ‘visible’ time for co-rating

• They never stop talking!



ISSUES… • Practicalities (incl. time)

• Teacher present or absent?

• Recording?

• Co-rating – how?

• Interlocutor/Partner effects

• Preparation time?

• Type of language – tasks – content…

• Balance production – Interaction

• Can teachers’ ratings be trusted – in general + of their own

students’ performances…?

EALTA* Guidelines for Good Practice in Language Testing and Assessment

Three categories/“audiences”:

• Teacher pre- and in-service training • Classroom assessment

• Test development in national or institutional testing units or centres

Basic principles the same for all categories: Respect for students/examinees, responsibility, fairness, reliability, validity, collaboration among the parties involved

* European Association for Language Testing and Assessment

www.ealta.eu.org



European student 2005:

“I think that a good language test/ assessment should get students chance to show what they know and devolope their skills. It should contain of listening and reading comprehations, as well as writing and grammar part. But the most important thing is to speak in that language, so I think there should be also oral exams. Speaking is the most important, because without it, we wouldn’t communicate. Because of that students should also learn pronunciation.”

“ Now have I not so much more to tell you… ”

! ! ##

THANK YOU ! TACK FÖR MIG !#

erickson: large-scale assessment of oral language proficiency - a swedish example

Education