Good Practice in Assessing Speaking Carol Spoettl, University of Innsbruck Nivja de Jong, Utrecht University Jayanti Banerjee, CaMLA

Session 2: The task

Session Overview • Construct • Test mandate • Test parameters • Elicitation methods • Task difficulty

Materials

2 handouts 4 activities Each HO used twice for different activities

SIG participants construct Results from activity one

What

How

expressing thoughts respond to situation getting the message through being able to communicate ability to communicate exchange information, opinions, emotions communicate goals make yourself understood communicate ideas & thoughts to others explaining thoughts transmit information (orally)

appropriately meaningful sounds spontaneously effectively accurately logically systematically clearly coherently understandably successfully not too fast, not too slow intelligibly

Frameworks – Nunan 1989 – Candlin 1987 – Weir 1993 – Bachman and Palmer 1996 – Fulcher 2003

for interpreting components of the construct – Teaching focus – Generic testing focus – Specific speaking focus

Frameworks and constructs

Major elements of CEFR model

Language use, embracing language learning, comprises the actions performed by persons who as individuals and as social agents develop a range of competences, both general and in particular communicative language competences. They draw on the competences at their disposal in various contexts under various conditions and under various constraints to engage in language activities involving language processes to produce and/or receive texts in relation to themes in specific domains, activating those strategies which seem most appropriate for carrying out the tasks to be accomplished. The monitoring of these actions by the participants leads to the reinforcement or modification of their competences. [p 10]

CEFR - An action-oriented approach It views users and learners of a language primarily as ‘social agents’, i.e. members of society who have tasks (not exclusively language-related) to accomplish in a given set of circumstances, in a specific environment and within a particular field of action. (CEFR p 10)

The CEFR, the construct and task design

Domains (on pages 48–49) Gives some examples of the above situational categories, classified according to domains, likely to be met in most European countries. The table is purely illustrative and suggestive. It makes no claim to be exhaustive.

CEFR table 5

Key aspects • Authenticity of items and tasks

– Situational authenticity – Interactional authenticity

• Discreteness with which competences are tested

Operationalising the CEFR model

• Situational authenticity • The accuracy with which the task represents language

activities from real life. – Identify key features of the real life task and replicate them

• Interactional authenticity • The naturalness of the interaction between the test taker

and task and the mental processes which accompany it. – Use situations and tasks that are familiar and relevant – Make the purpose for carrying out the task clear, together with

the intended audience, by providing an appropriate contexualisation

– Make clear the criterion for success in completeing the task

Authenticity

Identify the authentic speaking activity [CEFR] Identify an appropriate domain linked to that activity [familiar and relevant] Create a task that the candidate has to accomplish Stipulate the successful outcome

Possible stages in task design

• Provides a clear description of the speaking context • Defines clearly

– Situation [CEFR circumstances and environment] – Participants [social agents] – Topic [Table 5 Domains] – Purpose [CEFR functions and notions]

Possible criteria of a good prompt

Context and Purpose

No of participants

Formal / Informal

Input

Who were you talking to?

Why?

Where?

Dialogue D Group G

Formal F Informal I

What started the conversation?

Task authenticity: Speaking log

CEFR Manual Speaking grid: task input

Interaction type

Dialogue: paired candidates

Dialogue: grouped candidates

Dialogue: candidate/ examiner

Dialogue: simulated/ recorded prompts

Monolog

Repetition of prompt

Role play Reading aloud

React to prompt

Other:

Discourse mode (genre)

Interview Story telling (narration)

Speech, presentation Discussion/ conversation

Audience Assessor Other candidate

Teacher None Other:

A2

Can give a simple description or presentation of people, living or working conditions, daily routines. likes/dislikes etc. as a short series of simple phrases and sentences linked into a list.

Group exercise: HO 1 CEFR scales

B2

Can give clear, systematically developed descriptions and presentations, with appropriate highlighting of significant points, and relevant supporting detail. Can give clear, detailed descriptions and presentations on a wide range of subjects related to his/her field of interest, expanding and supporting ideas with subsidiary points and relevant examples.

• Construct • Holistic • Analytic

• Evidence • Construct

• Construct • Elicitation

• Model of language

• Model of the skill

Construct Task

Scale Performance

Test specifications

Popham's 5 component test specification model 1. GD - General description 2. PA - Prompt attributes 3. RA - Response attributes 4. SI - Sample item 5. SS - Specification supplement

Popham 1978

Iterative feedback for test revision

1. Select a skill

2. Write the specs

3. Write the item

4. Test/Pilot

Mandate

Lynch and Davidson, 1994

Group work

Groups: Teachers, researchers, test providers How do test specifications play a role in your test development?

How do test specifications play a role in the test development in your various contexts?

• Who develops them? o Individuals? Teams? Item writers? Researchers?

• Are they consulted? • Are they updated?

Groups

Test task characteristics

Frameworks for interpreting components of the construct – Nunan 1989 – Candlin 1987 – Weir 1993 – Bachman and Palmer 1996 – Fulcher 2003

Pedagogic Test generic Speaking specific

Format • Channel (aural, visual) • Form (language, non-language, both) • Language (native, target, both) • Length • Type (item, prompt) • Vehicle (live, reproduced, both)

Bachman and Palmer 1996:49

Characteristics of the input

• Processing under normal time constraints • Degree of participation in developing interaction • Purpose • Interlocutors • Setting • Role • Topic • Channel • Input dimensions • Size • Complexity • Range

Weir's Performance Conditions (1993:39)

select categories salient to testing context analysis of the task's success in eliciting language

samples for rating decisions made in task design should relate directly

to the test construct

Categories test a design process

Fulcher's framework for describing tasks

Task types Task description Example source Type of speech elicited?

Repeat the sentence Phonepass

Picture story

Map, graph, text ETS 2000

Picture prompt UCLES 2001

Information gap UCLES BEC 1997 (since replaced)

Role play Occupational english test, physiotherapists, from McNamara 1996

Task design

Where do we get the ideas for prompts? • TLU domain - real life language use tasks

• The most important • The most common • The most prototypical TLU tasks

Carr, 2011

Start by describing the TLU tasks • The setting of the activity • The purpose of the interaction • The identities of and relationships between the speakers • The time and place of the interaction • The language input received in real life • How the speaker would be expected to respond in real life • The length of the expected response

Task design

Item writer training

Test specifications

Task development

Item revision

Standard Setting

Item revision

Piloting

Piloting Speaking tasks Test development cycle

Item moderation

Task difficulty

• Expert judgements – individual work! • Using CEFR scales and subscales

• Allocate a CEFR level to the tasks in the handout

• Justify by citing CEFR descriptors

Task no: Rater number:

A2 B1 B1+ B2 B2+ C1 C2

Justfications: