Testing Testing Writing Writing andand SpeakingSpeaking

Jonathan MagdalenaJonathan Magdalena

What What are we trying to are we trying to MEASUREMEASURE??

The student’s abilityThe student’s ability to to interactinteract in the target language in the target language inin writtenwritten oror spokenspoken form. form. This involves This involves comprehensioncomprehension as well as as well as productionproduction.. (Hughes, 1991)(Hughes, 1991)

WhyWhy is it so is it so difficult to test difficult to test productive productive skillsskills??

Difficult to elicit behavior in early Difficult to elicit behavior in early stagesstages

Tasks should elicit a representative Tasks should elicit a representative sample of the student’s abilitysample of the student’s ability

Formal formats tend to elicit unreal Formal formats tend to elicit unreal behaviorbehavior

Teachers tend to score by impressionsTeachers tend to score by impressions Scoring criteria is often inconsistent Scoring criteria is often inconsistent

among testersamong testers

We can We can mend mend these problems by…these problems by…

Setting Setting appropriate tasksappropriate tasks Testing Testing onlyonly writing or speaking writing or speaking

abilities, and nothing elseabilities, and nothing else Restricting Restricting the student’s responsethe student’s response Selecting appropriate Selecting appropriate test formatstest formats Obtaining Obtaining reliable reliable scoresscores

Setting Setting appropriate tasks appropriate tasks means…means…

Considering Considering questions questions like these:like these: What is the What is the communicative function communicative function I I

want to elicit from the student? want to elicit from the student? (expressing… giving… describing…)(expressing… giving… describing…)

What is the What is the form of interaction form of interaction expected? (letter, postcard, email, voice expected? (letter, postcard, email, voice chat, interview, oral presentation) chat, interview, oral presentation)

What is the What is the contextcontext for the interaction? for the interaction? (formal, informal, real, unreal, school (formal, informal, real, unreal, school context, the street, home, abroad…)context, the street, home, abroad…)

Obtaining a Obtaining a representative sample representative sample means…means…

Setting a Setting a flexible flexible contextcontext Considering a Considering a varietyvariety of tasks of tasks Asking for authentic and Asking for authentic and meaningfulmeaningful

answersanswers Setting the Setting the criteriacriteria for production (1 for production (1

paragraph, the following aspects…)paragraph, the following aspects…) Giving the student Giving the student enough time enough time to to

complete the task.complete the task.

Restricting Restricting the student’s the student’s responseresponse means…means…

DefiningDefining tasks in detail (telling exactly tasks in detail (telling exactly what you expect from the students)what you expect from the students)

Establishing Establishing boundariesboundaries (controlling (controlling ideas that will stop students from going ideas that will stop students from going too far astray)too far astray)

Specifying the Specifying the minimumminimum requirements requirements for productionfor production

Provide Provide informationinformation or cues in the or cues in the form of notes, examples or picturesform of notes, examples or pictures

Testing the skill Testing the skill only only means…means…

Avoiding Avoiding the assessment of…the assessment of… OpinionsOpinions IQIQ Creativity or imaginationCreativity or imagination Previous/general knowledgePrevious/general knowledge Other skills (reading, listening…)Other skills (reading, listening…)

E.g. E.g. Read and explain this quote:“Bigotry Read and explain this quote:“Bigotry is the sin which most harms the sinner” is the sin which most harms the sinner”

Test formats Test formats forfor Speaking Speaking Oral questionnaireOral questionnaire Free interviews Free interviews Peer-interaction/role-playsPeer-interaction/role-plays Picture-cued tasks (Describing a picture, Picture-cued tasks (Describing a picture,

structuring a story, giving directions, structuring a story, giving directions, giving instructions…)giving instructions…)

Response to recordingsResponse to recordings Paraphrasing a story, a situation…Paraphrasing a story, a situation… Prepared monologue *Prepared monologue * Reading aloud *Reading aloud *

Test formats Test formats forfor Writing Writing Essay questionsEssay questions Guided writing (a letter, postcard, email, Guided writing (a letter, postcard, email,

note, form, advertisement…)note, form, advertisement…) Picture-cued tasks (Describing a picture, Picture-cued tasks (Describing a picture,

structuring a story, giving directions, structuring a story, giving directions, giving instructions…)giving instructions…)

Response to written/spoken stimuliResponse to written/spoken stimuli Paraphrasing a story, a situation…Paraphrasing a story, a situation… Prepared written production *Prepared written production * Dictation*Dictation*

Obtaining Obtaining reliable scores reliable scores means…means…

Having a well-structured Having a well-structured criteriacriteria based on based on bandsbands and and descriptorsdescriptors

55 Excellent writer Excellent writer 44 Good writer Good writer 33 Average writer Average writer 2 2 Substandard writer Substandard writer 11 Poor writer Poor writer

However, one However, one questionquestion arises… arises…

What makes a student an 'What makes a student an 'excellentexcellent' writer?' writer?

Is it Is it comprehensibilitycomprehensibility?? Is it grammatical Is it grammatical accuracyaccuracy?? Is it Is it spellingspelling?? Is it the way they Is it the way they organizeorganize the text? the text?

One One solutionsolution is… is…Either produce Either produce overall descriptionsoverall descriptions of writing and of writing and spoken ability… spoken ability… Or concentrate on Or concentrate on different aspects different aspects of writing and of writing and speaking and then separate these aspects out into speaking and then separate these aspects out into individual scales. individual scales.

Holistic scalesHolistic scales: those containing different : those containing different features of abilityfeatures of ability

Analytic scalesAnalytic scales: Those which separate out : Those which separate out different aspects of abilitydifferent aspects of ability

HolisticHolistic ScalesScales55 Constructs grammatically correct sentences and phrases and shows Constructs grammatically correct sentences and phrases and shows

full mastery of appropriate vocabulary. Textfull mastery of appropriate vocabulary. Text is organized coherently. is organized coherently. Correct spelling at all times. Excellent content and presentation. Correct spelling at all times. Excellent content and presentation. Message wholly relevantMessage wholly relevant

4 4 In general, grammatically correct sentences and phrases but some In general, grammatically correct sentences and phrases but some errors which do not affect understanding. Does not use. appropriate errors which do not affect understanding. Does not use. appropriate vocabulary at all times. Some difficulties with organization of text. vocabulary at all times. Some difficulties with organization of text. Some errors in spelling. Good content and presentation. Message Some errors in spelling. Good content and presentation. Message mostly relevant.mostly relevant.

3 3 Grammatical errors and use of vocabulary affects understanding as Grammatical errors and use of vocabulary affects understanding as does organization of text. Many errors in spelling. Satisfactory does organization of text. Many errors in spelling. Satisfactory content and presentation. Message not always relevant.content and presentation. Message not always relevant.

2 2 Text understood with difficulty due to inaccurate grammar and Text understood with difficulty due to inaccurate grammar and inappropriate use of vocabulary. Poor content and presentation. inappropriate use of vocabulary. Poor content and presentation. Message generally lacks relevancy. .Message generally lacks relevancy. .

1 1 Impossible to understand text due to frequency of grammatical errors Impossible to understand text due to frequency of grammatical errors and incoherence of organization. Poor spelling Message irrelevant.and incoherence of organization. Poor spelling Message irrelevant.

Why Why usingusing Holistic Holistic Scales?Scales? They are relatively easy to constructThey are relatively easy to construct Scoring takes less time than with the Scoring takes less time than with the

analytic method analytic method The integration of sub-skills tends The integration of sub-skills tends

concentrates on the overall effect of the concentrates on the overall effect of the piece of writingpiece of writing

Each work can be assessed more than onceEach work can be assessed more than once More bands can be added depending on the More bands can be added depending on the

context and the purpose of the testercontext and the purpose of the tester

AnalyticAnalytic ScalesScalesGrammarGrammar 6. 6. Few (if any) noticeable error of grammar or word order.Few (if any) noticeable error of grammar or word order.5. 5. Some errors of grammar and word order which do not Some errors of grammar and word order which do not

interfere with comprehension.interfere with comprehension.4. 4. Errors of grammar or word order fairly frequent; occasional Errors of grammar or word order fairly frequent; occasional

re-reading necessary for full comprehension.re-reading necessary for full comprehension.3. 3. Errors of grammar or word order frequent; efforts of Errors of grammar or word order frequent; efforts of

interpretation sometimes required on reader’s partinterpretation sometimes required on reader’s part2. 2. Errors of grammar and word order very frequent; reader often Errors of grammar and word order very frequent; reader often

has to rely on own interpretation.has to rely on own interpretation.1. 1. Errors of grammar or word order so severe as to make Errors of grammar or word order so severe as to make

comprehension virtually impossible. comprehension virtually impossible.

ScoreScore: Gramm____ Voc____ Mech____ Form____ : Gramm____ Voc____ Mech____ Form____ TOTALTOTAL________

Why Why usingusing Analytic Analytic Scales?Scales? They consider each student’s particular They consider each student’s particular

differences in terms of sub-skillsdifferences in terms of sub-skills Testers need to consider the different Testers need to consider the different

aspects of performance individuallyaspects of performance individually The sole fact that the scorer needs to give a The sole fact that the scorer needs to give a

number of individual marks makes the number of individual marks makes the scoring more reliablescoring more reliable

Each aspect can have a different weight Each aspect can have a different weight depending on the purpose of the testerdepending on the purpose of the tester

Some sub-skills might not fit together in a Some sub-skills might not fit together in a single bandsingle band

Holistic Holistic oror Analytic Analytic Scales?Scales?

What is the purpose of your test?What is the purpose of your test? Are you looking for diagnostic information?Are you looking for diagnostic information? Do you want to assess overall performance?Do you want to assess overall performance? Is it for a small and homogeneous group?Is it for a small and homogeneous group? Is it for a large heterogeneous group?Is it for a large heterogeneous group? Will it be carried by yourself?Will it be carried by yourself? Will it be applied in several different places?Will it be applied in several different places?