testing writing for the e8 standards
Post on 02-Jan-2017
217 Views
Preview:
TRANSCRIPT
Testing Writing for the E8 Standards
Technical Report 2011
Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller
Testing Writing for the E8 Standards
Technical Report 2011
Otmar GassnerClaudia MewaldRainer BrockFiona LackenbauerKlaus Siller
Bundesinstitut für Bildungsforschung, Innovation & Entwicklung des österreichischen Schulwesens Alpenstraße 121 / 5020 Salzburg
www.bifie.at
Testing Writing for the E8 Standards. Technical Report 2011.BIFIE Salzburg (Hrsg.), Salzburg 2011
Der Text sowie die Aufgabenbeispiele können für Zwecke des Unterrichts in österreichischen Schulen sowie von den Pädagogischen Hochschulen und Universitäten im Bereich der Lehrer aus-, Lehrerfort- und Lehrerweiterbildung in dem für die jeweilige Lehrveranstaltung erforderlichen Umfang von der Homepage (www.bifie.at) heruntergeladen, kopiert und ver-breitet werden. Ebenso ist die Vervielfältigung der Texte und Aufgabenbeispiele auf einem anderen Träger als Papier (z. B. im Rahmen von Power-Point Präsentationen) für Zwecke des Unterrichts gestattet.
Autorinnen und Autoren:
Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller
Contents
3 EmbeddingtheE8WritingTestinaNationalandInternationalContext
3 ThePlaceofWritinginAustrianLowerSecondarySchools
4 ValidityAspectswithregardtotheE8WritingTestConstruct
5 TestTakerCharacteristics6 CognitiveValidity6 WritingTheoryinBrief7 CognitiveProcessingintheE8WritingTest9 ContextValidity9 Setting:Task10 Setting:AdministrationofE8WritingTests11 LinguisticDemands:TaskInputandOutput12 ScoringValidity13 CriteriaandRatingScale15 RaterCharacteristics15 RatingProcess15 RatingConditions16 RaterTraining18 PostExamAdjustments18 ReportingResults19 ConsequentialValidity
20 E8WritingTestSpecificationsVersion03(July2011)
20 1.PurposeoftheTest20 2.DescriptionofTestTakers20 3.TestLevel20 4.TestConstructwithE8ConstructSpace23 5.StructureoftheTest23 6.TimeAllocation23 7.ItemFormats23 8.LanguageLevelforInstructionsandPrompts23 9.AssessmentwithWritingRatingScale28 10.PromptsandPerformanceSampleswithJustifications
37 ScaleInterpretations
37 ScaleInterpretation–TaskAchievement39 ScaleInterpretation–CoherenceandCohesion41 ScaleInterpretation–Grammar43 ScaleInterpretation–Vocabulary
45 Literature
48 Appendix
48 PromptInterpretation:LongPrompt50 PromptInterpretation:ShortPrompt53 InventoryofFunctions,NotionsandCommunicativeTasks
3Testing Writing for the E 8 Standards
EmbeddingtheE8WritingTestinaNationalandInternationalContext
ThePlaceofWritinginAustrianLowerSecondarySchools
Thereseemstobesomeagreementthatspeakingandlisteningaretheskillsmostneededwhentryingtosucceedinaforeignlanguageenvironmentandthatbeingable to read is next in priority. This leaves writing as the skill least necessary forsurvival.Nevertheless,writingistrainedfromyearoneofsecondaryeducationonaregularbasis.Insomecoursebooksitstartsoffwithmodelparagraphsthatareper-sonalisedbythelearnersandleadsontoopenwriting,mostlybasedonthecontentofthecoursebookunitinprogress.Itgoeswithoutsayingthatlowerabilitylearnersaregivenmoreguidance,withsomeofthemhardlyeverattemptinganopenwritingtask.
Inpost-beginnerclassestheimportanceattributedtowritingincreases.Itseemstobeawide-spreadbeliefamongteachersofEnglishthatwhenwritingskillsareassessed,otherdimensionsoflanguagecompetencelikevocabularyandgrammarknowledgecanbeassessedautomaticallyatthesametime.Therefore,thewritinggradegoesalongwaytowardstheoverallEnglishgradeforthatparticularstudent.
Whereasthisbeliefmightberesponsibleforthehighregardteachershaveforwrit-ing, the awareness of the complexity of assessment procedures for writing is stilllimited.Thereisnoperceivedneedforsharedstandardlevels,thereisnoagreementonhowwritingshouldbetested,markedandweightedinrelationtotheotherskills(reading, listening, speaking),1 there are a great number of idiosyncratic markingschemesinplace(evenwithinoneschool),andthereisnoagreementonanythinglikepassmarksorcutscoresforgrading.
In this situation there is roomforconstructivewashback in thecourseof the in-troductionofE8Standards.ItishopedthatthewaythetestsareconstructedandassessedwillimpactonthewaywritingistaughtandassessedinAustrianschools.
AlthoughmuchofwhathasbeensaidabovewasformulatedforthefirsteditionofthisTechnicalReportin2008,itisstillrelevantandwecancertainlyseesignificantsignsofchange.Aprogrammetotrainfourhundredwritingratersisinplaceandspreads expertise across the country; test specifications and a number of pilotingphaseshave led tovisible adaptations in thecoursebooksused; 'train the trainerprogrammes'onhowtoassesswrittenperformancesfunctionasstartingpointsforschool-basedprofessionaldevelopment.Finally, thereorganisationofacentralisedapproachtotheassessmentofwrittenperformancesatE12level(Matura)hascon-tributedalottoraisingawarenessofthecomplexityofassessingwrittenscripts.
1 ThislackofagreementisnoticeabledespiteaclearstatementintheAustriancurriculumaboutallfourskillstobetaughtandtrainedequallyintheclassroom;unfortunatelythecurriculumdoesnotsayanythingonweight-ingintests.(seeLehrplanderHauptschule.2008,p.2)
4 Testing Writing for the E 8 Standards
ValidityAspectswithregardtotheE8WritingTestConstruct
Shaw&Weir (2007)havedesignedacleargraphic to illustrate their“frameworkforconceptualisingwritingtestperformance”(seefigure1below).It takesall therelevantparametersintoaccountandcanserveastheblueprintforthedescriptionoftheE8WritingTestsandthetheoreticalframeworkonwhichtheyarebased.Withinthisframeworkthefocusofthediscussionwillbeonthefollowingaspects:testtakercharacteristics,cognitivevalidity,contextvalidity,scoringvalidity,andconsequentialvalidity.
Figure 1: Adapted from Shaw & Weir 2007, 4
6
Test-taker Characteristics
Cognitive Validity
Context Validity
Setting: Task
Response format Purpose Knowledge of criteria Weighting Text length Time constraints Writer-reader relationship
Setting: Administration
Physical conditions Uniformity of administration Security
Linguistic demands: (Task input and output)
Lexical resources Structural resources Discourse mode Functional resources Content knowledge
Response
Scoring Validity
Score
Cognitive Processes
Macro-planning Organisation Micro-planning Translation Monitoring Revising
Consequential Validity
Physical/Physiological Psychological Experiential
Rating
Criteria/rating scale Rater characteristics Rating process Rating conditions Rater training Post-exam adjustment Grading and awarding
Washback on individuals in classrooms Impact on institutions and society Avoidance of test bias
5Testing Writing for the E 8 Standards
TestTakerCharacteristics
It isobvious that test taker characteristicshave an influenceon theway a task isprocessedanda text iswritten.Three categorieshavebeen identifiedasphysical/physiological,psychologicalandexperientialcharacteristics(Shaw&Weir2007,5).
As regards the first category, any provisions made for schooling can be consideredsufficientfortheE8testsituationasalltesttakersarepupilsinthelastformoflowersecondaryschoolsinAustria.Toputitsimply,anypupilwhoisfitenoughtoattendEnglishclassesatanAustriansecondaryschoolandtobeassessedisfittotaketheE8WritingTest.
Psychological factors, however, are almost impossible to control. Most critical ismotivationasE8Standardsisalow-stakesexamthathasnoinfluencewhatsoeveron the individual test takers’marksoron their school career.Wecanexpect lowachieverstobemoreaffectedbylackofmotivation.Forthisreason,testresultsmightnotfullyrepresenttheactuallanguagecompetenceofthesestudents,buttheymightappeartobeatasignificantlylowerlevelbecauseafairnumberfromthisgroupoftesttakersmaychoosenottoshowwhattheycandoinEnglish.Aslongasthetesthasnopracticalimplicationsfortheindividualtesttaker,itwillbedifficulttogene-raterealinterestandmotivationinthosethatdisplaya'could-not-care-less'attitude.
In2013 theE8WritingTestwill be administerednation-wide for thefirst time.Thishasalreadyhadsomeimpactonteacherattitudeandmightalsohaveapositiveinfluenceonlearnermotivation.Preferredlearningstylesandpersonalitytraitsareotherfactorsthatarerelevant,butcannotbecateredforinthegiventestsituation.
Thethirdgroupoffactorsareexperientialcharacteristicsreferringtofamiliaritywiththetestformat.Whereasthetesttakersareallnewtothisparticulartypeoftesting,they shouldgenerallybe familiarwith the typeofpromptsused in theE8Writ-ingTest.Asdetails fromthe test specificationsconfirm(seepp.20–36),promptsusedarebasedontheBIST-Verordnung (Anlage zur Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungsstandards im Schulwesen – BGBl.II Nr.1/2009 v. 2.1.2009), theCEFR(Common European Framework of Reference for Languages: Learning, Teaching, Assessment) and theAustrian curriculum (LehrplanderHauptschule2008undLehrplanderAHS2006).
LearnerswhohaveonlydonetasksthatareheavilyscaffoldedwillfindtheE8promptschallenging.ThosewhohaveneverfacedopenwritingtasksintheirlearninghistorycannotbeexpectedtoperformwellintheE8WritingTestsorininternationaltests.Wewouldconsideritimportantwashbackifcoursebookauthorsand,consequently,alsoteachersweretorethinktheissuesinvolvedandalsoattemptunscaffoldedwrit-ingtaskswithALLpupils.AfterfouryearsofEnglishatsecondaryschoolandsome(very limited)writing atprimary level amounting tomore than500 lessons, anystudentshouldbeabletodoataskliketheonebelowsuccessfully:
6 Testing Writing for the E 8 Standards
You have come back from a one-week stay with a host family in Cambridge. At home you remember that you left your mobile phone in your room in Cambridge. Write a short email to your host family.
�Tell them where you are now.�Tell them about your mobile.�Ask them for the mobile.�Tell them how you liked your stay.
Figure 2: BIFIE Item Archive (http://www.bifie.at/freigegebene-items)
2011isthefirstyearwithanewgenerationofcoursebooksavailableforAustrianschoolstochoosefrom.Whatwasformulatedaboveasexpectedwashbackin2008hasmaterialised:ThenewcoursebooksincludewritingtasksthataregearedtotheE8WritingSpecificationswithanumberofthemextremelyclosetoactualE8Writ-ingPrompts.Eventhetimeconstraintsandthespecificationsregardinglengthhavebeen takenonboard.Another salient feature is the attempt to actually teach thestudentsaboutusingparagraphswhenproducing(longer)texts.
CognitiveValidity
“ThecognitivevalidityofaWriting task isameasureofhowclosely it representsthecognitiveprocessing involved inwritingcontextsbeyondthetest itself, i.e. inperformingthetaskinreallife”(Shaw&Weir2007,34).Whereasitisnotoriouslydifficulttodescribethecognitiveprocessesastheyarenotdirectlyaccessible,itseemsimportanttodescribeageneralwritingmodelthataccountsforwritinginareal-lifecontextaswellas inanexamsituation.However,onedifferenceshouldbenotedat the outset, namely that there is no time-constraint in most real-life situationswhereasintheE8testingsituationtime,topic,genre,andlengthofoutputarepre-determined.Thismightimposelimitationsontheplanningphaseaswellasonthewritingandrevisionphases.
WritingTheoryinBrief
Inthegivencontext,onlysketchyreferencesshallbemadetovarioussourcesthatpresentanddiscussthewritingprocessandmodelsofL1andL2writingindetail.AccordingtoGrabeandKaplan(1996,230–232),theplanningphase,whichtheycall“goalsetting”,involvesthesefivefactors:
�� anassessmentofthecontext�� apreliminaryrepresentationofthewritingproduct�� anevaluationofpossibleproblemsintaskexecution�� aninitialconsiderationofthegenrerequired�� anorganisationalplan
ShawandWeir(2007,37)makeapointofemphasisingtheadvantagesofamorepsycholinguisticallyorientedmodelofwritingover theGrabe andKaplanmodeland refer toField (2004)andKellogg (1994,1996). Interested readersmaywishto consult the detailed discussion there. The Field model (Field 2004, 329–331)involves
�� macro-planning�� organisation�� micro-planning
7Testing Writing for the E 8 Standards
�� translation�� execution�� monitoring�� editingandrevising
AreferencetoScardamaliaandBereiter(1987)isessentialhereastheyhavedescribedtwodifferentstrategiesusedbyskilledandlessskilledwritersintheplanningphase: knowledge telling and knowledge transformation.
Inknowledgetelling,novicewritersplanverylittle,andfocusongeneratingcontentfromwithinrememberedlinguisticresourcesinlinewiththetask,topic,orgenre.Knowledgetransformingbytheskilledwriterentailsaheightenedawarenessofproblemsasandwhentheyarise–whetherintheareasofideas,planningandorganisation(content),orinthoseofgoalsandreadership(rhetoric)[…](Shaw&Weir2007,43).
Whereas thisholds true for allwriting,L2writingposes additional cognitivede-mandsonthewritersasField(2005)argues.Attentiondirectedtowardslinguisticaspectslikelexicalretrieval,spelling,andsentencestructurescanimpedethefluencyofwritingandthecapacitytoorganiseandstructurethetextasawhole.Someideasmighthave to be abandoned in the executionphase on the grounds of languageconstraintsandlimitations.
CognitiveProcessingintheE8WritingTest
IntheE8contextwesuggestusingamodifiedGrabe/Kaplan-Fieldmodeltoillustratethewritingprocess,whichwillclearlybebasedonknowledge tellingandthushasaverybriefplanningphasemainlyconsistingofconsideringrelevantcontentpoints.
Thismodelincludesthefollowingphases:
�� assessmentofthecontext(whowritesaboutwhattowhomandwhy?)�� characteristicfeaturesofthegenrerequired�� preliminaryrepresentationofthewritingproduct�� selectionofcontentpoints�� evaluationofpossibleproblemsintaskexecution�� micro-planningatparagraphandsentencelevel�� translation�� monitoring�� revising
Figure3isagraphicrepresentationofthemodifiedGrabe/Kaplan-Fieldmodelhigh-lightingthethreemainsteps.
Figure 3: Modified Grabe/Kaplan-Field Model
Planning
ContextGenre
Preliminary representationContent
Problems
Writing
Micro-planningTranslationMonitoring
Editingand
Revising
8 Testing Writing for the E 8 Standards
IntheE8testsituation,theplanningphaseisclearlynotelaborateorextensive.Afteranassessmentofthecontext,whichincludesidentifyingthetopic,thesituationofthewriter,thetextpurposeandtheaddressee,mosttesttakerswillmovestraighttotheconsiderationofthegenrerequiredanddevelopa“preliminaryrepresentationofthewritingproduct”.Thenthebulletpointswillpre-structurethecontentelementstobeincluded.Anorganisationalplanisnotnecessaryasthetasksarefairlyshortandguidedbycontentpointswithlittleopportunityfordeviation.Especiallywiththeshorttask(40–70words)planningeffortswillbereducedtothebareminimumandberestrictedtothedecisiononwhichcontentpointstoelaborateandhowtoproceedinthatdirection.
Thewritingismorecloselyguidedthaninreallifeasanumberofcontentpointsaregivenintheprompt.Thismakesthewritingprocesssomewhateasierthaninreallife,butontheotherhand, it seemsunavoidable ifwewant toensure inter-raterreliability for thedimensionof taskachievement. Inadditionto this,providingacontentschemaforcandidatesatthislevelisnecessarybecausethecognitiveloadforsimultaneousactivitiesonamacroandmicrolevelwouldbetoogreatandthetasktoodemanding.
Ithasbecomeclearfromthepresentdiscussionthatmacro-planningandorganisa-tionplaynoroleinthegivenwritingcontextandthattheproductdeliveredwillbefirmlysetintheareaofknowledge telling.
Themicro-planningphase,thenextstepofthewritingprocess,mightbethepointwherepossibleproblemsintaskexecutionwillbeidentifiedbeforetheactualwritingbegins.Theproblemswillbecontent-relatedandhavetodowithknowledgeoftheworldandwhat(abstract)ideastousewiththecontentpointsgiven;theymightalsobeconnectedtotheattempttorecalltherequirementsofthegenreinquestionandwiththelanguagenecessarytoexpresstheideas.
Thisstageofidentifyinglanguageresourcesandtheirlimitationsisonlyafractionaway from actually putting pen to paper and undoubtedly is a central aspect ofmicro-planningfocusing
onthepartofthetextthatisabouttobeproduced.Here,theplanningtakesplaceonatleasttwolevels:thegoaloftheparagraph,itselfalignedwiththeoverallgoalofthewritingactivity;withintheparagraph,theimmediateneedtostructureanupcomingsentenceintermsofinformation(Shaw&Weir2007,39).
Micro-planningmergeswiththe translationphasewherepreviouslyabstract ideasonlyaccessibletothewriterhim/herselfaretranslatedintothepublicspacedefinedbyasharedlanguage.IncontrasttoShaw&WeirandField,weseemicro-planningandtranslationastwostagesthatareinterlinkedasthewritermightoscillatebetweentheoneandtheotheratsentenceleveloratparagraphlevel(Shaw&Weir2007,39–40).
Itisinthetranslationstagethatlanguagedecisionshavetobemadeandplanningdecisionshavetobeimplemented.Theactualproductionoftexttakesplaceundertheconstraintsofcontentschemata,genrerestrictionsandthelimitationsoflinguisticresourcesathandinL2.Whathasbeencalled“avoidancebehaviour”(e.g.avoidinglexisorstructuresthatseemunsafe)and“achievementbehaviour”(e.g.usingsimplerstructures,paraphrasing)byField(2004,66–67)needstobetakencareof intheassessmentphase,asdoestheabilitytoproducecoherentandcohesivetexts.
Thenextstepismonitoringalthoughthisisnotnecessarilysequentialandmightbeoscillatingwithphasesoftranslation.“Atabasiclevelmonitoringinvolveschecking
9Testing Writing for the E 8 Standards
themechanicalaccuracyofspelling,punctuationandsyntax”(Shaw&Weir2007,41).AtE8levelthis iswhatcanbeexpected, ifnotinthelowestsegmentoftesttakers.Inaddition,betterwriterswillalsocheckbackoncontentandgenrerequire-ments.Thesemonitoringactivitieswillleadtoeditingandrevisingifsomepartsofthetexthavebeenfoundunsatisfactory.Thismightinvolveadding,deletingormod-ifyingacontentpoint,addingcohesivedevices,replacingpoorwordsandphraseswithbetterones,orsimplycorrectingmistakesinspellingandstructure.
IntheE8context,writingiscertainlybasedontheknowledge-telling model (Scar-damalia&Bereiter1987);Hyland’ssummaryofthemodelepitomisesE8writingperformances:
A knowledge-telling model addressesthefactthatnovicewritersplanlessoftenthanexperts,reviselessoftenandlessextensively,andareprimarilyconcernedwithgeneratingcontent from their internal resources.Theirmaingoal is simply to tellwhat theycanrememberbasedontheassignment,thetopic,orthegenre(Hyland2002,28).
ContextValidity
Tests shouldbe as close aspossible to authentic real-life situations.Writing is anactivitythatisnormallyperformedbyindividualsatatimesetasideforit.Writershave a purpose and an audience; they have the freedom to interrupt the writingprocessandresumeitatatimeoftheirchoice,especiallyforeditingandrevising;andtheycannormallyusedictionariesandotherresources.Inthegiventestsetting,someconstraintswillbeoperative,butunavoidable.
Shaw&Weir2007(64–142)discussanumberofaspectsofcontextvalidityrelatedtothreeareas:
�� Setting:Task�� Setting:Administration�� LinguisticDemands:TaskInputandOutput
ThesepointswillstructurethediscussionofcontextvalidityoftheE8WritingTests.
Setting:Task
Theaspectstobediscussedhereareresponseformat,purpose,knowledgeofcriteria,weighting, text length, timeconstraints, andwriter-reader-relationship. In theE8WritingTestsauthenticityisoneofthemostprominentaimsofpromptconstruc-tion.However,incontrasttoreal-lifewritingthereisnoprovisionfortheuseofanyresourcematerialssuchasdictionaries.
ThewritingtasksaretargetedatpupilsofAustrianschoolsinyear8andnormallyagedfourteen.Thetasksaredesignedtoappealtothisagegroupandtoelicitscriptsthat show what test takers can do within the framework defined in the BIST-Ver-ordnung.Thedomainsandgenreshavebeencarefullyselectedfromthisframework,whichisbasedontheCEFR,andhavebeenfilteredfurtheronthebasisoftheAustriancurriculum.
Astheresponseformatmaywellplayasignificantroleintestperformance(Aldersonetal.1995),thedecisionhasbeentakentoincludetwoformatsintheE8WritingTest.Thereisashorttask(40–70words)andalongtask(120–180words),whichareassessedseparately.Bothareopenwritingtasks.Goodwritershaveabetterchancetoshowtheirbestinthelongtask,whichisbasedonaB1descriptor,takenfromthe
10 Testing Writing for the E 8 Standards
BIST-Verordnung.Lowerachieversareexpectedtodobetterintheshorttask,whichislimitedinscope,morecloselyguidedandbasedonanA2descriptor.However,both good andweakerwriters are expected to address both tasks as they arenotsupposedtochooseonlyoneofthetasks.
Instructions,deliveredbothorallyandinwritingtothetesttakersbeforetheactualtestbyatestadministrator,andrubrics thatgowitheachtaskpresentcandidateswithinformationregardingtextlength(seeabove)andtimeconstraints.Forcom-pletingbothtasksthetesttakershave30minutesofwritingtimeplus5minutesforeditingandrevisinginall.After35minutesthereissometimeforwordcountbythecandidates.Theactualpromptscontextualisethetaskbydefiningthewriter-reader-relationship,statingpurposeandgenre,andgivingcontentpointstobeincludedinthetext.Theshorttaskcontains3–4contentpoints,thelongone5–8.
Informationonthescoringcriteriausedandtheirweighting,includingtheratingscaleused,scaleinterpretationsandbenchmarkedsamplescripts,ispublishedinthisreport(seepp.12–14,23–27,28–36,37–44).Furthermore,sampleprompts,therating-scaleandbenchmarkedtextsarepubliclyavailableontheBIFIEwebsite2.
Setting:AdministrationofE8WritingTest
In its present form, thewriting testwasfirstpilotedon a sampleof ca. 800 testtakersin2007andin2009abaselinestudywascarriedout.Consequentlydetailedinformationonthe“pilotphase”between2006and2008andonthebaselinetestsin2009werepublishedinaTechnicalReport(Breit&Schreiner2010).Startingin2013,theE8WritingTestswillbesetnationwideeverythreeyearsandallAustrianschoolchildreningrade8willbetested.OnlySENpupils,i.e.thosewithspecialeducationalneeds,willbeexemptedfromdoingthetests.
Inordertoensurereliabletestresults,thecircumstancesunderwhichtheE8WritingTest takesplacemustbe similar.The stepsdiscussed inmoredetailhere concernphysical conditions,uniformityof administration, and test security,basedon theideasbyShaw&Weir(2007).
As thevenuesof theE8WritingTestareclassrooms inAustrianschools,physicaltestconditionsareofverysimilarstandardsandtesttakersshouldfindappropriateconditionsfortakingthetest.
In order to grant the uniformity of administration, the test must be conductedaccordingtostandardisedinstructionsbytrainedtestadministrators.Anextensivetestadministrator’smanualisprovidedduringthetestadministratortraining.Themanualincludesinformationonthebackgroundofthetest,checklistsandToDo’sbothforthepreparation,theactualsettingofthetest(e.g.startingtheexam,com-pletingdifferentlists,standardisedverbalinstructionsforthetestadministratoretc.),andtheconclusionoftheexamination.
Inanationwideexamtherearesomeadministrativeconstraints:apoliticaldecisionhasbeentakenregulatingtestadministrationintheyearstocome:in90%oftheclassestheE8Testswillbeadministeredbytheteachersoftheschool(internaltestadministration).Inafurther3%oftheclassesthetestswillalsobeadministeredinternally, but there will be external quality monitors to assure the correct andstandardisedadministrationofthetests.7%oftheclasseswillbetestedexternally.Alltestadministrators,bothinternalandexternalones,aretrainedtoadministerthe2 http://www.bifie.at/freigegebene-items[24June,2011]
11Testing Writing for the E 8 Standards
testsaccordingtoagreedstandardisedprocedures.However,itiswithintheresponsi-bilityof the schools’head teachers to take careof a correct and standardised testadministration,asthisistheonlywaytogetreliablefeedbackregardingtheperfor-manceoftheirpupilsandtoplanlocalmeasuresofqualitydevelopment.
Thepromptsused in futurewritingtestshaveallbeenwrittenbytheprospectiveraters,moderated, edited, and screenedby theBIFIEWritingTrainerTeam,pre-testedandstored in the itemarchive.Thetestbookletsaredesignedby the samegroupincooperationwiththepsychometricdepartmentatBIFIE.Theactualdistri-butionofalltestpaperstotheschoolsishandledcentrallybyBIFIESalzburg.
MoredetailedinformationontheadministrationoftheE8TestswillbepublishedinaTechnicalReportafterthefirstnationwidetestingin2013.
LinguisticDemands:TaskInputandOutput
IntheAustrianteachercommunitythecommunicativeapproachtolanguagelearn-ing(Canale&Swain1980asanimportantprecursoroftheBachman1990modelofcommunicativelanguageability)iswidelyaccepted,anditisalsosetdowninwritinginthenationalcurriculum.Asthelearningtasksaremodelledonreal-lifecontexts,thelearningenvironmentaimstomirrorreallifeascloselyaspossible.ExamssetintheAustriancontextneedtosharethesepremisesandtoreflecttheminthetasksset.
Shaw&Weir(2007,91),Alderson(2004,13)andotherscomplainthattheCEFRremainsvagueandwithholdsdetailswhenitcomestostructuresorvocabulary,usingtermslike“simple”inthedescriptors.Whilethisistrue,readingtheCEFRexten-sivelyratherthanfocusingonlyonthesectionscontainingthescalesproveshelpful.Inchapter3,thedevelopmentofthecommonreferencelevelsisexplainedanditismadeclearthattheyprogressinaverycoherentwayfrom“thelowestlevelofgener-ativelanguageuse”(CEFR2001,33)tosocialfunctionsand“descriptorsongettingoutandabout”(CEFR2001,34)basedonWaystage (A2)andasimplifiedversionofsometransactionallanguagefrom“’TheThresholdLevel’foradultslivingabroad”(CEFR2001,34).A2+doesnotsomuchincreasetherangeoftopics,butfocuseson“moreactiveparticipationinconversation”and“significantlymore[onthe]abilitytosustainmonologues”.B1reflectstheThreshold Levelandinvolves“theabilitytomaintaininteractionandgetacrosswhatyouwantto,inarangeofcontexts”aswellas“theabilitytocopeflexiblywithproblemsineverydaylife”(CEFR2001,34).B1+makesincreaseddemandsonthequantitiesofinformationtobehandled.
As this is theway the levelshavebeen constructed (i.e. fromWaystage toA2), itseems legitimate to move from A2 specifications back to Waystage. And here wehaveavocabularylistandalistofstructuresconsideredcharacteristicofthatlevel.AsUCLEShavealsoorientedthemselvesonvocabulary lists fromtheCouncilofEuropePublications(LexicalInventoryinWaystage,1980,45–62;andinThreshold, 1979,85–115),itcanbeconsideredausefulshortcuttopickupthevocabularylistspublishedonthewebforKET(A2)andPET(B1),especiallyas thesehavebeenupdatedonthebasisoffrequencyandusagedatafromlanguagecorpora.Generally,“[the] language specifications of KET are the same as those set out inWaystage1990”(KETHandbook2007,1).
Toresumethediscussionofthevaguenessofdescriptorsusingwordslike“simple”,“basic”or“sufficient”,itmaysufficetosaythatthisvaguenessneedstobecontextu-alised.IftheA2descriptoronGrammaticalAccuracyreads“Usessomesimplestruc-turescorrectly,butstillsystematicallymakesbasicmistakes”(CEFR2001,112),wecanexpectlearnerstousetherangeofstructureslistedinthe Structural Inventory of
12 Testing Writing for the E 8 Standards
Waystage (63–83)ortheKET Handbook(8–9)withseverelyrestrictedaccuracy.Inthissense,evenvaguetermslike“simple”arereasonablywell-definedsothatpromptwritersandratersknowwhattolookfor.
TheE8writingpromptsdonotrestricttesttakersintheiruseofspecificlexicalorstructuralresources,butgivethemtheopportunitytodemonstratetheirlinguisticabilitieswithinthetaskset.Theextentoftheirsuccessindoingsoisassessedaccord-ingtothegradeddescriptorsintheassessmentscale.
Whathas tobenoted,however, is thebasicorientationof theCEFRtowardsanadultlearnerandadominanceoftouristaspectsoflanguagelearning.ThisiswhytheAustrianE8Standardshavealso integrated the specifications setdown in theAustriancurriculumandadaptedtheCEFRdescriptorstotheagegroupofthetestpopulation.Thismainly reflects the selectionofdomains and transactional situa-tions.Ithasnoinfluenceonthestructuresincluded,thoughithassomeinfluenceonthewordlist.Generally,theschoolbooksusedinAustriatakethisintoaccount.AsthetestisexplicitlybasedontheAustriancurriculum,thelinguisticdemandsofthetestarefairforalltesttakers.
The writing prompts used often specify particular language functions to be per-formed,e.g.“invite…,apologise…,askfor…,giveadvice…”.Alistofthesefunc-tionshasbeenmadeavailabletotheteacherspreparingthetesttakerssothattheycanbeexpectedtobeawareofthem(seepp.53–54intheappendix).
Severalresearchpapershaveobservedaninteractionorevenaninterdependenceofcontentknowledgeontheonehand,andwritingperformanceandtestscoresontheother(Read1990,Papajohn1999,Weir2005).Provisionsforthishavebeenmadeby restricting topics toareas that can safelybeassumed tobe familiar to the testtakersastheyaresetdownintheAustriancurriculumandmusthavebeenincludedintheirEnglishlessons.However,thisstillleavesthefactthatsometesttakersmightfeel indisposed todealwith aparticular topic for anumberof reasons, themostcommonprobablybeinglackofmotivationandinterest.
Moredetailedinformationondiscoursemode(i.e.texttypes),functionalresources(i.e.intention/purpose),andcontentknowledge(i.e.topicarea)canbefoundinthetablesonpp.21–22representingtheE8ConstructSpace.
ScoringValidity
Scoringvalidityisconcernedwithalltheaspectsofthetestingprocessthatcanimpactonthereliabilityof test scores. […It] iscriterialbecause ifwecannotdependontheratingofexamscriptsitmatterslittlethatthetaskswedeveloparepotentiallyvalidintermsofbothcognitiveandcontextualparameters.Faultycriteriaorscales,unsuitableratersorprocedures,lackoftrainingandstandardisation,poororvariableconditionsforrating, inadequateprovision forpost examstatistical adjustment, andunsystematicorill-conceivedproceduresforgradingandawardingcanallleadtoareductioninscoringvalidityandtotheriskofconstructirrelevantvariance(Shaw&Weir2007,143–144).
Inthissectionweexamineeachoftherelevantparametersinsomedetail:criteriaandratingscale,ratercharacteristics,ratertraining,ratingprocess,ratingconditions,postexamadjustments,andgrading.
13Testing Writing for the E 8 Standards
CriteriaandRatingScale
Beforetheactualconstructionoftheratingscale,informationonexistingscaleswascollectedandtheusefulnessofthescalesintheframeworkofE8Testingwasana-lysed:Jacobsetal.scoringprofile1981(Weigle2002,116);TEEPattributewritingscales,Weir1990(Weigle2002,117);FCEScoringrubric1997(Weigle2002,152);TOEFLwritingscoringguide2000(Tankó2005,125);IELTSbands2002(Weigle2002, 159); Analytic writing scale developed by the Hungarian School-LeavingEnglishExaminationReformProject2005(Tankó2005,127).
LumleyreportsfindingsfromWeigle1994,whousedananalyticscaletohave30compositionsassessedbynoviceandexpertraters.Weiglefocusedonnoviceraters,whichisrelevanttotheE8situationinAustriawherearatingcultureisonlyjustevolving.
Shefoundthatraterreliabilityincreasedasaresultoftraining,andthattheimprovedagreementwastheresultofratersgainingbetterconsensualunderstandingofthetermsandlevelsrepresentedinthescale.Shefoundevidencethattraininghelpedclarificationoftheratingcriteria(Lumley2005,44).
Thissupportstheviewofthetestingteamthatinthegivencontextananalyticscalewouldbepreferabletoaholisticscale.ThisviewisalsosupportedbyWeigle2002,whomentionsseveraladvantagesofanalyticoverholisticscoring:
�� Itismoreusefulinratertrainingasinexperiencedraterscanmoreeasilyunderstandandapplythecriteriainseparatescales.�� Itisparticularlyusefulforsecond-languagelearners,whoaremorelikelytoshowa
markedorunevenprofile.�� Ascoringschemeinwhichmultiplescoresaregiventoeachscripttendstoimprove
reliability(Weigle2002,120).
AnotherreasonforrulingoutaholisticapproachwasthefactthatratingproceduresforscriptswithintheAustrianschoolsystemarenotregulated,showgreatvarietyandaretoalargeextentholistic,evenimpressionistic.Asassessmentproceduresforwriting inAustrian schools cannotbe taken as abasis for adisciplined approachtowardsratingscripts,breakingwiththistraditionseemedtobestguaranteeafreshapproachtoassessment.
Taking the general background of Austrian traditions in assessing writing intoaccountandinspiredbytheHungarianscale(Tankó2005,127),thedecisionwastaken to design an analytic scale measuring four dimensions:Task Achievement,CoherenceandCohesion,Grammar,andVocabulary.Whereasthreeofthesefourdimensionshave a strong recognitionvalue forAustrian teachers,Coherence andCohesion might appear unusual and reflects the high importance given to thisdimensionbytheCEFR.Thesefourdimensionspromisedtoyieldenoughdetailforaconstructivefeedbackprofileonindividualtesttakerperformance,informationforinstructionaswellasinformativedataforsystemmonitoring.
The assessment scale was constructed bearing in mind the fact that the overallmajorityofperformancescouldbeexpectedtobearoundA2/B1.ThismeantthatA2andB1descriptorsneededtobeincludedwhileanythingatandaboveB2couldbeneglected.WeareawareofthefactthatthiskindofscalecannotmeasureB2orC1performancesandwehavesettledforstatingthatperformancesabovetheupperendofthedescriptorsintheE8Scalearecalled“aboveA2”forshorttasksand“aboveB1”forlongtasks.But,generally,theapplicabilityofaparticulardescriptordoesnot
14 Testing Writing for the E 8 Standards
automaticallysignalthatascriptisatagivenCEFRlevel.Firstly,bandsconsistofmorethanonedescriptor,andsecondly,linkingwrittenperformancestotheCEFRisacomplexprocedurethatisbeyondthescopeofthisreportandwillbediscussedinaseparatepublication.
Thesecondconsiderationinscaleconstructionwasthecognitiveloadthatraterscanmanageintheratingprocess.Thedecisiontousefourdimensionsisalsoinagree-mentwiththeCEFRrecommendationtoreducethenumberofpossiblecategoriesto“afeasiblenumber”as“morethan4or5categoriesstartstocausecognitiveover-load”(CEFR2001,193).Wetakeitthatthiswarningalsoappliestothenumberofbandsanddescriptors that raters canhandle, sowehaveopted for fourbandssuppliedwithdescriptorsandthreeemptybandsinbetween,makingitaseven-bandscaleplusazeroband.
Atthatpointinscaleconstructionthescalesconsistedofthreecolumns:Thefirstbeingadeflateddescriptorforeachofthefourbands,thesecondbeingextendedandcontainingmoredetail,andthethirdquotingtherelatedCEFRdescriptor.AnimportantdecisionintheprocessofscaleconstructionwastheremovaloftheCEFRlevelsattheendoftheCEFRdescriptorsand,inasecondstep,theremovaloftheCEFR descriptors altogether. This was the logical step to take when some ratersawardedband7toascriptandarguedthatthescriptwasaB2performance.How-ever,suchanargumentisinadmissibleasthepromptsusedinthetestarewrittenonthebasisofA2orB1descriptorsandresponsestothesepromptssimplycannotrepresentperformancesaboveA2orB1respectivelyasonebasicfactoristhescopeofaperformancetogetherwiththegivenlimitationsofdomainsandgenres.SoevenwhentheCEFRB2descriptorforGrammaticalAccuracy“Showsarelativelyhighdegreeofgrammaticalcontrol.Doesnotmakemistakeswhichleadtomisunderstand-ing.”(CEFR2001,114)describestheperformancewell,itdoesnotmeanthatitisB2,butthattheA2/B1taskhasbeencarriedoutverywellandthatthe(grammar)performanceisaverygoodA2orB1performancerespectively.
Inanotherstep,thescalewascondensedtoonepagewithanextendedscaleeachforTaskAchievementShortandTaskAchievementLong.Asthisdeflatedscalemightnotcarryenoughinformationfortheratersatthebeginningoftheirtraining,scaleinterpretationshavebeenprovided(seepp.37–44).Thescalesthemselveshavebeenfine-tunedinthetrainingprocessinanongoingdialoguewiththeraters.Itfollowsfromthis that thescalesarewhathasbeencalledassessor-oriented(Weigle2002,122;CEFR2001,38).
Thewritingscriptsareassessedonfourdimensions:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary.WhereasthelastthreearebasedontheCEFR and the Austrian BIST-Verordnung, the CEFR does not contain anythingexplicitonTaskAchievement.ThedescriptorsofthescalesonOverall Written Pro-duction and Overall Written Interaction mainly refer to linguistic and pragmaticaspects(Can write a series of simple phrases and sentences linked with simple connectors …), whereasthesubscalesonlymakereferencestotexttypes,domainsandcontentaspects(Can write personal letters describing experiences …). Thesedescriptorscannotbeoperationalisedinassessmentterms.
Inourview,however,thecontentaspectofwritingiscentralandlargelyresponsiblefortheoverallqualityofascript.Nevertheless,theratersdonotgiveanoverallgradeforwriting,butallfourdimensionsareratedseparatelyandarereportedasaprofile,whichmoreoftenthannot isunevenormarked.For the feedbackprocedure,anoverallwritingscorewiththefourdimensionsoftheshortandthelongperformanceassessmentisgiven,basedonanequalweightingofalldimensions.
15Testing Writing for the E 8 Standards
RaterCharacteristics
Ithasbeenreportedthat“SubjectspecialistsandlanguagetrainedEFLteachersdem-onstrateatendencytoemployratinginstrumentsdifferently”(Elder1992,inShaw&Weir2007,169).InthisrespectthepresentsituationinAustriaisuncomplicatedasallratersareteachersofEnglishwhoteachinlowersecondaryschools.SomeofthesearenativespeakersnowlivingandworkinginAustria,somehaveauniversitybackground,otherswereeducatedatUniversityCollegesofTeacherEducation.
Althoughtheratersgo througha specific training that familiarises themwith theratingscalesandtheratingprocedures,differencesintheirexperientialbackgroundandintheirprofessionaltraininganddevelopmentmayleadtodifferingassessmentsofscripts.Inordertomakeratersawareofthisandtostartaprocessofself-reflection,allratersgetdetailedfeedbackontheirratingbehaviouratseveralpointsinthetrain-ingandparticularlyafterthelasttrainingsessionandaftertheadministrationofawritingtest.Theyareinformedabouttheirinter-raterreliabilityandraterseverity.Eventually,harshnessandleniencyofratersistakencareofthroughRaschmodell-ing.
RatingProcess
Milanovicetal.(1996)identifiedanumberofapproachesraterstakeintheprocessofratingascript.Inourtrainingsessionswegenerallyadviseagainstthe“readthrough”andthe“provisionalmarkapproach”,bothofwhicharebasedononereadingofthescript.Ratersareencouragedtoadopta“principledtwoscan/readapproach”totheprocesswithafocusonTaskAchievementandCoherenceandCohesioninthefirstreadingandonGrammarandVocabularyinthesecond.Thelengthofthescriptsseemstosupportthisapproach.
WeareawareofgroupeffectsonraterreliabilityasdescribedbyShaw&Weir(2007,174–175)andhavemadeanefforttousethemtoouradvantageinthestandard-isationmeetingsatthebeginningofthetrainingsessionsandtheratingsession.Inaddition to theprocedures recommended for standardisationmeetings (Alderson,Clapham&Wall1995,112–113)aconsiderableamountoftimeisspentonthedetailedinterpretationoftheprompts(seeappendix,pp.48–52)andanopendiscus-sionofanyquestionsthatmightberaisedbytheraterstakingintoconsiderationthatallratershavealsobeeninvolvedinthewritingofpromptsandtheirpiloting.Anadditionalsetoftenbenchmarks,gainedinanextensivebenchmarkingconferencewithtenbenchmarkers,playsavitalroleinthestandardisationsessions.
RatingConditions
In2013thewholeE8populationofsome90,000pupilswillbetested.InJune,allraters,whohavebeentrainedatdifferent intervals since2006,will takepart inaone-daystandardisationmeetingasdescribedonp.18intrainingphase6,followedupwithaone-dayratingsession.
TherewillberegionalstandardisationmeetingsforallraterswhomarkscriptsfromtheE8WritingTests.Inthesesessionsratersareupdatedon,forexample,anychangesregardingtheassessmentscaleused.Thentheywillcontinuewiththeon-siteratingsession,inwhichtheywillundergosupervisedratingwiththenewtestpromptsandactualscriptsofthetest.ThescriptshavebeencarefullycompiledinratingbookletsbytheBIFIESalzburgpsychometricdepartment.ThiswillprovideBIFIEwiththerelevantdataneededfortestanalysisandfeedback.
16 Testing Writing for the E 8 Standards
Itwilltakehalfaday’sworktodealwitheachpromptandgiveratersenoughtimeforon-site ratingandclarificationof ratingproblemsbasedon theparticular testprompts.Theremainingunratedscripts,approximatelythreequartersintotal,willberatedoff-sitewithinsixtoeightweeksattheraters’convenience.
RaterTraining
AccordingtoAlderson,Clapham&Wall,ratertrainingisoneofthemostessentialaspectsinanefforttoobtainreliableassessments(1995,105).Lumleyrefersto„agrowingbodyofworkthatshowstheeffectivenessofthetrainingprocessinallowingraterstodevelopadequatereliabilityoragreementintheuseofindividualscalesinrelationtospecifiedtesttasks“(2005,62).
ThishasbeentakenveryseriouslybytheBIFIEWritingTrainerTeam,whohavedevelopeda seven-months trainingprogramme for raters starting inOctoberandpreparingtheraters for themockratingsession inApril/May.Thisprogrammeisdescribedinsomedetailbelow.
RECRUITMENTIn the recruitment phase teachers in Austrian lower secondary schools are ap-proachedtobecomewritingraters.Asthetesttakerscomefromthetwodifferenttypesoflowersecondaryschools,theGeneralSecondarySchool(HauptschuleandNeueMittelschule)andtheAcademicSecondarySchool(AHS),carehasbeentakento ensure intakeof raters fromall threeof these school types.While recruitmentwasoriginallycarriedoutbyBIFIESalzburguntil2009,theadministrationoftherecruitmentprocesshassincebeenoutsourcedtotheregionalUniversityCollegesofTeacherEducation.
TRAINING PHASE 1: OCTOBER (1 DAY; FACE-TO-FACE SESSION)AstheCEFRisthemostrelevantbackgrounddocumentfortheE8Standards,thestar-tingpointofthefirsttrainingsessionisThe Common European Framework ingeneralandtheOverall Writing Scales for Production and Interactioninparticular.Thefamil-iarisationwiththeCEFRisimplementedonthebasisoftherecommendationsmadeintheManual on Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF) (2003, 70–77),includ-ingsortingtasks.Itismadeclearatthispointthatoneaspectofwritingisrelatedtocommunicativeactivitiesandstrategies,anotheronetolinguisticcompetences.
TheTestSpecificationsarepresentedanddiscussedindetail:domains,genres,constructspace,promptformat.Inthisphasethereisafocusonpromptproduction.Thereare Guidelines for prompt writers thatprovideassistanceintheprocessofpromptwriting.
Eachprospectiveratertakesonthetaskofwritingonelongoroneshortpromptinatrainingtandemduringtheweeksfollowingthefirsttrainingsession.Allpromptsaresentintothetrainerteamformoderation.Oncethepromptshavebeenscreened,theyarepilotedbythepromptwriters.Behindthisprocedurethereisthebeliefthatratersneedtoknowaboutthequalitiesofpromptsandwhatelementstheymustcontain.Thispreparesthemforbetterinteractionwiththetestpromptsintheactualratingphase.
Withregardtodifferentialvalidityitisimportanttostatethateveryprecautionistakenattheearlystageofpromptwritingtoavoidtestbias.Varyingculturalback-groundsandknowledgeof theworldhavebeentakenintoaccountaswellas thegivenvarietyofcognitivecharacteristics,mothertongue,ethnicity,andgender.
17Testing Writing for the E 8 Standards
AfteranintroductiontotheAustrianE8Standardstheratingscaleispresentedandexplainedonthebasisofwrittenscaleinterpretations(seepp.37–44).Unfoldingthesevenbandswithfourofthemdefinedandworkingthroughthefourdimensionstakestime.Theprocedureadoptedistolookatthesevenbandsofonedimension,whichleadstosometheoreticalunderstandingofthescales,butconnectionstoactualscriptsarestilltenuous.Sotheratersgettwoscriptseachforindividualmarkingonthefirstdimension(taskachievement).Theydiscusstheirassessmentinsmallgroupsandthetrainerdisclosesthebenchmarkedassessmentsandarguesthecase.Thispro-cedureisrepeatedfortheotherthreedimensions.
In the secondphase of the trainingworkshop theparticipants get sets of bench-markedscriptswhichtheyrateonallfourdimensions.Benchmarkjustificationsarepresentedindetailanddiscussedinplenarytoensurethattheratersunderstandthedescriptorsoftheratingscaleandcanhandletheratingagreementsthathavebeenformulatedforanumberofspecial ratingproblems(e.g.downgradingforbelow-lengthtexts).Afterdiscussionsandargumentationofthejudgementsforthebench-marked scripts theparticipantshavea rough ideaof thecomplexityof the ratingprocessandtheeffortittakestoarriveatreliablejudgements.Theratingsheetsfilledinbytheparticipantsprovideafirstsetofdatathathelpstomonitorintra-raterreliability.
TRAINING PHASE 2: OCTOBER – DECEMBERThesecondtrainingphaseisanopenonewithafairlyloosestructure.Allpartici-pantsfirstwritetheirpromptsandgetthembackfromthetestingteamasscreenedprompts(insomecasesthepromptsarereturnedtothewritersforrepair).Asafirstmeasuretowardsqualityassurancethepromptwritersproducearesponsetotheirownprompt.Thisshouldmakepromptwritersawareofthemoreobviousflawstheirpromptsmighthave.Thepromptsarethenpilotedinoneoftheirclassessothatallparticipantshavearound20scriptsbasedontheirprompt.
TRAINING PHASE 3: JANUARY – MARCHOncethepromptwritingandpilotingisfinished,theonlineratingphasestartsinJanuarywiththetrainerteamsendingoutscriptstotheratersforindividualratingoneachofthefourdimensions.Theratershaveaboutfourweekstodothisandsendintheirratings.Whenallscoringsheetshavebeensubmittedtothetestingteam,the benchmarks are sent out to the raters. In February/March the raters practisetheirratingskillsontheirownpilotscripts.Theyselecttwoscriptstobefedintothetrainingprocess.Theyratethesescriptsandwritejustificationsfortheirratings.Asecondonlineratingphasehelpstostandardisetheraters,whoareencouragedtomakefinaladjustmentstotheirscoresandjudgementsbeforesendingthemtothetrainertogetherwiththedigitalisedscripts.ThetrainergoesthroughthesescriptsandselectsinterestingsamplesfortheupcomingtrainingworkshopinApril/May.
TRAINING PHASE 4: APRIL/MAY (1 ½ DAY; FACE-TO-FACE SESSION)TrainingPhase4startswithadiscussionofopenquestionsfromprevioustrainingphases.Thenthereare twostandardisation sessionswith recycled scriptsandnewbenchmarksandthetimeisspentratingscriptsanddiscussingparticularproblemsarisingintheprocess.
Afterthesetwostandardisationsessionsthefirstpromptwritertandempresentstheirpromptandthewholegrouprates2–4scripts.Experiencefrompreviousratingses-sionshasshownthat,astheratershavetohandleanumberofdifferentpromptsinthisphase,theyneedmoreguidanceintheanalysisofthepromptsandthereforearealsoprovidedwithpromptinterpretations.Thepromptwritersthendisclosetheirjudgementsanddefendtheirscoresinadiscussionwiththewholegroupmonitored
18 Testing Writing for the E 8 Standards
bythetrainer.Thisprocedureisrepeatedsothatthemajorityoftheratershavethechancetodiscusstheirscriptsandtheirjudgementswiththewholegroup.Inter-raterreliabilityandintra-raterreliabilityaremonitoredandpertinentdataoneachrateriscollectedsystematically.
TRAINING PHASE 5: MOCK RATING SESSION: IMMEDIATELY AFTER TRAINING PHASE 4 (1 ½ DAY; FACE-TO-FACE SESSION)Thesecondimportantpartofthismeetingistheanalysisandinterpretationofnewprompts thatwerepilotedona representative sampleofprospectiveAustrian testtakersandwhichmightactuallybeusedinafuturetest.Ratersaregivendetailedinformationandareinvitedtodiscussanyissuesthatarestillunclear.
Thentheactualratingbegins.Theratersreceivebookletsofpilotedscriptswhichwerewritteninresponsetoashortoralongprompt.Thereisaratingplanwithoverlapformultiplerating.Afterabouttenscriptshavebeenrated,theratersmeetwiththesupervisortodiscussanycriticalissuesthatmayhavecomeupduringtherating.Thentheyproceedtoratetheotherscriptsofthatbooklet,whichinvolvessomefreetimemanagementfortheraters.Thisprocedureisthenrepeatedfortheotherbooklets.
Thescoringsheetsfilledinbytheparticipantsprovideasetofdatafortheanalysisofratingbehaviour.Thedataareusedtogiveextensivefeedbacktoallratersontheirinter-raterreliabilityandraterseverity.
TRAINING PHASE 6: UPDATE SESSION IN THE YEAR OF THE ACTUAL TEST (1 DAY; FACE-TO-FACE SESSION)ThereareregionalstandardisationmeetingsforallraterswhomarkscriptsfromtheE8WritingTests.Inthesesessionsratersareupdatedon,forexample,anychangesregardingtheassessmentscaleused.Thentimeisspentontheanalysisandinterpre-tationofthepromptsusedintheactualtest,andbenchmarkedscriptsbasedonthesepromptsarerated.
After theupdate and the standardisation session, theon-site rating session starts,followingthesameprocedureasdescribedintrainingphase5above.
PostExamAdjustments
Although considerable efforts are taken in the training programme to minimisediscrepancies inraterbehaviour,theratingsareadjustedforanyremainingdiffer-ences in rater severity by means of multifaceted Rasch analysis after the scriptshavebeenmarked.Thisbecomespossiblebyhavingacertainproportionofscriptsmarkedbytworaters(doublerating)andanotherproportionofthescriptsbyallraters(multiplerating)sothatraterbehaviourcanbeassessedintermsofmodelfitaswellasseverity.
ReportingResults
ThepurposeoftheE8StandardsisgivingfeedbackonthewritingcompetenceofAustrian pupils in grade 8. The aim, therefore, is system monitoring rather thancertificationorselectionatthelevelofindividualtesttakers.Cutscoreswillbeestab-lished,however,toenableindividualfeedbacktotesttakersandshowwhethertheobjectivesinthenationalcurriculumhavebeenmetasregardswritingcompetenceingeneralandallfourdimensionsinparticular.Consequently,whilethetestresultsarelinkedtotheCEFR,criticalcutscoresonwhichtobaseselectiondecisionsneednotbeestablishedbythetestconstructors.Itishopedthatbyprovidingresultstoindividualteachersandschoolsthisfeedbackwillinstigateaqualitativedevelopment
19Testing Writing for the E 8 Standards
thatwill radiatebeyond regionsand spread throughout thewhole school system.Thewayfeedbackontheresultsisgiventotesttakersandotherstakeholdersisbeingdevelopedatthemoment.Incompliancewithpoliticalrequirements,onlythetesttakersthemselveswillhaveaccesstotheirindividualresultsthroughacodetheywillbegivenwhensittingtheexam.Teachersandschoolprincipalswillreceiveaggre-gateddataforthegrouprelevanttothem(class,school)viaaninternetplatform.Educationalauthoritieswillreceiveaggregatedreports.
Theinformationthatresults fromthewritingtest isreportedonthefourdimen-sionsoftheWritingScale(TaskAchievement,CoherenceandCohesion,Grammar,Vocabulary).The results for eachdimensionare reportedona scale from0 to7,whichenablesreferencetotheCEFRuptoB1.Ratingsareadjustedfordifferencesin rater severity and taskdifficulty bymeans ofmultifacetedRasch analysis.Theresultsarethereforecomparableacrossalltesttakersregardlessofwhichraterratedtheperformanceandwhatparticularprompttheperformanceisbasedon.Theproc-essofstandardsettingandCEFR-linkingwillbedescribedinmoredetailinatech-nicalreportaftertheactualtestin2013.
ConsequentialValidity
Shaw&Weir(2007,218)taketheterm‘consequentialvalidity’fromMessik1989andinterpretitinthelightofrecentliteraturetoincludewashback(influencesonteaching, teachers, learning, curriculumandmaterials) and impact (influencesonthecommunityat large).TheE8Standardscanbeenvisagedasan instrumenttoinitiatechangesinthedirectionofpositiveorbeneficialwashback.
Inandaround2008newcoursebooksforteachingEnglishtothetargetgroupwerelaunchedandanumberof themclaimtobe informedby theCEFRand theE8Standards.ThismeansthattextbookwritersarewellawareoftheE8StandardsTestsandareadaptingtheirmaterialstowardsthem.
Therequirementsforthewritingtestareclearlylaiddowninthisreportanddemon-stratewhatkindsofwritingourlearnersareexpectedtodeliver.
Theexpectationsofthetestdesignersformulatedin2008havebeenlargelyfulfilled3.Three years later a greatnumberofwriting tasks in thenew coursebooksused inAustrianschoolshavechangedinthedirectionindicatedintheTechnicalReport4of2008.Thereismuchlessscaffoldedwriting;thetasksarerealisticandauthentic;texttyperequirements,variationintexttypes,textlengthandtimeconstraintsareallinlinewiththepresenttestspecifications.Somecoursebooksalsoemphasisetheuseofparagraphsinwriting,givehintsonhowtowritegoodparagraphs,andprovidecorre-spondingexercises.ThismeansthatAustriantesttakerswhosittheE8Testsafter2011willbefamiliarwiththetestformat,theparticularrequirements,andtheinstructions.
3 Firstedition,2008,p.19:„Itishopedthatthiswillleadtolessscaffoldedwriting,thusenhancinglearnerem-powerment.TheemphasisgiventocoherenceandcohesionintheCEFRandtheE8Standardsmightalsofocusteacherattentiononthisareaandentailimprovements.“
20 Testing Writing for the E 8 Standards
E8WritingTestSpecificationsVersion03(July2011)
TheguidingdocumentsforthedevelopmentofthewritingtestspecificationsfortheE8WritingTestaretheAustriancurriculum(AHS2006;APS2008),theBIST-Ver-ordnung (BGBl.II Nr.1/2009 v. 2.1.2009)andtheCEFR (CouncilofEurope,2001).Thefirsttwodocuments listwritingcompetencesatdifferentproficiencylevels intermsoftheCEFR.
1.PurposeoftheTest
Themainpurposeofthewritingtestistoidentifystrengthsandweaknessesintesttakerswritingcompetenceandtousethis informationbothforthe improvementofclassroomproceduresandforsystemmonitoring.Whatismore,individualanddetailed test results are reported to the test takers,which isof interest to the testtakersthemselvesandtheirparents.
2.DescriptionofTestTakers
The test takers are Austrian pupils in the two different types of lower secondaryschools,theGeneralSecondarySchool(HauptschuleandNeueMittelschule),andtheAcademicSecondarySchool (AHS), towards theendofgrade8(8.Schulstufe). Pupils from all three ability groups in APS will be tested. The majority of testtakerswillbeaged14.SENpupils,i.e.thosewithspecialeducationalneeds,willbeexemptedfromdoingthetests.
3.TestLevel
ThedifficultylevelofthetestissupposedtoencompasslevelsA2toB1intheCEFR.
4.TestConstructwithE8ConstructSpace
The tables on pp. 21–22 summarise the construct space relevant for item designwithalistoftheprompttypesusedtotestthewritingcompetencesasspecifiedinthe BIST-Verordnung, targetedatlevelsA2,A2+,andB1oftheCEFR.Thetasksattheselevelsaskfor(mostly)concretecontent.ThereforetopicsarerestrictedtoareasthatcansafelybeassumedtobefamiliartothetesttakersastheyaresetdownintheAustriancurriculumandmusthavebeenincludedintheirEnglishlessons.
More specifically, the tasks display various text types and writing intentions/pur-poses.Tables1and2provideanoverviewof the rangeof text types andwritingintentions/purposesfortheproficiencylevelstested.Fortheactualconstructionofwritingitemspromptwritersaregivenspecialpromptdesignspecifications,whichclearlylistwhatkindofprompt–intermsofprompttype,level,BIST-Descriptor,topicarea,andtexttype–thepromptwritersaresupposedtocreate.
21Testing Writing for the E 8 Standards
Pro
mp
t
Typ
e
CE
FR
Leve
l
CE
FR
Des
crip
tor
Des
krip
tor
aus
BIS
T-V
O:
Sch
üler
/inn
enk
önn
en…
Top
icA
rea
Text
Typ
esIn
tent
ion/
Pur
po
se
Pri
mar
y
Aud
ienc
e
LongPrompt
B1
¡
Can
writ
eac
coun
tso
f
ex
perie
nces
,des
crib
ing
fe
elin
gsa
ndre
actio
nsin
si
mpl
eco
nnec
ted
text
.
¡
Can
writ
ea
desc
riptio
nof
an
eve
nt,a
rece
nttr
ip–
real
or
imag
ined
.
¡
Can
nar
rate
as
tory
.
¡
Can
writ
epe
rson
alle
tter
s
desc
ribin
gex
perie
nces
,
fe
elin
gsa
nde
vent
sin
som
e
de
tail.
¡
Erfa
hrun
gsbe
richt
e
sc
hrei
ben,
ind
enen
Gef
ühle
un
dR
eakt
ione
nin
ein
em
ei
nfac
hen,
zus
amm
en-
hä
ngen
den
Text
wie
der-
ge
gebe
nw
erde
n
¡
eine
Bes
chre
ibun
gei
nes
re
alen
ode
rfik
tiven
Ere
ig-
ni
sses
,z. B
.ein
erR
eise
,
ve
rfass
en
¡
eine
Ges
chic
hte
erzä
hlen
¡
ausf
ührli
cher
eK
arte
n,
pe
rsön
liche
Brie
feu
ndE
-Mai
ls
sc
hrei
ben
und
darin
auc
hüb
er
E
reig
niss
e,E
rfahr
unge
nun
d
G
efüh
leb
eric
hten
¡
Fam
ilieu
ndF
reun
de
¡
Woh
nen
und
Um
gebu
ng
¡
Ess
enu
ndT
rinke
n
¡
Kle
idun
g
¡
Kör
per
und
Ges
undh
eit
¡
Jahr
es-
und
Tage
sabl
auf
¡
Fest
eun
dFe
iern
¡
Kin
dhei
tund
Erw
achs
enw
erde
n
¡
Sch
ule
und
Arb
eits
wel
t
¡
Hob
bys
und
Inte
ress
en
¡
Um
gang
mit
Gel
d
¡
Erle
bnis
seu
ndF
anta
siew
elt
¡
Ged
anke
n,E
mpfi
ndun
gen
und
G
efüh
le
¡
Ein
stel
lung
enu
ndW
erte
¡
Um
wel
tund
Ges
ells
chaf
t
¡
Kul
tur,
Med
ien
und
Lite
ratu
r
¡
Inte
rkul
ture
lleu
ndla
ndes
kund
liche
A
spek
te
¡
(Fic
tiona
l)B
iogr
aphi
es
¡
Sim
ple
(tech
nica
l)de
scrip
tions
¡
Dia
rye
ntrie
s
¡
Dire
ctio
nsa
ndin
stru
ctio
ns
¡
Em
ails
¡
Lette
rs(p
erso
nal,
advi
ce,a
pplic
atio
n)
¡
Mag
azin
ear
ticle
s
¡
Nar
rativ
ere
port
s
¡
Leng
thy
post
card
s
¡
Sta
tem
ents
ofp
erso
nalv
iew
san
d
op
inio
ns
¡
Sto
ries
(cre
ate
ane
ndin
g;g
iven
an
en
ding
–c
reat
ea
stor
y;u
sea
vi
sual
impu
lse
toc
reat
ea
stor
y;
pe
rson
al)
¡
Toc
onve
y
em
otio
ns,
fe
elin
gs
¡
Toin
form
¡
Toc
onvi
nce,
pe
rsua
de
¡
Toe
nter
tain
,
pl
ease
¡
Tok
eep
into
uch
¡
Tod
escr
ibe
¡
Tog
ive
di
rect
ions
and
in
stru
ctio
ns
¡
Sel
f
¡
Oth
ers
E8
Co
nstr
uct
Sp
ace
Tabl
e 1: C
onstr
uct S
pace
for L
ong
Prom
pts
22 Testing Writing for the E 8 Standards
Pro
mp
t
Typ
e
CE
FR
Leve
l
CE
FR
Des
crip
tor
Des
krip
tor
aus
BIS
T-V
O:
Sch
üler
/inn
enk
önn
en…
Top
icA
rea
Text
Typ
esIn
tent
ion/
Pur
po
se
Pri
mar
y
Aud
ienc
e
ShortPrompt
A2+
¡
Can
writ
eab
oute
very
day
as
pect
sof
his
/her
en
viro
nmen
t,e.
g.p
eopl
e,
pl
aces
,ajo
bor
stu
dy
ex
perie
nce
inli
nked
s
ente
nces
.
¡
Can
writ
eve
rys
hort
,bas
ic
de
scrip
tions
ofe
vent
s,p
ast
ac
tiviti
esa
ndp
erso
nal
ex
perie
nces
.
¡
inF
orm
ver
bund
ener
Sät
ze
et
was
übe
rda
sal
ltägl
iche
U
mfe
lds
chre
iben
,wie
z. B
.
üb
erF
amilie
,and
ere
M
ensc
hen,
Ort
e,S
chul
e
¡
Fam
ilieu
ndF
reun
de
¡
Woh
nen
und
Um
gebu
ng
¡
Ess
enu
ndT
rinke
n
¡
Kle
idun
g
¡
Kör
per
und
Ges
undh
eit
¡
Jahr
es-
und
Tage
sabl
auf
¡
Fest
eun
dFe
iern
¡
Kin
dhei
tund
Erw
achs
enw
erde
n
¡
Sch
ule
und
Arb
eits
wel
t
¡
Hob
bys
und
Inte
ress
en
¡
Um
gang
mit
Gel
d
¡
Erle
bnis
seu
ndF
anta
siew
elt
¡
Ged
anke
n,E
mpfi
ndun
gen
und
G
efüh
le
¡
Ein
stel
lung
enu
ndW
erte
¡
Um
wel
tund
Ges
ells
chaf
t
¡
Kul
tur,
Med
ien
und
Lite
ratu
r
¡
Inte
rkul
ture
lleu
ndla
ndes
kund
liche
A
spek
te
¡
(Fic
tiona
l)B
iogr
aphi
es
¡
Sim
ple
(tech
nica
l)de
scrip
tions
¡
Dia
rye
ntrie
s
¡
Dire
ctio
nsa
ndin
stru
ctio
ns
¡
Em
ails
¡
Lette
rs(p
erso
nal,
advi
ce,a
pplic
atio
n)
¡
Mag
azin
ear
ticle
s
¡
Not
es
¡
Not
ices
¡
Pos
tcar
ds
¡
Sta
tem
ents
ofp
erso
nalv
iew
san
d
op
inio
ns
¡
Toc
onve
y
em
otio
ns,
fe
elin
gs
¡
Toin
form
¡
Toc
onvi
nce,
pe
rsua
de
¡
Toe
nter
tain
,
pl
ease
¡
Tok
eep
into
uch
¡
Tod
escr
ibe
¡
Tog
ive
di
rect
ions
and
in
stru
ctio
ns
¡
Sel
f
¡
Oth
ers
A2
¡
Can
writ
esh
ort,
sim
ple
fo
rmul
aic
note
sre
latin
gto
m
atte
rsin
are
aso
f
im
med
iate
nee
d.
¡
Can
writ
esh
ort,
sim
ple
im
agin
ary
biog
raph
ies
and
si
mpl
epo
ems
abou
tpeo
ple.
¡
kurz
e,e
infa
che
Not
izen
und
M
ittei
lung
ens
chre
iben
,die
si
cha
ufu
nmitt
elba
re
B
edür
fnis
seb
ezie
hen
¡
einf
ache
Tex
tez
. B.z
u
B
ildim
puls
eno
der
S
chlü
ssel
wör
tern
(key
wor
ds)
sc
hrei
ben
¡
kurz
e,e
infa
che
Bio
grafi
enu
nd
a
nder
eei
nfac
hefi
ktio
nale
Te
xte
schr
eibe
n
E8
Co
nstr
uct
Sp
ace
Tabl
e 2: C
onstr
uct S
pace
for S
hort
Pro
mpt
s
23Testing Writing for the E 8 Standards
5.StructureoftheTest
Thetest contains two sections.Section1 consistsof a shortwriting taskwithanexpectedresponseof40to70words.Section2consistsofalongwritingtaskwithanexpectedresponseof120to180words.
ThetwotaskswillbeassessedseparatelyonthebasisofthefourdimensionsoftheWritingRatingScale.
6.TimeAllocation
Totaltestingtimeavailable:45minutes.Timeforadministrationatthebeginning(handingouttestbooklets):5minutes.Time for administration at the end (word count and collecting test booklets): 5minutes.Workingtime:35minutes.Theshorttaskshouldtakeabout10minutes,thelongtaskabout20,with5minutesforrevision.
7.ItemFormats
Thecandidates’scriptswillbehandwrittenonthepagesprovidedinthetestbook-let.Thewritingtaskisguidedbypromptsthatensurethatthecandidatesproduceenoughlanguagethatmakesreliableandvalidassessmentpossible.
Thepromptsmaycontainblackandwhitepicturesordrawings.TheyneedtobeappropriatefortheageandatalanguagelevelnohigherthanA2.Inputtextsshouldbe authentic, if at allpossible, andas longasnecessary to contextualise the task.Ideally,theyshouldnotbelongerthan50words(excludingcontentpoints).
Thepromptsthataredevelopedaretobefreeofstereotypes.Theyoffertheoppor-tunity towrite fromexperience,butaredesignednot to intrudeon the students’personalfeelings.
Promptsneedtostatethereasonforwriting,theintendedaudienceandtherequiredtext type.Theworking time available and thenumber ofwords for the requiredlengthofthetextswillbeindicatedintheinstructions.
8.LanguageLevelforInstructionsandPrompts
All relevant instructions andprompts are inEnglishwith additional informationgivenbythetestadministrator inGerman.However,theymustbeformulatedinlanguage that is well within reach of the candidates’ expected language level andtherefore easily understandable for all test takers.Test takers must not be put ata disadvantagebecause theyhavedifficultyunderstanding the instructions or theprompts.ThereadingcompetenceexpectedisCEFRlevelA2.
9.AssessmentwithWritingRatingScale
InE8testingthemostsignificantcompetencesneededforwritingareidentifiedforassessmentpurposes,thefirstandforemostofwhichisthecommunicativecompe-tencedemonstratedinanappropriateresponsetothetask.Inpracticaltermsthismeansthatallexpectedcontentpointsofthepromptaretobeclearlyandmeaning-fullymentionedbythetesttakers.Forthehigherbands,elaborationofsomecontentpointsisrequired.Thesecondistheabilitytoproducefluenttextbyusingadequatedevicestocreatecoherenceandcohesiononsentence,paragraphandtextlevel.The
Tabl
e 2: C
onstr
uct S
pace
for S
hort
Pro
mpt
s
24 Testing Writing for the E 8 Standards
thirdisagoodknowledgeofarangeofgrammaticalstructuresandtheabilitytousethemaccurately,andthefourthisthechoiceofvocabularythathasacertainrange,isaccurateandrelevanttothecontent.
Thereforethetestisdesignedtoelicitlanguagesamplesthatallowthecandidatestobeassessedinfourareas:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary. The two tasks are assessed separately by trained raters, using ananalytic rating scalebasedonthese fourdimensions.Multiple-ratinganddouble-ratingofasufficientlylargesampleofscriptsensurereliability.Differencesinraterseverityareadjustedforintheprocessofmulti-facetedRaschanalysis.
Pages25–27includethefourdimensionsoftheanalyticratingscaleforwritingandmoredetailedscalesforTaskAchievement.
25Testing Writing for the E 8 Standards
TaskAchievement CoherenceandCohesion Grammar Vocabulary
7
¡ completetask achievement
¡ cohesiononboth sentenceandparagraph levelusingalimited numberofcohesivedevices¡ clearandcoherenttext
¡ goodrangeofstructures¡ relativelyhighdegreeof grammaticalcontroland fewinaccuracieswhichdo notimpaircommunication¡ messageclear
¡ goodrangeofvocabulary communicatingclearideas¡ generallyaccurate vocabulary¡ formulationssometimes variedtoavoidrepetition
6
5
¡ goodtask achievement
¡ goodsentencelevel cohesionasalinear sequenceonasimple level¡ someparagraphlevel coherenceandcohesion¡ fairlyclearandcoherent text
¡ generallysufficientrange ofstructures¡ occasionalinaccuracies whichcanimpair communication¡ messageclear
¡ sufficientrangeof vocabularycommunicating clearideas¡ occasionallyinaccurate vocabulary¡ majorerrorspossible whenexpressingmore complexideas
4
3
¡ sufficienttask achievement
¡ somesimplesentence levelcohesionusing simpleconnectorslike ‘and’,‘but’and‘because’¡ frequentlackof coherenceandcohesion onparagraphlevel¡ textoftenlacksclarityand coherence
¡ limitedrangeofsimple structures¡ frequentlyinaccuratewith basicmistakes,generally withoutcausingbreak- downofcommunication¡ messageusuallyclear
¡ limitedrangeof vocabularymostly communicatingclear ideas¡ frequentlyinaccurate vocabularycontrollinga narrowlexicalrepertoire¡ tendencytousephrases fromtheprompt
2
1
¡ sometask achievement
¡ basiclinearconnectors (‘and’,‘then’)onwordor wordgrouplevel¡ textnotcoherent
¡ extremelylimitedrangeof simplestructures¡ limitedcontrolcausing frequentbreakdownof communication¡ messageseldomclear
¡ extremelylimitedrangeof vocabularycommunicating fewclearideas¡ mostlyinaccurate vocabularyfrequently causingbreakdownof communication¡ severalchunksliftedfrom theprompt
0¡ notask achievement
¡ notenoughassessable language
• notenoughassessable language
¡ notenoughassessable language
WritingRatingScale(July2011)
26 Testing Writing for the E 8 Standards
Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/ inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngrading byoneband(twobandsifrequirementsarenotmetatall).
Textlength:120–180words
Below-lengthanswers • Anythingbelow110wordswillbepenalised(downgradingbyoneband).• Fewerthan80words-downgradingbytwobands(ifthegeneralscoreisband3or above;otherwise,downgradebyoneband).• 50–79words:Assessmentisconfinedtobands1and2.• Answerscontainingfewerthan50wordsreceive0.
Over-lengthanswers• Morethan180words:thewholeanswerisassessed.
ExtendedScales
7¡ completetaskachievementwith
¡ allcontentpointsmentioned&threeormoreofthemelaborated
6
5¡ goodtaskachievementwith
¡ 85%ofthecontentpointsmentioned&twoorthreeelaborated¡ orallcontentpointsmentionedandoneortwoelaborated
4
3¡ sufficienttaskachievementwith
¡ 65%ofthecontentpointsmentioned&oneortwoelaborated¡ orallcontentpointsmentionedwithoutelaboration
2
1¡ sometaskachievementwith
¡ 50%ofthecontentpointsmentioned&noelaboration
0 ¡ notaskachievement
TaskAchievementLongTasks
27Testing Writing for the E 8 Standards
Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/ inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngrading byoneband(twobandsifrequirementsarenotmetatall).
Textlength:40–70words
Below-lengthanswers• Anythingbelow30wordswillbepenalised(downgradingbyoneband).• Fewerthan30words:Assessmentisconfinedtobands1and2.• Answerscontainingfewerthan20wordsreceive0.
Over-lengthanswers• Morethan80words:thewholeanswerisassessed.
ExtendedScales
7¡ completetaskachievementwith
¡ allcontentpointsmentioned&oneortwoofthemelaborated
6
5¡ goodtaskachievementwith
¡ allcontentpointsmentioned&possiblyoneelaborated
4
3¡ sufficienttaskachievementwith
¡ 70%ofthecontentpointsmentioned&noelaboration¡ or50%ofthecontentpointsmentioned&oneelaborated
2
1¡ sometaskachievementwith
¡ 50%ofthecontentpointsmentioned&noelaboration
0 ¡ notaskachievement
TaskAchievementShortTasks
28 Testing Writing for the E 8 Standards
10.PromptsandPerformanceSampleswithJustifications
Thetaskprototypesbelowaretakenfromthe2007E8writingtest.ItisimportantthatthetasksarestructuredandcontainanumberofcontentpointssothatTaskAchievementcanbemeasured.4
TheinstructionsbelowhavebeenreducedtoaminimumbecausemostthingsareannouncedbythetestadministratorinGermanbeforethetesttakersopentheirtestbooklets.TheinstructionsinEnglisharetoensurethattheykeepthemainpointsinmind,butalsothatlearnerswithafirstlanguageotherthanGermanhavethesamefairchancetodothetask.
10.1LongTask
Instructions
LongPromptfromE8WritingTest2007
10.1.1Script1
4 MorepromptsareavailableontheBIFIEwebsite:http://www.bifie.at/freigegebene-items
Read the instructions carefully and then write your text on the next page.Time: 20 minutesText: 120–180 wordsUse paragraphs.In your text, try not to use language from the task below.
You have just moved to another town/village. Write a letter to your American/English friend in which you tell him/her about your new situation.
Inform him/her about ● your new place of living. ● the reason for moving.Describe ● the town/village you’re living in now (buildings, people,…). ● your new home.Tell him/her about ● the first days of your ‘new life’ (new school, teachers,…). ● how you feel about your new situation.
Dear Bill, how are you? Now I’m living in Vienna. That is in Austria. It is a very big city with nice people. There is also a fun fair called “Prater”. My parents got divorce and so I’m living here with my mother. There are wonderful buildings in this city like the animal park “Schönbrunn” and many castles. I like the river “Donau” very much, because I often go swimming there. My new house is very big and next to it is a forrest. I like that. The first day of my “new life” was not so good. When I came into my class most of the pupils laughed at me but the teacher was nice. I hope you will write back.
Yours, Raphael
(124 words)
29Testing Writing for the E 8 Standards
Justifications
TaskAchievement 5
Thetextmeets the text type requirements, iswithin theword limit set, andusesan informal register suitable fora letter toa friend.Thecandidateworkshiswaythroughthecontentpoints,onlyjusttouchingonthelastone.HowhefeelsneedstobeinferredfromstatementslikeThe first day of my ‘new life’ was not so good, most of the pupils laughed at me, and the teacher was nice.Alltheothercontentpointsarementioned.Whereas content point 3 shows good elaboration, the elaboration ofcontentpoints1and4islesssuccessful.ThepassageNow I’m living in Vienna. That is in Austria. canbe taken asmentioning contentpoint1 and the following twosentences(It is a very big city with nice people. There is also a fun fair called “Prater”.) canbeseenasanattemptatelaboration.
Withcontentpoint4elaborationisjustasthin.Thestatement My new house is very bigmentionscontentpoint4,butdoesverylittleinthewayofdescribingthenewhome(likeforinstancehowmanyrooms,whathis/herroomlookslike,etc.)andthesentenceadded(next to it is a forrest. I like that.)isnotreallydescriptive.Sothewritercannotbegivenmuchcreditforthisattemptatelaboration.
Thismeetsthedescriptorintheratingscaleforband5:“goodtaskachievementwithallcontentpointsmentionedandoneortwoelaborated.“
CoherenceandCohesion 4
Fromtheverystarttheideasdonotconnectwell.Theintroductoryquestionhow are you?islefthangingtobefollowedbyNow I’m living in Vienna. ThethreeshortsentencesthatfollowaddsomeinformationaboutVienna.Butthenextidea(My parents got divorce)meetsthereaderunpreparedandthereisnolinktosmooththetransition. At this point it becomes clear that the three sentences about Viennashouldactuallyhavebeenmovedtocontentpoint3.Thiswaythefirstunitofthetextcomprisingcontentpoints1and2wouldhaveflowedbetter.Textorganisationisbasedonthesequenceofthecontentpointsprovidedwiththewriter’shandprac-ticallyinvisible.
Thetextiscertainlycohesiveatsentencelevelbuthardlyatparagraphlevel.Movingfromoneideatotheother(tellingaboutthenewplace–thereasonformoving–describingthenewtown)maybeimpliedbytheorderofthecontentpoints,buttheabruptwaythishasbeendoneshowsthewriter’slimitations.
However,thetextis“fairlyclearandcoherent”.Sentencelevelcohesionisgoodandsomebasicconnectorsareusedtodelivera“linearsequence[ofpoints]onasimplelevel”.Thereissomeparagraphlevelcoherence,buttherearenotransitionsorlink-ingdevicesbetweenthevariousideaspresented.Whileband5couldbeconsideredfor thisperformance, the lackofparagraphorganisation leadstodowngradingbyonebandtoband4.
Grammar 5
Atfirstsightthestructuresusedaregenerallysimple,manyofthesentencesareveryshort.Therearetwoidenticalcasesofthepresentprogressive(I’m living),fourusesofis, there is and there are,fourcasesofthepresentsimple(I like [2x], I often go, I hope), threesimplepasttenseforms(was, came, laughed)andonefuture(will write).
30 Testing Writing for the E 8 Standards
Althoughallofthesetenseformsarebasic,theyseemsufficienttoexpressthewriter’sideasandtheyhavebeenusedcorrectly.Moreover,thewriterusescomplexsentencescorrectly,whichhintsatmorecomplexlanguagecompetences.
Thereareactuallytwosubordinateclauses (because, when) andathirdwiththefinalsubordinatorthatomittedin I hope [THAT] you will write back. Moreover, ifonetakesacloselookattheothersentences,wefindgoodpost-modificationofnounphrases(big city WITH…, fun fair CALLED…, buildings…LIKE…).Eventheco-ordinateclauseshaveatwisttothem…and SO…, and NEXT TO IT….Sothereisdefinitelysufficientcomplexitytojustifyband5.
Vocabulary 5
Thecandidatehasasufficientrangeofvocabularytoexpresshimself.Ontheonehand,thereisratherunexpectedvocabularyused(nearly)correctly(got divorce, fun fair, laugh at)asevidenceofagoodrangeofvocabulary,ontheother,thereisalsorepetitionofverysimpleexpressionssuchasnice, big and like,andtherearesomeoccasional inaccuracies (got divorce, forrest). The majority of words belong to themost frequently used basic English vocabulary, but the ideas communicated arealwaysclear.
Agoodrangeofvocabularycanonlyrarelybeseen,asmostpartsofthetextdisplayasufficientrange.Whatisdefinitelylackingistheabilitytovaryformulationstoavoidrepetition(band7),sothetextisband5.
10.1.2Script2
Justifications
TaskAchievement 4
Thetextmentionsallcontentpointswiththeexceptionofcontentpoint2,wherereasonsformovingshouldbestated(=85%).Thereissomeelaborationofcontentpoint4bydescribingthesizeofthegardenandcontentpoint5bysupplyingthereaderwithsomeadditionalinformationabouthisnewfriends(they speak very good english)andhisachievementsatschool(lotsofEnglishlessonsandhimdoingwellinEnglish),althoughthisonlyimplicitlyreferstothecontentpoint“thefirstdaysofyour'newlife'”.Contentpoint6hasbeenliftedfromthepromptaddingthewordgood.
Ascontentpoints1and6haveonlybeendealtwithinanextremelybasicwaybymerelymentioningSalzburgand liftingaphrase fromthepromptandtheelabo-rationofcontentpoints4and5isratherweak,adowngradetoband4isthecon-sequence.
Hi Steven!In Salzburg it is very cool and I’m living in a small flat with 5 rooms two bedrooms a kitchen a livingroom and a bathroom. Our garden is not so big, but big enough for us. The building very beautiful and it give no skyscrapers and it is very hot. I always go in the garden and I lie in the sun. The people are very funny and they accept that I speak english. I have got two new friends and they speak very good english. The teachers are very good and we have a lot of english and I’m the best one, but in Deutsch I’m very bad. I feel very good with my new situation and I wish all my old friends and the teachers a good luck for the next time and I hope you always wish me a good next time.Yours Olav! (149 words)
31Testing Writing for the E 8 Standards
CoherenceandCohesion 2
Textorganisationisquitelowwithsimpleadditionasthedominatingstructuringprinciple.Thesimplest connectorandoccurswithundue frequency,proving thatideasaremostlystrungtogetherwithoutexpressinglogicalrelations.Apartfromthesecondsentence (Our garden is not so big, but big enough for us)thetextdoesnotreadwellbecausesomesentencesthatfolloweachotherhavelittleornoconnectionatcontentlevel,sothetextlackscoherence.Somechunksoflanguagethathavelittleincommonareoftenjoinedinonesentence,e.g.The building very beautiful and it give no skyscrapers and it is very hot.
Therefore,thistextischaracterisedbyanoticeablelackofclarityandcoherenceandsomeratherbasicsentencelevelcohesion.Thiswouldpointtowardsaweakband3.Astherearenoparagraphs,thetexthastobedowngradedtoband2.
Grammar 3
Thewriteruses aquite limited rangeof simple structures correctly, repeating thesamebasicpatternwith little variation.Heonlyusespresent tense structures, al-thoughcontentpoints2and5would invite theuseofpast tense.Therefore, therangeofstructurescannotbeconsideredsufficientforthepurposesofthetask.EvenwithinthenarrowframeofpresenttensesentencestructuresthereisinappropriateuseofthecontinuousforminI’m living in a small flat. Thesimplemessageisusuallyclear,althoughanerrorsuchasit give no skyscrapers causesbreakdownofcommuni-cation.Similarly,thephrasethe building very beautiful leavesusundecidedwhetheritshouldrefertothehousewherethewriterlivesorthebuildingsinSalzburg.Allthiswouldsuggestaveryweakband3,which,however,issupportedbytherelativelyhighdegreeofcorrectness.
Vocabulary 4
Thetextshowssuccessfulcontrolofalimitedrangeofvocabulary,withsomegoodphrasesstickingoutsuchasbig enough for us, skyscrapers, orthey accept that I speak english. Thesimplevocabularyusedinthefirstpartofthetextcommunicatesmostlyclear ideas,but in the last sentence thewriter seems tobeattempting toomuch,leavingthesafearea,andthisresultsinseveralbreakdowns(I wish them a good luck; L1:for the next time; I hope you always wish me a good next time)demonstratingthelimitations,asdoestheuseofDeutschforthesubjectGerman.Thenarrowlexicalrepertoireandalsothetendencytoliftphrasesfromtheprompt(I feel very good with my new situation)wouldindicateaband3,buttheoccasionalneatexpressionandthefactthatsevereproblemsappearonlywhentryingtoexpressamorecomplextrainofthought(toleratedatband5)justifyaweakband4.
32 Testing Writing for the E 8 Standards
10.2 Short Task
Instructions
Short prompt from E8 Writing Test 2007
10.2.1 Script 3
Justifications
Task Achievement 7
The register and the layout are clear indications that this text is an email. The salutation and closing formulas are most appropriate and for these reasons text type requirements are perfectly met. All content points have been mentioned and there is elaboration of content point 1 (It was so great!, I liked the games we played, the food was excellent) and content point 3 as the candidate makes enquiries about an up coming event in the near future (I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too?), so we have complete task achievement – band 7.
Coherence and Cohesion 7
The text admirably incorporates qualities of spoken English (Let’s talk about, Oh and, I nearly forgot, Okay I have to), which one would expect in an informal email, and which make it flow well. A number of cohesive devices are used to connect groups of sentences together very well, such as lexical cohesion (party-it-party), conjunctions (because-and), backward and forward referencing (we played-tell your mum that), and there is evidence of good linear sequencing of points making it a clear and coherent text, pointing it towards band 7. However, there are two abrupt changes in the linear sequence of the text What are you going to do and I nearly forgot, but as paragraphs are not expected in short texts, it remains a band 7.
Your friend’s birthday party was a few days ago. Write an email to tell him/her that you liked the party.
• Tell him/her why you liked the party.• Tell him/her what you liked best.• Ask your friend when you are going to meet again.• Suggest something for the next weekend.
Dear Daisy, how are you? Let’s talk about your party. It was so great! I liked the party best, because Lukas was there. And I liked the games we played. Oh and tell you mum, that the food was excellent! What are you going to do on Sunday? Maybe we can go to cinema or swimming. Tell me please if you have time. I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too? Okay I have to help my mum with dinner.Love you big kissYours,Aida (95 words)
Read the instructions carefully and then write your text on the next page.Time: 10 minutesText: 40–70 words In your text, try not to use language from the task below.
33Testing Writing for the E 8 Standards
Grammar 7
Thereisarelativelyhighdegreeofgrammaticalcontrolwithonlyoneslightslipinaccuracy.However,theomissionofthedefinitearticleinthephrasego to cinema doesnotimpaircommunicationandthemessagethroughoutthetextisclear,suggestingband7.Thecandidate’sgooduseofthepresent,past,going-to-future,Saxongeni-tive,andasubordinateclauseproveshe/sheisabletoaddressallthelanguagefunc-tionsincludedintheprompt:toinform,toask(how are you?, What are you going to do on Sunday? are you invited too?),andtosuggest (Maybe we can go to cinema).Thisindicatesthegoodrangeofstructuresthecandidateisabletouseaccurately,thussupportingastrongband7.TheverycasualsentenceOkay I have to help my mum with dinnerwithnear-nativeomissionofthedefinitearticle (with dinner),isfurtherevidencethatclearlypointstoaband7.
Vocabulary 7
The vocabulary elicited by the prompt points towards band 7. Not only does itcontainagoodvarietyofappropriateandaccuratecontentwordsexcellent, invite, dinner, but also many collocations that are equally appropriate and accurate andlenda certainnaturalness to the text (the food was excellent; what...going to do on Sunday; have time; in two weeks; I nearly forgot; love you big kiss).Anotherindicatorforband7isthecandidate’schoiceofwordswhichenablehertogethermessageacrossveryclearlythroughoutthetext.Someofthephrases(Let’s talk about; Oh and tell; I nearly forgot; Okay I have to)indicateacertainairof'chattiness'tothetext.Furthermore,theuseofLove you big kiss, asanalternativeoradditiontothecommonclosing lineYours, which exemplifies how the candidate canvary formulations toavoidrepetition,isadditionalevidencethatthisisaband7.
10.2.2Script4
Justifications
TaskAchievement 4
Thetextclearlyfollowstheorganisationofthecontentpoints(liked the party because it was cool/liked the girls and waterpipes best, do such a party tomorrow again?, go to the city …).Although thefirst contentpoint ishandled verybriefly and contentpoint3featuresasanindirectrequestonecanstillsaythatallcontentpointshavebeenmentioned,whichhintstowardsband5.Bands6or7cannotbetakenintoconsiderationbecausethereisnoelaboration.Althoughcontentpoint4consistsoftwosentences,theintroductoryquestiontocontentpoint4isnothingelsethanarepetitionofthequestionincontentpoint2(Can we do such a party again…?, … we can do it again, or?)andtheactualcontentpoint4iscoveredbythefollowingsuggestion(I have a better idea …).
Astheclosingformulaismissing,texttyperequirementsareonlypartlymet(saluta-tion:Hey Kevin!!). Thisleadstodowngradingbyonebandtoband4.
Hey Kevin!!! I’ve liked your party because it was very cool. What I liked most were all the nice girls and all the nice waterpipes. Can we do such a party again tomorrow? It would be very nice. But please, buy more waterpipes, and more grass! And next week-end, we can do it again, or? I have a better idea, we can go to the city and chillout at a concert.(65 words)
34 Testing Writing for the E 8 Standards
CoherenceandCohesion 5
Thefirstpartofthetext(sentences1–5)showsgoodsentencelevelcohesionthroughbackwardandforwardreferencing,thematicfrontingandlexicalrepetition(party- it; what I liked most; such a party; it would be ...; waterpipes).Thispart,thoughnotmarked as such in the layout, even shows some good paragraph level coherencewithinthenarrowscopeofashorttext.Theconcludingpart(And next weekend....)providesmorebackreferencinginitforparty,andthefollowingsuggestionisthematicallylinkedtothepreviousquestionthroughI have a better idea.
Therefore,wecanascertain“goodsentence level cohesion”and“somegoodpara-graphlevelcohesion”,whichjustifiesband5.
Grammar 5
Thetextclearlyshowsasufficientrangeofstructurestofulfilthetask,whichrequiresthedescriptionofapastevent,thequestionforafutureevent,andasuggestion.Thepasteventiscorrectlydescribed(was very cool; what I liked most were),thequestioniscorrectlyphrasedandfollowedbytwostatementsmakingcorrectuseofasubjunc-tive(would)andapoliteimperative(But please, buy more…). Thetextalsofeaturesacomparative,thecorrectuseofamodalverbform,andasubordinateclause.
Thetextisgenerallyveryaccurate;themistakesindescribingapastevent(I’ve liked) andtheL1interferenceintheuseofor?insteadofatagquestiondonotimpaircom-municationanditisclearwhatthewriterwantstosay.Hence,band5isappropriate.
Vocabulary 4
Thecandidateshowssufficientlexicalrangetofulfilthetaskandcommunicatehisideas,whichwouldhinttowardsband5.However,indoingsohemakesuseofverysimplevocabulary: very cool, all the nice girls, go to the city, concert, was/were, buy, more, againetc.Theonlywordsstickingoutare waterpipes, grass, such a and to chill out.Althoughthewordswaterandpipeshavebeencombinedandaresemantically‘new’,thewordsthemselvesarestillextremelybasic,asis grass.
Thisleavesuswithas littleas to chill out and such aastheonly lexical itemsthatwouldgobeyondalimitedrange.
Ontheotherhand,vocabularyisaccuratewithto do a partyastheonlyincorrectuse,whichdoesnotcreateanymisunderstanding.Takingintoconsiderationthatthewriterdoesnottakeanylexicalrisksandtheitemsusedaresimple,limitedinrangeandrepetitiveadowngradingfromband5toband4isappropriate.
10.2.3Script5
My friend have at 7.5. birthday. The birthday-party was very good. We had a lot of fun on the party. We play playstation and we went play football but the best was that we are ate pizza. We go at the weekend to a football match.(46 words)
35Testing Writing for the E 8 Standards
Justifications
TaskAchievement 05
Thecandidatetriestoworkhis/herwaythrough(mostof )thecontentpoints.Con-tentpoint1ismentionedasthestudentwritesthatthe party was very goodandthattheyhad a lot of fun. Thenhe/shementionsafewthingsthattheydidattheparty.Itcanbeassumedthat theseactivitiesare thereasonswhyhe/she likedtheparty.Contentpoint2isshortbutok(the best was that we are ate pizza). Contentpoint3ismissingcompletely,andcontentpoint4isnotrecognisableasasuggestion,thereforenotmentionedcorrectly (We go at the weekend to a football match).
Sothereissometaskachievementwith50%ofthecontentpointsmentionedbutnoelaboration,whichpointstowardsband1.
However, thetextdoesnotmeettexttyperequirementsatall.Thecandidatewasaskedtowriteanemailtoafriend.Thefirstthingthatisstrikinghereisthatthereisnosalutationandnoclosingformula.Allinall,thistextreadslikeareportratherthananemailmessage.Asaconsequence, the texthas tobedowngradedby twobands,whichmeansitistobeplacedatband0.
CoherenceandCohesion 2
Atfirstsightcoherenceandcohesioninthistextseemtoberatherpoor.However,thereissomesimplesentencelevelcohesionconnectingtheideasofcontentpoints1and2(friend’sbirthday–thebirthdayparty–funattheparty–activitiesattheparty). Wefindoneinstanceofacohesivedefinitearticle(theparty)andthelogicalconnectionbetweenWe had a lot of fun on the party. We play playstation and [...] isactuallyfairlyobvious.Moreover,inonesentencesomebasicconnectorsareusedfairlysuccessfully (and inordertolinktwomainclauses;correctuseofbut+that-clause).
Thelastsentencebreaksoffthecoherencethatwasthereanddisruptstheminimalqualityofflowthatthefirstpartofthetexthas.That is why coherence and cohesion is clearly better than band 1, but not goodenoughforband3,hence(aweak)band2.
Grammar 1
Thetextshowsanextremelylimitedrangeofsimplestructures.Mostsentencesfol-lowaverybasicsubject-predicate-objectpattern.
Thetaskbasicallyrequiresthedescriptionofapastevent,thequestionforafutureevent,andasuggestion,soaltogetherrathersimplestructures.However,evensomesimplestructuresareusedincorrectly (we went play football; we are ate pizza). Whatismore,thereisnoquestion,nophrasethatindicatesasuggestion,andeventheverybasicverbformstoindicateapresent,pastorfuturetimeaspectarequiteoftenusedincorrectly (my friend have; we play playstationfordescribingapastevent;We go at the weekend to a football match).
5 AsbandzeroforTaskAchievementhasnotbeenreacheddirectly,butafterdowngrading,theotherdimensionsareassessedwithoutanyrestrictions.Script6demonstratesthatwhenTaskAchievementisaclearzerofromtheoutset,theotherdimensionsarenotratedasthereisnotenoughassessablelanguage.
36 Testing Writing for the E 8 Standards
Inaddition,thereareexamplesofthemisuseofprepositionsinverybasicphrases(on the party; at 7.5.),andwrongwordorderinverybasicsentencestructures (My friend have at 7.5. birthday; We go at the weekend to a football match).
Althoughthereisnobreakdownofcommunicationandthemessageisusuallyclear,theextremelylimitedrangeandincorrectuseofsimplestructuresmakethistextaperformanceatband1.
Vocabulary 1
Thetext showsanextremely limited rangeofvocabulary,usingverybasicwords,e.g.friend, birthday, birthday party(mentionedintheprompt),a lot of fun, football match, pizza, andformsof have, be, play and go.
Althoughthecandidateisabletocommunicatehis/herverysimpleideassuccessfullyand(mostly)accurately,theextremelylimitedrangeofvocabularyoverrulesaccuracyandthismakestheperformanceband1.
10.2.4Script6
Justifications
TaskAchievement 0
Thereobviouslyisnotaskachievementasthecandidatehasmerelycopiedthegiveninputtextandthecontentpointsinsteadofdoingwhattheysaid.Theonlyadditionmadebythewriterisminimalandbearsnorelationtothetaskset.
Onthegroundsoftaskachievementbeingbandzero,theotherthreedimensionsarenotassessableaswecouldonlyassessthelanguageusedinthetaskdescription,butnotanyofthecandidate’scompetences.
CoherenceandCohesion 0
Noenoughassessablelanguageintermsofcoherenceandcohesion.
Grammar 0
Toolittleindependentlyproducedlanguagetoallowassessment.
Vocabulary 0
Toolittleindependentlyproducedlanguagetoallowassessment.
My friend’s birthday party was a few days ago. Write an email to tell ihm that you liked the party. Tell him why you liked the party. Tell him what you liked best. Ask your friend when you are going to meet again.Suggest something for the next weekend.My best friend’s. Name von my best friend’s is …(58 words)
37Testing Writing for the E 8 Standards
ScaleInterpretations
Intheratertrainingcoursesoverthepastthreeyears ithasbecomeclearthattheratingscaleasitstandsisbynomeansself-explanatoryandreadyforgeneraluse.Therefore,somecommentsonhowtoreadandinterpretthescaleareaddedhere.
ScaleInterpretation–TaskAchievement
The scale on Task Achievement has no direct correlation with the CEFR andassessesthecontentcomponentsofatextandtexttyperequirements.Thewaythedescriptorsareformulatedleavesroomfordifferinginterpretations,whichwillleadtodivergingassessments.Inordertoreachtheaimsofthetrainingcourseandthestandardisationmeetings,acommonunderstandingofwhatalltheelementsinthescalesmeanandhowtheyrelate toeachother is required.To improve inter-raterreliabilityweneedtoclarifythekeyterms.
CONTENTThefirst issuethatseemssimpleenoughistodecidewhetheracontentpointhasbeenmentioned ina scriptornot.Whereas this isquite straightforward inmostcases, there is room for confusion when, for instance, some key words from thepromptappearinthetext,butthelanguagearoundthemdoesnotmakemuchsense.Asitsaysinthetestspecificationsthat“allexpectedcontentpointsofthepromptaretobeclearly and meaningfullymentioned”(p.23),suchacontentpointwouldnotbeconsideredasbeingmentioned.
WhatratertraineeshavefoundmostchallengingandconfusingwhenassessingTaskAchievementisdistinguishingbetweenmentioningacontentpointandelaboratingit.Withreferencetothelongpromptonp.28thefollowingexample,takenfromoneofthescriptsusedinthetrainingsessions,canservetodemonstratewhat“men-tioningacontentpoint”means:We live now in New York near the Central Park, we moved because my mum had not found a job.Inthissentence,contentpoints1and2arementioned.Thetextgoeson:But in New York my mum has a good job. Thissentenceextendspoint2alittle,butasitislittlemorethanareformulationoftheprevioussentence,itcannotbeseenaselaboration.
Elaborationofthispointcouldhavebeensomethinglikethis:In New York she works as a secretary in a bank on the 35th floor of a high building in Manhattan and is quite happy. Or:In New York she sells pancakes in the streets, and she is happy.Goodelabo-rationtheninvolvestheintroductionofanewidea,arealextensionofwhathasbeensaidbefore.
Lesssuccessfulelaborationissomethingyourecognisewhenyouseeit.Lookatthisexample–Contentpoint5ofthelongpromptonp.28:“Tellhim/heraboutthefirstdaysofyour‘newlife’(newschool,teachers,…)”: I have new friends but you are forever my best friend. The new school is very big. The teachers are sometimes unfriendly. And the school colleagues are not polite.
Thefirstsentencementionsnewandoldfriends.Thenitmovesabruptlytothecuesfromthepromptandaddssomesimplewordstoeach.Finally,thereisanewsen-tencebasedonthesamepattern.–Thereissomeelaborationhere,nodoubt,butitisnotverygood.Thiswillbereflectedintheassessment.
Whereas on one level we can assess Task Achievement quantitatively by simplycountingthecontentpointsmentionedandelaboratedrespectively,thediscussion
38 Testing Writing for the E 8 Standards
abovemakesitclearthat,inadditiontothis,thereisalsoaqualitativecomponenttobeconsidered.ThefirstquestionisHow many?, butthesecondisHow good?.
TEXTS AT BAND ZEROInthefollowingcasesthereisnoassessmentinanyofthefourdimensions:
�� Textsthatdonotdealwiththegiventopicandthecontentpointslisted.�� Textsthatareextremelyrude,sexist,racist,orpropagatingviolence.�� Textsthatshowanattemptatdealingwiththetopicbutdonotcontainenoughassessable language, i.e. fewer than50words in long texts and fewer than20wordsinshorttexts.
Textsthatareplacedatbandzeroduetodowngrading(texttyperequirementsand/ortextlength)areassessedintheotherthreedimensions.
TEXT TYPE REQUIREMENTSAnothertermintheTaskAchievementscale thathas invitedfrequentdiscussionsin the training sessions is text type requirements. The issue, however, is greatlysimplifiedbythecontextofuse,whichistestedatE8level.Thissimplymeansthatinthegiventestsituationthereislittleroomforstylisticvariationonthepartofthetest takers. Inmostcasesan informalregister is theonlyonetheyhaveaccess to,andtesttakersarenotexpectedtointroducestylisticdifferencesrelatedtoparticulartexttypeslikemagazinearticles,reports,diaries,letters,oremails.Testtakerswillbeusingamoreorlessinformalstyleinalltheirtextsandtheonlythingtheyneedtoknowishowtoopenandclosealetteroranemail.Inaratermeetingithasbeendecidedtoconsideremailsasslightlymoreinformalthanletters,butstillrequiringsomekindofsalutationandclosingformula.
Inpracticalterms–andinthecontextofE8testing–thismeansthatmeetingoftexttyperequirementsisconsideredagivenrequirementsothetesttakersdonotgetanybonusforit.Itisonlyincaseofproblemsinthisareathatwetakethisintoconsid-erationandreactbydowngradingonebandortwobandsrespectively.Thefollowingguidelineshavebeendiscussedandagreedoninpastratermeetings:
�� Salutation AND/OR closing formula missing or wrong – downgrade by oneband.�� Seriousregister/styleproblems–downgradebyoneband.
TEXT LENGTHTextlengthisarelatedissue,whichhasbeensetdowninsufficientdetailasafoot-noteintheTaskAchievementscales.Themainpointisthatover-lengthtextsarenotpenalisedwhereastextsthataresignificantlybelowtherequestednumberofwords(110/80and30respectively)aredowngraded.Thisisbasedontheassumptionthatawriterwhoonlydeliverstwothirdsorlessofthelengthrequiredwillhaveseriousproblemstoproduceasubstantialtext.
Thebaselinetestingof2009hasshownthattextswithintherangeof80–109wordsrarelygetaprovisionalscorethatishigherthanband4,thosewithintherangeof50–79wordsgetnohigher scores thanband3.At these lowperformance levels,however,aparticularproblemhasarisen.If,forinstance,a“longtext”ofonly78wordsisprovisionallyplacedatband2,itwillbedowngradedbytwobandsfortextlength,sothefinalscorewouldbezero.Inacaselikethis(withthewordcountsoclose)ithasbeendecidedtodowngradeonlybyonebandfortextlengthsoastorecogniseoveralltaskachievementbyplacementinthelowestband.
39Testing Writing for the E 8 Standards
ScaleInterpretation–CoherenceandCohesion
Coherenceisaqualitycriterionthatreferstothelogicalarrangementofideasandargumentswithinatext.Inawell-written–thatis“coherent”–textthewritersuc-cessfullyarrangeshis/hersentencestoachieveapurpose,e.g.toreflectthechrono-logicalsequenceofeventsortodevelopaconvincinglineofargument.Acoherenttextmakesiteasyforthereadertofollowthewriter’strainofthoughtsothatthereisnoneedtostopandrereadinordertoestablishmeaningasideasandargumentsflowsmoothlyandlogically.Alesscoherenttext,however,impairsreadabilityandappearsjumpy.
Thetermcohesionrelatestotherelationshipsbetweenelementsofatext,whichisthewaywords,wordgroupsandindividualsentencesarelinked.Thereareseveralwaysinwhichcohesioncanbeestablished.Simplesentencescanbeconnectedbyusinglinkingwordssuchasand, but or because. Forexample,thesequenceMy holi-day was a disaster. It rained almost every day. canbereformulatedasMy holiday was a disaster because it rained almost every day. AnothersolutionwouldbekeepingthetwosentencesbutlinkingthembysayingIt rained almost every day. Therefore my holiday was a disaster. SomesuchcohesivedevicesthatwemayexpectwriterstouseatlevelA2/B1are:
Addition and,or,alsoTime when,after,before,Result so,thereforeContrast ontheonehand–ontheotherhand,althoughReason because,asExemplification forexampleSequence first,then,next,finally
Wewill,however,havetobearinmindthatatextcanbecoherentevenifveryfewofthesecohesivedevicesareusedandthat,ontheotherhand,thefrequentuseofcohesivedevicesdoesnotnecessarilyturnanincoherenttextintoacoherentone.
Othertechniquestomakeatextappearcohesivearereferencesbytheuseofpersonalpronouns,possessives,demonstratives,andcomparatives.Ataverysimplelevel,inthetwosentencesMy best friend is Michael. He is in the same class as I. cohesionisrealisedbyusingthepersonalpronoun ‘he’insteadofrepeatingthenameMichael. Similarly,inapassagesuchas My sister has the big room in the house. Mine is a lot smaller. thepossessiveminereferstotheroomintheprevioussentence,thuslinkingthetwosentences.Demonstrativescanservethesamepurpose.In I got a new camera for my birthday. That was my best present ever. thewordthatreferstothecamerathewritergot,therebylinkingthetwosentencessuccessfully.
Sentencescanalsobeconnectedbysubstitutingoneormorewordsinasentence.InWe have a lot of field trips in our school. The nicest one was to Schönbrunn Zoo. thewriterhasreplacedfield tripsbyoneintheimmediatelyfollowingsentence.In Girls are better at English. Everybody thinks so. thewordsorepresentsthewholeideathatgirlsaredoingbetteratlanguages.Aparticularlysuccessfulwayofestablishingcohe-sioninatextistheuseoflexicalchainsasexemplifiedinthefollowingtextpassage:
When I think of clothing I would say that T-shirts with crazy designs like dots, squares, skulls are definitely in. All my friends are wearing that and they think it’s the latest craze! This year wearing the ‘right’ shoes like ‘Converse’ or ‘Vans’ is very important. Everybody loves to wear them because it’s a must-have!
40 Testing Writing for the E 8 Standards
Inthefirstsentencethewriterusesthephrase definitely intodescribecertainkindsoffashionableT-shirts.Inthefollowingsentencethisideaiscontinuedbyusingthephrasethe latest craze. Attheendoftheparagraphthisconceptofafashionproductis reformulated as it’s a must-have. This establishes a lexical chain that binds thesentencestogetherandestablishesasmoothflowofideasintheparagraph.
IntheCEFR,coherenceandcohesion isanaspectof thepragmaticcompetencesof a language user. The discourse competences relevant for writing that are dealtwith in theCEFRare“flexibility tocircumstances“, “thematicdevelopment“and“coherenceandcohesion“.AsthelatteristhequalitycriterionthatisofparticularrelevancewithintherangeofA2/earlyB1writers,itistheonethatisrepresentedintheE8AssessmentScale.IntheE8ScaleforcoherenceandcohesiontheCEFR(seep.125)movesfromtheverybasicA1skillofbeingabletolinkwordswithlinearconnectorssuchasandorthentoA2,whichmeansalsosuccessfullyusingconnectorsthatexpressreason(because)andcontrast(but)tolinkwordsorwordgroups.A2+includestheabilitytousethesemostfrequentconnectorstodescribesomethingasa“listofpoints“,whereasonelevelfurtherupatB1theloosely“connectedlistofpointshasbecomeafullyconnectedlinearsequenceofpoints”.
OurE8AssessmentScale forwriting includes the aspects of both coherence andcohesion.Regardingcoherenceweexpectatexttobeessentiallyclearinitsmessageandcoherentatbands5to7,butacceptsomeamountofvaguenessandambiguityinband5.Band3 texts are characterisedby frequently incoherent text elements,noticeablyimpairingclarityandreadability,whileband1textsarenotcoherentatallandconsistofmostlydisconnectedchunksoflanguage.Insuchband1textsweonlyfindthemostbasic linearconnectorssuchasandorthenascohesivedevicesonwordgrouplevel,whileband3textsshouldalreadyshowsimplesentencelevelcohesionwithawiderrangeofconnectors.Aband3writerisabletolinksentencessuccessfullyusing simple connectors,butusually fails toproduce longer stretchesofconnectedlanguageatparagraphlevel,makingatextappearasachoppylistofpointsratherthanalongerconnectedsequence.Fromband5upwecandemandthislongerconnectedsequenceofsentences,withthewriterbeingabletolinksen-tencesintoclearparagraphs.Atband5wewanttoseethisabilityreflectedinatleastsomepartsofthetexts,whileatband7thewholeofthetextshouldreflectgoodsentenceaswellasparagraphlevelcohesion.Atthetopband7anexpertwriterwillprobablynotonlymanage to link sentences smoothly and logically toproduce acoherentparagraph,butmightalsoalreadyestablishlinksfromparagraphtopara-graph.Needlesstosay,thedegreeinwhichhandlingsuchissuesofcoherenceandcohesioncanbemasteredbyatesttakeralsodependsonthecomplexityoftheideasputforwardandmayhavetobetakenintoaccountbytherater.Themorecomplexandunexpectedideasthereareinatext,themorewehavetoacceptsomejumpinessinthewaytheyarepresented.
Thecontentpointsinthepromptwillalreadysuggestaparagraphorganisationtothewriter,butitisfinallythedecisionofthetesttakerhowhe/shechoosestoorgan-isehis/hertext.Theabilitytostructureatextofaround150wordsintomeaningfulparagraphs is an important skill that we expect test takers to demonstrate in theE8WritingTest. In long textswe expectparagraphs fromband3up, anda lackofindentationorvisualmarkingofparagraphswillresultindowngradingthetextbyoneband.Asequenceofindividualsentencesmarkedasaparagraphcannotbeacceptedassuccessfulparagraphingifthesentencesarearrangedinahaphazardandrandomway showingno connectionwhatsoever.The same applies toparagraphsconsistingofonesentenceonly.Inshorttextscoherenceandcohesionisgenerallymoredifficulttodemonstrate.Paragraphsarenotmandatoryand,ifusedtogoodeffect,couldbeconsideredareasonforupgradingthetext.
41Testing Writing for the E 8 Standards
ScaleInterpretation–Grammar
TheScaleforGrammarcomprisesdescriptorsforrange,control,andtheclarityofthemessage.Therefore,theratersevaluatethetesttakers’abilitytomakeuseofarangeofgrammaticalstructures,theleveloftheiraccuracyaswellastheirimpactonthemessage.Thefocusisongrammaticalformsthatcreatemeaningandthatarereasonablycorrecttoaccomplishsuccessfulcommunication.
ShorttasksaredesignedtobeA2tasksandtherangeofgrammaticalstructuresthatis likely tobeelicited in such tasks comprises structures typicallymasteredatA2level.6LongtasksshouldhavethepotentialtoproduceB1languageand,asaconse-quence,alsogrammaticalstructuresrepresentativeofB1level.7
THE CONCEPT OF GRAMMATICAL RANGEGrammaticalrangereferstothevarietyofgrammaticalstructuresfoundinaperfor-mance.Rangecansurfaceinthevarietyofgrammaticalforms(verbmodification,tense,aspect,comparativeforms,superlativeforms)andthecomplexityofsentences(mainclauses,subordinateclauses,conditionalorrelativeclauses)usedinatext.
IntheE8context,grammaticalrangemustbeseeninrelationtothetask.Wecannotexpectthetesttakerstousestructuresthatarenotmeaningfullyelicitedbythetask.Sincethewritingpromptsfocusexclusivelyonfamiliartopicsandhavetocaterforallabilitylevels,theyareasstraightforwardintheirset-upaspossible.Thisdoesnotautomaticallysuggestthattheresponsecannotbemorecomplexthanthestimulus.Evenifataskissimpleinnature,weexpectdifferentiationingrammaticalformsorclausetypes,suchasconditionalorrelativeclauses.
Verbs,forexample,canbemodified,markaspect,anddeterminevarioustypesofsen-tencefunctionsuchasstatement,question,negation,command,andexclamation.Moreover, they can be used in their active or passive forms, and test takers maychoosetousedirectorindirectspeech.
Inadditiontothespecificationsoftheprompt,whichwilltrytoelicitcertaingram-maticalstructuresfortaskfulfilment,thetimeallotmentandtheexpectednumberofwordswillalsohaveanimpactonrange.Thatis,shorttasksarelikelytoprovidefeweropportunitiestoshowgrammaticalrangethanlongtasks.
RANGE VERSUS ACCURACYInamistakesandcorrectiondriventraditionofteaching,theuseofgrammaticallychallenginglanguagecanbecomeaproblemforalearneriferrorsoccur.NotsoinE8testing.ItisE8testingpolicythatrangeoverrulesaccuracyinthesensethatrichgrammaticalrangethroughrisktakingisencouraged,whileminorinaccuraciesthatdonotimpairmeaningplayareducedrole.Themorevariedthegrammaticalrange,thehighertheband.Risktaking,whichresultsinrichstructuresbutreducedcon-trol,canevenbeareasonforupgradingatext.
Globalerrors, i.e.errorsthat interferewiththecomprehensibilityofthetext,willcausedowngradingortheplacementofatextatalowband.Localerrorswhichdonothindercommunicationwillnotautomaticallyleadtodowngradingunlesstheirfrequencyimpairsthemessageorthereadabilityofthetext.
6 ForaninventoryofgrammaticalareasatA2levelseeKETHandbook,8–9.Availableat:http://www.cambridgeesol.org/assets/pdf/resources/teacher/ket_handbook.pdf[24June,2011]
7 ForaninventoryofgrammaticalareasatB1levelseePETHandbook,7–8.Availableat:http://www.cambridgeesol.org/assets/pdf/resources/teacher/pet_handbook.pdf[24June,2011]
42 Testing Writing for the E 8 Standards
Testtakersareencouragedtomakeuseoftheirfullpotentialandthemorecreativethestructuralfeaturestheyshow,thebetter.Nevertheless,theuseofvariationshouldnotbeexaggeratedeither.Thetaskssuggestcertainscenarioswhichrequirespecialstructuralsolutions.Theseshouldproduceauthenticandnaturalvariationbutnotartificialtexts.
ASSESSMENT OF PERFORMANCES Theplacementofaperformanceatacertainbandreflectstherangeofgrammaticalstructuresandtheleveloftheircorrectnesswithinameaningfullyandsuccessfullyaccomplishedcommunicativetask.
Band7 texts feature goodgrammatical rangewhich createsmeaning andnaturallanguagewithintheframeworkofthetask.Thewritervariesthegrammaticalstruc-turesthepromptelicitsandmayoccasionallygobeyondtheobviousandexpected.However,anyenhancementshouldnotmakethetextsoundunnaturalorresultinexaggerationofgrammaticalstructures(rangeforthesakeofrange).Inadditiontogood rangea relativelyhighdegreeofgrammatical control is expected inband7texts.Afewinaccuraciescanoccurbuttheywillnotimpaircommunication.
Band 5 texts show sufficient range of grammatical structures. Sufficient range isachieved,ifthewritermakesenoughuseoftheprompt’sstructuralfeaturestomakethe required communication successful and if the grammatical forms used createappropriatemeaning.Occasionalinaccuracieswhichcanimpaircommunicationcanbetolerated.
Band3textsfeaturealimitedrangeofsimplegrammaticalstructures.Thismeansthatthegrammaticalstructuresarejustenoughtoachievesuccessfulcommunication.Mostlytheyareverysimple,repetitive,andhardlyvaried.Grammaticalstructuresinband3textscanbefrequentlyinaccurateandmayshowbasicmistakes.Generally,thesemistakesdonotcausebreakdownofcommunication.
Band1texts featureanextremely limitedrangeof simple structures.Thisusuallyforces the writer to compromise the message regarding meaning, content, andnaturalnessoflanguage.Extremelylimitedrangeresultsinstructuresthatarerepet-itive and invery simple subject-predicate-object sentencepatterns.The structuresusedhardlygobeyondthelearntrepertoireofbeginners.Inadditiontostructuralrestrictions,band1textsshowlimitedcontrolwhichfrequentlycausesbreakdownofcommunication.
43Testing Writing for the E 8 Standards
ScaleInterpretation–Vocabulary
When we assess vocabulary we are looking at content words (nouns, full verbs,adjectives,adverbs),collocationsandchunksoflanguagethatawriterusestoper-formawrittencommunicativetask.Whatweneedtoassessislexiscreatingmeaningthat isreasonablycorrect toaccomplishsuccessfulcommunication.Similar tothegrammar scale, the scale for vocabulary also comprises descriptors for range andcontrol.
THE CONCEPT OF LEXICAL RANGERangereferstothebreadthofvocabularyacandidateusesinawrittentext.IntheE8context,rangemustbeinterpretedinrelationtothepromptasraterscanassessonly the vocabulary actually elicited by the prompt. The time allotment and theexpectedtextlengthhaveanimpactonrange.Shorttasksarelikelytoprovidefeweropportunities todemonstrate vocabulary range than long tasks.Asmentioned inthepreviouschapter,shorttaskshavebeendesignedtobeA2tasksandlongtaskshavebeendesignedtobeB1tasks.ForthesereasonstherangeoflexicalitemsthatwecanexpectinshorttasksarewordsandphrasestypicallymasteredatA2level8,forlongtaskswecanexpectafairamountofwordsandphrasestypicallymasteredatB1level9.
Even if a task is simple innaturewemay expectdifferentiationwithin choiceofwords.Forexample,ifataskasksforanarrativedescriptionaboutthefirstfewdaysatanewschool,thetextswillprimarilycontainwordsrelatedtoschool,teachers,subjects,newfriendsetc.,which,however,canbevariedandmodified.Althoughthepromptlanguageisassimpleaspossible,writersmaywellproducearesponsethatexceedsthepromptstimulus.
RANGE VERSUS ACCURACYItisnotenoughforacandidatetousealargenumberofdifferentwordsinatexttoachieveahighbandinassessment.Thewordsacandidatechoosesmustberelevantandappropriatetothetopicandusedinawaythatthecandidateisabletoconveyhis/herideasmeaningfully.Atopwriteramongourtesttakerswillusevocabularythatisgenerallyaccurateenoughtoformulateevenamorecomplexideawithclarity.Test takerswhostay inabsolutely safe languageareas (e.g. languagepickedup inyearsoneandtwo)andavoidtakinganyriskhavelessevidenceofmistakes.How-ever,itisE8policytoencourageourcandidatestoventureoutoftheirsafelanguagezonebyrewardingrisktakingtocommunicatesuccessfully.
Textsthatshowagoodrangeofvocabularyatband7containagoodselectionofcontentwords andphrases thatdemonstrate that the candidate is able to expresshim/herselfclearlyandpreciselyandoccasionallycanevenvaryformulationssoasnottoappearrepetitive.Wemaywellexpectoneortheotherexpressiontostickoutandexceedwhatwetypicallyexpectfromtesttakersatthislevel.
Band5textscontainasufficientrangeofmostlyhigh-frequencywordsthatagainmeettheneedtocommunicateclearideasandaregenerallyusedaccurately.Theremaybesomeoccasionalmistakes,particularlywhenthecandidateistryingtocom-municateamorecomplexidea.
Inaband3textweexpectthelexicalrangetobelimited,containingonlyarathernarrowrepertoireofhigh-frequencywords,butstillthesimpleideasthatarecom-8 Availableat:https://www.teachers.cambridgeesol.org/ts/digitalAssets/113295_ket_vocablist09.pdf[24June,2011].9 Availableat:https://www.teachers.cambridgeesol.org/ts/digitalAssets/113298_pet_vocablist09.pdf[24June,2011].
44 Testing Writing for the E 8 Standards
municatedaremostlyunderstandableevenifthereisacertainamountofinaccuratevocabulary.Withband3candidateswearelikelytodetectexamplesofliftingphrasesfromtheprompttocompensatefortheirlexicallimitations.
Finally,inaband1textawriterwithextremelylimitedlexicalcompetenceinEng-lishwilldemonstratethisbyincludingonlyafewveryhigh-frequencycontentwordswhicharemoreoftenthannotinaccurateandinappropriate.Wecommonlyexpectband1writerstocompensatefortheirlackinlexicalrangebyheavilyliftingdirectlyfromthepromptandby interspersing their textwithL1words inorder to ‘keepgoing’,thushavingthe‘knockoneffect’offrequentlycausingbreakdownincom-munication.
Thenatureofsomepromptsmakesitalmostimpossibletoavoidliftingandratersmusttakecarenottofallintothetrapofautomaticallyplacingatextatband4orbelowbecausethereisevidenceofpromptlifting.Agoodwriterdoesnotjust‘copyandpaste’wordsorphrases,butcanadaptthemandincorporatethemsuccessfullyintothetexttoaccomplishthecommunicativetask.Thisisaskillthatneedstobeacknowledgedpositively.
Anaspectof lexicalaccuracythatratersneedtoaddress is spelling.It iscommonpractice amongst teachers to take marks off for incorrect spelling. However, theemphasisoncommunicatingmeaningsuccessfully iscentral to theE8context.Atext containing many spelling mistakes, in particular those mistakes that changethewholemeaningofaword,isverylikelytodisturbthereaderandcauseabreak-downofcommunication.Ratersneedtoassesstheextentofbreakdownandrateasisnecessary.However,asweencourageourwriterstotakerisks,slight‘slipsofthehand’andminorerrorsinspellingthatdonotchangethemeaningoftheword(e.g. pleas, tomorow)shouldnotbepenalised.Intheenditisthelexicalrangethatismoreimportantthanaccuracyandatextmightmeritoneofthehigherbandsdespitetheinaccuracies.
45Testing Writing for the E 8 Standards
Literature
Alderson,J.C.2004.Washbackinlanguagetesting.Researchcontextsandmethods.In:Cheng,L.,Watanabe,Y.&Curtis,A.(Eds.).Context and Method in Washback Research: The influence of language testing on teaching and learning. Mahwah, NJ:LawrenceErlbaum.
Alderson,J.C.,Clapham,C.&Wall,D.1995.Language Test Construction and Eva-luation. Cambridge:UniversityPress.
Bachman,L.F.1990.Fundamental Considerations in Language Testing.Oxford:Uni-versityPress.
BIST-Verordnung.2009.SeeVerordnung der Bundesministerin.
Breit,S.&Schreiner,C.(Eds.)2010.Bildungsstandards: Baseline 2009 (8. Schulstufe). Technischer Bericht. Salzburg:BIFIE.Availableasdownloadfromhttp://www.bifie.at/buch/1056[14.April,2011]
Brock,R.&Haslinger,U.(Eds.)2011.Bildungsstandards für Fremdsprachen (En glisch) 8. Schulstufe. Praxishandbuch(Neubearbeitung2011).ÖSZPraxisreihe4.BifieWien.
Canale,M.&Swain,M.1980.TheoreticalBasesofCommunicativeApproachestoSecondLanguageTeachingandTesting.In:Applied Linguistics 1(1),1–47.
CouncilofEurope(Ed.).2001.Common European Framework of Reference for Lan-guages: Learning, Teaching, Assessment.Cambridge:UniversityPress.
Ek,J.A.van[CouncilofEurope].1979.The Threshold Level for modern language learning in schools. Harlow:Longman.
Ek,J.A.van&Alexander,L.G.1980.Waystage English. Anintermediaryobjec-tivebelowThresholdLevel inaEuropeanunit/credit systemofmodern languagelearningbyadults.Preparedfor theCouncilofEurope inassociationwithM.A.Fitzpatrick.Oxford:PergamonPress.
Field,J.2004.Psycholinguistics. The Key Concepts.London:Routledge.
Field, J. 2005. Second language writing: a language problem or a writing problem? Paper presented at IATEFL Research SIG ‘Writing Revisited’ conference, Cam-bridge,25–27February2005.
Gassner,O.,Mewald,C.&Sigott,G.2007.Testing Reading. Specifications for the E8 Standards Reading Tests. LTC Technical Report 2. Wien:bm:ukk.Availableasdown-loadfrom:http://www.bifie.at/publist-07-05-14[24June,2011]
Grabe,W.&Kaplan,R.B.1996.Theory and Practice of Writing. An Applied Linguistic Perspective.London:Longman.
Hyland,K.2002.Teaching and Researching Writing. London:Longman.
Kellogg,R.T.1994.The Psychology of Writing. NewYork:OxfordUniversityPress.
46 Testing Writing for the E 8 Standards
Kellogg,R.T.1996.Amodelofworkingmemoryinwriting.In:Levy,C.M.&Rands-dell,S.(Eds.)The Science of Writing. Mahwah,NJ:LawrenceErlbaum,57–72.
KET Handbook. CambridgeUCLES.2009.Availableasdownloadfromhttp://www.cambridgeesol.org/assets/pdf/resources/teacher/ket_handbook.pdf[14April,2011]
KET Vocabulary List. Cambridge UCLES. 2006. Available as download fromhttps://www.teachers.cambridgeesol.org/ts/digitalAssets/113295_ket_vocablist09.pdf[14April,2011]
Lehrplan der Hauptschule: LebendeFremdsprache(alleSprachen)2008.Availableasdownloadfromhttp://www.bmukk.gv.at/medienpool/16682/bgbl_nr_ii_210_2008.pdf[14April,2011]
Lehrplan der AHS:LebendeFremdsprache(alleSprachen)2006.Availableasdown-loadfromhttp://www.bmukk.gv.at/medienpool/782/ahs8.pdf[14April,2011]
Lumley,T.2005.Assessing Second Language Writing. The Rater’s Perspective. Frank-furt:PeterLang.
Papajohn,D.1999.Theeffectoftopicvariationinperformancetesting:thecaseofthechemistryTEACHtestforinternationalteachingassistants.In:Language Testing 16(1),52–81.
PET Handbook.CambridgeUCLES,2009.Availableasdownloadfromhttp://www.cambridgeesol.org/assets/pdf/resources/teacher/pet_handbook.pdf[14April,2011]
PET Vocabulary List. CambridgeUCLES.2009.Availableasdownloadfromhttps://www.teachers.cambridgeesol.org/ts/digitalAssets/113298_pet_vocablist09.pdf[14April,2011]
Read,J.1990.ProvidingRelevantContentinanEAPWritingTest.In:English for Specific Purposes 9(2),109–21.
Scardamalia, M. & Bereiter, C. 1987. Knowledge telling and knowledge trans-forminginwrittencomposition.In:Rosenberg,S.(Ed.)Advances in Applied Psycho-linguistics, Vol. 2: Reading, writing, and language learning. Cambridge UniversityPress,142–175.
Shaw,S.D.&Weir,C.J.2007.Examining Writing. Research and practice in assessing second language writing.Cambridge:UniversityPress.
Sigott,G.,Gassner,O.,Mewald,C.&Siller,K.2007.E8 Standardstests. Entwick-lung der Tests für die rezeptiven Fertigkeiten: Überblick. LTC Technical Report 1. Lan-guageTesting Centre, Alpen-Adria-Universität Klagenfurt. Available as downloadfromhttp://www.bifie.at/publist-07-02-19[24June,2011]
Tankó,G.2005.Into Europe. The Writing Handbook.Budapest:TelekiLászlóFoun-dation.
Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungsstan-dards im Schulwesen.BGBl.IINr.1/2009v.2.1.2009.Availableasdownloadfromhttp://www.bmukk.gv.at/schulen/recht/erk/vo_bildungsstandards.xml [14. April,2011]
47Testing Writing for the E 8 Standards
Weigle,S.1994.EffectsofTrainingonRatersofESLCompositions.In:Language Testing 11(1),41–69.
Weigle,S.C.2002. Assessing Writing.Cambridge:UniversityPress.
Weir, C. J. 2005. Language Testing and Validation. An Evidence-Based Approach. Basingstoke:PalgraveMacmillan.
48 Testing Writing for the E 8 Standards
Appendix
PromptInterpretation:LongPrompt
TaskAchievement
Content
�� Althoughthefirstcontentpointcanbedealtwithverybriefly(newplace),thesecondpoint(reasonsformoving)allowsforsomeelaboration.�� Thenextthreepoints(describetown/village,newhome,andfirstdays)definitelyallowforelaborationandweexpectsomewhatdetaileddescriptions(points3,4,5).�� Thelastpointaboutfeelingsismoredifficultbecausethisrequiressomecreativethinking,whichiswhywedonotexpectmuchelaboration.Ifthereiselaborationhere,itneedstobespeciallyrecognised.
DegreesofTaskAchievement
Inthislongtext,allcontentpointsneedtobementionedforcomplete task achieve-mentandthreeelaborated.Elaborationinthelastpointaboutfeelingscannotbeexpectedasthisisconsideredtobedifficulttoachieveatthelevelofthetesttakers.
Good task achievementcanbeawardedevenifonecontentpointisnotdealtwith.‘85%’meansthatfivecontentpointsmustbementioned.Dependingonthequalityofelaboration,wewouldexpectatleasttwopointstobeelaboratedforgoodtaskachievement.Asecondoptionforband5istohaveallcontentpointsmentionedandoneortwoelaborated.
Forsufficient task achievementwewouldexpectfourcontentpointstobemen-tionedandoneortwotobeelaborated,dependingonthequalityofelaboration.
Asecondoptionforband3istohaveallcontentpointsmentioned,butnoelabo-ration.
Some task achievement wouldbegivenifthemessagewasclearenoughtoconveytheinformationthatthepersonhasmovedtoanotherplaceandsomebriefinforma-tionabouttwoothercontentpointstoreachthe50%.
You have just moved to another town/village. Write a letter to your American/English friend in which you tell him/her about your new situation.
Inform him/her about ● your new place of living. (1) ● the reason for moving. (2)Describe ● the town/village you’re living in now (buildings, people,…). (3) ● your new home. (4)Tell him/her about ● the first days of your ‘new life’ (new school, teachers,…). (5) ● how you feel about your new situation. (6)
49Testing Writing for the E 8 Standards
TextTypeRequirements
�� Thetext type isclearlymarkedasapersonal letter,which iswhyweexpectaninformalopeningandclosingformulalikeDear Jim/Hi Sue andYours/Love etc.�� Theregisterneedstobeinformal.�� Texttyperequirementsarenot met ifthereisnoopeningand/orclosingformulaANDiftheregisterisnotappropriate(e.g.rudelanguageormismatchofinfor-malsituationandformallanguage).Thiswillleadtodowngrading.�� Missingorincorrectopeningand/orclosingformulawillleadtodowngradingbyoneband.�� Inappropriateregister/tonealonewillleadtodowngradingbyoneortwobandsdependingonthequalityofthetext.Thesameappliestoanimpoliteoroffensivetone.
CoherenceandCohesion
Inalongtextwedefinitelyexpectparagraphing.Sincethecontentpointsaregroupedintothreeunits,thetesttakersareprovidedwithsufficienthintsastohowthetextcouldbestructuredintoparagraphs.Alackofparagraphorganisationinbands3to7leadstodowngradingbyoneband.
�� Testtakersneedtouseparagraphsandsomecohesivedevicestoreachband7.Thetextmustbeclearandcoherentatsentence,paragraphandtextlevel.�� Forband5,cohesionshouldbeachievedat leastat sentence level,and longerstretchesshouldshowsomeparagraphlevelcoherenceandcohesion.Forexample,thiscouldbedemonstratedinthedescriptionofthenewplaceand/orhome.�� Forband3,wewouldatleastexpectthetesttakerstousesimpleconnectors(e.g.because, but, and, then)whengivingreasonsformovingorforthedescriptionofplacesand/orthefirstdaysatschool.Sowecanexpectsentencelevelcohesion,butproblemsatparagraphlevel.Thetextmightnotbefullycoherent.�� Forband1,itisenoughifthesentencesarelinkedwithverybasicconnectors.Wedonotexpectparagraphingortextualcoherenceforband1.�� Iftheletteriswritteninchunksratherthansentences,wesuggestthatthereisnotenoughassessablelanguageforcoherenceandcohesion.
Grammar
Inthis longtext,writersareexpectedto informaboutthemovetoanotherplacemakinguseofpastverbforms.Thereasonformovingcaneitherbelinkedwiththisinformationandalsorealisedinthepast,orconnectedtotheinformationaboutthenewplaceandthereforefeaturepresenttenseverbforms.Thedescriptionofthenewplace,thenewhomeandthenewlifewillmostprobablyelicitpresenttenseverbforms.Verygoodsolutionsmightincludecomparisonsbetweenthenewandtheoldplace,thenewandtheoldsituation.However,thesecannotbeconsideredcompul-sorybecausetheyarenotexplicitlyrequiredbythetask.
Vocabulary
Theinstructionsandcontentpointsshouldenticethewritertousealotoffamiliarcontentwordstodescribethenewsituation.Wewouldexpectthewritertomentionthenameofacountry,city, townorvillageandusecontentwordstodescribe it,thebuildings,people,neighbours, andpossibly the countryside there.Thewriterhastheopportunitytodescribehis/herhomeandwordsrelatingtotypesofhouses,householdrooms,furniture,andperhapsagarden,willmostlikelybeinthetext.Wewouldcertainlyexpectthewritertousearangeofadjectivessuchasqualifiers
50 Testing Writing for the E 8 Standards
todescribeappearance,condition,and/orshape;alsoquantifierstodescribesize,inordertocomparethepresentsituationtotheformersituation.Thewritershouldgiveareasonformovingandhe/shewillprobablyusevocabularyrelatingtoapasteventinthefamily:aparentgetting/losingajob,startinganewlife,movingbacktoaformerplaceofresidence,orachangeinthefamilystructureduetoadivorce.
Thewritershouldalsowriteabouthis/hernewlifeandwecanexpectsomevocab-ularyaboutschool, teachers,classmates,subjects,classroom,and/orhomeworktoappear inthetext.Itmaywellbethatthewriterchoosesto ignorewritingaboutschoolandinterprets“thefirstdaysofyour'newlife'“byreferringtooutofschoolactivitiessuchasbuying/movingfurniture,meetingandplayingwithnewfriends,exploringthenewareaandotherfree-timeactivities.Onceagain,wewouldexpectthesentencestocontainamixtureofqualifiersandquantifiers.
Finally,wewouldexpectsomeofthesentencesaboutthenewsituationtoincludeadjectives to describe the positive, ambivalent and/or negative feelings he/she isexperiencinginthenewsituation.
PromptInterpretation:ShortPrompt
TaskAchievement
Content
�� Point1:Thewritersneedtogiveoneorseveralreasonsforlikingtheparty.Itisnotenoughtosaythattheylikedthepartybutthisshouldincludegivingreasons.�� Point 2: This can be done in a more or less elaborate way depending on thewriter’schoicesandabilities.�� Point3:Thisissimpleenoughandwillbedealtwithverybriefly.Mostprobablyitwillimmediatelyleadontopoint4.�� Point4isexpectedtobethemostchallengingoneasitrequiressomethinkingandaminimalamountofcreativity.Writersneedtoincludeatleastonesugges-tionreferringtothefollowingweekend.
DegreesofTaskAchievement
Withshorttasks,therelevantdescriptorneedstobeinterpretedverystrictly.Infact,withfourcontentpointsrequireditmakeslittlesensetotolerateamissingone.ItshouldbetakenforgrantedthatALLfourcontentpointsmustbedealtwithforcomplete or good task achievement. This would normally mean a text length ofaround60words.
�� Itwillbethedegreeofelaborationthatdistinguishesbetweencompleteandgoodtaskachievement.
Your friend’s birthday party was a few days ago. Write an email to tell him/her that you liked the party.
• Tell him/her why you liked the party. (1)• Tell him/her what you liked best. (2)• Ask your friend when you are going to meet again. (3)• Suggest something for the next weekend. (4)
51Testing Writing for the E 8 Standards
�� Withonecontentpointmissing,goodtaskachievementisnolongerpossible,butband4is.�� Ifonecontentpointismissing,wecanawardband3,i.e.sufficienttaskachieve-ment(justover70%).Ifthereissomeelaboration,itisband4.�� Iftwocontentpointsaremissing,wecannotexpectmuchelaborationeitheranditseemstopointtowardsband1.�� If,however,thereisgoodelaborationofonecontentpoint,itisband3.Ifelabo-rationisweak,itisband2.
TextTypeRequirements
�� WecanexpectaninformalopeningandclosingformulalikeDear Jim/Hi Sue and Yours/Loveetc.�� Forband7 theremustbe anappropriateopeningandclosing formula and theregistermustbeinformalanddemonstratethefamiliarityindicatedbyyour friend.�� Problemswiththebeginningorendingofthetextorminorproblemswiththeregisterwouldleadtodowngradingbyoneband.�� Whentheproblemsbecomemoreobvious,taskachievementwilldropbytwobands.�� Text type requirements are not met when there is no opening and/or closingformulaANDwhentheregisterisnotappropriate.(Downgradebytwobandsiftheproblemswithregisterareveryserious).
CoherenceandCohesion
�� Wedonotexpectanyparagraphingwithshorttasksalthoughgoodwriterswilluseparagraphingtostructuretheirshorttextsaswell.Sosomecreditshouldbegivenforsuccessfulparagraphing.�� Demonstratingthisdimensioninshorttextsisratherdifficult,butwecanexpectthewriterstodealwiththefourcontentpointsintheordergiven.Implicitandexplicitlinking(connectors)isnoteasytoplaceinthiskindoftextandshouldbespeciallyacknowledgedwhenitisusedsuccessfully.Contentpoints1and2sug-gesttheuseofbecauseand/orand.
Grammar
Thisshort taskasks for feedbackaboutapastevent.Thedescriptionofwhatwasliked andwhatwas likedbest opensuppossibilities for comparisonsor even thesuperlative. However, any other solution that implies the information about thelikedandbestlikedactivityatthepartyneedstobeacknowledged.Moreover,thetaskrequiresaquestionforameetinginthefutureandasuggestionforanactivityforthenextweekend.Whilethequestionisverylikelytoelicitafutureverbform(mostprobablythegoing tofutureasgivenintheprompt),thesuggestioncanalsoberealisedinthepresentormakinguseofmodalverbforms.Thelattercouldalsocomeacrossasanotherquestionfollowedbyasuggestionorastatement (What about next weekend? My brother has a birthday party. Would you like to come?)
Vocabulary
We expect the writer to use adjectives such as great, cool, funny and/or to refertonounssuchas the games, music, food, lights, friends that were at the party intheresponsetothefirstcontentpoint.Thewriterwillprobablyrephrasethesecondcon-tentpointandaddsimilarvocabularytostateclearlywhyhe/shelikedthepartybest.Theresponsetothethirdcontentpointwillmostlikelycontainoneofthefollowing:When can I see you again? When are we going to meet again? What are you doing at the
52 Testing Writing for the E 8 Standards
weekend? Thefinalpartofthetextwillmostlikelybeacombinationofasuggestionandaquestioncontainingwordsrelatedtocommonteenageleisureactivitiessuchasmeeting friends, going shopping/to a party, and/orplaying/watching football etc. Asthelastcontentpointreferstotheweekend,wecanexpectthewritertoincludewordssuchasweekend, Friday, Saturday, Sunday, morning, afternoon, evening and/ornight.
53Testing Writing for the E 8 Standards
InventoryofFunctions,NotionsandCommunicativeTasks1
ConveyingEmotions,Feelings
�� criticisingandcomplaining�� expressingneedsandwants�� expressingpreferences,likesanddislikes�� makingandrespondingtoapologiesandexcuses�� payingcompliments�� sympathising�� talkingaboutfeelings
Informing,Asking
�� askingandansweringquestionsaboutpersonalpossessions�� askingandtellingpeoplethetime,dayand/ordate�� askingforandgivinginformationaboutroutinesandhabits�� askingforandgivingpersonaldetails:(full)name,age,address,namesofrelativesandfriends,occupation,etc.�� askingforandgivingsimpleinformationaboutplaces�� completingformsgivingpersonaldetails�� expressing(in)abilityinthepresentandinthepast�� makingpredictions�� talkingabout(im)probabilityand(im)possibility�� talkingaboutfood�� talkingaboutfutureorimaginarysituations�� talkingaboutfutureplansorintentions�� talkingaboutone’shealth�� talkingabouttheweather�� talkingaboutpasteventsandstatesinthepast,recentactivitiesandcompletedactions�� talkingaboutwhatpeoplearedoingatthemoment�� writingdiariesgivinginformationabouteverydayactivities
Convincing,Persuading,ExpressingOpinions
�� askingandgiving/refusingpermissiontodosomething�� drawingsimpleconclusionsandmakingrecommendations�� expressingagreementanddisagreement,andcontradictingpeople�� expressingdegreesofcertaintyanddoubt�� expressingobligationandlackofobligation�� expressingopinionsandmakingchoices�� expressingpurpose,causeandresult,andgivingreasons�� givingadvice�� givingwarningsandprohibitions�� persuadingandasking/tellingpeopletodosomething
1 Adaptedfrom PET Handbook 2009andKET Handbook2009.Thelistisnotexhaustive,butservestoillustrateaspectsoftheConstructSpace.
54 Testing Writing for the E 8 Standards
Entertaining,Pleasing
�� talkingaboutpasteventsandstatesinthepast,recentactivitiesandcompletedactions�� talkingaboutwhatpeoplearedoingatthemoment�� understandingandproducingsimplenarratives
KeepinginTouch
�� expressingandrespondingtothanks�� givingandrespondingtoinvitations�� makingandgranting/refusingsimplerequests�� makingandrespondingtooffersandsuggestions�� writingletters/emailsgivinginformationabouteverydayactivities�� writingletters/emailsgivingpersonaldetails
Describing
�� buyingandsellingthings(costs,measurementsandamounts)�� describingeducationandskills�� describingpeople(personalappearance,qualities)�� describingsimpleprocesses�� identifyinganddescribingaccommodation(houses,flats,rooms,furniture,etc.)�� makingcomparisonsandexpressingdegreesofdifference
GivingDirectionsandInstructions
�� askingforandgivingtravelinformation�� askingthewayandgivingdirections�� followingandgivingsimpleinstructions�� identifyinganddescribingsimpleobjects(shape,size,weight,colour,purposeoruse,etc.)�� talkingabouthowtooperatethings
top related