linking exams to the cef levels: t he manual and the catalan experience
DESCRIPTION
Linking exams to the CEF levels: T he Manual and the Catalan experience. Neus Figueras [email protected]. Presentation Outline. Background information about the Catalan EOI Usefulness and relevance of the CEF and the Manual The project to link our exams to the CEF - PowerPoint PPT PresentationTRANSCRIPT
Linking exams to the CEF levels:The Manual and the Catalan experience
Neus [email protected]
Presentation Outline
● Background information about the Catalan EOI ● Usefulness and relevance of the CEF and the
Manual● The project to link our exams to the CEF● Using The Manual: problems encountered and
solutions adopted● A Manual for the future
Background
The EOI system in Catalonia● State funded language schools (+16)● Two levels defined Certificat de Cicle Elemental (aimed at B1) Certificat d’Aptitud (aimed at B2)
● 13 different languages (7 + 6)
● 15,000 thousand certificate exams every year
● Standardised language proficiency certificates in place since 1995, with exams
● developed centrally● related to published curricula● written according to set specifications● submitted to pre-testing● standardised administration and marking● analysed statistically
Issues to be solved in 2002
● Revise EOI certificate exams after 10 years.● Develop an item banked system.● Link levels to Common European Framework levels.
● Is examination difficulty equivalent across time?● Is the lower certificate consistently easier than the higher
certificate?● Are the different certificates in the different languages
comparable?
Exam revision Linking EOI & CEF levels
Teaching/learning context
To sum up
Why linking?
Why the CEF?
A link or a knot?
Linking or equating?
Adoption or adaptation?
A link or a knot?
Relevance of the Manual (2003)
● Clarified issues: tests and the CEF levels● Provided set of systematic procedures ● Stated what was right
● Followed CEF approach and philosophy● Self-justified claims of linkage to the CEF
The Manual offers guidance to users to...
● describe the examination coverage, administration and analysis procedures,
● relate results reported from the examination to the “Common Reference Levels” presented in Chapter 3 of the CEF,
● provide supporting evidence that reports the procedures followed to do so.
The Manual: steps
Four inter-related sets of procedures whichcontribute to the validation process:
● Familiarisation● Specification● Standardisation● Empirical Validation
VISUAL REPRESENTATION OF PROCEDURES TO RELATE EXAMINATIONS TO THE CEF
"BUILDING AN ARGUMENT"
THEORETICAL RATIONALE
QUALITATIVE VALIDATION PROCEDURE
QUALITATIVE VALIDATION PROCEDURE
QUANTATIVE VALIDATION PROCEDURE
SPECIFICATION OF EXAMINATION CONTENT
STANDARDISATION THROUGH TRAINING
EMPIRICAL VALIDATION THROUGH DATA ANALYSIS
Internal familiarisation with CEF
Familiarisation training
Data collection
Training in assessing performance to CEF using standardised samples
Training in judging the difficulty of test items to CEF standardised items
Internal validity: description and analysis of ° General examination content; ° Process of test development; ° Marking, grading, results; ° Test analysis and post-examination review.
Internal validation: Confirming the psychometric quality Of the test
Benchmarking local performance samples to CEF
Item difficulty Estimation to CEF
External validity: relate ° General examination description to CEFscale; ° Description of communicative activities tested to CEF scales; ° Description of aspects of communicative language competence tested to CEF scales.
Dissemination and implementation
External validation: Confirming the relationship to the CEF through an independent measure
CLAIM of link to CEF CLAIM (reinforced CLAIM (confirmation on basis of
on basis of specification) specification and standardisation)
Recommended procedures encourage alignment of examinations to CEF at differing degrees of rigour appropriate to different testing contexts and to the extent permitted by resources and expertise available.
And……
Prerequisites for linkage
1. If an examination result is not reliable, then the examination cannot be compared to the CEF.
2. If each time a new form of an examination is produced it varies according to content and difficulty, it is very difficult to compare the examination to the CEF since the examination does not present a stable standard.
3. If examination results vary by teacher and professor, by school or university, they cannot be matched to the CEF.
(Alderson 2002)
Exam revision Linking EOI & CEF levels
Teaching/learning context
Our case….
Our case: project Outline 2003-06
● July 2003 – 2004● Define objectives● Project design● Empirical scale development● Scale validation● Item banking (English) starts
● July 2004 –2005● Publication of scales ● Item banking (English) continues ● Linkage to CEF: Procedures in the Manual (specification)
● July 2005 – 2006● Linkage to the CEF:Procedures in the Manual (standardisation) ● Linkage to the CEF: “” “” (empirical validation)● Comparability across languages
Scale Development....à la mode du CEF
Step 1: selecting level descriptors (per skill)● CEF● Curriculum objectives for EOI● Finnish scales (YKI)
Step 2: translation of the descriptors into CatalanStep 3: mapping descriptors onto levels (CEF and EOI)
● Experts assign levels to descriptors ● Analysis of sorting exercises● Collating resulting “surviving” descriptors
Step 4: developing new scales● Drafting ● Validation● Scale revision● Validation● Final version
Linking CEF and EOI scales (www.xtec.net/eoi)
CEF - C1 Has a good command of a broad lexical repertoire allowing gaps to be readily overcome with circumlocutions; little obvious searching for expressions or avoidance strategies. Good command of idiomatic expressions and colloquialisms.
EOI – C1 Can understand and use a broad lexical repertoire, including synonyms and antonyms of less common words, idiomatic expressions, to express himself/herself correctly and appropriately on a wide range of general, academic, professional and leisure topics. Can understand word play based on multiple meanings and use prefixes and suffixes to form less common words. Less frequent words may cause some effort in comprehension.
Comparing EOI and CEF levels (Voc)
Linking CEF and EOI scales
Lessons learnt…implications
● The CEF is not always right● The CEF as a catalyst● Reading is not understanding● Continued training is necessary
● Appearances need to be checked empirically● Exact correspondence may not be possible, but is it
desirable?● Linkage does not mean equivalence
Item bank building (first batch, June 2004)
Step 1 ● Booklet development (Listening, Reading, Grammar & Vocabulary).
(Total items included: 396)
Step 2● Data collection (260-784 students per item)● Analyses: CTT and IRT (Total surviving items :301)
Step 3● Standard setting procedures: - test centered - examinee centered - annual average pass rate● Setting (provisional) cut off scores at Elemental (B1) and Aptitud (B2)
And now…back to the Manual
Problems encountered: Specification
● The gaps and inconsistencies in the CEF are evidencedList of communication themes “clearly not definitive” ( p.52)
Definition of communicative task/communicative activity (4.3, 4.4.2.2)
Definition of tasks (7.1, 7.2, 7.3)
● Grids useful as awareness raising but also frustrating● On what basis are the grids completed?(Testversion/Specs?)
● Who does what?
● Who checks it?
Solutions adopted: Specification
● Three languages● Team work in 5 steps (for each grid): Judges go and have a first attempt to fill up Joint work in understanding/agreeing on interpretation for each cell in each gridJudges go away and do the jobJudges meet again and discuss possible problemsExternal judge checks three languages for homogeneity.
How ?
● Extensive knowledge of CEF and Manual● Insiders’ information from the Dutch –CEF Construct project● No commercial (or other) interest in confirming our linkage ● No hurry to be on the “CEF claimers” map
Problems encountered: Standardisation
● Reference materials not ready until late 2005.● Different skills, different languages, different
nature.● “reference” with a small “r” (quantity and quality) ● Assessing task vs. item levels.● No “linguistic competence” reference materials.
Solutions adopted: Standardisation
● Taking advantage of judges/system used to standardisation procedures in speaking and writing.
● Using part of familiarisation process together with exam-centered standard setting procedures.
Empirical validation: Problems encountered, solutions still under way.
● Internal validation : back to
● External validation Anchoring and linkage: with small “r” materials Overcoming difficulties with design in languages with
small number of candidates.
A straightforward alternative…
The proposal from Dutch CEF project● Train in CEF + labelling instruments.● Describe the test (texts and items) using the dimensions
in the Final Grid.● Make a guess at the level of an item (guided by the
classification system), leading to the estimated CEF level.
● Pretest the items thus labelled.● Calibrate the items.● Do standard-setting to set the boundaries of the levels on
the scale coming from the calibration.● Assign a psychometric level to the items. ● Assign a definitive level to the items.
but beware….
An item can only be assigned to a definitive level if the psychometric level falls within the band ofthe estimated level (in other words if the estimation based on the content is comparable with the psychometric value)
?
And mind the gaps!
● Coverage in terms of CEF “relevant subscales”● Comparison across countries, institutions, education
levels…and viceversa● Comparison across skills and languages
The Manual for the future
● What’s it going to be like? Structure and Range● What currency?● Who’s going to use it? ● How (far)?● Will there be primus inter pares?
O mon âme, n’aspire pas à la vie
immortelle, mais épuise le champ du possible.
Pindare, 3e Pythique
Available tools so far…….- Dutch CEF construct Project www.ling.lancs.ac.uk/cefgrid
- Council of Europe Pilot samples www.coe.int
English speaking performances French speaking performances English, German, Spanish, Italian listening and reading tasks (Available from [email protected] )
- ALTE Writing Grid- ALTE Speaking Grid
- CEFTRAIN www.ceftrain.net