language proficiency evaluation: raters henry emery pricesg linguistic sub-group
TRANSCRIPT
Language proficiency evaluation: Raters
Henry Emery
PRICESG Linguistic Sub-Group
Session structure
• Define terms• Operational and linguistic raters • The need for adequate training• Rater competencies• A mechanism for international standardisation
of language proficiency assessment:
The Association of Language Raters
An examiner:
• delivers a direct language test• is trained in a specific test format • elicits speech • interacts with the candidate/candidates (an
interlocutor)• may examine and rate simultaneously
A rater:
• assesses language proficiency• applies the Rating Scale to speech• requires familiarity with the test format• may examine and rate simultaneously • may rate ‘live’ speech• may rate a pre-recorded sample
• Who is an operational expert?• What is the role of an operational expert?“to assess operational integrity and technical accuracy” (ICAO doc. 9835)
• When should an operational be employed? “test prompts should not be intended to evaluate specific technical
knowledge concerning operations” (http:www.icao.int/icao/en/trivia/peltrgFAQ.htm#27)
• In a test of plain English is there a role for an operational rater?
• Operational experts: tests which aim to assess both phraseology and plain English proficiency
Operational raters
• subjective impression• ‘pass’ or ‘no pass’ • operational integrity• technical accuracy• overall effectiveness• Holistic Descriptors ?
Who has the final say, the linguistic or the operational expert?
Linguistic raters
• more complex judgement
• judge components of performance separately
• less subjectivity
• justify assessment
• identify specific aspects of performance
• analytic Rating Scale
The complexities of the ICAO Rating Scale
Six key features
Feature 1
• The six operational levels of the rating scale are not of equal intervals
• Levels three and four are particularly broad
Level descriptors for Structure
Errors in basic grammatical structures rarely interfere with meaning
Errors in in basic grammatical structures frequently interfere with meaning
Limited control of a few simple memorized grammatical structures and sentence patterns
Performs below the elementary level
4
3
1
2
Feature 2
• A candidate is unlikely to fully ‘fit’ a descriptor at any level
• A candidate’s speech may demonstrate features of different levels at the same time
Interacts with ease in nearly all situations. Is sensitive to verbal and non-verbal cues, and responds to them appropriately
Responses are immediate, appropriate, and informative. Manages the speaker/listener relationship effectively
Responses are usually immediate, appropriate, and informative. Initiates and maintains exchanges even when dealing with an unexpected turn of events. Deals adequately with apparent misunderstandings by checking, confirming, or clarifying
Level descriptors for Interactions
6
5
4
Feature 3
• A candidate with a ‘flat’ profile across the six profiles in the Rating Scale will have a ‘jagged’ proficiency
...occasional loss of fluency... does not prevent effective communication
Vocabulary … sufficient to communicate effectively
Errors ... (in basic structures) … rarely interfere with meaning
Pronunciation, stress, rhythm, and intonation … sometimes interfere with ease of understanding
Extracts from level 4 descriptors
“Can we be certain that two or more ... speakers will find the utterance of a foreign speaker equally comprehensible…?”
(Harris 1977:81)
• Pronunciation is notoriously difficult to assess• What does good pronunciation of a second
language mean?• Standardise comprehensibility?
Feature 4
Feature 5
• Degree of ‘overlap’ between profiles • Linguistic behaviours described in one profile
appear in other profiles• Lack of proficiency in one profile will affect
performance in another
Feature 6
• Rating scale includes both ‘positive’ and ‘negative’ descriptors
...may not vary speech flow...
Can often paraphrase successfully...
Positive descriptor:
Negative descriptor:
Increase in positive descriptions moving from Levels 1-6
Decrease in negative descriptions moving from Levels 1-6
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
Positive descriptors
Negative descriptors
Reliable and accurate rating requires thorough training in the application of the ICAO Rating
Scale
It is not enough to say “ I can apply the Scale”. Raters must
demonstrate this ability
Linguistic Rater competencies:
• Qualified and experienced teacher of English to speakers of other languages?
• Experience in language assessment?• Experience in teaching English for aviation?• Native speaker/certificate of language
proficiency?• Background in aviation operations?• Familiar with R/T phraseology?
Standardisation: Making level 4 level 4
• Intra-rater reliability
• Inter-rater reliability:
a. same test format
b: different tests
c: international standardisation
• Standardisation of rating procedure
Raters must
• understand the criteria and the context in which the criteria occur
• agree upon the standardised procedures for the implementation of the criteria
• inspire trust and gain the confidence of the community and the candidates
ICAO Document 9835, Section 6.7.4
“A body of well-informed and experienced raters must be formed”
ICAO Document 9835, Section 6.7.4
ALRThe Association of
Language Raters
• rate the speech samples and write rationales for the ratings
• organise the rated speech samples into ‘assessment packs’
• produce an association ‘Code of Practice’• formalise guidance material for language
raters
Speech samples are gathered from organisations around the world
Regional ALR representatives (aviation SMEs and linguists) convene to:
Language Rater Accreditation Service
• A prospective language rater applies to ALR for Rater Accreditation
• The Association sends an assessment pack to the rater
• The rater rates speech samples contained in the assessment pack
• If the ratings agree with the ALR standard, an official Certificate of Accreditation is issued
Executive committee Association of Language Raters (ALR)
ICAO
Funding:o Governmental and commercial aviation organisations o Accreditation Service Fees
ALTE
Regional Representatives(Language Raters and Operations Personnel)
Local Rater
Team Leaders (In-country)
FEEDBACK
ILTA
Cyclical presidency
Local Rater
Regional Representatives(Language Raters and Operations Personnel)
Local Rater
Team Leaders (In-country)
Local Rater
STANDARDISATION
Rater Accreditation
• Certificates valid for two years
• On expiry, the rater must re-apply for ALR accreditation
• ALR-accredited raters for assessments for licensing purposes
Regional ALR representatives will convene every two years to
• Rate fresh speech samples for ‘assessment packs’ for the new accreditation period
• Review existing policy for rater accreditation
• Review feedback gathered by Team Leaders
• Produce a published bi-annual report for the international aviation community
• Language Test Accreditation
• Inspection and auditing of language testing organisations
• Training programs
• Regional seminars
Long-term activity
