using real aviation communications to create tasks for
TRANSCRIPT
INTERNATIONAL CIVIL AVIATION ENGLISH ASSOCIATION
supporting the use of English for aviation safety
Using real aviation communications
to create tasks for training and testing (2) Aline Pacheco (PUCRS)
Ana Lú cia Monteiro (ANAC, ICAEA) Angela Garcia (ANAC, Carleton)
Malila Prado (Fujian University of Technology) Patrícia Tosqui-Lucks (ICEA)
Collaborative Aeronautical English Corpus
• Webinar 5a: Let’s create a collaborative Aeronautical English corpus together?
AEROCORPUS
• Steps for compiling a corpus:
• Step 1 - Find a suitable RTF communication;
• Step 2 - Check if it has a transcript or transcribe it;
• Step 3 - Fill in the chart with the transcript;
• Step 4 - Fill in the chart with the information from the text;
• Step 5 - Save it at padlet. Done!
Today. Thank you!!!
New Challenge
• Let’s see how many scripts we can have by November, 24th – 2 weeks
https://padlet.com/patricialucks/a2rp4dol4uhpswjm
• We will share the template AND guidelines for posting in the chat - in the end of the webinar.
• We will put together all transcripts we get collaboratively and share them all with the people who participated in this project
• Send us an e-mail to receive the corpus – or if you have questions
Objectives
To discuss the usefulness of Corpus Linguistics (CL) to
Language Testing and Assessment;
To present some applications of Corpus Linguistics to task design in the assessment of pilots and controllers.
Guide
Definitions
Usefulness of Corpus Linguistics to Language Testing and Assessment
Our focus
Construct specification – speaking and listening
Task design:
1) listening tasks;
2) speaking tasks.
Conclusion
References
Definitions
“Corpus linguistics is a research method which employs corpora for data
extraction. A linguistic corpus is a bank of texts stored in computers,
which allow for a (semi-) automatic extraction of data by using statistical
analysis”.
(Prado & Tosqui-Lucks, 2019, p. 116)
Corpus Linguistics can be useful at all phases of test development and validation (Cushing, 2017).
• Comparative analyses of :
1) specialized corpus vs. general corpus
2) learner corpus vs. expert users corpus
3) test performances corpus
Usefulness
Usefulness
Language testers can also check intuitions against empirical corpus data (Cushing, 2017):
1) For task and item design:
- make decisions about what features of language are criterial at different levels of proficiency,
- check the prevalence of certain error types for creating plausible distractors for multiple choice questions,
- identify the features that make listening or reading texts more or less difficult.
2) For rating scale design: based on the close analysis of learner language.
Usefulness
Taylor and Barker (2008) add that “corpora of language test content (input)
and of test-taker performance (output) provide language testers with
important archives that enable them to address key issues such as:
• comparability across test forms,
• rater training and standardisation,
• standard setting and maintenance of standards over time,
• and investigation of test bias across different test-taker populations”(p. 247).
Our focus
To use a specialized corpus (from expert users) in order to inform the design of test tasks and items that authentically reflect the target language use (TLU) domain.
professional context of international aeronautical RT communications LSP testing = Language for Specific Purpose testing (Douglas, 2000) LAPP = Language Assessment for Professional Purposes (Knoch & Macqueen, 2020)
Test development process
Test Specs
Frameworks
Models Construct framework - What complex of
knowledge, skills, or other attributes
should be assessed?
Evidence Models - What behaviors or
performances should reveal those
constructs?
Task Models - What tasks or situations
should elicit those behaviors?
(Adapted from Fulcher and Davidson, 2007, 2009)
Construct specification – Monteiro (2019)
Source: Adapted from Monteiro (2019, p. 220)
Interactional Competence - Definition
Interactional Competence refers to “the pragmatic relationship between participants’ employment of linguistic and interactional resources and the contexts in which they are employed …[and] how those resources are employed mutually and reciprocally by all participants in a particular discursive practice” (Young, 2011, p. 428).
Interactional competence “is not what a person knows, it is what a person does together with others” (p. 430).
Construct specification of listening tests
What needs to be assessed in relation to the listening performed by pilots in RT communications with the ATCOs (Garcia & Fox, 2020): - To understand main ideas; - To recognize words and numbers; - To understand both plain English and phraseology; - To extract specific meaning from both short and long transmissions; - To understand messages that differ from what was expected; - To use strategies in order to understand.
Interactive Listening
Interactive listening – John Field (2020, Webinar) “Listening as an equal partner in a conversation is much more cognitively demanding than (e.g.) listening to a recorded conversation:
a) lexical search and parsing – to tag pieces of language in short term memory to link their next utterance to the one they are hearing; b) meaning construction – to recognize the intention of the speaker, interpret pronouns and infer information that has not been explicitly mentioned”.
under extreme time pressure – construction of his/her next utterance
Task design
Methodology:
• Corpus – CORPAC (non-routine events) – specialized corpus
Pacheco, 2021 (forthcoming)
• Software - Wordsmith
• Concordance analysis – “confirm” to explore: • type of info that required confirmation/clarification • contexts in which they appeared
Task design
Task design
48 instances of confirm/confirmed/confirming
in the corpus
13 were from the same transcript
Task design
1) Listening tasks
• Construct to be measured – Ability to recognize words and numbers, understand main ideas, and to extract specific meaning from both short and long transmissions.
• Goal – To develop test tasks that require pilots and ATCOs to understand transmissions, including headings, distances, emergency situations, clarification requests, etc.
Selected task types - Completing a table
- Answering multiple choice questions
- Reporting what happened in a non-routine situation
- Answering questions about a non-routine situation
Task design
Task 1: You will hear a short pilot/ATCO transmission. Complete the table below filling in the spaces in the right column.
PILOT CALL SIGN , […]
ATCO Unreadable, please say again.
PILOT CALL SIGN, requesting heading for ditching.
PILOT Ditching.
PILOT MAYDAY, we need vectoring for the sea for ditching. Away
from the trees.
ATCO Unreadable. Confirm you need vectors to land or UH ditch
fuel?
PILOT Vectors for ditching. Ditching. The aircraft is uncontrollable.
We need vectors to land ditching.
ATCO Roger sir, in that case you can fly to […] fly heading 260.
PILOT We need vectoring to the sea, please. We will be ditching.
Come on! Ditching.
ATCO CALL SIGN, fly heading 260.
PILOT 260.
PILOT CALL SIGN, please, request vectoring to the sea area,
please. We will be ditching. Aircraft is completely unstable.
ATCO CALL SIGN, you have the river, if you turn, by the left or
right heading 260. It’s the closest thing.
PILOT 260 for the river, okay. Maybe ocean? Not so far.
ATCO The ocean is 40 miles from your position.
PILOT Okay, please request back to the river.
ATCO CALL SIGN, left heading 260.
PILOT Left heading 260.
1. Pilot’s initial request to
ATCO
2. The pilot wanted to avoid
3.The ATCO asked the pilot to
confirm if he wanted to
or
4. Problem reported by pilot
5. Heading assigned
6. Distance to the ocean
7. Instructions to reach the
river
Adapted script:
Task design
Task 1: You will hear a short pilot/ATCO transmission. Complete the table below filling in the spaces in the right column.
1. Pilot’s initial request to
ATCO
2. The pilot wanted to avoid
3.The ATCO asked the pilot to
confirm if he wanted to
or
4. Problem reported by pilot
5. Heading assigned
6. Distance to the ocean
7. Instructions to reach the
river
PILOT American 2493 , […]
ATCO Unreadable, please say again.
PILOT MAYDAY MAYDAY MAYDAY, American 2493, we need
vectors to the sea for ditching. We need to avoid trees.
ATCO American 2493, unreadable. Confirm you need vectors to
land or to dump fuel?
PILOT American 2493, vectors for ditching. Ditching. The aircraft is
uncontrollable. We need vectors to ditch.
ATCO American 2493, Roger sir, in that case you can fly to KMIA,
fly heading 260.
PILOT We need vectoring to the sea, please. We will be ditching.
Ditching.
ATCO American 2493, fly heading 260.
PILOT Heading 260, American 2493.
PILOT American 2493, please, request vectors to the ocean,
please. We will be ditching. Aircraft is completely unstable.
ATCO American 2493, you have the river, if you turn, by the left or
right heading 260. It’s the closest thing.
PILOT Heading 260 for the river, American 2493. Maybe the
ocean? Is it too far?
ATCO American 2493, the ocean is 40 miles from your position.
PILOT Okay, please request back to the river.
ATCO American 2493, left heading 260.
PILOT Left heading 260, American 2493.
Adapted script:
Task design
The pilot requested vectors to the ATCO in order to ….
a) dump fuel over the sea
b) ditch the aircraft
c) land on a clear field
d) land in an airport by the sea
The pilots were facing ….
a) unstable engine pressure
b) uncontrollable fire onboard
c) loss of Control In-flight
d) shortage of fuel
The pilot’s final intention was to be vectored to….
a) the sea, as it was 40 miles from the airport
b) the sea, as it was not so far from their position
c) the river, as it was 40 miles from their position
d) the river, as it was the closest option
Task 2: Listen to a short recording (same as Task 3) and choose the best option to complete each statement:
Task design
CORPAC 43:
PILOT – Approach, CALL SIGN, MAYDAY MAYDAY MAYDAY. We lost control… 4000 feet on heading 030.
(…)
PILOT– MAYDAY, we need vectoring for the sea for ditching. Away from the trees.
ATCO – Unreadable. Confirm you need vectors to land or uh ditch fuel?
PILOT – Vectors for ditching. Ditching. The aircraft is uncontrollable. We need vectors to land ditching.
ATCO – Roger sir, in that case you can fly to uh fly heading 260.
Adapting:
PILOT: MAYDAY, MAYDAY, MAYDAY, Miami Approach, American 2493, we lost control of the aircraft. Request vectors to the
sea for ditching. Away from the trees.
ATCO: American 2493, Miami Approach, Roger Mayday. Confirm you need vectors to land or dump fuel?
PILOT: Miami Approach, American 2493, we need vectors for ditching. Ditching. The aircraft is uncontrollable. I say again, we
need vectors to land on the sea.
ATCO: Roger, 2493, in that case you can fly heading 260.
Task design
Task 3: Listen to a short recording. Then, tell the examiner, in your own words, everything you understood. Task 4: Listen to a short recording. The examiner will then ask you some questions. a) What was the reason for the Mayday call?
b) What was the pilot’s request to deal with this situation? c) What information did the ATCO ask the pilot to confirm? d) How important was this request for confirmation, in your opinion? e) What was the heading assigned by the ATCO?
Task design
2) Speaking tasks • Construct to be measured – Interactional competence = ability to deal
with apparent misunderstandings, by checking, confirming and clarifying • Goal – develop scenarios for test tasks that require pilots and ATCOs to
confirm or clarify information/instructions
• Selected task types - Brazilian test for pilots (SDEA), Part 2
- roleplay for ATCOs
Task design
SDEA Part 2 – Interacting as a pilot (Mock version - example)
Interlocutor: You are going to land at Frankfurt Airport. Listen to Frankfurt Center and read back. Recording 1: ANAC 123, [traffic is overtaken]. [Descend to flight level two niner zero]. Test-taker reads back. Interlocutor: Imagine you have just experienced a rapid decompression. Call Frankfurt Center to report the situation and say your intentions. Test-taker interacts with ATCO. Recording 2: ANAC 123, [descend to flight level zero niner zero], [I understand you had a loss of hydraulic pressure], confirm? Test-taker interacts with ATCO, confirming or clarifying. Interlocutor: What did the controller say?
Test-taker responds.
Task design
Step 1: Setting the scene: context and routine communication (Option 1)
Task:
Interlocutor: You are going to land at Miami Airport.
Listen to Miami Approach and read back.
TRACK 1
ANAC 123, [turn left] [heading two-
seven-zero]; [descend and maintain
10 thousand]. [Expect ILS approach
runway 09].
Test-taker reads back.
CORPAC – Transcript 2
ATCO – CALL SIGN, turn heading two-seven-zero;
descend and maintain one-zero, ten thousand.
Expect the ILS runway nine approach.
...
PILOT- Miami Approach, CALL SIGN is eight
thousand- I mean, we’re at nine thousand to eight
thousand.
Task design
Step 1: Setting the scene: context and routine communication (Option 2)
Task:
Interlocutor: You are flying from Lisbon to Cairo,
climbing after departure. Listen to Lisbon Approach
and read back.
TRACK 1
ANAC 123, [turn left] [heading zero-
six-zero]; [climb and maintain 4000
feet] .
Test-taker reads back.
CORPAC – Transcript 43
An (COMPANY) Embraer 190 performing flight from
LPPT to HECA was climbing after departure when
the pilots reported a flight control problem stating
that the aircraft was totally uncontrollable.
PILOT: XXX Approach, CALL SIGN, MAYDAY
MAYDAY MAYDAY. Climbing 4000 feet on heading
060. Request…
Task design Step 2a: Non-routine communication
Task:
Interlocutor: Imagine you have lost control of the
airplane and decided to ditch the airplane in the sea.
Call Miami Approach to report the situation and say
your intentions. Test-taker interacts with ATCO. TRACK 2
(Confirm)
ANAC 123, [Descend to 2500], [QNH
1010]. [Confirm you wish the sea and not
the river?]
Or
ANAC 123, [Descend to 2500], [QNH
1010]. [Confirm you’d like to proceed for a
ditching over the sea?]
Test-taker interacts with ATCO, confirming or clarifying.
CORPAC – Transcript 43
PILOT – CALL SIGN, requesting heading for ditching.
PILOT – Ditching.
PILOT – MAYDAY, we need vectoring for the sea for ditching.
Away from the trees.
ATCO – Unreadable. Confirm you need vectors to land or UH
ditch fuel?
PILOT – Vectors for ditching. Ditching. The aircraft is
uncontrollable. We need vectors to land ditching.
_________
ATCO – CALL SIGN, descend to 2500 feet, QNH 1010.
...
ATCO – Confirm you wish the sea and not the river?
…
ATCO – Confirm, you’d like to proceed for a ditching over the
sea?
Task design Step 2b: Non-routine communication
Task:
Interlocutor: Imagine you have lost control of the
airplane and decided to ditch the airplane in the sea.
Call Miami Approach to report the situation and say
your intentions. Test-taker interacts with ATCO. TRACK
2a
(Clarify)
ANAC 123, [Descend to 2500], [QNH
1010]. [Confirm you wish the river and
not the sea?]
Or
ANAC 123, [Descend to 2500], [QNH
1010]. [Confirm you need vectors to
dump fuel over the sea?]
Test-taker interacts with ATCO, confirming or clarifying.
CORPAC – Transcript 43
PILOT– CALL SIGN, requesting heading for ditching.
PILOT – Ditching.
PILOT – MAYDAY, we need vectoring for the sea for ditching.
Away from the trees.
ATCO – Unreadable. Confirm you need vectors to land or UH
ditch fuel?
PILOT – Vectors for ditching. Ditching. The aircraft is
uncontrollable. We need vectors to land ditching.
_________
ATCO – CALL SIGN, descend to 2500 feet, QNH 1010.
...
ATCO – Confirm you wish the sea and not the river?
…
ATCO – Confirm, you’d like to proceed for a ditching over the
sea?
Task design
Interlocutor: You are going to land at Miami Airport. Listen to Miami Approach and read back.
TRACK 1
ANAC 123, [turn left] [heading two-seven-zero]; [descend and maintain one-zero, ten thousand].
[Expect the ILS runway nine approach].
Test-taker reads back.
Interlocutor: Imagine you have lost control of the airplane and decided to ditch the airplane over the sea. Call
Miami Approach to report the situation and say your intentions. Test-taker interacts with ATCO.
TRACK 2a
(Confirm)
ANAC 123, [Descend to 2500], [QNH 1010]. [Confirm you wish the sea and not the river?]
Or
ANAC 123, [Descend to 2500], [QNH 1010]. [Confirm you’d like to proceed for a ditching over the
sea?]
TRACK 2b
(Clarify)
ANAC 123, [Descend to 2500], [QNH 1010]. [Confirm you wish the river and not the sea?]
Or
ANAC 123, [Descend to 2500], [QNH 1010]. [Confirm you need vectors to ditch fuel over the sea?]
Test-taker interacts with ATCO, confirming or clarifying.
Task design
Information: You are going to listen to a short conversation between a Departure Controller and the pilot of Diamond 453, who is experiencing some problems.
When the recording finishes, you will take the role of the ATCO and interact with the examiner, who is going to play the role of the pilot. Ask and answer questions as you would do in real life.
At the end of the roleplay, I will ask you some questions.
Roleplay – Test-taker = ATCO
Task design
(After the roleplay): Now, answer some questions about the situation in the roleplay: 1) What was the emergency the pilot was facing? 2) What did the pilot request? 3) How did you, as the controller, try to help the pilot? 4) How important was it to confirm the pilot’s intentions? 5) Have you ever faced a situation like this? What happened?
Task design
Roleplay task:
Listening part – recorded by SMEs, add authenticity (sound effects)
Examiner’s script (pilot) – guidance, but requires flexibility, because communication is co-constructed at the moment of interaction
Role of pilot – performed by an SME, who is knowledgeable to adapt his responses and questions according to previous utterances
Prompts to test-taker – computer screen (diagrams, maps), cards with relevant information, pictures, etc.
Conclusion
Present the usefulness of Corpus Linguistics to Language Testing and Assessment Design tasks based on the construct (e.g., Interactional Competence and
Interactive Listening) Identify features of the construct in transcripts of real communications using CL
analysis Create different task types: not interactive – somewhat interactive – totally interactive Adapt the transcript to create task input, questions and options Reminder: Corpus Linguistics is not prescriptive, it is descriptive Work closely with an SME, extract what is good and adapt to your needs
Conclusion
Corpus
linguistics
Training Testing
References
Cushing, S. T. (2017). Corpus linguistics in language testing research. Language Testing, 34(4), 441–449.
Douglas, D. (2000). Assessing languages for specific purposes. Cambridge: Cambridge University Press.
Field, J. (2020). Idle chatter: What really goes on in tests of interactive communication? CRELLA Symposium. Available at https://www.youtube.com/watch?v=ONM7xcJTPD0&feature=youtu.be
Fulcher, G., & Davidson, F. (2007). Language testing and assessment: An advanced resource book. London and New York: Routledge.
Fulcher, G., & Davidson, F. (2009). Test architecture, test retrofit. Language Testing, 26(1), 123-144. https://doi.org/10.1177/0265532208097339
References
Garcia, A. C. M., & Fox, J. (2020). Contexts and constructs: Implications for the testing of listening in pilots’ communication with air traffic controllers. The Especialist, 41(4), 1-33. DOI: https://doi.org/10.23925/2318-7115.2020v41i4a4
Knoch, U., & Macqueen, S. (2020). Assessing English for professional purposes. Milton: Routledge.
Monteiro. A. L. T. (2019). Reconsidering the measurement of proficiency in pilot and air traffic controller radiotelephony communication: From construct definition to task design (Unpublished doctoral dissertation). Carleton University, Ottawa, Canada.
Pacheco, A. (2021). Analyzing the use of personal pronouns in aeronautical communications through CORPAC (Corpus of Pilot and Air Traffic Controller Communications). RELIN: Revistas de Estudos da Linguagem, Corpus Linguistics: Achievements and Challenges, 29(2) (forthcoming).
References
Prado, M., & Tosqui-Lucks, P. (2019). Designing the Radiotelephony Plain English Corpus (RTPEC): A specialized spoken English language corpus towards a description of aeronautical communications in non-routine situations. Research in Corpus Linguistics 7,113–128.
Taylor L., & Barker F. (2008). Using Corpora for Language Assessment. In: Hornberger N. H. (Eds), Encyclopedia of Language and Education. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-30424-3_179
Young, R. (2011). Interactional competence in language learning, teaching and testing. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (Vol. 2, pp. 426–443). New York: Routledge.
