Computer-aided approach to language testing: How can computer-aided technology help increase authenticity in language tests?

Dr Dana Gablasova

Lancaster University

d.gablasova@lancaster.ac.uk

Language testing: Authenticity

Authenticity – directly related to the validity and usefulness of a test; at the core of authenticity – the relationship between the characteristics of the test task and the TLU task

“We consider authenticity to be an important test quality because it relates the test task to the domain of generalization to which we want our score interpretations to generalize” (Bachman & Palmer, 1996, p. 23-24)

“Another reason for considering authenticity to be important is because of its potential effect on test takers’ perceptions of the test and, hence, on their performance.[…] It is this relevance, as perceived by the test taker, that we believe helps promote a positive affective response to the test task and can thus help test takers perform at their best”. (Bachman & Palmer, 1996, p. 24).

How can computer-aided technology help increase authenticity in language tests?

CL & LT: Growing cooperation

Corpora and computer-aided analysis – a steady presence in the field of LT for several decades since 1990s (e.g. Alderson, 1996)

Major examination bodies commissioned and used corpora in creation or revision of their tests (e.g. Pearson, ETS, Trinity College London, the British Council, Cambridge English Language Assessment).

The topic of corpus approaches to LTA has increasingly appeared in major reference works in the fields of applied linguistics and language testing (e.g. Routledge Handbooks)

Research articles on corpus-based LTA analysis are on the increase in major

LT journals - Language Testing, Language Assessment Quarterly (e.g. Paquot, 2018; Cushing, 2017; Park, 2014)

AL journals - Applied Linguistics, the Modern Language Journal, Language Learning (Staples et al, 2017, 2018; Wisniewski, 2017)

What are corpora?

Corpora are large electronic collections of actual language use, used as evidence in empirical, (mostly) quantitative analysis of language

Corpora are representative

corpora can represent different modes and domains of language use (e.g. spoken, written, online communication, different domains – engineering, business communication, academic writing)

corpora can represent different groups of language users (e.g. of different gender, age, social status, L2 speakers of different proficiency levels, from different L2 backgrounds)

corpora are searchable according to these characteristics

Example: The British Academic Spoken English (BAWE) corpus

• proficient university-level writing – 6.5M words & almost 3,000 good-standard student assignments

• four disciplinary areas (Arts & Humanities, Social Sciences, Life Sciences and Physical Sciences)

Corpus methods

Corpora are typically large (millions & billions of words)

Corpus methods allow us to Process large amounts of data and manipulate them (sort, search,

filter, calculate statistics)

Annotate the data for linguistic, text-related or speaker-related information(e.g. grammatical structures such as all nouns, all premodified nouns; semantic information, e.g. all words related to emotions); search for language from younger or older speakers…

Identify patterns (regularities) in language use o identify features typical (recurring) in certain texts o relate the patterns to underlying (explanatory) variables (speakers’

proficiency or L1 background, previous instruction, text and task properties)

Corpus methods

Concordance analysis Frequency wordlists

Ellis & Simpson-Vlach, 2010

Collocation analysis Keyword analysis

Corpus evidence complements linguistic intuition

See McEnery and Wilson, 2001: 8

Expert + Native speaker

Corpus information (BNC)

damage death

disruption explosion failure

havoc grievous bodily harm

Corpora reveal (hidden) patterns

Corpora: Two types relevant to LT

Two types of corpora relevant for increasing authenticity in language testing:

1. Corpora representing target language use

2. Corpora representing L2 production under exam conditions

Corpora: Target language use (TLU)

L1 USER & EXPERT CORPORA

A. General corpora – typically very large – seek to represent a broadly defined domain (e.g. all of spoken British English) – samples from different genres/registers from this domain

Example: The British National Corpus 2014 • Built in collaboration btw Lancaster University & CUP • Contains a written (ongoing) and spoken part (completed) • Covers different genres of written British English (fiction,

academic prose, journalistic prose, poetry, drama, student essays, emails, twitter messages, …) + informal conversations (10M words)

Corpora: Target language use (TLU)

B. Specialised corpora – represent more narrowly defined domains (e.g. a particular genre such as written academic language or blogs)

• Mostly L1 use – but can be restricted to expert use (not all L1 users have the same level of language mastery, Hulstijn, 2015; Gablasova et al, 2017)

Example: The Pearson International Corpus of Academic English (PICAE)

• 37.1M words from four science areas

• used to develop the Pearson Test in English – Academic (PTE Academic)

Corpora: Target language use (TLU)

Language testing applications

General corpora:

Particularly useful in establishing overall distribution of linguistic features, providing info about frequency of (co)occurrence

Can be used to guide test writing as well as rating scale development

Specialised corpora:

Especially useful for exploring target language domains – to understand the language use in these TLD & to obtain examples for test items or input materials

They “enable test writers to base their testing tasks more closely on authentic rather than contrived language and texts” (Taylor & Barker, 2008, p. 246)

Example: the Pearson International Corpus of Academic English (PICAE) – used to develop the Pearson Test in English – Academic (PTE Academic) (Nesi, 2016)

Corpora: L2 Exam production

L2 (learner) corpora - a group of specialised corpora representing different types of L2 production

usually contain information about speaker characteristics (e.g. proficiency, L1 background)

usually contain information about the linguistic setting (task & genre)

Exam corpora – a special category of L2 (learner) corpora representing language used in an exam context

Corpora: Exam use

Example: Guangwai Lancaster Chinese Learner Corpus • Target language: Chinese

• Over 4.2M Chinese characters

• Contains both written and spoken language from the HSK examination (Chinese proficiency test)

• Different levels of proficiency, coded for errors

Example: Trinity Lancaster Corpus – built in cooperation with Trinity College London, based on the Graded Examination in Spoken English (GESE) Over 4M words and 2,000 exam transcripts Searchable according to a number of variables

-Speaker characteristics (proficiency, age, educational level, L1 background) -Exam characteristics (task, marks for each task) -Released in 2019 (free to use)

participant roles

backchannels

filled pauses

pauses

contextual clues

Corpora: Trinity Lancaster Corpus - GESE extracts

Authenticity, language testing & corpora

Authenticity

Test development

Definition of test construct

Creation of test materials

Validation

Language in the test and TLU

Test development

Understanding of the target language use (TLU) crucial for test development and for authenticity of the test – what is required for the test-taker to perform successfully in the target environment

A variety of ways to collect primary data about the TLU used in LT - corpus findings can complement these and enhance understanding of TLU considerably – moving beyond intuitions and experience with language

Corpus contribute to authenticity in LT to two major areas in test development

Definition of the test construct

Creation of test materials

Test development: Definition of the construct

Describing the test construct – an essential step in producing effective and useful tests (Bachman & Palmer, 1996)

Corpus data – detailed description of distinguishing linguistic features of the actual language use in the target domain evidence-based definition of the construct

Distinguishing linguistic features – typical/frequent

Words & word combinations (collocations)

Grammatical and lexico-grammatical features

Description of the domains

• New domains, not yet (fully) established (e.g. blog posts, online chats, twitter messages… as new(er) genres)

• Redefine description and construct based on existing domains (e.g. academic registers).

Test development: Description of new domains

Example: Language of blogs (Grieve, Biber, Friginal & Nekrasova, 2010)

• Compiled a corpus of American blogs – 2.3M words • Identified different types of blogs and their typical linguistic properties • ‘Personal diary blogs’ – very personal voice, concerned primarily with the

blogger’s own life, also addressee focused • ‘Expert blog’ – very formal and impersonal style, focus on conveying

information on a particular topic

Implications for language testing: • How do test takers write a blog? A corpus-based study of linguistic

features in performances based on the task type ‘Blog’ (Maurer, 2018)

Further Examples of corpus-based linguistic description: Searchable web genres, e.g. opinion pieces, Q&A forums (Biber, Egbert & Davies, 2015); a legal genre – the patent specification (Groom & Grieves, 2019), the language of dark web interactions (Grieve, forth.), patterns in American English (multiple studies)

Test development: Description of existing domains

Revisiting the description of existing domains PICAE – the Pearson International Corpus of Academic English

T2K-SWAL - the TOEFL 2000 Spoken and Written Academic Language – constructed to inform development of specific tests of academic English

Re-evaluation & further description of more general constructs Römer (2017) used L1 corpora to investigate formulaic language in

spoken communication & compared it to ratings scales of major exams (the TOEFL iBT, IELTS, Cambridge English: Advanced exam)

Results: formulaic units had a major role in spoken language in the corpora – they were not consistently included in the construct definition and rating scales of these exams

Test development: Test writing

L1/EXPERT CORPORA: benefits from general & specialised L1/expert corpora for test writing - most typical (frequent and recurring) features in the TLU – to be considered for inclusion in the test development

Direct use of L1/expert corpora – source of authentic examples in the TLU – used for input materials and test items – enhancing correspondence between the language in the test and the target environment

Indirect use of L1/expert corpora – to check how similar the test items are to the authentic language use

Test validation

Eliciting language similar to TLU necessary for

Authenticity and validity of the test

Ability to predict whether the test-takers will be able to successfully communicate in the TLU

Corpora are effective at establishing the correspondence btw language use elicited in tests and the TLU

Example: Comparison of language produced in the TOEFL iBT and a corpus representing the target domain (e.g. Staples, Biber & Reppen, 2018)

Compared lexico-grammatical characteristics of texts produced by L2 users in TOEFL iBT & academic texts produced by the same writers as part of their degree (e.g. argumentative essays, lab reports)

Reported a number of differences related to linguistic features such as stance-taking (e.g. use of personal pronouns and specific verbs), hedging

Authenticity: Impact on teaching materials

The final message: Who can use corpora?

ANYONE !

What about RESOURCES & AVAILABILITY?

The final message: Who can use corpora?

1. Corpora

• Freely available corpora through Sketch Engine, CQPweb, #LancsBox

Examples: Guangwai-Lancaster corpus, Trinity Lancaster Corpus, BNC,

BNC2014, Open Cambridge Learner Corpus, EFCAMDAT, British Academic Written English (BAWE), British Academic Spoken Corpus (BASE)

• DIY Corpora: Possible to build own ones (especially from materials in electronic format) & analyse them with #LancsBox

The final message: Who can use corpora?

2. Corpus analysis, skills & training

Interfaces: more user-friendly & Software for own analysis: Freely-downloadable

#LancsBox (http://corpora.lancs.ac.uk/lancsbox/):

You can load your own corpus (in any format, it will deal with it)

Using corpus tools is easy: Wordlists, Frequencies, Collocations, Keywords, Concordancing, etc.

Who can use corpora

lancaster.ac.uk/corpussummerschools/

Who can use corpora

Starting again: September 2019

Thank you!

Alderson , J. C. (1996) . Do corpora have a role in language assessment? In Thomas, J. A. & Short, M. H. (Eds.) Using corpora for language research (pp. 248–59). London: Longman.

Bachman, L. F., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.

Biber, D., Conrad, S., Reppen, R., Byrd, P., & Helt, M. (2006). The Spoken and written academic language (T2K-SWAL) corpus. University language: A corpus-based study of spoken and written registers, Amsterdam: John Benjamins.

Brown, A. (2003). Interviewer variation and the co-construction of speaking proficiency. Language Testing, 20(1), 1-25.

Brown, A. (2013). Interlocutor and rater training. In Fulcher, G. & Davidson, F. (Eds), The Routledge Handbook of Language Testing (pp. 427-439). New York: Routledge.

Cushing, S. T. (2017). Corpus linguistics in language testing research. Language Testing, 34(4), 441-449.

Gablasova, D., Brezina, V. & McEnery, T. (forth). The Trinity Lancaster Corpus: Development, description and application. International Journal of Learner Corpus Research.

Gablasova, D., Brezina, V., McEnery, T. & Boyd, E. (2017). Epistemic stance in spoken L2 English: The ef fect of task type and speaker style. Applied Linguistics, 38(5), 613-637

Hulstijn, J. H. (2015). Language proficiency in native and non-native speakers: Theory and research. Amsterdam: John Benjamins.

LaFlair, G. T., & Staples, S. (2017). Using corpus linguistics to examine the extrapolation inference in the validity argument for a high-stakes speaking assessment. Language Testing, 34(4), 451-475.

Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T. (2017). The Spoken BNC2014. International Journal of Corpus Linguistics, 22(3), 319-344.

McEnery, A. M., & Wilson, A. (2001). Corpus linguistics: an introduction. Edinburgh University Press.

Nesi, H. (2016). Corpus studies in EAP. In The Routledge Handbook of English for Academic Purposes (pp. 230-241). New York: Routledge.

Paquot, M. (2018). Phraseological competence: A missing component in university entrance language tests? Insights from a study of EFL learners’ use of statistical collocations. Language Assessment Quarterly, 15(1), 29-43.

Park, K. (2014). Corpora and language assessment: The state of the art. Language Assessment Quarterly, 11(1), 27-44.

Römer, U. (2017). Language assessment and the inseparability of lexis and grammar: Focus on the construct of speaking. Language Testing, 34(4), 477-492.

Staples, S., Biber, D., & Reppen, R. (2018). Using corpus‐based register analysis to explore the authenticity of high‐stakes language exams: A register comparison of TOEFL iBT and disciplinary writing tasks. The Modern Language Journal, 102(2), 310-332.

Staples, S., LaFlair, G. T., & Egbert, J. (2017). Comparing language use in oral proficiency interviews to target domains: Conversational, academic, and professional discourse. The Modern Language Journal, 101(1), 194-213.

Taylor, L., & Barker, F. (2008). Using corpora for language assessment. In E. Shohamy & N. H. Hornberger (Eds.), Encyclopedia of language and education: Language Testing and Assessment (pp. 241–254). New York: Springer.

Thewissen, J. (2013). Capturing L2 accuracy developmental patterns: Insights from an error‐tagged EFL learner corpus. The Modern Language Journal, 97(S1), 77-101.

Wisniewski, K. (2017). Empirical learner language and the levels of the Common European Framework of Reference. Language Learning, 67(S1), 232-253.