Issues in Assessing Language Issues in Assessing Language Proficiency and Academic AchievementProficiency and Academic Achievement

Lyle F. BachmanLyle F. BachmanDepartment of Applied Linguistics & Department of Applied Linguistics &

TESLTESL

University of California, Los AngelesUniversity of California, Los Angeles

OverviewOverview ESEA mandateESEA mandate Nature of language use and language abilityNature of language use and language ability Issues in assessing academic English proficiencyIssues in assessing academic English proficiency Measuring English proficiency as opposed to Measuring English proficiency as opposed to

academic achievementacademic achievement Current assessments of English proficiency for ELLsCurrent assessments of English proficiency for ELLs ConclusionConclusion

ESEA MandateESEA Mandate

Assess “academic English proficiency”Assess “academic English proficiency”• Oral languageOral language• ReadingReading• Writing skillsWriting skills

Assess at all levels, from beginning to advanced, K-12Assess at all levels, from beginning to advanced, K-12 Assess annuallyAssess annually Assessments must be sensitive to annual increases in Assessments must be sensitive to annual increases in

proficiencyproficiency

Nature of Language UseNature of Language Use Language is used to accomplish things in the Language is used to accomplish things in the

real world.real world. Language useLanguage use is the situated negotiation of is the situated negotiation of

meaning through language.meaning through language. Language use involves:Language use involves:

• Performing Performing functionsfunctions that are that are• Realized as Realized as stylesstyles or or registersregisters, and, and• Expressed through a Expressed through a formal codeformal code..

Language FunctionsLanguage Functions Language functions are the social uses to which we put Language functions are the social uses to which we put

language. (Halliday 1976). language. (Halliday 1976). Four macro-functions (Bachman, 1990)Four macro-functions (Bachman, 1990)

• IdeationalIdeational• ManipulativeManipulative• HeuristicHeuristic• ImaginativeImaginative

General classroom functions (Valdez Pierce and General classroom functions (Valdez Pierce and O’Malley, 1992)O’Malley, 1992)

Scientific language functions (Chamot & O’Malley, 1987)Scientific language functions (Chamot & O’Malley, 1987)

Register and StyleRegister and Style RegisterRegister: a variety of language that is : a variety of language that is

associated with a particular context or domain. associated with a particular context or domain. (Halliday et. al. 1964)(Halliday et. al. 1964)• Distinguished essentially by grammar and Distinguished essentially by grammar and

vocabularyvocabulary StyleStyle: a variety of language used by : a variety of language used by

individuals, appropriate to levels of formality.individuals, appropriate to levels of formality.• Ranges from intimate to consultative to Ranges from intimate to consultative to

oratorical (Joos 1967)oratorical (Joos 1967)

Formal CodeFormal Code FFormal codeormal code: the system of language that we use to : the system of language that we use to

express functions in specific contexts.express functions in specific contexts.• Grammatical structuresGrammatical structures• VocabularyVocabulary

TechnicalTechnicalSubtechnicalSubtechnicalGeneral, with field specific meaningGeneral, with field specific meaningGeneralGeneral

• Cohesive markersCohesive markers• Rhetorical organization/conversational structureRhetorical organization/conversational structure

Language ability (proficiency)Language ability (proficiency) Knowledge of functions, register, formal code Knowledge of functions, register, formal code Capacity for implementing this knowledge in Capacity for implementing this knowledge in

language uselanguage use

Issues in assessing academic English Issues in assessing academic English proficiencyproficiency

1.1. Defining the construct, “academic English Defining the construct, “academic English proficiency”proficiency”

2.2. Specifying assessment tasksSpecifying assessment tasks3.3. Setting performance standardsSetting performance standards

Defining the construct, “academic Defining the construct, “academic English proficiency”English proficiency”

Several different approaches to thisSeveral different approaches to this

1.1. Academic language as a compilation of Academic language as a compilation of unique language functions and structures unique language functions and structures that are difficult for language minority that are difficult for language minority students to master (Hamayan and Perlman, students to master (Hamayan and Perlman, 1990)1990)

Defining the construct, “academic Defining the construct, “academic English proficiency”English proficiency”

2.2. Academic language as distinct from Academic language as distinct from interpersonal conversational language interpersonal conversational language (Cummins, 1980, 1983; Mohan, 1986)(Cummins, 1980, 1983; Mohan, 1986)

• Amount of cognitive demandAmount of cognitive demand• Degree of contextualizationDegree of contextualization• Basic interpersonal communication skills (BICS): Basic interpersonal communication skills (BICS):

cognitively cognitively unundemanding and context-embeddeddemanding and context-embedded• Cognitive academic language proficiency (CALP): Cognitive academic language proficiency (CALP):

cognitively demanding and context-reduced cognitively demanding and context-reduced

Defining the construct, “academic Defining the construct, “academic English proficiency”English proficiency”

3.3. Academic language as language in interaction Academic language as language in interaction (e.g., Mehan, 1979)(e.g., Mehan, 1979)

• Turn-takingTurn-taking• Patterns of participationPatterns of participation• Initiation-Reply-Evaluation sequence Initiation-Reply-Evaluation sequence

characterizes classroom discoursecharacterizes classroom discourse

Defining the construct, “academic Defining the construct, “academic English proficiency”English proficiency”

4.4. Academic language as stylistic registers Academic language as stylistic registers (Solomon & Rhodes, 1995)(Solomon & Rhodes, 1995)

• Styles tied to specific academic tasksStyles tied to specific academic tasks• Stylistic registers associated with broad, Stylistic registers associated with broad,

discourse levels of language; not limited to discourse levels of language; not limited to sentence-level linguistic featuressentence-level linguistic features

Specifying assessment tasksSpecifying assessment tasks1.1. ““Skills” versus “activities”Skills” versus “activities”

• Another way to conceptualize speaking, Another way to conceptualize speaking, reading, writing: as activities we perform reading, writing: as activities we perform with languagewith language

• Most theoretical models of speaking, Most theoretical models of speaking, reading and writing are models of reading and writing are models of processes, rather than of skill or ability.processes, rather than of skill or ability.

Specifying assessment tasksSpecifying assessment tasks2.2. Language use taskLanguage use task

““an activity that involves people in using an activity that involves people in using language for the purpose of achieving a language for the purpose of achieving a particular goal in a particular situation”particular goal in a particular situation” (Bachman & Palmer, 1996)(Bachman & Palmer, 1996)

3.3. Use framework of task characteristics to Use framework of task characteristics to describe language assessment tasks as specific describe language assessment tasks as specific instances of language use tasksinstances of language use tasks

Language proficiency and Language proficiency and assessment tasksassessment tasks

The way we define the construct, “academic The way we define the construct, “academic English proficiency” andEnglish proficiency” and

the way we specify assessment tasksthe way we specify assessment taskswill determine the kinds of will determine the kinds of inferencesinferences we can we can

make, andmake, andthe domains to which these inferences the domains to which these inferences generalizegeneralize..

Setting performance standardsSetting performance standards1.1. ELLs vs. “native English speakers”ELLs vs. “native English speakers”

• Different standards?Different standards?• Continuous scale of ability?Continuous scale of ability?

2.2. Performance standardPerformance standard• ““Educated native speaker”?Educated native speaker”?• Based on professional content standards Based on professional content standards

(e.g., TESOL, 1997) or state standards?(e.g., TESOL, 1997) or state standards?

Measuring English proficiency as Measuring English proficiency as opposed to academic achievementopposed to academic achievement

Two-part conundrum:Two-part conundrum:1.1. Distinguishing English proficiency from Distinguishing English proficiency from

content knowledgecontent knowledge2.2. Designing assessment tasks that are Designing assessment tasks that are

appropriate for the construct definitionappropriate for the construct definition

Measuring English proficiency as Measuring English proficiency as opposed to academic achievementopposed to academic achievement

Distinguishing English proficiency from content Distinguishing English proficiency from content knowledge: three options (Bachman & Palmer, knowledge: three options (Bachman & Palmer, 1996, Bachman, in press)1996, Bachman, in press)

1.1. Define construct as English proficiency only (i.e., formal Define construct as English proficiency only (i.e., formal code, functions, register)code, functions, register)

2.2. Define construct as content knowledge only (e.g., Define construct as content knowledge only (e.g., knowledge of mathematics, history, science)knowledge of mathematics, history, science)

3.3. Define construct as combination:Define construct as combination:““ability to use English to process information about ability to use English to process information about academic content” academic content”

Measuring English proficiency as Measuring English proficiency as opposed to academic achievementopposed to academic achievement

Designing assessment tasks that are appropriate for the Designing assessment tasks that are appropriate for the construct definition. construct definition.

Two Hypotheses:Two Hypotheses: The more we design tasks to isolate English proficiency, the The more we design tasks to isolate English proficiency, the

more our assessment tasks look like non-language, and the more our assessment tasks look like non-language, and the more we limit our inferences to the formal language code, i.e., more we limit our inferences to the formal language code, i.e., grammar and vocabularygrammar and vocabulary

The more we try to design tasks to assess aspects of language The more we try to design tasks to assess aspects of language use (functions, cohesion, rhetorical /conversational use (functions, cohesion, rhetorical /conversational organization), the more topical content we include in our organization), the more topical content we include in our taskstasks

Measuring English proficiency as Measuring English proficiency as opposed to academic achievementopposed to academic achievement

Designing assessment tasks that are appropriate Designing assessment tasks that are appropriate for the construct definition. for the construct definition.

GrammarGrammar Markers of Markers of CohesionCohesion

VocabularyVocabulary Rhetorical Rhetorical OrganizationOrganization

Language Language FunctionsFunctions

RegisterRegister

Aspects of English Proficiency Aspects of English Proficiency AssessedAssessed

Involvement of Content Involvement of Content Knowledge/Topical ContentKnowledge/Topical Content

MoreMoreLessLess

Current assessments of English Current assessments of English proficiency for ELLsproficiency for ELLs

1.1. Great deal of variety in areas of language Great deal of variety in areas of language ability and assessment tasks includedability and assessment tasks included

2.2. Tasks generally reflect the hypotheses aboveTasks generally reflect the hypotheses above3.3. Typically difficult to distinguish English Typically difficult to distinguish English

proficiency assessment tasks from academic proficiency assessment tasks from academic achievement assessment tasks, especially in achievement assessment tasks, especially in the area of readingthe area of reading

ConclusionConclusion

1.1. Current approaches to assessing English Current approaches to assessing English proficiency for ELLs have, in my view, failed proficiency for ELLs have, in my view, failed to build an adequate validation argument to to build an adequate validation argument to provide evidence supporting score-based provide evidence supporting score-based inferences of English proficiency inferences of English proficiency onlyonly

2.2. Assessments of English proficiency based on Assessments of English proficiency based on current approaches (“first generation” current approaches (“first generation” assessments) are not likely to resolve this assessments) are not likely to resolve this issueissue..

ConclusionConclusion

3.3. For “next generation” assessments, need to For “next generation” assessments, need to conduct research that would investigateconduct research that would investigate::

• The differential effects of English proficiency and The differential effects of English proficiency and content knowledge on test performancecontent knowledge on test performance

• The differential effects of a wide range of The differential effects of a wide range of assessment task types on test performanceassessment task types on test performance

• Utilize recent developments in measurement Utilize recent developments in measurement theory and test design (e.g., Mislevy et. al. 2002a, theory and test design (e.g., Mislevy et. al. 2002a, 2002b) to disentangle student abilities from task 2002b) to disentangle student abilities from task characteristicscharacteristics