Transcript
Page 1: Davies Textbook Trends in Teaching Language Testing

http://ltj.sagepub.com

Language Testing

DOI: 10.1177/0265532208090156 2008; 25; 327 Language Testing

Alan Davies Textbook trends in teaching language testing

http://ltj.sagepub.com/cgi/content/abstract/25/3/327 The online version of this article can be found at:

Published by:

http://www.sagepublications.com

can be found at:Language Testing Additional services and information for

http://ltj.sagepub.com/cgi/alerts Email Alerts:

http://ltj.sagepub.com/subscriptions Subscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.co.uk/journalsPermissions.navPermissions:

http://ltj.sagepub.com/cgi/content/refs/25/3/327 Citations

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 2: Davies Textbook Trends in Teaching Language Testing

Language Testing 2008 25 (3) 327–347

© 2008 SAGE Publications (Los Angeles, London, New Delhi and Singapore) DOI:10.1177/0265532208090156

Textbook trends in teaching language testingAlan Davies University of Edinburgh, UK

The article examines changes in language testing textbooks in English sinceLado (1961) and proposes that two trends may be discerned. The first showshow the growing professionalism of the field has required an expansion inteaching materials to meet the need for new training programmes. What theexpansion also shows is the desire, again a mark of increasing professional-ism, to provide all teaching resources from within the profession so that forneeded skills (e.g. statistics and measurement) it is now less necessary toappeal to outsiders such as statisticians and psychometricians. The secondtrend explains the need for the profession to expand its view of the skillsneeded by its members. From Lado onwards, skills were always conjoinedwith knowledge about language and about testing. More recently, the profes-sion has explicitly declared a concern for principles with regard, for example,to validity and to ethics. The increasing professionalism comes at a cost: thatcost is twofold: in-housing all resources means that language testers areincreasingly insulated from other potentially rewarding disciplines. And thecomplete resource offerings in the later teaching materials means that studentsmay be denied empirical encounters with real language learners, spendingall (or much of) their training within the resource material. The article alsoquestions how far research has informed the changes in training materials.

Keywords: informed by research, knowledge, language testing textbooks,practical manuals, principles, professionalism, skills, teachers’ resources

In writing about the teaching of language testing, we can make useof any of the materials (printed, audio, video, DVD, etc.) that havebeen developed. But it will not be very helpful to do so, first becauseour critique becomes an indiscriminate survey of the literature, and,second, because teaching ceases to be a deliberate proactive presen-tation and becomes an exposure to the whole field. It is more useful,both for our understanding of teaching and in order to put limits on

Address for correspondence: Alan Davies, Linguistics and English Language, School ofPhilosophy, Psychology and Language Sciences, University of Edinburgh, Adam FergusonBuilding, George Square, Edinburgh EH8 9LL, UK; email: [email protected]

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 3: Davies Textbook Trends in Teaching Language Testing

328 Textbook trends in teaching language testing

the material discussed in this paper, to consider teaching as deliber-ate pedagogy.

By ‘deliberate pedagogy’, I mean the work that teachers do intheir professional pursuit of teaching: they plan and organize theirarea of expertise, which may be a language, a science or, in our case,language testing, in order to facilitate learning.

When I look back over the last 50 years two trends may be dis-cerned. The first trend charts the growing professionalism and expan-sion of the field alongside the attempt to develop all-in material,thereby relieving the student of the need in the teaching context to drawon material outside the textbook. As we shall see, psychometric issuesare still very important today – but see Fulcher and Davidson (2007).

The second trend reveals the move from the skills � knowledgeapproach to the current attempt to take account also of principles.Skills provide the training in necessary and appropriate methodology,including item writing, statistics, test analysis and increasingly software programmes for test delivery, analysis and reportage.Knowledge offers relevant background in measurement and languagedescription, as well as in context setting, and may involve an exam-ination of different models of language learning, of language teach-ing and of language testing such as communicative language testing,performance testing and nowadays, socio-cultural theory. Principlesconcern the proper use of language tests, their fairness and impact,including questions of ethics and professionalism, thus a consider-ation of the growing professionalism of language testing, of theresponsibilities of language testers and of the impact of their work ona range of stakeholders and of the ethical choices they must make. Inwhat follows, I reflect on key publications over the period and laterreturn to a consideration of a selection from those key publications ofrepresentative texts in terms of the two trends I have adumbrated.These representative texts are British, American and Australian andthey span the whole period under discussion. They are not intendedto be any more celebrated than any of the others referred to but areselected as representative largely because they illustrate my argumentof the move over these 40� years from skills � knowledge to aknowledge-informed skills and then to a principles-informed skills.

I First trend: Expansion

A three-way distinction can be made of materials produced for teach-ers. At the most discursive end we have (1) teachers’ resources,

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 4: Davies Textbook Trends in Teaching Language Testing

Alan Davies 329

including books, videos, DVD and computer software. These providea library for teachers, there to inform them and be made available,where appropriate, to their students. Next are (2) textbooks whichprovide a deliberately pedagogic approach, again aimed largely atteachers and intended to help them professionally. Then at the how-toend we have (3) practical manuals. Sometimes two of these elementsmay be combined. Robert Lado, whom Wood calls ‘luminary’ (Wood,1991, p. 238) gave language testing early credibility. It is instructiveto consult the list of references in Language Testing (Lado, 1961). Heis even-handed: he cites such influential psychometric texts asAnastasi (1954/1961), Cronbach (1949/1961) and Buros (1959), aswell as significant linguistic texts (Bloomfield, 1933; Gleason, 1955;Hockett, 1958; Sapir, 1921; Fries, 1945). There are no references toany other language testing authors and Lado’s linguistics referencesare all to theoretical and descriptive linguists, not to applied linguists.It is as though Lado is making his contribution to establishing thefield by maintaining that applied linguistics needs language and thatlanguage testing needs applied linguistics � measurement.

Lado’s book is thus firmly in the middle of my three-way distri-bution, among the textbooks. Lado introduces his book thus: ‘a comprehensive introduction to the construction and use of foreignlanguage tests. It incorporates modern linguistic knowledge into lan-guage testing as one of its chief contributions. … The material is pri-marily intended for teachers of foreign languages and of English asa foreign language’ (Lado, 1961, p. vii). While the book may be atextbook, in my use of the term here, there are some parts of the bookwhich approximate a practical manual, notably Part 2: ‘Testing theelements of language’.

There are those who do not value Lado’s contribution to languagetesting. But that is unjust. The book is a triumph of combining issues.McNamara, 40 years later, writes ‘his recommendations about testingdominated practice for nearly twenty years and are still influential inpowerful tests such as TOEFL’ (McNamara, 2000, p. 89). BernardSpolsky calls Lado’s 1961 volume ‘a pioneering book’ (Spolsky,1995, p. 353). He praises Lado’s work thus: ‘Lado’s explicit appeal totheory was a crucial step to the professionalization of the field. WithLado, and with the students and colleagues he gathered (in the 1950s)at Michigan, like Harris and Palmer the language testing professionhad taken a major first step’ (Spolsky, 1995, p. 150). And he urgesus to remember our pioneers, such as Lado: ‘Our field has beenremarkably ahistorical: we have too often satisfied ourselves withpatricidal fury on a named or unnamed predecessor before launching

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 5: Davies Textbook Trends in Teaching Language Testing

330 Textbook trends in teaching language testing

ourselves into our own rediscovery of a slightly circular wheel of ourown’ (Spolsky, 1995, p. 352). No later publication comes near thebreadth of Lado (1961), until perhaps Fulcher and Davidson (2007).What Lado was keenly aware of was that language teachers needto know about language as well as about language testing. Lado’ssuccessors have been less concerned with providing knowledge aboutlanguage, perhaps because in the last half century applied linguisticshas been more widely available. Here is part of Lado’s commentary:‘As language yields its secrets to linguistic analysis, lexicographicstudy, and quantitative research, it is more and more feasible to definespecifically the task of learning a foreign language. As we identifymore precisely the elements and patterns to be acquired by the speak-ers of a language in learning another, we will be able to test more pre-cisely the progress made by the student under given conditions’(Lado,1961, pp. 338–339). Lado may have been over-optimistic about thefuture of science, but his understanding of what was needed was just.

Later in the 1960s, Harris (1969) and Valette (1967, 2nd ed., 1977)followed on Lado’s example, Harris for ESL, Valette for modern for-eign languages, amplifying Lado in a specialist area and at the sametime dependent on him. Valette (1967, p. v) claimed that her inten-tion was ‘to introduce teachers to a diversity of testing techniques’,while Harris (1969) offered his book as ‘a short concise text on thetesting of ESL, a subject about which both classroom teachers andtrainers of teachers have shown an increasing concern’ (Harris, 1969,p. vii). Like Valette, Harris modelled himself on Lado, thus combin-ing an analytic approach to language and its uses in such sections as‘what is meant by reading comprehension’, ‘what is meant by writ-ing’, ‘what is meant by speaking a second language’ along with dis-cussion of test characteristics (reliability, validity, practicality), testconstruction, test administration, analysis of test results and followedby a separate section on the statistics needed to complete the task.Valette is narrower, understandably so since she offers a range of dif-ferent language examples, but in the main both the Harris and theValette are largely concerned with ‘how to’ develop tests, with Harrisgoing further in how to analyse the results.

While Lado combined the resource and the textbook and Valetteand Harris the textbook and the practical manual, all three primarilyoffer a textbook approach. CAL (1961) and Davies (1968), in theirprovision of historical accounts and views of language testing issues,provided texts which were resource-based and as such offered infor-mation and ideas to teachers and graduate students which could befollowed up. Applied linguistics in the 1960s was still in its early

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 6: Davies Textbook Trends in Teaching Language Testing

Alan Davies 331

days and therefore language teachers (who might or might not fol-low graduate courses in applied linguistics at some point) were thenecessary audience for applied linguistics developments.

In his important CAL paper, J. B. Carroll (1961) does not attemptto offer a language testing blueprint. Instead, he sets out to ‘readdressthe attention of the audience to certain basic and fundamental prob-lems and points of view, some of which may have been lost sight ofin the heat of enthusiasm for technical detail’ (Carroll, 1961, p. 31).And in his edited volume, Davies (1968) offers a range of views thatattempt to bring together the ‘three strands of language testing: lan-guage, learning and evaluation’ (Davies, 1968, p. 1), examining ‘thebasic disciplines and their relevance to language testing … uses andtypes of test … the influence of tests on education … the item analy-sis needed’ (Davies, 1968, p. 13). Again, as with the Carroll paper,the focus here is that of resource (with a glance at textbook material).Whether we locate them as resource materials or as textbooks, itseems the case that while the Lado and the Valette and the Harriscombine the what with the how to, with Valette and Harris more onthe how to side, the Carroll and the Davies are both very much onthe what side.

Through the 1960s and the 1970s, with the publication of the text-books of Clark (1972) and Allen and Davies (1977), along with thepublication of the Peace Corps’ Manual of language testing (pub-lished later as Anderson 1993), a deliberately practical field-guide,there was always the recognition that while these materials providedtextbooks and practical manuals, supporting psychometric and stat-istical back-ups were necessary. These were not language or appliedlinguistics specific but were generic to all testing such as Cronbach(1949), Anastasi (1954) and so on. And for statistical work therewere generic programmes such as SPSS.

The Edinburgh course in applied linguistics appeared in four vol-umes between 1973 and 1977 (Allen & Corder, 1973, 1974, 1975;Allen & Davies, 1977). Volume 4 (Allen & Davies, 1977), with thetitle Testing and experimental methods, argued that ideas in appliedlinguistics needed to be submitted to the rigour of hypothesis andexperimentation. Experiments, it was suggested, need tests whiletests are themselves kinds of experiment: ‘This book is an attempt todemonstrate our belief in the importance of this link’ (Allen &Davies, 1977, p. 10).

After the Introduction by Davies, two chapters (by Davies &Ingram) were devoted to testing, two (by Ruth Clark) to experimen-tal design and computation and one to statistical inference. There

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 7: Davies Textbook Trends in Teaching Language Testing

332 Textbook trends in teaching language testing

were also two appendices on statistical inference and tables. Practicalwork was provided at the end of each chapter. Ingram (1977) con-tributed a theoretical chapter on ‘Basic concepts in testing’, whileDavies contributed a more practical chapter on ‘The construction oflanguage tests’ (Davies, 1977). The book fits very neatly into our sec-ond (text-book) category. The emphasis throughout is very much onthe connection between skills and knowledge, both measurementknowledge and language knowledge being presented as part of theskills that the language tester (and researcher) needed to acquire. Thisvolume, The Edinburgh course in applied linguistics, Volume 4, wasa serious attempt to locate language testing firmly within applied lin-guistics and was possibly the first such attempt. All four volumeswere much used over the subsequent 10 years to teach applied lin-guistics and, in the case of Volume 4, to teach both language testingand research methodology.

In the 1980s, we see both an expansion and an enriching in lan-guage testing publications. This development was paralleled in otherareas of applied linguistics, an increasing number of research spe-cialists in fields such as second language acquisition and discourseanalysis expanded their research base and, necessarily, their teachingprovision to take account of the expansion. Thus in language testingwe see the explosion of communicative language teaching withCarroll and Hall’s (1985) teachers’ guide alongside a growth in moregeneral textbooks such as Madsen (1983), Hughes (1989) and theearlier Heaton (1975). B. J. Carroll (1985) promoted communicativelanguage testing procedures. ‘The purpose of this book’ Carrollmaintained ‘is to outline principles and techniques for specifying thecommunicative needs of a language learner and for assessing his lan-guage performance in terms of those needs’ (Carroll, 1985, p. 5).With hindsight it would be more appropriate to place Carroll’sTesting Communicative Performance in our skills � knowledge cat-egory. Carroll’s principles are more properly considered knowledgein my sense, with the proviso that this knowledge is quite ideologic-ally driven.

What we also see in the teaching programmes at graduate level isthat empirical work is still part of the core requirement so that stu-dents following language testing courses were required to carry outa small-scale testing project. For this they relied on the growingnumber of textbooks which were beginning to relate the necessaryresearch and analytic techniques to language and applied linguistics.Among these were Hatch and Farhady (1982), followed in a secondedition by Hatch and Lazaraton (1991) for statistics and research

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 8: Davies Textbook Trends in Teaching Language Testing

Alan Davies 333

design and the Henning (1987) which began to make IRT techniquesmeaningful to the field of language testing and applied linguistics.The 1980s also saw the start of the new journal Language Testing,a sure sign of the field’s growing research capacity. This journal wasvery deliberately not a teaching outlet.

In the 1990s we see the normal academic development of anemerging discipline, now maturing and showing that maturity bypublishing research monographs, regular surveys of the field(Davies, 1982; Skehan, 1988, 1989; Alderson & Banerjee, 2001,2002), both its development and its future trends. These develop-ments increased the coming together of the contributing disciplinesso that Bachman (1990) and Bachman and Palmer (1996) broughttogether research design, statistics, computer programmes, testpreparation and analyses, while Davies (1990) and Wood (1991)offered critiques of language testing which can at best be regarded asresource materials for teaching, but are not easily put directly to usein a training programme.

Since so much writing about language testing, up until the 1990s,and perhaps even today, concerns large-scale testing, Genesee andUpshur (1996) was very much to be welcomed, dealing as it did withthe very real, and very difficult context of classroom assessment.Genesee and Upshur termed their book ‘practical’ and it belongs tothe practical manual end of my textbook category, concerned primar-ily with skills and offering the knowledge necessary for employingthose skills but less concerned with principles.

As the field of language testing has grown, as courses have de-veloped, at the undergraduate and graduate as well as at the PhD level,and as those courses have specialized so that now there are a numberof master’s degrees in language testing itself, while formerly thesewere normally part of degrees in applied linguistics or in applied lan-guage studies, TESOL, and so forth, different teaching needs haveshown themselves. Extended resources deliberately designed forteaching are represented by the publication of self-help teachingmaterials in the shape of the University of Melbourne video seriesMark My Words (1997) and the ILTA Web-based interviews on lan-guage testing Video FAQs (Fulcher & Thrasher, 1999, 2000). A paral-lel development is represented by the publication of the Dictionary oflanguage testing (Davies, Brown, Elder, Hill, Lumley & McNamara,1999) and the Encyclopedic dictionary of language testing (Mousavi,2002). Maturing disciplines display and develop their maturitythrough the development of teaching materials, such as the videos andthrough the defining descriptive work of specific dictionaries. This

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 9: Davies Textbook Trends in Teaching Language Testing

descriptive work provides the self-help teaching resources that teach-ers and students rely on. Davies et al. (1999) labelled their Dictionaryof Language Testing a segmental dictionary: ‘as such, it retains itsprofessional/vocational/registral association and at the same time itsnormative/pedagogic purpose’ (Davies, 1996, p. 231).

A more traditional development, also in the 2000s, is representedby the Cambridge University Press (CUP) Language Assessmentseries which, beginning in 1999, has to date published 10 volumes.Seven are of particular relevance (Douglas, 1999; Alderson, 2000;Read, 2000; Buck, 2001; Cushing, 2002; Purpura, 2004; Luoma,2004) inasmuch as they replicate within their properly narrow con-fines the operation of the Bachman (1990) model of language test-ing. This series is deliberately pedagogic. The Series Editors’ Prefaceto Alderson (2000) concludes thus: ‘this book offers a principledapproach to the design, development and use of reading tests andthus exemplifies the purpose of this series to bring together theoryand research in applied linguistics in a way that is useful to languagetesting practitioners’ (Alderson, 2000, p. xi).

A separate but contemporary development may be found in thework of Pennycook (2001), Shohamy (2001), Hawkey (2006) andMcNamara and Roever (2006). Although none of these publicationsis a textbook, all are being widely used and excerpted in the teach-ing of language testing and in training programmes. Basing them-selves on a teleological foundation, on the judgement of test use(what Bachman & Palmer (1996) term ‘test usefulness’) and on aconcern for a professional attachment to ethics (Davies, 1997), theyall insist, in somewhat different ways, that test validity must takeaccount of how and where a test is used. Such critiques, based asthey are on an essentialist, relativist belief, may or may not be ten-able or indeed practical. But there is no doubt that their criticalattacks have penetrated into teaching programmes, giving pause tothe perhaps overly confident view that a language test is a languagetest, no matter where or for whom. This critical stand-off is linkedalso to the social constructivist critiques of positivist philosophies(Lantolf, 2000). In all cases, what we see is a genuine and worth-while attempt to reflect on what Shohamy (2001) calls the power oftests. Students and teachers who are working in and studying lan-guage testing need to know about these critiques so that they areaware that what they are involved in, language testing, has the poten-tial to harm, indeed destroy people, even though, of course, they maynot change what the students and teachers think or do.

334 Textbook trends in teaching language testing

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 10: Davies Textbook Trends in Teaching Language Testing

Alan Davies 335

II Second trend: Skills, knowledge and principles

The second trend reveals the move from the skills � knowledgeapproach to the current attempt to take account also of principles.Skills provide the training in necessary and appropriate method-ology, including item writing, statistics, test analysis and increa-singly software programmes for test delivery, analysis and reportage.Knowledge offers relevant background in measurement and lan-guage description as well as in context setting. Principles concernthe proper use of language tests, their fairness and impact, includingquestions of ethics and professionalism.

The movement over the last 40 odd years seems to be from skillsto skills � knowledge to skills � knowledge � principles. The trendis not consistent but overall seems to hold. We can argue as follows:what a new (applied) activity needs – quickly – is to disseminateskills. But it becomes apparent quite soon that skills are not sustain-able without knowledge since knowledge provides the context inwhich skills operate: if skills represent ‘how?’, then knowledge rep-resents ‘what?’. And then over time, as the activity becomes moreconfident and, as a profession, practising the activity grows, it isinevitable that the activity itself comes into question: externally – ofcourse, but that is an old critique (testing has always had its critics) –but now internally as the language testing professionals themselvesbegin to query their own professionalism, their ethical foundations.What then happens is that what had been a skill, such as item writ-ing, incorporates knowledge and so becomes skill � knowledgesince item writing requires understanding of the context and purposefor which the items are being written. Thus a test of LSP necessarilyrequires that item writers have the relevant knowledge of the lan-guage description of their area of special purpose. And further, asknowledge becomes more widely available in the profession, so theneed to explain, to justify and to judge becomes important. Thus theconcern for, let us say, validity moves from principles to knowledgeas validity itself takes on more than a concern to represent the idealdomain and becomes a recognition of the practical impact on the testin all its singular settings. In its turn, the new principles-informedknowledge is operationalized and incorporated into skills. Skills,meaning techniques and methodologies, on their own are no longerenough, skills � knowledge are inadequate without the addition ofprinciples. For teaching, as for learning, there is a need for careful bal-ancing of the practical (the skills) with the descriptive (the knowledge)

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 11: Davies Textbook Trends in Teaching Language Testing

336 Textbook trends in teaching language testing

and the theoretical (the principles). All are necessary but one withoutthe other(s) is likely to be misunderstood and/or trivialized.

The survey of language testing courses, reported in Bailey andBrown (1996), has now been updated for this volume by Brown andBailey who find that in the 10 years since their 1996 survey little haschanged apart from the choice of textbooks. They report ‘the pres-ence of a stable knowledge base that is evolving and expandingrather than shifting radically’ (Brown & Bailey, this volume: p. 371).In 1996, Bailey and Brown reported that ‘there is a great deal ofdiversity in the sorts of language testing preparation provided toteachers’ (Bailey & Brown, 1996, p. 250). This diversity is revealedin the list of required and optional textbooks supplied to them by the84 language testing teachers who returned their questionnaires.Bailey and Brown list 32 textbooks, half of which were listed byonly one respondent. The most common textbooks were as follows:

Henning, 1987Madsen, 1983Hughes, 1989Bachman, 1990Oller, 1979Shohamy, 1985

Bailey and Brown (1996, p. 247) comment that ‘there is a wide rangeof emphasis, from the very theoretical to the very practical in theassessment preparation language teachers receive’. However, of thesix textbooks listed above, the four most commonly used, (Henning,Hughes, Bachman and Oller) were very much on the theoretical side.It appears that there was a widely held view that a language testingtextbook should be inclusive, combining knowledge and skills. Thehigh ranking given to Oller (1979) confirms that choice of theoryplus practical textbook; indeed, we might speculate that the attrac-tion of the Oller is that it offered not just knowledge and skills butalso broached principles in the discussion of the nature of validity:‘What is the ultimate criterion of validity for language tests?’(Oller,1979, p. 404). No doubt Oller’s ideological adherence at that time tohis expectancy grammar and to the indivisibility hypothesis (or uni-factorial structure of language proficiency) may have made for aone-sided approach, but principles they are, nonetheless.

In their 2007 Survey, reported in this volume, Brown and Baileynoted that 29 textbooks were listed as in use, compared with the 32in the 1996 Survey. They write: ‘Interestingly, only six of the bookswere common to both studies and of those, four were in neweditions, while only two were in their original editions’ (Brown &Bailey, this volume: p. 371).

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 12: Davies Textbook Trends in Teaching Language Testing

Alan Davies 337

The six textbooks common to both surveys were as follows:Hughes (1989, 2002)Bachman (1990)Brown (new ed. 2005)Cohen (1994)Alderson, Clapham and Wall (1995)Bachman and Palmer (1996)

(There is, of course, a natural delay before a new textbook is takenup and an existing one laid down.) The 2007 list of frequency of useplaced five of these (Hughes, Bachman, Brown, Alderson et al.,Bachman and Palmer) at the head of its list. I noted above that of the sixmost commonly used textbooks listed in 1996 four were on the theor-etical side. These include the Hughes and the Bachman, the only text-books that appear as most commonly used in both lists. And two of thosemoving into the top position for the first time in 2007, the Alderson,Clapham and Wall and the Bachman and Palmer, also take up a theoret-ical approach, thereby combining, as I have argued, knowledge andskills. This combination of knowledge and skills is, it would appear,more likely to endure than the somewhat ephemeral practical manuals.

I now consider a small number of celebrated, perhaps iconic textspublished over the last 50 years, to illustrate what I regard as the skills,knowledge and principles trend in the concept of the teaching oflanguage testing. What I propose is, as foreshadowed earlier, that overthis period there has been an expansion from skills to skills �knowledge and then to skills � knowledge � principles. My illus-trative texts are:

Lado (1961)Allen and Davies (1977)Hughes (1989)Bachman (1990)Alderson, Clapham and Wall (1995)Bachman and Palmer (1996)Mark My Words (1997), the ILTA Video FAQs (Fulcher & Thrasher, 1999,2000), the Dictionary of language testing (Davies et al., 1999) – these threetaken togetherMcNamara (2000)Davidson & Lynch (2002)Weir (2005)McNamara and Roever (2006)Fulcher and Davidson (2007)

While there are indeed texts that deal entirely or perhaps mainlywith skills (for instance, Madsen, 1983; Carroll & Hall, 1985;Heaton, 1975), all the examples I want to discuss are more compre-hensive. Thus Lado (1961), Allen and Davies (1977), Hughes (1989)deal with both skills and knowledge.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 13: Davies Textbook Trends in Teaching Language Testing

338 Textbook trends in teaching language testing

Lado (1961) begins his book with knowledge: his Part 1 consistsof discussions of language, language learning, language testing, vari-ables and strategy of language testing, and critical evaluation oftests. The remaining 90% of the book examines the skills needed fordeveloping tests and for experiments using language tests.

Similarly, Allen and Davies (1977) consider both skills � know-ledge with chapters on ‘Basic concepts in testing’ and on ‘The con-struction of language tests’. There are then two chapters onexperiments plus a further chapter (and appendices) on the meaningand working of statistics used in experiments and testing.

Hughes (1989, now in its second edition, 2003) again deals withthe background knowledge needed in language testing and with thestatistical and item writing skills. What is of interest to us in this dis-cussion is that Hughes’s second edition does not move beyond theskills � knowledge position he took up in 1989 in spite of the pres-sures exercised by discussions in the language testing community onprinciples. This may suggest that there is less demand among teach-ers, Hughes’s target audience, for principles than I had assumed.

Bachman and Palmer (1996) and Davidson and Lynch (2002) followmuch the same pattern. However, Bachman and Palmer’s examinationof the conceptual basis of test development introduces what they term‘test usefulness’, ‘a kind of metric by which we can evaluate not onlythe tests that we develop and use, but also all aspects of test develop-ment and use’ (Bachman & Palmer, 1996, p. 17). This formulationI consider to be an incorporation of skills � knowledge such that theknowledge of test development and use now becomes a learnt skill. Butthe main purpose of their book is, they contend, ‘to enable the reader tobecome competent in the design, development and use of languagetests’ (Bachman & Palmer, 1996, p. 3). That is its primary purpose andthat is why I place the book in the skills � knowledge category.

Davidson and Lynch’s book (2003) also belongs in this category.The subtitle is: ‘A teacher’s guide to writing and using language testspecifications’. Davidson and Lynch maintain that existing languagetesting textbooks assume knowledge of testing while they aim toprovide an introduction to newcomers, focussing on test specifica-tions. And so their book offers guidance on the skills needed to writetest specifications but also shows how the knowledge behind thosespecifications can be viewed as – and taught as – skills in their ownright.

Bachman (1990), the text on which Bachman and Palmer (1996) isbased (as well as the series Cambridge Language Assessment – seeabove), does not deal with the skills of item writing and test analysis.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 14: Davies Textbook Trends in Teaching Language Testing

Alan Davies 339

These matters are, after all, taken up in the later Bachman and Palmer(1996). But what Bachman does in his 1990 text is to treat knowledgeas a form of skill and at the same time to move on to begin an exam-ination of principles. Hence the discussion of validity as a unitaryconcept, of the evidential nature of validity and of the consequentialand ethical basis of validity. Such discussions look forward to theBachman and Palmer (1996) concept of test usefulness.

Alderson, Clapham and Wall (1995) is much more rounded andless practical. ‘Something we do not do in this book is to describelanguage testing techniques in detail’ (Alderson, Clapham & Wall,1995, pp. 2–3). What they do deal with is the examination of valid-ation in all its aspects. Indeed, short on techniques though this volumemay be, through its in-depth discussion of validity and standards itsscope as a textbook is as wide-ranging as that of Lado’s before themand that of Fulcher and Davidson (2007) over ten years later.Alderson, Clapham and Wall (1995) ground much of their discussionby reference to the work of the UK EFL examination boards. Thishas the advantage of realism and makes a powerful argument for thedifferent concerns of academic language testing on the one hand andpublic or institutional (or indeed commercial) language testing onthe other. It is distressing for academics to learn that: ‘not all boardsunderstand what is meant by validation, validity and reliability’(Alderson et al., 1995, p. 257). But the very context specificity of thebook and its strong critique – almost ideological – of the boards maydetract from the overall concern of the learner who is unlikely tohave the same critical view as the authors.

The two video projects, Mark My Words (1997) and the ILTAVideo FAQs (Fulcher & Thrasher, 1999, 2000) are not primarily con-cerned with skills. Both deal largely with knowledge. Thus, MarkMy Words has the following topics in its series:

Language proficiency assessmentPrinciples of test development (‘principles’ is used somewhatdifferently in the present article)Objective and subjective assessmentStages of test analysisPerformance assessmentClassroom-based assessment

In this video series, knowledge is presented not so much as back-ground as part of the necessary skills behaviour in developing lan-guage tests. The Dictionary of Language Testing (Davies et al.,1999) goes further by incorporating, as is the nature of dictionaries,

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 15: Davies Textbook Trends in Teaching Language Testing

340 Textbook trends in teaching language testing

topics such as ethics, ethicality and impact and thus taking someaccount of the principles of language testing.

Like Alderson, Clapham and Wall (1995), Weir (2005) is less con-cerned with skills than with knowledge and in particular with valid-ation. What he very carefully does is to explain that validationevidence is required to demonstrate validity: in other words, whileothers have shaped knowledge into a kind of skill, what Weir does isto convert principles into first knowledge and then into a skill. Nodoubt there is a case for retaining a separation between knowledgeand skills and between principles and skills so that implementingthem requires thought, not just automaticity. But for teaching pur-poses, which is our concern here, the demonstration of how to makeboth knowledge and principles operational, that is skill-like, is peda-gogically very appealing.

McNamara (2000) and McNamara and Roever (2006) both elabo-rate the knowledge needed for professional language testers and dealin some depth with principles. Thus, in both texts there is concernwith ethics and social policy and responsibilities: indeed, the wholeof McNamara and Roever (2006) is concerned, as the title indicates,with social issues. McNamara and Roever argue that ‘language test-ing is … ripe for a broader view of assessment and its social aspects(while) … testers need … to reflect on test use’ (McNamara &Roever, 2006, p. 8). If McNamara (2000) and McNamara and Roever(2006) are concerned wholly with knowledge and principles in sucha way that principles become part of the knowledge needed, Fulcherand Davidson (2007) is yet more all-embracing, offering in one vol-ume what Bachman (1990) and Bachman and Palmer (1996) offer intwo. Fulcher and Davidson (2007) has the title: Language testing andassessment: An advanced resource book and is part of an AppliedLinguistics series of resource books in different areas.

Fulcher and Davidson (2007, p. xix) consider that their discussion‘is set within a new approach that we believe brings together testingpractice, theory, ethics and philosophy. At the heart of our newapproach is the concept of effect-driven testing. This is a view of testvalidity that is highly pragmatic. Our emphasis is on the outcome oftesting activities’.

The integrative nature of their text, comprising:

A) Introduction: 10 units dealing with the central concepts of languagetesting and assessment

B) Extension: readings from books and articles linked to the conceptsintroduced in Section A

C) Exploration: extended activities building on both A and B

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 16: Davies Textbook Trends in Teaching Language Testing

Alan Davies 341

situates the learner within the language testing enterprise. Of all thetexts examined in this paper, Fulcher and Davidson (2007) doesseem to provide the most complete coverage of skills, knowledgeand principles.

The development proposed above can be summarized as follows: Inthe 1960s (and earlier) language testing relied on external sources,particularly psychometric (Cronbach, 1949; Anastasi, 1961; Tyler,1963; Anstey, 1966). From the 1970s and onwards, the attempt wasmade internally to nativize the necessary skills and knowledge – butin separate texts, thus Hatch and Farhady (1982) and Hatch andLazaraton (1991) dealing with statistics and research design; Shohamy(2001) and possibly the external Pennycook (2001) to handle criticalapproaches, the ILTA Code of Ethics (2000) presenting the profes-sion’s ethical principles, and Bachman (2004), again dealing with stat-istics. Meanwhile, we have the internal sequence discussed abovefrom Lado (1961) to Fulcher and Davidson (2007) moving graduallybeyond the skills � knowledge scenario to the skills � knowledge �principles combination. We present this array in Table 1.

III Conclusion

The development in teaching materials examined in this paper comesas a result of the increasing professionalism of the field of languagetesting. That increasing professionalism has a cost: that cost is two-fold: in-housing all resources means that language testers areincreasingly excluded from other potentially rewarding disciplines.And the complete resource offerings in the later teaching materialsmeans that students are over-protected from exposure to empiricalencounters with real language learners, spending all (or much of)their training within the resource material.

This exclusion from external influences leads to an insularity, a reluctance to take up new ideas, as McNamara and Roever (2006)argue. They remind us that in spite of the social turn in the last twodecades the teaching of language testing is still largely psychomet-ric: ‘In terms of academic training, we stress the importance of awell-rounded training for language testers that goes beyond appliedpsychometrics … a training that includes a critical view of testingand social consequences’ (McNamara & Roever, 2006, p. 255).

A similar resistance may explain the reluctance to grapple withrecent research findings. Wood’s (1991) is an extreme view: ‘it isclear that innovation is not driven by research … (but) … it is import-ant to understand how (innovations) happened, and whether they

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 17: Davies Textbook Trends in Teaching Language Testing

342 Textbook trends in teaching language testing

were actually necessary, if only to appreciate how marginal the partresearch evidence plays in the decision’ (Wood, 1991, p. 248).Wood’sscepticism about the influence of research relates to innovation: ‘it isclear that innovation is not driven by research’, a view he exem-plifies in his comments on the English Language Testing Service test(Wood, 1991, pp. 235–236). In this paper, I have not, in any directway, discussed to what extent language testing textbooks make useof language testing research. This omission is deliberate. It is not, ofcourse, that there is a dearth of research in language testing: on thecontrary there is a great deal, reported in the journals such asLanguage Testing and Language Assessment Quarterly, in the encyc-lopedias such as Shohamy and Hornberger (2008) and Hinkel(2005), in the many monographs, notably the Alderson and Bachmanseries (see the reference above, p. 334), to the CUP LanguageAssessment Series: 2000–2004) and in the regular reports of

Table 1 Changes in English language testing textbooks

External Internal Internal Skills Knowledge Principlesseparate combined

1960s: Cronbach; Lado x xAnastasi;Tyler;Anstey

1970s: Hatch & Allen & x xFarhady; DaviesHatch & Lazaraton

Hughes x xBachman; x xAlderson, x xClapham & WallBachman & x xPalmerMark My ? x ?Words

2000s: Code of x x xEthicsShohamy McNamara x x

Pennycook Davidson & x xLynch; Weir

Bachman McNamara & x xRoeverFulcher & x x xDavidson

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 18: Davies Textbook Trends in Teaching Language Testing

Alan Davies 343

Cambridge ESOL and Educational Testing Service on TOEFL. Butwhile a textbook is properly informed by research, its primary pur-pose is not, as is that of a monograph, to report recent research.Textbooks consolidate while monographs are dynamic, reportingdevelopments in current research. There is an inevitable gap, a timelag between the publication of research and its incorporation in atextbook, by which time there may be a very different research need,as, for example, McNamara (2005), Rea-Dickins (2008) and Leungand Lewkowicz (2008) note, ‘in their references to classroom assess-ment’. McNamara’s labelling of this gap as ‘unbridgeable’(McNamara, 2005, p. 778) is apt and mistaken. Yes, there is a gapbut it is a necessary gap. Worthwhile training needs to be informedby mature understanding of research and not by the latest news fromthe PhD and the research project.

IV References

Alderson, C. (2000). Assessing reading. Cambridge: Cambridge UniversityPress.

Alderson, C. & Banerjee, J. (2001). Language testing and assessment (SurveyArticle). Language Teaching, 34, 213–236.

Alderson, C. & Banerjee, J. (2002) Language testing and assessment (SurveyArticle). Language Teaching, 35, 79–113.

Alderson, C., Clapham, C. & Wall, D. (1995). Language test constructionand evaluation. Cambridge: Cambridge University Press.

Alderson, C. & Bachman, L. F. (Eds.). (2000–2005). The Cambridge lan-guage assessment series. Cambridge: Cambridge University Press.

Allen, P. & Davies, A. (Eds.). (1977). Testing and experimental methods:Volume 4. The Edinburgh course in applied linguistics. London: OxfordUniversity Press.

Allen, P. & Corder, S. P. (Eds.). (1973). Readings for applied linguistics:Volume 1. The Edinburgh course in applied linguistics. London: OxfordUniversity Press.

Allen, P. & Corder, S. P. (Eds.). (1974). Techniques in applied linguistics:Volume 3. The Edinburgh course in applied linguistics. London: OxfordUniversity Press.

Allen, P. & Corder, S. Pit (Eds.). (1975). Papers in applied linguistics: Volume2. The Edinburgh course in applied linguistics. London: OxfordUniversity Press.

Anastasi, A. (1954). Psychological testing (1st ed.). New York: Macmillan.Anastasi, A. (1961). Psychological testing (2nd ed.). New York: Macmillan.Anderson, N. (1993). Handbook for classroom teachers in Peace Corps

language programs. Manual. Washington, DC: The Peace Corps.Anstey, E. (1966). Psychological tests. London: Macmillan.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 19: Davies Textbook Trends in Teaching Language Testing

344 Textbook trends in teaching language testing

Bachman, L. (1990). Fundamental considerations in language testing.Oxford: Oxford University Press.

Bachman, L. (2004). Statistical analyses for language assessment.Cambridge: Cambridge University Press (also with Antony Kunnan:Handbook and CD).

Bachman, L. & Palmer, A. (1996). Language testing in practice. Oxford:Oxford University Press.

Bailey, K. M. & Brown, J. D. (1996). Language testing courses: What are they? In A. Cumming & R. Berwick (Eds.), Validation in languagetesting (pp. 236–256). Clevedon: Multilingual Matters.

Bloomfield, L. (1933). Language. New York: Henry Holt.Brown, J. D. & Bailey, K. M. (2008). Language testing courses: What are they

in 2007? Language Testing (this issue).Buck, G. (2001). Assessing listening. Cambridge: Cambridge University Press.Buros, O. (1959). Fifth mental measurements yearbook. Highland Park, NJ:

Gryphon Press.CAL (Center for Applied Linguistics) (1961). Testing the English proficiency

of foreign students. Washington, DC: CAL.Carroll, B. J. (1985). Testing communicative performance. Oxford: Pergamon.Carroll, B. J. & Hall, P. (1985). Make your own language tests: A practical

guide to writing language performance tests. Oxford: Pergamon.Carroll, B. J. (1961). Fundamental considerations in testing for English

Language proficiency of foreign students. In CAL Testing the Englishproficiency of foreign students (pp. 30–40). Washington, DC: Center forApplied Linguistics.

Clark, J. (1972). Foreign language testing: Theory and practice. Philadelphia,PA: Center for Curriculum Inc.

Cobuild (1987). Collins Cobuild English language dictionary. London: Collins.COE (2000). ILTA Code of Ethics. http://www.iltaonline.com/code.pdfCohen, A. D. (1994) Assessing language ability in the classroom (2nd ed.).

New York: Heinle and Heinle.Cronbach, L. (1949). Essentials of psychological testing (1st ed.). New York:

Harper and Row International.Cronbach, L. (1961). Essentials of psychological testing (2nd ed.). New York:

Harper and Row International.Cumming, A. & Berwick, R. (Eds.). (1996). Validation in language testing.

Clevedon: Multilingual Matters.Cushing, S. W. (2002). Assessing writing. Cambridge: Cambridge University

Press.Davidson, F. & Lynch, B. (2002). Testcraft: A teacher’s guide to writing and

using language test specifications. New Haven, CT and London: YaleUniversity Press.

Davies, A. (Ed.). (1968). Language testing symposium: A psycholinguisticapproach. London: Oxford University Press.

Davies, A. (1977). The construction of language tests. In P. Allen & A. Davies(Eds.), Testing and experimental methods. Volume 4. Edinburgh course inapplied linguistics (pp. 38–104). London: Oxford University Press.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 20: Davies Textbook Trends in Teaching Language Testing

Alan Davies 345

Davies, A. (1982). Language testing survey. Parts 1 and 2. In V. Kinsella (Ed.),Surveys (pp. 127–159). Cambridge: Cambridge University Press.

Davies, A. (1990). Principles of language testing. Oxford: Oxford UniversityPress.

Davies, A. (1996). The role of the segmental dictionary in professional valid-ation: Constructing a dictionary of language testing. In A. Cumming andR. Berwick (Eds.), Validation in language testing (pp. 222–235).Clevedon: Multilingual Matters.

Davies, A. (1997). Demands of being professional in language testing.Language Testing, 14(3), 328–339.

Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T.(1999). Dictionary of language testing. Cambridge: CambridgeUniversity Press and UCLES.

Douglas, D. (1999). Assessing languages for specific purposes. Cambridge:Cambridge University Press.

Fries, C. (1945). Teaching and learning English as a foreign language. AnnArbor, MI: University of Michigan Press.

Fulcher, G. & Davidson, F. (2007). Language testing and assessment: Anadvanced resource book. London: Routledge.

Fulcher, G. & Thrasher, R. (1999/2000). Video FAQs. Introducing topics inlanguage testing. ILTA (online) http://www.le.ac.uk/education/ilta/faqs/main.html

Genesee, F. & Upshur, J. (1996). Classroom evaluation in second languageeducation. Cambridge: Cambridge University Press.

Gleason, H. (1955). An introduction to descriptive linguistics. New York:Henry Holt.

Harris, D. (1969). Testing English as a second language. New York: McGraw Hill.Hatch, E. & Lazaraton, A. (1991). The research manual: Design and statis-

tics for applied linguistics. New York: Newbury House.Hatch, E. & Farhady, H. (1982). Research design and statistics for applied

linguistics. New York: Newbury House.Hawkey, R. (2006). Impact theory and practice. Studies in language testing.

Cambridge: Cambridge University Press and UCLES.Heaton, B. (1975). Writing English language tests. London: Longman.Henning, G. (1987). A guide to language testing. Cambridge, MA: Newbury

House.Hinkel, E. (Ed.). (2005). Handbook of research in second language teaching

and learning. Mahwah, NJ: Lawrence Erlbaum.Hockett, C. (1958). A course in modern linguistics. New York: Macmillan.Hughes, A. (1989). Testing for language teachers (1st ed.). Cambridge:

Cambridge University Press.Hughes, A. (2003). Testing for language teachers (2nd ed.). Cambridge:

Cambridge University Press.Ingram, E. (1977). Basic concepts in testing. In J. P. B. Allen & A. Davies

(Eds.), Testing and experimental methods: Volume 4. Edinburgh coursein applied linguistics (pp. 11–37). London: Oxford University Press.

Lado, R. (1961)Language testing. London: Longmans.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 21: Davies Textbook Trends in Teaching Language Testing

346 Textbook trends in teaching language testing

Lantolf, J. (Ed.). (2000). Sociocultural theory and second language learning.Oxford: Oxford University Press.

Leung, C. & Lewkowicz, J. (2008). Assessing second/additional language ofdiverse populations. In E. Shohamy & N. H. Hornberger (Eds.),Encyclopedia of language and education: Volume 7. Language testingand assessment (2nd ed.), (pp. 301–317). New York: Springer Science &Business Media.

Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge UniversityPress.

Madsen, H. (1983). Techniques in testing. New York and Oxford: OxfordUniversity Press.

Mark My Words (1997). Video Series. Melbourne: University of MelbourneLanguage Testing Research Centre.

McNamara, T. (2000). Language testing. Oxford: Oxford University Press.McNamara, T. (2005). Introduction to Part V1: Second language testing and

assessment. In E. Hinkel. (Ed.), Handbook of research in second lan-guage teaching and learning (pp.775–778). Mahwah, NJ: LawrenceErlbaum.

McNamara, T. & Roever, C. (2006). Language testing: The social dimension.Malden, MA and Oxford: Blackwell.

Mousavi, A. (2002). An encyclopedic dictionary of language testing (3rd ed.).Taiwan: Tung Hua.

Oller, J. W., Jr. (1979). Language tests at school. London: Longman.Pennycook, A. (2001). Critical applied linguistics: A critical introduction.

Mahwah, NJ: Lawrence Erlbaum.Purpura, J. (2004). Assessing grammar. Cambridge: Cambridge University

Press.Read, J. (2000). Assessing vocabulary. Cambridge: Cambridge University

Press.Rea-Dickins, P. (2008). Classroom-based language assessment. In

E. Shohamy & N. H. Hornberger (Eds.), Encyclopedia of language andeducation: Volume 7. Language testing and assessment (2nd ed.). (pp. 257–271). New York: Springer Science & Business Media.

Sapir, E. (1921). Language: An introduction to the study of speech. New York:Harcourt Brace.

Shohamy, E. (1985). A Practical handbook in language testing for the secondlanguage teacher. Shaked: Ramat Aviv, Israel.

Shohamy, E. (2001). The power of tests: A critical perspective on the uses oflanguage tests. Harlow: Pearson.

Shohamy, E. & Hornberger, N. H. (Eds.). (2008) Encyclopedia of languageand education: Volume 7. Language testing and assessment (2nd ed.).New York: Springer Science & Business Media.

Skehan, P. (1988). Language Testing: Part 1. Language Teaching, 21(4),211–221.

Skehan, P. (1989). Language testing: Part 2. Language Teaching, 22(1), 1–13.Spolsky, B. (1995). Measured words. Oxford: Oxford University Press.Tyler, L. (1963). Tests and measurements. Englewood Cliffs, NJ: Prentice-Hall.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from

Page 22: Davies Textbook Trends in Teaching Language Testing

Alan Davies 347

Valette, R. (1967). Modern language tests: A handbook (1st ed.). New York:Harcourt Brace and World.

Valette, R. (1977). Modern language tests: A handbook (2nd ed.). New York:Harcourt Brace and World.

Weir, C. (2005). Language testing and validation: An evidence basedapproach. Houndmills, Basingstoke: Palgrave Macmillan.

Wood, R. (1991). Assessment and testing. Cambridge: Cambridge UniversityPress.

by Gilberto Berrios on April 19, 2009 http://ltj.sagepub.comDownloaded from


Top Related