issues and methods in the development of language assessment tools for bilingual welsh-english...

Issues and methods in the development of language assessment tools for bilingual

Welsh-English speakersEnlli Môn Thomas

Prifysgol Bangor University

ASN, Edinburgh 23 June, 2014

Structure of the talk• Background issues relating to bilingual language

development– Issues for minority and majority language assessment

• The development of receptive vocabulary tests– Prawf Geirfa Cymraeg, Fersiwn 7-11 (Gathercole & Thomas,

2007)– Prawf Geirfa Cymraeg, Fersiwn 11-15 (Gathercole, Thomas,

Hughes, Roberts & Owen, forthcoming)

• The development of Welsh verbal categorisation and verbal analogy tasks– Cognitive Abilities Task 4, Welsh Adaptation (GL Assessment,

2014)

Issues relating to bilingualsPattern of acquisition is similar across bilinguals and

monolinguals (e.g., De Houwer, 2005; Pearson & Fernández, 1994)

…although some unique bilingual characteristics sounds

naming

[2 words]3+ words (sentences)

Issues relating to bilingualsHowever…

Rate of acquisition slower in bilinguals than in monolinguals:

– Vocabulary (e.g., Umbel, Pearson, Ferńandez & Oller, 1992; Pearson, Ferńandez & Oller, 1993,1995; Pearson & Ferńandez, 1994; Bialystok, Luk, Peets & Yang, 2010);

– expressive > receptive (Thordardottir, 2011)

– Morphosyntax (e.g., Thomas et al., 2014; Kupisch, 2004; Kupisch, Müller & Cantone, 2002)

Why?• Vocabulary knowledge is ‘distributed’ across bilinguals’ two

languages (so called ‘Distributed Characteristic’ – Oller, 2005)• domain specific• item-by-item

• For morphosyntax: takes longer for bilinguals to achieve the “critical mass” of exposure (Maratsos, 2000) in any of their languages that is necessary in order to be able to abstract regularities from the input and generalise patterns in a productive manner.

• Once “critical mass” is achieved, differences between bilinguals and monolinguals should /can disappear (e.g., Oller & Eilers, 2002).

Catch-up?• ‘Vocabulary gap’ more prolonged than grammatical

difference – e.g., Oller et al. 2007; Paradis & Genesee, 1996; but see e.g.,

Thordardottir, Rothenberg, Rivard, & naves (2006) for equal performance for both vocab and grammar for balanced bilinguals

• But…largely dependent on amount of input, particularly in the minority language (e.g., Thordardottir, 2011; Thordardottir et al., 2006; Thomas et al., 2014) and use of the language

• Bilingual ‘catch-up’ is quicker and more likely in dominant community languages (Gathercole & Thomas, 2009).

81.28

57.83

73.47 71.39

86.11 85.39

60.44

70.83

96 95.3 95.9 100 100 100 98.8 95

0102030405060708090

100

L1 Welsh childrenL1 Welsh adults

58.41

34.141

26.41

44.1 44.7235

44.83

81.573.35

78.8

96.692 96

89.697.5

0102030405060708090

100

+suff

+suff/V

~suff

~suff/V -su

ff-su

ff/V V

Suppletive

2L1 children2L1 adults

45.61

32.39 36.39

24.5734.35 32.78

20.48

39.43

74.364.6 66.8

76.582

100

71.6

83.1

0102030405060708090

100

+suff

+suff/V

~suff

~suff/V -su

ff-su

ff/V V

Suppletive

L2 Welsh childrenL2 Welsh adults

Thomas, E. M., Williams, N., Jones, Ll. A., Davies, S., & Binks, H. (2014). Acquiring complex structures under minority language conditions: Bilingual acquisition of plural morphology in Welsh. Bilingualism: Language and Cognition, 17 (3), 478-494.

Catch-up?However…

•Bialystok & Luk (2012): prolonged differences between ‘balanced’ adult bilinguals and monolinguals , even on receptive knowledge of English vocabulary (dominant community language)

•This suggests that acquisition of vocabulary in a dominant community language is not necessarily ‘automatic’

•Such findings have clear implications for bilingual education and assessment, as well as for theoretical approaches to language production and comprehension

Rhys, M. & Thomas, E. M. (2013). Bilingual Welsh–English children's acquisition of vocabulary and reading: implications for bilingual education. International Journal of Bilingual Education and Bilingualism, 16 (6), 633-656.

Bilingual Control

L1 Welsh L1 English 2L1 Monolingual

7/8 36 28 19 17

10/11 46 27 19 15

Total (N=207):

82 55 38 32

Controlled for SES (all Fs > .115, all p > .077) and NVIQ (F(3, 196)= 2.619, p= .052).

English reading

Home language (F(3,193)= 7.400, p= .000) & age (F(1, 193)= 12.065, p= .001). Mono > L1 Welsh (p= .006) , 2L1 (p= .017)L1 English > L1 Welsh (p= .007), 2L1 (p= .024).

80

85

90

95

100

105

110

7 to 8 10 to 11

Control

L1 English

2L1

L1 Welsh

Standard Scores (NARA-II)

English vocabularyStandard Scores (BPVS)

Home language (F(3, 196)= 17.827, p= .000).Mono & L1 English > L1 Welsh(p= .006) , 2L1 (p= .017)

70

75

80

85

90

95

100

7 to 8 10 to 11

Control

L1 English

2L1

L1 Welsh

Implications of past research findings for Celtic bilinguals

• Bilinguals should be tested in both languages

• Bilinguals should be compared to a meaningful norm in both tests – English: existing tasks may be appropriate to use with L1

English bilinguals, but not 2L1 or L1 Welsh/Scottish Gaelic– Need to create bilingual norms for existing English tasks– Welsh/Scottish Gaelic: need language specific tests, with

meaningful norms– English & Welsh/Scottish Gaelic: tests need to deal with

unique features of bilingual speech – e.g. code-switching

Initial considerations

• Each word native Welsh word

• No cognates/ borrowings

• Avoid dialectal words

• Wide range of frequencies and complexities (Cronfa Electroneg o’r Gymraeg, Ellis et al., 2001)

• Able to be expressed easily in pictorial form

240 words

30 words each per 8 frequencyLevels:

•600-1000+ per 1,000,000•200- 599 "•150 - 199 "•100-149 "•75-99 "•50-74 "•25-49 "•< 25 " "

Cronfa Electroneg o’r Gymraeg (Ellis et al., 2001)

AB

CCH

[Target =Pren “wood/wooden”]

AB

C CH

[Target =Peryglus “dangerous”]

AB

C CH

[Ymosod “to attack”]

A B

C CH

[Cuddio “to hide”]

A

B

C CH

[Gwag “empty”]

A B

C CH

[Llygru “to pollute”]

A B

C CH

[Dwys “solemn”]

7- to 11-year-olds:

Age OWH WEH OEH Tot.7 43 36 29 1088 45 60 55 1609 36 43 44 123

10 54 40 46 14011 36 17 27 80

Tot. 214 196 201 611

Criteria for assignment to bilingual ‘type’:

Language spoken by Parent 1 to child

Language spoken by Parent 2 to child OWH WEH OEH

only W only W xonly W mostly W x

mostly W mostly W xW and E W and E xonly W only E xonly W W and E x

mostly W W and E xonly E mostly W xonly E W and E x

mostly E mostly W xmostly E W and E x

only E only E xmostly E mostly E x

Norming the data

Variables:• Age

– 7-11• School Year

– higher/lower• Home Language

– OWH, WEH, OEH

• Performance on 111/240 words differed across the variables

Raw score

General standardised score

+Age/Year x Home

Language standardised score

75.0

73.3

81.6

77.8

89.2

83.3 85

.4

94.1 97

.682

.084.8

75.1

68.070

.6

66.2

61.4

69.6

77.5

88.4

78.5

72.0

71.3

68.370.2

58.259

.3

59.4

0

10

20

30

40

50

60

70

80

90

100

1107

- Yr

3

8 -

Yr 3

8 -

Yr 4

9 -

Yr 4

9 -

Yr 5

10 -

Yr 5

10 -

Yr

6

11 -

Yr

6

11 -

Yr

7

Age - Year Group by Home Language

Num

ber o

f Wor

ds

OWH WEH OEH

Mean Number Correct Responses by Age - Year and Home Language on 111 discriminating words

AGE 7

AllAge 7 - Yr 3

OWH WEH OEH

8586878890

85 9186 9287 9388 94

85 90 9586 91 9787 92 9888 93 9989 94 10090 95 10191 97 10292 85 98 10394 86 99 10595 87 100 10796 88 101 10897 90 102 10998 91 103 11099 92 105 111

100 93 107 112101 94 108 113102 95 109 114103 97 110 115104 98 111105 99 112106 100 113107 101 114109 102 115110 103

434445464748495051525354555657585960616263646566676869707172737475

Gathercole, V. C. Mueller, Thomas, E. M., Hughes, E. K., Roberts, E. J., & Hughes, C.(in press) . Prawf Geirfa Cymraeg: Fersiwn 11-15. Bangor: School of Psychology, Bangor University.

Discussed in Chapter 9 (pp.175-193):

Thomas, E. M., Gathercole, V. C. Mueller, & Hughes, E. K. (2013). Sociolinguistic Influences on the Linguistic Achievement of Bilinguals: Issues for the Assessment of Minority Language Competence. In V. C. Mueller Gathercole (ed.), Solutions for the Assessment of Bilinguals. Clevedon: Multilingual Matters.

Method

• 242 words tested in total• 212/242 same as in the 7-11 version • Additional 30 less frequent words added

30

Age (in years)

11 12 13 14 15 Total

Home language

Welsh 47 57 48 56 52 260

Welsh & English

35 69 45 49 41 239

English 31 54 54 77 52 268

Total 113 180 147 182 145 767

Results

• Performance on 125/242 was at ceiling (over 90% correct) or at baseline (under 25% correct)

• 3/242 words non-discriminatory• Final test includes 114 words

31

Results

32

7680

84 84

89

8176

7066

76

6664

6965

63

0

10

20

30

40

50

60

70

80

90

100

110

11 12 13 14 15

Age Group

Num

ber o

f Wor

ds

OWH WEH OEH

Norming the data11-13, OWH. WEH, OEH

14-15 OWH, WEH

Raw score


+Age/Year x Home


Norming the data14-15 OEH

Raw score


+Age/Year x Home


+ community language standardised score

Results: Home Language, Age and Community Type

• Quantitative– Number analogies– Number series

• Non-verbal– Figure classification– Figure matrices

• Spatial– Figure analysis– Figure recognition

• Verbal

Verbal task 1: Verbal Classification

Verbal task 2: Verbal Analogies

Development of verbal tasks in Welsh

• All items translated• 3 researchers read all

items• Items coded as

– acceptable for Welsh– Not acceptable for

Welsh• Researchers met to

discuss and agree on items to keep

• Project leader (me)• Team of Welsh-speaking

researchers (Bangor University)

• Translation services (Atebol) • Teachers• Data collectors (GL

Assessment)• Statistician (GL Assessment)• Original author


• Non-acceptable items coded further:– Problems with target– Problems with options

• Problems usually of the following types:– Item irregular word– Item dialectal word– Not enough expressions

from within same category in Welsh

– Duplication






• Adapting the items– Some thrown out– Some included item

changes– Some needed to be

created from new






• Researchers worked on developing new items and new full entries

• Meetings to discuss items

• Final version sent to translators and teachers

• Few queries• Final decisions made





Results

issues and methods in the development of language assessment tools for bilingual welsh-english...

Documents