bulats –a toolfor measuringand...
TRANSCRIPT
![Page 1: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/1.jpg)
Seite 1
BULATS – A TOOL FORMEASURING ANDCOMPARING PROGRESS IN LANGUAGE LEARNING?
KATHRIN HAHN, GOETHE-INSTITUT (ZENTRALE, BEREICH 44)
DR. WINFRIED KOCH, BDS KOCH
![Page 2: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/2.jpg)
Seite 2THE PROJECT TABLET-CLASSROOM16.04.2014
Goethe-Institut für Thema
• Pilotproject for the implementation of tablets(iPads) in the forgein language classroom,
• Multimedial equipment for the Goethe-Institutes London, Dublin and Amsterdam,
• Preparation of special sequences and materialsusing tablets in class,
• Special training für the „iPad-Teachers“ at the testGoethe-Insitutes,
• Monitoring throughout the whole project: BULATS test for listening comprehension as one part ofthe monitoring.
![Page 3: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/3.jpg)
Seite 3
HOW DO TABLETS CHANGELANGUAGE TEACHING ANDLEARNING?
16.04.2014
Goethe-Institut für Thema
![Page 4: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/4.jpg)
Seite 4
16.04.2014
Goethe-Institut für Thema
Mobility in the classroom
![Page 5: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/5.jpg)
Seite 5
16.04.2014
Goethe-Institut für Thema
Encouragement of thelearner‘s initiative
![Page 6: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/6.jpg)
Seite 6
16.04.2014
Goethe-Institut für Thema
Multimedia products canbe easily produced
![Page 7: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/7.jpg)
Seite 7
16.04.2014
Goethe-Institut für Thema
Listening comprehension
![Page 8: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/8.jpg)
Seite 8
16.04.2014
Goethe-Institut für Thema
Working with authenticalmaterial
![Page 9: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/9.jpg)
Seite 9RESEARCH DESIGN16.04.2014
Goethe-Institut für Thema
Bulats-Test in Test
(iPad) andControl
Test
• B1.1
• 3 months
• 3 UE/week
Language Course
Bulats-Test in Test (iPad) and Control
Re-Test
![Page 10: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/10.jpg)
Seite 10NUMBER OF LEARNERS BYGOETHE-INSTITUTE
16.04.2014
Goethe-Institut für Thema
![Page 11: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/11.jpg)
Seite 11BULATS – FACTS AND FIGURES
� since 1997, the Goethe-Institut is a BULATS-Partner from
Cambridge ESOL, der Universidad Salamanca, der Alliance
française
� BULATS ist orientated on the Common European Framework
of Reference for Languages
� available for German, English, Spanish and French
� since 2008 adaptive online-version of BULATS:
all participants start with an introductory questions at the
same level. The program chooses each test-item on
the basis of the previous answer. The level of difficulty is
adapted to the learner's level of language.
16.04.2014
![Page 12: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/12.jpg)
Seite 12BULATS – FORMAT FOR TESTING
�Scaled (A1-C2)�Adaptive�Online�Reading, Listening Comprehension,Vocabulary und Grammar
16.04.2014
30 40 50 60 70 80
. .
YesYes
YesNo
No
Yes
the test is adaptive to the candidate's level of language.
![Page 13: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/13.jpg)
Seite 13BULATS – POINTS PER LEVEL16.04.2014
Goethe-Institut für Thema
Level 0-A1* A1 A2 B1 B2 C1 C2
Points 0-9 10-19 20-39 40-59 60-74 75-89 90-100
� the points scored in the test are represented in a scale from
0-100
� the points in the scale represent the different levels of the
Common European Framework of Reference for Languages
as shown in the table
* 0-A1 includes those learners who are on their way to level A1
![Page 14: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/14.jpg)
Seite 14OBJECTIVES OF STATISTICAL ANALYSIS
• Identifying Change/Progress in Listening Comprehension (LC)
• Null Hypothesis:No difference in change/progress in LC between Test (iPad) and Control Group
• Influence of the LC-results in the first test and of the Goethe-Institute tested
• Learn our lesson for future studies
![Page 15: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/15.jpg)
Seite 15STATISTICAL TESTING OF THE NULL-HYPOTHESIS
• Mean Difference = 3.795%-C.I. [-3.8, 11.2] n=48 (27 Test, 21 Control)
• Two-sided p values: - t-test p = 0.326 - Wilcoxon test p = 0.298
=>Null-Hypothesis notrejected
![Page 16: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/16.jpg)
Seite 16
![Page 17: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/17.jpg)
Seite 17
![Page 18: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/18.jpg)
Seite 18DEPENDENCY OF CHANGE OF LC FROM PRE-TEST LEVEL
• Progress in LC decreases with increasing pre-test level,
• Above pre-test level of 80: not enough lee-way for improvement
• Up to a LC pre-test level of up to70 the 95%-confidence bands of the linear fit of the Test Group are completely above 0 � enough leeway for differences to develop
![Page 19: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/19.jpg)
Seite 19TESTING OF THE NULL-HYPOTHESIS FOR LEARNERS WITH PRE-TEST<= 70
• Mean Difference = 8.0 95%-C.I. [-2.7, 18.7]n=29 (17 Test, 12 Control)
• t-test p = 0.137 Wilcoxon test p = 0.144
• Relative Effect Size = 0.578
⇒ Null-Hypothesis not rejected but p<0.2,
⇒ Mean difference of 8.0 relevant,
⇒ Signal of a relevant effect detected
![Page 20: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/20.jpg)
Seite 20
![Page 21: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/21.jpg)
Seite 21LISTENING COMPREHENSION MEAN CHANGES AND MEAN PRE-TEST VALUES BY INSTITUTE
• Baseline LC results are well comparable in Amsterdam (n=23) and London (n=15) less well in Dublin (n=10),
• At all three Goethe-Institutes the mean increase of LC is slightly higher in the Test Group compared to the Control Group,
• Amsterdam shows a good mean increase despite a high LC mean baseline level.
![Page 22: BULATS –A TOOLFOR MEASURINGAND …events.cambridgeenglish.org/alte-2014//docs/presentations/alte2014-kathrin-han-and-win...BULATS–FACTSANDFIGURES Seite 11 since 1997, the Goethe-Institut](https://reader030.vdocuments.us/reader030/viewer/2022040505/5e3b76550e597b0420156439/html5/thumbnails/22.jpg)
Seite 22SUMMARY AND CONCLUSIONS
• BULATS is a suitable test to study factors that may influence learning of German,
• Significant effects of pre-test LC (p=0.011) and Institute (p=0.061) could be discovered,
• Use of IPad had a positive effect (p<0.2 on the subset of pre-LC <=70),
• Include subjects up to a BULATS pre-test level of 70 or at most 80 into future trials,
• To discover a relative effect size of 0.5 with 80% power, 64 learners per group would be needed.