saint kliment ohridski university of sofia - uni-sofia.bg
TRANSCRIPT
![Page 1: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/1.jpg)
1
DEPARTMENT FOR LANGUAGE TEACHING AND INTERNATIONAL STUDENTS
ALTE Member
Saint Kliment Ohridski University of Sofia
![Page 2: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/2.jpg)
2
A Developmental PerspectiveonStandardized Language Testing
Andonyan M., Borissova A., Yontcheva G., Todorinova J.
![Page 3: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/3.jpg)
3
This presentation covers some important points regarding the examination through the Standardized Tests of Bulgarian:
![Page 4: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/4.jpg)
4
Points of interest:
Test development Test takersTest resultsImpact of the ALTE audit 2007
![Page 5: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/5.jpg)
5
The BeginningThe design and development of Bulgarian
standardized tests gradually evolved from 2000 onwards.
Forty-five years lasting experience of the Department in teaching Bulgarian to foreign students and citizens and the sharp increase in the interest towards its studying and certifying, have been the main factors that affected the process.
![Page 6: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/6.jpg)
6
The aim of the tests is: to meet the needs of the international
audience for a test of Bulgarian according to the standards of ALTE and corresponding to the Common European Framework (CEFR)
to serve as an educational or job entryrequirement
to serve as a standard outcome for the international students taught at the Department
to be a part of the procedure for acquiring Bulgarian citizenship.
![Page 7: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/7.jpg)
7
Proficiency LevelsBulgarian standardized tests cover 5 levels of language
proficiency: A2, B1, B2, C1 and C2, which are strictly aligned with the Common European Framework (CEFR) and ALTE Framework.
BASIC LEVEL – broadly corresponds to the Common European Framework of Reference for Languages, level A2
INTERMEDIATE LEVEL – B1 and B2 CEFR levels
ADVANCED LEVEL – C1 and C2 CEFR levels
![Page 8: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/8.jpg)
8
Test Construction
Test construct, task types and format that measure four essential receptive, productive and interactive skills are based on the model of communicative competence for Bulgarian languageknown as Bachman-Tacheva’s model.
![Page 9: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/9.jpg)
9
Test Developers A team of experienced Bulgarian
language teachers from the Department has been established to develop the design and content of the exam.
The team members are teachers and lecturers with scientific interests in the field of language testing. Most of them have academic degrees and are authors of textbooks, handbooks and manuals for language testing.
![Page 10: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/10.jpg)
10
Test ContentTest content has been collected from
a great number of authentic sources: newspapersmagazines reference or history books brochures, booklets fiction, etc. Key principles for text selection are given
in the guidelines for item writers.
![Page 11: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/11.jpg)
11
Content Selection CriteriaThe main selection criteria are: external context and the language use
for different levels according to CEFR the level of skills to be tested typical subject or topic areas for different
levels socio-cultural context vocabulary range grammar and structures specific difficulties and obstacles for
foreign learners of Bulgarian
![Page 12: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/12.jpg)
12
Pre-testing
Due to some objective factors, related to test population and administration, real-time pre-testing has been difficult to be conducted.
The quality and validity of the tasks and items are evaluated by in-class testing and expert analyses, organized by the Examination Board.
![Page 13: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/13.jpg)
13
New Materials
Prior to using, the new examination materials are checked and approved by two working committees:
The Content revising committeechecks and approves the linguistic and theme content.
The Ethics committee monitors for avoiding the insensitive language usage.
![Page 14: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/14.jpg)
14
Training the Personnel
The Examination Board holds seminars, training workshopsand sessions on a regular basis addressed to:
test constructors and item writers examiners and supervisors experts, involved in the language
testing process.
![Page 15: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/15.jpg)
15
Examination CentresTwo operating centres administer the test: Testing Centre at the Department Foreign Examination Centre at IMXA
Institute – Thessaloniki (Greece)We have also contracts for establishing new
centres in: Pizza (Italy) and Nantes (France)but they have not operated yet.
![Page 16: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/16.jpg)
16
MarkingMarking is carried out according to very
clear and detailed assessment instructions both for the oral and written parts of the exam.
These instructions describe in detail: procedures of marking assessment criteria cut-off scores for different grades grading scale.
![Page 17: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/17.jpg)
17
Scoring
Multiple choice items are scored by a prescribed number of points, given for the right answer only. No score reduction is made for
either the wrong or missing answers.
![Page 18: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/18.jpg)
18
Scoring
Speaking is assessed and assigned a grading mark according to a special assessment instruction for the oral part of the examination only.
This part of the examination is fully audio-recorded.
![Page 19: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/19.jpg)
19
AccuracyAll components are marked immediately
after the examination. To assure the accuracy and quality of the
marking, only professional examiners are assigned to conduct the examinations. They have been selected among the most qualified full-time language teachers at the Department.
The team of trained examiners is annually approved by the Examination Board.
![Page 20: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/20.jpg)
20
Training the examiners
During the marking process examiners have to follow strictly defined rating criteria. These criteria and grading procedures are clarified in special training sessions, which take place at least once a year.
![Page 21: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/21.jpg)
21
Examination supervisor
The accuracy of marking is monitored by an examination supervisor, who has the right to confirm or disprove the marks assigned by the examiners.
In case of disapproval, the supervisor is authorized to correct the assigned marks and to report the case to the Examination Board.
![Page 22: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/22.jpg)
22
The Overall Grade
Speaking25%Writing
38%
Reading15%
Listening22%
The overall grade of the examination is formed by the results from different parts as follows:
![Page 23: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/23.jpg)
23
Test timing and cut-off score
The test has to be completed within 3 hours.
The cut-off score is set to 60% of the total.
Candidates are provided with leaflets describing the scope of the examination and a user guide how to apply and how to interpret the results.
![Page 24: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/24.jpg)
24
Post-test studies
Several post-test bias analyses are carried out to make sure the results are not influenced by any extraneous factors.
These studies also reveal test-takers’ profile and draw conclusions based on the data, collected from both the pre-test and post-test questionnaires.
![Page 25: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/25.jpg)
25
QuestionnairesResponses to the questionnaires enable
the Examination Board to gather information about:
test-taker’s educational background the purpose and use of the exam pre-testing study and preparation for
the exam immediate impression and degree of
satisfaction from the examination
![Page 26: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/26.jpg)
26
Distribution of the tested population over the period 2000-2007
Total = 419
43 5062
90
174
0
20
40
60
80
100
120
140
160
180
2000/2003 2004 2005 2006 20070
20
40
60
80
100
120
140
160
180
200
![Page 27: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/27.jpg)
27
Purpose of the examination
General3%
Work34%
Citizenship15%
Education48%
The examination is mostly used for education, workand citizenship.
![Page 28: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/28.jpg)
28
B2 level is mostly selected for education
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
![Page 29: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/29.jpg)
29
B1 level is mostly selected for citizenship
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
![Page 30: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/30.jpg)
30
B1, B2, C1 and C2 levelsare selected for work
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
![Page 31: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/31.jpg)
31
Purpose of the examination distributed by age
2126
3540
education general work citizenship
![Page 32: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/32.jpg)
32
Purpose of the examination distributed by gender
7
53
4
4332
75
4
4639
128
8
89
0
20
40
60
80
100
120
140
citizenship education general work
female male totalSignificantly more men than women apply for citizenship
![Page 33: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/33.jpg)
33
Purpose of the examination distributed by nationality
Turkish, 39%
Greek, 64%
Greek, 35% Other, 38%
Other, 26%
Education Work
![Page 34: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/34.jpg)
34
Tested population distributed by age groups
26 - 4035%
- 188%
40 +17%
19 - 2540%
![Page 35: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/35.jpg)
35
Tested population distributed by gender
male53%
female47%
![Page 36: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/36.jpg)
36
Which are the most popular test information sources?
Advertissment
Test Book
Friends
IMHA Institute
Internet
None
Office
Staff
40+ years26-40 years19-25 years-18 years
![Page 37: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/37.jpg)
37
Is preliminary test information sufficient for the test-takers?
YES50%NO
19%
No answer18%
Neither13%
![Page 38: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/38.jpg)
38
Which factors best supporttest preparation?
0 10 20 30 40 50 60
-18 years
19-25 years
26-40 years
40+ years
TeacherSelf-practiceText book
Teacher is the main test preparation factor.
![Page 39: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/39.jpg)
39
How do the test-takers rate the effectiveness of their test preparation?
-18 years old
19-25 yearsold
26-40 yearsold
40+ years old
positivenegative
Young people demonstrate negative attitude towards test preparation.
![Page 40: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/40.jpg)
40
What is the average grade for different testing levels?
4.75
4.72
4.60
5.18
5.43
4.00 4.50 5.00 5.50 6.00
A2
B1
B2
C1
C2
![Page 41: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/41.jpg)
41
Statistics of the overall grade by age groups
5.1
4.6
4.99
4.93
4.84
5.38
4.75
5
5
5
4 4.5 5 5.5 6
-18
year
s19
-25
year
s26
-40
year
s40
+ye
ars
All
Mean Median
Teenagers demonstrate higher results
![Page 42: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/42.jpg)
42
Statistics of the overall grade by gender
5.01
5
4.68
4.75
2 3 4 5 6
Mea
nM
edia
n
MaleFemale
![Page 43: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/43.jpg)
43
Average grade and L1
3.00 3.50 4.00 4.50 5.00 5.50 6.00
Turkish
Asian
Greek
Hebrew
Arabic
French
German
English
Italian
Spanish
Slavic
![Page 44: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/44.jpg)
44
How do the test-takers qualify their examination result?
19%
15%
54%
12%
no answer
worse
as expected
better
![Page 45: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/45.jpg)
45
Self-rating of the results distributed by levels
0% 20% 40% 60% 80% 100%
A2
B1
B2
C1
C2
worseas expectedbetter
27% of C1 level candidates qualify their result as worse than expected
![Page 46: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/46.jpg)
46
How do the test-takers rate the quality of examination?
no answer17%
very good39%
good8%
satisfactory1%
unsatisfactory1%
excellent34%
81% of the test-takers appreciate the quality of examination
![Page 47: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/47.jpg)
47
ALTE audit 2007
B2 level test was audited in June 2007.
This ALTE audit revealed some drawbacks and also inspired immediate actions for improving the quality and administration of the Standardized Tests of Bulgarian.
![Page 48: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/48.jpg)
48
Increasing the number of candidates The Council of the Department decided
to allow the international students free of charge examination as a regular outcome of their studies at the end of the academic year 2006/2007.
90 students took advantage of this proposal (the sponsorship cost the Department nearly € 13 500).
![Page 49: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/49.jpg)
49
Improving pre-testing
Pre-testing in real-examination conditions is to be carried out parallel with in-class pre-testing from 2008.
Preliminary test is to be introduced to help pre-testing and avoid misleading the candidates by identifying their approximate level.
![Page 50: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/50.jpg)
50
Reliability
Insertion of anchor itemsacross different variants to handle the reliability of the test.
Increasing the numberof test-takers.
![Page 51: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/51.jpg)
51
Calibration
To continue the process of developing calibrated examples for evaluation of various models for the different levels of the test.
![Page 52: Saint Kliment Ohridski University of Sofia - uni-sofia.bg](https://reader034.vdocuments.us/reader034/viewer/2022042802/62683c63866c305391069d74/html5/thumbnails/52.jpg)
52
Thank you for your attention