Maurice Grinbergwww.nbu.bg

evaluation.nbu.bg

ALTE meeting, Lisbon, November 12-15

Modern private university Year 1991: 4 000 students in 30

programs Year 2008: 15 000 students in more than

100 programs Bachelor, Masters, and PhD programs Distance learning For the first time in Bulgaria:

General Education Test as entrance exam four times a year

Assessment CenterALTE meeting, Lisbon, November 12-15

Since 1995: General Education Test as admission exam (more than 10 000 candidates per year)

Since 1998 for all NBU students:• L2 exams at the end of a course of study,

levels from A2 to B2 • 6 languages: English, German, Italian, Russian,

Spanish, and French • L1 exam for writing skills in Bulgarian language• Computer skills exam

Since 2002: placement testsALTE meeting, Lisbon, November 12-15

Staff: Experts in testing, psychometrics, cognitive science, languages

Advisory board:• Department of Psychology and Cognitive

Science, and department of Foreign Languages• NBU QA department• External consultant:

prof. D. Dimitrov, George Mason University Projects and research

• Leonardo da Vinci program (WELKOM)• Erasmus program (NETTLE)• Assessment in e-learning • Cognitive assessment, small group assessment

ALTE meeting, Lisbon, November 12-15

Monitoring and development of all examination procedures in NBU

Development of exams for NBU students Analysis and interpretation of test

results: test score reports to test takers and university administration

Item-banking and storage of data Training for item-writers and

standardized workshops for examinersALTE meeting, Lisbon, November 12-15

Fairness and equity• Standardized administration• Monitoring during examination• Special accommodation for candidates with

disabilities• Information about the content and format of

the examination: question types, rubrics and appropriate test-taking strategies, test samples

ALTE meeting, Lisbon, November 12-15

B2 is a required level in NBU for all students

Goals: use the language to express and understand opinions, to build up and understand arguments, to communicate with students from other countries in English, Spanish, Italian, French, German or Russian.

Certification decisions and graduation

ALTE meeting, Lisbon, November 12-15

Test takers: Bulgarian native speakers with different topical knowledge and second language experience

Tasks: both from the instructional and real-life domain

Scores are reported to candidates in grades

Tested abilities: speaking, listening, reading, and writing

ALTE meeting, Lisbon, November 12-15

Psychometrics: • IRT used for evaluation of test item

properties • Many-facet Rasch analysis in assessing essay

writing task (facets: topic, raters, and assessment criteria)

• Estimation of candidates’ abilities, raters’ severity, relative difficulty of tasks, and relative difficulty of assessment criteria

• Bias/interaction analysisALTE meeting, Lisbon, November 12-15

Meet all ALTE standards for quality in language tests: • Revise existing foreign language exams• Achieve a consensus about test qualities• Develop a new test for Bulgarian as a

foreign language

Exchange of best practices: • Investigation and assurance of test

reliability• Improvement of validation argumentation• Elaboration of procedures to involve all

stakeholders in the examination processesALTE meeting, Lisbon, November 12-15

Development of tests for Bulgarian as a foreign language• Situational analysis: need for a A1 test for

foreign university students in Europe and Asia and related textbooks

• Planning phase: users and test takers identified, resource balance performed

ALTE meeting, Lisbon, November 12-15

Development of tests for Bulgarian as a foreign language (cont.)• Design phase: theoretical background of the

test• Approaches considered:

Bachman model of CLA will underlie each task as a needed executive resource

CEFR for the domain, communicative goal and language function

Weir’s Frameworks for Developing and Validating tests

ALTE meeting, Lisbon, November 12-15

NBU: http://www.nbu.bg Assessment center:

http://evaluation.nbu.bg(in construction!)

ALTE meeting, Lisbon, November 12-15

About Impact: Our strive to achieve positive impact involves

Providing public information and advisory services for students

Seminars to familiarize university teachers with our research on test qualities

Training for item writers and examinersReports to the University BoardCooperative work with the Department of

Foreign Languages to improve continuous assessment in language classes

Problems: we have not investigated the educational impact of our examinations

Decisions: start collecting evidence about “face validity”

About Practicality: The Assessment Center develops and implements the appropriate procedures for managing all aspects of the examination process.

Problems: need to increase research studies in all aspects of test production

Decisions: identification of suitable experts

ALTE meeting, Lisbon, November 12-15