eAssessment ScotlandDundeeBert Wylin - b.wylin@televic.com

Overview

References – use cases

France France UKBelgium

Belgium Belgium BelgiumFrance

Kenya UKBelgium Belgium

References – use cases

France France UKBelgium

Belgium Belgium BelgiumFrance

Kenya UKBelgium Belgium

Case studies for today

FranceBelgium

Case studies for today

FranceBelgium

Case study - Selor

• Scientific screening, adaptation and extension of the

national language exam for federal employees

• Introduction of smart (technological) question types allowed for the use of

authentic language materials (with multimedia) in the exam.

• New online preparatory learning environment,including thousands of exercises and simulation exams,with tracking and logging of people’s progress and results,allowing a useful training before the exam.

• High stakes: job selection, promotion, bonuses

The Belgian minister of Public Services and Enterprises getting information

about the Salto language tests beforetaking an exam herself.

Case study - Selor

Facts and figures

• Belgian federal government selection and recruitment office.

• + 200.000 exams per year• + 500 job selections per year• 2 exam rooms with 600 pc’s for

synchronous e-testing• Decentralized exam rooms in all provinces.• Online subscription for candidates• SSO login with the Selor Admin system• Exam results automatically returned into Selor

Admin SystemThe Belgian minister of Public Services

and Enterprises getting information about the Salto language tests before

taking an exam herself.

Case study - Airbus

Airbus is the world leader in airplane manufacturing. The project

Airbus screens and permanently evaluates all new candidates for the new A350 composite airplane.

Airbus organizes specialized training, both external and in-company and screens all staff members involved in this process.

Edumatic Exam is used for both evaluation and self-assessment. 

Facts and figures

• AMA-T Airbus Manufacturing Academy – Toulouseorganizes and manages more than 35.000 trainings per year

• For the new A350 FAL (Final Assembly Line) 1.000 new employeesare needed.

• Edumatic Exam is self-operated by Airbus AMA-T and the externaltraining centers

Airbus is preparing for the manufacturing ofa new composite airplane, the A350.

Case study – K.U.Leuven

Edumatic as one of the systems in the university’s Toledo platform for e-learning.

Edumatic was co-developed with research groups from various faculties:• Language faculty for the unique translation correction algorithm, the

voice recording questions and the questions including ASR and TTS• The Law faculty for the unique Open question (with fully automated

correction and feedback)• The Faculty of Business and Economics for the Accounting question

types.

Hundreds of teachers (professors and assistants) are working as authors to create questions and as coaches to do student follow-up.

Case study – K.U.Leuven

 Facts and figures• Students: 36 923 students• Staff: K.U.Leuven employs 9 560 people.

• Senior Academic staff: 1 463 people• Junior academic staff: 5 136• Administrative and Technical Staff : 3 098 people

• One undergraduate campus in Kortrijk• E-learning with Edumatic:

• More than 10.000 students online every year• 228 different courses online• More than 92.000 active (used) items in the

database• Peak uses in exam periods, more than 3.000

student online (really) simultaneously.• Fully automatic Single Sign On (SSO) webservice

with Blackboard learning platform.• E-testing with Edumatic Exam:

More than 47.760 online exams for the faculty of Business and Economics since 2008.

Students taking an exam at K.U.Leuven.

Trends in eAssessment

…triggered byCustomer’s needs

more assessment than evermore tracking and logging than evermore flexibility asked than evermore technological means/devices than ever…

…Technology evolutionsallowing for previously impossible innovative approaches

wifi, mobile, tablets, cloud computing, bandwidth available, multimedia options (webcam, audio recording…), trackpads, touchpads, data warehousing, webservices, multiplatform, standards, tagging, social networks,…

Trends in eAssessment

…solutions in eAssessment1. SSO or webservices based user management2. Webservices based results exchange3. Tag based structuring and filtering

create test scenarios based on metadata4. High performance correction algorithms

open questions5. Full multimedia playing and recording

authentic question types and materials6. Mobility, wireless and new devices7. Scalability

How do you solve…?

User management• Manually• By group import

with Excel sheets• By SSO – Single Sign On

Webservices that connect existing systemsPeople only have to login once (SSO!)

Deeplinking as an interesting feature

• SSO with GEAF• Login and results

• SSO with Selor ADMIN and website• Launched from Selor Monitoring tool

How do you solve…?

Organizing/structuring your DB• Tree structured

• Branches• Metadata• Media library

• Shows linked items• Link to external (IPR protected) resources

• Metadata for items• Metadata for screening external

training centers

• Metadata mgmt for legal issues (ml)• Content input from external partners

How do you solve…?

“We want it like the paper version…”• Navigation (linear, non-linear)

• Skip/Remember this one

• Calculator

• …

• Brainteaser/Casse-tête optionat the end of each series

• Brainteaser option evaluated asextremely useful

• Calculator

How do you solve…?

Open questions• Developed an “Open question” question type

• Based on keywords recognition(including sophisticated scoring)

• Scoring suggestion by the systemand/orhuman correction/editing

• Now also with ASR and TTS (FR, EN, NL)

• Not used• Fully testing the efficiency of open question evaluation

How do you solve…?

Voice recording(and face recording)• Developed a “Recording” question type

• No installation (Flash based)• Saved and scored (suggestion) by the system (with ASR)

and/orhuman correction/editing

• Excellent for interpreter training

• Used for self-assessment andsurvey questions“Give your comments on…”

• Voice recording for speaking skills• Voice and face recording for interpreters

(Selor for EPSO)

How do you solve…?

New devicesSmartphones and tablets• Omnipresent and cheap (smartphones)• Connected• Mobile web version

• iOS limitations (no Flash)• “Touch” limitations• Small screen limitations

• App version• Android and iOS (free or paid)• Connect, download, work• In-app (purchase of) modules• Upload/Synchronize results

How do you solve…?

Unequipped rooms/Large groups• Voting application• Edumatic download to standalone• Televic AvidaNet Teacher Control Center

• Project teacher’s screen• Students have voting devices (all iOS and Android devices)

• Many question types available• Real time answer monitoring• Full scoring, tracking and logging• Results sync back to server

• Unequipped rooms onwork floor

• Large groups (+ 1000!)

How do you solve…?

Test room security• Televic AvidaNet classroom software

• AvidaNet• Selor Monitoring Tool

How do you solve…?

Crashes, electricity cuts,toilet breaks…• Innovative features

• “Resume” function• Don’t come back function (or See only once)• Pause function

• Add time option

• Selor Monitoring Tool• Resume

How do you solve…?

Scalability• Decentralized assessments or exam• Huge groups (> 1000 people)

• Cloud computing• Switching on/off servers (energy saving)• For extra capacity and reliability• Pay what you use/need

• Synchronize between the local and the cloud system

• Azure Cloud• Decentral• Huge groups

• Decentral (all over Europe)

How do you solve…?

Reporting Services• Huge tracking and logging database• Very difficult to produce “generic” reports that satisfy all customers• Very specific combinations of data in many different views

• Reporting Web Services• “Harvesting”: customer can harvest in our database

• Selor Admin is connected via Webservice • Airbus GEAF is connected

Edumatic RoadmapRELEASE 3.6 (Sept)• COBALT• Reporting

• Per person• Per group• Per metadata

• Dialog (ASR & TTS)• Voting devicesRELEASE 3.7 (Autumn 2011)• Survey questions and management• Adaptive testing

• Custom paths• IRT algorithm paths

• Math question type +RELEASE 3.8 (Spring 2012)• Offline and standalone version

• Including standalone authoring

Edumatic MobileAll releases• Mobile candidate/student version

• iOS and Android compatible• Voting

Edumatic SMS• Spring 2012

EDUMATICHow to… (example)