tlc2016 - turning blackboard learn into a digital examination platform: lessons learned from 8 years...

Turning Blackboard Learn into a Digital Examination Platform

Lessons learned through 8 years of practice

Geert De Schutter, Educational technologistWalter Roosels, Educational technologist

Exam in progress

Digital assessment/examination: Why?

• Strategic reasonsImage building

• Efficiency reasonsSaving time (large groups)

• Financial motivesStaff – student ratio

• Educational motivesObjectivity, quality of assessment

Digital assessment/examination: University of Antwerp

• Focus on efficiency and educational benefitsSaving time (large groups)

• Authenticity of assessmentUse various types of questions & multimedia

• Accurate, objective and quick correctionAutomating the repetitive character of correcting

Facts and figures (4): students and examinations

5

Total amount of exams per student

Total amount of students taking an exam

6

Facts and figures (1): Evolution since reference year 2009 - 2010

7

Facts and figures (3): 2014-2015

Capturing the vibe…

9

Question 1: What about infrastructure?

Availability of infrastructure

Campus CDE135 (4 rooms)

Campus CGB 90 (2 rooms)

Campus City275 (8 rooms)

Campus Zuid85 (3 rooms)

Campus PM77 (4 rooms)

Total # 2015-16662 pc’s

New (2016-17) campus CDE

200 (5 rooms)

New (2016-17)campus CGB

328 (8 rooms)

Total # 2016-171190 pc’s

Anti-cheat sheet

Availability vs suitability: preventing fraud and cheating

• Anti-cheat sheet PC

• Assign PC at random (numbered cards)

• Identity control (student card & identity card)

• Banning of watches and (smart)phones

• Use of Building Block examination course

• Consecutive sessions (without time interval)

Availability vs suitability: preventing fraud and cheating (2)

13

Availability vs suitability: preventing fraud and cheating (3)

• Block Internet access & access to file-shares

• FILTER per room (router configuration):– No internet access– Blackboard only

14

Question 2: What about software?

15

Does our software meet the expectations?

Blackboard?

16

Beneficial: student-friendly/authentic examination

• Questioning: use different types of questions– Not only MC questions

– Questions pertaining to knowledge and insight (<> easy correction)

• Enriched Content– Images in question text and answers

– Video/audio file

• Students– Essay: text editor, spelling checker, word count– Access to Test Results and Feedback

– Magnify text and images

– SPRINT (assistive software)

17

Beneficial (2): coping with security and anti-fraud measures

Test options:• Use random blocks of questions

• Show answers in Random Order• Randomise questions

• Use of different series (≠bgcolour) for questions

• Present all questions at once or one question at a time & prohibit backtracking

• Define Test availability exceptions

• Set Timer• Use password

• Restrict location=> IP-filter

But what about other courses? Content collection? …

Adaptive rule ?

18

Beneficial (3): quality assurance & reusability

• Quality control– Attempts statistics– Column statistics– Item Analysis– Download results=> run statistics

• Reusing questions– Electronic archive of pools of questions & tests

19

Beneficial (4): scoring and processing of results

• Grade questions

• Grade with usernames hidden

• Essay:– digital text is readable, better structure of text, word count– use of rubrics– print to HTML/WORD/PDF (program)?

• Use negative scores?

• Fill-in questions?– partial credit alternatives and recalculate score unsatisfactorily

20

Additional Software

• Building Blocks – BB Examination course– BB negative scores– BB Rename & report (feedback CC)

• In-house software– Application WordtoBB– Application MFIB (collect unique answers)– Program: Print Essay & word count– AlephQ (test software)

21

Building Block: Examination course

22

Why?

• Need of secure environment

• Avoid fraud via access to Blackboard– Courses– Organisations – Content Collection– Calendar– Social tools

23

What?

• Course tool in regular course

• Generates New course based on specific template

Link: Start ExamLink: End Exam

• Transfer of students & instructors regular course >> examination course(daily synchronisation)

24

Examination course “Start Exam”

How?

• Student clicks on the link “HIER”

• Alternative: Instructor changes mode for one/multiple/all students in Course tools, Examencursusregular student => exam student

What happens?• Courses and organisations unavailable

• Content Collection tab unavailable: (disable auha_contentcollection portal role)

• Grade Centre column “Examenlink” gets marked “1”

25

Before

“start exam”

After

No acces to Content Collection tab

Only exam course in course list

26

Examination course “End exam”

How?• Student clicks on link “End Exam”

• Alternative: Instructor changes mode for one/multiple/all students

What happens?• Courses and organisations available

• Content Collection tab available: (enable auha_contentcollection portal role)

• Grade Centre column “Examenlink” gets marked “0”

27

Examination course

Start exam

Access to test

Different tests

Instructions: what to do after submissionInstructions

Random block of MC, sorting, combining, jumbled sentence

Essay

Information

End exam

28

Question 3: What about support?

29

Administrative Support

Rooms:

• Availability, capacity, software, headphones, …

Communication towards students:• Sample exam, Practice test, • Use of Announcements • Course menu link “practical matters”:

– link to Education and Examination Regulation – guidelines & instructions concerning the procedure– video about the procedure– course information: competences, evaluation criteria– application of negative scores

Technical Support

• Exam course– Activate – Prepare tests & assignments– Include pictures & multimedia– Prepare settings/adaptive release

• Supervisors:– Procedure via VIDEO/PPS– Certified training – Checklist procedure/problems

• During exam– Start & finish

31

Lessons learned & advice

32

Lessons learned

• Good organization & preparation is key!

• Success of digital exams creates new demand– Demand for BB exams continues to grow– What about the capacity of computer rooms?

• 4 weeks of examinations: – Stressful - exhausting – intensive for BB-team

33

Downsides

• What to do with network problems/Blackboard problems?– Keep calm and call IT colleagues

• Loss of work:– File was not saved correctly when downloading/uploading– File could not be retrieved

• Procedure of Exam– Rather complicated– Difficult to explain to some teachers=> need for technical support!

• Settings for a test in different places : – Test options– Adaptive release– No password/IP-range for assignments

34

Key considerations

• Need for atypical questions– Chemistry, accountancy, languages,…

• Need for external correction procedures– Fill-in fields – Negative scores – Essay word count– Different tests for different types of questions

35

Conclusion

Goals achieved for students and teachers:

• Enriched questions with multimedia

• Improved readability and structure of essays

• Accurate, objective, quick processing and fast correction of answers

=>Saving a lot of time (large groups)

• Quality assurance

• Secure environment

36

Nevertheless:

In order to:• Integrate all security settings in one location• Avoid loss of work and network problems (DOS Attack)• Combine authentic examination and powerful correction

Take your assessments and examinations to the next level with Blackboard and AlephQ

So, don’t leave just yet, but stay for the next presentation!!!