Recommendation for a new testing application

Advisory report

ICTO Large-scale digital testing project

Document history

Date Version Description Author

29 November 2012

0.1 Initial version Leen van Kaam

30 November 2012

0.2 Including scenarios and changes referred to in the agreement of 29 November 2012

Leen van Kaam

13 December 2012

1.0 Recommended version as approved by SIG Digitaal Toetsen

Leen van Kaam, Alwin Wullink

Amendments to the previous version are highlighted in yellow.

Content

Introduction .................................................................................................................................... 4

Recommendation ............................................................................................................................... 5

Showstoppers for all demands and requirements, including traffic light reports ............................... 5

Discriminatory demands and requirements in order of priority ......................................................... 6

Conclusion ...................................................................................................................................... 8

The test scenarios .............................................................................................................................. 9

Scenario 1: Cost analysis, implementation ....................................................................................... 9

Scenario 2: Technical infrastructure implementation ...................................................................... 13

Scenario 3: Continuity Implementation .......................................................................................... 15

Scenario 4: Support and help function Implementation .................................................................. 18

Scenario 5: Setting up domains within the application ................................................................... 21

Scenario 6: Roles for each academy and workflow ......................................................................... 23

Scenario 7: Security ....................................................................................................................... 29

Scenario 8: User management Implementation .............................................................................. 32

Scenario 9: Converting test questions ............................................................................................ 35

Scenario 10: Developing new test questions .................................................................................. 36

Scenario 11: Test matrix ............................................................................................................... 40

Scenario 12: Preparing to take a test ............................................................................................. 44

Scenario 13: Large-scale testing.................................................................................................... 49

Scenario 14: Supervising testing .................................................................................................... 52

Scenario 15: Test requirements ..................................................................................................... 53

Scenario 16: Multilingualism Implementation ................................................................................. 55

Scenario 17: Test inspection.......................................................................................................... 57

Scenario 18: Report on test questions, tests and testing Implementation ....................................... 62

Scenario 19: Analysis .................................................................................................................... 64

Scenario 20: Assessing caesura, scores and results........................................................................ 66

Scenario 21: Archiving .................................................................................................................. 71

Scenario 22: Capacity for secured skills testing ............................................................................. 72

Appendix 1 Those involved in this recommendation ...................................................................... 74

Introduction This is an advisory report for Saxion’s digital testing environment project. The report is intended to present a recommendation for Saxion's choice from a shortlist of three testing applications: Maple TA, Question Mark Perception and Surpass. The advisory report ‘Advies_Testscenario_05_20092102’ describes the process, participants and reasoning behind this choice. This advisory report documents the selection process conducted using the method described here and makes a final, substantiated recommendation to the ProgramControlGroup.

It is based in part on the participation of the academies now conducting electronic digital testing or which plan to do so in the near future. These participants represent a wide range of roles in the testing process, including those of instructor, educational expert, academy examination desk, central examination office, functional administrator of digital testing, technical administrator of digital testing, academy ICT&O contact person and the academy's representative for SIG digital testing.

The recommendation consists of twenty-two scenarios in the report, in most cases based on the discussions in two meetings with the participants referred to above. It must be pointed out that neither the recommendation nor the traffic light report are intended to be a general evaluation of the testing applications on the shortlist. The purpose of this evaluation is to answer the question of whether and to what extent the testing applications on the short list are compatible with Saxion's digital testing procedures.

Recommendation

Showstoppers for all demands and requirements, including traffic light reports The test scenarios include a number of 'showstoppers'. A showstopper is defined as a scenario with a level of priority for Saxion so high that if the application fails in this scenario, it can be stopped and further testing of scenarios for that application is not required (indicated by a red light on the traffic light report). The results of the analysis of all showstopper requirements from the testing scenarios are shown in the following table:

Questions Priorities Maple TA QMP Surpass

2 Technical infrastructure Server installation in Saxion 1 SaaS server installation 1 Workstation installation 3 Continuity Proven technology 1 5 Setting up domains within the

application Each academy's own domain 1

6 Roles per academy Roles sufficiently finely honed 1 7 Security Minimum CIA standards 1 Secure delivery 1 Test schedule per student 1 8 User management Blocking student accounts for

management rights 1

11 Test matrix Use of a test matrix 1 12 Preparation for taking the tests Central exam office or test desk

options 1

13 Large-scale testing Are 500-700 simultaneous tests possible?

1

18 Reporting on tests, test questions and those taking the tests

Reporting tool 1

19 Analysis Test analysis data 1 Question analysis data

This makes it clear that Maple TA is inadequate as a testing application for meeting Saxion's summative digital testing needs. Question Mark Perception exhibits 1 showstopper, namely that the application does not currently support a test matrix. This showstopper is critical because Saxion's digital testing method in TestVision is entirely designed around working with a test matrix. Additionally, we see that running 500-700 test-takers on Question Mark Perception is risky (yellow). Looking a little more closely at this aspect, we observe that QMP would allow 500-700 simultaneous logins, as long as these participants' logins were spread out over a period of five minutes and there was a suitable infrastructure.

The illustration below outlines the infrastructure that would be required to run 800 simultaneous tests in QMP and 6000 (Silver Installation) or 10,000 (Golden Installation) simultaneous tests in Surpass. The illustration makes clear that the infrastructure required to support QMP at an adequate level of

performance for Saxion would be fairly significant and would presumably have a correspondingly significant price tag.

Applicatieserver = application server Auteur = author Afname = (taking the) test

Surpass emerges from the showstopper analysis as the clear winner.

Discriminatory demands and requirements in order of priority If we take Maple TA out of the equation, then order all demands and requirements by priority and remove those demands and requirements where QMP and Surpass have the same score, we can see how both applications are compatible with Saxion’s digital testing procedures. The view that Surpass is the most compatible with Saxion’s procedures is then confirmed.

If you order all demands and requirements by priority, it is striking that Surpass is the most compatible with Saxion’s procedures even from this perspective. Surpass mostly has a green score but has a somewhat lower score only in terms of functionality with a lower priority.

There are still a number of points on which Surpass emerges with a yellow or red score:

• Use of arithmetical elements/formulae: The roadmap for Surpass’s new interface also includes a tool for processing arithmetical elements and formulae more efficiently. We shall be bringing to BTL’s attention the fact that this functionality is a huge priority for Saxion.

• Dutch/German-language version: The roadmap for Surpass’s new interface also states that it will be available in Dutch in the summer of 2012. The German version is scheduled somewhat later on their roadmap.

• Mobile app available: The roadmap for Surpass’s new interface also includes a mobile app. • PC-dependent testing: This cannot be adjusted in the application, but can be arranged – if

required – from the technical infrastructure. • Adaptive testing: Saxion will expressly inform the supplier when this functionality will be

required from Saxion. • Scoring on competences is not in fact a test application functionality. Parantion’s Scorion tool

is used for this purpose for Saxion.

Questions Priorities QMP Surpass

11 Test matrix Use of a test matrix 1 13 Large-scale testing Are 500-700 tests possible at the same time? 1 4 Support and help function Support from supplier 1 5 Setting up domains for the application Is the application still complete? 1 7 Security Access code test 1 8 User management Automatic read-in of all users 1 10 Developing new test questions Mathematical elements/formulae 1 Other types of test questions 1 11 Test matrix Use of metadata 1 Use of folder structure/learning tree 1 16 Multilingual versions Dutch 1 21 Archiving Suitability for archiving 1 2 Technical infrastructure Mobile 2 6 Roles for each academy and workflow Assigning capabilities or screen each role 2 Developing workflow of test items and testing 2 Workflow administrative support 2 9 Converting test questions Including analysis data 2 Converting test questions 2 12 Preparing to take the tests Automatically extended test period 2 13 Large-scale testing Solutions in the event of disasters 2 14 Guidance for taking tests Monitoring testing (remote) 2 16 Multilingual versions Multilingual (German) 2 7 Security PC-dependent testing 3 11 Test matrix Questions meet conditions 3 11 Adaptive testing 22 Opportunity for secure skills testing Possibility of scoring (part) competencies 3

Conclusion The outcome to this selection process is that Surpass is unanimously recommended as the new testing application for Saxion to replace TestVision.

The test scenarios

Scenario 1: Cost analysis, implementation

Procedure Saxion’s digital testing requirements are described using 3 key elements.

1. Licence; 2. Infrastructure and technical administration and technical application management; 3. One-off implementation costs.

The description contains a baseline on each of these key elements. This baseline contains Saxion’s minimum requirements and the testing application will offer an appropriate solution. In addition, a number of value-added criteria are defined for each key element. A supplier can stand out by doing well on these value-added criteria. The supplier indicates whether there are similar value-added criteria for the relevant testing application and at what additional cost. Points with clear weighting are awarded to the baseline for each key element and the value-added criteria.

Key element 1: Licence

In the baseline for key element 1, we ask for the licence costs of 2 different situations to be detailed separately:

Situation 1: Long-term situation

The basic assumption is a campus licence for the testing application of a minimum of 3 years, including an option to extend it for a further 3 years. Saxion has around 23,000 students and 1,700 instructors. There is a maximum of 50-60 highly specialised testing developers for Saxion (12 academies each with 5 specialists or ‘test deskers’), who need access to specialist development of test questions. The licence must state that testing takes place in a secure environment. Saxion assumes that the licence includes availability of an acceptance and development server in addition to a ‘production server’. Assessment with the testing application may be both summative as well as formative in the long term. We assume that the licence costs are given for the environment that the supplier recommends as the optimum testing environment (cf. also key element 2).

Situation 2: Short-term situation

During the first year of implementation and also for 1 or 2 years afterwards, not all academies and therefore all students will be using digital testing. The basic assumption is the current digital testing capacity for Saxion, which means that around half the students at most use the current testing application. That means that we request a price quotation for a licence for 12,000 students, 800 instructors and 25 highly specialised testing developers. As expected, a maximum of 30,000 tests a year are taken in the short term (i.e. 1 test taken by 1 student) with the testing application. The licence must also state that testing takes place in a secure environment. Saxion assumes that the licence also states that an acceptance and development server will be available in addition to a ‘production server’.

The testing application is mainly used summatively. We assume that the licence costs are given for the environment that the supplier recommends as the optimum testing environment.

As regards the value-added criteria for key element 1, we ask for the costs of the following to be detailed.

It must be possible to offer extra modules in the testing application in order to produce one of the scenarios from the attached appendix. Consideration should be given here to:

• the costs of an extra licence for providing a flexible reporting tool in which Saxion is itself able to define reports;

• costs of an extra licence to have the option of deleting test questions from the score after analysis of the test results, or else to be able to score as was originally intended;

• costs of an extra licence to have the option of using mobile services combined with the testing application.

• Saxion uses a print & scan solution for absorbing capacity shortfalls for digital testing. The costs of an extra licence for having – if present – the print & scan functionality on the application.

Key element 2 Infrastructure

The infrastructure must be capable of handling with no performance problems 500 - 700 (maximum 1000) simultaneous tests when there is a fair number of archived test and results data (e.g. 3000 tests) in the database. We expect the supplier to provide advice about the optimum infrastructure given the scalability and security requirements (see scenario 7) and experiences with other clients.

The SLA offers management during the test weeks in a window from 7 am to 11 pm on normal working days including a minimum uptime guarantee of 99.5% provided during these hours. The window for the SLA is from 9 am until 5 pm outside test weeks. One year of study equates to 15 test weeks.

In the baseline for key element 2, we ask for the licence costs of 2 different situations to be detailed separately:

SaaS solution:

Technical administration and technical application management are organised externally. The supplier is responsible for the entire technical administration of the application.

Partial in-house management:

Servers are physically located at Saxion. Server management (and OS management) is by Saxion, but technical application is managed by the supplier.

As regards the value-added criteria for key element 2, we ask for the cost of any options the supplier may offer for the SAAS solution to be detailed, for example a bronze, silver and gold SLA. Infrastructure solutions offering extra options for security are especially appreciated.

Key element 3: One-off implementation costs

On the baseline for key element 3. We would like to set up user management automatically. User management automatically includes (at least once a day) the creation of all students and instructors in the testing application based on data from the LDAP/Active Directory. In addition, we would like to authenticate these in the LDAP/Active Directory. The costs of doing this also need to be detailed for the baseline.

As regards the value-added criteria for key element 3:

• We ask each supplier to outline the costs of extra specialist training for specialist test developers. In other words, the costs of 4 1-day specialist training sessions for 10 authors.

• Each supplier has received a package containing 3 tests in various formats as they may be exported from TestVision. We want to find out from the suppliers:

o What % of the test questions can be automatically converted? o What are the costs of converting this package including 3 types of test questions? o Can there be any deviation from the folder structure in which the tests are stored in

TestVision? o Is it possible to include values for a p-value, for example, or an RIT value? If so, how? o How are illustrations or other multimedia included? o A methodology is produced that can preferably be implemented by Saxion itself.

• Saxion uses an existing print & scan solution from Teleform to absorb capacity shortfalls for digital testing. One part of the test is often carried out digitally and one part is dealt with using forms that can be scanned. We would want to be able to read the scanned results into the database so that they can be part of an analysis together with the results of that part of the test that is taken digitally. In order to prepare scan forms prior to testing using print & scan, login details, the name of the person taking the test and the test name are merged with the forms and students therefore receive pre-printed forms. We would like to find out what the costs are - if possible - of both options for reading back scanning results into the database and for merging user data. If only one of the two options is possible, however, we shall ask for these to be detailed.

Saxion expects a specified offer. However, in order to provide a simple comparison of the offers, we also need the costs to be summarised in the table below:

Key element Cost Heading Summary of Costs

1. Licences Long-term licensing solution (campus licence)

Short-term licensing solution (used by half the campus)

Flexible reporting tool for licence

Licence for re-marking tool

Mobile licence

Print & scan licence

2. Infrastructure and technical administration

SaaS solution

Partial management by Saxion

Extra profiling for ...................................................................... ...................................................................... ...................................................................... ......................................................................

3. One-off implementation costs

Automatic alignment of user administration and authentication on LDAP/Active Directory

Specialist training

Conversion

Print&Scan alignment

* Cost analysis

Questions

Implementation The procurement office has received 3 offers. We are not able to publish the full offers here now. These are still subject to negotiation through purchase once the choice is made for Saxion. We shall

therefore list a number of characteristic differences between the offers in a general sense to clarify the point in each case.

Maple TA: The licence comes under a completely different price range to QMP and Surpass. The costs of key elements 1 and 2 together for Maple TA are approximately 1/6 of the costs of QMP and Surpass. No specific mobile app is included (but you can open Maple TA from the browser on mobile devices), nor a separate reporting tool or scoring tool. The supplier offers only the baseline for key elements 1 and 2. One-off costs are half those of Surpass and approximately 1/3 of QMP's.

Question Mark Perception and Surpass are in the same price range for key elements 1 and 2. The difference in pricing model is striking. Efinity provides one transparent pricing model for Surpass based on a price per user per year. Stoas provides four different pricing models for QMP in which the structure is not totally transparent due to the wide variety of licence versions. For key element 3, we see that QMP is approximately 30% more expensive than Surpass. A major difference is the fact that Efinity does not calculate any costs for a tool for conversion from TestVision to Surpass, where Stoas does do that for QMP. All the value added points required for all the key elements have been filled in both for Surpass and for QMP. The offer for QMP was complete in every way and included an SLA example and a DAP example. A version of the SLA is missing in Surpass.

Scenario 2: Technical infrastructure implementation

Procedures The assessment will be made as to whether the application is compatible for Saxion’s technical infrastructure based on the supplier’s documentation and a teleconference.

Questions 1, showstopper

Is the server installation compatible with Saxion’s technical infrastructure

1, showstopper

Is the equipment needed for a Saxion work station compatible with Saxion’s technical infrastructure?

2 Uptake of tests using MacOS 2 Developing tests and reports using MacOS 2 Development in the near future for use with Mobile devices

Implementation Maple TA: The server installation runs on Windows 2008 or a Red Hat Linux installation. This is compatible with the Saxion infrastructure. Maple TA works with the Progress database. This is not supported in Saxion. Installation at Saxion is thus not desirable and an SAAS solution (hosted environment) is the only conceivable solution. The supplier still needs to indicate whether site-to-site VPN is supported. This is necessary if users are automatically sent to the application. A site-to-site VPN will also be used for the LDAP traffic. As a result the connection is additionally secured.

The testing environment runs on Windows OS, Linux and MacOS. The Firefox and Internet Explorer browsers are supported, but Google Chrome has not yet been tested. The (open) testing environment also works with Safari. The management environment does not. Delft Technical University has

developed a secure browser. This is supported by Delft Technical University, not by the supplier. That is a risk for Saxion because Saxion wants to use the tool mainly for secure summative testing. The secure browser has been installed within the Saxion infrastructure and tested. It works in the infrastructure.

Maple TA works as a standard application and on Mobile devices with Android operating systems. This has not been extensively tested. It does not work, however, on an iPad or iPhone as Maple TA uses JAVA significantly. No Mobile App is being developed.

Question Mark Perception: The server installation operates from Windows 2008 and IIS 7.5. The database server is an MS SQL 2008 SP2. QMP is also provided on Oracle. Stoas does not recommend this because it is a secondary option. Oracle upgrades usually appear much later than those for Windows. QMP is a .NET application with a PHP frontend. Stoas supports a site-to-site VPN (which one is not yet clear) for ensuring that user data can be automatically created securely and authenticated securely on the LDAP. The server installation is thus compatible with the Saxion infrastructure.

The authoring software only works under Windows, both on Internet Explorer (6, 7, 8, and 9) and Firefox (3, 4, 5, 6, 7, 8, 9, 10, 11) and on Safari 4. QMP does not work on Google Chrome. This is compatible with the Saxion infrastructure. The secure browser QM Secure does not yet work on MacOS. Beta testing is being carried out for this. Given that all summative tests are taken with the secure browser, this does not yet work on MacOS. A QMP Mobile app for Android and iPhone/iPad is designed for taking tests.

The secure browser has been installed in the Saxion infrastructure and has already been operating during APO testing with 60 students.

Surpass: The server installation operates from Windows 2008 IIS 7,5 and MS SQL 2005 or 2008. QMP is a .net application. Site-to-site VPN is not necessary because user data can be retrieved from a secure connection at Saxion and do not therefore have to be placed on the testing environment at Surpass. The server installation is thus compatible with the Saxion infrastructure.

The authoring client works on a Windows7 installation, as well as the secure browser client application. The management environment works with Windows Internet Explorer 7, 8 and 9, Firefox 3, 4, and 6 and Google Chrome. This is compatible with the Saxion infrastructure. It is not yet possible to take summative tests on a MacOS, any more than it is to develop test questions. The new interface for Surpass, however, is being expressly developed for MacOS. Their roadmap indicates summer 2013 as the deadline for this new interface. (See also http://saxion.hosted.panopto.com/Panopto/Pages/Viewer/Default.aspx?id=44d70c15-6705-43c8-8cac-6e1422a21bbc ).

The secure browser client has been installed for the Saxion infrastructure and tested. It works.

The secure browser is not just simply a browser. It is a client application in which data are stored temporarily encrypted in a local database. The application does not yet have a Mobile app. A new

interface is also being developed for Surpass with this in mind. A mobile interface is therefore expected next summer for the management software.

Results:

Maple TA QMP Surpass (1) Server installation (showstopper) for Saxion

(1) SAAS server installation

(1) Workstation installation (Showstopper)

(2) MAC testing (2) MAC management

(2) Mobile

Scenario 3: Continuity Implementation

Procedures: Consultation with the supplier, retrieval of the supplier’s annual report. Retrieval of supplier’s references. The manner in which the testing application is used is illustrated for ‘proven technology’. The opinion of one or more reference clients of each testing application is requested for this showstopper. Consultation with the supplier, retrieval of the supplier’s company annual report. Retrieval of supplier’s references. The educational institutions that use the testing application are shown for ‘proven technology’. A questionnaire of around 10 questions has been drawn up. These 10 questions are put to 3 people per application.

• QMP: Referees are sought at Hanze University, Windesheim and VU University • Maple TA: Referees are sought at Delft Technical University, HAN and NHL. • Surpass: provide referees from an educational situation in England that is the closest to

Saxion.

The questionnaire as sent to the different referees can be found on Blackboard under Saxion community ICTO> SIG Testing > Recommendation process testing application > scenario 3 Continuity. The answers on the questionnaire can also be found there.

Questions * Description of the company’s history * Scope of the department responsible for development and management 1 Showstopper

Proven technology

Maple TA: Maplesoft is a company that focuses mainly on developing software for mathematical applications. Maplesoft is already 25 years old and was until recently a private company. For some years now it has been part of a larger Japanese distributor Cybernet Systems Co. Ltd. See also

http://www.maplesoft.com/company/about/index.aspx. The description of Maplesoft is omitted in the annual report of this Japanese company. In addition, Maplesoft is first and foremost developing the Maple mathematical programme. Maple TA is a relatively small by-product that relies on Maple for a number of functions. Of the 70 or so people working for Maple Soft, 5 of them work as full-time developers/managers at Maple TA. 5 extra freelance people are sometimes hired to develop the package.

Maple TA is used at a number of higher education institutions:

Technical University of Delft Arnhem Nijmegen University of Applied Sciences Northern University of Applied Sciences of Leeuwarden Utrecht University of Applied Sciences

Rotterdam University of Applied Sciences * Saxion University of Applied Sciences University of Amsterdam Van Hall Larenstein Amsterdam University of Applied Sciences * Inholland University of Applied Sciences * Hanze University of Applied Sciences * Fontys University of Applied Sciences

It is striking that very many of the institutions referred to here also use QMP. It is also striking that Saxion is listed. Limited use is made of Maple TA for Saxion within the LED academy. English-speaking instructors from MIM are also carrying out a pilot scheme with Maple TA. It appears that Maple TA is used to a limited extent in several HE institutions mainly for mathematical test questions. In short, Maple TA is proven technology because it is being used. The consultants’ comments indicate whether users are satisfied with the application for the contexts in which they use the application.

CAN diensten is Maple TA’s Dutch supplier. It is a small company that sells mathematical, analytical and statistical software. One of the products is Maple TA. See also http://amsterdam.digicity.nl/CAN_Diensten-1589989900id.html.

Summary of the referees’ comments about Maple TA:

Maple TA is used across the institution only at TU Delft (Delft Technical University) Maple TA is used in one faculty at the NHL. Question Mark Perception is used across the institution at HAN. According to the central service, the use of Maple TA is an initiative by a few instructors within a training course and is not supported by ICT&O. The situation in the HAN is therefore similar to the use of Maple TA in Saxion.

Question Mark Perception: Questionmark (QM) was set up in 1988 and was initially a pioneer in digital assessments. On the basis of these experiences and client feedback, the first global digital testing system came onto the market in 1995, resulting in the launch of the first web-based version, Questionmark Perception ( ) in 1998.

More than 2,500 organisations across the world have opted for QMP as their strategic management system for digital testing with clients from educational bodies, profit and not-for-profit organisations as well as government agencies.

QMP is in use at many universities in the Netherlands, an overview of existing educational institutions follows.

Academic Medical Centre Amsterdam (AMC) GP training in the Netherlands Christian University of Applied Sciences in Ede Marnix Academy Windesheim University of Applied Sciences Open University Fontys University of Applied Sciences* Stenden University of Applied Sciences The Hague University of Applied Sciences University of Amsterdam * Hanze University of Applied Sciences * University of Maastricht Council for Higher Professional Education University of Twente Arnhem Nijmegen University of Applied Sciences * VU University InHolland University of Applied Sciences * Wageningen University Rotterdam University of Applied Sciences * Various regional training centres (15) Amsterdam University of Applied Sciences * Various knowledge and trade associations

Zuyd University of Applied Sciences Education-related

It is striking that a number of the institutions listed here also use Maple TA. These are marked by *.

Product development (and continuing development) is actually based on two key elements. On the one hand, client questions/user groups that come directly within or via the QM distribution channel are carefully heeded. On the other hand, QM has a management team that has won its spurs internationally in the field of visionary ICT - and developments in education. These two approaches ensure QMP’s continuing development.

In all, there are roughly 100 employees actively involved in QM’s product development. Some of them (25) are directly involved with the strategic and tactical developments (vision, technical management, support) and the other employees are active at operational level (software development and programming). In some countries, QM takes care of distribution directly and in the Netherlands, Stoas Learning has been QMP’s distributor/partner for more than 20 years. With an annual turnover of 10 million or so Euros (2010 financial year), Stoas lists Questionmark as a sound company. QMP’s operational and technical knowledge is present to a large extent in Stoas Learning. Around 7 people work with QMP every day at Stoas in a technical and operational capacity. Saxion has been working with Stoas since 2009 because Stoas runs the technical application management on Blackboard for Saxion. See also http://www.stoas.nl. Question Mark is a privately organised company and does not have an annual report.

Summary of the referees’ comments about QMP:

QMP is used across the institutions within the reference group at the HAN, Windesheim University of Applied Sciences, Hanze University of Applied Sciences and VU University. The VU University would definitely applaud if a competitor were to come onto the market for Question mark.

Surpass: Efinity has been Surpass’s Dutch representative since autumn 2011. Efinity has close ties with other universities, is a member for instance of the NVE (Dutch Examinations Association) and seeks collaboration in SURF. BTL is the parent company that Surpass supplies. BTL was founded in 1985. The company originally focused on e-learning services. Although Surpass has been involved with testing since 1985, the Surpass platform only came into existence in 2008. BTL is a private company with

around 70 employees. The company has traditionally focused on a partnership with very large companies only, the so-called awarding companies. One very large client, for example, is the Scottish government that has all examinations tested using Surpass. In England, a number of very large examination boards use Surpass.

For further information, see: http://www.btl.com . See also http://saxion.hosted.panopto.com/Panopto/Pages/Viewer/Default.aspx?id=6531a02e-ccf7-47e0-9566-3ab798d919a3.

Until the end of 2011, BTL restricted itself to the English market alone. BTL is a financially sound company. Information about the company can be found on Blackboard under Saxion community ICTO> SIG Testing > Recommendation process testing application > scenario 3 Continuity.

Result Maple TA QMP Surpass 1 Showstopper

Proven technology

Scenario 4: Support and help function Implementation

Procedure ICT&O functional administrators are able to get a good picture during scenario testing of the support provided by the supplier because they can carry out all the tests with the academies and have to be able to find their way in the application and discuss various topics with the supplier. In addition, a number of reference clients are asked to evaluate for each application.

Questions 2 Does the application have a good help function? 2 Is support possible for FB ICTO and test desks irrespective of time and place? 1 Support from the supplier

Implementation Maple TA: CAN diensten provides support in the Netherlands. For functional support they use the freelance services of Metha Kamminga, former instructor at the Northern University of Applied Sciences of Leeuwarden, a mathematician and long-term user of Maple TA. See http://www.methakamminga.nl/ . Metha has retired early as an instructor. She is the only person offering functional support for Maple TA. She is frequently abroad from around the end of April until October. We consider the fact that functional support depends on a single person who is already retired to be risky for Saxion. CAN diensten mainly provides technical support. CAN diensten also diverts calls regarding technical problems through to Maplesoft’s support desk. FB ICTO and test desks are able to access every part of the application, irrespective of time and place, because it is a web application.

The help function is not context sensitive. The application’s user guide can be downloaded. You can search in it, you can look up the part where an explanation is needed using the table of contents. And there is an index on the help function. There is also a web-based administrator, instructor, supervisor and student help.

Summary of the referees’ comments about Maple TA:

Technical University of Delft on support:” We do not have an SLA with the supplier, but support issues are properly responded to. There are occasionally support issues with technical management, but these arise much more regularly in the area of functionality, or tips/tricks relating to test questions”. There is no help desk relationship either with the supplier at the NHL.

Question Mark Perception: Stoas provides 2nd-line support in the Netherlands and sometimes diverts calls through to Question Mark’s support desk. Stoas employs consultants both for functional as well as technical support. Around 7 people work on at Stoas every day with Question Mark Perception. Stoas has good intentions as regards support. However, it was found that the tiered structure of Stoas 2nd line – QM 3rd line means that the support does not always offer an immediate solution. We note that there were a number of problems during the scenario tests, which took a relatively long time to resolve, particularly at the time when Stoas was not able to solve the problem (e.g. access to QM Live, downloading secure browser, access to communities, information on communities, website not up to date). In addition, the QMP infrastructure seemed relatively tricky in the sense that unclear error messages appeared with relatively simple functionality, for which a total solution is not immediately apparent (secure browser error message, sluggish response from administration environment, performance testing). We regard this as a risk.

The help function is not context sensitive. You can use the help function to open a user guide for the section in which you are working (e.g. authoring or scheduling tests, etc .....). The user guide is in English. FB ICTO and test desks are able to access every part of the application, irrespective of time and place, because it is a web application. Space is also provided in the form of the Question Mark

Communities where substantive information can be found on testing as well as practical examples. This may also be didactic support. See http://www.qmark.com .

Summary of the referees’ comments about QMP:

Windesheim on support provided by QMP: “Initially handled by Stoas. Very dependent on how the call gets assigned. In addition, calls must be assigned through clients (electronic system for handling complaints). We have often already explored all the options and avenues before we insert a report. Stoas then starts unravelling the same problem again from scratch and you will therefore get a repetition of steps. Takes a great deal of time and energy and is also frustrating. For some genuinely technical issues, it is often more convenient to have a technical consultant look into our environment. A lot of water has already flowed under the bridge before you’re ready. If Stoas does not manage to resolve it, send a message to Questionmark and you’ll get the same process once again. Questionmark does not respond quickly to (potential) queries. All in all, it can sometimes be weeks or months before you have a solution to your problem. Some problems are also brushed aside under the guise that they are (or will be) on the roadmap. In this case, it may be a year or longer before a bottleneck is addressed. For example: the call for a report with which a score rating can be calculated. It is still not there and I think it was requested 10 years ago now.”

VU: “Application has a lot of bugs and nasty glitches. These cannot always be resolved. Everything goes through a call system and that’s not nice.”

Surpass: This also has tiered support. Efinity provides 2nd line of support and sometimes diverts calls through BTL’s 3rd line of support. BTL states on the website that it prefers to think of itself as a partner of the institutions Surpass uses rather than a supplier. This atmosphere is the breath of support. FB ICTO has had a great deal more contact with consultants at BTL than with those from Question Mark or Maplesoft. This was not because there have been more problems with Surpass, but because BTL has already enlisted Saxion directly as a partner for developing the new interface for the software. BTL has also approached Saxion at its own initiative to poll people about the new interface. They were then also prepared to give a presentation at short notice. Not only has BTL investigated whether it was possible to convert questions from TestVision, but they have also created a tool that is demonstrated in a Skype session and made the tool available for testing. Similarly, they have also started creating a solution for secure skills testing with MS Office.

The help function is not context sensitive. You can use the help function to open a user guide for the section in which you are working (e.g. authoring or scheduling tests, etc.). The user guide is in English. FB ICTO and test desks are able to access every part of the application irrespective of time and place because it is a web application. Substantive information about testing and didactic support can also be found in a community at Surpass. See http://www.btl.com/community/ .

Result Maple TA QMP Surpass 2 help function 2 support irrespective of

time and place

1 Supplier support

Scenario 5: Setting up domains within the application

Procedures This scenario is closely linked to scenario 6 with regard to roles for each academy and workflow. Both scenarios together provide a good picture of the possibilities for working with the application.

We do not develop a whole model domain, but look at whether a model domain can be set up for each application with the tools of the application.

There are 2 different areas here where a domain needs to be set up. Each academy wants to have a single learning tree available with test questions and testing as a whole, but areas with different rights must be able to be assigned to different people within the learning tree.

Example: the test desk of an academy has read, write and admin access to the entire learning tree of the academy, but each test administrator has read, write and admin access to the tests and test questions of their own modules/subjects. Instructors can only provide test questions for their own subject areas.

The second area is the area of preparing for the tests before they are taken.

Example: the test administrator for a certain test provides a test matrix. This is registered in BISON. The test desk devises this test matrix according to a test with test questions in the test application. The test administrator checks whether the matrix has been devised correctly and checks the test versions on the basis of the test matrix. The test desk therefore has read, write and admin rights to compile test matrices and test versions based on those test matrices. The test administrator has minimal read rights for obtaining a preview in each case. The central exam office ensures that the test is scheduled in time and that students are enrolled for the test. The academy test desk may also be responsible for this. The test is supervised either by the central exam office or the academy's test desk. The test administrator must ultimately be able to see the results of the test again.

Questions 1 showstopper

Can each academy create its own domain without this producing administration problems (e.g. 12 installations of the same software)?

2 Are there also certain containers that can be shielded off within a certain domain (e.g. for each training course or for each year of study, etc.)?

1 Are there opportunities for producing collaborations between academies within the academy interface?

2 Is it possible to create access to the right areas within the collective academy interface for the exam office and ICT&O Functional Administration?

1 Is the application still complete despite the use of domains? Do users not need to import and export files between miscellaneous sections of the application?

Implementation Maple TA: Maple TA works with classes. Test questions and testing exist within a certain class and cannot be exchanged without copying between classes. Within a class a user can acquire the rights to administer tests and test questions or as a participant in taking a test. Within a class you can protect rights to sections of the content by creating sub-classes. Instructors with administration rights to a (sub) class can edit all test questions and tests within that (sub) class and prepare them prior to the test being taken. You cannot create any new containers within a (sub) class. Those taking a test are logged on to the class and are then enrolled for all the tests in that class. This means in the Saxion context that in order to register only participants for the right test, you would need to create a (sub) class for each test matrix. Test questions are then virtually interchangeable between classes without that same test question being copied to other classes. The number of classes will therefore increase just as the number of tests being taken will. Seen from the sections and the overview of the questions in the learning tree, this far-reaching and rigid format is not desirable.

For academies to work together, separate classes or sub-classes need to be re-created. Exchanging test questions is still a question of copying a test question.

The test centre and ICT&O Functional Administration would then need to have all rights within all classes just like the test desk in order to plan tests and administer enrolments. However, all instructors in a class are also entitled to plan and administer enrolments.

There is still an option for each academy to carry out their own installation of the software. This creates problems with collaboration within academies and the need for manual import and export between domains where there is collaboration.

It is therefore clear that the format of the application in domains is difficult at the very least and contains a lot of risks.

Question Mark Perception: Questions and tests are stored in a folder structure. Read, write and/or admin rights can be granted to certain people at senior level. The various modules/themes from which tests are compiled therefore form the basis for the folder structure at senior level. This structure resembles the one in TestVision and has proved to be workable in the past.

Whenever academies collaborate, they are able to share certain folders, but an academy employee can also obtain read rights, for example, to another academy’s folders. Planning and enrolling students for a test is organised on the basis of roster groups. Each test desk of the academy can obtain rights to enrol their own roster groups for a test and schedule a test into the time. The central exam office obtains write access to certain roster groups and tests, but not to the entire folder structure. The exam office is therefore able to schedule tests for all academies, but not change the questions and tests.

It appears from the questionnaires on QMP that the majority of HE institutions working with QMP operate in practice with a separate environment, in which instructors develop their tests and test questions and a separate environment in which the tests are taken. Stoas recommends this to prevent test questions being changed by mistake while these are being used in tests. This structure also safeguards performance. Test questions and tests are supplied to the testing environment by manually

exporting and importing the development environment. This is a structure that we actually do not want for Saxion.

Surpass: The application has test centres. A test centre is an academy or a training centre, for example. Tests are compiled in a test centre based on qualifications. A qualification is a (training course) diploma, for example, for which someone undergoes training. A qualification may also be a competency. The folders with test questions are linked to the qualifications. The tests are also linked to qualifications. A test can be linked to several qualifications and several centres. Users can only see the qualifications, test questions and tests for which they have obtained access rights.

You can search for tests and test questions under qualifications and centres.

A root folder (a project) can be set up for each test matrix where the folders including test questions are kept and in which instructors can compile test questions. All kinds of user rights are established for each folder and characteristics of the field in which the test questions can be compiled. Users only see those projects to which they have access rights. Test questions may also be compiled there.

To compile test matrices and test versions, the test desk is only able to find those that have been released for publication and are linked to certain centres and qualifications. It does not matter which project these test questions are in as they cannot be changed here, only viewed.

The central exam office obtains access to certain centres and qualifications, but only for scheduling test versions (e.g. language test-quartile 1-1st chance, Language test-quartile2-repeat). The exam office is therefore able to schedule tests for all academies and organise enrolment, but not to change the questions and tests or test matrices.

Results:

Maple TA QMP Surpass 1 SS

Each academy’s own domain

2 Screening in domain 1 Collaboration between

academies

2 Entry for test desk, central exam office, FB ICTO.

1 Application is a single whole with no import and export

Scenario 6: Roles for each academy and workflow

Procedures We do not develop a whole model system of roles and a model workflow, but we look at whether adequate roles and an appropriate workflow can be provided for each application with the tools of the application. Testing the workflow based on the entire process is detailed at an initial meeting with those concerned.

Roles Task

Test administrator Point of contact for validity, reliability and acceptance of the test: test matrix. Test analysis and statement of results

Instructor Responsible for (part of) the test questions, review

Examining board Quality assurance of the test

Test desk Digitising test matrix and generating tests from the test matrix

Rostering and planning Rostering test and location

Central exam office Coordinating implementation of the tests and enrolment of participants.

Invigilator Taking the test

Expert Explanation of the analysis of the test and test questions

IT centre Technical management and infrastructure support

ICT&O functional administration

Application management and support

Establishing test matrix Test administrator and

Exam board

Creating and reviewing test questions

Instructors

Digitising test matrix Test desk

Generating test from

test matrix Test desk

Carrying out test analysis

Test desk

Rostering P & R

Test schedule Exam office

Testing Exam office and

invigilators

Test analysis Test administrator and education

expert

Not every academy in Saxion uses an equally detailed workflow or the same workflow. A high degree of flexibility is therefore needed from the application for operating with a workflow and with roles to please the 12 academies.

Questions 1 showstopper

Do roles include capabilities that are sufficiently finely honed for use in Saxion?

2 Are these roles to be set up because FB ICTO is able to assign or screen capabilities for a role? Are new roles to be created within the application?

2 Is it possible to organise a workflow for developing test items testing within the application?

2 Is it possible to organise a workflow for administrative support within the test application?

Implementation Maple TA: Maple TA works with classes. 3 different roles are available within a class, i.e.:

• Administrator: The role administrator has all the rights to all Maple TA’s classes. The administrator is able to assign users to roles within classes at system level.

• Instructor: The role for those developing test questions and tests within a class then schedule and enrol students in the class.

• Supervisor: Invigilator, able to grant students individual access for a test. • Student: The student takes part in the test.

These roles are fixed within the application. This means that the capacities of the roles cannot be changed. Nor can any new roles be added. There is in fact only 1 role for developing and managing test questions and tests and for scheduling and enrolling participants. This means that Maple TA is too limited in role differentiation to be able to function within Saxion’s organisation.

There is no tool available for creating a workflow within the application. On the basis of the available roles, anyone with rights to certain classes other than participants can do anything both in terms of developing questions and tests and in scheduling and enrolling those taking the tests. A workflow is

embedded in the application’s interface and this assumes that the same person compiles the test, sets security options and plans the time. This is highly undesirable as a security method.

Question Mark Perception: There are 4 rights available for developing questions and tests for each folder with test questions or for each folder with tests, which an administrator may assign to a user account:

• No access • Full access • View/Use • Edit/Use

A user account may also be given a role referred to as a profile. The administrator may also create new profiles.

Powers are also defined within a profile, which may or may not be assigned to the profile. These powers are relatively crudely defined. They seem to be sufficiently usable in the Saxion context, but there are also limitations in defining roles due to the clustering of powers.

There is a tool within Question Mark Perception for developing a workflow for questions and tests. This workflow tool, however, is not fully integrated in the application. A special sort of root directory needs to be created if the workflow is to be used at all. A workflow is not possible in normal folders. It is therefore not possible in the context of implementation simply to start by developing questions and to switch over at a later stage to operating with a workflow. It is possible to create several workflows and to assign a different workflow for each folder (for another academy). The workflow operates with a number of question statuses, i.e.:

• Normal (can be used in a test) • Retired (can no longer be used in a test) • Incomplete (under construction, cannot be used) • Beta (can be used in a test) • Experimental (may already be used in a test, but does not count in final score)

In Saxion’s test license, workflow was not operating when we received it. It appears that a separate licence is needed for the workflow editor. The separate licence appears to be no longer available. The workflow only concerns the compilation of test questions and testing and may only be used in the client application, the so-called authoring tool (see scenario 10). Powers can also be linked to roles in the administrative field. There is no tool for creating an administrative workflow. An administrative workflow can only be setup via appointments and procedures.

Surpass: An administrator can define roles with matching powers for each project when developing questions and tests and then also define workflow statuses.

And they attach workflow statuses to roles.

There may be a number of question statuses in the workflow, i.e.:

• Draft • Ready for review

• Proof read • Ready for approval • Approved • Released • Withdrawn

These question statuses do not need to be used at all. A question is developed as a draft by default and is only available for compiling tests if the question has released the status. This means that different workflows may be defined for all the academies for developing test questions.

When a test is being developed, there are several roles available with a finely tuned system of powers. Several roles are also available for scheduling a test to be taken and enrolling those taking it. These roles may be very finely tuned by the administrators. New roles an also be created.

As a result, there are more than enough tools for creating administrative workflows as well, including the roles needed for Saxion.

Roles and rights are carefully assigned based on a certain standard. The scale of possibilities makes it necessary to be careful when creating a Saxion-specific functional structure from the application. The supplier’s advice is not superfluous in this.

Results:

Maple TA QMP Surpass 1 SS

Finely tuned roles with powers

2 Personally creating roles and new roles possible

2 Workflow for developing test questions and testing.

2 Workflow for administrative support

Scenario 7: Security

Procedures: A discussion is held with all 3 suppliers about the possibilities for the IPI matrix and the specific security of the application prior to testing. Tests are carried out by the ICT&O functional administration regarding the specific security prior to testing. These tests are not done from the point of view of breaking through the security, but from the point of view of understanding how the security is organised.

Kai Haveman from ACT with a group of students was asked for a white hat hacker test.

Questions 1 showstopper

Minimum compliance with the IPI standards

* Opportunities for meeting extra security requirements from the IPI matrix * White hackers test (if ACT does not carry it out, look for another way of carrying it out) 1 showstopper

Secure delivery of the test

1 Access code for the test 2 Individual control of access 3 Available only on certain premises (on IP number) 1 showstopper

Arranging how often the test may be opened per student and per test, on what dates and at what times and with which access code.

2 Storage process from which PCs a test is/has been created. 2 Secured access to tests by students is possible

Implementation In principle. the ACT academy has pledged its commitment for the white hat hacker test, but the time at which this test comes seems awkward for the course programme. This means that the test has unfortunately not yet taken place.

Maple TA: Maple TA is fully scalable and completely redundant in the next release. The next release is expected in the spring of 2013. There are several Tomcat servers, an incremental backup of which is made every day. A full backup is made every month. There is an emergency power supply. Maple has 24-hour monitoring on the servers. However, support in Canada during the night consists solely of re-starting a machine if other actions do not succeed. This is however daytime in the Netherlands! CAN diensten in Amsterdam provides technical support between 9 am and 5 pm. SSL connection is possible. User data are supplied to Maple TA via a secure VPN connection. With regard to performance (see also scenario 12), it is known that 600 users can log in at the same time on a server with 8 CPU and 12 GB RAM. There are therefore a number of risks for the IPI matrix particularly in long-term scalability and redundancy and the technical support.

A summative test is delivered via a secure browser. This secure browser is developed and administered by the Delft Technical University. The supplier cannot therefore be approached during secure testing. This is a risk. A test is linked to a class in Maple TA. Access to the class establishes that a student has access to all the tests in that class. The test can be set in time. This time cannot be adjusted per student, only per class. For students with an extended test time, for example, a copy of the test must

be prepared to include the extended time. If students have to take the test individually, login may be requested by a supervisor. The supervisor must provide a login code and password before the student can log in. It is not possible to provide an access code during the test. It is possible to make the test available for certain IP addresses alone. The reports do not state the IP address under which a student has taken the test. Access to a test has a great number of opportunities for limiting this access. Access does not appear to be secured.

The test is stored per question at the time a new question is retrieved. Open questions can therefore be stored until timed out. No test is possible without an internet facility. There is no offline test version and, if the internet connection fails during the test, the rest of the test will no longer be available.

Personal login name and password are possible. There is no extra strong authentication externally.

Question Mark Perception: The test environment is redundant. There are several application servers that can replace each other and which are backed up every day. The database is also backed up every week with daily incremental backups. There is a redundant internet connection. A failover service is provided in the short term at a different location. There is an emergency power supply. Stoas has 24-hour monitoring on the servers. The service window for support has a maximum run-time from 5 am in the morning until 11 pm in the evening on weekdays. In relation to the performance (see also scenario 12), a robust infrastructure is needed on servers to guarantee simultaneous availability for 500-700 users. A summative test is delivered via a secure browser. Groups and individual students are logged on as participants in the test. This means that the test can be set in time individually and per student. If students need to take the test individually, login may be requested by a supervisor. The supervisor must then provide a login code and password before the student can log in. It is not possible to obtain an access code during the test. It is possible to make the test available for certain test premises alone. The IP address on which a student has taken the test is recorded and a report can be created based on it. The test can be accessed via the secure browser. We have not been able to reproduce testing in practice (see also scenario 17) to check whether this secure access method is also compatible with Saxion procedures. A customised coaching report can be created, however, for giving feedback to a student. The test is stored per question at the time a new question is retrieved. When working on a question for a long period, Stoas recommends saving it periodically in order to prevent timeout. It is also recommended that an open question should not be recorded as the last question in a test to prevent the answer getting lost. It is possible to take a test without an internet facility because there is an offline test version, for which a separate licence must be purchased. If the internet connection fails during the test, the rest of the test will no longer be available. SSL connection is possible. User details are delivered to QMP via a secured VPN connection. Personal login name and password are possible. There is no extra strong authentication externally. A summative test can be securely accessed.

Surpass: The environment is duplicated in the SaaS solution. The test environment is delivered in a load-balanced cluster with several application servers that can replace each other. 15,000 users a day can be tested per application server. 5000 students can start the test at the same time on a Golden configuration. An on-line backup of the application servers is created every day. An incremental backup of the database is created every hour. An offline backup is created every week. A separate failover capability is provided at a different location. There is a redundant internet connection. There is an emergency power supply. It is possible to take a test without an internet facility (there is an offline test version).

Surpass works in a different way to QMP and Maple TA. A secure browser is not used on the local workstations alongside a central Surpass server, but a secure client application is. At the time the test starts in this local client application, the entire test is downloaded with all the questions. If the test has started, the local client delivers the test questions for submission to the student. The answers are initially stored in the local client’s database. These data are stored encrypted. The local client synchronises the answers given with the central server every 30 seconds. If the internet connection fails during the test, the rest of the test that has been downloaded to the local client is usually still available. The results are stored locally and synchronisation is resumed once the internet connection is restored. The servers are monitored 24 hours a day. There are several SLA versions for support on the environment. The server configuration would be for the SaaS solution in England, as a result of which support is guaranteed during Saxion’s working hours. Performance is good, even with large numbers of students (see also scenario 12) and Surpass can therefore be arranged on 1 server for Saxion in terms of performance. A load-balanced infrastructure is created for failover purposes.

SSL connection is possible. A summative test can be accessed securely because a summative test is delivered via a secure client. Groups and individual students are logged on as participants in the test. This means that the test can be set in time individually and per student. There are 3 levels of test security:

• Those taking the test receive a key code for logging on to the test. This is a randomly generated code (of numbers and letters) that applies for a single test at a single point in time for a single student. Those taking the test then have to ‘sign’ a declaration stating that they are who they say they are and are not committing fraud.

• An access code can be requested for the test. • If students need to take the test individually, login may be requested by a monitor. The

monitor must then provide a login code and password before the student can log in. This may be done locally, but also remotely.

It is not possible to have the on-line test only available on certain test premises. It is possible with the offline version. The IP address on which a student has taken the test is stored and a report can be created on it. Access to a test is restricted, especially for a summative test. The test can be accessed via the secured browser. A report may be compiled for access to the test.

Personal login name and password are possible. A secure tool is used by Saxion to retrieve user data and authentication Saxion from the Active Directory. There is no extra strong authentication externally.

Result Maple TA QMP Surpass 1 SS

Minimum compliance with the IPI standards

1 SS

Secure delivery of the test

1 Access code for the test 2 Individual control of access 3 Available only on certain premises (on IP

number)

1 SS

Arranging how often each test may be opened, on what dates and at what times

Maple TA QMP Surpass and with which access code.

1 SS

Arranging how often the test may be opened, on what dates and at what times and with which access code for each student.

2 Storage from which PCs a test is/has been created.

2 It is possible for students to access tests securely

Scenario 8: User management Implementation

Procedures The assessment is made as to whether user management can be automated and authentication is possible on the LDAP based on the supplier’s documentation and a teleconference.

Other sections are tested in the application.

Questions 1 All users are automatically read in every day 1 LDAP authentication 1 No anonymous accounts 2 Can a student account be blocked in order to obtain other rights to the test application

on that account, followed by testing and test inspection? 1 showstopper

Several people may act as administrators at the same time in the same domain.

Implementation Maple TA puts users into classes. The program has an API whereby it can import users to classes. A Building Block has therefore been constructed on this API, for example, for Blackboard. Links to other software can also be constructed. If users are automatically loaded via the API and the account is created automatically, no LDAP authentication is subsequently possible. The program from which the users are loaded (e.g. Blackboard) carries out the authentication. If the accounts are created by manually loading CSV files, it is possible to authenticate them on the LDAP. Secure authentication is possible via SSL. A connection with Blackboard, for example, is only desirable for formative testing, not for summative testing. The combination that Saxion wants, automatic reading in of student accounts followed by authentication, does not exist in Maple TA. An e-mail address may occur only once when creating users. This is undesirable, for example, in accessing the e-mail address noreply@saxion.nl that is used for test and demo accounts.

It is not necessary to create anonymous administrative accounts. Several administrators work simultaneously in a class or in the application. The role of administrator is added by name to an account. Students receive a different role when adding users to a class, but this is not (physically) separate from the process of adding administrators. Making mistakes is therefore a risk.

Question Mark Perception contains a supplementary QMWise application that contains functionality for linking the user administrator via web services. This link is customised because the infrastructure of each institution is different. There is no experience with this link because the majority of institutions still choose to import users with manual batch files. It is possible to authenticate these securely via SSL on the LDAP. There has been good experience at the VU, for instance. It is still unclear what software has to be used for a secure connection in order to create users.

Administrative roles may be assigned to accounts in personal names. Anonymous accounts are therefore not necessary. There are several accounts in a certain area that may acquire an administrative role, sometimes with different administrative roles as well. The student role prior to testing and the different administrative roles for creating questions and tests are assigned at different places in the application and are therefore physically separate. Participants create a test (see illustration) and administrators can create test questions or tests, or schedule tests, or assign users to tests.

Surpass can automatically read in users from the Active Directory. There is a special tool for this. Surpass has experience with reading in from the Active Directory, but because the infrastructure of all the institutions is different, this means customisation. Surpass is able to retrieve the user files securely on the Saxion servers and therefore sending user files securely via a site-to-site VPN is not necessary.

It is possible to authenticate the Active Directory securely on the LDAP (via SSL). There has also been experience in doing this.

Administrators are assigned because a specific role is assigned to a user account. It is not necessary to work with an anonymous administrative account. Several administrators can be assigned to specific sections (referred to as centres) and sometimes in various roles as well. Administrators (users in Surpass) acquire administration rights at a different place in the application to those taking part in tests (candidates in Surpass).

Result 1 Maple TA QMP Surpass 1 Automatically

importing users

1 LDAP authentication

2 No anonymous accounts

1 showstopper

Blocking student accounts for administrative rights

1 Several administrators domain.

Scenario 9: Converting test questions

Procedures 3 tests are chosen from 3 academies. These are Multiple Choice, Multiple Answer and Yes/No questions in particular. There are also questions with blanks to be filled in. All export forms of these test questions have gone to the suppliers with the question as to whether this can be converted, to what extent these will then be connected and at what cost. We therefore want to find out whether everything is can be converted.

Questions * What percentage of the test questions can we convert to the test application in a freely

automated way? What is the cost associated with conversion? 2 Is automated conversion of test questions possible? 2 Can analysis data be included? * What quality check is possible when converting tests and test questions?

Implementation Maple TA: Conversion is possible at an additional cost. No conversion is made by the supplier in this test phase because no payment will be made prior to conversion until Saxion has made the decision. It is not known whether the conversion can be carried out in sections by ICT&O Functional Administration, or whether the conversion will be carried out by Maple. We do not therefore know whether we can do a quality check during export and import. We do not know whether analysis data can be included.

Question Mark Perception: Conversion is feasible based on txt export of test questions from TestVision. Additional costs have been charged for converting all Multiple Choice, Multiple Answer, Yes/No and questions with blanks to be filled in. The offer includes an extra cost heading for this. Illustrations and other multimedia are not included in the conversion. It is not known whether the conversion can be carried out in sections by ICT&O Functional Administration, or whether the conversion will be carried out by Stoas. We do not therefore know whether we can do a quality check during export and import. The conversion, as far as we know at present, may not include any analysis data such as the level of difficulty of the question (p value).

Surpass: A conversion tool has been built for Multiple Choice, Multiple Answer and Yes/No questions based on a TestVision export. The conversion tool is not yet suitable for questions with blanks to be filled in. The tool is available at no extra charge. Illustrations and other multimedia files can be included in the tool, although this requires an extra manual procedure in which the underlying multimedia files are transferred to an appropriate spot in Surpass. The test questions are designed based on the layout template used for the different types of question and the design is therefore in our own hands. The tool for conversion is transferred to ICT&O functional administration and they can used it for parts of the test questions. Thus it is possible to carry out a quality check with the conversion in consultation with the academy without our first having to clean up the test questions in TestVision.

Analysis data can also be imported. These, however, are not present in the TestVision export that can be converted, but could be manually imported into the Excel file, which forms the basis for importing TestVision questions in Surpass.

Result Maple TA QMP Surpass 2 Conversion

possible?

2 Can analysis data be included?

Scenario 10: Developing new test questions

Procedures Test questions are imported by instructors through one academy and only by a test desk at other academies. We want to be able to import the following types of question in an automated manner:

- Multiple Choice - Multiple Answer - True/False - Fill in the Blank - Essay

An easier way of importing questions is needed for instructors: a pretty basic method from MS Word for all those academies where instructors do not enter the test application at all. A suitable interface is also needed for those academies where instructors do enter the test application. The approach to questions must be based on the test matrix. The test desk can then look at how many questions are available and can then control which questions need to be created. It would be nice if this could be determined in the application so that a message automatically appears if questions need to be created. Complicated questions such as hotspot questions are more difficult to create. This type of question needs to be created by experts alone, such as the test desks or ICT experts from the academies. Mathematics questions are trialled in all applications. We create a number of multimedia questions in the applications and display them in a test to be created. Questions are also processed in the test that includes multimedia questions with the most common advanced questions that are possible in the application. A description is written of the possible types of question for each application.

Questions 1 Simple submission of test questions by instructors with no extensive training 1 Capacity for recording mathematics elements/formulae in test questions and test answers 1 Use of multimedia in test questions and test answers 1 Use of other types of test question * Description of which types of question there are in the test application

Implementation Maple TA: Questions are created for each class or sub-class. You cannot re-use the same question in the same

version for different (sub)classes. You must then copy questions. Questions cannot be imported into a class in Maple TA from an MS Word or Excel file. Only exported questions can be imported from Maple TA. Creating questions in the interface is relatively easy. The interface is not intuitive in every respect, but is easy to handle for instructors who regularly work on it.

The application has been specifically developed for creating mathematical questions. A lot can therefore be done in it, both as regards mathematics/physics/chemistry elements/formulae in test questions and test answers. In order to be able to make genuinely exhaustive use of the opportunities, a great deal of very specialist knowledge is required since this has to be programmed. This is certainly not the preserve of a standard user, but can in fact only be done by experts with a knowledge of mathematics, physics or chemistry.

Multimedia may be used, but not in every format. Moving images, for example cannot be used in formats such as AVI, MPEG and WMV. Moving images can only be used in Flash format. Illustrations and smileys can also be used.

The types of question available are listed in the appendix. The possibilities are sufficient in themselves, partly because different types of question can also be combined in a single question (e.g. MC + Essay, or Yes/No + question with blanks to be filled in). Maple TA has a limited number of tools for creating adaptive testing.

Question Mark Perception: Questions can be imported from a txt file. Macros can be created in MS Word or MS Excel for generating that txt file. Multimedia cannot be included using this txt import. A limited number of types of question can be created in the web-based interface. This methodology can be used for instructors who are not highly skilled and who work in the test application, but it will disappear in the future.

It will be replaced by the so-called LIVE interface. What this interface will look like can be seen in the Question Mark communities, where this interface is already available for users. See https://www.questionmark.com/uk/live/Pages/default.aspx . However, this interface is only integrated in the Question Mark Perception software version 6. This means that questions created manually in this interface have to be imported into the Saxion database. QMP 6.0 will be released in a 0 version in summer 2013. Stoas does not generally recommend switching to the 0 version. A later version, to which switching is recommended, will not be available before summer 2014.

Questions are created for version 6 in what is referred to as the Authoring Manager. This client application is installed locally and the data are on-line in the Saxion database. The Authoring Manager’s interface is intended for experts and/or more advanced instructors. The application works with wizards for standard questions and a question created using a wizard can be edited in the Editor. Editing using the QuestionEditor is complicated and irreversible. A question once edited in the Editor can no longer be revised in the Wizard. The Authoring Manager has all the tools for developing questions, tests, question content, saving multimedia files, adaptive testing and making workflows accessible for all users. That makes the tool complicated for not very advanced users. See also https://www.questionmark.com/us/perception/Pages/authoring_windows.aspx.

Mathematical symbols can be inserted using a MathML editor. Mathematical formulae are displayed as an image in the question with this editor, but can still be revised in MathML. Mathematical formulae cannot be recorded in the answers. Because MathML editor works with images, QMP’s possibilities are regarded as insufficient for Saxion in this respect by those present.

There are sufficient tools available for using multimedia in test questions.

Other types of questions exist, but only according to the fixed structures as recorded in the application. There are a lot of different types of question, although a separate licence must be purchased for a number of question types. However, several types of question cannot be combined in a single question as in Maple TA.

Surpass: There is a tool for creating Multiple Choice, Multiple Answer and Yes/No questions easily from MS Word and MS Excel and importing them into Surpass. It will also be possible in the future to create questions with blanks to be filled in this way. Metadata, level of difficulty and a link to multimedia can be included immediately using this import.

There is also a client application, called content producer, in which experts can create questions. This client application is relatively complicated to use for advanced instructors. In order to have less advanced instructors create questions, question templates can be constructed in the content producer. A less advanced instructor can also construct questions easily based on a one-off template, because a number of items are already covered (think of scoring rules, rules to be applied to the test matrix, construction and layout of the question). BTL is developing a new instructor interface, which will need to be available in the summer of 2013 alongside the interfaces that already exist.

Both are possible for mathematical questions. A MathML editor is also available here.

Question tools are also available for using charts (lattice chart, column chart, points chart) as an answer and for using tables in the answer. There is also the option of having the application calculate a composite value based on an answer previously given. This means that if a step at the beginning goes wrong in the calculation, but subsequent steps go well, points are automatically given anyway for the subsequent steps. Hence the functionality that exists in Surpass for mathematics, physics, chemistry and technical use seems very reasonable. Whether this is sufficient for more technical academies, only a detailed examination by an expert will be able to show. Mathematical formulae cannot be used in answers.

It is possible to use multimedia extensively in Surpass for the questions and answers. For example illustrations, video, tables, sound, animations in Flash, charts and autoshapes. You can see a few examples of the use of multimedia in the video fragment http://saxion.hosted.panopto.com/Panopto/Pages/Viewer/Default.aspx?id=626df95d-65bc-4f3a-90e5-cf878723e7ee, where a BTL employee among others demonstrates a few test questions.

Because BTL does not work using types of question, but only using tools with which types of question can be developed, 120 types of question are possible in the application.

Result Maple TA QMP Surpass 1 Importing test questions

from MS Word or MS Excel

1 Creating test questions easily in the interface

1 Capacity for recording mathematics elements/formulae in test questions and test answers

1 Use of multimedia in test questions and test answers

1 Use of other types of test question

Scenario 11: Test matrix

Procedures It has been agreed in the first instance that we examine whether there are enough tools in the 3 test applications to compile a test matrix that pulls questions into a test application based on the theoretical test matrices of AMM and ABR from BISON and to develop test versions (examples of test matrices in Appendix 2). In the second instance, we would then examine whether there are academies that have a test matrix that differs from the test matrices of ABR and AMM to such an extent that we need to examine whether there are tools in the application for constructing those test matrices in the test application. On the basis of this analysis, we would then be able to identify which application provides the best test matrix with the minimum risk.

A test matrix in Saxion’s terminology is not the same as the possibility of compiling a test in the test application. A test matrix provides a basic matrix that compiles several different tests for several different exam times from the same imported principles and rules, while all the tests taken get the same test for each exam time (and have therefore had the same questions for the analysis). In other words, the digital test matrix in the test application makes it possible in Saxion’s view to generate with one click a version of the test based on the same test matrix, whereby all students get the same questions.

Questions 1, showstopper

Use of a test matrix

3 Available questions that meet the conditions, visible in the test matrix 1 Use of metadata for compiling tests from the test matrix 1 Use of folder structure/learning tree for compiling tests from the test matrix 1 3 ways of randomising are possible in testing 1 Suitable for formative testing

2 Able to separate summative and formative test questions adequately 3 Suitable for adaptive testing

Implementation Maple TA: Tests can be compiled in Maple TA from sets of questions pulled at random, or on the basis of questions inserted into the test. Questions in the test can also be put in random order as well as alternative answers for each test question. A test is therefore either compiled manually and the questions are then the same for all individual participants, or the test is individually generated at random from the available questions on the basis of each student’s start-up and, as a result, the test questions are different for each individual taking the test. It is not possible to create a test matrix with which a new version with the same questions is generated for each group of students or chance, as is used for Saxion.

No home-grown metadata can be stored in Maple TA for a test question. Tests are compiled from the folder structure for a class. The application is suitable for formative testing and also has specific forms of testing for formative testing. Feedback can be given especially for formative testing on a right answer and a wrong answer to the questions.

The only way of separating summative and formative test questions is based on split classes or split folders in the learning tree or folder structure. Questions can then still be drawn in a test in that case. Maple TA contains tools for adaptive testing.

Question Mark Perception:

Tests can be compiled in QMP from sets of questions pulled at random or on the basis of questions inserted into the test. Questions in the test can also be put in random order as well as alternative answers for each test question. A test is therefore either compiled manually and the questions are then the same for all individual participants, or the test is individually generated at random from the available questions on the basis of each student’s start-up and, as a result, the test questions are different for each individual taking the test. It is not possible to create a test matrix with which a new version with the same questions is generated for each group of students or chance, as is used for Saxion.

No metadata can be stored in a test question for QMP. Test questions can be pulled at random based on folders, types of question and those metadata stored in the test questions. It is not possible to pull test questions at random on the basis of psychometric empirical data, such as the level of difficulty.

QMP has specific forms of test for formative and summative testing. You may separate test questions for formative and summative testing using the folder structure and metadata. The task of separating these, however, still needs to be done manually. Feedback can be given especially for formative testing on a right answer and a wrong answer to the questions. Feedback can also be given on each alternative answer.

QMP has tools for adaptive testing. These work with jump blocks. This means that, based on the results of a block of questions, the participants can be sent to a new block of questions. If the result is positive above a certain limit, the participant is sent to a block that includes more difficult questions

for example. If the result is below a certain limit, the participant can be sent to a block that includes easier questions, etc. Creating adaptive tests such as this is highly labour-intensive because the test developer has to consider all the test paths including compatible test questions and set them up.

Surpass: Exams can be created in Surpass. You can create several exam versions from one exam. If an exam is defined as a test that pulls test questions at random from folders or on the basis of metadata, an exam such as this is a test matrix. Test questions can be pulled in the test matrix based on the type of question, certain metadata in a test question, location in the folders as well as on the basis of empirical data such as a question’s level of difficulty (p value). Several rules can be combined transparently combined for selecting the right set of questions. The interface shows the questions that are available on the basis of the rules compiled for selecting questions at random.

Several test versions can be generated on the basis of this exam and are saved as a separate test version that includes fixed questions. The test questions that are available for pulling and that are ultimately inserted into the test are clearly visible when generating the versions. These test questions can also be seen in the interface. The tools for pulling test questions at random in a test (matrix) are the most extensive in Surpass of the 3 applications.

These exam versions can be scheduled and students can be enrolled on them. It may be specified that test questions be displayed in random order in both the test matrix and in the test versions. It may be specified that alternative questions (e.g. a Multiple Choice question) be displayed in random order in the questions.

Formative testing can also be created in Surpass. The questions for this formative testing are difficult to separate from the questions for summative testing. This would have to be on the basis of metadata or place in the folder structure. These are tasks to be done manually.

Surpass does not yet have tools for adaptive testing.

Result Maple TA QMP Surpass 1 SS

Use of a test matrix

3 Available questions visible in the test matrix

1 Use of metadata for compiling tests from the test matrix

1 Use of folder structure/learning tree for compiling tests from the test matrix

1 3 possible ways of randomising in tests

1 Suitable for formative testing 2 Capacity for separating summative

and formative test questions adequately

3 Suitable for adaptive testing

Scenario 12: Preparing to take a test

Procedures The group discusses the workflow before preparing a test. There are different roles for each academy and the workflow is different. We display and analyse the most elaborate situation here. There are several academies. The roles and workflow are not always the same. The academies are at different stages of development as regards the organisation of testing. That is why we analyse the most complicated organisation.

Roles Task

Test administrator Point of contact for validity, reliability and acceptance of the test: test matrix. Test analysis and statement of results

Instructor Responsible for (part of) the test questions, review

Examining board Quality assurance of the test

Test desk Digitising test matrix and generating tests from the test matrix

Rostering and planning Rostering test and location

Central exam office Coordinating implementation of the tests and enrolment of participants

Invigilator Taking the test

Expert Explanation of the analysis of the test and test questions

IT centre Technical management and infrastructure support

ICT&O functional administration

Application management and support

Establishing test matrix Test administrator and

Exam board

Creating and reviewing test questions

Instructors

Digitising test matrix Test desk

Generating test from

test matrix Test desk

Carrying out test analysis

Test desk

Rostering P & R

Test schedule Exam office

Testing Exam office and

invigilators

Test analysis Test administrator and education

expert

After the test matrix has been digitised by the academy’s Test Desk, versions are generated from it. If such a version has been reviewed and approved by the exam board, the academy prepares for this test to be taken. The central exam office receives a report from test desk of the academy that prepares the test. The exam office links the date to the test and also registers the students enrolled for the test in the student information system prior to the test. Only the academy’s test desk and the exam office have the right to administer those enrolments. The instructors do not. If a student still needs to be enrolled for the test after the enrolment period has closed, the test desk or the exam office will need to be able to do this quickly for an individual student.

There is an acknowledged group of students to whom extended exam time needs to be granted. Those students are awarded a 15-minute extension for a test lasting 60 minutes.

Questions 1, showstopper

Does the exam office or test desk have an adequate procedure for logging students on for a test?

1 Fast enrolment is possible 1 Capacity to read in logons automatically from a source application. 1 Easily possible to schedule in students with an extended exam time. 2 Application automatically assigns extended exam time to students flagged up for this.

Implementation Maple TA: Test is prepared in the class. Students can be imported using a CSV file in the application. Students are allocated to the class in which the test takes place. All the tests in a class are visible to all the students in the class. Saxion’s procedure is possible in practice. However, the procedure is not only to be assigned to the exam office or the test desk. Only a system administrator or someone who is allowed to import students creates students in the system. But every instructor can enrol students for a class and log them in for this test in the system. Instructors who have developed the questions

and have had to review the test can also log students on. Hence the procedures are similar to those familiar from Blackboard.

Logons can be read in automatically (see also scenario 8). However, the group for testing is the same as the group enrolled for a test. That is not the same as a class. The class structure makes the Saxion procedure difficult. Individual students can be enrolled even later for the class and therefore for the test.

A single student cannot be automatically granted an extended exam time. If a single student has an extended exam time on a test, all the students enrolled for that test will either have extended exam time or a separate test will have to be prepared for that student with extended exam time. That would have to be different again in a separate class where all the students have the same test available twice with all the attendant problems.

Question Mark Perception: Students can be automatically read in with QMWise (see also scenario 8) and assigned to groups. The groups can then be logged on for the tests. By groups we do not mean roster groups here, but the group of all those enrolled at a certain exam time. The right to schedule testing and enrol groups for testing can only be assigned to the academy’s test desk and to the exam office.

Fast enrolment is possible. Extended exam time can be granted, but not automatically. Extended exam time must be granted for participants needing extended exam time by subsequently editing the participant’s schedule and allocating a different start and end time.

Surpass: Students can be automatically read in (see also scenario 8) and assigned to groups (referred to as cohorts). These cohorts can then be assigned to the tests. By groups we do not mean roster groups here, but the group of all those enrolled at a certain exam time. The right to schedule tests and enrol groups for tests can only be assigned to the academy’s test desk and to the exam office. Individuals can be enrolled for the test at the last minute.

The system can specify whether someone taking a test needs this extended exam time and if so, how much.

You can define how long the extended exam time is in the test.

This extended exam time is then assigned automatically for this participant for all exam times.

Result Maple TA QMP Surpass 1 SS

Does the exam office or test desk have an adequate procedure for logging students on for a test?

1 Fast enrolment is possible 1 Capacity to read in logons automatically from a source

application.

1 Easily possible to schedule in students with an extended exam time.

2 Application (automatically) assigns extended exam time

Maple TA QMP Surpass for students flagged up for it.

Scenario 13: Large-scale testing

Procedures The performance of the application is important when there are large numbers of people taking the test at the same time. It may be necessary in the long term for Saxion to be able to start 500-700 tests at the same time. All suppliers are asked to carry out a performance test in the presence of IC Saxion staff and ICTO Functional Administration. The conditions under which the test takes place are important for any performance test. 3 elements are of interest here:

• How has the technical infrastructure been set up? Is there 1 application server or more? What was this server’s capacity?

• How has the test been compiled? Is it a test with only a few MC questions or several types of question? If so, how many questions are there and of what type?

• Is it one set of test questions selected at random per student or a fixed set? • Does the performance test take place on a full database (with previously created tests and

results per student) or on an empty database? Virtually every test system performs well with an empty database.

In the event of problems at the exam time itself, solutions that are as effective as possible must be available to be able to continue the test still in progress.

Questions 1, SS Are 500-700 simultaneous tests possible? 1 Paper printout 2 Solutions in the event of disasters 1 Application ensures that a test being taken is always finished.

Implementation

Maple TA: Maple TA always works on the basis of a centrally installed server. A (secure) browser is used on the local workstation as a tool for displaying the test, but the central server does all the work. At the time a test starts, the test is compiled on that server for each participant and then the first test question is displayed on the local workstation in the (secure) browser. If the test question is filled in and the participant moves on to the next test question, the test question is saved on the central Maple TA server and the next question is retrieved by the same central server and again displayed on the local workstation. This means that all the Maple TA program tasks are carried out on a central server. Disasters may occur if the central server is unreachable or unreachable in time. If there is a disruption on the Internet, a test that has started is disrupted. Students are no longer able to save answers or retrieve new questions. Only if a paper copy has been produced can you proceed with the test on paper. Maple TA has no print option to MS Word or another word processor, but does enable screen printing.

Because for open questions the answer is only stored if the participant goes on to the next question, a time out may occur in the event of a long open question (where the student has to give a long answer), whereupon the student is thrown out of the test. The answer given to the last question up to that point is not saved.

The student closes a Maple TA test if he or she clicks on ‘Grade’. Only at that moment does the test become available for assessment. If the length of time is fixed for an exam, a student may choose ‘Save & Quit’, but the exam time will continue. At the end of the exam time, the test is automatically closed for assessment.

Maple TA did not want to carry out any performance test for Saxion. It appears from the completion of the questionnaire by Delft Technical University and college contacts that Delft Technical University is busy carrying out a number of simultaneous tests on correctly scaling the application. This is in collaboration with the supplier. It seems from our contact with each other that the environment is still performing for 480 simultaneous tests, but not for 520 simultaneous tests. The infrastructure used is unfamiliar. Maple TA claims that 500 simultaneous tests are possible on 1 server. How those tests are compiled is not known. It is not known whether the database was full. Due to the lack of performance and scaling data (concerning the infrastructure of the hardware), we have to reject the application for this scenario.

Question Mark Perception: QMP works on the basis of a centrally installed server. A (secure) browser is used on the local workstation as a tool for displaying the test, but the central server does all the work. At the time a test starts up, the test is compiled on that server for each participant and then the first test question is displayed on the local workstation in the (secure) browser. If the test question is filled in and the participant moves on to the next test question, the test question is saved on the central QMP server and the next question is retrieved by the same central server and again displayed on the local work station. This means that all the QMP program tasks are carried out on a central server. Disasters may occur if the central server is unreachable or unreachable in time. If there is a disruption on the Internet, a test that has started is disrupted. Students are no longer able to save any answers or retrieve new questions. Nor can a paper printout still be produced at that point in time because the central server cannot be reached. Only if a paper copy has been created can you proceed with the test on paper. QMP has options for printing a test on paper. If the exam time has expired, the answers are automatically saved for assessment, irrespective of the question as to whether the student has closed the test correctly.

If the Internet is never available in a test location, the test can be taken offline using a separate ‘Perception to GO’ licence. See https://www.questionmark.com/ned/perception/Paginas/p2g.aspx .

Because for open questions the answer is only stored if the participant goes on to the next question, a time out may occur in the event of a long open question (where the student has to give a long answer), whereupon the student is thrown out of the test. The answer given to the last question up to that point is not saved. Stoas also recommends ending each test with a final page instead of with the last question, thus also ensuring that the answer to the last question is always stored in the database.

Stoas has carried out a performance test in the presence of 2 people from Saxion in Wageningen. This test failed. A second test was subsequently carried out at Stoas using the same parameters, with no Saxion staff present. A separate test report on this test is attached. This test took place with the following configuration:

• 2 application servers (2 QABS) • 4 presentation servers (4 QPLA)

Our main conclusion is that all 800 competing participants/students receive the tests containing 80 questions on the screen within 4 seconds, as long as the participants’ logins are spread over 5 minutes. The fact that spread (alone) is important is evidenced by the fact that if 700 competing students with a login spread of 3 minutes take the test, there is a maximum response time of 33 seconds. 13 of the 800 tests went wrong on 3 presentation servers. The test consisted of 1 block of questions with 80 MC questions selected at random, including illustrations in a number of questions. The stress tests referred to are carried out with an empty database. As soon is the database is full, the response time increases by 50%. The conclusion is that there remains some uncertainty as to whether the numbers of simultaneous tests that Saxion wants to achieve are feasible with QMP. It is difficult to spread the start of a test over 5 minutes if students start a test at several locations on several test premises. If the database is full, the response time increases quickly too. Many (6) servers are needed for some sort of satisfactory performance result. Surpass: Surpass works in a different way to QMP and MapleTA. A secure browser is not used on the local workstations alongside a central Surpass server, but a secure client application is. At the time the test starts in this local client application, the entire test has been downloaded with all the questions. If the test has started, the local client provides the test questions for submission to the student. The answers are initially saved in the local client’s database. The local client synchronises the answers given with the central server every 30 seconds. The program tasks are therefore shared during a test between the local clients and the central server. If the exam time has expired, the answers to the test are automatically saved for assessment, irrespective of the question as to whether the student has closed the test correctly. Surpass has options for printing a test on paper. If there is no Internet connection on the test premises, a test can be taken offline with Surpass within the same configuration and licence. If there is a disruption on the Internet, an Internet test that has started will not be disrupted. All the questions are nevertheless downloaded to the local secure client and the answers are saved there. The student is usually able to complete the test. In the event of a disruption to the Internet, the answers can only be synchronised if the disruption to the Internet is over. The answers are then saved encrypted in the local client. However long an open question may be, it a time out cannot occur because synchronisation takes place every 30 seconds and activity can therefore be measured. Surpass has carried out a performance test in the on-line presence of 2 Saxion staff members using teamviewer. A single application environment was used, a so-called ‘silver’ installation (silver covers the capacity of the application and database server or the service level; there is also a gold level) and a test containing 40 MC questions, each answered within 1.5 minutes, 8 of which use video (4) or an illustration (4). It was a fixed set of questions. The database was full of test data. 2 tests were done. One with 750 and one with 3000 participants taking the test at the same time. Conclusions: The number of participants made no difference to the response times. The response time is still limited to 1 second. Login does not need to be spread out. Only the database server was loaded with more users at a higher level. The results of the performance tests are attached.

Result Maple TA QMP Surpass 1, SS

Are 500-700 tests possible at the same time?

1 Paper printout 2 Solutions in the event of disasters

1 Application ensures that a test taken always ends.

Scenario 14: Supervising testing

Procedures The exam office and the academy test desk at 2 academies supervise testing. They enlist invigilators. In the meantime, they want to be able to monitor the progress of the test(s) taken by the various participants, both by class and individually. In doing so, it is useful if students also have a digital complaints form for the test and inspection of the test.

Questions 2 Functionality for (remotely) monitoring testing. 2 Functionality to enable testing as part of the individual pathway. 2 Tool for students to be able to provide feedback/make complaints when taking the test

and during inspection of the test.

Implementation Maple TA: Monitoring the active users is limited to one test. Active users can be seen in the class, but not whether they are involved in a test. The test can be taken individually with the help of the proctor. The proctor has to log into the test at the local station itself before the student is able to open the test.

There is no facility for giving students the opportunity to raise complaints.

Question Mark Perception: Monitoring during tests is not possible. Active users are not visible. QMP has the Monitor function. The monitor has to log into a test at the local station before a student is able to take the test. Feedback or complaints cannot be displayed at question or test level during testing or inspection.

Surpass: Monitoring during tests is not possible. There are a lot of statuses:

Locked Test ready for the participant, but is still not visible to the participant, e.g. because the test is scheduled for later.

Locked for invigilator

The test is ready for the participant and if the invigilator logs in to the test, the participant is able to take the test

Locked by Pin If the participant types in the access code for the test, the participant is able to take the test

Ready The test is ready for the participant. In progress The participant is taking the test. Paused The invigilator has temporarily paused the exam for this participant. Awaiting upload The test has been downloaded so that it can be taken offline. The answers need to

be uploaded.

Finished The participant has ended the test. Voided The participant has not taken the test and is being withdrawn from the test. Ready for download

The test is ready to be downloaded for an offline test.

Surpass has the invigilator function. The invigilator has to log in to a test before a student is able to take the test. The invigilator can also do this remotely. Feedback or complaints cannot be displayed at question or test level during testing or inspection.

Result Maple TA QMP Surpass 2 Functionality for (remotely) monitoring

testing.

2 Functionality to enable testing as part of the individual pathway.

2 Tool for students to be able to provide feedback/make complaints when taking the test and during inspection of the test.

Scenario 15: Test requirements

Procedures See http://www.accessibility.nl/nieuws/2012/10/wcag-2.0-is-iso-standaard . We check whether participants with a disability are able to make 3 test applications.

Questions 1 Functionality for students with limitations with regard to enlarging typefaces 1 Functionality for students with limitations with regard to enhancing contrast 1 Functionality for students with limitations with regard to reading aloud 1 Functionality for students with limitations with regard to dictation (Dragon) 1 Service using a laptop 1 Secure availability of tools for a test, such as a calculator or exercise paper. 1 Compliance with the regulations on accessibility.

Implementation

Maple TA: Maple TA’s black and white layout is simple. The browser zoom options work in the application. However, the application is not specifically equipped for working accessibly. For a screen reader such as Jaws, go to: http://www.maplesoft.com/support/faqs/detail.aspx?sid=32659 On overall accessibility, see: http://www.maplesoft.com/support/VPAT.html . The Dragon operation, for example, has not been tested on Maple TA, but it does seem feasible.

Maple TA works via the Internet and a browser and can also therefore be served using a laptop.

No tools such as a calculator can be added to the questions in the application.

Question Mark Perception: contains options for improving the accessibility of assessments and aligning them to the needs of participants with a limitation. When a test is being taken, QMP displays

optimised HTML for assistive technologies such as screen readers. In addition, assessment templates contain control elements with which participants can adjust the font size and contrast.

There are options for navigation through assessments using the keyboard and/or alternative equipment for participants who are unable to use a mouse. The Dragon operation, for example, has not been tested on QMP, but it does seem feasible. QMP meets the WCAG accessibility standard. See https://www.questionmark.com/ned/perception/Paginas/whats-new-v5.aspx

QMP works via the Internet and a browser and can also therefore be served using a laptop.

Tools such as a calculator can be added for a test question. The user is free to define the tools to be added.

Surpass: BTL has a separate accessibility guide: http://www.btl.com/surpass/accessibility/.

Larger typefaces can be set using the browser. When taking a test, participants can opt for all kinds of preferences by selecting ‘Preferences’ so that they can see screen better. For the visually impaired, Jaws is recommended for reading texts. Surpass itself will also allow questions to be read out loud digitally.

Surpass operates the same on a laptop as via a fixed network.

Surpass operates as much as possible on the basis of generally defined standards for making the software accessible as well. Surpass has been given the ‘Accredited Plus’ award after being tested by people with various disabilities. Surpass meets WCAG 2.0 AA. The Dragon operation, for example, has not been tested, but does seem feasible.

Tools such as a calculator can be added for a test question. The user is free to define the tools to be added. These tools are provided securely.

Result Maple TA QMP Surpass 1 Functionality for enlarging typefaces for

students with limitations

1 Functionality for enhancing contrast for students with limitations

1 Functionality for reading out loud for students with limitations

1 Functionality for dictation (Dragon) for students with limitations

1 Service using a laptop 1 Secure availability of tools for a test, such

as a calculator or exercise paper.

1 Compliance with the regulations on accessibility.

Scenario 16: Multilingualism Implementation

Procedures Dutch and English must be available or included in the Help function. If there is still no multilingual version available, but the roadmap states that it is on its way, the issue is highlighted in yellow. The language is not assessed in terms of quality. A distinction is made between the development environment and testing environment. The languages in the testing environment are more important than in the interface for administrators and test developers. In the testing environment we also look at whether answer buttons change in the language of the interface. We also investigate whether the option of different language versions has been built into the software. If this is possible, it is an advantage, even though the language is not yet available.

When creating questions, we look at whether 1 question can be imported in several languages without it turning into separate questions. This would mean that the question appears to depend on the user’s chosen language in the test. Therefore if a user has set the language to Dutch, the question appears in Dutch and if the user has set the language to English or German, the same question appears in English or German, provided, of course, that the questions have been drawn up in these languages.

Questions 1 Available in English 1 Dutch 2 Multilingualism

Implementation According to the supplier, Maple TA is available in 3 languages. Quotation from the CAN diensten supplier’s e-mail: “As far as languages are concerned, only English, French and Chinese are currently available and I don’t think that Dutch or German will be added within the foreseeable future.”

Question Mark Perception is multilingual. Both the testing and the development environment are multilingual. A user can select his or her language in the web-based environment via a dropdown box in the top right-hand corner.

Top right-hand corner

Question Mark also has what is referred to as a Translation Management System. This is not included in the standard licence. A separate licence is needed for this. This makes it easier to provide multilingual questions. See also https://www.questionmark.com/ned/perception/Paginas/tms.aspx.

Surpass has only been rolled out in England until recently. They are only now making the transition to Europe with a Dutch representative company. There is therefore no Dutch and no German version. The program itself, however, is multilingual. The Dutch version announced for the total Surpass environment is being brought in as a new interface. This version is on the roadmap for summer 2013. (See also http://saxion.hosted.panopto.com/Panopto/Pages/Viewer/Default.aspx?id=44d70c15-6705-43c8-8cac-6e1422a21bbc ) It is not clear whether a German version will therefore be or have to be available for the entire program for Saxion.

Above image of language options applies to the testing environment. The Dutch and German glossaries are still not available at present for the testing environment. Gaelic and Cymraeg (Welsh) are available in addition to English. It has been agreed that Saxion will provide the Dutch and German glossary for the testing environment. When the testing environment becomes multilingual will therefore depend on Saxion.

Result

Maple TA QMP Surpass 1 English 1 Dutch

2 Multilingual (German)

Scenario 17: Test inspection

Procedures Generating test feedback on a test. When and how will you be able to see this?

Questions 1 Test inspection functionality 2 Possible to inspect the test, which will help the student learn from what went wrong

and, on the other hand, create minimum risk of making the test questions unusable because they are familiar to students.

2 Tool for students to be able to provide feedback/make complaints when taking the test and during inspection of the test.

2 Test can be inspected from home.

Implementation Maple TA: Maple TA offers the user many opportunities for providing feedback on a test, both during and after the test.

During the test, the user sees a ‘how did I do?’ link showing the feedback.

After the test has come to an end, a participant can look at a test result in the grade book.

Whether the inspection tools are sufficient for enabling the student to learn from his or her mistakes, but without giving away the questions on the other hand, remains somewhat uncertain for the summative secured tests. That cannot be postulated with complete certainty during the tests because there is no real practical experience. Test inspection is web-based and therefore accessible anywhere. There is no tool for writing down complaints on the questions during inspection of the test.

Question Mark Perception: QMP offers the user many feedback options. No feedback can be provided during the test. The participant can see feedback immediately after the test. It can indicate whether a pass/fail score is also available.

The participant can look at a coaching report for a little longer after the test. Saxion is itself able to decide what is and what is not shown to the participant in that coaching report.

This means that students can inspect the test. Whether the inspection tools are sufficient for enabling the student to learn from his or her mistakes, but without giving away the questions on the other hand, remains somewhat uncertain for the summative secured tests. That cannot be postulated with complete certainty during the tests because there is no real practical experience.

Test inspection is web-based and therefore accessible anywhere, even from home. There is no tool for writing down complaints on the questions during inspection of the test.

Surpass: In Surpass, you can also define in all kinds of ways how feedback on the test should be given.

Feedback can be provided during the test. (Show feedback button during exam). Immediately after the test has come to an end, you will see several tabs including feedback on a screen:

• Exam result: Pass/fail with score in % • Feedback tab for each question (see below) • Feedback tab where the questions are ordered per competency/learning objective (display

feedback by reference, not in illustration below)

Clicking on a question indicates whether the answer is right or wrong. The correct answer is not listed. Feedback appears on a separate information screen. This feedback only appears immediately after the test becomes available.

At a later point in time after the test, there is an inspection option in the form of a report. Whether the inspection tools are sufficient for enabling the student to learn from his or her mistakes, but without giving away the questions on the other hand, remains somewhat uncertain for the summative secured tests. That cannot be postulated with complete certainty for the tests because there is no real practical experience.

There is no tool for raising complaints when accessing the test. The application is web-based and the test can therefore be inspected, even from home.

Result Maple TA QMP Surpass 1 Test inspection functionality 2 Possible to inspect the test, which will

help the student learn from what went wrong and, on the other hand, create minimum risk of making the test questions unusable because they are familiar to students.

2 Tool for students to be able to provide feedback/make complaints when taking the test and during inspection of the test.

2 Test can be inspected from home.

Scenario 18: Report on test questions, tests and testing Implementation

Procedures There are 2 types of report. Reports that you can create and need immediately in real time during the operation and reports that do not need to be available immediately. The performance of testing is very important in a summative test application. Because the creation of reports on large numbers of results selects performance from many users, many reports are often produced using a reporting tool that works based on a copy of the database. A new up-to-date copy of the results database is always produced at night. Users are then able to use reports as required without disrupting the testing process.

We check whether a number of reports are available in real time:

• Overviews of available tests • Overviews of students logged on for tests • Overviews of questions in a test • Overview of questions in the tree structure • Overview of psychometric values of questions in a test and in the tree structure • Overview of the students’ results for a test.

We also check whether sufficient reports are available afterwards and to what extent Saxion itself has influenced the provision of the reports.

In this scenario we do not check whether we have all the reports listed, but we mainly check whether we have confidence that they will be available.

Questions 1 Flexible reporting tool 1, showstopper Reporting tool available

Implementation Maple TA The report herein is very limited. The reports requested are in real time. The reporting unit, however, is the class. Usability is therefore reduced. The question overview shows which questions are used in which tests. Folders with questions, however, are stored per class.

Psychometric values of the questions, such as p value and RIT value, are visible, but only for the test analysis, not apart from a test. There is a Grade Report in which students’ results from several tests are alongside each other. This Grade Report can be exported to Excel.

These reports are thus generated directly on the database. This can influence performance when the tests are being taken. There is no genuine reporting tool and certainly no flexible tool with which reports can be independently set up on the database.

QuestionMark: All real time reports required in real time are available. A document is created from these directly available reports. This document states which data are available and how they can be viewed in the application. This document is provided in the ICTO Saxion community > SIG testing >

Selection process test application> Map Scenario 18 reports. These reports can themselves be set up within fixed limits by Saxion or the academy.

These reports are real-time reports and it is well-known that they sometimes influence the performance of the database. It is not known whether these reports will remain available in this way because Question Mark is moving its focus toward a different method of reporting. QuestionMark Analytics has recently become available for the more complicated reports. This works on the basis of a separate database that is updated every night with the latest data.

The reports are fixed in the main and can only be set up within certain limits. Some of these data can be selected. For further information, see:

https://www.questionmark.com/ned/perception/Paginas/reporting.aspx and

https://www.questionmark.com/ned/perception/Paginas/reporting_analytics.aspx .

Surpass:

The analysis of the test is available in real time. It is convenient that the available test questions for a specific section can be retrieved in the test matrix, including a number of psychometric values, such as level of difficulty, number of times used and other metadata. There are real-time reports on exam schedules, student access codes, students’ exam results, what answers are given for each question, etc. Tests are also directly available from the application for printing on paper prior to testing. Overviews can be created of the tests, which test a specific competence.

The real-time report on testing when the test is being taken is very convenient in Surpass. The status of what the student is doing can be clearly seen in the report and can be directly influenced.

Surpass Analytics has subsequently been developed for reports. It includes a number of standard reports and you can compile your own reports. A document showing its implementation can be found in the ICTO Saxion community > SIG testing > Selection process test application> Map Scenario 18 reports.

Surpass can do more than QMP in terms of reports. The reports available in QMP, however, are more than satisfactory. Both Surpass and QMP are satisfactory as regards reports, Maple TA is not. The option of being able to generate one’s own reports, however, is an advantage of Surpass, especially in view of an unpredictable future. We do not know what we shall need then, but Surpass already has more to offer.

Result Maple TA QMP Surpass 1 Flexible reporting tool 1 SS

Reporting tool

Scenario 19: Analysis

Procedures Analyses can be provided in real time or with a specific processing time. We want to be able to retrieve the analysis on the test in real time. The analysis on tests (over several tests) or other advanced analyses do not need to be available in real time.

The values that the analyses need to include are equal to the current analysis that is provided from TestVision. The values analysed must also be able to be exported to other applications (for example Excel, Access, SPSS, etc.). We come to agreements about the test on which the analysis is carried out.

Questions 1, showstopper

Availability of test analysis data

1 showstopper

Availability of analysis data for the test questions

3 Reading back scores on questions and test level of Print & Scan tests

Implementation Maple TA: In addition to the score, there are no other values available on the test as a whole. For each test, the p value, RIT value and RIG value are available for the questions. Other values are not available. All reporting is in real time. There are no other reports. The analysis data are not visible when compiling the test. Analysis data of the questions for each test can be exported to CSV.

To read back scores from Print & Scan to Maple TA, custom-made work is needed.

Question Mark Perception: There are real-time reports with the conventional test analysis data for each test and for the different questions for each test or on back testing.

QuestionMark Analytics also provides a subsequent analysis report. (See https://www.questionmark.com/ned/perception/Paginas/reporting_analytics.aspx ) The analysis data are not visible when compiling the test. They can be exported to CSV and to TXT.

To read back scores from Print & Scan to Maple TA, custom-made work is needed.

Surpass: There are real-time reports with the conventional test analysis data for each test and for the different questions for each test or for questions on back testing. Surpass Analytics also provides subsequent analysis reports.

When compiling the test, analysis data are visible as measured on all previous testing of those questions.

They can be exported to CSV. To read back scores from Print & Scan to Maple TA, they need to be customised.

Result

Maple TA

Surpass QMP

1 SS

Availability of test analysis data

1 SS

Availability of analysis data for the test questions

3 Reading back scores on questions and test level of Print & Scan tests

?? ?? ??

Scenario 20: Assessing caesura, scores and results

Procedures Ida sends a test with open questions. We will look at all the items for checking open questions in the applications. In this scenario, it is important how the scores to questions are produced and whether they can be changed. Finally, it is also important whether and how you move from a score to a figure or result that is imported into BISON. Results in the Netherlands are displayed with figures and percentages and not in letters, as in the US. The caesura has to be imported in percentages or in points. The final figure has to be rounded off. We have to consider here when and how figures are rounded off and whether decimal points are kept behind the scenes or whether the rounded-off figures are used for the calculation.

Ben e-mails all those concerned with a number of scoring methods and caesura criteria, including the instructions and those used for Saxion. Ida and Alexander look at these and provide comments/supplements.

It must be possible to obtain a question from the scoring. It must also be possible to change a question’s score for everyone. Correct answers to questions must be able to be adjusted after the test ends. The people who are allowed to adjust the scores in the application do not have a fixed role because this is different for the academies. The authorisation for this must be linked to a role that can be linked to individual accounts.

The possibility of exporting results that can be read into another application is also being considered.

Questions 1 Tool for scoring/checking open questions 1 Tools for scoring questions again per student and per test 2 Report for the caesura criteria 3 Exporting results enables batch processing of results in BISON.

Implementation Maple TA: The test used is one with a case (casus). Only questions can be added. A case (casus) or description cannot be inserted as a separate item. This then becomes a separate question or the text needs to be inserted into each question. There may be several questions on 1 page, but this is then prepared for the entire test. The answer to the question is only saved at the time that the person taking the test moves on to the next question. If a long text has to be completed, this means that it may be timed out and the student will be thrown out of the test (experience with MIM).

There is a tool for scoring open questions.

There is 1 text field for comments, but no form or rubrics therein. Scoring is in figures and not in %. Scores can only be adjusted per question and per student. You may adjust the score for 1 question for all students, but the person marking the test would then have to select for which students the score will be adjusted. There is therefore no automatic processing of this adjustment for all the students affected on a specific question.

Filling in a caesura may indicate when a student is successful only by the number of points. The score includes decimals. There is 1 way of establishing caesura and that is an absolute caesura in points. The correct answer to a question cannot be adjusted after the test has been taken. Maple TA provides scores on test questions and a final score. An instructor may on the basis of this score arrive at figures or results in very many different ways. This kind of conversion table is not usually included in test programs because the diversity is too great. (In TestVision, this calculation is included in an Excel export and TestVision itself produces scores). The scores can be exported to a CSV. This file contains little information, only the participant data and the final score.

Question Mark Perception: There may be a description/case (casus). The answer to an open question is stored if the participant moves on to the next question. To avoid the last question of a test not being stored, Stoas recommends inserting a final page after the last question in the test. A scoring model may be designed for an open question, including rubrics, if desired.

The same scoring model is referred to for several open questions. The model must be linked to the question referred to. When scoring the open question, different tools such as the rubrics, feedback and scoring can be filled in. The tools are differentiated with tabs. This does not make it entirely clear.

Scoring is in points and not in %. Filling in a caesura should be stated in % when a student is successful (with no decimals). There is 1 way of establishing an absolute caesura in %. The score is not in decimals. Re-scoring may occur per student. The archive then similarly shows that there has been a deviation from the original score and this can also be seen in the figure displayed.

The roles are used to determine those accounts in which participants may be re-scored and in which groups and tests.

april 5, 2012 = 5 April 2012

* this result has been modified manually

QMP had a separate Results Management System (RMS) for adjusting the score of a question for all students simultaneously. A separate licence was needed for this. The system works because the students’ scores are exported to another database and the score is edited here. The edited score is saved in this separate system and QMP’s results database still contains the original score (including any manual adjustments on individual students). Question Mark will be replacing the RMS with a QMP-integrated system. The licence for the RMS is therefore no longer available just like that and the current system is not being developed any further from now on.

QMP provides scores on test questions and a final score. On the basis of this final score, figures or results may be arrived at in very many different ways. This kind of conversion table is not usually included in test programs. The caesura itself may well be adjusted. Various scores and sub-scores can be exported in Excel and txt.

Surpass: A scoring model may be designed for an open question. The scoring model is linked to the question and is part of the development of the question. The scoring model is displayed in Word. When scoring an open question, a comments field can be filled in (no form) and there are a number of call characters for providing feedback (e.g. ticks, red bars, outlining). Each action when scoring is recorded in what is referred to as Marking History.

The scores can be adjusted for each student. Scoring is in points and not in %. Filling in a caesura should be in % and in points when a student is successful (with no decimals). There is 1 way of establishing an absolute caesura. Scores are given with no decimals. Re-scoring may occur per student. Re-scoring can be assigned to a second reviser by the test administrator, for example. A specific workflow can be created here by the role structure that has been devised to include capabilities. Re-scoring is only open for those students assigned by the test administrator. The roles are used to determine those accounts in which participants may be re-scored and in which groups and tests. Archiving in the marking history then similarly shows that there has been a deviation from the original score, by whom and for what reason. An adjusted score appears in the overviews in addition to the original score. Re-scoring a question for all students simultaneously is not (yet) possible. There are plans afoot, however, to do this on the roadmap. Surpass provides scores on test questions and a final score. On the basis of this score, figures or results may be arrived at in very many different ways. This kind of conversion table is not usually included in Surpass. The caesura itself may well be adjusted. Various reports on the results are possible for the caesura criteria. Because the reports can be developed by the institution itself, reports including conversion tables may also be possible. Various exports of scores and sub-scores are possible in Excel (CSV).

Result Maple

TA QMP Surpass

1 Tool for scoring/checking open questions

1 Tools for scoring questions again per student. 1 Tools for scoring questions again per test for several students

simultaneously.

2 Report for the caesura criteria 3 Conversion of scores into results. 3 Exporting results enables batch processing of results in BISON.

Scenario 21: Archiving

Procedures In connection with OER, (automatic) archiving of each test taken is needed for auditing and specifically the accreditation. Analysis of the application, presentation of the result to the working group and joint compilation of the report.

Questions 1 Suitability of the application for archiving. 2 Archiving that is sufficiently secure for an SaaS solution

Implementation

Maple TA: Each instructor can do anything they want with each test. Rights cannot be protected properly in a class (see also scenario 6 'Roles and Workflow’). If someone with sufficient authorisation for an assignment in a class selects ‘delete’, not only the test and the characteristics of the questions in that test will disappear, but also all of the students’ results. The fact that very many instructors can do that is not a good basis for archiving. If the application can be stored according to the CIA matrix (see scenario 7), the archiving will be sufficiently secure.

Question Mark Perception: All the scores on tests are saved in QMP’s results database. The right to delete results may be assigned or kept for each role. If, however, questions are deleted, the precise formulation of the question on which a test result is obtained from the past may be unclear, even though the result itself has been saved. The right to delete questions cannot be denied to instructors, who can also create new questions and edit existing questions. Thus there is a basis for archiving, but there are a few risks that can only be dispelled with good verbal agreements. There is still also the problem that there are 2 ways (see scenario 20) of subsequently changing the scores on the basis of the analysis. Results can be adjusted per each individual student. This is done in the database and is traceable and therefore archived. If the score for the entire question has to be erased from the results, this can only be done in the Results Management System (RMS) (see https://www.questionmark.com/us/news/pressreleases/Pages/rms_mar_2008.aspx ). This works on the basis of a separate database, however, which contains the modified scores in all the modified versions. The original scores still exist in QMP’s original results database. Question Mark is working on a new solution instead of this RMS. The question therefore is whether and how long this system will continue to work and how long the licence for it will continue to be available. Using this system means that there are 2 databases in which the final scores of a test have been archived. If the RMS is used, it contains by definition other scores for the same test. What is used is not clear because an instructor can trial several possible situations in the RMS. If the application can be stored according to the CIA matrix (see scenario 7), the archiving will be sufficiently secure.

Surpass: Surpass does not have the capacity for identifying results or candidates with their results, even for the administrators at the highest levels of authorisation. Results are always saved and an administrator cannot make any change to them (apart from starting again with an empty database). Test questions are also saved in all the versions if results are obtained for them. Even which answer was given in the past to which version of the question is saved. The right to erase questions exists only for questions that have not yet been published prior to use. Questions once published can only be withdrawn and they will no longer therefore be recorded in tests. The senior administrator may only assign the right to withdraw questions to specific users. This is therefore a secure basis for archiving. If the application can be stored according to the CIA matrix (see scenario 7), the archiving will be sufficiently secure.

Result Maple TA QMP Surpass 1 Suitability of the application for

archiving.

2 Archiving that is sufficiently secure for an SaaS solution

Scenario 22: Capacity for secured skills testing

Procedures Skills tests are held regularly in Saxion. In doing these, a student proves that he can work out a case (casus) in, for example, a budget, an advertising text, a computation, an analysis etc., working with Excel, Word or SPSS, for example. It is the intention that this version of Excel or Word will then be sufficiently secure for students, for example, not to be able to communicate digitally with each other on the questions or answers. In the end, the detailed file must be handed in. You could see this process as a form of open question in a digital test in the test application. Is that feasible in the new test application? We have to investigate the possibility of reviews of assessments by the assessor. This is for scoring competences.

Questions 3 Secured skills tests in an application 3 Possibility of scoring (partial) competences for an assessment

Implementation Maple TA: Delft Technical University has built the secure browser for Maple TA. It is therefore not possible to speak to the supplier about this secure browser. Delft Technical University wishes to incorporate secured skills testing into this secure browser. How and which ones are not known at present. Partial competences on a student’s skills cannot be scored in Maple TA.

Question Mark Perception: There are plans to create the possibility in QMP’s secure browser of white listing specific programs that can then be opened for the secure browser. This is a step that may lead to secured skills testing. QMP has the option of an instructor scoring (partial) competences of an assessment for a specific student. This tool is also available via mobile devices. See https://www.questionmark.com/us/perception/Pages/delivery-workplace-assessments.aspx .

Surpass: BTL is in the process of creating the possibility in the Surpass secure application of taking skills tests based on white listing and what is referred to as the file delivery issue. See http://saxion.hosted.panopto.com/Panopto/Pages/Viewer/Default.aspx?id=67e4de95-3a0f-4958-9769-12dad0d0c7ba . Partial competences on a student’s skills cannot be scored in Surpass.

Result Maple TA QMP Surpass 3 Secured skills tests in an application 3 Possibility of scoring (partial)

competencies for an assessment

Appendix 1 Those involved in this recommendation

Those involved from the academies/departments All those involved in setting up and carrying out test scenarios are listed on the list below together with their role.

Academy Those involved Role ABO Peter Arends Test contact person on behalf of

SBRM, instructor Rien Poyck Test contact person, instructor ABR Michel de Klein Test desk Paul Schunselaar Expert AGZ Ben Aardema ICT administrator AMA Alexander Kremers Test desk Lieke Hegemann Test desk Marco Farfan Galindo ICT&O contact person,

instructor AMM Ineke Doeschot Test desk Ida ten Tusscher Test desk Ger Reekers Functional administration,

instructor ACT Jannie Braber Educational expert Dion Wielens ICT&O contact person APO Paul Hilferink ICT&O contact person FEM Amber Kornet Test contact person, educational

expert Luuk Beumer Test contact person, instructor ROB Gerko Hassink ELO contact person, instructor LED Herman Surink ICT&O contact person,

instructor ICT&O Henny Groot Zwaaftink ICT&O Business Information

Manager SCV Purchases Jaques Heeren Buyer SCV IC Koert Tuijtel ICT&O Account Manager Erwin Rensink Technical application

management and infrastructure SVC CTB Thijs van Velzen Exam office coordinator Pascal Huiskes Exam office member of staff ICT&O functional administration Alwin Wullink Functional administrator for

speciality testing Leen van Kaam ICT&O functional administration

team leader

Those involved in drawing up the recommendation The people below were involved in assessing the recommendation at the meeting of 29 November 2012 and unanimously confirmed the written result of this advice, also in writing.

Academy Name

ABO Rien Poyck Peter Arends ABR Michel de Klein ABR Paul Schunselaar ACT Jannie den Braber AMA Alexander Kremers AMM Ger Reekers Ineke Doeschot Ida ten Tussscher AGZ Ben Aardema APO Paul Hilferink

FEM Amber Kornet Hans Leijenaar LED Herman Surink

MIM Anne Klarenbeek ROB Gerko Hassink SVC CTB Pascal Huiskes SVC IC Koert Tuijtel

Erwin Rensink

Present at SIG Digitaal Toetsen (SIG Digital Tests) On 13 December 2012, SIG Digitaal Toetsen gave their unanimous consent to the recommendation. The members of SIG Digitaal Toetsen are there to be instructed and consulted by the academy.

Academy Name ABO Rien Poyck ABR Henny Groot Zwaaftink AGZ Ben Aardema AMM Ineke Doeschot AMA Written consent by Alexander Kremers on

behalf of AMA due to illness. HBS Richard Sueters LED Herman Surink ROB Wietse Mensonides Gerko Hassink SCV Pascal Huiskes ICT&O Heino Logtenberg Leen van Kaam Alwin Wullink (Chairman) Peter Heidemann (Secretary)

Program control group The recommendation was discussed in the program control group on Wednesday 5 December. There was unanimous support for the advice here, too.

Academy Name MIM Rik Eijsink (Chairman) AGZ Stephan van der Voort APO Jan Auwke Diepenhorst FEM Bert Velt LED Peter van Dam ICT&O Heino Logtenberg

External review Michiel van Geloven, the external reviewer, endorses this recommendation.