1

e-Assessment Acceptance & Usage 2005-2006

Geoff ChapmanThomson Prometric

Thomson Prometric

OverviewThe UK e-Assessment marketCompare 2005 with 2006Measure industry change in acceptance and usageDocument new emergent issues

Thomson Prometric

Methodology

Qualitative (face-to-face) interviewsDocument key and emergent issues

Formulate quantitative study Quantitative (telephone) interviewsCompare and contrast with 2005

Thomson Prometric

The 2006 market position

• 38% of ABs surveyed are delivering e-assessment in some form• 33% of the remaining 62% are developing or piloting• 70% believe e-Assessment will deliver a Return on Investment

• 72% believe QCA vision of full implementation by 2009 is not achievable– 11% believe their entire program will be delivered through

e-assessment

Thomson Prometric

Industry issuesCandidate satisfaction Candidate identity & eligibility authenticationValue of multiple-choice at higher levels of assessment‘Test When Ready’ versus Fixed Testing WindowsUse of psychometrics in developing item banks & data management

Thomson Prometric

Advantages of e-Assessment 2005-2006

8%

8%

28%

23%

37%

41%

16%

48%

10%

11%

27%

28%

29%

30%

33%

55%

Flexibility (of questions)

Security

Time flexibility/convenience

Cost effectiveness

Ease ofmarking/consistency/accuracy

Geographicflexibility/Accessibility

Ease of administration

Speed/immediate feedback

2006

2005

Q. What are the three main advantages or benefits of using e-Assessment, in your opinion, overall?

Base: 87 (2005), 93 (2006)

Thomson Prometric

Disadvantages of e-Assessment 2005-2006

26%

7%

33%

19%

21%

17%

13%

38%

8%

12%

13%

17%

17%

19%

26%

30%

Technical failure/issues

Lack of flexibility in question type

Inability to assess sophisticatedquestions/creativity

User resistance/inability

Security/Confidentiality/Cheating

Candidate validity/authenticity

Access to IT/Accessibility

Cost

2006

2005

Q. What are the three main disadvantages of using e-Assessment, in your opinion, overall?

Base: 87 (2005), 93 (2006)

Thomson Prometric

Q. What risks do you associate with the introduction of e-Assessment, if any?

E-Assessment risks 2005 - 2006

7%

6%

12%

10%

13%

23%

18%

40%

5%

7%

8%

8%

9%

13%

17%

28%

Risk not fit forpurpose/appropriateness

Ineffective assessment/qualityassurance

Publicperception/mindset/confidence

Cheating/invigilation

Financial risk/cost/investment

Technology/Hardware/infrastructureavailability/suitability/quality

Candidate authenticity/identity fraud

Security of database/hacking

2006

2005

Base: 87 (2005), 93 (2006)

Thomson Prometric

E-Assessment Importance Factors 2005-2006

Base: 87 (2005), 93 (2006)

7.5 7.4 7.47.1 6.9

6.6 6.6 6.5 6.46.1

5.95.6

8.4

7.9 7.8

7.2 7.27.5

6.8 6.8 6.8

6.3

5.8

4.04.55.05.56.06.57.07.58.08.59.0

2006 importance 2005 importance

Q. On a scale of 1-10 where 10 is extremely important and 1 is not important at all, how important are each of the following factors to you in relation to e-Assessment?

Thomson Prometric

Driving Uptake 2005-2006Q. Across the assessment industry, what do you believe to be the top three factors currently driving the uptake of e-Assessment?

13%

9%

11%

18%

10%

9%

26%

27%

24%

38%

8%

10%

10%

11%

17%

17%

17%

21%

37%

45%

Improvements in technology &access

Better quality ofassessment/usability

Competitors doingit/fashion/the future

Convenience/availability/testingon-demand

Reducing admin

Candidate/customer demand

Geographicscope/accessibility

Speed of results/feedback/e-Marking

Regulator/governmentpolicy/agenda/targets

Cost/savings/value for money

2006

2005

Base: 87 (2005), 93 (2006)

Thomson Prometric

Suitability of Multiple Choice – higher learning

Responses from the remaining 48% are opposite

Q. Do you believe that MCQs are suitable for higher level learning?

52% of respondents believed that MCQ’s are suitable for use in higher level learning

6%

9%

9%

16%

19%

44%

Too easy/less challenging

Suitable for assessment, notlearning

Choice/guesswork element

Used as wider testing/insome areas

Tests basicknowledge/facts/limitations

Not able to testopinion/thinking/interpretation

Base: 93

Thomson Prometric

Suitability of Multiple Choice – high stakes examsQ. Do you believe that MCQs are suitable for high stakes exams?

54% of respondents believed that MCQs are suitable for use in high stakes exams

8%

8%

11%

16%

27%

32%

Too easy/not challenging

As part of wider assessment only

Can only test knowledge/facts

Narrow scope/lack ofbreadth/depth of testing

Doesn't allowjudgement/understanding/compet

ence

Due to elements ofguesswork/choice

Base: 93

Responses from the remaining 46% are opposite

Thomson Prometric

Use of PsychometricsQ. What do you understand to be the implications or advantages of using psychometrics to develop exam questions?

3%

4%

4%

7%

7%

7%

Enhances publicperception/seems

fair/objectivity

Not exactscience/unsure of

reliability

Discriminatesbetween

knowledge levels

Helps set upbalanced/robust

test

For measurementof success of

questions

Enhancesreliability/validity

Base: 93

Thomson Prometric

Return on InvestmentQ. Do you believe that e-Assessment will deliver a Return on Investment?

Yes 70%No 7%Don’t know 23%

Q. If yes, when do you believe you will, or did you, see ROI?

8%

11%

15%

17%

22%

23%

Within 1 year

Don't know

It depends

5 years plus

Within 5 years

Within 3 years ofimplementation

Base: 93 Base: 65

Thomson Prometric

The Future – view from 2005 vs. 2006Q. Approximately what percentage of your assessment programme do you realistically expect to be e-Assessment by 2009?

2%

7%

15%

31%

10%

11%

12%

20%

31%

39%

0%

100%

60 - 99%

31 - 59%

1 - 30%

20062005

Base: 87 (2005), 93 (2006)

Thomson Prometric

Summary

Acceptance & usage grown at a substantial rate since 2005Strong confidence in e-Assessment being able to deliver ROI MCQs suitability at higher levels more widely acceptedEmergent usage of psychometricsNeeds of candidates/learners remain at the forefront

Copies of the research are available on the Thomson Prometric stand or download at www.prometric.com/uk or http://tinyurl.com/fyhbt

17

Acceptance & Usage -e-Assessment in 2006

Geoff ChapmanThomson Prometric

geoff.chapman@thomson.com