implementing large scale and rapid growth assessment programmes suzana lopes commercial director
DESCRIPTION
Implementing large scale and rapid growth assessment programmes Suzana Lopes Commercial Director. Presentation Outline. Pearson Company Overview Case Studies: Large Scale Assessment Programme DSA – Driving Standards Agency (UK) DVTA – Driver and Vehicle Testing Agency (Northern Ireland) - PowerPoint PPT PresentationTRANSCRIPT
4 July 2004 CAA Conference 1 of 19
Assessments & Testing
Implementing large scale and rapid growth assessment programmes
Suzana LopesCommercial Director
4 July 2004 CAA Conference 2 of 19
Assessments & Testing
Presentation Outline
Pearson Company Overview
Case Studies:
Large Scale Assessment Programme DSA – Driving Standards Agency (UK) DVTA – Driver and Vehicle Testing Agency
(Northern Ireland)
Rapid Growth Assessment Programme City and Guilds
4 July 2004 CAA Conference 3 of 19
Assessments & Testing
Pearson Company Overview
Pearson plcPearson plc
£4 billion annual sales£4 billion annual sales
World’s # 1 education World’s # 1 education businessbusiness
Over 30,000 employeesOver 30,000 employees
Pearson VUE - The UK Market Leader in e-Assessment Pearson VUE - The UK Market Leader in e-Assessment
Acquired Promissor in January 2006 to expand assessment serviceAcquired Promissor in January 2006 to expand assessment service
Flexible and robust content development and delivery solutionsFlexible and robust content development and delivery solutions
Test centres in 145 countries and easy to deploy customer site testingTest centres in 145 countries and easy to deploy customer site testing
A commitment to quality and outstanding serviceA commitment to quality and outstanding service
4 July 2004 CAA Conference 4 of 19
Assessments & Testing
Case Study: The Driving Theory Test
• Contract awarded 31 October 2003 by Joint Authorities
• Our contract is for delivery of theory tests
• Nearly 1.8 million tests delivered annually
• Learner tests for: Car Motorcycle Truck Buses Driving instructor tests Driving examiner tests
4 July 2004 CAA Conference 5 of 19
Assessments & Testing
What needed to be achieved
• Establish a contact centre to handle: Telephone bookings Mail and fax bookings
• Build DDA compliant test centres at 151 locations England Scotland Northern Ireland Wales
• Build and publish the tests• Develop bespoke booking and reporting systems• Real-time eligibility checks• Hire, train and transfer staff from previous vendor
4 July 2004 CAA Conference 6 of 19
Assessments & Testing
What Needed To Be Achieved - The Timeframe
Nov 03 Dec 03 Jan 04 Feb 04 Mar 04 Apr 04 May 04 Jun 04 Jul 04 Aug 04 Sep 04
Contract awarded31 Oct 2003
Model centre launched1 March 2004
Contact centre opened1 Jul 2004
Registration andscheduling begins
26 Jul 2004
Testing ends withprevious contractor
3 Sep 2004
Testing beginswith Pearson4 Sep 2004
4 July 2004 CAA Conference 7 of 19
Assessments & Testing
Test Centres
Requirement: 80% of the UK population to be no more than 20 miles from their nearest test centre
• Difficult to find DDA-compliant (accessible) locations in some areas
• Northern Scotland was particularly a problem
• Decision was made to use coaches as mobile test centres
• Eventually found locations for 151 test centres Space that worked with the test centre plans Safe and appropriate Easily accessible by public transport
• Flexible opening hours
4 July 2004 CAA Conference 8 of 19
Assessments & Testing
Publishing The Tests
Requirement: Publish tests in English and Welsh - with voiceovers in 20 languages and available in British Sign Language
• First understand the presentation and scoring requirements
• Multiple choice and hazard perception components
• Thousands of the items built into a variety of tests
• Multiple forms of each test
• Hundreds of video clips for HPT test with specialized scoring
• Inventory the items and materials carefully
• Identify missing components
• Build the tests
• Test, test and test
4 July 2004 CAA Conference 9 of 19
Assessments & Testing
Test Centre Design and Test Delivery
Requirement: Design highly secure test centre with invigilator station
CCTV and audio monitoring and biometrics during check in process
• Produced working prototype of test centre meeting all requirements
• Opened on 1 March 2004
• Loaded Hazard Perception Test video clips on 30 stations to demonstrate smooth operation
• Test registration and scheduling
• Standardised procedures at the centres
• Hired contractors to kit out 151 test centres
and mobile centres
• Hired and trained staff
4 July 2004 CAA Conference 10 of 19
Assessments & Testing
Example Test Centre Floor Plan
4 July 2004 CAA Conference 11 of 19
Assessments & Testing
Critical Success Factors
• Excellent Project Management
• Working groups Software Testing centres Contact centre Test publishing Security
• Senior business group
• Experienced test development and delivery staff Members from Pearson VUE, DSA and DVTA
• Weekly update meetings
4 July 2004 CAA Conference 12 of 19
Assessments & Testing
Rapid Growth Assessment Case Study: City & Guilds
• 125 year old not for profit company• Royal Charter• UK’s largest vocational awarding body• 1.2 million learners• 8500 centres • 600 awards• 102 Countries • Operating in 22 sectors
4 July 2004 CAA Conference 13 of 19
Assessments & Testing
Overview
• 2001 City & Guilds decided to launch assessment and learning portal
• Plan to move centres to online registration and booking and develop computer based tests
• Global search for test partner
• Detailed specification
• Reliability, scalability and flexibility ranked highly among the attributes required
• 6 companies competed
• Promissor awarded the contract in 2002
4 July 2004 CAA Conference 14 of 19
Assessments & Testing
Attributes Of The Market
• Almost entirely paper based.
• Candidate choice was limited but growing
• Centres were cost focussed
• Constant policy and funding changes
• Backdrop of slow uptake of on line services
across industry
4 July 2004 CAA Conference 15 of 19
Assessments & Testing
Test Development
• Deployment team established• CBT item writers appointed by
City and Guilds• Training in Promissor Builder
Easy to Use – non expert interface
Web based tool – collaborative
Track and view content changes
Store multiple classification schemes
Create tests based on balanced content criteria
Allows randomized tests, test pooled from an item pool and adaptive test
4 July 2004 CAA Conference 16 of 19
Assessments & Testing
Test Delivery
• Customers consulted about design of system
• Regional training events held for customers
• Online training provided to use registration system
• Documentation provided for surveillance and invigilation at centre
• Technical support line
• Customer service support
Centre Approval
Promissor provide
Centre Criteria
Test Centre Installation
Test Centre Demo Test
Promissor make LIVE
4 July 2004 CAA Conference 17 of 19
Assessments & Testing
www.catglobal.com
Solution Overview
(ENTERPRISE)SITE MANAGER
www.catglobal.com
Candidate Checks-in
Booking
Test Administrator
Test Administrator
Test Administrator
CandidateTakes Test
Test Published for Delivery
Item Bank
BUILDERCandidateRegistration
Registration
C&GSystem
Scheduling
Candidates scheduled
on-line
Results
C&GSystem
Test CentreSet-up
Test Centre
4 July 2004 CAA Conference 18 of 19
Assessments & Testing
Rapid Growth
• 2002: 200 tests delivered
• 2003: Plan to move 80% of objective assessment to CBT
65k tests delivered
• 2004: 187k tests delivered
2006
• 100k test per month
• Over 250 computer based tests developed
• 6000 registered centres
GOLA - Monthly Running Totals 2002 to 2006
839199
708219
394273
253073
127761
64120
15294394
0
100000
200000
300000
400000
500000
600000
700000
800000
Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06
Month
Tes
t V
olu
me
4 July 2004 CAA Conference 19 of 19
Assessments & Testing
Critical Success Factors
• Senior management support
• Efficiency and value has to be proven internally as well as externally
• On-screen must be mandatory
• Technology right first time
• Training must be easily available
• Extensive support is needed