planning, design, executionww.w.cedma-europe.org/newsletter articles/webinars/performance-based...
Post on 19-Mar-2020
1 Views
Preview:
TRANSCRIPT
CEdMA SIG Meeting Sep 2013
Performance-based Testing Planning, Design, Execution
Kieran Milne, Juniper Networks Certification Program
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception
• PBT Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
2
CEdMA SIG Meeting Sep 2013
• Juniper Networks
– Computer networking; routing, switching, security
– Competitors include Cisco, Alcatel-Lucent, Extreme,
HP, etc.
• JNCP team
– 6 employees
• Liz Burns, Director
• Kieran Milne, Program Manager / Tech Lead
• 2 Tech Leads / Exam Proctors
• 2 Operations folks
Company and Certification team
3
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception
• PBT Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
4
CEdMA SIG Meeting Sep 2013
• Candidate reach: global
– Serves customers, partners, employees
• Content scope: validate technical expertise/depth
– Not for sales/pre-sales
• Certification/exam scope:
– 19 written exams
– 4 lab exams
Juniper Networks Certification Program
(JNCP)
5
CEdMA SIG Meeting Sep 2013
JNCP Framework
6
Asso
cia
te
Pro
fessio
nal
Sp
ec
ialist
Ex
pe
rt
JNCIA -
Junos
JNCIS-SP JNCIS-
SEC
JNCIS-
ENT
JNCIP-SP JNCIP-
ENT
JNCIP-
SEC
JNCIE-SP JNCIE-
ENT
JNCIE-
SEC
JNCSP-
SP
JNCSP-
ENT
JNCSP-
SEC
JNCIP-SP JNCIP-ENT JNCIP-SEC P P P
Network Engineer Support Specialist
JNCIA-
E
JNCIS-
E
JNCIP-
E
JNCIS-
FWV
JNCIS-
WLAN
JNCIS-
SA
JNCIS-
AC
JNCIS-
QF
JNCIA-
IDP
P Out-of-track Prerequisite
=
Product/Technology Track
JNCIS-ENT P
Information current as of September 2013
Service Provider
Routing and Switching
Enterprise Routing
and Switching
Junos Security
Service Provider Routing and
Switching
Enterprise Routing
and Switching
Junos Security
E Series Firewall/ VPN
Wireless LAN
Junos Pulse
Secure Access
Junos Pulse
Access Control
QFabric IDP Series
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
7
CEdMA SIG Meeting Sep 2013
• Factors in deciding to undertake a lab exam effort
– Desire to validate skills at highest level
• Attempt to replicate the actual skills being validated
– Desire to provide a high-level reference point for
expertise to the marketplace
– Desire to meet customer demand
• PBT candidates are most committed to the program
– ‘Me-too’ factor
PBT Conception
8
CEdMA SIG Meeting Sep 2013
• Planning elements to consider
– Cost
• Input: resourcing, capex, opex
• Output: ROI? Strategic value?
– Scale
• Simulations in written exam?
• Full hands-on ‘lab’ exam?
• Live panel / adjudication?
– Staffing/resourcing
• Is there staff available to spend time on this?
• Does staff require specific skills?
PBT Considerations
9
CEdMA SIG Meeting Sep 2013
• Planning elements to consider
– Time commitment
• Exam development time can be substantial
• Staffing requirement can be substantial
– Supporting/study resources
• Courseware?
• Self-study guides?
• Practice tests?
– Logistical constraints
• Equipment availability?
• Lab access limitations?
PBT Considerations
10
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
11
CEdMA SIG Meeting Sep 2013
• Pre-design/pre-development
– Establish some level of scope
• Scope of job role
• Scope of topics/content
• Scope of products
• Etc…
– Tends to be some degree of ‘business reality’ for
vendor-specific exams
• Cannot be all things to all people
• Cannot truly cover all aspects of a job role
– Role is often too diverse for one vendor
Exam Design & Development
12
CEdMA SIG Meeting Sep 2013
• Exam development workshop (1 week)
– Mini-JTA / task expansion / blueprint
– Topology creation
– Task creation / development
– Technical review
– Fixes / edits / changes
• Subject matter experts (SMEs)
– Cross-section of experts (internals/externals) who:
• Do (or have done) the job role
• Have significant exposure to the environment of the job role
Exam Design & Development
13
CEdMA SIG Meeting Sep 2013
• Post-workshop
– Tech Lead takes on remaining steps
• Assemble tasks/sections into cohesive exam
• Translate network topology and tasks onto actual equipment
• Run alpha/beta testing
• Update exams based on alpha/beta feedback
• Create supporting documentation
– Answer guides / grading docs / grading
• Create supporting elements for lab equipment
– Starting-configs, start-up/reload/reset scripts, grading scripts
Exam Design & Development
14
Post-workshop effort is generally much greater than workshop itself
CEdMA SIG Meeting Sep 2013
• Post-workshop
– Sample task list
Exam Design & Development
15
Planning note:
Timelines may be significantly
extended when tasks are
integrated among other work
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
16
CEdMA SIG Meeting Sep 2013
• Lab equipment
– Juniper uses real equipment
• One rack per candidate
– All hosted in Sunnyvale, CA
• No need to ‘travel’ the equipment
– Considerations:
• Equipment cost
• Space, power requirements
• Installation
• Location of equipment <> candidates
– Considering virtualized equipment
for future
Exam Logistics
17
CEdMA SIG Meeting Sep 2013
• Testing locations
– Global coverage
– Combination of Juniper and training partner venues
– Can go onsite to a customer if required
• Lab access
– Secure/VPN access to Sunnyvale, CA
• Good network connection required
– Use ‘trusted’ PCs/laptops at testing location
• Never use candidate laptops
Exam Logistics
18
Internet
Cert lab
VPN
Access portal Lab racks
CEdMA SIG Meeting Sep 2013
• Exam forms
– Provided to candidates
• Multiple sections with tasks, supporting info/exhibits
• Network topology diagram
– Currently paper-based
• Created as PDFs; provided to proctors, to be printed
• Hand out at exam start; collect at exam end
– Electronic forms are interesting
• Increase security; easier to manage
• Increase capex/opex
Exam Logistics
19
CEdMA SIG Meeting Sep 2013
• Supporting documentation
– Provided to exam proctors
• Detailed Description doc
– ‘Walk-through’ guide of exam tasks and answers
• Grading doc
– Instructions to check correctness of exam tasks
– Strict criteria for correctness
• Infrastructure details doc
– Diagram of the underlying physical lab environment
– Useful for troubleshooting network issues
Exam Logistics
20
CEdMA SIG Meeting Sep 2013
• Operations planning
– Develop global exam delivery schedule that coordinates
• Proctors (most not dedicated proctors)
• Lab equipment and PCs/laptops
• Facilities
– Provide an exam registration and payment system/process
• Communications/marketing
– Announcements to certified audience
– Releases to industry press
– Promotions to likely lab exam candidates
– Develop sense of ‘prestige’ around these certifications
Exam Logistics
21
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
22
CEdMA SIG Meeting Sep 2013
• Role of a proctor
– Ensure that every candidate receives a consistent and
fair testing experience
– Provide security for the lab exam event
– Onsite ‘representative’ for JNCP
– Protect the integrity of the JNCP and all certification
credentials
– Provide technical support/expertise
• Juniper proctors all JNCIE certified
Exam Proctoring
23
CEdMA SIG Meeting Sep 2013
• Proctor tasks
– Prepare the testing environment
– Manage and control the distribution of proprietary
documents
– Support candidates during the exam • Verify the operation of lab equipment
• Answer candidates' questions
– Backup candidates’ work after the exam
– Grade exams
Exam Proctoring
24
CEdMA SIG Meeting Sep 2013
• Building a proctor team
– Internal team
• Cert team staff
– External team
• Other internal employees
• Training partner instructors
• Channel partner engineers
– Primary consideration: the TRUST factor
Exam Proctoring
25
CEdMA SIG Meeting Sep 2013
• Training a proctor team
– Pre-requisites
• Juniper requires that proctors have passed the exam
• Excellent way to expose proctors to the ‘front’/candidate side of
the exam
– Training materials and guides
• Proctor Training doc
• Policies and Guidelines doc
• Proctor Guide doc
• Lab Access doc
– Practice grading
• Excellent way to expose proctors to the ‘back’ side of the exam
Exam Proctoring
26
Full proctors are
3xJNCIEs.. a
small/select group
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
27
CEdMA SIG Meeting Sep 2013
• Candidate scheduling and payment
– Online exam registration page
• Candidates can choose exam, select location and date, pay
• JNCP Ops team can add/edit/open/close events, assign proctors
Exam Operations
28
CEdMA SIG Meeting Sep 2013
• Lab equipment usage
– Online lab equipment scheduling tool
• Scheduled candidates > rack reservations
– Pre-scheduled exam sessions automatically start up
morning of exam
– Can borrow racks
from training as
needed
Exam Operations
29
CEdMA SIG Meeting Sep 2013
• Grading/results
– Grading performed by proctors
– Grades sent to JNCP Ops team
– Ops team sends out results by email
• Allows Ops team to track candidate end to end
• Grading scripts/tools
– Script replaces manual grading
• Connects to lab equipment, checks candidate’s work, calculates
grade, provides output for review (text file, Web page)
– Significant reduction in grading time
• But must be measured against ROI
Exam Operations
30
CEdMA SIG Meeting Sep 2013
• Granting credentials/rewards
– Results are manually entered into JNCP candidate
management tool
– Plaque sent out via fulfillment vendor
– Candidate is granted a ‘certification number’
• Ex. JNCIE-SP #380
• Important factor to candidates – ‘lower is better’
– Certificate available online
Exam Operations
31
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
32
CEdMA SIG Meeting Sep 2013
• For candidates
– Increased skillset, marketability
– ‘Marquee’ line item for resume
• For employers
– Provides a point of ‘known reference’ for high-end hiring
– Validates a high level of skills/knowledge/expertise
• For company/vendor
– Extends brand recognition
– Expands pool of expertise in marketplace
– Expands mindshare effort upward, towards decision-
makers
Benefits of PBT
33
CEdMA SIG Meeting Sep 2013
• Company and Certification Team
• Juniper Networks Certification Program
• PBT Conception and Considerations
• Exam Design & Development
• Exam Logistics
• Exam Proctoring
• Exam Operations
• Benefits of PBT
• Lessons Learned
Agenda
34
CEdMA SIG Meeting Sep 2013
• Do not implement PBT unless committed to the
required investment in money and resources
• Consider carefully why implementing PBT
– Is there a compelling business return on investment?
• ‘Cascade effect’ of exam refreshes/updates
– Exam forms are just one element
– May also require a full workshop
– May also impact lab equipment, supporting docs,
grading scripts, proctor training, etc.
Lessons Learned
35
CEdMA SIG Meeting Sep 2013
• Amount of support documentation required
– Don’t underestimate the time required to develop
and/or update necessary docs to successfully roll out
a PBT
• Trade-offs with equipment location
– Centralized, single location
• Easier to manage, support
• Latency can cause testing issues when testing far from
equipment
– Distributed, multiple locations
• Improved performance as candidates are closer to equipment
• Harder to manage, support
Lessons Learned
36
CEdMA SIG Meeting Sep 2013
• Security of documents
– PBT requires a significant investment
– Exam documentation is the tangible output of that
investment
– Consider how much security is ‘enough’ for your
environment
Lessons Learned
37
Questions Kieran Milne
kmilne@juniper.net
top related