testing an olympic challengemanagers of test activities and defects by application, sport, intel,...
TRANSCRIPT
![Page 1: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/1.jpg)
Testing – An Olympic Challenge
Magnus Alvarsson, Atos Origin,
Canada
Europe’s Premier Software Testing Event
World Forum Convention Centre, The Hague, Netherlands
WWW.QUALTECHCONFERENCES.COM
“The Future of Software Testing”
![Page 2: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/2.jpg)
Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006
© 2006 Atos Origin
Testing an Olympic Challenge
EuroSTAR, The Hague, 12 November 2008
Magnus Alvarsson
![Page 3: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/3.jpg)
3
The Olympic Challenge
» There is no second chance» Proven solutions» Reliable solutions» Meet the Users’
expectations
» World Wide Visibility
» Large IT Deployment
» Multiple partners, suppliers and
technology
![Page 4: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/4.jpg)
4
Agenda
» Introduction
» The Olympic Challenge
» Framework
» Project Timeline
» Organization
» Testing
» Strategy
» Testing environments
» Process
![Page 5: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/5.jpg)
5
Objective
» Ensure stable Games systems that:» Conform to functional requirements» Satisfy or exceed agreed service levels» Support the business of the customers
AND:
» Ensure the integration of:» The numerous and varied Games
systems » The partners, people and processes
supporting these systems
Such that ….
» The systems and business processes complement each other to ensure a seamless delivery of service to our customers
![Page 6: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/6.jpg)
6
The Olympic Challenge - Scope
» Atos Origin Systems- 9 Core Games Management Systems (CGS)- 9 Info Diffusion Systems (IDS)- The Olympic Family Intranet in 2 or 3 languages- A large number of interfaces between these systems
» On Venue Results- One On Venue Result System per sport (15/36)- One Timing and Scoring System per Sport and/or Venue- One Scoreboard per Venue- Minimum of 1 TV-Graphics solution per sport
» External Interfaces- On Venue Results- 8 World News Press Agencies- 20 Internet Data Feeds clients (Rights Holding Broadcasters, Media, IF’s and
NOC’s)- Weather- Transportation- Biography and Background Provider(s)- Background Check
![Page 7: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/7.jpg)
7
Integration of Partners, People, Processes
» Organizing Committee
» Timing and Scoring / On Venue Results Provider
» Hardware providers
» Network
» Servers
» Desk tops
» Cameras
» Score boards / Video boards
» Mobile devices
» Software Providers
» Databases
» Operating Systems
» Office suites
» Etc
The Olympic Challenge – Scope (Cont’d)
![Page 8: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/8.jpg)
8
» Operations» Software Distribution Solution» Call Management System» Monitoring» etc
» External Stakeholders» International Olympic Committee» International Sporting Federations» Rights-holding Broadcasters and News Agencies» OCOG Functional Areas» etc
The Olympic Challenge – Scope (cont’d)
![Page 9: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/9.jpg)
9
The Olympic Challenge - Scope
INTEGRATION
People, Partners
& Processes
• Integration of
sponsors/partners
• Customers (VANOC,
NBC, WNPAs etc)
• Atos TS, Ops, MEP
• Roles & Resp.
• Policies &
Procedures
• Communication
• Project / Risk / Issue
Management
• Omega – On-Venue
Results Systems
• TV Graphics :
Olympic Broadcast
Services
• Scoreboard
• RCMP, Immigration
• Data providers
Non-Atos
Systems
Core Games
Management Systems
• Sport Entries &
Qualification
• Accreditation
• Workforce
• Medical
• Arrivals &Departures
• Transportation
• Commentator
Information
System
• INFO 2010
• Internet Data Feed
• Results Data Feed
to WNPAs
• Print Distribution
System
• Central Repository
System
Information
Diffusion Systems
Testing / Integration Test Lab
![Page 10: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/10.jpg)
10
Project Timeline
2006
Design
Planning
Business Process
Reviews
GAP Process
Functional
Requirements
Interface Definition
Software development
Lab build
Staff build-up
Software deliveries
Test Tools Acquisition
Test Case preparation
Build
Connectivity Tests
Functional Tests
Sport/Test Events
User Accesptance
System Tests
Multi Sport Tests
Test
Technical Rehearsals
Games TOC operational
Applications in production
Venue Deployment
Maximum staff numbers
Operate
Feb 2010Olympic Games
2010
40
100
200
1,200Resources
![Page 11: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/11.jpg)
11
The Challenges
» Challenges» Applications from suppliers worldwide
» Synchronizing deliveries and compatible code
» Management and control of:
- defects
- testing environments
- the numerous & varied Integration Test Lab activities
![Page 12: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/12.jpg)
12
The Answers
» The answers» A purpose-built Test Lab (500 m2; 5000 ft2)
» A dedicated Testing team (80+ people including partners)
» A comprehensive multi-faceted test program
» Strict test procedures and defect tracking
» > 200,000+ hours of testing; > 9,000+ test cases: happy v unhappy path
» Support from Technical Services and Operations
» Communication & Coordination
Testing: the single most important success factor!
![Page 13: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/13.jpg)
13
Objective of Integration testing
Ensure that:
» the Games systems work together seamlessly
» Fulfill the required functionality and service levels
To successfully manage and support the Olympic and Paraolympic
Games.
![Page 14: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/14.jpg)
14
To achieve the objective…
» Test functionality of different systems
» Availability test
» Verify operating procedures of the systems
» Test systems under extreme situations
» Focus the tests on data flowing end-to-end
through the systems.
» Simulate games
![Page 15: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/15.jpg)
15
Integration Test Program
» Pre-Delivery Testing of:
» Supplier Development
» Supplier “Connectivity”
» Supplier Technical
» Integration Testing of:
» Delivery Acceptance
» Stand-alone
» Interface
» End-to-End
» Regression System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Entire System
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Supplier
Development
Testing
Supplier
Connectivity
Testing
Delivery
Acceptance
Standalone
End-to-End
System Test
Interface
IOC Homologation,
CIS & VANOC UAT
Test Events, Multi-
Sport Tests, Tech
Rehearsals,
Software Freeze
![Page 16: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/16.jpg)
16
Integration Test Program
Integration Testing of:
» Delivery Acceptance
» Stand-alone
» Interface
» End-to-End
» Regression
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Entire System
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Supplier
Development
Testing
Supplier
Connectivity
Testing
Delivery
Acceptance
Standalone
End-to-End
System Test
Interface
IOC Homologation,
CIS & VANOC UAT
Test Events, Multi-
Sport Tests, Tech
Rehearsals,
Software Freeze
![Page 17: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/17.jpg)
17
Integration Test Program
» System Testing (3 cycles
of > 20 days each per
application)
» Load
» Stress
» Fault Tolerance
» etc
» Acceptance Testing
» Test Cases supplied by
the IOC and the Sports
Federations or the end
users
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Entire System
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Supplier
Development
Testing
Supplier
Connectivity
Testing
Delivery
Acceptance
Standalone
End-to-End
System Test
Interface
IOC Homologation,
CIS & VANOC UAT
Test Events, Multi-
Sport Tests, Tech
Rehearsals,
Software Freeze
![Page 18: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/18.jpg)
18
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Entire System
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Supplier
Development
Testing
Supplier
Connectivity
Testing
Delivery
Acceptance
Standalone
End-to-End
System Test
Interface
IOC Homologation,
CIS & VANOC UAT
Test Events, Multi-
Sport Tests, Tech
Rehearsals,
Software Freeze
Integration Test Program
» Test Events
» Every Sport
» Elite level competition
» Real Games conditions
» 1 year before
» Multi-sport Tests
» Three cycles
» Subset of Games schedule each
time
» Overall system performance and
functionality
» 2 months period
![Page 19: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/19.jpg)
19
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Entire System
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
System 1
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Application
1
Application
2
Application
3
Supplier
Development
Testing
Supplier
Connectivity
Testing
Delivery
Acceptance
Standalone
End-to-End
System Test
Interface
IOC Homologation,
CIS & VANOC UAT
Test Events, Multi-
Sport Tests, Tech
Rehearsals,
Software Freeze
Integration Test Program
» Technical Rehearsal (TR1 & TR2)
» Two cycles
» Subset of Games schedule
each time
» Operational readiness of
systems and people
» 10 days period
![Page 20: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/20.jpg)
20
Technical Rehearsal main objectives
Technical rehearsals primary focus is to:
» Practice and evaluate:
» operational policies and procedures
applied during Games
» communication processes
» Establish relationships
» Verify roles and responsibilities
» Test staff utilizing systems/applications
in a Games time environment
Beijing example
35 Sports
39 Venues
500 Scenarios
2000 Staff
![Page 21: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/21.jpg)
21
Test Organization
Integrated team made up with the following groups:
» Integration Test Board - Atos Origin / OCOG / Omega
Oversees, Highest Point of Escalation
» Integration Test Manager and Test Management Team - Atos Origin
Definer & Coordinator of testing program & lab
» Test Group Leaders - Atos Origin / OCOG
Test plan managers & delivery managers, escalation point for Test Team Leaders
» Test Team Leaders - Atos Origin / OCOG
Managers of test activities and defects by Application, Sport, Intel, Unix, Security
» Testers - Atos Origin / OCOG
Perform tests, log and track defects
![Page 22: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/22.jpg)
22
Test Organization (cont’d)
A set of integration testing plans, policies and procedures ensuring quality and
consistency in the testing process, including:
» Integration Testing Strategy
» Integration Test Plan
» Defect Tracking Process
» Software Delivery Procedure
» Change Management Process
» Test plans for each sport/system
» etc
![Page 23: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/23.jpg)
23
Test Lab Characteristics
» Production like environment
» Tools used for testing activities» Test Management tool (Quality Center)
- Test Cases and Scenarios
- Defect Tracking
» Test automation tool (QuickTest Professional)
- Test scripts
» Load and Performance testing tools
(Load Runner / Win Runner)
» Staff» 35 Atos Origin» 30 VANOC» 6 Omega» 10 Others
![Page 24: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/24.jpg)
24
» Physical Infrastructure» 300 PCs» 100 laptops» 50 printers» 32 switches» 450 servers
(30 ITL, 420 TER)
Customer Engagement
Test Lab Characteristics
![Page 25: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/25.jpg)
25
Test Cell
![Page 26: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/26.jpg)
26
Criteria for Environment Strategy
» Meet objectives stated in the Integration Test
Strategy
» Integration and operation activities can be
smoothly executed.
» Minimize the total number of environments
» Consider the needs for conducting several
continuous tests in parallel without delays.
» Use experience gained from previous
Olympic Games.
![Page 27: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/27.jpg)
27
Environments
» All applications are installed on all environments.
» Three environments must be production like
» Three environments must have the Games-time configuration
» The rest environments will meet the minimum requirements of the Integration team
» Four environments will be available during the Games,
Date
Environment J F M A M J J A S O N D J F M A M J J A S O N D J F M
PDC
E2E
T1
T2
SDC
PAR
PPP
MEP BCN
Paralympic Testing
Para PreProduction
Testing
Testing Test Events System / MSTs TRs Games
Testing
Testing
2008 2009 2010
Systems Test / DR DR
Homologation PreProduction
![Page 28: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/28.jpg)
28
Integration Control
Keys to Integration Testing success for big scale projects:
1. Correlation with Project Plan
2. Defect Tracking Process
3. Control of Software Deliveries
4. Change Management Process
5. Clear Testing Strategy
6. Partner management/Stakeholder agreement
7. Support Tools
8. Common policies & procedures
To minimise risk, the overall IT
system and applications must be
exhaustively tested
![Page 29: Testing an Olympic ChallengeManagers of test activities and defects by Application, Sport, Intel, Unix, Security » Testers - Atos Origin / OCOG Perform tests, log and track defects](https://reader034.vdocuments.us/reader034/viewer/2022052008/601c95b6257e6b61f90e98e1/html5/thumbnails/29.jpg)
Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006
© 2006 Atos Origin
Questions and Answers
Atos Origin Major Events
Av. Diagonal, 210-218
08018 Barcelona (Spain)
www.atosorigin.com
Magnus Alvarsson
+1 778 328 3636