nts game software compatibility / functionality test outline

8
NTS Testing Labs Game Software Test Lab Compatibility & Functionality Test Outline Revision 1.0 NATIONAL TECHNICAL SYSTEMS The NTS Mission: Assisting our Clients in Navigating a Short Course to Global Markets Written by: Max Macias, Project Manager Tracking No. TO201009 Rev. 1.0 Approved by: Osman Sakr Date 2/22/02 Quality Assurance Approval Derek Fakehany Date 2/22/02 FileName NTS Test Outline - Game Software - TO201009.doc Copyright © 2002, National Technical Systems, Inc. This document is provided to clients and potential clients of National Technical Systems (NTS) for informational purposes only. It may not be reproduced or stored, in whole or in part by any means, with out the express written permission of NTS. Document Tracking # TO201009 NTS Test Outline - Game Software - TO201009.doc 1.0 MM 02/02

Upload: softwarecentral

Post on 29-Jun-2015

271 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: NTS Game Software Compatibility / Functionality Test Outline

NTS Testing Labs Game Software Test Lab

Compatibility & Functionality Test Outline

Revision 1.0

NATIONAL TECHNICAL SYSTEMS

The NTS Mission:

Assisting our Clients in Navigating a Short Course to Global Markets

Written by: Max Macias, Project Manager

Tracking No. TO201009

Rev. 1.0

Approved by: Osman Sakr

Date 2/22/02

Quality Assurance Approval Derek Fakehany

Date 2/22/02

FileName NTS Test Outline - Game Software - TO201009.doc

Copyright © 2002, National Technical Systems, Inc. This document is provided to clients and potential clients of National Technical Systems (NTS) for informational purposes only. It may not be reproduced or stored, in whole or

in part by any means, with out the express written permission of NTS.

Document Tracking # TO201009 NTS Test Outline - Game Software - TO201009.doc 1.0 MM 02/02

Page 2: NTS Game Software Compatibility / Functionality Test Outline

Table of Contents 1. Introduction .........................................................................................................................................................1 2. Scope of Work .....................................................................................................................................................1 2.1 Overview ...........................................................................................................................................................1 2.2 Test Procedures................................................................................................................................................2

2.2.1 Compatibility Testing ....................................................................................................................................2 2.2.2 Functionality Testing.....................................................................................................................................3 2.2.3 Content Verification Testing..........................................................................................................................3

3. Test Environment ................................................................................................................................................4 4. Project Deliverables ............................................................................................................................................4 4.1 NTS Test Reports .............................................................................................................................................5 4.2 Test Matrix ........................................................................................................................................................5 4.3 Project Documentation Report ..........................................................................................................................5 4.4 Report Transmission.........................................................................................................................................5 4.5 Project Leadership ............................................................................................................................................5 5. Conclusion...........................................................................................................................................................5

i

Page 3: NTS Game Software Compatibility / Functionality Test Outline

National Technical Systems Game Software Test Plan

1. Introduction National Technical Systems (NTS) Testing Laboratories currently conducts software compatibility / functionality testing for a wide array of software developers. This test outline will summarize and define the strategy and approach taken to perform testing on Game Software. The following Test Outline will describe the methodologies and test procedures used in Game Software Compatibility / Functionality Testing.

2. Scope of Work

2.1 Overview NTS typically conducts compatibility and functionality across a minimum of twenty (20) test platforms. The platform selection will be a random mixture of both desktop and laptop platforms. The platforms’ selection will be chosen from ‘Tier 1’ vendors (i.e. vendors that hold the largest PC market share). These platforms will range in processor speeds from 266Mhz to 2Ghz, using Intel’s Pentium-II, Celeron, Pentium-III, Pentium-4, and AMD’s KX and Athlon series. The purpose for the wide array of internal equipment variants is to expose the product-under-test to a variety of BIOS manufactures, motherboard chipsets, graphics controllers (either on-board or add-in), sound controllers (either on-board or add-in), and I/O controllers. Testing under these diverse hardware configurations will assure this type of coverage and should uncover issues that might exist that could be “platform specific” or “configuration specific” in nature. For PC type platforms, each test platform will have a selected operating system installed using the Microsoft’s 9x and NT kernels. The specific operating systems used during the compatibility test under the Microsoft’s 9x kernel will be Microsoft Windows 95 OSR2.5, Microsoft Windows 98 Gold Edition, Microsoft Windows 98 Second Edition, and Microsoft Windows Millennium Edition. Under the Microsoft NT kernel, the specific operating systems using during the compatibility test will be Microsoft Windows 2000 Professional and Microsoft Windows XP (which was in Beta as of the date of this document), and in future revisions to this or similar documents, UNIX based operating systems (i.e. Linux variations, Solaris, etc.). For Macintosh platforms, each test platform will have a selected MacOS operating system installed. The specific operating systems used during the compatibility testing will be Apple MacOS 8.6, Apple MacOS 9.04, Apple MacOS 9.1, and Apple MacOS X. Performing compatibility testing under these diverse environments should expose the product under test to possible Operating System specific issues or issues specific to Kernel variation. See section 3, ‘ ,’ for more detailed information.

Test Environment

Once the operating system is installed on a selected platform, NTS will execute a quality assurance test suite consisting of three (3) functionality areas. The test areas include Compatibility, Functionality, and Content Verification. If critical errors are encountered, then the software will fail its compatibility test. If no critical errors were encountered, then the software will pass its compatibility test. During compatibility testing, NTS will “shotgun” an in-depth functionality test across the test platforms. The reason for this approach is to hopefully reveal any critical errors that might be hidden during basic functionality testing. Additionally, “exception processing” or “boundary” testing will also “shotgunned” across test platforms. The purpose of this testing technique is to place the device under test in an extreme stress environment that will hopefully uncover or reveal errors that might be hidden during basic functionality testing. If critical errors are encountered during this in-depth functionality test or “exception processing” test, it will not affect the results of the compatibility test. These results are intended to provide the client with an in-depth knowledge of the functional capabilities and “boundaries” of their product that might not have otherwise been noted from a pure compatibility test.

February 22, 2002 National Technical Systems TO201009-1.0 - 1 -

Page 4: NTS Game Software Compatibility / Functionality Test Outline

National Technical Systems Game Software Test Plan 2.2 Test Procedures Before NTS can begin testing, platforms must be setup and configured with the appropriate operating system. Once the platforms are configured, NTS Testing Engineers will confirm that no “bangs” are listed in the Device Manager – if applicable – and the operating system is functioning as expected. The quality assurance test consists of three (3) functionality areas: Compatibility, Functionality, and Content Verification. Below are those quality assurance areas and the test procedure outlines that coincide with each area.

2.2.1 Compatibility Testing Game software testing will be conducted to ensure that the software provided behaves and performs as expected by the client. Based on the specifications required by the client, a variety of platforms and operating systems will be used. Third-party applications may also be used to ensure that installation gaming software does not affect third-party applications or vice versa. Installation Testing

◊ ◊ ◊ ◊

◊ ◊ ◊ ◊

◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊

Install the game under different operating systems such as Microsoft Windows 95 OEM, Microsoft Windows 95 OSR1, Microsoft Windows 95 OSR2, Microsoft Windows 98 Gold Edition, Microsoft Windows 98 Second Edition, Microsoft Windows NT, Microsoft Windows Millennium Edition, and Microsoft Windows 2000 Professional Install third-party applications Verify the Autoplay feature (if available) Install game under different installation modes (i.e. Custom, Typical, etc.) Verify error handling on invalid entry (i.e. entering alphabets on a numeric field, using incorrect filename convention, improperly using the CD drive as the destination path, etc.) Verify platform can be rebooted after game installation Install game under a different partition Verify game can be uninstalled Verify if third-party applications are properly working

Performance Testing Performance testing verifies the performance of the game under different video resolutions, computer modes, multiple applications opened at the same time, etc. This type of test ensures that the game is under common and unusually stressful conditions. Some of the tests performed for performance testing include:

Run the program simultaneously with other applications Play the game under the minimum system specifications. Play under different modes given by the game Play the different levels of the game Verify printing option Use latest patch for game Play game under different resolutions and color depths Verify game can be launched with native drivers Verify game can be launched with updated drivers Play the game as a single and multi-player Let the demo of the game run for a long period of time Let the game play in auto-mode if it is supported Find ways that might crash the game Perform task-switching, switch between different applications and look for any corruption.

Online Game Testing In case games have to be tested over the Internet, the following tests will be performed:

Verify if the users can login in the game’s web site and initiate a multi-player game.

February 22, 2002 National Technical Systems TO201009-1.0 - 2 -

Page 5: NTS Game Software Compatibility / Functionality Test Outline

National Technical Systems Game Software Test Plan

◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊

◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊

◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊ ◊

◊ ◊ ◊ ◊ ◊

Verify error handling of the web site when users login. Verify if the user is able to access the different areas of the game’s website. Verify if the user obtain the expected score after playing the game. Verify if the game’s website offers the player enough tips on how to play the game. Verify if by clicking on different areas of the game, the website does not crash the test platform. Test the game under a T1, DSL, ISDN, Cable Modem and dial-up connection Play the game under different levels. Verify if game is available and playable from online gaming services (i.e. MSN Gaming Zone, GameSpy, etc.)

Playability Testing This part of testing concentrates on how well the game can be played

Verify if the game can be played with only one user Verify if the game can be played with multiple users If the game can be played over the internet Verify if the game can be played using different types of video and sound cards Make sure that the game is realistic Verify the logical flow of the game (if client will provide this information) Play game using different game control device (i.e. mouse, keyboard, joystick, gamepad, etc.) Verify game is playable with the use of hardware (i.e. DirectX or OpenGL) or software acceleration Verify game’s playability/performance under different versions of DirectX

2.2.2 Functionality Testing Functionality testing provides tests verifying every feature of the program executes properly without any anomalies, difficulties or discrepancies due to the test configuration. This can include:

Verifying demo software runs correctly Verifying registration process Verifying main web links launch the correct web page Verifying graphic acceleration using software or hardware acceleration Verifying internet game play Verifying menus and all areas and levels of game are covered during testing Verifying that game functions properly in a networked environment Verify if the game does not have misspelled words or comments Study the manual and explore other and on other sections that are not displayed on the main menu screen of the game If the game is being run from a CD-ROM, try to copy all the data from the CD-ROM and run it from a second partition. Try to upgrade the game with a new version (if available)

2.2.3 Content Verification Testing Upon the client’s request, tests are provided for verification of the Client’s software to verify that the contents of the game software contain only necessary information. In addition, NTS will document the contents that are contained on each CD.

Scan game CDs for known computer viruses Review files on each CD for hidden messages, vulgar expressions or improper words or phrases Identify the actual contents of each CD comparing contents expected to be on the CD Identifying software products not listed on the CD Identify hidden or extraneous files on game CD

February 22, 2002 National Technical Systems TO201009-1.0 - 3 -

Page 6: NTS Game Software Compatibility / Functionality Test Outline

National Technical Systems Game Software Test Plan 3. Test Environment Below is an example listing of test platforms and operating systems typically deployed by NTS when performing compatibility testing. When configuring the test environment, NTS Testing Laboratories selects platforms and operating system configurations which represent “real world” conditions. Sample of PC Test Platforms

Test Platform Operating System 1. Compaq Presario 2286 Cyrix MII-333 MMX Microsoft Windows 98 SP1 2. Acer AcerPower 8600 Pentium III/800 Microsoft Windows Millennium Edition 3. Acer Aspire 3230 AMD K62/366 MicroTower Microsoft Windows Millennium Edition 4. Compaq Presario 7998 AMD Athlon/1GHz Microsoft Windows 2000 Professional 5. IBM Aptiva K7 650Mhz Microsoft Windows 98 Second Edition 6. Acer TravelMate 737TLV P3700 - Microsoft Windows 2000 Professional 7. Compaq: Presario 1275 AMD K6-2 366MHz Microsoft Windows 2000 Professional 8. Dell Latitude 3500 CPi300XT Pentium II/300Mhz Microsoft Windows 98 Second Edition 9. Hewlett-Packard Pavilion 8650 Celeron/533 Microsoft Windows 2000 Professional 10. Hewlett-Packard Vectra VL8 Pentium II/450 Microsoft Windows 98 Second Edition 11. AST Bravo 500A-1 Pentium III/500MHz Microsoft Windows 98 Gold Edition 12. Compaq Presario 5900Z AMD Athlon/700Mhz Microsoft Windows 2000 Professional 13. Compaq Prosignia 310 Celeron 400Mhz Microsoft Windows 98 Gold Edition 14. Gateway G6-400C Intel Celeron/400 Microsoft Windows 98 SP1 15. Compaq ProSignia 330 Pentium III/500Mhz Microsoft Windows 98 16. Hewlett-Packard Omnibook 7150 PII/300 Microsoft Windows Millennium Edition 17. Hewlett-Packard Brio MT Celeron/333MHz Microsoft Windows 98 Gold Edition 18. Hewlett-Packard Pavillion 4430 AMD K6-2/300 Microsoft Windows 98 Second Edition 19. Hewlett-Packard Pavillion 6460 Celeron/400 Microsoft Windows 98 SP1 20. Hewlett-Packard Pavillion 9695C AMD Athlon/850 Microsoft Windows 2000 Professional

Sample of Macintosh Test Platforms

Test Platform Operating System 1. Apple PowerBook G3 PPC G3/400 Apple MacOS X 2. Apple iBook Graphite 466MHz Apple MacOS 9.1 3. Apple iMac 333 MHz Apple MacOS X 4. Apple iMac G3 Power PC 233 MHz Apple MacOS 8.6 5. Apple PowerMac G4 Cube/450 Apple MacOS 9.1

4. Project Deliverables NTS clients receive the following standard NTS project deliverables during or at the conclusion of tests performed by NTS. (Project specific deliverables may also apply depending upon the specific project type).

1. NTS Test Reports (daily / as generated) 2. Test Matrix and/or Test Status Report 3. Project Documentation Report

The Test Matrix and NTS Test / Problem Reports are transmitted to client daily or as developed. The Project Documentation Report is delivered following the conclusion of testing.

February 22, 2002 National Technical Systems TO201009-1.0 - 4 -

Page 7: NTS Game Software Compatibility / Functionality Test Outline

National Technical Systems Game Software Test Plan 4.1 NTS Test Reports During testing, NTS generates reporting to document all testing performed. NTS test reporting is designed to capture configuration information, general procedures, test measurements and other test results related to all tests performed.

4.2 Test Matrix The NTS Test Matrix provides a quick overview of the status of entire project. The Test Matrix is a master grid outlining individual tests or test milestone coverage on the project. The matrix provides a quick overview of tests performed to date, with columns specifying the product(s) tested, a pass/fail/warn status, the date performed, test engineer identification, and any related Test Report numbers.

4.3 Project Documentation Report NTS binds the final copy of the project test matrix along with all NTS test reports and delivers this to clients following the completion of testing.

4.4 Report Transmission NTS uses Internet e-mail as much as possible to transmit status reports, test reports, test matrices, documentation reports, etc. In some cases, materials may also be faxed or send via FedEx. If additional security is required, NTS will utilize encryption to prepare and send files in batches, using an agreed upon encryption mechanism (to be decided between NTS and a particular client prior to test execution). Catastrophic and critical problems are prioritized and reported upon as soon as possible.

4.5 Project Leadership A project leader is assigned to projects NTS performs for clients. The NTS project leader communicates with clients regarding test progress, status, and findings. The project leader’s primary responsibility is to insure that testing is being executed as agreed upon, to provide management and technical expertise, and to work with the Client to make any required changes. The project leaders are supervised by NTS’ Operations Manager. If major revisions to a test plan is required, or if a client is not satisfied with the testing process, the client is encouraged to contact the Lab Quality Manager immediately. It is NTS’ goal to provide testing that meets Client’s needs as accurately as possible.

5. Conclusion NTS Testing Laboratories has decades of experience and unsurpassed expertise in the area of compatibility and interoperability testing. Fewer product recalls, faster time to market, reduced technical support requirements and higher customer satisfaction are but a few of the many potential benefits of testing a product with NTS. Software compatibility and functionality testing fill the need for “real-world” testing. Testing the product-under-test will be performed under the areas of Compatibility, Functionality, and Interoperability. Testing will assure that the client’s product will function correctly in a “real world” environment. The benefactors of this type of compatibility and functionality testing are lower product recall rates, lower volume of customer support calls and higher consumer satisfaction.

February 22, 2002 National Technical Systems TO201009-1.0 - 5 -

Page 8: NTS Game Software Compatibility / Functionality Test Outline

NATIONAL TECHNICAL SYSTEMS

National Technical Systems (NTS) is one of the largest consulting and certification laboratories in the United States. NTS operates extensive test facilities, throughout the United States and Japan, covering operational environment, stress environment, compatibility, interoperability, broadband network testing, EMI/EMC, FCC, NEBS, USB, WHQL, Product Safety testing and ISO Audit/Certification (we certify the National Space Administration for ISO compliance). NTS is a Nationally Recognized Testing Laboratory (NRTL) - as certified by the United States Government. We are accredited to ISO 17025 by the American Association for Laboratory Accreditation. NTS holds many certifications from foreign governments as a Conformity Assessment Body (CAB). We are the only “one stop” certification and compliance facility offering a broad range of testing capabilities. These capabilities include significant financial resources, numerous certification programs and a high level of lab security. We support extensive test procedures, operating systems and hardware. We are fully equipped for broadband testing and offer such services as DOCSIS “third-party” pre-certification and DSLAM compatibility and functionality testing. A variety of PSTN and Internet access methods are available ranging in speed from dial-up to 1Gb/sec. Several independent T1 Internet access points and frame relay circuits are available. An OC-12 link permits access ranging from standard T1 to OC-12 (644 Mb/sec) in a variety of formats: ATM, Frame Relay, SuperTrunk, HiCap and various other architectures. Our particular testing laboratory is located in Culver City, California. This facility is centrally located and is just 15 minutes from Los Angeles International Airport. IT IS EASY TO START! Send us your “wish list” covering the areas that you would like addressed in your test, and we will design a test proposal specific to your needs that covers only what you want tested. To turn your request into a formal (no cost to you) proposal, we need the following:

1. Product description and or specification. 2. Target Market – What market your product will sell into. 3. Start and end date – When you need to start and end this test. 4. Company information – Name, contact, address, phone, fax and e-mail. 5. Any concerns about your product that you have or you know about.

Let us know if you want a test proposal for your particular project. Typically, our Proposal Department can prepare one for you within five (5) working days, although more complex products may require additional time. The proposal will contain our test methodology, project schedule, defect classifications, regression information, and the project cost. This proposal should be considered a “working document” that you can change or modify to fit your exact test requirement and budget – Our goal is to work together so that you can achieve your objectives. CONTACT US National Technical Systems Computer Testing Division 5730 Buckingham Parkway Culver City, CA 90230 Telephone: (310) 641-7700 Fax: (310) 641-5827 Web: http://www.ntscorp.com/ E-mail: [email protected]