tl 9000 quality system metrics book two - · pdf filethe tl 9000 quality system metrics book...

Quality Excellence for Suppliers ofTelecommunications Forum

(QuEST Forum)

TL 9000 Quality SystemMetrics

Book TwoRelease 2.5

Copyright

TL 9000 Quality System Metrics

ii

Copyright 1999 Quality Excellence for Suppliers of Telecommunications Forum

For further information, see the QuEST Forum Web page at: www.questforum.org

TL 9000 is a registered trademark of the Quality for Excellence for Suppliers ofTelecommunications Forum.

Sections of this document contain copyrighted material from a variety of sources, these sourcesare identified in the Bibliography of this handbook.

To the memory of our friend Henry A. Maleca founding member of the QuEST Forum

iii

ForewordThe TL 9000 Quality System Metrics Book Two was prepared in a cooperativeeffort by the members of the Quality Excellence for Suppliers ofTelecommunications Forum. The Forum’s goal from the outset has been todevelop a consistent set of quality system requirements and metrics that, whenimplemented, will help provide telecommunications customers with faster, betterand more cost-effective services.

This book compliments the TL 9000 Quality System Requirements Book One withmetrics that reflect the performance of the industry and its products. Forummembers, including service providers and suppliers alike, utilize measurementsthat are collected under the provisions of this handbook to help improve theirprocesses. By improving processes, the industry becomes more efficient andtelecommunications customers, worldwide, derive the benefit of improved services.

The Forum is pleased to present this book in a common spirit of delighting ourcustomers.

QuEST Forum – Executive Board

Steve Welch, SBCQuEST Forum - Chairman

Krish Prabhu, AlcatelQuEST Forum – Vice Chair

Pierre Blouin, Bell CanadaQuEST Forum – Board Member

Barry D’Amour, Nortel NetworksQuEST Forum – Treasurer

John Pittman, Lucent TechnologiesQuEST Forum – Board Member

Olga Striltschuk, MotorolaQuEST Forum – Board Member

George Via, Bell AtlanticQuEST Forum – Board Member

Tim Wedemyer, BellSouthQuEST Forum – Board Member

Copyright


iv

Preface


v

PrefaceThe Quality Excellence for Suppliers of Telecommunications Forum (“QuESTForum”) was founded to foster continued improvements to the quality andreliability of telecommunications service. The founders took the critical initialstep of establishing a common set of quality system requirements andmetrics by creating the TL 9000 Quality System Requirements Handbook andthe TL 9000 Quality Systems Metrics Handbook. These handbooks are theresult of a cooperative effort among members of the telecommunicationsindustry.

The work of the QuEST Forum yields benefits to customers, theirsubscribers, and their suppliers. Membership is comprised of members ofthe telecommunications industry. Members fund and participate in theForum, have defined voting rights, and are expected to contribute to the workof the Forum. Members vote on adoption of the TL 9000 structure, content,administration, and other questions coming before the Forum.

The Forum establishes and maintains a common set of quality systemrequirements and metrics built upon currently used industry standardsincluding ISO 9001. The requirements promote consistency and efficiency,reduce redundancy and improve customer satisfaction. They also enablesuppliers to improve quality and reliability, reduce costs, and increasecompetitiveness.

Acknowledgments


vi

Acknowledgments

Many members of the Quality Excellence for Suppliers of TelecommunicationsForum (“QuEST Forum”) contributed to the development of theTL 9000handbooks. These handbooks are truly a team effort by the experts that make upthe Forum’s membership. The following individuals participated in thedevelopment of the TL 9000 Quality System Metrics as representatives of theircompanies.

I would like to thank the members of the QuEST Forum and the volunteerregistrars for their dedication and cooperation in creating the TL 9000 Handbooksand the Quality Excellence for Suppliers of Telecommunications Forum.

Jim McDonnellQuEST Forum Project Director

Daniel Avery SymmetricomGalen Aycock Bell AtlanticChandan Banerjee Newbridge NetworksBret Barclay Marconi CommunicationsJim Barke ADC TelecommunicationsDavid Barski NEC AmericaAlan Beaudry Bell AtlanticGary Bishop Sumitomo ElectricDave Bisone Lucent TechnologiesReg Blake BSI & IAARTerry Blok UnisysMichele Boulanger MotorolaPaula Brackman Telcordia TechnologiesBob Brigham Telcordia TechnologiesJoan Brough-Kerrebyn SCCJack Burgen MCI WorldComLarry Busch Lucent TechnologiesJerry Cates SiecorBob Cicelski Newbridge NetworksGlenn Cooley BroadBand TechnologiesBud Cuthbert QMI & IAAR

Duke Dahmen TellabsAshok Dandekar FujitsuPaula DeCarlo PulsecomE. deVries Telkom South AfricaThierno Diallo 3M Telecom SystemsBob Dietz Marconi CommunicationsTim Dinneen Sprint North SupplyKim Dobson MotorolaJean-Normand Drouin Bell CanadaJim Dumouchelle Nortel NetworksGreg Feldman MotorolaMehmet Ficici QUASARPaul Fortlage RABTilman Foust Hekimian LaboratoriesEd Franck AmeritechKen Gale Nortel NetworksMike Gericke BellSouthJoe Gibbs Lucent TechnologiesFrank Gray, Jr. MotorolaRobert Gray AT&TGlen Groenewold Motorola

Acknowledgments


vii

Dennis Grousosky GraybarJack Growe CTDISteve Hackett SBCJean Hardwig Marconi CommunicationsJeff Harpe Nortel NetworksDebbie Hearn Telcordia TechnologiesRich Helmuth GTERolf Hendricksen Nortel NetworksRon Hershberger EricssonGene Hutchison SBCAri Jain Lucent TechnologiesAlka Jarvis Cisco SystemsJohn Jennings Marconi CommunicationsJohn J. Johnson IV PulsecomWally Jubran PirelliDuane Knecht CTDIKen Koffman SiemensJeff Lanham SprintJerry Lefever ADC TelecommunicationsSandy Liebesman Lucent TechnologiesMatt Lindsay TellabsRon Luttrull AlcatelRichard Lycette Newbridge NetworksNick Magro UL & IAARHenry Malec 3COMMichael Manning Newbridge NetworksSue Maxwell Excel Switching Corp.Tama McBride MotorolaJim McDonnell SBCBob McKeown QMI & IAARGreg Miller MotorolaPaul Miller SiemensKen Molinelli Marconi CommunicationsDon Moore FujitsuMark Moore AlcatelArt Morrical Lucent TechnologiesRosemarie Moskow SBCPat Muirragui SiemensStephen Newsome Marconi CommunicationsWilliam J. Novak Lucent TechnologiesCharles O’Donnell ADTRANRobert Oakley Nortel NetworksKathy Parker AT&TRobert Paschke MCI WorldComBrendan Pelan Bell AtlanticRichard Pierrie Nortel NetworksTex Prater BellSouthRichard Pratt Ericsson

Judy Przekop AmeritechMisha Ptak PulsecomMustafa Pulat Lucent TechnologiesSoundar Rajan PairGain TechnologiesGary Reams NEC AmericaJoel Reece SiecorJerry Reichert Marconi CommunicationsDonna Reinsch Marconi CommunicationsMichael Rippe AlcatelJeffrey Rose Sumitomo ElectricJohn Rosenow MCI WorldComGwynne Roshon-Larsen Cisco SystemsMarty Rudnick Lucent TechnologiesH. Pierre Salle KEMA & IAARMike Samocki TeltrendVictor Sandoval EricssonDonna Schilling SymmetricomSue Schwab MotorolaTom Scurlock Lucent TechnologiesDarryl Seeley Marconi CommunicationsChris Shillito Eagle RegistrationsDavid Siebrasse AntecDoug Smith EricssonJohn Smith TellabsJohn Snarr Marconi CommunicationsRob Stewart Cisco SystemsOlga Striltschuk MotorolaSteve Stroup SBCJoel Sullivan BellSouthBill Taylor AntecJoe Taylor TellabsDoug Thompson Newbridge NetworksDavid Turtletaub ElcotelJennifer Vidos Marconi CommunicationsAnthony Vitucci HekimianArlen Waldrop ComplasVicki Walker TellabsJohn Walz Lucent TechnologiesGrace Weaver QualcommRick Werth SBCRandal Whorton ADTRANDon Wilford US WESTNorwita Williams-Powell Pacific Network SupplyJohn Wronka Lucent TechnologiesLeslie Wolf Telcordia TechnologiesTom Yohe AlcatelLen Young CorningMark Young Siemens

Table of Contents


ix

Table of ContentsSECTION 1 - INTRODUCTION.......................................................................................1-1

SECTION 2 - STRUCTURE ............................................................................................2-1

SECTION 3 - METRICS USAGE AND RESPONSIBILITIES...........................................3-1

SECTION 4 - GENERAL METRICS REQUIREMENTS...................................................4-1

SECTION 5 - COMMON METRICS.................................................................................5-1

5.1 Number of Problem Reports (NPR) (C)...........................................................5-15.2 Problem Report Fix Response Time (FRT) (C) and Overdue Problem

Report Fix Responsiveness (OFR) (C) Measurements ...................................5-85.3 On Time Delivery (OTD) (C) .........................................................................5-205.4 System Outage Measurement (SO) (HS)......................................................5-27

SECTION 6 - HARDWARE METRICS ............................................................................6-1

6.1 Return Rates (RR) (H)....................................................................................6-1

SECTION 7 - SOFTWARE METRICS .............................................................................7-1

7.1 Software Update Quality (SWU) (S)................................................................7-17.2 Release Application Aborts (RAA) (S).............................................................7-77.3 Corrective Patch Quality (CPQ) (S) and Feature Patch Quality (FPQ) (S) .....7-14

SECTION 8 - SERVICES METRICS ...............................................................................8-1

8.1 Service Quality (SW) (V) ................................................................................8-1

APPENDIX A: PRODUCT CATEGORY TABLES..............................................................1

APPENDIX B: TL 9000 CUSTOMER SATISFACTION METRICS GUIDELINES ...............1

GLOSSARY OF ABBREVIATIONS, ACRONYMS AND DEFINITIONS ........ GLOSSARY-1

BIBLIOGRAPHY ...................................................................................BIBLIOGRAPHY-1

List of Figures


x

List of FiguresFigure 2.1-1. The TL 9000 Model .......................................................................2-1Figure 2.2-1. TL 9000 Measurement Data Flow and Usage................................2-2Figure 6.1-1. Shipping Date Groups for Computing Return Rates ......................6-6

List of Tables


xi

List of TablesTable 5.1-1. Number of Problem Reports (NPR) Metric Identifiers and

Formulas.........................................................................................5-3Table 5.1-2. Number of Problem Reports – RQMS Alternative Metrics (IPR) .....5-4Table 5.1-3. NPR Data Table..............................................................................5-4Table 5.1-4. RQMS Alternative NPR Data Table (IPR) .......................................5-5Table 5.1-5. Example 1: NPR Data Report .........................................................5-6Table 5.1-6. Example 1 NPR Source Data and Measurements ..........................5-6Table 5.1-7. Example 2: NPR Data Report .........................................................5-7Table 5.1-8. Example 2: NPR Source Data and Measurements ........................5-7Table 5.2-1. Problem Report Fix Response Time (FRT) and Overdue

Problem Report Fix Responsiveness (OFR) Metric Identifiers andFormulas.......................................................................................5-12

Table 5.2-2. Problem Report Fix Response Time and Overdue ProblemReport Fix Responsiveness RQMS Alternative Metrics(ORT and OPR) ............................................................................5-12

Table 5.2-3. FRT or OFR Data Table ...............................................................5-13Table 5.2-4. RQMS Alternative FRT Data Table (ORT).....................................5-13Table 5.2-5. RQMS Alternative OFR Data Table (OPR) ...................................5-14Table 5.2-6. Example FRT Data Report............................................................5-16Table 5.2-7. Example FRT Source Data and Measurement Calculation ...........5-16Table 5.2-8. Example OFR Data Report ...........................................................5-17Table 5.2-9. Example OFR Source Data and Measurement Calculation...........5-17Table 5.2-10. Example FRT Data Report (Services).........................................5-18Table 5.2-11. Example FRT Source Data and Measurement Calculation .........5-18Table 5.2-12. Example OFR Data Report .........................................................5-19Table 5.2-13. Example OFR Source Data and Measurement Calculation for a

Services Product ........................................................................5-19Table 5.3-1. On Time Delivery (OTD) Metric Identifiers and Formulas..............5-22Table 5.3-2. OTD Data Table ...........................................................................5-22Table 5.3-3. Example of Installed System OTD ................................................5-23Table 5.3-4. Example of On Time Service Delivery OTD ..................................5-24Table 5.3-5. Example of On Time Item Delivery OTD.......................................5-25Table 5.3-6. Example On Time Delivery Data Report (OTD) ............................5-26Table 5.4-1. System Outage Measurement (SO) Metric Identifiers and

Formulas.......................................................................................5-29Table 5.4-2. System Outage Measurements (SOE) RQMS Alternative Metrics

End Office and/or Tandem Office, Wireless Products, andNGDLC Products ..........................................................................5-30

Table 5.4-3. System Outage Measurements (SOG) RQMS Alternative MetricsGeneral Series ..............................................................................5-30

Table 5.4-4. System Outage (SO) Data Table ..................................................5-36

List of Figures


xii

Table 5.4-5. RQMS Alternative System Outage (SOE) End and/or TandemOffice, Wireless Products, or NGDLC Products Data Table...........5-37

Table 5.4-6. RQMS Alternative System Outage (SOG) General Series DataTable.............................................................................................5-37

Table 5.4-7. Example Non-Uniform System Calculation of SO .........................5-39Table 5.4-8. Example Normalized SO for Non-Uniform Systems......................5-39Table 5.4-9. Example System Outage (SO) Data Report for Example a.,

March 2001 ...................................................................................5-40Table 5.4-10. Example Transport System SO Measurement Calculation..........5-41Table 5.4-11. Example Normalized SO for Transport Systems.........................5-41Table 6.1-1. Return Rates (FYR and LTR) Metric Identifiers and Formulas .......6-4Table 6.1-2. Return Rate (RR) Data Table .........................................................6-5Table 6.1-3. Example Returns ............................................................................6-6Table 6.1-4. Return Rate Example Data Table .................................................6-10Table 7.1-1. Software Update Quality (SWU0, SWU1, and SWU2) Metric

Identifiers and Formulas..................................................................7-3Table 7.1-2. Software Update Quality RQMS Alternative Metrics (DSU)............7-3Table 7.1-3. SWU Data Table.............................................................................7-4Table 7.1-4. SWU Data Table RQMS Alternative Data (DSU) ............................7-5Table 7.1-5. Example SWU Source Data............................................................7-6Table 7.1-6. Example SWU Data Table Report for June, 2000...........................7-6Table 7.2-1. Release Application Aborts (RAA) Metric Identifiers and

Formulas.........................................................................................7-9Table 7.2-2. Release Application Aborts RQMS Alternative Metrics (RAQ) .......7-9Table 7.2-3. RAA Data Table............................................................................7-10Table 7.2-4. RAA Data Table RQMS Alternative Data (RAQ) ...........................7-10Table 7.2-5. Example RAA Source Data...........................................................7-12Table 7.2-6. Example RAA TL 9000 Data Report .............................................7-13Table 7.3-1. Patch Quality (CPQ and FPQ) ......................................................7-16Table 7.3-2. Patch Quality RQMS Alternative Metrics (DCP and DFP).............7-17Table 7.3-3. CPQ or FPQ Data Table...............................................................7-17Table 7.3-4. CPQ or FPQ Data Table RQMS Alternative Data (DCP or DFP) .7-18Table 7.3-5. Example CPQ Source Data ..........................................................7-19Table 7.3-6. Example CQP Data Report...........................................................7-20Table 8.1-1. Definitions of Defects, Service Volume. and Units of Measure by

Service Product Categories for Service Quality Metrics..................8-2Table 8.1-2. Service Quality (SQ) Metric Identifiers and Formulas......................8-3Table 8.1-3. Service Quality (SQ) Data Table.....................................................8-3Table 8.1-4. SQ Data Sources............................................................................8-4Table 8.1-5. Example Source Data for Installation SQ........................................8-4Table 8.1-6. Example Data Report for Installation SQ ........................................8-5Table 8.1-7. Example Source Data for Repair SQ .............................................8-5Table 8.1-8. Example Source Data for Maintenance SQ ....................................8-5Table 8.1-9. Example Source Data for Call Center SQ.......................................8-6Table 8.1-10. Example Source Data for Support Services SQ............................8-6

List of Tables


xiii

Table A-1. Product Category Definitions.............................................................A-3Table A-2. Metric Applicability Table (Normalized Units) ..................................A-16Table A-3. Transmission Standard Designations and Conversions...................A-23Table A-4. Optical and Electrical Equivalency...................................................A-24Table A-5. Metrics Summary Listing .................................................................A-25

Section 1 - Introduction


1-1

Section 1 - Introduction

The TL 9000 handbooks (the TL 9000 Quality System RequirementsHandbook and the TL 9000 Quality System Metrics Handbook) are designedspecifically for the telecommunications industry to document industry qualitysystem requirements and metrics.

The TL 9000 Quality System Requirements Handbook consists of four (4)major sections with appendices. It establishes a common set of qualitysystem requirements for suppliers of telecommunications products: hardware,software, and services. The requirements are built upon existing industrystandards, including ISO 9001. The TL 9000 Quality System MetricsHandbook defines a minimum set of performance metrics. The metrics areselected to measure progress and evaluate results of quality systemimplementation.

The goals of TL 9000 are to:

• Foster quality systems that effectively and efficiently protect the integrityand use of telecommunications products: hardware, software or services

• Establish and maintain a common set of quality system requirements• Reduce the number of telecommunications quality system

standards• Define effective cost and performance-based metrics to guide progress

and evaluate results of quality system implementation• Drive continuous improvement• Enhance customer-supplier relationships• Leverage industry conformity assessment processes.

The purpose of TL 9000 is to define the telecommunications quality systemrequirements for the design, development, production, delivery, installation,and maintenance of products: hardware, software or services. Included in TL9000 are performance-based metrics that measure reliability and qualityperformance of these products. Long-term goals include both cost andperformance-based metrics. Telecommunications service providers, their subscribers and all customerswill benefit as a result of implementing TL 9000. Expected benefits are: • Continuous improvement of service to subscribers• Enhanced customer-supplier relationships

1.1 Goal

1.2 Purpose

1.3 Benefits ofImplementation

Section 1 – Introduction


1-2

• Standardization of quality system requirements• Efficient management of external audits and site visits• Uniform metrics• Overall cost reduction and increased competitiveness• Enhanced management and improvement of suppliers’ performance• Industry benchmarks for TL 9000 metrics. The QuEST Forum maintains compatibility with other sets of requirementsand standards. TL 9000 provides a telecommunications-specific set ofrequirements built upon an ISO 9001:1994 framework. See the Bibliographyfor the standards and requirements that were considered during thedevelopment of TL 9000.

Characteristics of the TL 9000 relationship to other requirements are:

• TL 9000 includes ISO 9001:1994. Any future revisions toISO 9001 will be incorporated;

• Conformance to TL 9000 constitutes conformance tocorresponding ISO 9001 requirements; and

• It is the intent of the QuEST Forum that conformance toTL 9000 will eliminate the need for conformance to multipletelecommunications quality management standards.

The QuEST Forum is responsible for the development, publication, distributionand maintenance of the TL 9000 handbooks that are publicly available. Changerequests for the handbooks, following initial publication, are to be submitted tothe Forum Administrator. Any user of the handbooks may submit changerequests. Change requests will be forwarded to the appropriate Handbooksection chairperson by the Forum Administrator and will be considered for thenext revision. A change request/feedback form is located at the back of thishandbook. Final approval of all changes to TL 9000 handbooks will be by vote of theQuEST Forum voting members in accordance with the Forum’s bylaws. Re-issue of the TL 9000 handbooks will be determined by the Forum, but not toexceed five (5) years following the last issue date. When the Forumdetermines there are issues in the TL 9000 that could impact third partyregistration, then addenda or similar communication mechanisms will beemployed to inform the industry of corrections and updates to the standard.

1.5 Developingand Maintainingthe Handbook(s)

1.4 Relationshipto ISO 9001 andOtherRequirements

Section 2 - Structure


2-1

Section 2 - Structure TL 9000 is structured in layers (See Figure 2.1-1): 1. International Standard ISO 90012. Common TL 9000 Requirements3. Hardware, Software, and Services Specific Quality Systems

Requirements4. Common TL 9000 Metrics5. Hardware, Software, and Services Specific Quality System Metrics

The TL 9000 Model

International Standard ISO 9001

Common TL 9000 Requirements

Hardware SpecificRequirements

Software SpecificRequirements

Services SpecificRequirements

Hardware SpecificMetrics

Software SpecificMetrics

Services SpecificMetrics

Common TL 9000 Metrics

TL

9000

Figure 2.1-1. The TL 9000 Model The QuEST Forum retains full control over the content except for materialthat is copyrighted by others. The use of terms in TL 9000, such as shall, should, may and can, followsthe standard ISO 9000 glossary definitions. The word shall indicatesmandatory requirements. The word should indicates a preferred approach.Suppliers choosing other approaches must be able to show that theirapproach meets the intent of TL 9000. Where the words typical andexamples are used, an appropriate alternative for the particular commodityor process should be chosen. Paragraphs marked NOTE are for guidanceand not subject to audit.

2.1 Overall Structure

Section 2 – Structure


2-2

Figure 2.2-1 illustrates the data flow and usage of TL 9000 Quality SystemMetrics as described in this handbook.

QuEST ForumWeb Site

ContinuousImprovement

Programs

Customer Supplier

MetricsRepository

System (MRS)

Industry Statistics

MRS ReportedTL 9000 Data

Figure 2.2-1. TL 9000 Measurement Data Flow and Usage

The use of metrics should be designed to meet the principles of the QuESTForum, which are stated in subsection 3.2.

2.2 Flow and Usageof Metrics

Section 2 - Structure


2-3

Usage Approach - Figure 2.2-1 depicts an environment in whichimprovement opportunities are identified by customer-supplier exchangesand from the TL 9000 information.

a. Metrics may be used between supplier and customer to set mutualtargets to improve products. This helps build customer and supplierrelationships and establishes targets that best meet the needs of bothparties.

b. Some of the TL 9000 metrics may be used as improvement measures byindividual suppliers. These metrics undergo careful review to determinethat the measures are indeed comparable. These measurements aremonitored by the Metrics Administrator to assure that aggregation acrosssuppliers into summary statistics are valid and meaningful. The summarystatistics definitions will be revised as needed. The definition of thesemetrics includes the designation “compared data.”

c. Other metrics include the designation “research data.” Research datashall not be used for comparison purposes. However, the MetricsAdministrator will analyze the data to reveal possible industry trends.These analyses are reported only to the metrics work groups forexamination for possible future uses.

d. The product category performance is improved as each suppliercompares its measures against the summary statistics and improves itsperformance.

e. The QuEST Forum measurements database is not intended for use as asupplier management tool, but as a data repository. Output from thedatabase shall consist of statistical summary reports derived from the TL9000 Metrics Repository System (MRS) for each metric by productcategory.

Section 3 – Metrics Usage and Responsibilities


3-1

Section 3 - Metrics Usage and Responsibilities

TL 9000 registration requires the fulfillment of the TL 9000 Quality SystemRequirements and the reporting of the TL 9000 Quality System Metrics dataspecific to that TL 9000 registration to the Metrics Administrator. The following principles for processing and using the measurements aremeant to foster an environment where customers and suppliers can worktogether to drive continuous improvement:

a. All applicable metrics for a product category as defined in the MetricsApplicability Table, Appendix A, Table A-2 shall be reported.

b. Valid reasons for the exclusion of specific metrics from the scope ofregistration must be documented by the supplier and available to theregistrar and customer upon request.

c. Suppliers shall provide TL 9000 measurement data to the MetricsAdministrator who will compile the data and calculate product categorystatistics, such as “Industry Mean,” “Standard Deviation,” “Median,”“Range,” “Number of Data Points,” and “Best in Industry,” for eachproduct category, as appropriate. Metrics Administrator-produced resultsor reports will not identify individual suppliers.

d. Customers who are members of the QuEST Forum shall provide thenecessary TL 9000 field performance data to the suppliers in order tocalculate the specific measurements.

e. Customers may request their suppliers to provide the TL 9000measurements specific to that customer. This information exchange willtake place strictly between the supplier and the customer per mutualagreement. The Forum Administrator and Metrics Administrator will notbe involved in any way.

f. There will be no ranking of suppliers by the QuEST Forum Administrator.g. The use of measurements shall not compromise the proprietary nature of

the data.

The intended usage of TL 9000 measurements is to:

a. Provide industry performance information suitable for benchmarking;b. Improve telecommunications processes and products;c. Identify customer-supplier improvement opportunities; andd. Standardize customer report cards or assessments.

3.2.1 Levels of Usage

In order to fully meet the requirements of this handbook and the companionTL 9000 Quality System Requirements Handbook, the metrics defined herewill be used at three (3) levels. These levels are:

3.1 Principles ofMetrics Usage

3.2 Metrics Usage



3-2

a. Internally as a part of the supplier’s continuous improvement programsand management reports;

b. As appropriate, in customer-supplier exchanges and continuousimprovement programs; and

c. Where indicated, reported to the Metrics Administrator for inclusion in thedatabase.

3.3.1 Aggregation of Products

If a supplier wishes to register multiple products in the same product categoryand clearly identifies them as separate in the registration scope, the suppliermay report the data for each product separately. Similarly, if a supplierregisters a business unit or a location, the supplier has the option todetermine which products will be registered and how the data will beaggregated. 3.3.2 Customer Base

a. Customer base refers to the defined group of customers that thesupplier’s measurement data encompasses. The customer base optionsare as follows:(1) Forum Members: Only the supplier’s customers who are members of

the QuEST Forum; or(2) Total: All of the supplier’s customers for the product(s) to which the

metric applies.b. The customer base shall be reported in each measurement data

submission for each metric as indicated in the metric profile.c. The supplier shall report measurement data from only one (1) customer

base per individual metric. 3.4.1 Forum Administrator Responsibilities The Forum Administrator shall:a. Maintain complete security and confidentiality of company information;b. Develop, implement, publish and maintain formal operating procedures

defining the TL 9000 metric process;c. Receive “Data Confirmation Report” from the Metrics Administrator and

forward the “Data Confirmation Report” on to the supplier;d. Communicate to the supplier missing or questionable data concerns as

reported by the Metrics Administrator;e. Maintain a membership database that includes registration and metrics

submission history;f. Publish and maintain industry reportable statistics by:

(1) Product category with data from a Total customer base;(2) Product category with data from only a QuEST Forum customer base;and(3) All available data for a product category (i.e., 1 and 2);

3.4 Responsibilities

3.3 Metrics DataAggregation andCustomer Base



3-3

g. Provide and control access to the measurement data output through theQuEST Forum web site;

h. Develop and implement a disaster recovery plan for QuEST ForumAdministrator related operations;

i. Support external audit or oversight of Forum Administrator activities; andj. Advise registered suppliers and effected companies when updates to

Appendix A, Product Category Tables, are released on the QuEST Forumweb site (www.questforum.org).

3.4.2 Metrics Administrator Responsibilities The Metrics Administrator shall:a. Maintain complete security and confidentiality of the data;b. Develop, implement, publish, and maintain formal operating procedures

defining the TL 9000 metric process tools and techniques;c. Receive and acknowledge receipt of data from suppliers, including

identifying missing or inaccurate data and reporting back to the ForumAdministrator;

d. Calculate the industry statistics, such as “Industry Mean,” “StandardDeviation,” “Median,” “Range,” “Number of Data Points” and “Best inIndustry,” as appropriate, by product category using the appropriate dataelements for each metric that has compared data;

e. Compute industry statistics by:(1) Product category with data from a total customer base;(2) Product category with data from only a QuEST Forum customer base;and,(3) All available data for a product category (i.e., 1 and 2);

f. Post compared data output to the web page at least quarterly;g. Develop and implement a disaster recovery plan for related operations;h. Support external audit or oversight of activities;i. Determine when sufficient data has been collected per metric product

category to publish statistically valid results;j. Be responsible for the accurate representation of provided data;k. Create and maintain user manuals;l. Propose aggregation of product categories to produce meaningful

measurements as a result of analysis of the inputs;m. Analyze “research data” to reveal industry trends and report only to the

metrics work groups;n. Analyze “research data” to determine if there are conditions under which

the data could be compared and make recommendations only to themetrics work groups to achieve comparability.

3.4.3 Supplier Responsibilities The supplier shall:a. Have documented processes in place to capture and validate applicable

measurement data such that source data records are available;b. Collect, validate, and submit data per the defined metric definitions to the

Metrics Administrator;



3-4

c. Submit a minimum of three (3) consecutive months worth of data to theMetrics Administrator and receive a “Data Confirmation Report”acknowledging valid submissions to obtain TL 9000 registration;

d. Submit data on metrics that are within its scope of registration;e. Provide measurement data for new products within six (6) months from

General Availability of the product, if it falls within the scope ofregistration;

f. Compare internal measurements to the industry statistics and take stepsto improve products and practices as appropriate;

g. Provide regular TL 9000 Quality System Metrics reports to its responsiblemanagement;

h. Correct any data discrepancies; andi. Re-submit corrected data for any erroneous data submitted within the

previous two (2) years. 3.4.4 Customer Responsibilities

The customer shall:a. Provide the necessary data to allow suppliers to generate the TL 9000

measurements;b. Have processes in place to capture and validate applicable measurement

data;c. Use the TL 9000 metrics definitions for standardizing the supplier

performance review process (e.g., report cards);d. Establish joint customer-supplier improvement teams and objectives

based on TL 9000 metrics and other required performance objectives;and

e. Consider using TL 9000 metrics as an input when determining life cyclecosts.

3.4.5 QuEST Forum Responsibilities The QuEST Forum shall:a. Be responsible for the administration of the TL 9000 Quality System

Metrics Handbook;b. Ensure that the TL 9000 Quality System Metrics Handbook is publicly

available. Publication, distribution and maintenance are carried out underthe direction of the QuEST Forum, which retains its copyright;

c. Be responsible for assuring the availability of appropriate training to helpusers correctly and consistently interpret the TL 9000 requirements andreport the TL 9000 measurements;

d. Provide metrics process oversight;e. Address all issues and concerns relating to the metrics process and

provide a summary and recommendations to the Forum; andf. Review proposed aggregations of product categories submitted by the

Metrics Administrator.



3-5

3.4.6 Registrar Responsibilities

During each audit, the registrars shall verify that:a. Processes are in place to ensure data validity and integrity in accordance

with the TL 9000 Quality System Metrics definitions and requirements;b. All supplier responsibilities are met; andc. All metric process non-conformances are corrected within the registrar

specified timeframe.

Section 4 –General Metrics Requirements


4-1

Section 4 - General Metrics Requirements

Title Handbook Section

Common Metrics (C) 5Number of Problem Reports (NPR) (C) 5.1Problem Report Fix Response Time (FRT) (C) 5.2

and Overdue Problem ReportFix Responsiveness Measurements (OFR) (C)

On Time Delivery (OTD) (C) 5.3System Outage Measurement (SO) (HS) 5.4

Hardware Metrics (H) 6Return Rates 6.1

Software Metrics (S) 7Software Update Quality (SWU) 7.1Release Application Aborts (RAA) 7.2Corrective Patch Quality (CPQ) 7.3

and Feature Patch Quality (FPQ)

Services Metrics (V) 8Service Quality (SQ) 8.1

4.1 Metrics Listing

Section 4 – General Metrics Requirements


4-2

4.2.1 Conformance to Metrics Profile

The supplier shall generate and distribute the measurement data to theMetrics Administrator (and to customers upon request) as described by theprofiles in this handbook for the applicable product categories. Themeasurement data shall conform to the requirements in the correspondingprofile. Changes to reported data that are needed to comply with a newmaster version of Appendix A or a new version of the metrics handbook shallbe completed within six (6) months of their release.

NOTE: Text within quotation marks (“text”) in the data tables is a variablestring that is to be reported exactly as shown.

4.2.2 Applicable Product Categories

For each product, the supplier shall identify product categories and applicablemetrics according to Appendix A. Appendix A is current as of the release ofthis handbook. See the QuEST Forum web site (www.questforum.org) forthe master. The master shall be used in conjunction with registrations and forall data submittals to the QuEST Forum database.

4.3.1 Customer Source Data

When the customer does not provide data that are required for a metric, thesupplier shall not be required to report the measurement for that customer.

NOTE: For a supplier to not be required to submit data for a metric to theMetrics Administrator all of the supplier’s customers must not be providing therequired information. If any customers do provide the information, then thesupplier shall submit data for the metric.

4.3.2 Acceptable Alternative Metrics

When the metric profile states that RQMS alternative reporting is acceptable,under the “Method of Delivery and Reporting” topic in the profile, the followingshall apply:

a. RQMS Data Acceptability

If a supplier is using the methods outlined in the latest issue of TelcordiaTechnologies document GR-929-CORE, Reliability and QualityMeasurements for Telecommunications Systems (RQMS)[1], to calculate aspecific measurement, those methods and the resulting data will beaccepted in lieu of the TL 9000 definition, if the following conditions aremet:(1) The data used for reporting to the QuEST Forum and its members

include all applicable data as prescribed by the TL 9000 definition of

4.2 MetricsReportingRequirements

4.3 Metrics Dataand Reports

Section 4 –General Metrics Requirements


4-3

the metric and it is not limited to the RQMS client company subset ofdata.

(2) For product categories not subject to RQMS reporting, the TL 9000definition shall be used.

b. TL 9000 Data Preference

In all cases, the TL 9000 definition of the metric is the preferred method.Where none of a supplier’s customers requires the supplier to generatethe RQMS reports, then the TL 9000 method shall be used. The suppliershall state which method is used when reporting this measurement.

NOTE: The intent of RQMS alternative metrics is to minimize redundanteffort by suppliers when both RQMS and TL 9000 metrics arecontractually required. In that case, compliance audits would accept anRQMS-based procedure coupled with meeting the conditions listed aboveas valid for calculating this measurement.

4.3.3 Report Frequency and Method

Unless otherwise specified by the profile, the supplier shall collect datamonthly and report the required results quarterly to the QuEST Forumdatabase. The supplier is free to use whatever time periods or formats thatare appropriate for reporting internally and to its customers. The quarterlyupdate shall include new data points from the preceding three (3) months.

4.3.4 Use of Fiscal Periods and Calendar Days

The supplier shall report TL 9000 measurement data either based oncalendar months or on defined fiscal months. The supplier shall use thechosen method consistently. The supplier shall notify customers and theMetrics Administrator prior to changing from one method to the other. Thesupplier shall use calendar days for the metrics that involve number of days.

4.3.5 Reporting of Compared Data and Research Data

The supplier shall report data for all applicable metrics defined in thishandbook to the Metrics Administrator according to agreed rules. Thisreporting requirement applies whether the supplier uses the TL 9000 methodor the RQMS alternative reporting and whether the metric includes thedesignation “compared data” (CD) or “research data” (RD). See Table A-5 inthe appendix.

NOTE: The designation “compared data” in the Method of Delivery andReporting section of the profile means that industry statistics may beavailable from the Forum Administrator. However, the designation “researchdata” indicates that no comparable industry statistics is available and the

Section 4 – General Metrics Requirements


4-4

Metrics Administrator will report analyses of industry trends only to theQuEST Forum metrics workgroups.

4.3.6 Product Exclusions

The supplier may exclude data on products that are no longer supported to itsgeneral customer base. Formal notification of placement of the product on“Additions and Maintenance (A&M)” or “Manufacturing Discontinued (MD)”status shall have been made to the customers for this exclusion to apply. 4.3.7 Metric Applicability Unless otherwise stated, metrics shall only apply to products during GeneralAvailability.

Section 5 – Common Metrics


5-1

Section 5 - Common Metrics

Common metrics are metrics that apply to all products: hardware, software,and services.

5.1 Number of Problem Reports (NPR) (C)

5.1.1 General Description and Title

Total Problem Reports (Complaints) Measurement is a measure of totalproblem reports as specified in the Metric Applicability (Normalized Units)Table A-2. This measurement is adapted from RQMS [1] and applied to allproducts: hardware (H), software (S), and services.

5.1.2 Purpose

The purpose of this metric is to measure the number of customer originatedproblem reports (complaints) that are indicative of the quality of the productdelivered during the operating life cycle of that product. This is being done inorder to reduce the number of problem reports, associated costs, andrevenue losses. Problem reports may have a negative impact to the supplier(such as rework), to the customer (such as scheduling repeat site visits) andmay reduce end user loyalty.


This metric applies to product categories as shown in Appendix A.

5.1.4 Detailed Description

a. Terminology – The Glossary includes definitions for the followingterms used for the NPR Metric:

• Customer• Duplicate Problem Report• General Availability (GA)• In-Service Release• Information Request (IR)• No Trouble Found (NTF)• Official Fix• Problem – Critical H/S• Problem – Major H/S



5-2

• Customer Problem – Minor H/S• Problem Report (PR)• Service Problem Reports• Severity Level• Release.

b. Counting Rules

(1) In the case of hardware or software, problem reports associatedwith any and all in-service supported release versions shall becounted.

(2) In the case of services, any formal report of a problem (complaint)after or during delivery of a service shall be counted.

(3) Only customer-originated problem reports shall be counted.(4) For hardware or software, any problem report after General

Availability of each release shall be counted unless otherwisespecified, regardless of whether it is related to software,hardware, documentation, no trouble found, procedural,duplicate, or other.

(5) Identical problem reports, i.e., multiple reports of the sameoccurrence of the same problem at the same location at the sametime, shall be counted as one (1) problem report.

(6) Duplicate problem reports, i.e., the same fault has occurred eitherat a different customer location or at another time, shall each becounted as separate problem reports.

(7) By agreement between the supplier and customer, a problem reportdetermined to represent an information request (IR) or request forfeature shall not be counted as a problem report.

(8) Multiple problems recorded on the same problem report (as in aproblem report form or screen) shall be counted separately,unless in the customer's view these problems are all related tothe same manifestation of failure experienced by the customer.

(9) Routine (expected maintenance) circuit pack, hardware returns(or associated paperwork), or software upgrades shall not becounted as problem reports. Circuit pack returns are measuredseparately. Problem reports on aspects of hardware design,software design, performance, service, and so on shall becounted.

(10) Routine written problem reports of outages, such as ServiceFailure Analysis Reports (SFAR), shall not be counted.

(11) The problem report shall be counted in the severity classificationat the time of reporting.

(12) Temporary fixes, such as temporary patches or workarounds, arefrequently used to resolve critical software or hardware problems.The official fix is often developed under a subsequent or“follow up” major or minor problem report that references theoriginal critical problem report. A critical problem report of this



5-3

type shall not be reclassified and shall be reported as a criticalproblem report. The subsequent major or minor problem reportshall not be counted.

(13) The supplier shall map its software or hardware severitycategories according to the classification in the Glossary.

c. Calculations and Formulas

(1) The measurement (See NPR1, NPR2 NPR3 and NPR4 inTable 5.1-1.) shall be calculated monthly as the total number ofincoming problem reports for all product categories listed inAppendix A, divided by the normalization factor listed in theProduct Category Table, Appendix A.

(2) In the hardware and software product categories where thecustomer cannot readily identify the quantity of product in usefactors, the problem report measurement (NPR) will not becalculated. The supplier will still be required to track the numberof problem reports and their resolution (In this case enternormalization factor = “none” in Table 5.1-3).

(3) When reporting RQMS alternative metrics for hardware and/orsoftware, suppliers shall refer to IPR1, IPR2, and IPR3 in Table5.1-2 to determine reporting conventions.

Notation

NU = Normalization Unit (NU) from Appendix A, Table A-2S = Normalization Factor; the total NU countNp1 = Number of Critical H/S Problem ReportsNp2 = Number of Major H/S Problem ReportsNp3 = Number of Minor H/S Problem ReportsNp4 = Number of Service Problem Reports

Table 5.1-1. Number of Problem Reports (NPR)Metric Identifiers and Formulas

Identifier Title FormulaNPR1 H/S Critical Problem Reports per NU Np1 / SNPR2 H/S Major Problem Reports per NU Np2 / SNPR3 H/S Minor Problem Reports per NU Np3 / SNPR4 Service Problem Reports per NU Np4 / S



5-4

Table 5.1-2. Number of Problem Reports –RQMS Alternative Metrics (IPR)

Identifier TitleIPR1 Incoming Critical Problem Reports per system per monthIPR2 Incoming Major Problem Reports per system per monthIPR3 Incoming Minor Problem Reports per system per month

d. Reported Data and Format

(1) Data shall be reported quarterly. Each report shall include datafor the three (3) months in the quarter.

(2) TL 9000 Data NPR Table (Table 5.1-3) – The NPR measurementshall be reported with data elements (or equivalent as defined bythe Forum Administrator) for each month and each productcategory as follows:

Table 5.1-3. NPR Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: (From Appendix A)Metric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersNormalization Factor: S or noneAnnualization Factor: Afactor (See Glossary.)Metric Identifier: NPRNPR1 Numerator: Np1NPR2 Numerator: Np2NPR3 Numerator: Np3NPR4 Numerator: Np4

(3) RQMS Alternative Data Table (Table 5.1-4) – The RQMSalternative metrics shall be reported with data elements (orequivalent as defined by the Metrics Administrator) for each monthand each product category as follows:



5-5

Table 5.1-4. RQMS Alternative NPR Data Table (IPR)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: (From Appendix A)Metric Methodology: RQMSCustomer Base: Either (1) Total or (2) Forum MembersNormalization Factor: Number of systems in serviceMetric Identifier: IPRIPR1 Numerator: # of incoming critical problem reportsIPR2 Numerator: # of incoming major problem reportsIPR3 Numerator: # of incoming minor problem reports

5.1.5 Sources of Data

Data for the NPR measurement is derived from information provided bycustomers and from supplier analysis as follows:

a. Customers• Report problems to the supplier• Report normalizing information for hardware or software

categories to the supplier according to the Product CategoryTable, Appendix A.

b. Suppliers• Count reported problems by product category and customer

base and convert to “number of problem reports” according tothe applicable counting rules

• For service products, track and report service normalizationunit

• Calculate the normalization factor.

5.1.6 Method of Delivery or Reporting

a. Compared Data (CD) or Research Data (RD):

Critical Problem Reports per NU: CDMajor Problem Reports per NU: CDMinor Problem Reports per NU: CDService Problem Reports per NU: CD

b. RQMS Alternative Reporting:

Critical Problem Reports per NU: YESMajor Problem Reports per NU: YESMinor Problem Reports per NU: YESService Problem Reports per NU: NO



5-6

5.1.7 Example Calculations

a. Example 1 – NPR for H/S Products

(1) Consider one (1) month’s data for a supplier of a particular operationalsupport system (OSS) sold to both members and non-members of theQuEST Forum. There are 30 systems in service during the entiremonth and NU is “systems in service.”

(2) The data reported is shown in Table 5.1-5.

Table 5.1-5. Example 1: NPR Data Report

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 4.2Metric Methodology: TL 9000Customer Base: TotalNormalization Factor: 30Annualization Factor: 12Metric Identifier: NPRNPR1 Numerator: 0NPR2 Numerator: 3NPR3 Numerator: 45NPR4 Numerator: Not applicable

(3) The calculation of the measurement would be:

Table 5.1-6. Example 1 NPR Source Data and Measurements

ProblemReports Severity

NormalizationFactor (S)

YYYYMM NPRMeasurement (NPR1,2,3,4)

0 (Np1) Critical 30 0 Critical Problem Reports per system per month

3 (Np2) Major 30 0.1 Major Problem Reports per system per month

45 (Np3) Minor 30 1.5 Minor Problem Reports per system per month

(Np4) Service Problem Reportsare not applicable for thisproduct

b. Example 2 – NPR for Services Products

(1) Consider one (1) month’s data for a supplier of a particularmaintenance service sold to both members and non-members of the



5-7

QuEST Forum. There are 20 units maintained during the entiremonth and NU is “units served.”

(2) Data reported is shown in Table 5.1-7

Table 5.1-7. Example 2: NPR Data Report

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 7.3Metric Methodology: TL 9000Customer Base: TotalNormalization Factor: 20Annualization Factor: 12Metric Identifier: NPRNPR1 Numerator: Not applicableNPR2 Numerator: Not applicableNPR3 Numerator: Not applicableNPR4 Numerator: 30

(3) The calculation of the measurement is shown in Table 5.1-8.

Table 5.1-8. Example 2: NPR Source Data and Measurements

ProblemReports Severity

NormalizationFactor (S)

YYYYMM NPRMeasurement (NPR1,2,3,4)

(Np1) Critical – H/S Critical ProblemReports are not applicablefor services(NPR1)

(Np2) Major – H/S Major ProblemReports are not applicablefor this product (NPR2)

(Np3) Minor – H/S Minor ProblemReports are not applicablefor this product (NPR3)

30(Np4)

Notapplicablefor Services

20 1.5 Service Problem Reportsper unit maintained permonth (NPR4)



5-8

5.2 Problem Report Fix Response Time (FRT) (C) and Overdue ProblemReport Fix Responsiveness (OFR) (C) Measurements


Problem Report Fix Response Time (FRT) is the overall responsiveness ofthe supplier to reported problems. The Problem Report Fix Response Timeapplies only to major and minor hardware/software (H/S) problem reports andto all service problem reports. Critical H/S problem reports are excludedbecause critical problems are worked on a high priority basis until they areclosed.

Overdue Problem Report Fix Responsiveness (OFR) is the rate of closure ofoverdue major and minor H/S problem reports and all service problemreports.

These measurements are adapted from RQMS. [1]

5.2.2 Purpose

The purpose of this metric is to quantify the responsiveness to problemreports and facilitate prompt fixes and closures of problem reports.


These metrics apply to product categories as shown in Appendix A.


a. Terminology – The Glossary includes definitions for the followingterms used for the NPR Metrics:

• Closure Criteria• Closure Interval• Closure Date• Fix• Fix Response Time• Information Request (IR)• Official Fix• Overdue Problem Report• Problem - Critical H/S• Problem - Major H/S



5-9

• Problem - Minor H/S• Problem Report PR• Severity Level• Temporary Fix.

b. Counting Rules

(1) Only Problem Reports (PRs) that are originated by a customerand meet the criteria for Number of Problem Reports shall beincluded in the responsiveness measurement.

(2) The start of the interval for calculating FRT shall be considered asthe receipt of that problem report by the supplier. Routine writtenreports of outages (such as SFARs) shall be excluded.

(3) The end of the interval for calculating FRT shall be considered asthe date that the official fix or closure criteria is made available.Should the problem report originator later reject the fix asincomplete or causing side effects, the problem report shall be re-classified as open.

(4) If, during the processing of a problem report, the severity of theproblem is reclassified (such as from critical to major and frommajor to minor) the FRT shall be the total time accumulated fromthe time the problem was originally opened. The total FRT shallbe reported in the severity classification at the time of closure,unless specified otherwise.

(5) The customer has the final determination that a problem report isresolved. All resolutions must be acknowledged by the customerthat the solution provided by the supplier meets the customer’srequirements. This is particularly relevant to the resolution ofduplicate problem reports where the criteria may vary byindividual customer.

(6) Since this measurement is intended to measure the supplier’s fixresponse time, any extraordinary delays in the closure of aproblem report caused by the customer may be deleted from theoverall closure time. The supplier shall keep records of suchdelays with specific start and stop dates. Examples of this type ofevent are:- Excess delay in testing of a proposed solution due to

customer staffing constraints;- After opening a problem report and being requested for

needed data by the supplier, the customer delays supplyingsufficient information for the supplier to commence problemresolution;

- The supplier is unable to access a facility to resolve a serviceproblem report.

(7) If the deployment of the fix is delayed (or does not occur)specifically at customer request (and not because of supplierproblems), the interval is defined as ending when the official fix isfirst made available for delivery. The delay interval shall not beincluded in the FRT calculation.



5-10

(8) If, with customer consent, the implementation of a fix is deferred(such as if the customer agrees to wait for the next softwareupdate versus a patch) then the deferral interval shall not beincluded.

(9) A problem report determined to represent an information request(IR) or a request for feature shall not be counted as a problemreport.

(10) The suppliers shall map their severity categories according to theseverity level definitions in the Glossary.

(11) Critical problem reports shall not be counted in this metric.(12) The following rules shall apply to Overdue Problem Report Fix

Responsiveness:i. Overdue problem reports are those that are open beyond the

due threshold time. The due threshold time is defined as:- 30 calendar days for major H/S problem reports and- 180 calendar days for minor H/S problem reports- A closure date agreement made between the customer

and the supplier for all service problem reports.(Expected closure intervals for services may bepredetermined by a contractual agreement).

ii. Open Problem Reports shall be counted as overdue in eachmonth during which they are open and overdue.

For example, if a problem report number 123 is open andoverdue in month m and did not close by the last day ofmonth m, then it shall count as overdue in month m andoverdue in month m+1 even if it closed on day one (1) ofmonth m+1.

(13) Penalty problem reports are counted in the OFR metric and areapplicable only to hardware and software products. A penaltyproblem report is defined as:

- For majors, all problem reports with age since opening whichexceed 180 calendar days;

- For minors, all problem reports with age since opening whichexceed 270 calendar days;

- Penalty problem reports shall also be counted as overdueproblem reports (that is, double counting constitutes the“penalty”).


(1) Each of the FRT measurements (See FRT2, FRT3, and FRT4 inTable 5.2-1.) shall be calculated monthly as the percentage of thetotal number of overall problems due to be closed during themonth and that were delivered on time by the due threshold timeas defined in the counting rules.

(2) Each of the OFR measurements (See OFR in Table 5.2-1) shallbe calculated as follows:



5-11

i. The sum of penalty problem reports for the month shall beadded to the sum of the overdue problem reports for themonth.

ii. The number of overdue problem reports closed are thoseoverdue problem reports that were closed in the month.

iii. The measurement is computed as the number of overdueproblem reports closed divided by the sum of overdue problemreports and the total number of penalty problem reports. Theresult shall be expressed as a percentage.

iv. The measurement shall be reported as 100% in the casewhere there are no overdue problem reports during the period.

(3) When reporting RQMS alternative metrics for FRT metrics,suppliers shall refer to ORT2 and ORT3 in Table 5.2-2 todetermine reporting conventions.

Notation

Fr2 = Major H/S fixes delivered on timeFr3 = Minor H/S fixes delivered on timeFr4 = Service problem reports resolved on timeFrd2 = Number of major H/S fixes due to be closedFrd3 = Number of minor H/S fixes due to be closedFrd4 = Number of service problem reports due to be closedPro2 = Number of overdue major H/S problem reportsPro3 = Number of overdue minor H/S problem reportsPro4 = Number of overdue service problem reportsPrp2 = Number of major H/S penalty problem reportsPrp3 = Number of minor H/S penalty problem reportsPrc2 = Number of overdue major H/S problem reports closedPrc3 = Number of overdue minor H/S problem reports closedPrc4 = Number of overdue service problem reports closed



5-12

Table 5.2-1. Problem Report Fix Response Time (FRT) andOverdue Problem Report Fix Responsiveness (OFR)Metric Identifiers and Formulas

Identifier Title Formula NoteFRT2 H/S Major Problem

Reports Fix ResponseTime

100(Fr2 / Frd2) % delivered on time

FRT3 H/S Minor ProblemReports Fix ResponseTime

100(Fr3 / Frd3) % delivered on time

FRT4 Service ProblemReportsFix Response Time

100(Fr4 / Frd4) % resolved on time

OFR2 H/S Major OverdueProblem Report FixResponsiveness

100(Prc2/[Pro2+Prp2]) % closed

OFR3 H/S Minor OverdueProblem Report FixResponsiveness

100(Prc3/[Pro3+Prp3]) % closed

OFR4 Service OverdueProblemReport FixResponsiveness

100(Prc4/Pro4) % closed

Table 5.2-2. Problem Report Fix Response Time and OverdueProblem Report Fix Responsiveness RQMS AlternativeMetrics (ORT and OPR)

Identifier TitleORT2 % Major Problems Closed on TimeORT3 % Minor Problems Closed on TimeOPR2 % Rate of Closures of Overdue Problem Reports – MajorOPR3 % Rate of Closures of Overdue Problem Reports – Minor



(2) TL 9000 FRT Data Table – The FRT and OFR metrics shall bereported with data elements (or equivalent as defined by theMetrics Administrator) for each month and each product categoryas shown in Table 5.2-3.



5-13

Table 5.2-3. FRT or OFR Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersAnnualization Factor Afactor (See Glossary.)Metric Identifier: FRT or OFRFRT2 or OFR2 Numerator: Fr2 or Prc2FRT3 or OFR3 Numerator: Fr3 or Prc3FRT4 or OFR4 Numerator: Fr4 pr Prc4FRT2 or OFR2 Denominator: Frd2 or Pro2FRT3 or OFR3 Denominator: Frd3 or Pro3FRT4 or OFR4 Denominator Frd4 or Pro4OFR2 Denominator 2nd Term: Prp2OFR3 Denominator 2nd Term: Prp3

(3) RQMS Alternative Data Table – The RQMS alternative metrics shallbe reported with data elements (or equivalent as defined by theMetrics Administrator) for each month and each product category asshown in Table 5.2-4 and Table 5.2-5.

Table 5.2-4. RQMS Alternative FRT Data Table (ORT)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: (From Appendix A)Metric Methodology: “RQMS”Customer Base: Either (1) Total or (2) Forum MembersMetric Identifier: ORT

ORT2 Numerator: The total number of major fixes due to be closedduring the three (3)-month window that weredelivered on time

ORT2 Denominator: The total number of major fixes that were due tobe delivered during the three (3)-month window

ORT3 Numerator: The total number of minor fixes due to be closedduring the three (3)-month window that weredelivered on time

ORT3 Denominator The total number of minor fixes that were due tobe delivered during the three (3) month window



5-14

Table 5.2-5. RQMS Alternative OFR Data Table (OPR)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: (From Appendix A)Metric Methodology: RQMSCustomer Base: Either (1) Total or (2) Forum MembersMetric Identifier: OPR

OPR2 Numerator: The sum of the overdue major problemreports closed in the three (3)-month period

OPR2 Denominator: The sum of penalty major problem reports forthe three (3)-month period added to the sumof the overdue major problem reports for thesame period

OPR3 Numerator: The sum of the overdue minor problemreports closed in the three (3)-month period

OPR3 Denominator: The sum of penalty minor problem reports forthe three (3)-month period added to the sumof the overdue minor problem reports for thesame period


The data for the FRT measurement are derived from information providedby customers and from supplier analysis as follows:

a. Customers

• Report problems to the supplier• Confer with supplier to establish severity classification for H/S• Agreed upon closure interval for service problem reports• Agree with decision for problem report closure.

b. Suppliers

• Track problem reports, their severity (H/S) the agreed uponclosure interval (services), and actual closure dates

• Count due, overdue and on-time fixes and problem reports, andcompute the measurements according to the stated rules.



5-15


a. Compared data (CD) or research data (RD):

Major H/S Problem Report Fix Response Time: CDMinor H/S Problem Report Fix Response Time: CDMajor H/S Overdue Problem Report Fix Responsiveness: RDMinor H/S Overdue Problem Report Fix Responsiveness: RDService Problem Report Fix Response Time: CDService Overdue Problem Report Fix Responsiveness: RD


Major H/S Problem Report Fix Response Time: YESMinor H/S Problem Report Fix Response Time: YESMajor H/S Overdue Problem Report Fix Responsiveness: YESMinor H/S Overdue Problem Report Fix Responsiveness: YESService Problem Report Fix Response Time: NOService Overdue Problem Report Fix Responsiveness: NO


a. Example 1 – FRT for an H/S Product

(1) Consider one (1) month’s data for a supplier of a particular OSS soldto both members and non-members of the QuEST Forum. There arefive (5) fixes to major problem reports due to be closed during themonth and all five (5) were delivered on time. There are 25 fixes tominor H/S problem reports due and 20 were delivered on time.

(2) The FRT data reported is shown in Table 5.2-6.



5-16

Table 5.2-6. Example FRT Data Report

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 4.2Metric Methodology: TL 9000Customer Base: TotalAnnualization Factor: 12Metric Identifier: FRTFRT2 Numerator: 5FRT3 Numerator: 20FRT4 Numerator Not applicableFRT2 Denominator: 5FRT3 Denominator: 25FRT4 Denominator Not applicable

(3) The calculation of the FRT measurements would be:

Table 5.2-7. Example FRT Source Data and MeasurementCalculation

On-TimeFixes Severity Fixes Due

YYYYMM FRTMeasurement(FRT2,3)

5 (Fr2) Major 5 (Frd2) 100% Major H/S ProblemReport Fixes DeliveredOn Time

20 (Fr3) Minor 25 (Frd3) 80% Minor H/S ProblemReport Fixes DeliveredOn Time

(Fr4) Services (Frd4) – Service ProblemReports are notapplicable for thisproduct.

b. Example 2 – OFR for an H/S Product

(1) At the beginning of the month, there were six (6) major H/S problemreports that were overdue (age > 30 calendar days). One (1) of thesebecame a penalty major H/S problem report during the month (age >180 calendar days). Two (2) of the six (6) overdue reports wereclosed during the month. There was no overdue minor H/S problemreport at the beginning of the month. However, by the end of themonth five (5) minor H/S problem reports for which fixes had beendue during the month had become overdue. One (1) of these overdueminor H/S problem reports was closed before the end of the month.

(2) OPR data reported is shown in Table 5.2-8.



5-17

Table 5.2-8. Example OFR Data Report

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 4.2Metric Methodology: TL 9000Customer Base: TotalAnnualization Factor: 12Metric Identifier: OFROFR2 Numerator: 2OFR3 Numerator: 1OFR4 Numerator: Not applicableOFR2 Denominator: 6OFR3 Denominator: 5OFR4 Denominator: Not applicableOFR2 Denominator – 2nd Factor: 1OFR3 Denominator – 2nd Factor: Not applicable

(3) The calculation of the OFR measurements for the month is shown inTable 5.2-9.

Table 5.2-9. Example OFR Source Data and MeasurementCalculation

ClosedOverdueProblems Severity

FixesOverdue

PenaltyProblemReports

OPRMeasurement Result

2 (Prc2) Major 6 (Pro2) 1 (Prp2) 100 x 2 / 7 = 28.6% %Overdue Major Problem Reports Closed

1 (Prc3) Minor 5 (Pro3) 0 (Prp3) 100 x 1 / 5 = 20% % Overdue Minor Problem Reports Closed

(Prc4) Services (Pro4) notapplicable

Service Problem Reportsare not applicable for thisproduct.

c. Calculations: Example 2 (Services)

(1) Consider one (1) month’s data for a supplier of a particular installationservice sold to both members and non-members of the QuESTForum. There are 20 service problem reports due to be closed duringthe month and 16 were resolved on time.

(2) FRT data reported are shown in Table 5.2-10.



5-18

Table 5.2-10. Example FRT Data Report (Services)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 7.1Metric Methodology: TL 9000Customer Base: TotalAnnualization Factor: 12Metric Identifier: FRTFRT2 Numerator: Not applicableFRT3 Numerator: Not applicableFRT4 Numerator 16FRT2 Denominator: Not applicableFRT3 Denominator: Not applicableFRT4 Denominator: 20

c. The calculation of the FRT measurements is shown in Table 5.2-11.

Table 5.2-11. Example FRT Source Data and MeasurementCalculation

On-TimeClosures

FixesDue

YYYYMM FRTMeasurement(FRT4)

16(Fr4) 20(Frd4) 80% Service Problem ReportsResolved On Time

d. Example 4 – OFR for a Services Product

(1) At the beginning of the month, there were two (2) service problemreports that were overdue (age greater than the agreed closureinterval). One (1) of the two (2) overdue reports was closed during themonth.

(2) OPR data reported would be as shown in Table 5.2-12.



5-19

Table 5.2-12. Example OFR Data Report

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: 7.1Metric Methodology: TL 9000Customer Base: TotalAnnualization Factor: 12Metric Identifier: OFROFR2 Numerator: Not applicableOFR3 Numerator: Not applicableOFR4 Numerator: 1OFR2 Denominator: Not applicableOFR3 Denominator: Not applicableOFR4 Denominator: 2OFR2 Denominator – 2nd Factor: Not applicableOFR3 Denominator – 2nd Factor: Not applicable

(3) The calculation of the OFR measurements for the month is shown inTable 5.2-13.

Table 5.2-13. Example OFR Source Data and MeasurementCalculation for a Services Product

Closed Severity Fixes Penalty OPROverdue Fixes Problem MetricProblems Over-due Reports Result1 (Prc4) Not 2 (Pro4) Not 100 x 1 / 2 = 50%

applicable applicable %Overdue ServiceProblem Reports Closed



5-20

5.3 On Time Delivery (OTD) (C)


On Time Delivery (OTD) is a measure of timeliness of all product ordersdelivered to customers.

5.3.2 Purpose

The purpose of this metric is to evaluate the supplier’s on time deliveryperformance in order to meet the customer’s need for timely product deliveryand to meet end customer expectations.


This metric applies to product categories as shown in Appendix A. It does notapply to continuous services (e.g., call centers) where service is measured byservice problem reports.


a. Terminology – The Glossary includes definitions for the following termsused for the OTD metric:

• Customer Requested Date (CRD)• Date Order Received• Late Orders Received (LOR)• Customer Requested On-Job Date (CROJD)• Customer Requested Completion Date (CRCD)• Actual Completion Date (ACD)• Actual On-Job Date (AOJD)• Installed System Orders• Item Orders• Service Orders.

b. Counting Rules

(1) A system that includes any combination of hardware, software andservice applications are counted as one (1) order.

(2) Acceptance shall be defined according to purchase order and/orcontract terms and conditions unless notified otherwise by thecustomer.



5-21

(3) Due dates and delivery dates are considered to be one (1) 24-hourperiod (customer’s calendar day).

(4) Early order completions or deliveries are considered to have missedthe delivery date unless authorized by the customer.

(5) Late Orders Received (LOR) are those for which CRD is earlier thanDate Order Received, and are excluded from the metric.

(6) Actual Completion Date (ACD) is the date when service is completeat a job site and accepted by the customer.

(7) CRD is the initial CRD date or, in the case of customer requestedchanges, the revised date.

(8) The monthly OTD data shall include all orders having CRD occurringduring the same month.

(9) Actual On-Job Date (AOJD) identifies the date when the shipmentactually was delivered at the ship to address. This date may bederived by adding the transportation interval to the actual ship date.

(10) CRD is either CRCD or CROJD depending on order type.(11) Order types can be: installed system, items, or service.(12) Compound orders designated by the customer for a single delivery

(“must ship complete” orders) shall be treated in aggregate. If one(1) line item is late, then all line items shall be counted as late.


(1) On Time Delivery (OTD) (See OTD in Table 5.3-1.) is the percentageof orders/items accepted on the Customer Requested Date (CRD)where CRD is equal to either ACD or AOJD depending on order type.

(2) OTD is calculated as 100 multiplied by the number of orders/itemsaccepted on the CRD during the month divided by the number oforders/items for which CRD occurred during the month.

(3) OTD is comprised of three (3) measurements of order fulfillment, asfollows:i. Percentage of installed systems orders accepted on Customer

Requested Completion Date (CRCD),ii. Percentage of line items accepted on Customer Requested On-

Job Date (CROJD), andiii. Percentage of service orders accepted on Customer Requested

Completion Date (CRCD).

Notation

Cs = Number of installed systems for which CRCD occurred during themonth

Ss = Number of installed systems accepted on the CRCD during themonth

Ci = Number of items for which CROJD occurred during the monthSi = Number of items accepted on the CROJD during the monthCv = Number of service orders for which CRCD occurred during the

month



5-22

Sv = Number of service orders accepted on the CRCD during themonth

Table 5.3-1. On Time Delivery (OTD)Metric Identifiers and Formulas

Identifier Title Formula NoteOTIS On Time Installed

System Delivery100(Ss / Cs) % accepted

on CRDOTI On Time Items

Delivery100(Si /Ci) % accepted

on CRDOTS On Time Service

Delivery100(Sv /Cv) % accepted

on CRD


(1) Data shall be reported quarterly. Each report shall include data forthe three (3) months in the quarter.

(2) TL 9000 OTD Data Table – The OTD measurements shall be reportedwith data elements (or equivalent as defined by the MetricsAdministrator) for each month and each product or product/servicecategory as follows (Table 5.3-2):

Table 5.3-2. OTD Data Table

Year and Month: YYYYMMReporting ID: Provided by the Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersMetric Identifier: OTDOTIS Numerator: SsOTI Numerator: SiOTS Numerator: SvOTIS Denominator: CsOTI Denominator: CiOTS Denominator: Cv


The data from which OTD is derived is obtained from one (1) or more of thefollowing sources:

a. Supplier’s order entry department;b. Installation teams; andc. Customer data.



5-23


a. Compared Data(CD) or Research Data (RD)

On Time Installed System Delivery: CDOn Time Items Delivery: CDOn Time Service Delivery: CD

5.3.7 Examples

a. Table 5.3-3 illustrates computation of OTD measurement from a series ofinstallations of systems per purchase order (PO).

Table 5.3-3. Example of Installed System OTD

PurchaseOrder

CRDmm/dd

LineItem

QuantityOrdered

QuantityInstalled

Date In-stalled

DateAccepted

On TimeInstallations

Note

A 03/10 1 5 5 3/10 12 6 6 3/103 4 4 3/10 3/10

B 03/20 1 8 4 3/22 0 14 3/23

2 12 6 3/226 3/25 3/25

C 03/21 1 2 2 3/21 02 2 1 3/21

1 3/22 3/22D 02/15 1 7 7 3/15 na 2

2 1 1 3/15 3/15E 03/25 1 1 1 3/15 3/15 0 3

TOTALS: Number ofOrders

# SystemCRDs Due inMonth (Cs)

lineItem

Count

Line ItemQuantitiesOrdered

On TimePurchase

Orders (Ss)

5 4 10 48 1March OTD

(Ss/Cs)25.0%(OTIS)

4

NOTES:1. Order B – 2 line items were split into 4 partial installations – each with a separate date installed.

2. PO system D CRD was not counted in the total of 4 for March as it had a February CRD.3. PO E Service Order was completed early. Thus, CRD was not met.4. The CRD installed system OTDI performance for March was 25% or 1(CRD met) / 4 (CRDs

due).

b. Table 5.3-4 illustrates computation of OTD measurement from a series ofservices per purchase order (PO).



5-24

Table 5.3-4. Example of On Time Service Delivery OTD

PO CRDmm/dd

LineItem

QuantityOrdered

QuantityComple-

ted

Compl.Date

mm/dd

Accep-tancemm/dd

OTSCRD

Note

F 3/10 1 5 5 3/10 12 6 6 3/103 4 4 3/10 3/10

G 3/20 1 8 4 3/22 04 3/23

2 12 6 3/226 3/25 3/25

H 3/21 1 2 2 3/21 02 2 1 3/21

1 3/22 3/22I 2/15 1 7 7 3/15 na 1

2 1 1 3/15 3/15J 3/25 1 1 1 3/15 3/15 2

Numberof

Orders

CRDsDue inMarch

Cv

# ofLine

Items

TotalQuantitie

s

On TimeOrders

SV

TOTAL: 5 4 9 48 1March OTD

Sv/Cv25%(OTS)

3

NOTES:1. PO system I CRD was not counted in the total of 3 for March as it had a

February CRD date and it was previously counted.2. Service Order was completed early. Thus, CRD was not met.3. The CRD OTD performance for March was 25% or 1(CRD met) / 4 (CRDs

due).



5-25

c. Table 5.3-5 illustrates computation of OTD measurement from a series ofdelivered line items per purchase order (PO).

Table 5.3-5. Example of On Time Item Delivery OTD

PO LineItem

QtyOrdered

CRD SplitOrder

QtyDelivered

Actual OTDCRD

Note

K 1 5 3/10 5 3/10 12 6 3/12 6 3/13 03 4 3/17 4 3/18 0

L 1 8 3/20 8 3/22 02 12 3/22 y 6 3/22 0 1

y 6 3/25 0 13 2 3/29 2 ? 04 2 3/30 2 3/30 1

M 1 7 2/15 7 3/15 na 22 1 2/15 1 3/15 na 2

Numberof

Orders

# LineItems

TotalQuantities

CRDsDue inMarch

Ci

OnTime

OrdersSi

TOTAL: 3 9 47 7 2March OTD

Si/Ci29%(OTI)

3

NOTES:1. Line item L2 was not on time for CRD because only ½ of the items were

delivered to CRD.2. PO line items M1 and M2 CRDs were not counted in the total of 7 for

March as they had Feb CRD dates and were previously counted.3. The CRD OTD performance for March was 29% or 2 (CRD met) / 7

(CRDs due).



5-26

d. The data that would be reported for the for the above examples is inTable 5.3-6.

Table 5.3-6. Example On Time DeliveryData Report (OTD)

Year and Month: 200103Reporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: TotalMetric Identifier: OTDOTIS Numerator: 1OTIS Denominator: 4OTI Numerator: 2OTI Denominator: 7OTS Numerator: 1OTS Denominator: 4



5-27

5.4 System Outage Measurement (SO) (HS)


System Outage (SO) applies only to hardware and software products. SO isa measurement of complete loss of functionality of all or part of anytelecommunications system (not just switching). Partial loss of functionality isnot measured. The SO measurement expresses the annualized total number(outage frequency) and total duration (downtime) of outages experienced byan average system. These measures translate directly into system MeanTime Between Failures (MTBF) and system availability respectively. The SOmeasurements are calculated for both overall (all causes) and for supplier-attributable causes. Supplier-attributable availability/unavailability is oftenspecified in system reliability performance requirements. The SOmeasurement provides the method of calculation for various system types.

5.4.2 Purpose

The purpose of this metric is to evaluate the downtime performance andoutage frequency during field operation in order to reduce both thefrequency and duration of outages and their associated cost and revenueimpact.


This metric applies to product categories as shown in Appendix A.Products specifically not included are single circuit packs or non-systemproducts for which the term outage has no meaning.


The supplier shall provide two (2) sets of measurements for each productcategory code: (1) overall outage frequency and downtime and (2)supplier attributable outage frequency and downtime. This metric countsonly complete loss of functionality of all or part of any telecommunicationssystem.

a. Terminology – The Glossary includes definitions for the followingterms used for the SO metric:• Supplier Attributable Outage• Customer Attributable Outage• Customer Base• Downtime Performance Measurement (DPM) (only required for

suppliers that elect RQMS equivalent reporting)



5-28

• Outage Frequency Measurement (OFM) (only required forsuppliers that elect RQMS equivalent reporting).

b. Counting Rules

(1) Outages Counted – Unless an exception applies, all outages andonly outages representing complete loss of functionality of all orpart of any telecommunications system shall be counted asfollows:i. Complete loss of functionality of any duration greater than 30

seconds is counted.ii. Scheduled outages are counted unless the system is allocated

a maintenance window during which the system is not requiredto be in service.

iii. Outages attributed to the customer are counted as part of theoverall metric.

iv. For systems that are not operational 24X7, count outages andduration that occur during the operational window.

Examples of complete loss of functionality of all or part of anytelecommunications system include:

− In switching systems, loss of origination and terminationcapability for all or part of the office is counted.

− In a tandem office or Signaling Transfer Point (STP), anytotal loss of Common Channel Signaling (CCS) is countedas total loss of functionality.

− In a mated pair Service Control Point (SCP), only binodaloutages resulting in complete loss of processing arecounted.

− Service order processing system cannot process any ordersand new orders cannot be entered.

(2) Outages Not Counted, Exceptions – The exceptions to countingall outages are as follows:

i. Outages due to natural disasters are not counted.ii. A remote system in stand-alone mode is not considered out of

service. (Some functionality exists.)

iii. A CCS outage in an end office is not counted. (Somefunctionality exists.)

iv. Loss of feature functionality, such as Calling Number Delivery,etc., is not counted.

v. Individual termination downtime is not counted.(3) Counting by Release and by Host/Remote Systems - The

following shall apply:

i. Performance of individual releases is not counted separately.



5-29

ii. Performance of host systems and remote systems of a producttype is not counted separately.


(1) The measurement (See SO1, SO2, SO3, and SO4 in Table 5.4-1.)is calculated monthly for overall and supplier attributable outages.

(2) When reporting RQMS alternative metrics for product categorieswhere one (1) of the RQMS designations, “end and/or tandemoffice,” “wireless,” or “next generation digital loop carrier (NGDLC)”applies, suppliers shall refer to Table 5.4-2 for reportingrequirements.

(3) When reporting RQMS alternative metrics for product categorieswhere the RQMS designations in (2) above do not apply, suppliersshall refer to Table 5.4-3 for reporting requirements.

Notation

P = Overall Weighted Outage MinutesPs = Supplier Attributable Weighted Outage MinutesQ = Overall Weighted OutagesQs = Supplier Attributable Weighted OutagesS = Normalization Factor, the total number of normalization units

that are in service during the monthNU = Normalization Unit from Appendix A

Table 5.4-1. System Outage Measurement (SO)Metric Identifiers and Formulas

Identifier Title Formula NoteSO1 Annualized Weighted

Outage Frequency12 (Q / S) Weighted outages

per NU per yearSO2 Annualized Weighted

Downtime12 (P / S) Minutes

per NU per yearSO3 Annualized Supplier

Attributable Outage Frequency

12 (Qs / S) Weighted outages per NU per year

SO4 Annualized Supplier Attributable Downtime

12 (Ps / S) Minutes per NU per year



5-30

Table 5.4-2. System Outage Measurements (SOE) RQMS AlternativeMetrics End Office and/or Tandem Office, WirelessProducts, and NGDLC Products

NOTE: Report only the measurements in this table that areapplicable to the specific product as defined by RQMS. [1]

Identifier TitlerDPMs Supplier Attributable Total Outage Minutes per System per Year

– remote onlyhDPMs Supplier Attributable Total Outage Minutes per System per Year

– host onlyrDPMc Service Provider Attributable Total Outage Minutes per System

per Year – remote onlyhDPMc Service Provider Attributable Total Outage Minutes per System

per Year – host onlyrOFMs Supplier Attributable Total Outages per System per Year

– remoteshOFMs Supplier Attributable Total Outages per System per Year – hostsrOFMc Service Provider Attributable Total Outages per System per Year

– remoteshOFMc Service Provider Attributable Total Outages per System per Year

– hosts

Table 5.4-3. System Outage Measurements (SOG) RQMS AlternativeMetrics General Series

Identifier TitleDPM Total Outage Minutes Per System Per Year –overallDPMs Total Outage Minutes Per System Per Year –supplier attributableOFM Total Outages Per Year – overallOFMs Total Outages Per Year – supplier attributable

(4) Detailed formulas for the downtime numerator quantities P, Ps, Q,and Qs and for the normalization factor S are given in thefollowing analysis.

Notation

M = number of months in outage measurement (M=1 for monthlymeasurements)

N = number of systems in service at the end of the monthm = number of outagesPi = duration of the ith outage



5-31

AI = number of normalization units (lines, DS1s, etc.) affected inoutage i

Si = number of normalization units (lines, DS1s, etc) in the systemthat were affected by outage i from Appendix A, that is, thesystem size

=S = average of number of normalization units (lines, DS1s, etc)from Appendix A per system

∑=

=N

1n

nN

SS

fi = fraction of normalization units (lines, DS1s, etc.)from Appendix A affected in outage I

i

ii S

Af =

Downtime Formulas

(i) General Downtime Formula

DT = annualized system downtime metric

( )

NS

SPfx

M12

DT

m

1i

iii∑

=

= (5.4-1)

Substituting Ai / Si for fi,

∑=

=m

1i

iis

S

PAPP, (5.4-2)

NS = (5.4-3)

Then, for monthly data, downtime may be expressed as:

SP

12DT x = = overall downtime (5.4-4)

SP

12DT SS x = = supplier attributable downtime

(5.4-5)

This general formula is used for products of varying system size suchas switching products. (See Non-Uniform System example – EndOffice, subsection 5.4.7.a.) The two (2) special cases that follow areincluded to show simplified versions of the formula.



5-32

(ii) Downtime Special Case 1

If products are of uniform system size and if only total system outages

are possible, then Ai = Si = S and the formula reduces to theannualized sum of minutes of downtime normalized to number ofsystems in service. This is comparable to the current RQMScalculation. For monthly data:

( )[ ]N

Pi12x

N

SSPi Si

Ai

12xDT

m

1i

m

1i

i ∑∑== =

=

(5.4-6)

Then the formulas for P and S are:

∑=

=m

iis PPP

1

, (5.4-7)

S = N (5.4-8)

(iii) Downtime Special Case 2

For products that express downtime in terms of normalization units,

substitute Σ (Sn / N) for S to eliminate N. The formula then reducesto the annualized sum of minutes of downtime multiplied by thenormalization units affected normalized to total number ofnormalization units in the population. (See example of TransportSystem – Digital Cross Connect, subsection 5.4.7.b.) For monthlydata:

( )[ ]

( )[ ]

N

NS

1Pi 1Ai

x 12

NS

SPi SiAi

x 12DT

N

1n

ni

m

1i

m

1i

i

=

=

∑∑

∑

=

=

=

∑

∑

=

==N

1nn

m

1iii

S

PAx 12DT (5.4-9)



5-33

For this case the formulas for P and S are:

∑=

=m

1iiis PAPP, (5.4-10)

∑=

=N

1nns SSS, (5.4-11)

Outage Frequency Formulas

(i) General Outage Frequency

OF = annualized outage frequency metric

NS

Sfx

M12

OF

m

1i

ii∑

=

= (5.4-12)

Substituting Ai / Si for fi,

∑=

=m

1i

is

S

AQQ, (5.4-13)

S, Ss = N (5.4-14)

Then for monthly data,

SQ

x 12OF = = Overall Outage Frequency (5.4-15)

S

Qx 12OF S

S = = Supplier Attributable Outage Frequency

(5.4-16)

This general formula is used for products of varying system size suchas switching products. (See Example of Non-uniform System OutageReporting – End Office, subsection 5.4.7.a.) The two (2) specialcases below show simplified versions of the formula.



5-34

(ii) Outage Frequency Special Case 1

If products are of uniform system size and if only total system outages

are possible, then Ai = Si = S and the formula reduces to annualizedcount of outages normalized to number of systems in service. This iscomparable to the current RQMS calculation. For monthly data:

( )Nm

x 12N

SS

SiAi

x 12OF

m

1i

i

=

=∑

= (5.4-17)

For this case

mQ = (5.4-18)

Ss mQ = (5.4-19)S = N (5.4-20)

(iii) Outage Frequency Special Case 2

For products which express downtime in terms of normalization units,

by substituting Σ (Sn / N) for S to eliminate N, the formula reduces toannualized count of normalization units affected normalized to totalnumber of normalization units in the population. (See example ofTransport System – Digital Cross Connect, subsection 5.4.7.b.) Formonthly data:

( )( )

N

NS

11

Ai

x 12N

SS

SiAi

x 12OF

m

1iN

1n

nm

1i

i

∑∑∑ =

==

=

=

∑

∑

=

==N

1nn

m

1ii

S

Ax 12OF (5.4-21)

In this case,



5-35

∑=

=m

1iis AQQ, (5.4-22)

∑=

=N

1nns SSS, (5.4-23)

(Only supplier attributable outages are summed for QS.)

NOTE: The following analysis provides formulas to convert outagefrequency and downtime to other reliability measurements forreference. Equation (5.4-25) provides conversion from downtime(expressed in minutes) to system availability. Equation (5.4-24)provides conversion from outage frequency to mean time betweenfailures (MTBF). System availability / unavailability and MTBF arealternative expressions of system reliability found in somerequirement specifications.

Reliability Conversions

MTBF = Mean Time Between Failures (reference only)

MTBF = [(365)(24)] ÷ OF = mean hours to failure (5.4-24)

This calculation represents the mean number of hours betweensystem outages.

A = Availability (Reference only)

A = Probability that the system is operational when required = Up time ÷ Total timeA = [(365)(24)(60) – DT] ÷ [(365)(24)(60)]A = (525,600 – DT) ÷ 525,600 (5.4-25)

NOTE: Availability is often expressed as a percentage rather than asshown above.

U = Unavailability (reference only)

U = Probability that the system is not operational when required

= 1 – A (5.4-26)

For five (5) minutes per system per year of downtime, availability is0.9999905, or “five nines.” and unavailability is 1 – A = 9.5 x E-6. For50 minutes of downtime, A = 0.999905, or “four nines,” andunavailability is 1 – A = 9.5 x E-5.



5-36

Customer Aggregation

A customer can determine the overall system availability delivered toits end users by aggregating the system availability from his varioussuppliers as follows, where Ax, Ay, or Az is the availability of systemtype X, Y, or Z; where Px, Py, or Pz is the probability that atermination is served by system type X, Y, or Z (determined by ratio ofterminations or systems); and where Px + Py + Pz = 1.

ZZYYXXTSP PAPAPAA ++= (5.4-27)



(2) TL 9000 SO Data Table – The SO measurement shall be reportedwith data elements (or equivalent as defined by the MetricsAdministrator) for each month and each product category asfollows:

Table 5.4-4. System Outage (SO) Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersNormalization Factor: SAnnualization Factor: (See Glossary.)Metric Identifier: SOSO1 Numerator: QSO2 Numerator: PSO3 Numerator: QsSO4 Numerator: Ps

(3) RQMS Alternative Data Table – The RQMS alternative metricsshall be reported with data elements (or equivalent as defined bythe Metrics Administrator) from the applicable Table 5.4-5 or Table5.4-6, for each month as follows:



5-37

Table 5.4-5. RQMS Alternative System Outage (SOE) Endand/or Tandem Office, Wireless Products, orNGDLC Products Data Table

NOTE: If separation of host and remote does not apply, reportall items under the host category.

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: RQMSCustomer Base: Either (1) Total or (2) Forum MembersMetric Identifier: SOErDPMs Numerator: Annualized supplier attributable total outage

minutes – remote onlyhDPMs Numerator: Annualized supplier attributable total outage

minutes – host onlyrDPMc Numerator: Annualized service provider attributable total

outage minutes – remote onlyhDPMc Numerator: Annualized service provider attributable total

outage minutes – host onlyrOFMs Numerator: Annualized supplier attributable total outages

– remote onlyhOFMs Numerator: Annualized supplier attributable total outages

– host onlyrOFMc Numerator: Annualized service provider attributable total

outages – remote onlyhOFMc Numerator: Annualized service provider attributable total

outages – host onlyr Denominator: (Total systems deployed per RQMS - remotes)h Denominator: (Total systems deployed per RQMS - hosts)

Table 5.4-6. RQMS Alternative System Outage (SOG) GeneralSeries Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: RQMSCustomer Base: Either (1) Total or (2) Forum MembersNormalization Factor: : Total Systems Deployed per RQMSMetric Identifier: SOGDPM Numerator: Annualized overall outage minutesDPMs Numerator: Annualized supplier attributable outage minutesOFM: Annualized overall outagesOFMs Numerator: Annualized supplier attributable outages



5-38


Customers shall report all outage data and system population of their endusers to the supplier. If outage data is not supplied, then the supplier isnot responsible for reporting this measurement.



Overall System Downtime: CDOverall System Outage Frequency: CDSupplier Attributable System Downtime: CDSupplier Attributable System Outage Frequency: CD


Overall System Downtime: YESOverall System Outage Frequency: YESSupplier Attributable System Downtime: YESSupplier Attributable System Outage Frequency: YES


a. Example of Non-Uniform System Outage Reporting – End Office

Consider a population of four (4) central office systems comprising 1600terminations distributed as 100, 200, 300, and 1000 terminations persystem. The 100- and 1000-termination systems are host systems andeach experience one (1) 10-minute total outage during the month. The200- and 300-termination switches are remote systems. The 200-termination system incurred a 20-minute outage affecting 50 terminations.All of the outages are attributed to the supplier. The calculation is givenbelow for average switch size = 400 terminations. Table 5.4-7 isconstructed to calculate formulas 5.4-4 and 5.4-15. See example c for asimpler calculation.



5-39

Table 5.4-7. Example Non-Uniform System Calculation of SO

OutageDuration(minutes)

Pi

Termina-tions

Affected

EquivalentSwitchFactor

SA i

Numberof

Outages

WeightedFrequency

SA i

WeightedDowntime =

S

PA ii

10 100 100 / 400 1 .25 2.510 1000 1000 / 400 1 2.5 2520 50 50 / 400 1 .125 2.5

Total Month m,Q,P m =3 Q =2.875* P = 30*Annualized 36 34.5 360

*reported items (See Table 5.4-9.)

Dividing the annualized totals by the population of 4 systems(N=S=4 systems), the normalized numbers are:

Table 5.4-8. Example Normalized SO for Non-Uniform Systems

End Office Overall Supplier Attributable UnweightedDowntime (DT)(min / sys / yr)

90.0 90.0

Outage Freq (OF)(# / sys / yr)

8.625 8.625 9

This measurement translates directly to real performance delivered to theend user. A typical system (or a typical line on a typical system) willexperience 8.625 outages totaling 90.0 minutes in a year based onperformance in the current month. From equation 5.4-22, the availabilityfor this system is (525,600 – 90) / 525,600 = 0.999829. Thismeasurement also translates to performance delivered to customer. Thecustomer (service provider) will experience 9.0 outages per system peryear (not weighted by duration of outage or by size of system).



5-40

Table 5.4-9. Example System Outage (SO) Data Reportfor Example a., March 2001

Year and Month: 200103Reporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Forum MembersNormalization Factor: 4Annualization Factor: 12Metric Identifier: SOSO1 Numerator: 2.875SO2 Numerator: 30SO3 Numerator: 2.875SO4 Numerator: 30

Note that SO1 is the same as SO3 because all of the outages weresupplier attributable.

b. Example of Transport System - Digital Cross Connect

Consider a population of a given product consisting of various sized crossconnects interfacing with the network at various signal levels. From thequantity of each type of port card and its traffic capacity in terms of DS1equivalents, excluding units used for protection, the total averagecapacity of these systems during the month can be determined. Note thatthis is a “per DS1-equivalent port” number. For this example, assumethat in the fielded systems there are 200 OC-3 port cards (16,800 DS1equivalents), 400 DS3 / STS1 units (11,200 DS1 equivalents), and 1,000units with 4 DS1 ports each (4,000 DS1 equivalents) for a total capacity of32,000 DS1 equivalent ports. The outages for the month are all supplierattributable and are given in the following table. The table is constructedto calculate formulas 5.4-9 and 5.4-21.



5-41

Table 5.4-10. Example Transport System SO MeasurementCalculation

OutageLength

(minutes)

(Pi)

SignalType

SignalQuantity

DS1Equi-

valents

(Ai)

Numberof

Outages

WeightedFrequency

(DS1 x

number)Q=Si

WeightedTime

(DS1 x

minutes)(Ai Pi)

60 DS3 1 1 x 28 1 28 16803 DS1 8 8 x1 1 8 24

16 DS1 1 1 x 1 1 1 165 OC-3 1 1 x 84 1 84 420

Total Month m,Q,P m =4 Q =121* P = 2140*Annualized 48 1452 25680

*reported itemsDividing the annualized totals by the 32,000 DS1 capacity (S = 32,000 DS1Equivalents), the normalized downtime numbers are:

Table 5.4-11. Example Normalized SO for Transport Systems

Digital Cross-Connect Overall Supplier Attributable UnweightedDowntime(min / Eq DS1 / yr)

0.8025 0.8025

Outage Frequency(# / Eq DS1 / yr)

0.045 0.045 0.0015

This represents delivered annualized performance on a per-DS1 basis.Each equivalent DS1 will experience 0.045 outages totaling 0.8025minutes in a year. From Equation 5.4-25, availability for this system is(525,600 – 0.8025) / 525,600 = 0.9999984. Due to the mix of port cardsin this example, the four (4) outages experienced by the TSP in the monthrepresent 1.5 outages per 1,000 equivalent DS1s per year (unweighted).

c. Alternative/ Simplified Example of Non-uniform SystemReliability Reporting (End Office)

Using the data from example a and the method used in example b asimplified calculation for an end office can be performed using equations5.4-9 and 5.4-21. These formulas are derived from the general formula5.4-1 with no simplifying assumptions and are therefore equally valid.The perspective of termination performance is equivalent to systemperformance because each termination appears on exactly one (1)system, so the termination performance always equals the systemperformance.

DT = 12 [(10 min)(100 terms)+(10 min)(1000 terms)+(20 min)(50 terms)]÷ 1600 terms

DT = 12 (1000 + 10,000 + 1000) ÷ 1600 = 4 x 12,000 ÷ 1600 = 90.0 min



5-42

OF = 12 (100 terms + 1000 terms + 50 terms) ÷ 1600 termsOF = 12 (1150) ÷ 1600 = 8.625

Section 6 – Hardware Metrics


6-1

Section 6 - Hardware Metrics

6.1 Return Rates (RR) (H)


This profile defines two (2) return rate measurements:

a. First Year Return Rate (FYR) - return rate of units during the first yearafter initial shipment (months zero (0) through 12 of shipment); and

b. Long-Term Return Rate (LTR)- return rate of units after the first year afterinitial shipment (months 13 and later after shipment).

6.1.2 Purpose

The purpose of this metric is to:

a. Provide a measure of the quality of the product as initially received by thecustomer and during subsequent in-service operation;

b. Determine areas needing corrective action or most likely benefiting fromcontinuous improvement activity; and

c. Provide input data needed to calculate equipment life cycle costs. 6.1.3 Applicable Product Categories

a. This metric applies to product categories as shown in Appendix A.b. In general, these measurements apply to:

• Any system comprised of field replaceable units (FRUs)• A system which itself is an FRU• The individual FRUs themselves.

c. These measurements apply equally well to any FRU shipped either in asystem or separately.

d. These measurements are not intended for items shipped in bulk such as:• Cable• Optical fiber• Mechanical hardware.

NOTE: The First Year Return Rate measurement for items warehousedoutside of the supplier’s control, for an extended period before placement inservice, may not accurately reflect the actual return rate for product inservice. This may also be true of items sold through distributors.

NOTE: Long-Term Return Rates may become inaccurate for older productsas units are taken out of service.



6-2

NOTE: The return rate for low cost items after the expiration of any warrantyperiod is likely to be inaccurate if purchasing a new item is no moreexpensive than repairing the failed one. 6.1.4 Detailed Descriptions

a. Terminology – The Glossary includes definitions for the following

terms used for this metric:

• Annualization Factor (Afactor)• Basis Shipping Period

- First Year Return Rate (FYR)- Long-Term Return Rate (LTR)

• Field Replaceable Unit (FRU)• Return.

b. Counting Rules

(1) All returns except returns from laboratory systems and/or First OfficeApplication (FOA) systems are counted in these calculations. See“return” in the Glossary for the exact definition used here.

(2) Only returns from the basis shipping period corresponding to thespecific metric shall be counted.

(3) The supplier shall document, for the specific measurement, themethod of determining which of the returns are from thecorresponding basis shipping period. This may be determined by anyof the following methods:• Serialized shipment records of the returned unit;• A shipment or warranty start date code marked on the unit;• A shipment date associated with a customer order; and• A manufactured date associated with a lot number.

NOTE: The last method would require the determination of anaccounting for a standard time delay between the date of manufactureand shipment.



6-3

NOTE: Returns and shipments should only be reported once when submittingdata to the Forum database. When a unit may be used in more than one (1)product, it may not be practical or possible to identify with which product areturn or shipment is associated. In such cases, the supplier shouldapportion the returns and shipments appropriately among all products inwhich the unit is used.


(1) The measurements shall be calculated according to the formulasshown in Table 6.1-1. The formula for LTR is not normalized but isexpressed in percentage returns per year. The formula for FYR isnormalized with the normalization units given in the MetricsApplicability Table, Appendix A, Table A-2.

(2) The return rates are annualized.(3) Normalization of System Numbers – The First Year Return Rates

shall be normalized with units defined in the Metric ApplicabilityTable, Appendix A, Table A-2 when reported to the MetricsAdministrator.

• A general formula (normalized return rate) of this normalization ofa return rate metric would take the form of:

Normalized return rate = Returns x Afactor / Normalization Factor.

• Example Calculations (See 6.1.7.b.) illustrate computation ofnormalized return rates.

(4) First Year Return Rate (FYR) – The First Year Return Ratemeasures the rate of return of product during its first year aftershipment. It is based on the number of returns during thereporting period from the population of units shipped during thecurrent month through twelve months prior to the reporting period.This basis shipping period is assumed to represent the initialreturn rate of the product during installation, turn-up, testing andoperation during any infant morality period. Returns from unitsshipped during the current month are also included.

(5) Long-Term Return Rate (LTR) – The Long-Term Return Ratemeasures the rate of return of product after its first year from timeof shipment. It is based on the number of returns during thereporting period from the population of units shipped more thantwelve months prior to the reporting period. This rate representsthe mature return rate of the product.

(6) Normalized First Year Return Rate– The normalization of the FirstYear Return Rate allows this circuit pack return measure to becompared between like products with different architecture.

Notation

NU = Normalization Unit (NU) from Appendix AS = Normalization Factor; the total NU count shipped in the basis



6-4

shipping period not counting the current reporting month.Rf = Number of returns in the FYR basis shipping periodRt = Number of returns in the LTR basis shipping periodAfactor = The number of reporting periods in a year

(the factor applied to annualize the return rate)

Examples of this factor are:

Report Period Type Afactor

Calendar Month 124 Week Fiscal Month 135 Week Fiscal Month 10.46 Week Fiscal Month 8.7

Sf = Number of FRUs shipped during the FYR basis shipping periodSt = Number of FRUs shipped during the LTR basis shipping period

Table 6.1-1. Return Rates (FYR and LTR)Metric Identifiers and Formulas

Identifier Title Formula* NoteFYR Normalized First

Year Return Rate(12)(100)Rf / S Returns per NU

LTR Long-Term Return Rate (12)(100)Rt / St % per year _____________ * Afactor = 12 in these formulas because calendar month data is used



(2) Data shall be reported for FYR and LTR. Compared data industrystatistics are based only on the normalized FYR metric.

(3) TL 9000 Return Rate Data Table - The return rates shall be reportedwith data elements (or equivalent as defined by the MetricsAdministrator) for each month and each product category as follows:



6-5

Table 6.1-2. Return Rate (RR) Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersNormalization Factor: S (value for computing FYR)Annualization Factor: (See Glossary.)Metric Identifier: RRFYR Numerator: RfLTR Numerator: RtFYR Denominator: SfLTR Denominator: St

6.1.5 Source(s) of Data – The supplier should have available as a part of itsdata systems the information listed above needed to calculate thesemeasurements. This includes:a. FRU shipping records – These are required to determine which units

received for repair are “first year returns” or “long- term returns” anddetermine the respective populations.

b. FRU returns records – The supplier’s return records shall include theidentifier necessary to match returns with shipment records.

c. Third party returns records – Units returned to a third party repair agencyby the customer or repaired by the customer itself shall be included in thereturn counts when available. To have accurate measurements, it isnecessary for the customer to make it a contractual requirement of theirthird party repair agencies to supply this data to the original equipmentmanufacturers.


At present, only FYR is considered to have compared data. The LTR datashall also be reported to the Metrics Administrator for future use.

Compared Data (CD) or Research Data (RD):First Year Return Rate: CDLong-Term Return Rate: RD


a. Example Without Normalization

In a given reporting month, all returns are divided into two (2) groups,according to when they were shipped. For example, for the reporting monthof January 1999, returns are divided into the following groups (as illustrated inFigure 6.1-1):



6-6

First Year: Returns of units shipped in the period from January 1, 1998through January 31, 1999.

Long-Term: Returns of units shipped prior to January 1, 1998.

1997 1998 1999

OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN

Long-TermReturns First-Year Returns

Figure 6.1-1. Shipping Date Groups for Computing Return Rates

Table 6.1-3 below shows shipments for January 1998 through December1999, plus all shipments prior to January 1998. In addition, it shows returnsfor January 1999 through December 1999, broken out by month of shipmentas determined by shipping records. The highlighted first row of data inTable 6.1-3 shows the breakdown by month of shipment for the 265 returnsreceived during January 1999.

Table 6.1-3. Example Returns

ShipDate →ReturnMo.↓

Dec-97&

before

Jan-98 Feb-98 Mar-98 Apr-98 May-98 Jun-98 Jul-98 Aug-98 Sep-98 Oct-98 Nov-98 Dec-98

Jan-99 58 11 7 14 10 6 6 14 16 20 39 36 23Feb-99 77 11 9 15 9 11 13 12 6 18 24 26 30Mar-99 58 7 8 12 14 12 12 14 14 15 18 24 20Apr-99 75 9 11 8 10 11 16 12 17 18 7 23 22

May-99 59 9 10 10 9 16 11 12 14 16 15 12 25Jun-99 70 9 8 15 8 9 7 14 14 6 15 13 15Jul-99 71 9 5 4 10 13 9 14 11 12 17 6 15

Aug-99 46 13 5 12 6 16 12 15 14 19 16 13 15Sep-99 69 6 3 7 12 8 14 11 12 12 13 9 16Oct-99 66 3 14 9 11 13 13 10 12 6 6 9 12Nov-99 53 6 7 5 11 10 7 11 8 16 19 20 16Dec-99 47 3 9 12 10 3 13 8 13 11 9 12 11

Ship-ments:

45996 6263 6436 7244 7275 7396 8263 8833 8954 9368 9818 9787 10528



6-7

Table 6.1-3 (continued). Example Returns

ShipDate →ReturnMo.↓

Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99 Aug-99 Sep-99 Oct-99 Nov-99 Dec-99 TotalReturns:

Jan-99 5 265Feb-99 33 1 295Mar-99 23 31 5 287Apr-99 25 23 27 2 316

May-99 22 27 26 33 4 330Jun-99 30 24 20 28 27 1 333Jul-99 18 24 29 26 27 31 1 352

Aug-99 15 11 38 26 28 26 35 5 386Sep-99 14 13 17 16 31 28 25 33 4 373Oct-99 19 22 14 19 18 26 32 25 30 4 393Nov-99 14 11 19 19 13 22 35 28 23 34 3 410Dec-99 19 16 12 12 24 15 16 21 22 22 36 4 390Ship-ments

10644 11321 11332 11674 12151 12460 13494 13670 13933 13725 14467 14905

The annualized return rates for the month of January 1999, are calculated as:

First YearReturn Rate = (Returns of units shipped Jan-98 through Jan-99) x 12 x100 Total Shipments for Jan-98 through Dec-98

= (11+7+14+10+6+6+14+16+20+39+36+23+5) x12 x100 _

(6263+6436+7244+7275+7396+8263+8833+8954+9368+9818+9787+10528)

= 2.48%

During January 1999, the number of returned units was calculated as follows:11 returns of units shipped in January 1998,7 returns of units shipped in February 1998,14 returns of units shipped in March 1998, and so on, including5 returns of units shipped in the month of January 1999,

for a total number of first year returns of 207.

The corresponding field population is determined by the sum of the shipmentquantities shown in the bottom row of Table 6.1-3 for the months of January1998 through December 1998. Note that the returns of units shipped inJanuary are included in order to count all returns during the month, and to bealerted to any developing problems. However, shipments during January areexcluded because most units will not have been placed into operation.

Long Term Return Rate calculations:

Long-TermReturn Rate = (Returns from shipments prior to Jan-98) x 12 x 100

Total Shipments prior to Jan-98



6-8

= 58 x 12 x10045996

= 1.51%

Calculating the return rates for all months in 1999 gives:

FirstYear

ReturnRate

Long-Term

ReturnRate

JAN-99 2.48% 1.51%FEB-99 2.38% 2.02%MAR-99 2.35% 1.49%APR-99 2.25% 1.87%MAY-99 2.37% 1.59%JUN-99 2.09% 1.77%JUL-99 2.18% 1.63%AUG-99 2.38% 1.54%SEP-99 2.03% 1.60%OCT-99 2.01% 1.62%NOV-99 2.13% 1.46%DEC-99 1.93% 1.33%

b. Examples With Normalization

(1) Normalized First Year Return Rate - Example 1:

A supplier makes an HDSL transmission system consisting of thefollowing products:

i. HDSL Central Office Transceiver Unit (HTU-C) – One (1) HTU-Cis required per HDSL line deployed.

ii. HDSL Remote Transceiver Unit (HTU-R) – One (1) HTU-R isrequired per HDSL line deployed.

iii. HDSL Range Extender (HRE) – Zero to two (2) HREs may beused per HDSL line deployed.

iv. HDSL Fuse / Alarm Controller (HFAC) – One (1) HFAC is requiredper HDSL shelf, which may be used to deploy up to 13 HDSLlines.

v. HDSL Central Office Terminal Controller (HCOT-CTL) – One (1)HCOT-CTL can control up to 100 shelves.

vi. HDSL E220 Shelf – One (1) shelf can accommodate up to 13HDSL transceiver units.

Only products i through v are field replaceable units.



6-9

To calculate the normalized FYR, returns are aggregated for theentire HDSL product category and the normalizing factor is applied tothe category as a whole:

i. HDSL Central Office Transceiver Unit (HTU-C)Returns in the reporting period (one [1] month): 50Shipments in the basis period (one [1] year): 100,000

ii. HDSL Remote Transceiver Unit (HTU-R)Returns in the reporting period: 40Shipments in the basis period: 100,000

iii. HDSL Range ExtenderReturns in the reporting period: 5Shipments in the basis period: 50,000

iv. HDSL Fuse/Alarm ControllerReturns in the reporting period: 3Shipments in the basis period: 10,000

v. HDSL Central Office Terminal ControllerReturns in the reporting period: 0Shipments in the basis period: 500

The normalizing factor for xDSL products is the number of DSL linesdeployed. Since one (1) HTU-C and one (1) HTU-R are required todeploy a single HDSL line, the total number of lines deployed in thebasis period is 100,000.

The normalized First Year Return Rate would be:

[(50 + 40 + 5 + 3 + 0) x 12] / 100,000 = 0.012 returns / yr. / DSL line

(2) Normalized First Year Return Rate - Example 2:

A supplier makes a local switch consisting of the following products:

i. POTS line card – Each POTS line card has 16 POTS lines.ii. Trunk line card – Each trunk line card has four (4) trunk lines.iii. ISDN line card – Each ISDN line card has eight (8) basic rate

ISDN lines, each of which has two (2) terminations.iv. Miscellaneous circuit packs – Besides the previous three (3)

circuit packs there are 30 other pack codes. They do not supplytermination service, but are needed to support the office.

During the basis period for the FYR, this supplier installed one (1)switch with the line cards and other circuit packs listed below. Tocalculate the normalized FYR, returns are aggregated for the entireswitch and the normalizing factor is applied to the category as awhole.



6-10

i. POTS Line CardReturns in the reporting period (one [1] month): 10Shipments in the basis period (one [1] year): 1,000

ii. Trunk Line CardReturns in the reporting period: 5Shipments in the basis period: 500

iii. ISDN Line CardReturns in the reporting period: 2Shipments in the basis period: 250

iv. Miscellaneous Circuit PacksReturns in the reporting period: 2

The normalizing factor for Switching and Routing System Elements is10,000 terminations. The total of all terminations for this switch is:

(1,000 x 16) + (500 x 4) + (250 x 8 x 2) = 22,000

The normalized First Year Return Rate per 10,000 terminations for theswitch circuit pack shipments is:

(10 + 5 + 2 + 2) x 12 x 10,000/ 22,000 = 104 returns / yr / 10,000terminations

FYR Denominator = 1,000 + 500 + 250 + 2 = 1,752

(3) Data Table for Example 2

The following table shows how data from the above example would bereported to the Metrics Administrator. For completeness, the reportincludes examples of LTR data that were not discussed in theexample.

Table 6.1-4. Return Rate Example Data Table

Year and Month: 199901Reporting ID: Provided by Forum AdministratorProduct Category Code: 1.1.1Metric Methodology: TL 9000Customer Base: TotalNormalization Factor: 22,000Annualization Factor: 12Metric Identifier: RRFYR Numerator: 19LTR Numerator: 30FYR Denominator: 1752LTR Denominator: 4500

Section 7 – Software Metrics


7-1

Section 7 - Software Metrics

7.1 Software Update Quality (SWU) (S)


Software Update Quality (SWU) tracks the percentage of software updatesthat are defective. This measurement is adapted from RQMS. [1]

NOTE: Software updates differ from patching in the manner in which softwarechanges are made to a system. A software update is a set of changes to arelease and is commonly referred to as a “dot” or “point” release. Softwareupdating replaces existing product code with a new load as opposed toentering patches into a load. A software update completely replaces theexisting software in the product. This measurement is designed to helpcustomers understand their likelihood for problems when implementingsoftware updates. Some products do not employ patching to implementsoftware fixes.

7.1.2 Purpose The purpose of this metric is to evaluate the level of defective softwareupdates in order to minimize associated customer risks.

7.1.3 Applicable Product Categories This metric applies to product categories as shown in Appendix A.


a. Terminology – The Glossary includes definitions for the followingterms used for software update metrics:

• General Availability (GA)• Release• Software Update• Official Software Update• Defective Software Update.

b. Software Update Counting Rules – The following rules shall applyto these measurements:

(1) A software update is counted upon General Availability.



7-2

(2) Software updates shall be considered delivered when they aredelivered to the destination(s) designated by the customer.For example, software updates are considered delivered when on-site, or ready for system installation, or available for downloadingby the customer to the site.

(3) Software updates shall be considered part of a release if includedin the release by the supplier prior to the shipment of that releasefor the first customer General Availability. Software updatesincluded with a release that require additional effort to implementshall be counted as software updates.

(4) The data shall include the three (3) most dominant releases foreach product being reported. If fewer than three (3) releasesexist, the data shall include all existing releases.

c. Defective Software Update Counting Rules – A defective softwareupdate is one that cannot be installed or is withdrawn because ofpotential or actual problems. The following rules shall apply tocounting defective software updates:

(1) Only supplier attributable aborts shall be counted as defectivesoftware updates.

(2) A defective software update shall be counted against the monthduring which the software update was found defective and therelease for which it was intended to update.

(3) The data shall include the three (3) most dominant releases foreach product being reported. If fewer than three (3) releasesexist, the data shall include all existing releases.

d. Calculations and Formulas

(1) The measurement (See SWU0, SWU1 and SWU2 in Table 7.1-1.)shall be calculated monthly as the cumulative percentage ofdefective software updates by release since General Availability.

(2) The percentage for each month shall be calculated by dividing thecumulative number of defective software updates by thecumulative number of software updates deployed for the release.

(3) The supplier shall provide, by release, the total monthly number ofsoftware updates delivered and the number of defective softwareupdates identified.

(4) When reporting RQMS alternative metrics, suppliers shall refer toDSU0, DSU1, DSU2, Sur0, Sur1, and Sur2 in Table 7.1-2 todetermine reporting conventions.

(5) The reported data and each of the computed measurements aretotaled/aggregated to one (1) value per registered entity perproduct category per month.



7-3

Notation

Release N: The most recent of the three (3) dominant releases reported.Release N-1 is the previous release reported. Release N-2 isthe release previous to N-1 that is reported.

Du2 = Cumulative number of defective software updates forrelease N-2

Du1 = Cumulative number of defective software updates forrelease N-1

Du0 = Cumulative number of defective software updates for release NMd2 = Monthly number of defective software updates for

release N-2Md1 = Monthly number of defective software updates for

release N-1Md0 = Monthly number of defective software updates for release NUs2 = Cumulative number of software updates for release N-2Us1 = Cumulative number of software updates for release N-1Us0 = Cumulative number of software updates for release NMu2 = Monthly number of software updates for release N-2Mu1 = Monthly number of software updates for release N-1Mu0 = Monthly number of software updates for release N

Table 7.1-1. Software Update Quality (SWU0, SWU1, and SWU2)Metric Identifiers and Formulas

Identifier Title Formula NoteSWU2 Defective Software Updates

- Release N-2100(Du2 / Us2) Cumulative

% defectiveSWU1 Defective Software Updates

- Release N-1100(Du1 / Us1) Cumulative

% defectiveSWU0 Defective Software Updates

- Release N100(Du0 / Us0) Cumulative

% defective

Table 7.1-2. Software Update QualityRQMS Alternative Metrics (DSU)

Identifier TitleDSU2 Cumulative % Defective Software Updates – Release N-2DSU1 Cumulative % Defective Software Updates – Release N-1DSU0 Cumulative % Defective Software Updates – Release NSUr2 Cumulative Number of Software Updates Deployed – Release N-2SUr1 Cumulative Number of Software Updates Deployed – Release N-1SUr0 Cumulative Number of Software Updates Deployed – Release N



7-4

e. Reported Data and Format


(2) TL 9000 Data SWU Table – The SWU measurements shall bereported with data elements (or equivalent as defined by theMetrics Administrator) for each month and each product categorycombination as follows:

Table 7.1-3. SWU Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: (From Appendix A)Metric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersMetric Identifier: SWUSWU2 Numerator: Du2SWU1 Numerator: Du1SWU0 Numerator: Du0SWU2 Denominator: Us2SWU1 Denominator: Us1SWU0 Denominator: Us0SWU2 Monthly Numerator: Md2SWU1 Monthly Numerator: Md1SWU0 Monthly Numerator: Md0SWU2 Monthly Denominator: Mu2SWU1Monthly Denominator: Mu1SWU0 Monthly Denominator: Mu0

(3) RQMS Alternative Data Table – The RQMS alternative metricsshall be reported with data elements (or equivalent as defined bythe Metrics Administrator) for each month and each productcategory combination as follows:



7-5

Table 7.1-4. SWU Data Table RQMS Alternative Data (DSU)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: RQMSCustomer Base: Either (1) Total” or (2) Forum

MembersMetric Identifier: DSUDSU2 Numerator: Cumulative number of defective software

updates for release N-2DSU1 Numerator: Cumulative number of defective software

updates for release N-1DSU0 Numerator: Cumulative number of defective software

updates for release NDSU2 Denominator: Cumulative number of software updates

for release N-2 (SUr2)DSU1 Denominator: Cumulative number of software updates

for release N-1 (SUr1)DSU0 Denominator: Cumulative number of software updates

for release N (SUr0)


Customers shall provide feedback to the supplier on the results(successful or unsuccessful) of any customer installed software update.In addition, customers shall confirm to the supplier that a supplierinstalled software update was still in operation 24 hours after theinstallation. Suppliers shall collect all data necessary to report thesemeasurements to the Metrics Administrator.


a. Compared data (CD) or research data (RD):

Defective Software Updates CD


Defective Software Updates: YES


A supplier of a software driven network element distributes softwareupdates in the interim between releases. Table 7.1-5 shows the history of



7-6

updates for a 6-month period. Table 7.1-6 shows the data report for June2000.

Table 7.1-5. Example SWU Source Data

(Product Category 2.1)Month: Jan Feb Mar Apr May Jun

Release ID # Updates 2000 2000 2000 2000 2000 2000C25 (N-2) Cumulative (Us2) 11 15 18 25 28 29C25 (N-2) Cumul. Defectives (Du2) 1 1 1 2 4 4C25 (N-2) Current (Mu2) 2 4 3 7 3 1C25 (N-2) Current Defectives (Md2) 0 0 0 1 2 0SWU2 Cumulative

% Defective9.1% 6.7% 5.6% 8.0% 14.3% 13.8%

D2 (N-1) Cumulative (Us1) 10 15 30 50D2 (N-1) Cumul. Defectives (Du1) 2 3 4 6D2 (N-1) Current (Mu1) 10 5 15 20D2 (N-1) Current Defectives (Md1) 2 1 1 2SWU1 Cumulative

% Defective20% 20% 13.3% 12%

D4 (N) Cumulative (Us0) 25 55D4 (N) Cumul. Defectives (Du0) 2 4D4 (N) Current (Mu0) 25 30D4 (N) Current Defectives (Md0) 2 2SWU0 Cumulative

% Defective8% 7.3%

Table 7.1-6. Example SWU Data Table Report for June, 2000

Year and Month: 200006Reporting ID: Provided by Forum AdministratorProduct Category Code: 2.1Metric Methodology: TL 9000Customer Base: Forum MembersMetric Identifier: SWUSWU2 Numerator: 4SWU1 Numerator: 6SWU0 Numerator: 4SWU2 Denominator: 29SWU1 Denominator: 50SWU0 Denominator: 55SWU2 Monthly Numerator: 0SWU1 Monthly Numerator: 2SWU0 Monthly Numerator: 2SWU2 Monthly Denominator: 1SWU1 Monthly Denominator: 20SWU0 Monthly Denominator: 30



7-7

7.2 Release Application Aborts (RAA) (S)


The Release Application Aborts metric (RAA) is the percentage of releaseapplications with aborts. This measurement is derived from RQMS. [1]

7.2.2 Purpose The purpose of this metric is to evaluate the percentage of releaseapplications with aborts in order to minimize the service provider risk ofaborts when applying a software release.

7.2.3 Applicable Product Categories This metric applies to product categories as shown in Appendix A.


a. Terminology – The Glossary contains definitions for the following termsused for the RAA metric:

• Cutover• General Availability (GA)• Release• Release Application• Release Application Abort• Dominant Release• Software Update.

b. Counting Rules

(1) A release is counted upon General Availability.(2) Only supplier attributable aborts shall be counted.(3) The application/installation interval (also the release application

window) shall start 24 hours prior to scheduled cutover and shall endseven (7) days after cutover.

(4) The percentage for each month shall be calculated using thecumulative number of release application attempts at the end of themonth for that release.

(5) The data shall include the three (3) most dominant releases for eachproduct being reported. If fewer than three (3) releases exist, thedata shall include all existing releases.



7-8


(1) The measurement shall be calculated monthly for each release asthe percentage of the cumulative number of application attempts forwhich a new release has been applied or committed to be appliedand for which a release application abort has occurred.

(2) For each of the three (3) most dominant releases, the supplier shallprovide the number of release application attempts for the month andthe number of systems that encountered any abort during the releaseapplication window. The supplier shall report this data monthly.

(3) When reporting RQMS alternative metrics, suppliers shall refer toRAQ0, RAQ1, RAQ2, Rar0, Rar1, Rar2, in Table 7.2-2 to determinereporting conventions.

(4) The reported data and each of the computed measurements aretotaled/aggregated to one (1) value per registered entity per productcategory per month.

Notation

Release N: The most recent dominant release reported. Release N-1 is theprevious release reported. Release N-2 is the release previousto N-1 that is reported.

Ra2 = Cumulative number of release application attempts for release N-2Ar2 = Cumulative number of release application aborts for release N-2Ra1 = Cumulative number of release application attempts for release N-1Ar1 = Cumulative number of release application aborts for release N-1Ra0 = Cumulative number of release application attempts for release NAr0 = Cumulative number of release application aborts for release NMa2 = Monthly number of release application attempts for release N-2Mr2 = Monthly number of systems that encountered at least one (1) abort

during the application window for release N-2Ma1 = Monthly number of release application attempts for release N-1Mr1 = Monthly number of systems that encountered at least one (1) abort

during the application window for release N-1Ma0 = Monthly number of release application attempts for release NMr0 = Monthly number of systems that encountered at least one (1) abort

during the application window for release N



7-9

Table 7.2-1. Release Application Aborts (RAA)Metric Identifiers and Formulas

Identifier Title Formula NoteRAA2 Release Application Aborts –

Release N-2100(Ar2 / Ra2) % of systems

with abortsRAA1 Release Application Aborts –

Release N-1100(Ar1 / Ra1) (same as above)

RAA0 Release Application Aborts –Release N

100(Ar0 / Ra0) (same as above)

Table 7.2-2. Release Application AbortsRQMS Alternative Metrics (RAQ)

Identifier TitleRAQ2 Cumulative % of Systems Experiencing an Abort during Release

Application – Release N-2RAQ1 Cumulative % of Systems Experiencing an Abort during Release

Application – Release N-1RAQ0 Cumulative % of Systems Experiencing an Abort during Release

Application – Release NRar2 Cumulative Number of Release Application Attempts –

Release N-2Rar1 Cumulative Number of Release Application Attempts –

Release N-1Rar0 Cumulative Number of Release Application Attempts –

Release N



(2) TL 9000 RAA Data Table – The RAA measurements shall bereported with data elements (or equivalent as defined by the MetricsAdministrator) for each month and each product categorycombination as in Table 7.2-3.



7-10

Table 7.2-3. RAA Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersMetric Identifier: RAARAA0 Numerator: Ar0RAA1 Numerator: Ar1RAA2 Numerator: Ar2RAA0 Denominator: Ra0RAA1 Denominator: Ra1RAA2 Denominator: Ra2RAA0 Monthly Numerator: Mr0RAA1 Monthly Numerator: Mr1RAA2 Monthly Numerator: Mr2RAA0 Monthly Denominator: Ma0RAA1 Monthly Denominator: Ma1RAA2 Monthly Denominator: Ma2

(3) RQMS Alternative Data Table – The RQMS alternative metricsshall be reported with data elements (or equivalent as defined bythe Metrics Administrator) for each month and each productcategory combination as defined in Table 7.2-4.

Table 7.2-4. RAA Data Table RQMS Alternative Data (RAQ)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: RQMSCustomer Base: Either (1) Total or (2) Forum MembersMetric Identifier: RAQRAQ2 Numerator: Cumulative number of systems experiencing an

abort during release application for release N-2RAQ1 Numerator: Cumulative number of systems experiencing an

abort during release application for release N-1RAQ0 Numerator: Cumulative number of systems experiencing an

abort during release application for release NRAQ2 Denominator: Cumulative number of release application

attempts for release N-2 (Rar2)RAQ1 Denominator: Cumulative number of software updates for

release N-1 (Rar1)RAQ0 Denominator: Cumulative number of software updates for

release N (Rar0)



7-11


a. Suppliers shall capture data relative to numbers of software upgrades andaborts.

b. Customers shall provide the suppliers (via the mutually agreed uponprocedure) with timely feedback related to any aborts which wereencountered. If customers perform the upgrades, they must provide thesupplier with the planned and actual dates for each software upgrade andidentify the applications that aborted due to supplier attributable causes.



Release Application Aborts: CD

b. RQMS alternative reporting:

Release Application Aborts: YES


a. For example, a supplier is upgrading three (3) active releases (from priorreleases to release N, N-1 and N-2). Software upgrade counts and thoseencountering release application aborts are shown in Table 7.2-5.



7-12

Table 7.2-5. Example RAA Source Data

Month 1 2 3 4 5 6 7 8 9 10 11 12 13Dec2000

Jan2001

Feb2001

Mar2001

Apr2001

May2001

Jun2001

Jul2001

Aug2001

Sep2001

Oct2001

Nov2001

Dec2001

Number of Upgrades In MonthMa2 Release N-2 1 2 5 10 20 40 45 46 45 43 45 30 24Ma1 Release N-1 1 3 5 12 20 39 46 51 52 45 48 33 29Ma0 Release N 1 4 6 14 22 39 45 52 54 50 47 36 30

Number of Upgrades that Encountered ReleaseApplication Aborts in MonthMr2 Release N-2 0 0 0 1 0 1 1 1 1 1 1 1 0Mr1 Release N-1 0 0 0 1 0 1 1 1 1 1 0 0 1Mr0 Release N 0 0 0 0 1 0 1 0 1 0 1 0 0

Cumulative Release Application AttemptsRa2 Release N-2 1 3 8 18 38 78 123 169 214 257 301 331 355Ra1 Release N-1 1 4 9 21 41 80 126 177 229 274 322 355 384Ra0 Release N 1 5 11 25 47 86 131 183 237 287 334 370 400

Cumulative Release Application AbortsAr2 Release N-2 0 0 0 1 1 2 3 4 5 6 7 8 8Ar1 Release N-1 0 0 0 1 1 2 3 4 5 6 6 6 7Ar0 Release N 0 0 0 0 1 1 2 2 3 3 4 4 4

Release Application Aborts Measurement (Cumulative %)RAA2 Release N-2 0.00 0.00 0.00 5.56 2.63 2.56 2.44 2.37 2.34 2.33 2.32 2.41 2.25RAA1 Release N-1 0.00 0.00 0.00 4.76 2.44 2.50 2.38 2.26 2.18 2.19 1.86 1.69 1.82RAA0 Release N 0.00 0.00 0.00 0.00 2.13 1.16 1.53 1.09 1.27 1.05 1.20 1.08 1.00



7-13

b. For the month of December 2001, the TL 9000 data reported for theabove example is shown in Table 7.2-6.

Table 7.2-6. Example RAA TL 9000 Data Report

Year and Month: 200112Reporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Forum MembersMetric Identifier: RAARAA2 Numerator: 8RAA1 Numerator: 7RAA0 Numerator: 4RAA2 Denominator: 355RAA1 Denominator: 384RAA0 Denominator: 400RAA2 Monthly Numerator: 0RAA1 Monthly Numerator: 1RAA0 Monthly Numerator: 0RAA2 Monthly Denominator: 24RAA1 Monthly Denominator: 29RAA0 Monthly Denominator: 30



7-14

7.3 Corrective Patch Quality (CPQ) (S) and Feature Patch Quality (FPQ) (S)

7.3.1 Title and General Description Corrective Patch Quality is the percentage of official corrective patches thatare determined to be defective. Feature Patch Quality is the percentage ofofficial feature patches that are determined to be defective. Thesemeasurements are adapted from RQMS. [1]

7.3.2 Purpose The purpose of this metric is to evaluate the percentage of defective officialpatches in order to minimize service provider risk of failure. 7.3.3 Applicable Product Types This metric applies to product categories as shown in Appendix A. 7.3.4 Detailed Description

a. Terminology – The Glossary includes definitions for the following termsused for these metrics:

• Corrective Patch• Defective Corrective Patch• Defective Feature Patch• Dominant Release• Feature Patch• General Availability (GA)• Official Patch• Patch.

b. Counting Rules

(1) Non-identical patches packaged together in one (1) administrativeunit shall be counted individually even if the package can be installedduring one (1) craftsperson task.

(2) Identical patches distributed to multiple processors (or units) in thesame system shall be counted only once provided they can beinstalled during one (1) craftsperson task.

(3) If several separate patches are provided to affect a single change(such as covering different parts of the code) that are separatelyidentifiable to the customer, they shall each be counted separately.



7-15

(4) A patch is counted upon General Availability. For example, patchesare counted when either (1) on-site and ready for system installationor (2) available for downloading by the customer to the site.

(5) Patches shall be considered part of a release if included in therelease by the supplier prior to the shipment of that release for thefirst Service Provider General Availability. Patches included with arelease that require additional effort to implement shall be counted aspatches.

(6) A defective patch shall be counted against the month during whichthe patch was found defective.

(7) The data shall include the three (3) most dominant releases for eachproduct being reported. If fewer than three (3) releases exist, thedata shall include all existing releases.


(1) These measurements (See CPQ and FPQ in Table 7.3-1.) shall becalculated monthly by release. Each measurement is calculated bymultiplying 100 by the number of defective patches identified duringthe month and dividing by the number of patches that becameavailable for general release during the month.

(2) For CPQ, the supplier shall provide, by release, the total monthlynumber of official corrective patches delivered and the number ofofficial corrective patches identified as defective.

(3) For FPQ, the supplier shall provide, by release, the total monthlynumber of official feature patches delivered and the number of officialfeature patches identified as defective.

(4) When reporting RQMS alternative metrics, suppliers shall refer toTable 7.3-2 Patch Quality – RQMS Alternative Metrics (DCP andDFP) to determine reporting conventions.

Notation

Release N: The most recent dominant release reported: Release N-1 isthe previous release reported. Release N-2 is the releaseprevious to N-1 that is reported.

DPc2 = Number of defective corrective patches for the month forrelease N-2

DPc1 = Number of defective corrective patches for the month forrelease N-1

DPc0 = Number of defective corrective patches for the month forrelease N

Pc2 = Total number of corrective patches that became availablefor general release during the month for release N-2

Pc1 = Total number of corrective patches that became availablefor general release during the month for release N-1

Pc0 = Total number of corrective patches that became available



7-16

for general release during the month for release NDPf2 = Number of defective feature patches for the month for

release N-2DPf1 = Number of defective feature patches for the month for

release N-1DPf0 = Number of defective feature patches for the month for

release NPf2 = Total number of feature patches that became available

for general release during the month for release N-2Pf1 = Total number of feature patches that became available

for general release during the month for release N-1Pf0 = Total number of feature patches that became available

for general release during the month for release N

Table 7.3-1. Patch Quality (CPQ and FPQ)

Identifier Title Formula NoteCPQ0 Defective Corrective Patches

– Release N100(DPc0 / Pc0) %defective

per monthCPQ1 Defective Corrective Patches

– Release N-1100(DPc1 / Pc1) (same as

above)CPQ2 Defective Corrective Patches

– Release N-2100(DPc2 / Pc2) (same as

above)FPQ0 Defective Feature Patches

– Release N100(DPf0 / Pf0) (same as

above)FPQ1 Defective Feature Patches

– Release N-1

100(DPf1 / Pf1) (same asabove)

FPQ2 Defective Feature Patches – Release N-2

100(DPf2 / Pf2) (same asabove)



7-17

Table 7.3-2. Patch Quality RQMS Alternative Metrics (DCPand DFP)

Identifier TitleDCP0 Monthly Number of Defective Corrective Patches Identified –

Release NDCP1 Monthly Number of Defective Corrective Patches Identified –

Release N-1DCP2 Monthly Number of Defective Corrective Patches Identified –

Release N-2DFP0 Monthly Number of Defective Feature Patches Identified –

Release NDFP1 Monthly Number of Defective Feature Patches Identified –

Release N-1DFP2 Monthly Number of Defective Feature Patches Identified –

Release N-2CPr0 Monthly Number of Corrective Patches Delivered – Release NCPr1 Monthly Number of Corrective Patches Delivered – Release N-1CPr2 Monthly Number of Corrective Patches Delivered – Release N-2FPr0 Monthly Number of Feature Patches Delivered – Release NFPr1 Monthly Number of Feature Patches Delivered – Release N-1FPr2 Monthly Number of Feature Patches Delivered – Release N-2



(2) TL 9000 Data CPQ or FPQ Table – The CPQ and FPQmeasurements shall be reported with data elements (or equivalentas defined by the Metrics Administrator) for each month and eachproduct category combination as follows:

Table 7.3-3. CPQ or FPQ Data Table

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Either (1) Total or (2) Forum MembersMetric Identifier: CPQ or FPQCPQ0 or FPQ0 Numerator: DPc0 (for CQP) or DPf0 (for FPQ)CPQ1 or FPQ1 Numerator: DPc1 (for CQP) or DPf1 (for FPQ)CPQ2 or FPQ2 Numerator: DPc2 (for CQP) or DPf2 (for FPQ)CPQ0 or FPQ0 Denominator: Pc0 (for CQP) or Pf0 (for FPQ)CPQ1 or FPQ1 Denominator: Pc1 (for CQP) or Pf1 (for FPQ)CPQ2 or FPQ2 Denominator: Pc2 (for CQP) or Pf2 (for FPQ)



7-18

(3) RQMS Alternative Data Table – The RQMS alternative metrics forDPQ and FPQ shall be reported with data elements (or equivalentas defined by the Metrics Administrator) for each month and eachproduct category combination as follows:

Table 7.3-4. CPQ or FPQ Data TableRQMS Alternative Data (DCP or DFP)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: RQMSCustomer Base: Either (1) Total” or (2) Forum MembersMetric Identifier: DCP or DFPDCP0 or DFP0 Numerator: Number of defective (corrective / feature)

patches for release NDCP1 or DFP1 Numerator: Number of defective (corrective / feature)

patches for release N-1DCP2 or DFP2 Numerator: Number of defective (corrective / feature)

patches for release N-2DCP0 or DFP0 Denominator: Number of (corrective / feature) patches

delivered for release N (CPr0 or FPr0)DCP1 or DFP1 Denominator: Number of (corrective / feature) patches

delivered for release N-1 (CPr1 or FPr1)DCP2 or DFP2 Denominator: Number of (corrective / feature) patches

delivered for release N-2 (CPr2 or FPr2)


Suppliers shall collect all data necessary to support this metric.


a. Compared Data (CD) or Research Data (RD)

Defective Corrective Patches: CDDefective Feature Patches: RD


Defective Corrective Patches: YESDefective Feature Patches: YES



7-19


a. The following example illustrates calculation of the corrective patchquality measurement. Calculation of the feature patch qualitymeasurement is analogous.

Example of Corrective Patch Quality Metric:

A supplier has three (3) active releases (N, N-1, and N-2). Corrective patchdistribution and bad corrective patch counts have been as per the tablebelow, Table 7.3-5.

Table 7.3-5. Example CPQ Source Data

Month: 1 2 3 4 5 6 7 8 9 10 11 12 13Jan

2000Feb2000

Mar2000

Apr2000

May2000

Jun2000

Jul2000

Aug2000

Sep2000

Oct2000

Nov2000

Dec2000

Jan2001

Number of Corrective PatchesIssued In MonthPc2 Release N-2 60 55 50 47 42 35 35 31 32 30 29 27 25Pc1 Release N-1 55 55 50 40 36 32 34 36 33 32 26 24 24Pc0 Release N 52 53 48 35 34 34 32 30 28 30 25 24 22

Number of Defective Corrective PatchesIdentified in MonthDPc2 Release N-2 1 0 0 2 0 0 0 1 0 1 1 0 0DPc1 Release N-1 1 0 0 1 0 0 1 0 0 0 1 0 1DPc0 Release N 0 1 0 0 0 1 0 0 0 0 0 1 0

Defective Corrective Patch Measurement -% DefectiveCPQ2 Release N-2 1.67 0 0 4.26 0 0 0 3.23 0 3.33 .345 0 0CPQ1 Release N-1 1.82 0 0 2.50 0 0 2.94 0 0 0 3.85 0 4.17CPQ0 Release N 0 1.89 0 0 0 2.94 0 0 0 0 0 4.17 0

b. For the month of November 2000, the TL 9000 CPQ datareported is shown in Table 7.3-6.



7-20

Table 7.3-6. Example CQP Data Report

Year and Month: 200011Reporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix AMetric Methodology: TL 9000Customer Base: Forum MembersMetric Identifier: CPQCPQ0 Numerator: 0CPQ1 Numerator: 1CPQ2 Numerator: 1CPQ0 Denominator: 25CPQ1 Denominator: 26CPQ2 Denominator: 29

Section 8 – Services Metrics


8-1

Section 8 - Services Metrics

8.1 Service Quality (SW) (V)

8.1.1 Description and Title

Service Quality is a measure of conformance of a service to specified criteria.

8.1.2 Purpose

The purpose of this metric is to provide quality measurement information forestablishing the evaluation and continuous improvement of the service.


This metric applies to service categories as shown in Appendix A.


a. Terminology – The Glossary includes definitions for the following termsused for these metrics:

• Installation Audit• Maintenance Callback• Maintenance Visit (repair, preventive)• Non-Conforming Installation Audit• Resolutions (call center)• Rework• Service Volume• Total Calls (call center).

b. Counting Rules

(1) Failure of any unit during the repair warranty period shall be countedas a defective repair unit.

(2) Maintenance visits or callback shall not be counted if it is determinedthat they were attributable to incorrect information supplied by thecustomer as mutually agreed between parties.

(3) Definitions for defects, service volume (Normalization Unit) andmeasurement units for the five (5) applicable product categories aregiven in the following matrix:


TL 9000 Metrics Handbook

8-2

Table 8.1-1. Definitions of Defects, Service Volume and Units ofMeasure by Service Product Categories for ServiceQuality Metrics

Service CategoryInstallation Maintenance Repair Call

CenterSupportService

CountedItem(defect)

# of non-conformingaudits

# ofcallbacks

# ofdefectiverepairwarrantyunits

# ofresolutionsexceedingagreedupon time

# ofdefects

ServiceVolume

(Abbreviation)

Total # ofinstallationaudits

(# audits)

Total # ofmaintenancevisits

(# visits)

Total # ofrepairedunits

(# repairs)

Total # ofcalls

(# calls)

Total # oftrans-actions(# trans-actions)

% SQ % SQ % SQ % SQ % SQ % SQ


The method to compute service quality is percentage conforming (% SQ). Todetermine the percentage conforming to specified criteria the percentage ofnon-conformances shall be counted and subtracted from 100%. Percentageof non-conformance shall be calculated as the total number of defects dividedby the total number of opportunities for defects.

% Service Quality (% SQ) = 100 x (1 – counted item/service volume)

For detailed service quality metrics formulas, see SQ1, SQ2, SQ3, SQ4, andSQ5 in Table 8.1-2.

Notation

(Items are counted according to above stated rules)

NU = Service volume unit (normalization unit)S1 = Installation service volumeS3 = Repair service volumeS2 = Maintenance service volumeS4 = Call center service volumeS5 = Support service volumeSd1 = Number of installation non-conformancesSd3 = Number of defective repair warranty unitsSd2 = Number of maintenance callbacksSd4 = Number of call center resolutions exceeding specified

agreed upon timeSd5 = Number of support service defects



8-3

Table 8.1-2. Service Quality (SQ) Metric Identifiers and Formulas

Identifier Title Formula Note

SQ1 Conforming Installations 100 (1-Sd1/S1) % auditsconforming

SQ2 Successful MaintenanceVisits

100 (1-S23/S2) % visits withoutcallbacks

SQ3 Successful Repairs 100 (1-Sd3/S3) % of successfulrepairs

SQ4 Conforming Call CenterResolutions

100 (1- Sd4/S4) % calls resolvedwithin agreedupon time

SQ5 Conforming Support ServiceTransactions

100 (1-Sd5/S5) % transactionswithout Defect



(2) SQ Data Table – The SQ measurement shall be reported withdata elements (or equivalent as defined by the MetricsAdministrator) for each month and each product category asshown in Table 8.1-3.

Table 8.1-3. Service Quality (SQ) Data Table

(Report one (1) value each for S1-S5, SQ1-SQ5, and Sd1-Sd5)

Year and Month: YYYYMMReporting ID: Provided by Forum AdministratorProduct Category Code: From Appendix ACustomer Base: Either (1) Total or (2) Forum MembersNormalization Factor: S1, S2, S3, S4, or S5

(as appropriate for the specific measurementreported)

Metric Identifier: SQ1, SQ2, SQ3, SQ4, or SQ5SQ Numerator: Sd1, Sd2, Sd3, Sd4, or Sd5

(as appropriate)



8-4


See table 8.1-4.

Table 8.1-4. SQ Data Sources

Category Source of DataInstallation Supplier to count number of non-

conforming supplier installationaudits

Maintenance Supplier to count maintenancerevisits

Repair Supplier to count number of repairedunits that failed within repairwarranty period

Call Center Supplier to count number of call center resolutionsexceeding specified time

SupportService

Supplier to count number of supportservice defects


a. Compared Data (CD) or Research Data (RD)

Conforming Installations: RDSuccessful Repairs: CDSuccessful Maintenance Visits: CDConforming Call Center Resolutions: CDConforming support Service Transactions: RD


a. Installation Example

(1) Data Collected and Results

Table 8.1-5. Example Source Data for Installation SQ

January February March April# of Non ConformingInstallation Audits

5 1 0 6

Total Number of InstallationAudits

100 50 75 80

Service Quality Measurement 95% 98% 100% 92.5%



8-5

(2) Computation for the month of January:100 x (1-5/100) = 95%

(3) Data Report for January, 2001 is shown in Table 8.1-6.

Table 8.1-6. Example Data Report for Installation SQ

Year and Month: 200101Reporting ID: Provided by Forum AdministratorProduct Category Code: 7.1Customer Base: Forum MembersNormalization Factor: 100Metric Identifier: SQ1SQ Numerator: 5

b. Repair Example:


Table 8.1-7. Example Source Data for Repair SQ

January February March April# of Defective Repaired Unitswithin Repair Warranty

2 0 1 4

Total Number of Repaired Units 30 20 75 120

Service Quality Measurement 93.3% 100% 98.6% 96.6%

(2) Computation for the month of January:100 x (1-2/30) = 93.3%

(3) Data report is analogous to the installation example.

c. Maintenance Example:


Table 8.1-8. Example Source Data for Maintenance SQ

January February March April

# of Callbacks 2 0 1 4# of Maintenance Visits 30 20 75 120Quality Service Measurement 93.3% 100% 98.6% 96.6%



8-6


(3) Data report is analogous to the Installation example.

d. Call Center Example:


Table 8.1-9. Example Source Data for Call Center SQ

January February March April# of Call Resolutions thatExceeded the SpecifiedTime Allotment

15 40 10 4

Total # of Calls whichCame into the CallCenter

2000 5000 2750 3000

Service QualityMeasurement

99.25% 99.2% 99.6% 99.8%

(2) Computations for the month of January:100 x (1-15/2000) = 99.25%

(3) Data report is analogous to the Installation example.

e. Support Services Example

This example references a cable locator service with a defined defect as acut cable due to incorrect identification.

(1) Data collected and results

Table 8.1-10. Example Source Data for Support Services SQ

January February March AprilCut Cables(# of Defects)

5 2 0 4

# of Cables Identified(# of Opportunities for Defects)

1000 500 750 300

Service Quality ConformanceMeasurement

99.5% 99.6% 100% 98.6%


(3) Data report is analogous to the installation example.

Appendix A


A-1

Appendix A: Product Category Tables

This appendix is current as of release of this handbook. See the QuESTForum web site (www.questforum.org) for the latest version. The latestversion shall be used in conjunction with registrations.

Suppliers shall classify their products and report measurements according tothe listed product categories. The Metrics Applicability Table (NormalizedUnits) lists specific metrics that apply to each category as well as thenormalized units and other information necessary for compiling measurementreports.

1. List of Tables

Table A-1. Product Category DefinitionsTable A-2. Metrics Applicability Table (Normalized Units)Table A-3. Transmission Standard Designations and ConversionsTable A-4. Optical and Electrical EquivalencyTable A-5. Cost Indicator Weighting Factors (under development)

2. Product Category Definitions

Table A-1 contains definitions of product categories to be used bysuppliers in categorizing their products.

2.1 Rules for Classification of Products

a. A supplier will not be required to report measurements for a given productin multiple product categories. Therefore, any product from a given suppliermust be classified in exactly one (1) product category.

b. General purpose products (e.g., computers) will be classified by specificfunction (e.g., signaling) when provided as a system designed for thatfunction. Otherwise, they will be classified in a separate category, (e.g.,Common Systems-Computers) designed for the general purpose product.

c. A product will be classified according to its primary function. For example,a digital transmission facility product with performance monitoring will beclassified as a transmission product, not an operations and maintenanceproduct.

d. The standard for classification is the product category, not the possibleuses to which the product may be put. For example, if a product classificationfalls in the Outside Plant category, all products that are consistent with thatcategory will be classified as such, even if the exact same product is

Appendix A


A-2

sometimes used in the customer premises and even if a particular supplier'sproduct is sold primarily into the customer premises market.

2.2 Principles for Construction of the Product Category Table

a. Product categories should fall into a clearly defined hierarchy ofclassification.

b. There are well established rules for classification.

c. Product categories should not be separated artificially if they can belogically aggregated.

d. Product categories should have clear definitions, which lend themselves tounambiguous interpretation.

e. For each category, the level to which metrics may be aggregated shall bedefined.

f. For each category, the level to which measurements may be aggregatedshall be defined.

g. Each product category specification shall consist of standard elements.

h. The placement of the product in the hierarchy will reflect the dominant useof the product.

Table A-1. Product Category Definitions (next page)

Appendix A


A-3

Table A-1. Product Category DefinitionsCategoryCode

Category: Definition: Examples:

1. Switching Equipment for the physical or virtual interconnection of communication channels inresponse to a signaling system. The switching category is broadly defined to includepacket or circuit switched architectures.

1.1 Circuit Switch Equipment for the termination of subscriber lines and/or trunk lines and the dynamicinterconnection of these ports or channels in a digital transmission facility. A circuitswitch establishes a dedicated circuit, as opposed to a virtual circuit, in response to asignal. Stored Program Control (SPC) is the most common type of switchingequipment used at end offices and tandem offices. These systems use either analogor digital switching. The switching system used must have the capability to send,receive and be actuated by signals, e.g., access line signals, or inter-office in-bandor common-channel signaling. This category includes all circuit switches regardlessof transmission medium, i.e., wireline, or wireless.

• End-office• Tandem• Tandem access• Remote• Service Switching Point

[SSP]• Mobile Switching Center

[MSC]

1.2 Packet Switch Equipment for switching or routing data on virtual, as opposed to dedicated, circuits.The service is packet switched in that the customer’s data is transported as asequence of data blocks (packets) that do not exceed a specified size. Thispacketization permits data from many data conversations to share a giventransmission facility economically through statistical multiplexing. Such dataconversations are known as virtual circuits, which are full duplex and connection-oriented.

1.2.1 Public PacketSwitched Network(PPSN)

Equipment for the provision of connection-oriented, packet-switched communicationservices designed to provide economical data transport based on internationallystandardized packet protocols. The packet switch is the primary switching elementof the network allowing efficient connectivity to many customers. The accessconcentrator concentrates traffic from lower-speed access lines for more efficientpacket-switch port usage and performs any necessary protocol conversion via thePacket Assembler/Disassembler (PAD) function.

• X.25 packet switch• Access concentrator /

PAD

1.2.2 IP Packet Switch /Router

Equipment which moves variable-length IP (Internet Protocol) packets from source todestination. Routing generally uses software algorithms to optimize one (1) or acombination of data-transport “metrics” such as delay, the use of reliable paths,“hops” between servers, etc. Switching is generally faster than routers since thedecision as to where to send the packet is done by hardware, but are also limited toless sophisticated algorithms than are routers to determine which path the packetsshould use. Most systems provide a combination of routing and switching, asappropriate, to best serve the needs of the user.

Appendix A


A-4



1.2.3 AsynchronousTransfer Mode (ATM)Switch

Switching equipment which operates at OSI Level 2 (hardware layer) to move fixed-length (53-byte) data cells from source to destination over virtual paths or channels.ATM is designed to support mixed data types (voice, video, computercommunications, etc.), provides selectable Quality of Service guarantees and easilyenables billing for data switching services. Throughput of up to 622 Mbps iscommonly available in ATM Switches.

1.2.4 Frame Relay Switch Switching equipment which operates at OSI Level 2(hardware) to move variable-length Frame Relay Frames over virtual circuits from source to destination. Data aremoved without data integrity checks or flow control at up to T3 rates.

2. Signaling Equipment for the provision of signaling, i.e., states applied to operate and controlthe component groups of a telecommunications circuit to cause it to perform itsintended function. Generally speaking, there are five (5) basic categories of "signals"commonly used in the telecommunications network. Included are supervisorysignals, information signals, address signals, control signals, and alerting signals.This category includes those signaling products that function within thetelecommunications network and excludes (possibly similar) products which wouldnormally provide enhanced services outside the network, or on the customerpremises such as ACD, IVR, or voice messaging systems.

2.1 Service Control Point(SCP)

A signaling point that functions as a database to provide information to another SCPor Service Switching Point (SSP). Transaction Capabilities Application Part (TCAP)queries and responses are used to communicate with the SCP as is done for 800Data Base Service and ABS. SCPs may support one (1) or more services per SCPand SCPs may be deployed singularly as stand-alone nodes, as mated pairs, or asmultiple replicates (more than 2) to increase their availability. SCPs, connected toSTPs, are associated with applications that consist of service-specific software and adatabase of customer-related information. This product category includesconventional SCP equipment, plus other platforms such as service nodes, intelligentperipherals, or service resource facilities, which may combine capabilities of a SCP,SSP or which may be used to provide AIN functionality or other enhanced serviceswithin the network.

• Service Control Point• Service nodes• Service resource

facilities

2.2 Signaling TransferPoint (STP)

A signaling point with the function of transferring signaling messages from onesignaling link to another and considered exclusively from the viewpoint of thetransfer. An STP is a specialized routing signaling point (SP). It is an SS7-basedpacket switch that transfers SS7 messages to and from other SPs and is alwaysdeployed in mated pairs for reliability. The STP uses the Message Transfer Part(MTP) and the Signaling Connection Control Part (SCCP) of the SS7 protocol to

Appendix A


A-5



screen and route messages destined for other nodes in the SS7 network. Itfunctions as an SS7 network routing hub, interfacing with SPs only through SS7 linksand not voice or data trunks. Within the LEC CCS network structure, STPs arearchitecturally referred to as either Local STPs (LSTPs) or Regional STPs (RSTPs).

2.3 Home LocationRegister (HLR)

Equipment to provide a permanent database used in wireless applications to identifya subscriber and to contain subscriber data related to features and services. Itstores information such as service profiles, location and routing information forroamers, service qualification, interface for moves, adds and changes. Itcommunicates with other HLRs and provides access to maintenance functions suchas fault information, performance data, and configuration parameters.

3. Transmission Equipment for the connection of the switched and interoffice networks with individualcustomers. An integral part of the distribution network is the loop, which connectsthe customer to the local central office (CO), thus providing access to the interofficenetwork.

3.1 Outside Plant The part of the telecommunications that is physically located outside of telephonecompany buildings. This includes cables, supporting structures, and certainequipment items such as load coils. Microwave towers, antennas, and cable systemrepeaters are not considered outside plant.

3.1.1 Transmission Medium Optical fiber, metallic cable, or other physical medium for the transmission of analogor digital communications.

3.1.1.1 Metallic Products Metallic as opposed to optical or wireless transmission media.

3.1.1.1.1 Metallic ConductorCable

Metallic pairs of conductors housed in a protective cable • Metallic cable• Central office coaxial

cable• Hybrid coaxial/twisted

pair drop 3.1.1.1.2 Metallic Connectors Devices used to terminate a metallic cable. • Coaxial connectors

• Coaxial distributionconnectors

3.1.1.2 Optical Fiber Products Optical, as opposed to metallic or wireless transmission media.

Appendix A


A-6



3.1.1.2.1 Optical Fiber A glass fiber wherein light is propagated and any associated covering. • Multimode fiber• Standard single mode

Fiber• Dispersion shifted fiber• Non-zero dispersion

shifted fiber• Loose tube cable• Central tube cable

3.1.1.2.2 Optical Connectors Device used to terminate an optical cable • Optical SC,ST, or MTconnectors

• Connectorized cableassemblies, e.g., opticalfiber ribbon fanouts

3.1.1.3 Transmission

Components Components embedded in the transmission medium other than cable or connectors

3.1.1.3.1 Active Components Metallic components containing electronics • Coaxial drop amplifiers• Fiber optic data links

3.1.1.3.2 Passive OpticalComponents

Optical components containing no electronics • Wavelength DivisionMultiplexer [WDM]

• Add drop multiplexers• Fiber optic dispersion

compensators• Optical isolators• Filters• Attenuators

3.1.1.3.3 Ancillary Equipment Other transmission components not specifically covered in other transmissioncomponent categories. Typically passive.

• Surge protectors• Bonding and grounding

hardware• Taps

3.1.2 Physical Structure Equipment for the support of telephone transmission media. These physicalstructures include poles, towers, conduits, equipment enclosures such as huts.

3.1.2.1 Enclosures Enclosures for network equipment located in the outside plant. • Fiber optic spliceenclosures

Appendix A


A-7



• ONU enclosures• Organizer assemblies• Seal assemblies• Controlled environment

Vaults 3.1.2.2 Support structures Products for the physical support of transmission media or enclosures. • Telephone poles

• Pedestals• Microwave / radio towers

3.1.2.3 Conduits Channels for the containment of optical fiber or metallic cable. • Innerduct• Multi-bore conduit• PVC pipe

3.2 Transport Equipment Equipment located in the central office or at the customer premises, but inside thenetwork demarcation point, for the transmission of digital or analog communicationover transmission media. This product category includes equipment for terminating,interconnecting, and multiplexing communications circuits.

3.2.1 Cross ConnectSystems

Equipment to provide a physical termination point for physical cables and individualconductors. They can be manual or automated, metallic or optical. Cross-connectsystems, such as distributing frames, Digital Signal Cross Connects (DSXs) and Fiber Distributing Frames (FDFs), provide the following basic functions: cross-connection of network distribution facilities and equipment in the central office,electrical protection for conductive media, test access, temporary disconnection, andtermination points for facilities and equipment.

• Digital Signal CrossConnect (DSX)

• Digital Interface System(DIS)

• Fiber Distribution Frame(FDF)

• Feeder DistributionInterface (FDI)

• Digital Cross-connectSystem (DCS)

• Electronic DSX• Optical DSX• Optical Interface Systems

3.2.2 Carrier Systems /Multiplexers

Equipment for transmitting multiple communication channels over a singletransmission facility. This category includes equipment for transmission overinteroffice trunks, for example, from central to remote offices.

3.2.2.1 Interoffice / Long Haul Equipment for transmission between central offices, between exchanges, or betweencarriers, as opposed to transmission between an end office and a remote location,typical of a loop carrier.

3.2.2.1.1 Metallic Carrier Carrier system that uses metallic transmission medium. • Analog carrier (N-,L-

Appendix A


A-8



System carrier)• D4, D5 digital carrier

3.2.2.1.2 Optical Carrier System Carrier system that uses optical transmission medium. SONET Muxes, WDM3.2.2.1.2.1

SONET / SDHTransport Systems

Fully featured digital transmission system employing optical medium • OC-3, 12, 48, or 192SONET equipmentconfigurable as linear orring.

• Similar for STM-x SDHequipment

3.2.2.1.2.2

WDM / DWDM /Optical AmplificationProducts

Shelf level sub-systems used for multiplexing, de-multiplexing, or amplification ofoptical signals. Lack the built in protection, electrical conversion and other featuresof a SONET Transport System.

• Wavelength DivisionMultiplexer [WDM]

• Dense WavelengthDivision Mulitplexer

3.2.2.1.3 Microwave Carrier system that employs fixed microwave transmission. 6, 8, 11, or 18 gigahertzmicrowave radio

3.2.2.2 Loop Carrier Equipment for deploying multiple voice or digital channels over fewer physicalchannels than would be otherwise required (a “pair gain” function). ). Loop carriersare typically digital systems which employ time-domain multiplexing (TDM) but mayinclude analog systems as well. Loop carrier systems consist of a Central OfficeTerminal (COT) located near the switching system, a Remote Terminal (RT) locatednear the customer to be served and a transmission facility connecting the COT to theRT. Individual communications circuits (such as POTS and Foreign Exchange [FX])are accepted as separate inputs at the COT (RT), time-division multiplexed (in adigital loop carrier) by the loop carrier system and reproduced at the RT (COT). There is an analog-to-digital (A/D) conversion of analog inputs to the DLC and thesesignals, which are carried digitally within the DLC, undergo a digital-to-analog (D / A)conversion when output at the COT or RT. The transmission facility used by a loopcarrier may be metallic cable pairs, repeated metallic cable pairs, or optical fibers.

• Digital loop carrier (DLC)• Universal digital loop

carrier (UDLC)• SLC remote terminal• Integrated digital loop

carrier• Analog loop carrier

3.2.3 Line TerminatingEquipment /Distributing Frames

Equipment to provide the termination point for voice-grade and voice-gradecompatible facilities and equipment in a central office. It is composed of protectors,connectors and terminal strips or blocks. Distributing frames are categorized aseither conventional or modular.

• Tall conventionaldistributing frames

• Low-Profile ConventionalDistribution Frames(LPCDFs)

• Conventional protectorframes

Appendix A


A-9



• Combined MainDistributing Frame(CMDF)

• Subscriber MainDistributing Frame(SMDF)

• Trunk Main DistributingFrame (TMDF)

• Intermediate DistributingFrame (IDF)

• Tie-Pair DistributingFrame (TPDF).

• Office repeater bays 3.2.4 Digital Subscriber Line

(DSL) Equipment for the transport of high-speed digital data on the embedded copperplant. DSL typically will operate over nonrepeatered, POTS-like, conditionedunloaded loops out to CSA ranges. This product category includes central office andremote units, regenerators or range extenders, and supporting equipment.

• ISDN• HDSL• ADSL• DDS

3.3 Wireless Transmission Equipment for analog or digital transmission to the subscriber unique to wirelessservices. This category does not include interoffice or long-haul wireless carriersystems.

3.3.1 Base StationEquipment

Equipment which provides the interface between wireless systems and the PublicSwitched Telephone Network (PSTN). It provides, for example, electrical signalingisolation as well as switching, routing, billing, and features capabilities. It providessubsystems for vocoding and selecting hand off decision.

• BSC• BSS

3.3.2 Base TransceiverSystem (BTS)

Equipment which provides the radio link to the mobile subscribers. It is connected tothe BSC though a backhaul interface between the BSC and BTS for both vocodedand overhead packet traffic.

• BTS

3.3.3 Pilot Beacon Unit(PBU)

Equipment whose primary purpose is to transmit and ANSI J-STD-008 Pilot channeland ANSI J- STD-008 Sync channel and a partial ANSI J-STD-008 Paging channel.The PBU is intended to notify a mobil unit of a change in CDMA coverage and canbe used to assist in the execution of cellular CDMA-AMPS and inter-frequencyCDMA-CDMA hand-off. It is designed with the capability for extended temperatureand environmental operation ranges.

4. Operations &Maintenance

Equipment, systems, and services for the management, upkeep, diagnosis andrepair of the communications network.

Appendix A


A-10



4.1 Test Systems Equipment to support testing of the network. This category includes permanently

installed equipment used to provide a centralized test capability or local test access,as opposed to portable equipment, as might be carried by a craftsperson.

4.1.1 Test AccessEquipment

Equipment to provide test access to transmission circuits. Test access equipment isin series with the customer circuit at all times and therefore directly affects the circuitreliability. This equipment is designed with transmission equipment issues in mind.This equipment may have analog and perhaps a variety of digital (i.e., T1, E1) types.

4.1.2 Test Equipment,Embedded

Equipment to perform tests on transmission circuits. This equipment is designedwith transmission equipment issues in mind. Test equipment is NOT generally inseries with the customer circuit and may be connected to a variety of accessequipment and network elements with integral access features. This equipment mayhave analog and perhaps a variety of digital (i.e., T1, E1) types. Failure of thisequipment doesn't bring down customer circuits; however, it inhibits the ability tomaintain the network and to restore lost service.

4.1.3 Test Support Software Computer software that runs on a general purpose computer (office environment)and perhaps the maintenance network that the computer uses to communicate withthe CO access and test equipment.

4.2 Operations SupportSystems

Systems that provide TMN (Telecommunication Management Network) compliant,flexible, scaleable, and interoperable solutions to automate service activation,service assurance, and network capacity management processes to worldwideexisting and emerging network services and equipment providers.

4.2.1 On Line Critical Real time systems, demanding high availability, typically 24 hours a day and 7 daysper week.

• Network trafficmanagement

• Surveillance of 911• Fire alarms

4.2.2 On Line Non-critical Real time systems with lower availability demands than on line critical systems. • Provisioning• Dispatch• Maintenance

4.2.3 Off Line Traditional business systems that are run off line sometimes in batch mode, typicallyovernight and do not have high availability expectations.

• Inventory• Billing records• Service creation platform

5. Common systems Any of a variety of specialized generic, shared equipment to support networkelements. Common systems include power systems and the Network Equipment-Building System (NEBS) that provides space and environmental support for network

Appendix A


A-11



elements. These systems are located in central offices and remote buildinglocations.

5.1 Synchronization Equipment for operating digital systems at a common clock rate (frequencysynchronization). This category includes primary reference sources and other timingsignal generators that produce a timing signal traceable to UTC.

• Stratum 1,2, 3Edomestic, TNC, LNC andType 1 International

• GPS timing receivers,cesium, loran, or CDMARF pilot timing referencegenerators.

5.2 General PurposeComputers

A category reserved for computer complexes (one or more interconnectedmachines) that perform general business functions for a TSP but which do notprovide any telephony transmission or storage service to subscribers or other TSPcustomers, or which may provide such services, but are not sold to the serviceprovider as part of a system designed exclusively for that purpose. The purposes towhich such machines may be put include but are not limited to:• Accounting systems• Billing systems• Legal systems• Ordering systems• Business Information systems• HR functions• Engineering and support functions• Marketing and Sales functions

• Terminals• PCs• Workstations• Mini, mid, mainframes

5.3 Power Systems Equipment for the provision of power to network equipment. Power systems providetwo (2) principal functions: the conversion of the commercial AC powersource to DC voltages required by the network equipment and the generation anddistribution of emergency (reserve) power when the commercial power is interrupted.This category also includes the ringing plant, a redundant plant which supplies theringing voltage, frequency, tones, and interrupter patterns

• AC rectifiers/batterychargers

• Battery systems• Uninterruptable Power

Supplies (UPS)• DC to AC inverters• DC to DC bulk converters• AC and DC switch gear• Ring generator• Power distribution panels

6. Customer Premises Equipment installed beyond the network demarcation point. Although commonlyinstalled on the subscriber’s premises, equipment with essentially identical functioninstalled in the service provider’s facility may also be classified as customer

Appendix A


A-12



premises equipment.6.1 Enhanced Services

PlatformsSystems that provide an environment in which service-specific application programscan execute and an infrastructure by which those application programs can provideenhanced services. Although each enhanced services platform has a correspondingservice creation environment, that creation environment is sometimes packagedseparately and may execute on a different platform.

6.1.1 Interactive VoiceResponse (IVR)Platforms

Equipment used to allow menu navigation and information retrieval, often fromlegacy databases external to the IVR platform itself.

6.1.2 Messaging Platforms Equipment for storage and retrieval of voice and/or fax messages Voice mail systems6.1.3 Multi-Application

PlatformsEquipment which provides an environment rich in capabilities so that multiple,possible disparate services can be provided concurrently.

Unified/UniversalMessaging (systemproviding a subscriber themeans, from a givendevice, to manipulatemessages originated on likeor different devices. Suchdevices include, but are notlimited to, conventionaltelephone handsets,wireless handsets, PCterminals, fax machines,and email)

6.2 Terminal Equipment Equipment connected to the network demarcation point which provides a service tothe subscriber. Terminal equipment includes telephone sets, whether wireline,cordless, cellular, PCS, or other voice terminals, fax machines, answering machines,modems, data service units (DSUs), or ISDN terminal adapters.

6.2.1 Voice Terminals Conventional, wireless, cellular, PCS, or other voice terminal equipment.6.2.1.1 Wireline Telephone

SetsTelephone sets connected to conventional wireline (POTS) circuits. • POTS telephone sets

• Cordless telephones6.2.1.2 Wireless Subscriber

User TerminalsThe subscriber user terminal made to transmit and receive voice and/or datacommunication using Telecommunication Infrastructure equipment not requiring hardlines as a means of transport. User terminals may be of any functional technologyavailable for public use.

• Wireless single modeuser terminal

• Wireless mobile userterminal

• Wireless stationary userterminal

Appendix A


A-13



• Wireless multi-mode userterminal

• Wireless multi-purposeuser terminal

• Wireless Global userterminal

6.2.2 Fax equipment Equipment for sending or receiving facsimile (fax) over conventional voice-gradelines.

6.2.3 Data Modems Equipment for digital communications over voice-grade lines6.2.4 Digital Data Service

UnitsEquipment for the interconnection of data terminal equipment (DTE) with a digitalcommunications service. Such equipment typically provides a network interface andone or more DTE interfaces and may be configurable.

• DDS CSU / DSU• ISDN CSU / DSU• IDSN terminal adapter• T1 CSU DSU

6.3 Automatic CallDistribution (ACD)systems

Equipment for the distribution of incoming calls to any of a number of destinationsbased on some programmed logic. ACD systems are typically used in customersupport call centers or sales centers.

6.4 Private BranchExchange (PBX)

Equipment to provide circuit switched voice and fax communications services,optimized for medium to large sized customer sites. Now is evolving to utilize ATMand IP networks and support multimedia communications.

6.5 SmallCommunicationsSystem (KeyTelephone System)

Equipment to provide circuit switched voice and FAX communications services,optimized from small to medium sized customer sites. Now is evolving to utilize IPnetworks.

7. Services Result generated by activities at the interface between the supplier and the customerand by supplier internal activities to meet the customer needs.NOTES:

1. The supplier or the customer may be represented at the interface bypersonnel or equipment;

2. Customer activities at the interface with the supplier may be essential to theservice delivery;

3. Delivery or use of tangible products may form part of the service delivery;and

4. A service may be linked with the manufacture and supply of tangibleproduct.[4]

7.1 Installation Service Contracted service to position, configure, and/or adjust a product.

Appendix A


A-14



7.2 Engineering Service Contracted service to design and/or develop a product. This includes, but is notlimited to, the documentation necessary for positioning, configuring , connecting,and/or adjusting.

7.3 Maintenance Service Contracted service to maintain customer’s equipment and/or systems.7.4 Repair Service Contracted service to repair customer’s equipment and/or systems7.5 Call Center Contracted service to process customer requests. This service may include problem

response, problem resolution, and/or information sharing.7.6 Support Service Contracted service that is not included in another product category.

Appendix A


A-15

3. Metric Applicability Table (Normalized Units)

3.1 Metrics Without Normalization Factors

The following metrics: Fix Response Time (FRT), Overdue FixResponsiveness (OFR), and On-Time Delivery (OTD) are applicable andrequired for ALL product categories. They do not require product specificnormalization. In the interest of saving space, they are not listed in thefollowing table but data must be submitted for each of these three (3)metrics for all products. Use the following table to determine thenormalization units and applicability of the rest of the metrics.

3.2 Other Rules and References

Software metrics are based on the three (3) most dominant releases.% = 100 x Quantity Defective / Total Quantity. “%” is applicable to"Software Only" metrics.

“NA” means the metric is not applicable for the product category.

“None” means that no common normalization factor has been identified forthe product category; however, data shall be submitted for the metric.

“NR” means that, although data is collected and the measurement iscalculated by the supplier, the measurement data is not reportable to theMetrics Administrator.

The column headings in Table A-2 are general descriptions coveringseveral sub-metrics in some cases. For cross references to the detaileddescriptions of the metrics elsewhere in this document, findmetric/submetric symbols in Table A-5. Metric Summary Listing.

3.3 Metric Summary Listing

Table A-5 is a listing of the metrics included in this handbook with thesymbols used in data reporting, the applicability to hardware, software,and/or services (H,S,V), and a reference to the table in this handbook withdata reporting details. The symbols listed here are referenced by thenormalization unit and applicability table to clarify the general descriptionsused as column headings.

Table A-2 Metric Applicability Table (Normalized Units) (next page)

Appendix A


A-16

Table A-2. Metric Applicability Table (Normalized Units)Product Category Hardware and Common

Hardware SoftwareSoftware Only

Code Description DowntimePerformance

H, S

OutageFrequency

H, S

Return Rate

H

ProblemReports

H,S

CorrectivePatch

Quality

FeaturePatch

Quality

SoftwareUpdateQuality

ReleaseApplication

Aborts

TL 9000 Metric/Submetric Symbols (See Table A-5.) SO2; SO4; SO1;SO3; RR (all) NPR (all) CPQ (all) FPQ (all) SWU (all) RAA (all)

RQMS Alternative Metric Symbols (See Table A-5.) r,h,DPMs,c_ r,h,OFMs,c IPR (all) DPQ (all) DFP (all) DSU (all) RAQ (all)

1 Switching1.1 Circuit Switch Min/sys/yr. Outages

/sys/yr.returns/

10,000 Terminations/yr.

Prob/sys/mo.

% NR % %

1.2 Packet Switch Min/sys/yr. Outages/sys/yr.

returns/10,000 Ter

minations/yr.

Prob/sys/mo.

% NR % %

1.2.1 Public Packet Switched Network(PPSN)

Min/sys/yr. Outages/sys/yr.

returns/10,000 Ter

minations/yr.

Prob/sys/mo.

% % NA %

1.2.2 IP Packet Switch/Router Min/NC/yr Outages/NC/yr.

returns/10,000 Ter

minations/yr.

Prob/sys/mo.

NA NA % NA

1.2.3 Asynchronous Transport Mode(ATM) Switch

Min/OC-1//yr.

Outages/OC-1/yr.

returns/10,000 Ter

minations/yr.

Prob/sys/mo.

NA NA % NA

1.2.4 Frame Relay Switch Min/sys/yr. Outages/sys/yr.

returns/10,000 Ter

minations/yr.

Prob/sys/mo.

NA NA % NA

2. Signaling

2.1 Service Control Point (SCP)

Min/sys/yr. Outages/sys/yr.

Returns perSystem/yr.

Prob/sys/mo.

% NR % %

2.2 Signaling Transfer Point (STP) Min/sys/yr. Outages/sys/yr.

Returns perSystem/yr.

Prob/sys/mo.

% NR % %

2.3 Home Location Register (HLR) NA NA NA Prob/sys/mo.

% NR % %

3 Transmission

3.1 Outside Plant

Appendix A


A-17




H, S

OutageFrequency

H, S

Return Rate

H

ProblemReports

H,S

CorrectivePatch

Quality

FeaturePatch

Quality


ReleaseApplication

Aborts



3.1.1 Transmission Medium

3.1.1.1 Metallic Cable Products 3.1.1.1.1 Metallic Conductor Cable NA NA NA None NA NA NA NA

3.1.1.1.2 Metallic Connectors NA NA NA Prob/10,0000units/yr.

NA NA NA NA

3.1.1.2 Optical Fiber Products

3.1.1.2.1 Optical Fiber NA NA NA None NA NA NA NA

3.1.1.2.2 Optical connectors NA NA NA Prob/10,000units/yr.

NA NA NA NA

3.1.1.3 Transmission Components

3.1.1.3.1 Active Components NA NA Returns/unit/yr.

Prob/Unit/yr..

NA NA NA NA

3.1.1.3.2 Passive Optical Components NA NA Returns/unit/yr.

Prob/Unit/yr.

NA NA NA NA

3.1.1.3.3 Ancillary Equipment NA NA Returns/unit/yr.

Prob/Unit/yr.

NA NA NA NA

3.1.2 Physical Structure

3.1.2.1 Enclosures NA NA Returns/unit/yr.

Prob/Unit/yr.

NA NA NA NA

3.1.2.2 Support Structures NA NA Returns/unit/yr.

Prob/Untie/yr.

NA NA NA NA

3.1.2.3 Conduits NA NA Returns/unit/yr.

Prob/meter/yr.

NA NA NA NA

Appendix A


A-18




H, S

OutageFrequency

H, S

Return Rate

H

ProblemReports

H,S

CorrectivePatch

Quality

FeaturePatch

Quality


ReleaseApplication

Aborts



3.2 Transport Equipment 3.2.1 Cross Connect Systems Min/DS1/yr. Outages/

DS1/yr.Returns per1000 DS1/

yr.

Prob/sys/yr.

NA NA % %

3.2.2 Carrier Systems/Multiplexers3.2.2.1 Interoffice/Long Haul

3.2.2.1.1 Metallic Carrier System Min/DS1/yr. Outages/DS1/yr.

Returns perDS1/yr.

Prob/sys/yr.

NA NA % %

3.2.2.1.2 Optical Carrier System Min/sys/yr. Outages/sys/yr.

Returns/unit/yr.

Prob/sys/yr.

NA NA % %

3.2.2.1.2.1 SONET/SDH Transport Systems Min/OC-1/yr. Outage /OC-1/yr.

Returns perOC-1/ yr.

Prob/networkelement

/yr

NA NA % %

3.2.2.1.2.2 WDM/DWDM/OpticalAmplification Products

Min/OC-1/yr. Outages/OC-1/yr.

Returns perOC-1/ yr.

Prob/networkelement

/yr

NA NA % %

3.2.2.1.3 Microwave Min/DS1/yr. Outages/DS1/yr.

Returns perDS1/yr.

Prob/networkelement

/yr

NA NA % %

3.2.2.2 Loop Carrier Min/DS1/yr. Outages/DS1/yr.

Returns perDS1/yr.

Prob/DS1/yr.

NA NA % %

3.2.3 Line TerminatingEquipment/Distributing Frames

NA NA Returns/10,000

terminations/yr.

Prob/10,000

terminations/yr.

NA NA % NA

Appendix A


A-19




H, S

OutageFrequency

H, S

Return Rate

H

ProblemReports

H,S

CorrectivePatch

Quality

FeaturePatch

Quality


ReleaseApplication

Aborts



Appendix A


A-22

Table A-2. Metric Applicability Table (Normalization Units)Product Category Applicability and Normalization Units for Services

Code Description Service Problem Reports Service Quality

TL 9000 Metric/Submetric Symbols (See Table A-5.) NPR (all) SQ

RQMS Alternative Metric Symbols (See Table A-5.) NA NA

7 Services

7.1 Installation Service Prob/job/yr. % audits conforming

7.2 Engineering Service Prob/job/yr. NA

7.3 Maintenance Service Prob/unit maintained/yr. % visits without callbacks

7.4 Repair Service Prob/unit repaired/yr. % of successful repairs

7.5 Call Center Prob/1000 calls/yr. % calls resolved within agreed upon time

7.6 Support Service Prob/unit /yr. % transactions without defect

Appendix A


A-23

4. Equivalency Tables

Tables A-3 and A-4 are included for convenience only.

Table A-3. Transmission Standard Designations and Conversions

Electrical Frequency EquivalentNORTH AMERICAN

DS0 64 Kb 1 TrunkDS1 1.544 Mb 24 DS0

VT 1.5 1.728 Mb 1 DS1, 24 DS0DS1C 3.152 Mb 2 DS1, 48 DS0DS2 6.312 Mb 4 DS1, 96 DS0DS3 44.736 Mb 28 DS1, 672 DS0

STS-1 51.84 Mb 1 DS3, 28 DS1, 672 DS0STS-3 155.52 Mb 3 DS3, 84 DS1, 2,016 DS0

STS-12 622.08 Mb 12 DS3, 336 DS1, 8,064 DS0STS-48 2488.32 Mb 48 DS3, 1,344 DS1, 32,256 DS0STS-192 9953.28 Mb 192 DS3, 5,376 DS1, 129,024 DS0

INTERNATIONAL (PDH)E1 - 2 Mbits/sec 2,048 Mb 30 64 Kb ChannelsE2 - 8 Mbits/sec 8,448 Mb 4 2 Mbit/s, 120 64 Kb Channels

E3 - 34Mbits/sec

34,368 Mb 4 8 Mbit/s, 16 2 Mbit/s, 480 64 KbChannels

E4 - 140Mbits/sec

139,264 Mb 4 34 Mbits/s, 64 2 Mbit/s, 1,920 64 KbChannels

565 Mbits/sec 636,000 Mb 4 140 Mbit/s, 16 34 Mbit/s, 64 8 Mbit/s, 2562 Mbit/s, 7,680 64 Kb Channels

Appendix A


A-24

Table A-4. Optical and Electrical Equivalency

Optical Electrical Frequency EquivalentNORTH AMERICAN (SONET)

OC-1 STS-1 51.84 Mb 1 OC-1, 1 DS3, 28 DS1, 672 DS0OC-3 STS-3 155.52 Mb 3 OC-1, 3 DS3, 84 DS1, 2,016 DS0OC-12 STS-12 622.08 Mb 12 OC-1, 12 DS3, 336 DS1, 8,064 DS0OC-48 STS-48 2,488.32 Mb 48 OC-1, 48 DS3, 1,344 DS1, 32,256 DS0

OC-192 STS-192 9,953.28 Mb 192 OC-1,192 DS3, 5,376 DS1, 129,024DS0

OC-768 Not available 39,680 Mb Not availableOC-1536 158,720Mb Not available

INTERNATIONAL (SDH)STM-1o (OC-3) STM-1e 155.52 Mb 1 E4, 4 E3, 64 E1, 1,920 ChannelsSTM-4o (OC-

12)STM-4e 622.08 Mb 4 E4, 16 E3, 256 E1, 7,680 Channels

STM-16o (OC-48)

STM-16e 2,488.32 Mb 16 E4, 48 E3, 1,024 E1, 30,720 Channels

STM-64o (OC-192)

STM-64e 9,953.28 Mb 64 E4, 192 E3, 4,096 E1, 122,024Channels

Not applicable VC-11 (VT1.5) 1.644 Mb (1.544 Mb) 1 DS1Not applicable VC-12 (E1) 2.240 Mb (2.048 Mb) 1 E1 (2 Mb)Not applicable VC-2 (VT6) 6.784 Mb (6.312 Mb)Not applicable VC-3 (E3) 48.960 Mb (34.368 Mb) 1 E3 (34 Mb)Not applicable VC-4 (E4) 150.336 Mb (139.264 Mb) 1 E4 (140 Mb)

5. Metric Summary Listing

Table A-5 is a listing of the metrics included in this handbook showing: (1) thesymbols used in data reporting; (2) the applicability to hardware, software,and/or services (H,S,V); and, (3) a reference to the table with data reportingdetails. The symbols listed here are also included in Table A-2, MetricApplicability Table (Normalized Units), to clarify the general descriptions inthe column headings.

Appendix A


A-25

Table A-5. Metrics Summary Listing

Table A-5. Metrics Summary Listing.Para-graph

MetricSub-Metric

MetricSymbol

Sub -metric

Symbol

Applic-ability

(H/S/V)

ReportedItems

(Table)

Comparedor Research

Data

5.1 Number of Problem Reports Formulas: Table 5.1-1

NPR H,S,V 5.1-3

H/S Critical Problem Reports perNormalization Unit

NPR1 H,S compared

H/S Major Problem Reports perNormalization Unit

NPR2 H,S compared

H/S Minor Problem Reports perNormalization Unit

NPR3 H,S compared

Service Problem Reports per NormalizationUnit

NPR4 V compared

Number of Problem Reports - RQMS Alternative Formulas: Table 5.1-2

IPR H,S 5.1-4

Incoming Critical Problem Reports perSystem per Month

IPR1 H,S compared

Incoming Major Problem Reports perSystem per Month

IPR2 H,S compared

Incoming Minor Problem Reports perSystem per Month

IPR3 H,S compared

5.2 Problem Report Fix Response Time Formulas: Table 5.2-1

FRT H,S,V 5.2-3

H/S Major Problem Reports Fix ResponseTime

FRT2 H,S compared

H/S Minor Problem Reports Fix ResponseTime

FRT3 H,S compared

Service Problem Reports Fix ResponseTime

FRT4 V compared

Problem Report Fix Response Time - RQMSAlternative Formulas: Table 5.2-2

ORT H,S 5.2-4

% Major Problems Closed on Time ORT2 H,S compared% Minor Problems Closed on Time ORT3 H,S compared

5.2 Overdue Problem Report Fix Responsiveness Formulas: Table 5.2-1

OFR H,S,V 5.2-3

H/S Major Overdue Problem Reports FixResponsiveness

OFR2 H,S research

H/S Minor Overdue Problem Reports FixResponsiveness

OFR3 H,S research

H/S Service Overdue Problem Reports FixResponsiveness

OFR4 V research

Overdue Problem Report Fix Responsiveness -RQMS Alternative Formulas: Table 5.2-2

OPR H,S 5.2- 5

% Rate of Closures of Overdue ProblemReports - Major

OPR2 H,S research

% Rate of Closures of Overdue ProblemReports - Minor

OPR3 H,S research

Appendix A


A-26


MetricSub-Metric

MetricSymbol

Sub -metric

Symbol

Applic-ability

(H/S/V)

ReportedItems

(Table)

Comparedor Research

Data

5.3 On Time Delivery Formulas: Table 5.3-1

OTD H,S,V 5.3-2

On Time Installed System Delivery OTIS H,S,V comparedOn Time Items Delivery OTI H,S comparedOn Time Service Delivery OTS V compared

5.4 System Outage Measurement Formulas: Table 5.4-1

SO H,S 5.4-4

Annualized Weighted Outage Frequency SO1 H,S comparedAnnualized Weighted Downtime SO2 H,S comparedAnnualized Supplier Attributable OutageFrequency

SO3 H,S compared

Annualized Supplier Attributable Downtime SO4 H,S comparedSystem Outage Measurement - RQMS AlternativeMetrics for End Office and/or Tandem Office,Wireless Products and NGDLC Products Formulas: Table 5.4-2

SOE H,S 5.4-5

Supplier Attributable Total Outage Minutesper System per Year - Remote Only

rDPMs H,S compared

Supplier Attributable Total Outage Minutesper System per Year - Host Only

hDPMs H,S compared

Service provider Attributable Total OutageMinutes per System per Year - RemoteOnly

rDPMc H,S compared

Service provider Attributable Total OutageMinutes per System per Year - Host Only

hDPMc H,S compared

Supplier Attributable Total Outages perSystem per Year - Remotes

rOFMs H,S compared

Supplier Attributable Total Outages perSystem per Year - Hosts

hOFMs H,S compared

Service Provider Attributable Total Outagesper System per Year - Remotes

rOFMc H,S compared

Service Provider Attributable Total Outagesper System per Year - Hosts

hOFMc H,S compared

System Outage Measurement - RQMS AlternativeMetrics - General Series Formulas: Table 5.4-3

SOG H,S 5.4-6

Total Outage Minutes per System per Year -Overall

DPM H,S compared

Total Outage Minutes per System per Year -Supplier Attributable

DPMs H,S compared

Total Outages per Year - Overall OFM H,S comparedTotal Outages Per Year - SupplierAttributable

OFMs H,S compared

6.1 Return Rates Formulas: Table 6.1-1

RR H 6.1-2

Normalized First Year Return Rate FYR H comparedLong-Term Return Rate LTR H research

Appendix A


A-27


MetricSub-Metric

MetricSymbol

Sub -metric

Symbol

Applic-ability

(H/S/V)

ReportedItems

(Table)

Comparedor Research

Data

7.1 Software Update Quality Formulas: Table 7.1-1

SWU S 7.1-2

Defective Software Updates - Release N-2 SWU2 S comparedDefective Software Updates - Release N-1 SWU1 S comparedDefective Software Updates - Release N SWU0 S compared

7.2 Release Application Aborts Formulas: Table 7.2-1

RAA S 7.2-3

Release Application Aborts- Release N-2 RAA2 S comparedRelease Application Aborts - Release N-1 RAA1 S comparedRelease Application Aborts - Release N RAA0 S compared

Release Application Aborts - RQMS AlternativeMetrics Formulas: Table 7.2-2

RAQ S 7.2-4

Cumulative % of Systems Experiencing anAbort during Release Application - ReleaseN-2

RAQ2 S compared

Cumulative % o Systems Experiencing anAbort during Release Application - ReleaseN-1

RAQ1 S compared

Cumulative % of Systems Experiencing anAbort during Release Application - ReleaseN

RAQ0 S compared

Cumulative Number of Release ApplicationAttempts - Release N-2

Rar2 S compared

Cumulative Number of Release ApplicationAttempts - Release N-1

Rar1 S compared

Cumulative Number of Release ApplicationAttempts - Release N

Rar0 S compared

7.3 Corrective Patch Quality Formulas: Table 7.3-1

CPQ S 7.3-3

Defective Corrective Patches - Release N CPQ0 S comparedDefective Corrective Patches - Release N-1 CPQ1 S comparedDefective Corrective Patches - Release N-2 CPQ2 S compared

Corrective Patch Quality - RQMS AlternativeMetrics Formulas: Table 7.3-2

DCP S 7.3-4

Monthly Number of Defective CorrectivePatches Identified - Release N

DCP0 S compared

Monthly Number of Defective CorrectivePatches Identified - Release N-1

DCP1 S compared

Monthly Number of Defective CorrectivePatches Identified - Release N-2

DCP2 S compared

Monthly Number of Corrective PatchesDelivered - Release N

CPr0 S compared

Monthly Number of Corrective PatchesDelivered - Release N-1

CPr1 S compared

Monthly Number of Corrective PatchesDelivered - Release N-2

CPr2 S compared

Appendix A


A-28


MetricSub-Metric

MetricSymbol

Sub -metric

Symbol

Applic-ability

(H/S/V)

ReportedItems

(Table)

Comparedor Research

Data

7.3 Feature Patch Quality Formulas: Table 7.3-1

FPQ S 7.3-3

Defective Feature Patches - Release N FPQ0 S researchDefective Feature Patches - Release N-1 FPQ1 S researchDefective Feature Patches - Release N-2 FPQ2 S research

Feature patch Quality - RQMS Alternative Metrics Formulas: Table 7.3-2

DFP S 7.3-4

Monthly Number of Defective FeaturePatches Identified - Release N

DFP0 S research

Monthly Number of Defective FeaturePatches Identified - Release N-1

DFP1 S research

Monthly Number of Defective FeaturePatches Identified - Release N-2

DFP2 S research

Monthly Number of Feature PatchesDelivered - Release N

FPr0 S research

Monthly Number of Feature PatchesDelivered - Release N-1

FPr1 S research

Monthly Number of Feature PatchesDelivered - Release N-2

FPr2 S research

8.1 Service Quality Formulas: Table 8.1-2

SQ V 8.1-3

Conforming Installations SQ1 V researchSuccessful Repairs SQ2 V comparedSuccessful Maintenance Visits SQ3 V comparedConforming Call Center Resolutions SQ4 V comparedConforming Support Service Transactions SQ5 V research

Appendix B


B-1

Appendix B: TL 9000 Customer Satisfaction Metrics Guidelines

The TL 9000 Quality System Requirements Handbook contains requirementsfor measuring customer satisfaction. The design of the mechanism forcollecting data from customers will necessarily be unique to each supplier.This appendix offers guidelines to assist suppliers in the design or review oftheir own customer feedback program.

B.1 Profile for Customer Satisfaction Metrics – The following metrics profileprovides basic guidelines for a customer feedback mechanism and referencesa detailed example of a customer satisfaction survey. Results may beprovided to customer organizations that have direct experience with thesupplier’s products or performance that these organizations may include, forexample, Quality, Purchasing, Operations, Engineering, Planning, Logistics,and Technical Support.

B.1.1 Purpose

The purpose of these metrics are to measure and improve the degree ofcustomer satisfaction with a supplier and its products from a customer point ofview to help the supplier to improve the satisfaction of its customers.

B.1.2 Applicable Product Categories

All products delivered through a purchase order and fulfillment process. Thisshould include stock items as well as items that are made-to-order.

B.1.3 Detailed Description

Feedback is obtained through various mechanisms (such as satisfactionsurveys and front line customer technical support input). The surveys shoulddetermine the importance of the items surveyed as well as how satisfiedcustomers are. Analysis should include trends and rates of improvement.

B.1.4 Sources of Data

Both customers and suppliers collect data on satisfaction with a supplier’sproducts.

B.1.5 Method of Delivery or Reporting

Both customers and suppliers should administer the mechanism fordetermining customer satisfaction. Results should be obtained at least once

B. Metrics Profilefor CustomerSatisfactionMechanism

Appendix B


B-2

per year and reported according to each customer or supplier firm’s ownformats and procedures.

B.1.6 Example

For an example survey, see the QuEST Forum web site. The following aretypical survey topics:

Quality of Delivery• Delivers on time• Meets due date without constant follow-up• Lead time competitiveness• Delivers proper items• Delivers proper quantities• Accurate documentation and identification• Handles emergency deliveries

Quality of Pricing• Competitive pricing• Price stability• Price accuracy• Advance notice on price changes

Quality of Customer Service• Compliance to contract terms• Supplier representatives have sincere desire to serve• Provides feedback from factory• Recognizes cost effectiveness• Market insight• Training provided on equipment/products• Support on professional and technical matters• Invoicing efficiency• Issuing credit notes• Order acknowledgement• Adherence to company policy

Quality of Product• Product reliability/durability/meets specifications• Product documentation, instructions, technology• Product packaging, suitability, environmental aspects• Contract service quality

Glossary


Glossary-1

Glossary of Abbreviations, Acronyms and Definitions

This Glossary contains a list of abbreviations and acronyms followed bydefinitions of terms. Definitions of terms that appear only in the ProductCategory Table, Table A-1, are not provided here.

ABBREVIATIONS and ACRONYMSA&M Additions and MaintenanceABS Alternate Billing ServiceACD Actual Completion DateACD Automatic Call DistributionAfactor Annualization FactorAIN Advanced Intelligent NetworkAOJD Actual On-Job DateATM Asynchronous Transfer ModeBSC Base Station ControllerBSS Base Station SystemBTS Base Transceiver SystemCCS Common Channel SignalingCO Central OfficeCOT Central Office TerminalCPQ Corrective Patch QualityCRCD Customer Requested Completion DateCRD Customer Requested DateCROJD Customer Requested On-Job DateDCS Digital Cross Connect SystemDIS Digital Interface SystemDPM Downtime Performance MeasurementDS(x) Digital Signal LevelDSX Digital Signal Cross ConnectDWDM Dense Wavelength Division MultiplexerE(x) European Digital RateFAX Facsimile (Electronic)FDF Fiber Distribution FrameFDI Feeder Distribution InterfaceFPQ Feature Patch QualityFRT Fix Response TimeFRU Field Replaceable UnitFYR First year Return rateGA General AvailabilityH HardwareH/S Hardware Common to SoftwareHLR Home Location RegisterIP Internet ProtocolIP Intelligent PeripheralIR Information RequestISDN Integrated Services Digital Network

Glossary


Glossary-2

ABBREVIATIONS and ACRONYMSIVR Interactive Voice ResponseLEC Local Exchange CarrierLOR Late Orders ReceivedLSTP Local Signaling Transfer PointLTR Long-term Return RateMD Manufacturing DiscontinuedMSC Mobile Switching CenterMTBF Mean Time Between FailureNGDLC Next Generation Digital Loop CarrierNPR Number of Problem ReportsNTF No Trouble FoundOC-(xxx) North American Equivalent Optical RateOFM Outage Frequency MeasurementONU Optical Network UnitOPR Overdue Problem ReportOSS Operational Support SystemOTD On Time DeliveryOTI On Time Item DeliveryOTIS On Time Installed System DeliveryOTS On Time Service DeliveryPBX Private Branch ExchangePDH Plesiochronous Digital HierarchyPO Purchase OrderPOTS Plain Old Telephone ServiceRAA Release Application AbortsRQMS Reliability and Quality Measurements for Telecommunications SystemsRSTP Regional Signaling Transfer PointRT Remote TerminalS SoftwareSCP Service Control PointSDH Synchronous Digital HierarchySFAR Service Failure Analysis ReportSLC Subscriber Line ConcentratorSO System OutageSONET Synchronous Optical Network ElementSPC Stored Program ControlSQ Service QualitySS7 Signaling System 7SSP Service Switching PointSTM-(x)e Synchronous Transport Module, ElectricalSTM-(x)o Synchronous Transport Module, OpticalSTP Signaling Transfer PointSTS Synchronous Transport SignalSWU Software Update QualityTCAP Transactional Capabilities Application PartUDLC Universal Digital Loop CarrierV ServiceVC Virtual ContainerVT Virtual TributaryWDM Wave Division Multiplexers

Glossary


Glossary-3

ABBREVIATIONS and ACRONYMSxDLC Digital Loop CarrierxDSL Digital Subscriber Line

Glossary


Glossary-4

Definitions

Actual CompletionDate (ACD)

The date when service is complete at a job site and accepted by thecustomer.

Actual On-Job Date(AOJD)

The date when the shipment actually was delivered at the ship toaddress. The derived on-job date is calculated by adding thecalculated transportation interval to the actual ship date.

Additions andMaintenance (A&M)

New equipment available in the form of plug-in modules andselective sub-assemblies that is intended specifically to expandpreviously delivered hard-wired but only partially equipped systemsand to support module replacement requirements.

Annualization Factor(Afactor)

The Afactor is the factor applied to annualize the return rate. It is thenumber of reporting periods in one (1) year.

For example, the Afactor would be 12 for data collected on a calendarmonth basis. For fiscal month data, the factor would be 52 dividedby the number of weeks in the fiscal month. A measurement ofreturns from any period could be converted to an annual rate bymultiplying the returns by 365 and dividing by the number of days inthe reporting period.

Basis Shipping Period A length of time during which FRUs were shipped to the customer.Specifically the period during which the FRUs that comprise thepopulation for determining the return rate were shipped. Assumingthe return rate measurement is being figured for the current month,the basis shipping periods used for the return rate metrics in thishandbook are:

• First Year Return Rate (FYR): current month and 12previous month

• Long-Term Return Rate (LTR): after the 12th month

Closure Criteria Specific results of supplier actions that the customer agrees aresufficient to resolve the customer’s service problem report.

Closure Date The date on which a service problem report is resolved, asacknowledged by the customer.

Closure Interval The length of time from origination of a service problem report tothe agreed upon expected closure date.

Glossary


Glossary-5

Common ChannelSignaling (CCS)Isolation Outage

A Common Channel Signaling (CCS) Isolation Outage for an End,Tandem or Combined office, includes the loss of signaling capabilityin all CCS links for a period longer than 30 seconds. CCS Isolationis not included in the definition of Partial Outage.

Compared Data Measurements that are adequately consistent across suppliers andappropriately normalized such that comparisons to aggregateindustry statistics are valid. Only industry statistics based on“compared data” as designated within each metric profile are postedon the QuEST Forum Web Site. See also research data.

Corrective Patch A patch that is intended to correct a fault.

Customer The current or potential buyer/user of products or services.Customers may include service providers, systems integrators, orsuppliers that purchase components or subsystems to include in afinal product or to help deliver telecommunications services.

Customer AttributableOutage

A customer attributable outage is an outage that is primarilyattributable to the customer’s system or support activities such astranslation or procedural errors or office environment problems.

Customer Base The defined group of customers that the supplier’s metrics dataencompasses. The customer base options are as follows:

(1) Forum Members: only the supplier’s customers that aremembers of the QuEST Forum. This is the minimum customerbase.(2) Total: all of the supplier’s customers for the product(s) towhich the metric applies.

Customer RequestedDate (CRD)

From the customer's purchase order, the CRD is the date of desireddelivery of ordered items or systems and/or, if applicable, completionof the services ordered by the customer. Equal to either CRCD orCROJD depending on order type. Order types can be: installedsystem, items, or service.

Glossary


Glossary-6

Cutover Cutover is the placement of a network element or networkconfiguration into service for subscribers.

Date Order Received The calendar date on which a supplier receives an order for productsor services. Used to compute the OTD measurements.

Defective CorrectivePatch

An official corrective patch that includes any of the following:• Cannot be installed,• Does not correct the intended problem;• Is withdrawn by the supplier; and• Corrects the intended problem but causes an additional critical or

major problem

Defective FeaturePatch

A Feature Patch that• Cannot be installed, or• Is withdrawn because of potential or actual problems.• Fails to provide the intended feature capability.• Provides the intended feature capability but causes an additional

critical or major problem.

Defective SoftwareUpdate

A software update that cannot be installed or is withdrawn becauseof potential or actual problems.

Design Design is defined as either of the following(1) The process of defining the architecture, components, interfacesand other characteristics of a system, component, or service.(2) The result of the process in (1).

Dominant Release Dominant Release indicates the relative degree of deployment in thefield. One release is more dominant than another if it is deployed ata greater number of installations.

Duplicate ProblemReports

Problem reports that upon investigation reveal the same fault hasoccurred either at a different customer location or at another time.

End Customer A user of telecommunications services.

Glossary


Glossary-7

EngineeringComplaint (EC)

A mechanism used to document a problem to the supplier forresolution. Problems reported may include unsatisfactory conditionsor performance of a supplier products or services. Exampleproblems reported: alleged fire safety or environmental hazard;failure to meet performance specifications; need for excessive fieldmaintenance; installation errors; and others as defined inGR-230-CORE [2]. A generic EC process flow for use by customersand suppliers to achieve resolution is suggested in [2].

Fault A defect that prevents or impairs a system from correctly performinga function and typically requires a change to the product set(hardware, software, documentation, or procedures). A faultexcludes problems that are: information requests, procedural errorsby the customer or supplier, requests for design changes, orenhancements (feature request). Also excluded are problems thatoccur after GA but before implementation due to shipment orlogistics, or from a duplicate cause.

Feature Patch A patch that adds functionality and is not a corrective patch.

Field Replaceable Unit(FRU)

A distinctly separate part that has been designed so that it may beexchanged at its site of use for the purposes of maintenance orservice adjustment. An example of a FRU is a plug-in circuit board.

First Year Return Rate(FYR)

The FYR is the return rate of units during first year of operation(months 4 through 12 after shipment.

Fix A fix is a correction to a problem. A fix temporarily or permanentlycorrects a fault.

Fix Response Time(FRT)

For major and minor hardware/software problems, FRT is theinterval from the receipt of the original problem report by the supplierto the time of the first delivery of the official fix.

General Availability(GA)

Life cycle phase during which a product or release is available to allsites at which it could be used.

Identical ProblemReports

Problem reports which are multiple reports of the same occurrenceof the same problem at the same location at the same time.

Glossary


Glossary-8

Information Request(IR)

An IR is a service request for which a customer with minimaltechnical expertise and acquaintance with the product could havedetermined or resolved on their own. An IR may have one (1) ormore of the following characteristics:• It documents a problem that the customer could have resolved

independently of the supplier.• The customer asks a question on procedures that are covered in

the documentation, which is shipped with the product.• The customer asks for information on the supplier’s product that

will be used to help interface the supplier’s product with acompetitor’s product.

• The customer asks for help on a “problem” that turns out not tobe a problem, bug, or failure, but rather is due to a lack ofunderstanding of the product.

In-Service Release For metrics associated with outages, a software release or a systemis “in service” when it handles end-user traffic or transactions. Thisincludes field trials prior to General Availability where end customersare affected. For all other metrics, in service is synonymous withGeneral Availability.

Installation Audit A supplier’s final internal audit of an installation project. Thisaudit takes place prior to customer acceptance.

Installed System A system installed by the supplier of the system hardware and/orsoftware or by another supplier of installation services.

Installed SystemOrder

An order for a system engineered, furnished and installed (EF&I) bythe supplier of the system hardware and/or software and having aCRCD.

Item Order An order for line items in a non-System order to be furnished (F)having a CROJD.

Late Order Received(LOR)

An order for products or services that is received after the customerrequested date (CRD) for delivery of the products or completion ofthe services. As used in the on time delivery (OTD) Metric, LORmeans that CRD is before Date Order Received. LORs areexcluded from the OTD Metrics.

Long-Term ReturnRate (LTR)

The LTR is the return rate of units after the first year of operation(months 16 and later) after shipment.

Glossary


Glossary-9

Maintenance Any activity intended to keep a functional unit in satisfactory workingcondition. The term includes tests, measurements, replacements,adjustments, and repairs.

Maintenance Callback A Maintenance Callback is a supplier personnel customer site visitfor maintenance rework.

Maintenance Visit(repair, preventive)

Supplier personnel customer site visit for maintenance.

ManufacturingDiscontinued (MD)

An MD product is one that is no longer available for purchase.

No Trouble Found(NTF)

An NTF is a supplier tested returned item where no trouble is found.

Non-ConformingInstallation Audit

An installation audit that fails to satisfy specified acceptancerequirements.

Normalization Factor The Normalization Factor is the total number of normalization units inthe product or product population to which a metric is applied. It isused in the metric denominator to reduce measurements on differentpopulations to comparable per unit values.

Normalization Unit The conventional increment of measure used in this handbook todescribe the size of a product or product population to which a metricis applied. See also Normalization Factor.

Official Fix A fix approved by the supplier and made available for generaldistribution.

Official Patch A corrective or feature patch for which delivery to all affecteddeployed systems has begun.

Official SoftwareUpdate

A software update for which delivery to all affected deployedsystems has begun.

Glossary


Glossary-10

On Time Delivery Collection of OTS, OTIS, and OTV.

On Time InstalledSystem Delivery(OTIS)

Percentage of Installed Systems delivered on-time to CustomerRequested Completion Date.

On Time Item Delivery Percentage of items delivered on-time to Customer Requested On-Job Date.

On Time ServiceDelivery

Percentage of Services completed on-time to Customer RequiredCompletion Date.

Overdue ServiceProblem Report

A service problem report that has not been resolved on or before aparticular date as agreed by the customer and supplier.

Patch A patch is any interim software change between releases (deliveredor made available for delivery to the field). It consists of one (1) ormore changes to affected parts of the program. Patches may becoded in either source code or some other language.

Problem - Critical H/S Critical problems severely affect service, capacity/traffic, billing, andmaintenance capabilities and require immediate corrective action,regardless of time of day or day of the week as viewed by acustomer upon discussion with the supplier. For example:

- A loss of service that is comparable to the total loss of effectivefunctional capability of an entire switching or transport system

- A reduction in capacity or traffic handling capability such thatexpected loads cannot be handled

- Any loss of safety or emergency capability (e.g., 911 calls).

Glossary


Glossary-11

Problem - Major H/S Major problems cause conditions that seriously affect systemoperation, maintenance and administration, etc. and requireimmediate attention as viewed by the customer upon discussion withthe supplier. The urgency is less than in critical situations becauseof a lesser immediate or impending effect on system performance,customers, and the customer’s operation and revenue. Forexample:

- Reduction in any capacity/traffic measurement function- Any loss of functional visibility and/or diagnostic capability- Short outages equivalent to system or subsystem outages, with

accumulated duration of greater than 2 minutes in any 24-hourperiod, or that continue to repeat during longer periods

- Repeated degradation of DS1 or higher rate spans or connections- Prevention of access for routine administrative activity- Degradation of access for maintenance or recovery operations- Degradation of the system’s ability to provide any required critical

or major trouble notification- Any significant increase in product related customer trouble reports- Billing error rates that exceed specifications- Corruption of system or billing databases.

Problem - Minor H/S Other problems that a customer does not view as critical or majorare considered minor. Minor problems do not significantly impairthe functioning of the system and do not significantly affect serviceto customers. These problems are tolerable during system use.

Engineering complaints are classified as minor unless otherwisenegotiated between the customer and supplier.

Problem Report A problem report includes all forms of problem reporting andcomplaints registered from the customer such as written reports,letters and telephone calls that are recorded manually or enteredinto an automated problem reporting tracking system.

Product Result of activities or processes.NOTES:1. A product may include service, hardware, processed materials,

software, or a combination thereof.2. A product can be tangible (e.g., assemblies or processed

materials), intangible (e.g., knowledge or concepts), or acombination thereof.

3. A product can be either intended (e.g., offering to customers) orunintended (e.g., pollutant or unwanted effects).

Glossary


Glossary-12

Product Category A grouping of products that a supplier considers as one (1) class forthe purpose of reporting measurements of quality. The recognizedgroupings for reporting TL 9000 metrics are listed in Appendix A ofthis handbook. Some examples of product categories are:emergency AC power systems, digital cross connect, and signalingtransfer point (STP). See also service categories.

Release A combination of software, hardware, firmware, and/ordocumentation which is issued by the supplier to provide new,improved, or changed functionality.

Release Application The process of installing a generally available release in acustomer’s in-service product.

Release ApplicationAbort

A release application is the regressing to a previous release withinseven (7) days of cut-over or the reschedule of the releaseapplication within 24 hours of cut-over.

Release ApplicationWindow

The release application window is the time period beginning 21 daysprior to scheduled cut-over and ending seven (7) days after cut-over.

Research Data Measurements that are not consistent from one supplier to anotherand/or are not possible to normalize and consequently cannot becompared to aggregate industry statistics. Industry statistics fromresearch data are analyzed for trends and reported to the metricswork groups. See also “compared data.”

Resolutions (callcenter)

Resolution to customer’s call.

Glossary


Glossary-13

Return • A return is any unit returned for repair or replacement due to anysuspected mechanical or electrical defect occurring duringnormal installation, testing, or in-service operation of theequipment.

• Working or untested units returned as part of a formal rotation orrecall program are not considered returns for the purposes ofthese metrics.

• Units, which fail due to the problem corrected by the recall beforethey can be rotated, are to be counted as returns.

• Returns include units damaged during normal shipping handlingwhere the container itself is not damaged due to abnormalshipping conditions.

• No trouble found (NTF) units, i.e., returned units which aredetermined by the supplier to meet supplier’s specifications, shallbe included in return totals.

• Units damaged during shipping or while in service due tovehicular accidents, water leakage, electrical spikes outside ofspecified limits, or other environmental factors outside thoseconditions for which the equipment was designed may beexcluded.

• Items that were ordered in error, purposely ordered in excess, orconsignment items which have not been found defective mayalso be excluded from the measure.

• All returns from laboratory systems and/or First Office Application(FOA) systems may be excluded from these measurements.

Rework Rework is an action taken on a non-conforming product.

Scheduled Outage A scheduled outage results from a scheduled or plannedmaintenance, installation, or manual initialization. This includes suchactivities as parameter loads, software/firmware changes, andsystem growth.

Service A service is a result generated by activities at the interface betweenthe supplier and the customer and by supplier internal activities tomeet the customer needs.NOTES:1. The supplier or the customer may be represented at the interface

by personnel or equipment;2. Customer activities at the interface with the supplier may be

essential to the service delivery;3. Delivery or use of tangible products may form part of the service

delivery; and4. A service may be linked with the manufacture and supply of

tangible product.

Glossary


Glossary-14

Service Categories Product categories listed in Appendix A that refer to services.Service categories include, for example, installation service, repairand maintenance service, engineering service, network monitoringservice, call center, training service, and support services.

Service Order An order for service having a CRCD, but not an installed systemsorder. Services may include engineering (E) or installation (I).

Service ProblemReport

A formal report of dissatisfaction because a contracted servicerequirement was not met. Service problems may be either tangibleor intangible. Tangible problems are those indicating dissatisfactionwith the result of the service. For example:

• Installation of an H/S product was poor.• Service was not provided in a timely fashion.• Service was below quality expectations.• Documentation (MOPS, procedures) associated with a

service contained errors/deficiencies.Intangible problems are those indicating dissatisfaction withpersonnel. For example:

• An individual had a poor attitude.• Human interactions were below expectations.

Service problem reports may be reported via any media such aspaper, telephone, email, or Internet. They shall include theoriginator’s name and a feedback mechanism for closure.

Service Provider A company that provides telecommunications services to the public.

Service ProviderAttributable Outage

A service provider attributable outage is an outage that is primarilyattributable to the service provider’s system or support activities suchas translation or procedural errors or office environment problems.

Service Unit A service unit is a quantifiable unit to measure the volume of theservice category. The normalization unit as applied to services. See“Normalization Unit.”

Service Volume Service volume is a measure of the amount of service delivered.

Severity Level Severity level refers to the classification of a problem report ascritical, major or minor. See “Problem – Critical H/S;” “Problem –Major H/S;” and “Problem – Minor H/S.”

Glossary


Glossary-15

Software Update A software update is a set of changes to a release to fix problemswhich may also include new features that a supplier wishes to deployon a timely basis rather than wait for the next base release. Asoftware update completely replaces the existing software in theproduct.

Supplier AttributableOutage

A supplier attributable outage is an outage primarily triggered by 1)the system design, hardware, software, components, or other partsof the system; or 2) scheduled events necessitated by the design ofthe system; or 3) supplier support activities including documentation,training, engineering, ordering, installation, maintenance, technicalassistance, software or hardware change actions, etc.

System A system is an assemblage of all facilities, united by some form ofregular interaction or interdependence, required to accomplish acomprehensive function or functions. In this handbook, a system istypically considered to perform or support functions for atelecommunications network such as transmission, operationssupport, switching, or power.

Telecommunications Telecommunications is the conveyance of information, such asvoice, images, or data, through a transmission channel withoutalteration of the message content.

Temporary Fix A fix that is delivered to a limited number of systems in the field forthe purposes of verification or to solve system problems requiringimmediate attention. A temporary fix is usually followed by an officialfix.

Temporary Patch A patch that is delivered to a limited number of systems in the fieldfor the purposes of verification or to solve system problems requiringimmediate attention. A temporary patch is usually followed by anofficial patch.

Total Calls (callcenter)

Number of calls initiated to the call center.

Total System Outage A failure that results in the loss of functionality of the entire system.

Glossary


Glossary-16

Transaction The complete cycle from a service request through the completion of theservice by the supplier.

Bibliography


Bibliography-1

Bibliography

1. GR-929-CORE Reliability and Quality Measurments for Telecommunications Systems(RQMS), Morristown, NJ, Telcordia Technologies, Issue 4, December1998.

2. GR-230-CORE Generic Requirements for Engineering Complaints, Morristown, NJ,Telcordia Technologies, Issue 2, December 1997.

3. GR-1323-CORE Supplier Data—Comprehensive Generic Requirements, Morristown,NJ, Telcordia Technologies, Issue 1, December 1995.

4. ISO 8402:1994 Quality Management Assurance – Vocabulary, Geneva, Switzerland,International Organization for Standardization.

5. TR-NWT-000284 Reliability and Quality Switching Systems Generic Requirements(RQSSGR), Morristown, NJ, Telcordia Technologies, Issue 2, October1990.

Review Report


Review Report TemplateName:

Company:

Phone:

E-mail:

Directions – In the table below, please fill in the first four (4) columns.• “Section” - Which Handbook and which section of the TL 9000 Handbook(s) your comment applies to;• “Company” - Your company name;• “Code” – Please use one (1) of the codes listed below to identify the classification / disposition of your suggestion / comment• Identify your “Specific Suggestions / Comments” with your recommended wording changes.

Code – G = Grammar, Spelling & CosmeticC = Critical – Show Stopper – specify required changes to obtain your approval.S = Suggestion

This form is available electronically on the QuEST Forum Web page www.questforum.org.

Section Company CodeG/C/S

Specific Suggestions / Comments Incorp.*Y/N

Work Group Response*

1. 2. 3.

4. 5. 6. 7. 8. 9. 10. 11.

This must be returned to QuEST Forum either via email [email protected] or fax 414-765-8665

tl 9000 quality system metrics book two - · pdf filethe tl 9000 quality system metrics book...

Documents