global business services © 2007 ibm corporation considerations for developing a performance...

Global Business Services

© 2007 IBM Corporation

Considerations for Developing a Performance Measurement System

MTAC Workgroup #114April 13, 2007

© 2007 IBM Corporation2 Performance Measurement Methodology 4/13/2007

Agenda

Introduction

Performance Measurement Methodology

General Study Considerations

Sample Design

Mail Fabrication

Induction

Receipt

Data Analysis

Reporting

Summary


Introduction The USPS requested that we present today based on the

requirement from the Postal Accountability and Enhancement Act to implement performance measurement for all Market Dominant Products

IBM’s Background in External Performance Measurement– Have conducted USPS transit-time measurement studies since 1990

– Mail measurement experience in more than 35 countries on various domestic and international initiatives

– Methodology developed specifically for this work• Simple in concept

• Complexity is in all the details


Performance Measurement Methodology

Quality Assurance

StatisticalDesign

Test MailFabrication

Test MailInduction

Test MailReceipt

DataAnalysis

Reporting

Panel Management


General Study Considerations

What objectives need to be met?

Who are the various stakeholders?– What are their interests and priorities?

Standards and best practices– What are other countries doing?


Sample Design

Representative of real mail flows – Base the design on real data (e.g. ODIS)

Need a defined measurement period and precision level– Monthly, quarterly, annual estimates to +/- X%

What are you interested in measuring?– Usually on-time rate– Could also consider measuring:

• Percent of mail delivered within a delivery window• Percent early, on time and late• Percent delivered within J+1 days, J+2 days…


Provide for continuous measurement – Measure across the period you are interested in – Include all days, weeks, months in the period

Follow live mailing patterns to the extent possible– Across days of the week– Across weeks of the quarter/year

Define the field of study that you want to measure– Performance of a mailing– Performance of letters, cards, flats– Performance of all Standard Mail

Sample Design (cont)


Meet desired stratification and geographic coverage by spreading inductions and receipts accordingly

– Could use available ODIS data:• Origin/destination ZIP Code pairs• Address type (handwritten, machine-addressed) • Barcoded• Shape or weight• Postage type• Day of week mailed/received

– Could also use other characteristics:• Physical appearance (color, size, window)• Presort level• Sender type • Method of acceptance/collection• Receiver type



Provide for a geographic stratification to ensure coverage across all areas covered by the product

– Some systems place o/d links into sample groups and sample links from the groups to represent the population• Major metro to major metro• Major metro to rural areas of size X• Rural areas to rural areas

– Some systems seek to estimate performance for:• All mail originating from an area with

certain precision • All mail destinating to an area • Origin/destination combination is more complex



Sample size required is largely determined by:– Number of sample groups to be measured– Desired level of precision for more detailed cuts of the data– Time period for which precision is needed– Variability of the population

Larger sample sizes can improve the precision– A simple random sample size of 1,067 achieves precision of +/-3% – A sample size of 4 times that size achieves +/-1.5%– Sample must represent the field of study well to be accurate



What else drives sample size?

– Variability of the results • Closer to 50% on time - more sample needed

• Closer to 100% on time - less sample needed

– Number of and correlation between test pieces mailed at the same time • The more pieces are correlated, the less information

you have

– Number of usable pieces out of your sample (after business rules are applied)



How are test pieces created / chosen– Sampling– Created by mailers in different locations– Centrally by a third party– Will test pieces need to be shipped for induction

Matching test pieces to participants– Seeding receiver names into mailings– Timing and coordination

What addressing standards are needed– Quality of address:

• Avoiding mail forwarding and other delays– What checks are done up front versus later

in the process

Mail Fabrication


How will you manage senders / mailers– Where are they needed based on design

– Recruiting, training, and performance monitoring

What types of mail acceptance are needed – Collection boxes

– Post Office

– DDU, SCF, ADC, AADC, BMC or BMEU

How will the Start-the-Clock data be validated– What will the induction rules be

– When will induction data be removed

Induction


Are receivers needed and how many– What types of delivery addresses are needed

– Recruiting, training, and monitoring

– Handling performance issues

How many pieces can be delivered to the same address: – Without compromising the confidentiality of the study

– Without overburdening receivers

How will the Stop-the-Clock data be validated– What will the rules be

– When will receipt data be removed

Receipt


Processes will be needed to validate the statistical assumptions and results– Design will rely on having data about the characteristics of the population to

ensure it is representative

– Weighting methodology for metrics will need to be validated

– Must develop a methodology for estimation that is agreed to and understood

Business rules about what data are suspicious and how to identify, investigate and decide whether or not to keep data are needed– Pieces missing valid start-the-clock or stop-the-clock information

– Pieces that were forwarded, returned to sender, etc

– A receiver’s data when there is a pattern of “bad” reporting

– Extreme circumstances (e.g. hurricanes, blizzards, blackouts)

Data Analysis


What deliverables will be needed– Reports– Data files

Who are the recipients– USPS– Mailers– PRC / Congress

What level of reporting is needed– National– Mailer-level– By sample group

How frequently should deliverables be released– Preliminary versus final results

Reporting


Credibility of the measurement system depends on quality processes

Must be able to withstand scrutiny:

– Mailer and USPS stakeholders

– GAO

– Congress

Ability to identify and respondto issues

Quality


Summary

There are numerous considerations in external measurement that vary depending on:– Objectives and priorities of your stakeholders

– Which product you are measuring

– What level of detail you require

Quality Assurance

StatisticalDesign

Test MailFabrication

Test MailInduction

Test MailReceipt

DataAnalysis

Reporting

global business services © 2007 ibm corporation considerations for developing a performance...

Documents

mail measurement experience

flats performance

defined measurement

ibm corporation considerations

continuous measurement

days slide

mailing performance

percent of mail