How to Use Mistakes to Improve Credibility

By Leon Østergaard, Statistics Denmark

UNECE Work Sessionon Statistical Dissemination and Communication

12-14 September 2006Washington D.C., USA

The problem with errors

• Knowledge is not gathered systematically– We lack the information necessary to

learn from our mistakes– We are not able to quality label our statistics

• Hiding mistakes puts credibility at risk– Not admitting errors is contrary to

user experience– Disguising errors is even worse!

Making mistakes visible

• Gathering information on errors– Being able to learn from mistakes– Making mistakes visible internally

• Publishing errors loud and clear– A chance to show that errors are not hidden– The error in itself will not add to credibility,– but a limited amount, corrected loud and

clear, will increase credibility!

Classification of errors

• Blemishes– users not affected– never in statistical figures

• Minor errors– unlikely that users are misled– only minor groups at risk

• Serious errors– real possibility that more than minor groups

are misled

Blemishes

• Users not affected• Never in statistical figures• Examples:

– Faulty links in pdf-documents– Reference period not updated in figure

(but everywhere else)– Spelling errors and punctuation

Dealing with blemishes

• Corrected online• Print not corrected• No mention of the correction

Minor errors

• Unlikely that users are misled• Only minor groups at risk• Examples:

– Percentage changes in regional unemployment were wrong. Comment and actual unemployment figures were correct

– Two nationalities were mixed up in statistics on overnight stays. Comment and all other figures were correct

– In a table on males and females, ’total’ and ’females’ were mixed up. Comment and all other figures were correct

Dealing with minor errors

• Corrected online• Note inserted in corrected statistics• Note inserted in list of statistics• Print not corrected (but books includes

link to updated version)• No action to actively forward information

Serious errors

• Real possibility that more than minor groups are misled

• Examples:– Error in both total and regional figures

on overnight stays– Unemployment figures by sex and age

groups were wrong. Comment and other figures were correct

– Sale of beer in 2005 dropped by 0.8 per cent, not by 26 per cent(!)

Dealing with serious errors

• Corrected online – or removed• Note inserted on website frontpage• Paragraph on the error inserted

in corrected statistics• Note inserted in list of statistics• All known users receive

corrected statistics

Collecting information on errors

• User interface on intranet asking:– type of publication?– description of error?– discovered when and by whom?– classification of error?– causes and responsibility?– actions taken to correct and when?– actions to prevent future errors?

Collecting information on errors

• Reporting done by news release editor• Information stored automatically• E-mails generated to statisticians,

communication and management• Transparency of the process

Setting a standard for errors

• Blemishes do not count• In 2010: Minor or serious errors in

less than 1 per cent of statistics• In 2006: Minor or serious errors in

less than 1.5 per cent of statistics• Not more than 15 errors in 2006

A preliminary statistics on errors

• First seven months:– 19 blemishes– 5 minor errors– 7 serious errors

• Forecast for 2006:– blemishes in 3 per cent of publications– errors in 2 per cent of publications

What’s next?

• Using experience to reduce the occurrence of errors

• Develop similar tool for collecting information on errors in databanks

• Setting a standard for errors in databanks

• Continued monitoring of credibility• More in-depth monitoring