A Basis for Better Decision Making: Improvements to Quality Reviews and Reporting

Catherine Bremner and Sarah Tucker

Office for National Statistics

Motivation

• Quality Centre is Centre of methodological expertise in statistical quality and respondent burden

• Working over the past 12 months to streamline current quality initiatives and drive innovation

Outline

• Introduction to quality reporting• Quality reporting developments• Using quality reports to drive quality

improvements• The new quality review process at ONS

Quality Reporting Overview

• Why do we report on Quality?

• Give users enough information to decide on suitable uses for the data

• Meet obligations under Code of Practice • Meet requirements from Eurostat

Quality and Methodology Information

• Static quality information• Updated on annual basis or when there are

major methodological changes• Includes:

• history of output • details of statistical process • reports against European Statistical System

Quality Dimensions

• Extensively quality assured

Quality and Methodology Information

Quality and Methodology Information

• Current format 5 years old – time to review • Changes to website – accessibility• More information on user personas - Inquiring

Citizen, Data Forager and Expert Analyst• Need to change the way we present the data

– content unlikely to change much as meets our obligations

Reviewing Presentation of Quality Information

• Aims to extend the reach of quality information to user personas – alongside current QMIs

• Investigations currently focussing on information that users need so they can’t misuse data

• First steps: consulting with data producers, methodologists and other stakeholders

Reviewing Presentation of Quality Information

• One size does not fit all• Pool of suggested pieces of information to

choose from• Fit together to give picture of how not to

misuse data• Held focus groups/workshops and individual

meetings to gain thoughts on what should be considered for inclusion

Internal Consultation Ongoing – Points Raised

• Four focus groups have been held - 2 each for output managers and quality assurers

• Individual meetings with output managers and other stakeholders

• Some themes are coming through consistently: uncertainty – is it an estimate etc, periodicity, coherence, accuracy and health warnings.

• As expected, emphasis is different across type of outputs – admin data vs. Survey

Next steps

• Compiling a shortlist of these vital pieces of information

• Work with data producers to produce pilot• User test• Review results• Propose action

Quality Reviews at ONS

• Previously carried out Annual Quality Reviews

• Made use of Quality, Methodology and Harmonisation Tool (QMHT)

• Responding to feedback, developed a new process during summer 2014

Regular Quality Reviews (RQR)

• Face to face meeting between Output manager, Deputy Director, G6/G7 Methodologist and Quality Centre representative

• Makes use of existing information from Value Engineering and Quality and Methodology Information (QMI) reports, which are discussed at the meeting

• Quality Assurance (DD) walk-through incorporated into the meeting

Regular Quality Reviews (RQR)

• G6/G7 methodologist produces bespoke recommendations, which will be presented at Management/Project boards

• Output reviewed once every 3 years• Supports the Code of Practice • Piloted on 3 outputs during summer 2014• Process rolled out in November 2014; 13

reviews conducted to date

Why are RQRs required?

• Support requirements of the Code of Practice• Assesses the output• Adds value to output through bespoke

recommendations• Allows recommendations to be addressed,

improving the output

What do we expect?

• What we already require - QMI, Quality Assurance (DD) walkthrough, Quality checklist

• Quality Centre collate all the relevant documents e.g. QMI, Value Engineering, Assessment Reports to be reviewed

• Output Manager, DD, and team members (as required) to meeting

Recommendations – Adding Value

• Aim to write up within two weeks of the meeting

• Assigned a RAG status• Prioritised from a Methodological and ONS

view• Timetable of when/how recommendations are

dealt with depend on what they are• Range from a quick win – update QMI or

longer term plan of getting work onto business plan

• Role for Quality Centre in monitoring

Relevance for GSS

• Regular Quality Review approach is a tool that could be used by GSS

• Expected that senior statistician could carry out review

• Alternatively, support from Methodology Advisory Service

• The approach will be included in the GSS quality resources

Contacts

• Quality Reporting Sarah Tucker

email: ons.metadata.project@ons.gov.uk

• Regular Quality Reviews

Catherine Bremner, email: ons.quality.centre@ons.gov.uk