institut de la statistique du québec (isq) surveys: compendium of good practices and quality...

Institut de la statistique du Québec (ISQ) surveys: compendium of good practices and quality self-assessment checklist

European Conference on Quality in Official Statistics

Rome, July 8-11, 2008

Louise BourqueFrance Lapointe

Institut de la statistique du Québec

2

Presentation outline

• Context

• Compendium of good survey practices– Development

– Content

• Survey quality self-assessment checklist– Objectives

– Development

– A few observations by users (pre-test)

• Challenges in applying the tools

• Conclusion

3

Context

• The Institut de la statistique du Québec (ISQ)– Its enabling legislation

“... to provide reliable and objective statistical information” “... in such a manner that comparisons in and outside Québec

may be made”

– Organization of work A number of units work together in conducting a survey

4

Context (cont.)

• Specific work dealing with quality incorporated into everyday activities – Variation according to the project and unit (neither systematic

nor harmonized)

• The ISQ has adopted an Integrated Quality Management Framework– General Quality Management Policy (November 2005)

5

Context (cont.)• General definition of quality

Quality is composed of all the characteristics that affect the capacity to meet the needs of users

• Six dimensions of quality– Relevance– Reliability and objectivity– Comparability– Timeliness– Intelligibility– Accessibility

6

Context (cont.)

• The policy refers to many documents (policies, procedures, programs, reports, strategic declarations, etc.)

• More specifically, survey quality assurance tools– Compendium of good practices

– Self-assessment checklist

7

Compendium of good survey practices

• Objective– Promote the use of good practices in conducting surveys, at

all stages, in order to ensure the quality of the results

• Development– Multidisciplinary team

▪ Bringing together all the entities involved in the carrying out of surveys

▪ Mandated by top manager▪ Work directed and monitored by a senior management

committee

8

Compendium of good survey practices (cont.)– Inventory of reference practices

Statistics Canada, Statistics Finland ... Division of the survey process into 16 stages

– Observation of ISQ practices 10 or so surveys selected as representative of diversity Application of a common observation checklist

o Adaptation of the European DESAP checklist

– Comparison ISQ practices vs. reference practices Choice of selected practices

9

Compendium of good survey practices (cont.)• Content of a section related to a stage

– Introduction Overview of the work to be done and links with quality dimensions

– Definitions (non-specialized reader)

– Principles (reader knows how a survey is conducted) General quality assurance policy directions

– Good practices (specialized reader) with ratings(1) Essential practices(2) Highly recommended practices(3) Recommended practices

10

Self-assessment checklist

• Objectives– Survey self-assessment and quality measurement according to

the six quality dimensions selected

– Checklist at the time of design, and throughout the conducting of a survey

– Verification of the suitable application of the compendium’s good practices

– Training for new employees

– Etc.

11

Self-assessment checklist (cont.)

• Development– Multidisciplinary team

– In three stages Adapt the DESAP checklist for observation of practices at the

ISQ (done) Review adaptation following use for observation (pre-test) with

the comments received (in progress) Ensure consistency with the compendium's good practices (in

future)

12

Self-assessment checklist (cont.)

• A few observations further to the pre-test– Logical structure of the questionnaire

– Recognized usefulness of the checklist

– High response burden Long questionnaire Complex questions, difficult to interpret

– Questionnaire format to be revised

13

Tool application challenges

• The tools must ...– Be in keeping with the ISQ’s reality– Gain the approval of the various entities

• The tools must be developed ...– At the least cost

– While ensuring that everyday activities continue

14

Tool application challenges (cont.)

• Tool introduction must ...

– Ensure the tools’ appropriation and use by employees

• Tool use must not ...– Be too constraining

– Require too many resources

15

Tool application challenges (cont.)

• General approach selected – Not to reinvent the wheel

– Actively involve the units concerned

– Be guided and supported by senior management

– Inform and train the staff

– Incorporate means of monitoring application

16

Conclusion• Current and future work ...

– Compendium of good practices Information and training activities for employees (in progress)

– Finalize the self-assessment checklist (in progress)

– Develop and adopt a survey quality policy (in future) Monitor the use of the compendium and checklist

17

Thank you!

Louise Bourque

louise.bourque@stat.gouv.qc.ca

France Lapointe

france.lapointe@stat.gouv.qc.ca