Dissemination to support Research & Analysis

John Cornish

Global trends• Users around the world are seeking access to more detailed data for research &

analysis– More analytical/quantitative academic research– Evidence-based policy – More international comparisons and exchange of information amongst

researchers, incl for monitoring MDGs

• More NSOs are finding ways of providing access to detailed data, including microdata– Better understanding of user needs– Improved usability & functionality of software interfaces– Improved database technology– Growth of Data Archives

• Trend towards ‘self help’ for most users

• 2003 Conference of European Statisticians agreed that more needed to be done to support research and analysis– But must not go beyond what is legal or acceptable to the public

Ways of supporting users doing research & analysis

• Have relevant data which is fit for use– Detailed aggregated data & Microdata

• Good documentation, metadata and archiving - users need to find what is available and all about it

• Access mechanisms

• Use standards, maintain consistency• Partnerships• Staff who understand researcher needs• Involvement with professional organisations (eg

population associations, economists)

Macrodata

• Data more detailed than in publications– Detailed tables – Datacubes– With or without tabulation software

• Detailed Indicators - ability to ‘drill down’

• Thematic databases– Time series– Geographic/area based/international– industry

Indicators

• Web ideal for allowing users to drill down, draw graphs/maps

• More for assessing trends rather than reasons for trends

Microdata

• There are limits to analysis which can be done with macrodata

• Users want more flexibility with different cross-classifiers, different aggregations, etc

• Lots of models of analysis require working at the micro level

• Linking of datasets

International Guidelines

• Guidelines on “Managing Statistical Confidentiality and Microdata Access” by Trewin et al

• Contains many case studies on good practices

• Available on the UNSD website

Guideline Principles

• It is appropriate to use microdata for statistical research & analysis as long as confidentiality is protected

• Microdata should only be made available for statistical purposes

• Provision must be legal

• The procedures for researcher access to microdata, as well as the uses and users, should be transparent and publicly available

Confidentiality• Usefulness & confidentiality risk are inversely

related• Usefulness increases with detail• Confidentiality risk increases with detail• Usefulness reduces with confidentiality

adjustments (eg collapsing of detail)

Confidentiality risk = f (sampling fraction + file detail + adjustments + management of access) also quality

Greatest risk is with business records & families/households

Options for microdata access

• Release of confidentialised microdata files-direct to user or via data archive

• Remote access to more detailed confidentialised microdata files

• Microdata services - submit jobs, NSO runs & vets

• Data Laboratories

• Research collaborations

Managing the Risks

• Microdata Review Panel

• Provide a variety of access methods

• Legally enforceable undertaking is important

• Withdrawal of service

• Training of researchers before allowing access

• Audits of actual practices

Data Archives & Microdata Services

• International Census Microdata Service (IPUMS) provided by University of Minnesota

• SADA, Data First

• Key benefits to NSOs- Archiving and data recovery services- Harmonisation of variables to facilitate

comparisons across countries and time- Allows access to international data sets- Documentation of microdata in English

AASDA

Infrastructure issues

• Data warehouse/data management - consistency of outputs

• Metadata• Standards - data and presentation• Confidentiality knowledge & measures• Interface tools• Access facilities & vetting• Archiving

Some success factors• Good user engagement

• Sound data management

• Management strategies for risk- risk management, not risk avoidance

• Overall vision for website & a transition path

• Research & analysis done by NSO