abstractsabstracts mainz, germany, castle of the prince elector, 24–26 may 2004 federal...

Abstracts

Mainz, Germany, Castle of the Prince Elector, 24–26 May 2004

Federal Statistical Office Germany

European Conference on Quality and Methodology in Official Statistics (Q2004)


- 51 -

© Federal Statistical Office Germany

Abstracts

Session 2: Invited DACSEIS Session Großer Saal Chair: Chris Skinner, University of Southampton, United Kingdom

Technical and Organisational Design of POULPE by Jean-Claude Deville, CREST/ENSAI, France

POULPE is a software written in SAS-macro language destined to compute estimations of variance for arbitrary complex design, arbitrary linear estimator (weighting) and arbitrary complex estimators. It rely on very few simple basic concepts. First of all we have to get an estimator of the variance for a total estimated by the Horvitz-Thompson estimator. In fact, formally, there are only three basic designs:

- Poisson sampling (and Bernoulli as a particular case)

- Fixed size design of maximum entropy (and simple random sampling as particular case).

- Systematic sampling (with equal or unequal probabilities).

Complex design (stratified, multistage) are treated by using the well known recursive formulas. Two or three phases sampling (with application for complete non-response) is also available for many second phase designs (or, equivalently for many models of non-response).

Complex estimators of totals are tackled by the use of the residual trick as far as they can be seen as calibrated (generalized eventually) estimators. The variance of non-linear indicators (means, ratios, correlations, matrix eigenvalues and vectors, concentration indicators) is estimated as those of the total of the linearized variable associated to the given indicator.

Variance computation when some data are imputed has not been included because the project has been cancelled by the administrative authorities before its end. This is one of the reasons for which the software is not very easy to learn and to use. The other reason is that a correct use of the software requires a rather deep knowledge of the subtleties of variance estimation (which is an art as well as a science!). However, this is a necessity even for very simple surveys (for instance because we have to deal with non-response). In an organism such as a National Statistical Institute, POULPE requires a person (not necessary full-time) having those capabilities, for preparing the parameterisation of the software before it can be used mechanically by a survey manager.

Contact: [email protected]


- 52 -


Complex Auxiliary Information by Carl-Erik Särndal, University of Montreal, Canada

Complex sampling design and complex parameter are familiar concepts in survey sampling. Less familiar is complex auxiliary information, discussed in this presentation. The term is used to contrast with the typical standard situation, where a known population auxiliary vector total is used for estimation, with calibrated weights, for a single phase sampling design.

Complex auxiliary information is present when several components from different sources contribute to the total information. They may come from the population or its subgroups, from one or more samples in one or more different surveys. Complex auxiliary information is often found in connection with sampling in two or more stages or in two or more phases. One wishes to benefit to the fullest extent possible from the total information, in a computation of calibrated weights and point estimates, and their associated variance estimates.

It is a characteristic of complex auxiliary information that the same given information can be used in more than one way when the calibrated weights are computed. “The best use” is not always apparent, a priori. The statistician must exercise judgement. Computational issues also enter into consideration: the computations, with suitable software, should be rapid without foregoing any important information.

Calibration estimators are non-linear. Their variance cannot be expressed in simple exact form. In order to assess their variance, we need a simple approximate procedure. A tool for this is automated linearization. We explain this technique and apply it in a few examples involving complex auxiliary information.


Session 3: Metadata I Spiegelsaal Chair: Clementina Ivan Ungureanu, National Institute of Statistics, Romania

Metadata as a Tool for Enhancing Data Quality in a Statistical Agency by Alice Born, Statistics Canada

Statistics Canada defines the quality of information in terms of its fitness for use. This is a multidimensional concept embracing both the relevance of information to users’ needs and characteristics of the information such as accuracy, timeliness, accessibility, interpretability and coherence. This paper examines the role of metadata in the management of data quality as an important component of overall management of a statistical agency.


- 53 -


One key role of metadata is to support of the interpretability of statistical information. This normally covers the underlying concepts, variables and classifications; methodology of data collection and processing; and measures of data accuracy. To fulfil this role, Statistics Canada has built a corporate metadata repository referred to as the Integrated Metadatabase (IMDB), which stores metadata for over 700 surveys. This information is accessible to users, including survey respondents, survey managers and analysts, on Statistics Canada’s website. The IMDB data model is an adaptation of ISO 1179 and corporate metadata repository (CMR) models, which are standards for managing data elements (variables) and other components in metadata registries.

Coherence of statistical information reflects the use of standard concepts, classifi-cations, variables, statistical units, and conceptual frameworks. The IMDB also serves a data management tool for definitional metadata. A framework has been developed to provide a way of organizing all the statistical units, variables and related classifications disseminated in the Agency that allows it to improve the coherence of the definitions of variables and use of classifications across subject matter areas.

For the metadata program, relevance means providing the right metadata at the right level of detail in order for users to be able to interpret the data that Agency disseminates. Feedback is obtained from client satisfaction surveys, advisory com-mittees, subject matter areas, survey managers and monitoring the number of hits on the metadata web pages. Since parts of the IMDB are still under development, input on the type and quality of metadata from Statistics Canada divisions is critical at this time.

In terms of the accuracy of the metadata itself, the Agency regularly reports on the level of completeness and correctness of the metadata for each IMDB record. This information is used to prioritize efforts and areas for further improvement of the Agency’s metadata program. It is also used to measure compliance to Statistics Canada’s policy on informing users as well as reporting to Parliament and other federal government agencies.



- 54 -


Conceptual metadata and process metadata: key elements to measure quality components of the statistical system by Max Booleman, Statistics Netherlands

With the use of the Dutch metadata model the transparency of the statistical system and the dissemination policy of Statistics Netherlands will be improved. After a short introduction of the Dutch metadata model the consequences and additional possibilities to measure the quality components coherence and comparability of the statistical information are up for discussion.

The metadata model structures the statistical information in such a way that it provides information about coherence and consistency of the statistical information. With the help of this information indicators can be developed to measure the coherence of the subsets of the statistical system.

Remarks: The views expressed in this paper are those of the author and do not necessarily reflect the policies of Statistics Netherlands.


Quality and Metadata – Some Connections and Reflections by Eva Elvers, Statistics Sweden

The European Statistical System (ESS) emphasises quality. The main quality dimen-sions are: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, and coherence. Standard quality reports have been developed as a means of communication between the National Statistical Institutes and Eurostat. Moreover, quality indicators for the ESS are under development. It is desirable to have a highly automatic system, especially for indicators.

The finite population parameters, which are estimated, build on statistical units with associated variable values and sub-populations (domains of estimation). Variables can have several roles when micro-data are aggregated to macro-data. Information about the finite population parameters – the contents of the statistics – is essential. Typically, statistics are accompanied by explanatory information in table headings, footnotes, and surrounding text. The standard quality reports do not say much about the contents, though, since there are Regulations.

Concepts are fundamental, and they are clearly needed when coherence and coherence deficiencies are investigated. Methods have a strong influence too. This is the case not only for accuracy but also for comparability and coherence. Information about statistics – statistical metadata – should, of course, include the quality dimensions. Since many quality aspects are difficult to measure, there are different sorts of facts, including quality indicators and descriptive information.


- 55 -


The requests for metadata grow with an increasing demand on statistics and analyses based on different sets of statistics. Metadata should be collected along the production process. Some choices are clear, but much more experience is needed on what facts to collect and save for future use. There are several descriptions for metadata systems, but a lot remains in practice.

Methodologists devote much work to quality and quality information. A search in the directory of the Journal of Official Statistics (since 1985) returns 82 articles with ‘quality’ in the abstract. However, there is only one article with ‘metadata’ in the abstract; one more article has ‘metadata’ as keyword. Quality and metadata are both buzzwords, but they are perhaps not as strongly connected, as they ought to be. Some reflections are made on quality, metadata, and connections between the two.


Quality and Metadata in Statistics Norway by Anne Gro Hustoft, Jenny Linnerud and Hans Viggo Sæbø, Statistics Norway

Many statistical institutions have launched systematic quality work, with user needs as the point of departure for defining quality. One of the main quality dimensions of statistics is accessibility and clarity of statistics. Statistical metadata defined as systematic documentation of statistics are necessary for users to find, understand and use the statistics. In addition metadata linked to production processes are necessary to improve these. Hence, metadata has become an important issue in the work on improving quality and efficiency in statistical institutions. The paper considers the role of metadata within the framework of systematic quality work. Typical challenges a statistical institution faces in this field are how to provide different metadata to different external and internal users on different levels of detail, and how to implement and link various metadata systems to promote efficiency in production processes and dissemination.

Statistics Norway has developed many different metadata systems to serve different purposes and different user groups. These systems have often been developed in isolation from each other in a traditional "stovepipe" environment. This has led to the same information being stored several times in several places making the availability of updated and consistent information difficult. In the last two years, there has been a strong focus on the need to link existing systems and a requirement that new metadata systems should not be built in isolation. Our aim is that metadata should be updated in one place and accessible everywhere. Our metadata systems should also be useful as tools for the harmonisation and standardisation of our documentation. The paper will discuss the linking of different metadata systems e.g. system for variables documentation, file description database, classification server, dissemination database and facts about the statistics or quality declarations for end users.

Contact: Anne Gro Hustoft: [email protected], Jenny Linnerud: [email protected] and Hans Viggo Sæbø: [email protected]


- 56 -


Metadata Description System in Polish Official Statistics as Attitude to Quality Improvement of Official Statistics by Irena Kwasna, Central Statistical Office, Poland Issue of system integration is one of conditions for quality improvement of statistical System. It doesn‘ t cover integration of surveys results only but assurance of detailed information concerning surveys also. Nowadays there are many info databases in polish statistical system.: Statistical Surveys Programme, Polish Public Statistics, Statistical Studies Outline.

Database of Files Description. Database of Statistical Questionnaires, Database of Statistical Information Flowcharts Metadata System of Administrative Data Sources, Publication Schedule, Documentation of Statistical Surveys, Glossary of Terms, Classifications. GUS aim is to integrate all these elements within the confines of common system for statistical meta data and to set up resources to open relevant project.

Statistical Metadata System will be a core part of integration of Statistical Databases System. It will provide users with detailed knowledge concerning databases content, browsing methods, data selection and it will secure full data description. Defined system consists of:

- subsystems that are sources for loading process of Data Warehouse metabase, it means: Glossary of Terms, Classifications, Database, of Statistical Units

- Information database that are working within the, confines of statistical system.

It has to be state quite clearly that creation of the system that enables to meet internal and external needs requires more time resources. Workshops concerning conception of the system are running. System model respects general guidelines of project for surveys documentation system. Uniform description of surveys is very important for efficiency of surveys documentation system. Activities concerning surveys description unification, at the some level of details including correlations between surveys should:

- secure 2 forms of descriptions for the some survey

- 1. that covers all phrases of surveys life cycle (planning, execution, data study, analysis, results exhibition)

- 2. that covers meta data (terms, definitions, classifications, groupings)

- secure comparability of surveys description with international standards

- establish rules of statistical surveys descriptions and impose needed power inside public statistics.

Documentation of statistical processes improves data exploitation and assures their high quality.


- 57 -


There is a idea to use DDS (Dispoitif de Documentation Structuree) - system for common data description for metadata system creation.


Statistical Metadata - How far (or close) have we got by Ranta Jaakko, Statistics Finland

During the past few years an increasing amount effort has been put to throw light on the various aspects of statistical metadata. A number of models and even practical applications have been developed in several projects on national and international level. The contents of the metadata these projects have arrived at differ from each other substantially. Moreover we are still facing the growing user complaint of the insufficient metadata in the dissemination of statistics.

The purpose of this paper is to compare the results reached so far from the point of view of the most essential metadata needed for the user to able to understand the information in a statistical presentation e.g. a statistical table. To be able to make such a comparison we should define the user needs. In the paper we look into this problem and state arguments for the definition we adopted.



- 58 -


Session 4: Questionnaire Design and Testing I Mozartsaal Chair: Sharleen Forbes, Statistics New Zealand

Development of Questionnaires - Current Practices of Destatis and Perspectives Hans Joachim Schwamb and Elmar Wein, Federal Statistical Office, Germany

Statistics Germany has improved it's methodology and processes for the development of questionnaires during the last years. Starting from the preconditions of questionnaire development within a decentralised statistical system and current developments, an overview of the practices used for questionnaire development and optimisation will be given. The contribution will treat the most important aspects of a revised manual for the development of questionnaires. Special attention will be given to the development and simple tests of electronic questionnaires. The considerations will be supplemented by short examples for standardised paper-and-pencil questionnaires. At the end the perspectives for further developments will be described.

Contact: [email protected], [email protected]

How People Answer LFS Questions About Economic Inactivity - Main Findings from a Qualitative Study Exploring the Answers given to Labour Force Survey Questions on Economic Inactivity by Daniel Guinea and Peter Betts, Office for National Statistics, United Kingdom

Both nationally and internationally there is a strong interest in the potential supply of labour and in being able to identify a single group within the economically inactive, sometimes called the 'labour reserve'. The UK Labour Market Framework Review, completed in 2002, highlighted a need to look in more detail at issues around labour market attachment. Although the four main questions asked in the UK's Labour Force Survey (LFS) to establish economic inactivity have been in use for many years, they do not allow analysts readily to distinguish between sub-groups of the economically inactive in terms of their potential for becoming active suppliers of labour.

The qualitative study reported in this presentation involved cognitive testing of the four questions to explore respondents' understanding of the questions and how they formulated their answers. The study also explored respondents' understanding of questions about their future intention to work. Twenty purposively sampled re-spondents to the LFS were interviewed.

Findings showed that when respondents were asked if they were looking for work in the previous four weeks, they interpreted this to mean were they 'actively' looking for work.


- 59 -


This implies that those who were 'passively' looking or 'keeping an eye open' answered 'no'.

When those not looking for work were asked if they would like to have a regular paid job at the moment, some answered 'realistically' and some 'idealistically'. The realistic answered in terms of need and ability: the idealistic answered in terms of the ideal dimension implicit in the phrase 'would you like'.

Some respondents interpreted 'at the moment' to mean at some point in the future and gave a positive response to whether they would like to have a regular paid job at the moment. This interpretation meant that they were asked why they were not looking for work and were not given the option of the possible answers to the question on the reason for not wanting work at the moment that additionally include 'retired from paid work' and 'doesn't need employment'.

People in similar situations might be classified differently concerning the main reason they were not looking for work, or did not want work; and the same person, in the same circumstances, could be classified differently from wave to wave.

Finally, in-depth interviewing about economically inactive people's future intentions to work led to the development of five categories on a continuum of how likely people were to work in the future.

Contact: [email protected] and [email protected]

The Quality of a Quality Control Instrument: Quality Aspects of Questionnaire Pretesting by Dirkjan Beukenhorst, Statistics Netherlands

The quality of a survey may be greatly improved by pre-testing the questionnaire before bringing the survey into the field. A small scale qualitative test with real respondents can uncover problems respondents may encounter while answering the questions. These problems may concern comprehension of the question’s meaning, retrieval of the information, choosing the right answer category or formulating the answer. Pretests can improve the quality of both household and establishment surveys.

Examples of pre-testing techniques are cognitive interviewing, observation, behaviour coding, and thinking aloud while filling in a questionnaire or answering a question posed by an interviewer. Depending on the phase of development or the nature of anticipated problems, different testing techniques are applied.

The quality of a pre-test can not simply be measured by subtracting the number of anticipated problems from the number of problems revealed by a certain test.. Several mechanisms operate during such pre-tests, which can elicit artificial problems that


- 60 -


would occur in the field never or only rarely. Test-respondents may be biased and atypical, the interviewing technique may provoke socially desirable answers or stimulate respondents to list minor problems, the interpretation of observations may be prejudiced etc.

This paper will provide guidelines on how to avoid artificial or negligible results while pre-testing questionnaires, and recommendations on how to ensure that most relevant, unanticipated, problems will be found in an efficient way. The guidelines concern among other things the careful selection of respondents, the rigorous execution of the test protocol, detailed recording of the test, interpretation of recordings by different researchers, and the triangulation of results.

contact: [email protected]

Coding Analysis of Cognitive Interviews: Benefits and Drawbacks by Kristen Miller, Beth Canfield and Lisa Moses, National Center for Health Statistics, U.S.A.

While cognitive interviewing is a widely used method for evaluating survey questions, there is little consensus among practitioners regarding the standards or criteria that make for quality cognitive evaluations. This deficiency has raised skepticism regarding the replicability, falsibiability and, ultimately, the validity of cognitive interview findings. The vast majority of cognitive evaluation literature pertains only to interviewing techniques (e.g. pre-scripted vs. emergent probing, concurrent vs. retrospective probing). Very little focus has attended to the actual analysis of cognitive interviews. That is, how are findings from individual cognitive interviews to be used in order to make conclusions regarding the quality of the question?

This paper will describe a National Center for Health Statistics project in which over 70 cognitive interviews were conducted in both English and Spanish and then coded to allow for quantitative analysis of interviews. Codes were based on the question-response model, depicting 1) R’s interpretations of key terms, 2) R’s process for judging, and 3) the presence of response error and response error type. The paper will describe the possible analyses that can be conducted with this coding system. In presenting the findings from this project, it will show how the impact of language, income level of respondent, and interviewer on the question response process was investigated. Finally, the paper will describe the strengths and weaknesses and methodological utility of this analytic technique.



- 61 -


Session 5: Administrative Data Museum Chair: Peter Lohauß, Statistical Office of Berlin, Germany

Quality Control and Quality Assurance for Administrative Data of Federal States and Communities by Official Statistics by Winfried Gruber, Statistical Office of Baden-Württemberg, Germany

In the German system of official statistics the regional decentralized Statistical offices of each state deal with surveys including data processing and presentation no matter if federal or state statistics are concerned. The Statistical Office of the state of Baden-Württemberg processes 226 statistics, most of them being produced with paper questionnaires. However, these are usually questionnaires for individuals/homes or enterprises, whereas the majority of data (in terms of data volume) from public institutions are already collected electronically.

In any case, paper questionnaire or electronic transmission, data need to be checked carefully for their validity or credibility when being processed. For efficiency reasons and also due to a shortage of financial resources, public institutions that are bound to provide statistical data to the Statistical Office are supposed to turn in pre-checked data. By using selected financial, fiscal, and social surveys it is shown in detail that ad-ministrative registers contain statistical inaccuracies and inconsistencies, because purpose, structure, and use of the registers usually do not correspond to the statistical demands. On the other hand, extensive checks, especially on the lower regional level, are not only necessary for statistical purposes, but also for administrative purposes, e.g. for the vertical fiscal distribution or the financial equalization system for communities that is common in Germany. For an efficient use of register data it is therefore necessary to require the admini-stration to carry out prior statistical checks and if necessary corrections without modifying the registers themselves. It is possible to oblige the administration both to perform credibility tests according to statistical rules and to deliver the data electronically. This would require a legal regulation concerning electronic form and statistical plausibility of the data.

Even so, for the time being the ideal case of delivery of electronic and credible data cannot be extended to all administrative parts. But in order to approach this goal, the Baden-Württemberg Statistical Office intends to reach service level agreements with data providers on all federal levels in a cooperative and voluntary way. In this case data providers get feedbacks from the Statistical Office about the type and frequency of errors for any kind of statistics they deliver. It is possible to improve continuously the quality of the data with regular communication and cooperation of the Statistical Office and the authority responsible for the administrative register.


- 62 -


In case of cooperation of the data providers the Statistical Office will inforce electronic data collection and plausibility methods. In this scenario both parties will mutually benefit as far as the effort in time is concerned and at the same time the quality of the data can be assured or even improved. The Statistical Office has established a meta database to estimate this development. By using this database there will be regular update informations about the improvements of data deliveries in the context of progress reports.


Quality Measurement of the Norwegian Labour Force Register by Aslaug Hurlen Foss, Statistics Norway

Labour force statistics were published from the labour force register within the 2001 Census. In earlier censuses, labour force data was collected from a survey together with data for household and living conditions. The Labour Force Register is made from several different registers. The core is the Central Population Register. From this register all persons living in Norway on the 3rd of November 2001 with the age between 16-74 years (at the end of the year) are subtracted. The main registers for making labour force variables are the Central Register of Employers and Employees, the Wage Register, the Tax Return Register and the Business Register.

The quality of the Labour Force Register is measured by comparing it with the Labour Force Survey on both macro and micro levels. By the use of the unique Personal Identity Number (PIN) the Labour Force Survey is merged with the Labour Force Register.

There are three difficulties in doing an analysis like this. For the first the Labour Force Survey and the Register have different time references. The Labour Force Register has week 44 as a reference period and the Labour Force Survey has weeks 40-52 as reference periods. The second problem is that the survey and the register may have different definitions or specifications of variables. The third problem is that errors exist in both data sources and they are different. A logistic regression analysis shows that the different reference periods have little impact on the quality. However the different register sources have different quality. This analysis also shows also that the use of indirect interview in the Labour Force Survey is a source of error.

In the analysis of the measurement errors of the Labour Force Register, the Labour Force Survey is assumed to be the truth; even though it is known that this is not always correct. These analyses show that:

- The measurements error is small for the main numbers on employment.

- The bias is relatively small for employment by gender, counties and industrial classification, (Nace).


- 63 -


- The bias is relatively large for employment among young people (20-24 years old) and for old people (67-74 years old). This is probably caused by difficulties with classification in the register for people with part time employment.

- The bias is relatively large for employment in public/private sector. This is probably caused by the choice of main employment in the Labour Force Register.

- The number of employees per local kind of activity unit differs significantly between the Labour Force Register and the Labour Force Survey. The explanation for this is probably measurement error in the Labour Force Survey.


Quality Measurement of a Register for Structural Business Survey: Application to Ukrainian Data by Michel Grun-Réhomme, Université Paris 2 and Olga A. Vasechko, Research Institute of Statistics, Ukraine Users of statistical business registers want those registers to be of "good quality”. It is therefore important to establish what level of quality is required and to measure the quality of the register.

We know the general quality criteria of Eurostat (relevance, accuracy, timeliness,...) and the quality standards for which register quality can be measured (rate of multiple registration, rate of falsely active units, ...). Moreover, we know the different methods of measuring register quality (control surveys, coverage checks,...).

In this paper, we propose, in the first part, to show in practice, where it's difficult to follow and apply these criteria. In the second part, we define a new quality indicator on the stratification variables. This indicator gives a synthetic measure of the difference between register as sampling frame and survey at the level of stratification variables.

In the third part, we use this indicator to measure the quality of Ukrainian register, on the stratification variables. And we try to explain the results with a logistic model.



- 64 -


Data Takeover for Statisticals Purposes from Administrative Sources in Serbia Need to Provide the Quality by Srdjan Stefanovic, Statistical Office of the Republic of Serbia

The new Law on official statistics in the Republic of Serbia predicts movement of focus from data collecting from reporting unit to data takeover from administrative sources. This is the answer to the recommendations of Eurostat and other statistics of developed countries that the registers and the other administrative databases have to be intensively used in statistical surveys. In this way the new approach in organization of Statistics demands creation of new data collecting methodology or revision of the old ones. Also, it is the opportunity that the new concepts of the data quality controls and procedures in data collecting are included in the new methodology. The paper is a contribution to the study and introducing the management quality in the Statistical Office of the Republic of Serbia who learns his first lessons on this subject.

In theoretical approach, in this paper, one of the possible answers will be given to the question: How to assure data quality in the takeover of the administrative data? Two interrelated methods are in the basis of the answer:

1) The data takeover requests preventive measures and provision of some precondi-tions:

- Using the data from databases for statistical purposes should be defined by laws

- Memorandum of understanding should be determined as well as the protocol of agreement in statistics information interchange system between national Statistics and some Administration

- Detail rules for takeover of data and metadata should be defined

- Authorities of the administrative sources should be drawn into approving their own data quality on the basis of the statistical results.

2) Use of the knowledge and standard procedures in the date analyses, transformations and quality assessments which are necessary in short time acceptance of data takeover for further performance of statistical surveys. The algorithm of series of the activities each ended with assessments of quality (data and procedures) and decision if the level of quality is enough to continue with he next steps will be introduced. For each of the activities data quality measures, as well as the suggested methods for their approval must be defined. The activities are:

- Formal data analyses (control)

- Logical data analyses

- Data transformation and adjustment of the data to their further use

- Connecting data analyses with other data sources (administrative sources, statistical surveys, previous stage in the data base).


- 65 -


For the illustration of the problem in the data takeover from the administrative sources the Statistical business register is chosen as the best example for the short time data quality harmonization of various data sources.


A Study to Analyse Possibilities to Establish a Detailed Fisheries Employment Statistics on the Basis of Existing Statistics and Registers by Anssi Ahvonen et al., Finnish Game and Fisheries Research Institute

Fisheries can be considered as a marginal phenomena in the modern society. However the demand for statistics and other information of fisheries is much greater than the share of the sector measured by common socio-economical parameters. The basic need for fisheries statistics originates from the fish resource management. In the European Union fishery is supported and maintained heavily by the Common Fisheries Policy, such as agriculture by CAP. The control of the policy implementation and policy planning encourages various, and often detailed needs for the subject statistics. One of the recent much emphasized need is fisheries employment statistics that should be considered in more details, and in a more harmonized way. The task for statisticians is challenging, as nationwide employment statistics are seldom enough precise for the needs. The quality of nationwide data may also be a limiting factor when register information is splitted more deeply and diversifiedly than expected in the more common use of the data. On the other hand, the subject surveys of the fisheries sector may give more accurate estimates, but they probably are not properly scaled and harmonized in the society level of the employment.

In the paper we analyse the practical possibilities to combine different data sources and statistics of fisheries employment. A condition for the work is to avoid new surveys. The goal is to build up a system that may satisfy the needs to divide the phenomena by fisheries activity (fishing, aquaculture, fish processing and ancillary industries), by region, by production environment (marine, inland), by type of employment (full-time, part-time), by age and sex of the employees and by same other identifiers. The estimation methods used in combination would be modelling of the dependency of different variables and calibration to most representative margin distributions. The data comes from Finland, where Finnish Game and Fisheries Research Institute is a responsible authority in the field of official statistics of the fishery and hunting.



- 66 -


Reducing Response Burden While Collecting Environmental Data for Statistical and Administrative Purposes by Bernd Becker, Federal Statistical Office, Germany The target to reduce the burden of statistical inquiries on enterprises is on the forefront of the political agenda: The German Federal Minister of Economics and Labour has set up a project group which should facilitate administrative processes including statistics. In particular in the field of statistics, the Statistical Advisory Committee („Statistischer Beirat“) has recommended to increase the use of administrative records in the field of environmental statistics.

Using administrative data for statistical purposes has many advantages. However, the German experiences on the usefulness of administrative data sources for envi-ronmental statistics are mixed: On the one hand, administrative sources of envi-ronmental data provide relevant statistical information on key indicators of envi-ronmental policy. On the other hand, the statistical use of administrative data on environmental issues lacks many quality criteria such as timeliness, accuracy, comparability, coherence and completeness.

Based on these deficiencies, it is advisable to develop a new framework for admin-istrative and statistical data management. Besides the demand to harmonise the needs of administrative and statistical data with regard to terminology, concepts, classifications and periodicity, one has also to extent substantially the right of access of statistical agencies for administrative data. In the following paper, however, procedures are described on how to collect environmental data only once which may serve simultaneously statistical and administrative purposes.


Measuring the Quality of the Economic Activity Code in the Italian SBR by Livia Calabrese and Monica Consalvi, ISTAT, Italy Different methods are available to assess the quality of the statistical business register characters. With reference to the economic activity of small enterprises, this paper compares a classic approach with a new one based on the Clamour project.

For the former data from the 2001 Italian census of industry and services and the annual survey on balance sheets of small businesses are used. For the latter data from a recent annual survey carried out by the Ministry of Finance are made use of. The peculiar features of this source are pointed out. The data are used as an imple-mentation of the Clamour model to describe the activities of the businesses.

Both the approaches are applied for tracking and evaluating the quality of the eco-nomic activity variable in the Italian business register. There are two main results. The quality surveys that the classic approach consists of are significantly less informative


- 67 -


than the innovative method. When assessing the quality of the economic activity variable it is important to take into account the failures of the nomenclature.

The process by which the activity code is attributed in the Italian business register is complex. Some modifications to be brought in it can be prompt by handling the codes constructed according the new method as reference values for the economic activity variable.


Quality Assessment of Structural Business Statistics Preliminary Estimates by Salvatore Filiberti and Mariagrazia Rinaldi, ISTAT, Italy

Use of administrative data for statistical purposes has been introduced into Istat Structural Business Statistics (SBS) production process. In fact, information coming from administrative sources has been used in order to reduce the impact of item and unit non responses for major SBS surveys. Administrative sources have also represented the most important data source in the production of SBS Preliminary Estimates (PE) concerning economic and employment variables, according to the Council Regulation No 58/97. To this end a new methodology has been implemented to combine administrative and statistical sources. Moreover, preliminary data estimation process is also based on use of auxiliary information provided by Business Register.

Quality assessment of a statistical product like SBS PE should take into account the evaluation of the adequacy of both data sources and statistical methodologies used to obtain such estimates. Analyses have been carried out in order to assess the quality issues concerning the sources of data, the administrative and statistical sources combination methodologies and estimation methodology.

Aim of the paper is to focus on quality assessment of SBS PE as a statistical product and show main results. In particular, stress is put on SBS PE coherence with SBS final estimates (ex-post analysis), accuracy, timeliness. Contact: [email protected], [email protected]

Improving the Surveys on Internal and International Migrations and on Foreign Population by Domenico Gabrielli, ISTAT, Italy

The surveys concerned are those on migrations (internal and international) and on socio-demographic characteristics of foreign population. The most important survey is the one for the measurement of internal and international migratory flows, based on the collection of administrative data on changes of residence


- 68 -


between Italian municipalities and between these with other countries, which has been extensively restructured in recent times.

The quality of the data produced has been considerably improved, through, in the first place, improvements in the way of collecting data, especially in those territorial units which showed the greatest faults. Afterwards a continuous system of monitoring of the survey has been set up. The quality checks include the production of a great number of tables to check the consistency of the data. Improvements in the analysis of the data were introduced as well, and now a greater number of ad hoc indicators on residential mobility is available. Individual data can be send by municipalities also through an electronic model in the frame of a system of collecting demographic data named ISI-ISTATEL.

In1993, as a consequence of the evolution in migratory trends, a new survey was launched, the one on resident foreign citizens; it is based on population registers, as the one on changes of residence but flow (and stock) data are aggregated at municipal level. As an improvement in the collection of data an electronic survey model has been also introduced, in order, in the first place to strengthen and consolidate the data quality improvement process, thanks to consistency automatic checks.

Together with surveys based on population registers, processing of data on foreign citizens' residence permits is carried out, as part of the plan to make use of admin-istrative data produced also by other bodies, in this case by the Ministry of the Interiors. Thanks to Istat's elaboration of the data, statistically significant information are produced, starting with the amount of documents still in course, net of expired and duplicated ones; to this purpose also the names and surnames of the permit holders are transmitted in the individual records but codified in such a way that foreign citizens cannot be identified. Also, consistency of the main variables is checked and eventually corrected (in a deterministic way).

In order to improve the quality of the data from population registers a linkage has been foreseen with the residence permits archives, so that the expiry date of the permit can be reported in all the records of the registers. That can be useful, for example, for preventing foreigners who are no longer in Italy to remain registered; in fact, this system was conceived for being implemented before the Census, in order to arrive at Census survey with population registers not inflated by fake residents, but unfortunately it wasn’t adopted by a significant number of Municipalities. Therefore efforts have to be made in order to improve co-operation with the Interiors Ministry so that the linkage between the population registers and the residence permits archives can be fully implemented. Contact: [email protected]


- 69 -


The Survey on Italian Population Resident Abroad by A. Silvestrini and G. Cariani, ISTAT, Italy

Data on Italian population living abroad are based on two different sources. Every municipality has a civil register where people who transfer their permanent residence abroad are registered. Moreover the Minister of interiors has a national register that has unified every local one.

The second source of data is Consular Registers. The law about Census of Italian living abroad has been changed tow years ago: it has been stated that the Census of 2001 will be substituted by the use of data coming out by Consular Registers, as they result on the 31s of march 2003, after a statistical treatment. This means the confrontation of data with other sources and in particular with the national civil register. Moreover, it has been planed to confront data with results of National Census held around the year 2000 in the major countries where Italian lives, according to the two Italian sources. The paper will explain the kind of controls that have been planed to validate data. Contact: [email protected]

The New Data Collection System on Population Statistics by A. Silvestrini, R. Bartoli, F. Tarantola and M. Venturi, ISTAT, Italy

Italian demographic data are collected from 8100 municipalities and based on civil registers. The system of data collection is mix: municipalities have the possibility to fill and electronic or a paper questionnaire or to send data using a particular software connected with the electronic civil register. The National Institute for Statistics collects individual data on events (births, deaths, internal and international migrations) and stock data for resident population. Moreover there are monthly questionnaires that summarise the total flows of events by present and by resident population.

These latest questionnaires produce the number of resident population for every municipality at the end of every month and are an important tool to control the number of individual questionnaires.

In each region ISTAT has a bureau that is involved in data collection at local level. This system of data collection exists since some years, but is now totally renewed. This paper aim to present the new system focussing in particular on the instruments for data quality control. These controls are based principally on a controlled registration of questionnaires and on reports that compare data of different sources, times and territorial distribution.



- 70 -


Automatic Evaluation Procedures for Elementary Administrative Data. The Case of Missing Values and Outliers in Non Pension Cash Benefits Archives by Vincenzo Spinelli and Massimiliano Tancioni, ISTAT, Italy

This paper contains a detailed description of some of the automatic procedures implemented for the acquisition and conditioning of administrative elementary data from the DM10 archive of INPS (Italy’s National Social Security Institute). This archive contains monthly information on the universe of the expenditure anticipated by private sector employers for Non Pension Cash Benefits (NPCB). The attention is focused on two fundamental aspects of the process:

i. logical-longitudinal identification and estimation of missing data on expenditure at the firm level;

ii. nonparametric identification and correction of potential outliers in the employee data at the firm level.

The first concerns the reconstruction of the time series data and their consolidation on an annual basis. Missing values identification and imputation is operated via automatic procedures employing both logical reasoning and longitudinal information. By the fact that the events by which the expenditure originates have scarce persistence, data are highly volatile over time. Hence, procedures relying on information defined exclusively in the time domain may result unsatisfying. In fact, they cannot consider elements - other than time memory - that are logically decisive in the reconstruction of observed processes. In order to account for these potential pitfalls, and assuming information from other sources to be unavailable, we have experimented the introduction of a sequence of logical constraints operating as preconditions for the use of longitudinal techniques. The satisfaction of each logical stage defines a reduction of the starting set. The last condition being satisfied, we obtain the set of the elementary data we deem to include missing values. Longitudinal identification and estimation techniques for missing values are then operated on this reduced set.

The second aspect concerns the identification and correction of potential outliers in the employee data. The use of employment data is compulsory to the production of statistics broken down by dimensional classes. However, the available source is not directly related to our set of interest, and data comparability is problematic if a dimension-expenditure proportionality assumption is adopted as an evaluation criterion.

Instead of espousing a traditional confidence-based approach, we have developed an iterative selection procedure for outliers based on almost neutral optimality (in this framework, termination) conditions. The substantial neutrality of the selection relies on the use of a generally accepted judgmental criterion for the termination conditions,


- 71 -


which evaluation is iteratively derived from the observables. This approach, in our view, reduces the role of the subjective choices in the identification process, since the key aspect is neither a distributional nor a maximum dispersion prior. We assume NPCB information to be complete and exact, in other terms, to be the reference validation information for the employees data. We also assume the existence of a probabilistic relationship between NPCB expenditure and number of employee, thus the existence of a distribution that is identifiable in its moments. These assumptions allow the use of the average individual expenditure as the reference variable and, once the outliers are identified, the correction of the data through the ratio between actual expenditure and average expenditure per employee.


Session 6: Survey and Census Design I Forstersaal Chair: Irena Krizman, National Statistical Office, Slovenia

Improving Timeliness and Quality in Short-Term Statistics: An Application to Italian Building Permits by Alleva G., University of Rome "La Sapienza" , Italy; Fabio Bacchini, P.D. Falorsi and R. Iannaccone, ISTAT, Italy

After the establishment of the monetary union, growing attention has been paid by economic Institutions and policy makers to the availability, comparability and time-liness of European short-term business statistics. In May 1998 was published the Council Regulation n.1165/98 for short term statistics (henceforth STS Regulation) For statistics on industry, construction, trade and tourism, the STS regulation foresees which indicators to produce and their timeliness. Henceforth the National Statistics Institute (NSI) started a revision of their statistics according to the new standards. For the Italian NSI this has implied the need of new data collection method for short-term statistics in the construction sector. Particularly, information on building permits are required at 90 days from the reference period.

The previous data collection method based on a monthly census of the Italian mu-nicipalities could not make possible to elaborate the index with the delay required. At first reason is that the survey was organized in a two step level (from municipalities to provincial level unit and then to Istat) to collect information. Moreover it was difficult to undertake a reminder activity for all the municipalities. At first improvement has been reached selecting a sample of 814 municipalities and collecting information without the intermediate step of provincial level. Moreover for the other 7.286 municipalities the survey continues as monthly census. This design implies that for the month t at time t + 90 days two different sources of information are available: respondent in the sample and respondent not included in the sample.


- 72 -


An estimation method (Alleva and Falorsi, 2003) is implemented to use all data available. The proposal is based on two-phase sampling theory and on the hypothesis that the response mechanism is the result of random process whose parameters can be suitably estimated. After illustration of the proposal methodology we show that, for the building permits case, the proposed estimation performs better than that based only on sample information. Moreover this methodology seems to be easy to generalized to the NSI context where preliminary information on microdata is always available. Contact: [email protected]

Optimizing Stratified Random Samples with Multiple Objectives by Volker Bosch and Raimund Wildner, GfK AG, Germany

Stratified random samples are frequently used in survey research. With given sample size or given cost it is essential to minimize the error variance (i.e., sampling error) of the estimated population statistics in order to maximize efficiency. This is achieved by determining the optimum allocation of observations to the strata of the sample. For this optimization problem, a variety of well known solutions currently exist that minimize the variance of the population estimate(s). In real life, however, a survey's objective is never just the sole estimation of the population mean of a single variable. In every practical case, means are to be calculated for a multitude of variables. Moreover, mean estimates are not only important for the total population. Rather, estimates restricted to subsets of the population (i.e., segments) are to be calculated in addition. However, available optimization procedures only partially address these issues.

To overcome these shortcomings, the authors developed a method for optimum allocation that explicitly considers the estimation of any number of mean values as defined by variable and segment (Bosch & Wildner, 2003, Comm. in Stat.: T&M, 32(10)). The proposed method is a generalization and unification of formulae provided by Neyman (1934, J. Amer. Statist. Assoc.,), Tschuprow (1923, Metron), Stuart (1954, J. Roy. Stat. Soc.), and Yates (1960, Sampling Methods for Censuses and Surveys).

An example from business-to-business market research demonstrates the usefulness of the proposed method. Here, segments are commonly defined according to company size or sector of activity. Variables of interest may be expenses (e.g., investments for telecommunication) and choice behaviour (e.g., provider chosen for telecommunication services). In terms of variance, these two types of variables behave differently across segments. Applying the proposed optimization method yielded a sample where each estimator of interest was sufficiently precise whereas any other method mentioned above failed to produce sufficient precision for the entire set of estimates.

In conclusion, the incorporation of all estimates of interest into the optimization procedure provides the researcher with a substantial improvement of the estimates’ joint precision. As is obvious from the example, error variance is not just shifted from


- 73 -


one estimate to another as in a zero-sum game to find some kind of compromise. Rather, considerable improvements of the precision of particular estimates may result from only small losses of other estimates’ precision. This fact accounts for the high effectiveness of the procedure. Contact: [email protected]

An Incremental 2-Stage Sampling Plan for a Flemish Hepatitis Prevalence Study: Accumulation of Respondents over Successive Waves by Peter Slock, Camille Vanderhoeft, Statistics Belgium and Sophie Quoilin, Scientific Institute of Public Health, Belgium

Often when designing a sample survey the main objective is to get a representative sample. A simple but widespread technique is stratification with proportional allo-cation, which allows obtaining representative samples in a strict mathematical sense.

When several stratification criteria are involved − say A, B, C − problems may arise if the complete cross-classification A×B×C is considered: small or empty strata, too many strata … and statistical consequences in estimation. Moreover, non-response can heavily distort the intended sample design. Statistics Belgium (Statbel) tries to remedy non-response problems by incorporating notions from quota sampling − some units that explicitly have expressed their will to participate are refused − and working with so-called replacement units − to be contacted only if some quota are not met. The Scientific Institute of Public Health (IPH), which closely collaborates with Statbel for sampling purposes, applies quite different but probabilistic sampling schemes. One such scheme, recently applied for a survey on hepatitis prevalence in Flanders, will be examined rigorously in this paper. Through simulations, we further compare this approach with the above-mentioned ad hoc sampling designs applied for some household and individual surveys at Statbel.

IPH’s approach is as follows. Only one criterion A is used for classical stratification, and proportional allocation is applied to the A-stratification. Within each A-stratum the remaining criteria (B and C) are used to define PSUs (groups of individuals), rather then strata. The next step is to select PSUs, e.g. by PPS (probability proportional to size, with replacement) sampling. Sorting of PSUs and systematic sampling is used to enhance representativity. Finally, each draw of a PSU gives rise to the sampling (e.g. by simple random sampling) of a fixed number G of SSUs (individuals), called a “cluster”. So far the initial sample was considered, the size of which is determined taking some global response rate into account.

Next, instead of sending an advance letter to all selected SSUs at once, letters are sent in successive waves. More precisely, if G = W × C, where W is the number of waves, then


- 74 -


C is the number of selected SSUs per wave and per cluster which will be sent an invitation. In the first wave, exactly C letters are sent in each cluster; in the next wave(s) either C or no letter at all will be sent per cluster. If for a given cluster in a specific wave at least one SSU responds positively, the cluster is eliminated and its remaining SSUs will not be sent letters in further waves; if no positive response is recorded for a given cluster in a specific wave, another C SSUs in the cluster will be sent a letter in the next wave. The procedure is repeated until a pre-specified number of positive respondents are found, or until the maximum number W of waves is reached.

It is shown that the above procedure allows careful planning of the survey, provided realistic assumptions can be made about response patterns. The final respondent sample size and its representativity can be controlled very well, while time aspects in planning the survey can optimally be taken into account.

Contact: [email protected], [email protected], [email protected]

Use of Tax Data for Sample Design Under Confidentiality Restrictions

by Juana Porras, Jorge Saralegui-Gil, National Institute of Statistics (INE), Spain.

It is well known that quality of households surveys design can be substantially improved when ancillary information from administrative registers related to the same target population is available for use at the different stages of the project, from sampling units definition and stratification to the estimation of final results. Such use of administrative data for statistical purposes is nevertheless conditioned by several characteristics of the external source , as units definitions, scope, concepts and variables, internal quality, reference periods, and others which are permanently object of concern for official statisticians . The modality of access to the source is an aspect extremely important too, when , as it may be the case, the source is regulated by confidentiality rules which prevent the access to microdata , being consequently available in aggregates only, what may not be satisfactory for the intended statistical use. This scenario is particularly common in the field of social surveys which collect monetary variables of individuals, highly correlated to data stored within Tax Administration files.

The paper describes procedures implemented in the Survey of Income and Wealth of Households, sponsored by the Bank of Spain and designed by Ine. The method makes an efficient use for sample design of microdata information from the Personal Income Tax and Tax on Wealth files kept by the Tax Agency , under strict confidentiality rules. Through a series of data flows from and to the tax Agency, Ine never receives microdata, but its survey statisticians are in the position to benefit from the outputs produced within the Tax Agency premises by carrying out instructions and algorithms elaborated by Ine. Preliminary studies of the structure of the information made it possible to define strata boundaries, providing a base to over-represent the social groups with the highest position in the income and wealth distribution, as well as to approximate the optimal


- 75 -


allocation on the remaining sub-populations. Thus Ine samplers know that affluent households are in the sample, but not who they are. In subsequent steps, additional protocols are used to calculate non response weights to be sent back to Ine.

Contact: [email protected],[email protected]

Improving Quality by Integrating Statistical Surveys with Adminis-trative Surveys: Industry and Services Census in Italy by Corrado Abbate, ISTAT, Italy

The quality of a census is given, first of all, by its capacity to be exhaustive, providing a precise quantification of the active units in a territory. The coverage area survey for the 1991 Census of Italian industry and services, and the analysis in terms of the year companies were established highlighted the presence of an under-coverage phenomenon. The growth of service companies, which often do not have a fixed office and thus cannot be identified easily on the territory, and more generally, the growing number of very small companies with one or two employees, increase the difficulty of running a traditional census.

The VIII General Industry and Services Census (CIS) in 2001 was thus designed and carried out using a mixed technique that combines the traditional “door-to-door” technique with the technique per list, which is made possible by the availability of centralized archives on the economic units. The units identified through the collection phase on territory were matched with the units in the available administrative and statistical archives: active companies (ASIA), public administration bodies and non-profit institutions. In order to distinguish between the units that make up differences due to over-coverage from those that make up under-coverage differences, when comparing CIS-ASIA, a latent classes model was used where the latent variable can have a value of 1 if the unit is active, and 0 if the unit has non-activity indicators as at 22nd October 2001. The variables observed were chosen from the data made available by the census and that contained in administrative archives.

In short, after having eliminated the errors due to wrong linkage between the two sources (because of companies having two or more tax numbers) the number of companies active at the date of the census was estimated to be about 4,050,000 including 650,000 companies that were not surveyed on the territory, but that are to be found in administrative archives, without including another 300,000 companies considered as active in the archives but not identified on the territory nor defined as active in terms of the model. These results confirm those that came out of previous analyses on the under-coverage phenomenon in the construction and service sector, in individual companies of


- 76 -


autonomous workers and freelance professionals, in very small companies and centres of larger sizes. In small municipalities, where the census operators have maximum knowledge of the territory, only fractional over-coverage errors were to be found.

The census results were therefore not corrected by choosing one source as being better than another source, but on the basis of the sources’ integration principle that makes it possible to limit estimation errors to a minimum, maintaining the largest degree possible of territorial detail on census data. The final result of the integration of data surveyed in the VIII general industry and services Census in 2001, marks an important step taken by Istat in improving the quality of statistical information.


The Quality of Official Mortality Statistics in Tunisia by Sofiane Bouhdiba, University of Tunis, Tunisia

The Arab Conference on the Implementation of the ICPD held in Beirut in September 1998 recommended, among its major future actions, to promote statistical and management informations systems to make them compatible with the national needs, and to upgrade local capacities in collecting, processing and analysing data for the purpose of monitoring and evaluating.

In Tunisia, although the government has made great efforts to improve the official statistics quality, sharp problems still exist. I would like, in my paper, to focus on one of the main problems we have today in the field of tunisian official statistics : the quality of the official statistics concerning mortality. For example, we still do not know what are the main causes of death, we have not precise datas concerning infant mortality, there are no official statistics on the link between mortality and social level, and so on. More detailed mortality datas may require a more interdisciplinary approach, involving scientists (medicals, demographers, sociologists, ethnologists,...), policy makers, community leaders, NGO, ... I am convinced that such measures are crucial for effective management and decision making, especially at the level of health programs. I propose, in my paper, to examine the mortality data quality (age confusions, validity of responses, causes of death, gender datas,...) and try to propose some solutions to improve the situation.

For that purpose, my paper is divided into three parts : first, I will make a flash-back to see how was the situation a few decades ago, and I will compare it to the today situation, so that we can see the difference.

Then, in the second part, I will explain why we still have a rather bad quality of official mortality statistics : are the problems at a social level, or a cultural one ?


- 77 -


In the last part of the paper, which is the most important one, I will discuss the methods used to correct the official mortality datas, by eliminating or at least reducing errors. I will propose, too, some recommendation in order to improve the official statistics.


The Statistical Units and the Quality of Business Statistics by Lise Dervieux, INSEE, France

The assessment of quality of a survey should concentrate first on the units surveyed and insure they fit in with the goal of that survey. The statistical units are not a goal in themselves. They are a conceptual as well as operational tool that should enable to collect and compile statistics, and to make progress towards establishing links between data of individual agents and macroeconomic analysis. Nevertheless their definition and implementation must reach a compromise between concept and practicability.

In the case of business statistics, the concept is looking for economic relevance. The enterprise, as defined at the European level, is meant to have a complete set of production factors and of internal functions for its production process.

When you look more closely at the evoluting structures of large enterprise groups, it is not so obvious that each unit of the group fulfills these conditions : in order to better follow their performances, their costs or to deal with the globalization of their activities, the businesses are very creative in conceiving juridical structures that lead to the multiplication of internal flows inside the group and, taken separately, are not accurate : indeed some legal units don’t have a complete set of production factors or an access to the market because they work only for other units of the group. Therefore they shouldn’t be considered economically relevant and should be grouped with other units in order to get consolidated figures.

The delineation of relevant statistical units means to follow the evolutions of the structures of businesses and their consequences on the compilation of statistics. It implies to be able to know the operating structure of a business and to identify the data available from the accounting records. Therefore it means a good comprehension of the businesses which are surveyed. Available data have to be found that are the most able to fulfill the goal of economic relevance, statisticians then have to look for the practicability of the concept.

Taking into account these structures inside statistical systems may have some considerable consequences for the quality of economic statistics. As such units have an important weight inside their industry, the change of unit may have an impact even at the level of national accounts.


- 78 -


The paper will focus mainly on one important case showing how the change of internal structure inside a group can have a strong impact on the quality of data and their relevance. It will also underline the difficulties in obtaining the expected data from the businesses and to incorporate the collected data based on the newly obtained structures into existing survey processes. Other cases will also be mentioned more briefly. Contact: [email protected]

Dual-System Estimation of Census Under-Coverage in Presence of Record Linkage Errors by Marco Fortini, ISTAT, Italy

The capture recapture model is a standard method, originally developed in the wildlife biology, useful in estimating the unknown amount of a population by means of at least two independent enumerations of its component units, where each of the enumerations is subject to undercount. When applied to human population studies the method is also known as dual-system estimation and, in many countries, it is assumed as the reference method for census coverage evaluation. The model requires several assumptions that, when not followed, can cause severe bias in the estimates. Among these hypotheses it is essential the one which requires the exact identification of the number of times that each unit is enumerated during the different capture occurrences. Unfortunately in real cases the identification procedures can produce an incorrect recognition of the units identity in one or more survey occasions. While these problems can hardly arise in animal abundance estimation studies, where the loss of the labels used to count the number of times a unit has been captured is a rare event, they take up more frequently in human population studies, due to the errors arising in the personal identification variables that are used as keys to link the subjects among archives or survey occasions. Even though a perfect recognition procedure cannot be implemented, a great effort has to be made to reduce the incidence of errors in data given that the more they are, the higher is the expected bias of the estimators. In human population studies, that will be considered here in the context of census coverage estimation, the recognition of the enumeration state of the units through the survey occurrences is commonly carried out by means of record linkage techniques, applied to two or more archives via common identifiers. With reference to studies based on only two captures, the linkage errors can produce two types of errors, namely: a false match of different people which are erroneously considered as the same person counted in both the occasions, a failed match of two records regarding the same person that are so considered as two people each one enumerated only in one of the two survey occurrences. Of course the assortment of the types of errors becomes more and more complicated when the number of captures increases. In this paper a Bayesian approach to the capture recapture method is proposed in order


- 79 -


to correct the estimates for the presence of linkage error in data. Since only capture recapture with two sources will be considered, no extra information is available for a direct estimation of the linkage error from the data. For this reason, information about the linkage error has to be conveyed through a proper prior distribution. After an explanation of the method, an example based on the Italian census post-enumeration survey data will be given. Contact: [email protected]

Integrating the 2001 Population Census and Post-Enumeration Survey Data: The Italian Experience by Alessandra Nuccitelli, ISTAT, Italy

Approximately one month after the 2001 Italian Population Census, the Post-Enu-meration Survey (PES) was undertaken in an attempt to re-enumerate all the house-holds and individuals living in a two-stage sample of 1122 Enumeration Areas (EAs). The data collected in the Census in the same selected EAs are compared with those collected in the PES and used to estimate the quality of the first enumeration using the capture/recapture methodology. For obtaining ‘good’ under-coverage estimates, it is important to establish with the minimum matching error the statistical units enumerated in both the Census and the PES. Since a few matching errors may be of critical importance, the whole process of comparison between the data was designed to be highly accurate.

Exact (i.e. deterministic) matching based on the common variables is not sufficient because of errors in both the sets of data, caused by misunderstandings, incorrect recording, the time gap between the Census and the PES; besides, a full hand matching is time consuming and costly. So, the matching process was designed to be a largely automated process involving record linkage (or probabilistic matching). In the last few years a Bayesian methodology and related software for record linkage have been developed in Italy at the National Statistical Institute primarily to support census coverage evaluation efforts.

The Bayesian approach for record linkage enables to compute: a) the probability that each pair of statistical units is a match, conditionally on the observed data; b) the probability of a set of matches, conditionally on the observed data. In this approach a matrix, which indicates the pattern of matches in the two sets of data, is considered as the parameter of interest. The analytic use of the posterior distribution for this matrix is practically impossible, but a sample from the posterior distribution can be generated by using a Metropolis-Hastings algorithm. Then, some inferential summaries can be used as point estimates of the parameter of interest.


- 80 -


The proposed strategy for integrating the Census and PES data uses a combination of automated and clerical matching. In order to achieve more accurate results, a pre-processing of data is performed. Since the Census and PES records are matched sequentially for each EA, as a preliminary geographic coding errors have to be detected. Moreover, free-form information is standardised; the individuals’ first names and surnames are transformed according to the Italian taxpayer’s code algorithm; the addresses are standardised by using a specially made dictionary of all the streets in the sampled EAs.

The main key stages of the matching process are as follows: 1) the households are automatically matched using exact and probabilistic matching; 2) the individuals within the matched households are matched using exact and clerical matching; 3) the individuals who remain unmatched are automatically matched using exact and probabilistic matching; 4) the individuals who remain unmatched are clerically reviewed.

In order to make the matching operations easier and more accurate, suitable user interfaces were designed. Some preliminary results based on the up to now linked data are presented. Contact: [email protected]

Evaluation of the Survey Swedish National and International Road Goods Transport by Per Anders Paulson and Johan Eriksson, Statistics Sweden Commissioned by the Swedish Institute for Transport and Communications Analysis (SIKA), the government agency responsible for official transport statistics, Statistics Sweden earlier carried out two separate sample surveys of goods transport with Swedish lorries, one for national and one for international transport. In year 2000 the two surveys were merged into one, "Swedish national and international road goods transports" (Inrikes och utrikes trafik med svenska lastbilar). Now, after a couple of rounds for the new survey, its design has been scrutinized for possible re - design. This report presents results from a study based on data and experiences from the 2000 and 2001 surveys.

A main goal was to enhance the accuracy of the statistics on international transports. The chief contribution to that effect was a so called "Blue card study". It concerned improvement of the information about "lorries likely to carry out international transports". The study comprised enterprises with more than 15 permits for international lorry transports, so called "blue cards". They were asked to give, for each one of their lorries, an estimate of how much the lorry will be used in international haulages during 2003. Chief findings from the study were : (i) This kind of information


- 81 -


is expected to yield substantial accuracy gains for the statistics on international transports. (ii) The enterprises cooperated willingly to provide this non - compulsory extra information.

It was, of course, also desirable to improve the accuracy of the statistics on national transports. An issue of particular interest in that context was if Statistics Sweden's newly (= early in the year 2002) completed "Mileage database" (Körsträckedatabasen) can provide valuable auxiliary information. This turned out to be the case.

Based on findings from the Blue card study and the Mileage database a new design for stratification and sample allocation was worked out, and set into operation in the 2003 survey. Evaluations indicate that in an over - all perspective the new design will lead to considerable accuracy improvements of the statistics on international as well as national transports. Contact: [email protected]

Session 7: Quality Assessment Leibnizsaal Chair: Willem de Vries, UN Statistics Division

Using Self Assessments for Data Quality Management – DESAP Ex-periences by Johanna Laiho, Statistics Finland and Anja Nimmergut, Federal Statistical Office Germany

Self assessments rank among the most important tools that could be used for a critical evaluation of statistical processes and products. Compared to other assessment tools (like audits or quality reports), they offer the advantage of a relatively low burden and cost, they are a systematic but in part subjective measure of statistical products and processes. Furthermore, self assessments are particularly useful in the context of the creation of a quality culture, as improvements are based on the intrinsic motivation of subject matter area specialists (survey managers) instead of external control.

DESAP is the first generic checklist for a simple self assessment programme for survey managers in the ESS and goes back to recommendation no. 15 of the Leadership Group (LEG) on Quality. It provides a standardised European tool for survey managers for both assessing the quality of their statistics and considering improvement measures. It is fully compliant with the ESS quality criteria, comprises the main aspects relevant to the quality of statistical data. The checklist applies to individual statistics collecting micro data. It is typically carried out by survey managers working with sampling surveys or censuses. It aims to cover issues relevant to surveys using primary data collection, auxiliary information or a combination of both.


- 82 -


The DESAP checklist has been developed on the basis of existing literature, material from different NSIs and from a survey carried out among 25 NSIs. Building on that, a prototype questionnaire and user instructions were developed and tested in a pilot study including 20 different statistics from the NSIs participating in DESAP. The evaluation was accomplished in a systematic and comparable way and the analysis, judgement, and documentation of the test results constituted the main input for the final revision of the checklist.

The DESAP checklist is structured in a process oriented way (in seven process steps and 24 elements) in order to enable the survey managers (and other users of the checklist) to relate the quality dimensions and aims of quality measures directly to specific steps during the compilation of a statistic. A sophisticated routing tailors the data collection section to specific statistics to arrange it clearly and reduce the time of completion. The checklist is composed of (1) assessment questions providing a simple assessment of the statistics with regard to the ESS quality criteria, (2) improvement / documentation questions stimulating improvements in the production process and (3) open questions to motivate the survey managers themselves to identify and prioritise future development projects, for example.

The DESAP checklist is available since October 2003. This paper introduces the concept and summarises the experiences made in the pilot study carried out for the evaluation of the checklist. The paper aims to give motivation for all European NSIs to implement the DESAP as a harmonised and standard tool for quality evaluation in the production of statistics. It emphasises the benefits of a harmonised and tailored quality assessment approach. The potentiality and usability of DESAP for systematic quality improvement within the NSIs and across the ESS is also discussed in this paper. Contact: [email protected], [email protected]

Self-Assessments in Total Quality Management – A Diagnosis Tool for the Organisational Development at the Federal Statistical Office by Sebastian Czajka, Federal Statistical Office, Germany

TQM and self-assessments at the Federal Statistical Office (FSO)

In 1999 the FSO started its TQM initiative called "Qualitätsoffensive Statistik". The initiative is based on the EFQM Model for Excellence and intends to improve efficiency, product quality, user satisfaction and staff satisfaction. In 1999, at the beginning of the TQM initiative, the FSO analysed in a short self-assessment its main strengths and weaknesses. During the last years the FSO has implemented numerous improvement activities, e. g. corporate image, strategic targets, process analyses, cost accounting, controlling and personnel development concept.


- 83 -


In 2003, after these activities, the FSO has analysed its current strengths and weak-nesses by a self-assessment according to the EFQM Excellence Model.

The targets and concept of the self-assessment

The targets of the self-assessment are:

1. evaluating the strengths and weaknesses, measured by the EFQM Model scale,

2. developing proposals to continue the improvement process and

3. transforming the proposals into new activities. To implement the self-assessment, the FSO has chosen the workshop method sup-ported by an external consultant. The concept and the operation of the self-assessment is described by the following steps:

- setting up a working group (step 1),

- training the workshop team (step 2),

- preparing the workshop, collecting information (step 3),

- implementing the self-assessment in a workshop (step 4) and

- integrating the self-assessment results into the cycle of planning and controlling (step 5).

The self-assessment was carried out in September 2003.

Integrating the self-assessment into the FSO cycle of planning and controlling

To transform the self-assessment proposals into new activities it was necessary to integrate the results of the self-assessment into the cycle of planning and controlling. The cycle of planning and controlling of the FSO contains four instruments: strategic planning workshop, annual goal agreements, controlling and annual reporting.

The results of the self-assessments were part of the strategic planning workshop in October 2003. In the workshop the results were presented and after discussions some new activities for the continuation of the improvement process were adopted in a binding manner.

Results

The results of the self-assessments have shown that in comparison with 1999 the FSO has increased its quality level. Moreover the working group has identified many potentials for further improvement activities.


- 84 -


In the strategic planning workshop, activities for the most important potentials to enhance the organisational quality were adopted in a binding manner.


Self Evaluation Form as a Tool for Continuous Quality Improvement Work at OECD by Lars Thygesen, OECD, Paris

The OECD Quality Framework was set up in 2003. It sets standards for the work with quality of new as well as existing statistics.

OECD views quality of statistical products in terms of eight dimensions: relevance; accuracy; credibility; timeliness; punctuality; accessibility; interpretability; and coherence. An additional factor is that of cost-efficiency. Quality is seen as a result of the production processes. Consequently, the essence of the quality framework is quality guidelines for different phases of the statistical production process.

In order to secure the quality of statistical products already in production, Quality Reviews have to be carried out on a regular basis. A distinctive new tool in theses reviews is a self evaluation questionnaire to be filled in by the officials in charge of the activity. The use of the questionnaire is voluntary. It should help statisticians to compile a summary of quality problems and a prioritised list of possible improvements to be discussed with users and other directorates of the OECD.

This paper focuses on the experience with the self evaluation forms in the first 8 reviews made in 2003. It discusses the usefulness of the concept, as well as its practical implementation: Does the questionnaire help the author in uncovering of the most important quality problems? Is it conducive to communication and discussion with internal and external users of statistics? Which types of questions should be asked, and how many? And should the questionnaire be mandatory or voluntary?


Quality Audits at Statistics Sweden by Gunlög Eiderbrant-Nilsson, Statistics Sweden

In November 2002 Statistics Sweden launched a Statistical Quality Audit Activity. Over a period of five years most of the agency’s approximately200 statistical products are to be audited in a standardized way.

The audit procedure consists of three main steps:


- 85 -


1) the staff of the audited product fills out a self-assessment form containing questions on e.g. management and planning, staff competency and relations to users and customers as well as on survey design, data collection, data processing, dissemination and other processes.

2) a team of three persons visits the product during one week (five working days), studying documentation, reports and publications associated with the product and discussing the product with the staff. The team members’ combined expertise covers statistical methods, statistics production and software systems. The evaluation results in an Audit Report, which describes different aspects of the production, identifies good examples and ends with a number of improvement proposals.

3) the product staff is responsible for taking measures according to the proposals from the team.

The audit work is organized by a small secretariat responsible for recruiting and training auditors, composing teams, scheduling audits, following-up of measures taken and compiling and presenting the good examples found.

So far 36 audits have been carried out, resulting in some 200 amendment proposals. This paper will describe the auditing process in more detail and present some of the major findings.



- 86 -


Session 8: Backstage Presenting Statistical Data

Chair: Len Cook, Office for National Statistics, United Kingdom

On Communicating Quality of Sample Estimates in Media Releases by Beat Hulliger, Swiss Federal Statistical Office

Media releases are the shortest statistical informations that are issued by statistical agencies. Necessarily they have to concentrate on the salient features. However, the quality of sample based estimates should not be completely omitted. Information on quality is essential for the user to make appropriate use of the statistical information. Thus one may postulate that whatever the statistical information that is given to the public a standard proportion, for example 10%, should be devoted to the information on its quality. This is by far not the case in media releases. Some examples of media releases show that the quality of the statistical information is rarely addressed.

We propose a short list of questions on a sample survey that even in a media release should be answered. Who commissioned the survey? What is the underlying population? Is the sample random or not? What is the net sample size? How and when has the information been collected? How much non-response has there been? Is there anything else the reader should know to be able to use the statistical infor-mation correctly (e.g. known bias, definition changes)?

The variability of estimators should be communicated for key figures in the text, for example by giving an estimate of the standard deviation in parenthesis. This should indicate to the user that there is uncertainty in the statistical information. The in-formation on variability must be part of the main text, i.e. must not be set aside in boxes and additional information. Of course the interpretations given in the media release should be statistically sound and uncertainty, for example in the increase of retail trade, should be reflected in the wording.

When displaying results in tables the user should get intuitive information on the quality of the estimates. We propose a simple system of three classes of quality of estimates: satisfactory, limited, or poor. For positive variables the classes may be based on the coefficient of variation (cv). The classes might then be satisfactory if cv ( 5% limited if 5% < cv ( 15% poor if cv > 15%.


- 87 -


Of course the limits are open for discussion and should be chosen in accordance with the problem and with needs of the users.

For proportions we propose to base the limits of the classes on a normalized standard error � EMBED Equation.3 , where s is the estimated standard deviation of the proportion p. Then the limits of the classes can be established by replacing the coefficient of variation by d in the inequalities above. Examples of tables show that the distinction into three classes can be communicated through parenthesis or stars, which occupy little space in a media release. Contact: [email protected]

Customer-Driven Data Suppression Decisions by Rich Allen, National Agricultural Statistics Service, U.S.A.

Data confidentiality and suppression decisions are often reduced to only a mathe-matical process of preventing the possibility of identifying a specific individual or operation from a data summary. However, more useful data summaries might be created by considering data users interests and applications and adjusting summaries to better serve those needs.

This paper will highlight the use of alternative size groups, geographic boundaries, and detailed classifications to generate more consistent published time series data sets. It will also discuss the use of statistical defensibility as a key suppression element. The use of informed consent to maximize the amount of summary data released will also be covered.

Most examples will be from sample surveys. However, decisions related to censuses of small populations and broader economic censuses will be included. This paper is a follow-on to the „Customer-Driven Quality, Revisited“ presentation at the International Conference on Quality in Official Statistics in Stockholm 2001.


A Brave New World by Sónia Patrícia F. C. B. Quaresma Gonçalves, National Statistical Institute, Portugal

Every time a NSI launches a new survey, several services have to be prepared to deal with it. In particular its databases must be able to accommodate the incoming data. Currently we use very large databases to store this information and we call them data warehouses.


- 88 -


The perceived difference between a normal database and a data warehouse is that the first is optimal for putting data in the system and the later is better for retrieving it. This is true, and there are specific database configurations and customizations for each case. Also the structure’s design differs. While on OnLine Transaction Processing systems (OLTP) we must avoid data replications, for better quality and performance when entering or altering data, in OnLine Analytical Processing (OLAP) systems we must endure some data redundancy in order to allow quick access to the information.

The value of the data warehouses is their ability to support business intelligence. These specialized analytical databases typically provide support for complex multi-dimensional calculations and data aggregations, and they are able to do them in an acceptable amount of time. This would be impossible using OLTP systems. So it doesn’t come as a surprise that organizations regard their data warehouses as a tool for dissemination. Data dissemination is regarded as an important characteristic of quality. If only for this, data warehouses are tools of undeniable importance in NSI’s.

However, as we will show, data warehouses can and should be making greater contributions for the overall quality in statistical institutes. When data is transported from the OLTP systems to the OLAP systems, it should be cleansed and integrated with all the other data available in the OLAP system. It’s through the process we will explain in this paper that simple data shall be transformed into meaningful information, comparable not only across the dimensions of it’s own project, but also with transverse variables whose source could be distinct surveys.

In a rapidly changing world this is not an advantage anymore but a necessity. We need to report quickly but also be flexible on our analysis. Making integrated information available is providing a quality service to our users.

At the Portuguese NSI we have begun our journey through this path of excellence. We developed several techniques we shall present you, and already deployed some fully integrated systems. However this is a painful road. We shall discuss some of the difficulties and the techniques commonly used to overcome them.

When integrating information, sometimes we have to recover old information, which is difficult, and we may have to work with different classifications, trying to build bridges between projects and helping people to compromise. Each case is unique! It is a hard process for all the people involved but it is worth the effort, because this is the approach that will lead us to a new perception of reality, and will ultimately help us build our brave new world. Contact: [email protected]


- 89 -


Disclosing Semantics in Statistical Data Using Topic Maps Technology by Dagmara Wilschut, Statistics Netherlands

In the present form the output database of Statistics Netherlands consists of a collection of tables, which each contain data from one statistical process. Apart from the information that can be derived from the table’s position in the thematic hierarchy users don’t get any information about semantic relations between different tables or the statistical data therein. The accessibility of the statistical data can be greatly improved by making explicit the semantics of the statistical data and particularly the semantic relations between data; this will enable more intuitive and complete navigation of our data.

Topic maps is a new information technology which makes it possible to structure vast amounts of (digital) information resources. A topic map is a network which represents the knowledge embodied in the resources. By placing a network over the information and linking this network to the resources, a view on the information is created. Because the choice of nodes and connections in the network is free the topic map can model any view on the information resources.

Statistics Netherlands has researched whether topic maps could help make explicit the semantics in statistical data. The idea was to create a semantic view of the output database by modelling the semantic relations that exist between the statistical data in a topic map. The hope was that by using a topic map far more relations that exist between statistical data could be made visible.

Another powerful aspect of topic maps is the scope feature: knowledge is typically valid within a certain context. The topic maps technology offers a technique with which this context dependibility can be modelled. Information in the topic map can be labelled as being relevant within a certain scope. While navigating the topic map a user can choose his or her “context” from the collection of implemented scopes and thus limit the amount of information offered.

A pilot topic map was made for a part of the statistical domain. The first results show that the navigating possibilities opened by the topic map are far more intuitive and complete than the present day search mechanisms. The precise relation between the statistical data and the topic map, as well as some aspects of topic map technology are subject of further research.



- 90 -


Using Portal Technology to Improve Quality at the U.S. Census Bureau - Presentation and Demonstration of Portal Technology by Cheryl Landman and Deborah Stempowski, U.S. Census Bureau

The U.S. Census Bureau is the preeminent collector and provider of timely, relevant and quality data about the people and economy of the United States. With its main goal “To provide the best mix of timeliness, relevancy, quality and cost for the data we collect and services we provide”, it was natural for the Bureau to turn to Information Technology to help achieve this goal. During 2000, the Census Bureau began experimenting with portal technology in order to improve the quality of our data and our business processes. This paper will document our progress to date and lessons learned with the use of portal technology to improve quality at the U.S. Census Bureau.

As part of the Quality program at the U.S. Census Bureau, we needed a vehicle to better communicate and share knowledge about quality throughout the organization. We saw a need to establish a portal that would provide users with the Census Bureau’s quality standards, guidelines, and best practices while also providing a repository for documents that others could use in developing their quality components. At the same time, we saw an opportunity to link the quality program with another major process improvement activity at the U.S. Census Bureau– Project Management Initiative. We designed the Quality and the Project Management Portals using Oracle WebDB. We modified these portals over the last year based on customer feedback. The portal tool provides for the management of data quality as well as a means to share best practices, standards, and guidelines for data quality.

As we progressed with the use of portal technology, we began designing portals to help analysts improve their work processes and environment. These two survey portals give analysts the ability to easily review data from disparate sources, compare data from multiple surveys or censuses on one screen, annotate analytical reports electronically and share notes across reports, as well integrate search results from multiple engines. The U.S. Census Bureau’s data processing and analyses are enhanced with the use of portal technology. The final release of the two survey portals will be available in Spring 2003, however, analysts are working with the portals now.

This paper and presentation will demonstrate the portal technology as well as provide background about how we developed the sites, used customer feedback and the lessons learned through this process.

Contact : [email protected]


- 91 -


Interactive Visualisation and Publication for Official Statistics by Hans-Joachim Mittag, Eurostat

Due to the breakneck speed of advances in computer and internet technologies, the dissemination policy of national as well as of supranational statistical offices has changed considerably during recent years. There has been a transition from a chiefly paper-based distribution of data sets and publications to an electronic distribution via the Internet. The few remaining key publications in printed form are complemented by CD-ROMs or DVDs. This development is driven by growing user expectations with regard to the immediate accessibility and comprehensibility of data as well as by policy targets like “Towards an Information Society for All”.

In spite of the already existing technological capabilities, the potential of the new dissemination channels is barely exploited in official statistics. The Internet is mainly used for transporting static information without identifiable added value compared to printed material. Interactivity seldom means more than the provision of navigation tools supporting queries or extraction from databases. Flexible tools for the immediate visualisation of user-defined data subsets and interactive publications offering environments for exploratory analysis of data are still largely missing. Such tools and user-friendly environments address citizens at large and offer the unique opportunity to increase the visibility of official statistics in society and to improve its public image. In addition, they could play an important role in furthering statistical literacy and linking the rarely connected worlds of educational and official statistics.

In order to illustrate the huge potentials for official statistics connected with interactive visualisation and to improve the accessibility of survey results for educational institutions and the interested citizen in general, several Java applets based on Eurostat data sets have recently been developed within the framework of a German multimedia project “New Statistics”. These elements aim at communicating the messages behind the data by means of self-explaining and user-controlled graphs to a broad audience. The elements are self-contained with built-in methodological comments and hints for optimal use. Interactive and multilingual Java applets could be used online as well as offline on CD-ROMs as marketing instruments (“appetizers”). They could be started from PowerPoint presentations in training courses or press conferences. Eurostat already made offline use of interactive visualisation by incorporating a trilingual applet in the CD-ROM, to complement the Yearbook 2003 on Regions. Repositories of multilingual interactive visualisation tools could be shared by national and supranational statistical offices as well as by educational institutions.

Java applets could also be used for interactive publications. These link static text with user-friendly experimental working environments, for exploratory ad-hoc data analysis, or with official data bases in order to provide automatically updated information.


- 92 -


The lecture will focus on presenting selected Java applets based on European data sets from social statistics, and on outlining different scenarios for use, including interactive publication.

Web references: http://www.neuestatistik.de http://www.neuestatistik.de/presentation.php?HP=4 http://www.fernuni-hagen.de/e-statistik http://forum.europa.eu.int/Public/irc/dsis/wages/information


Session 10: Leibnizsaal

Variance Estimation in the Presence of Imputation (DACSEIS I) Chair: Anthony Davison, Swiss Federal Institute of Technology Lausanne

Presentation of the Swiss Household Budget Survey (HBS) in DACSEIS: A Focus on Nonresponse by Jean-Pierre Renfer, Ueli Oetliker, Swiss Federal Statistical Office

In the DACSEIS project, several data sets were made available to the partners for conducting a simulation study. The Swiss Statistical Office (SFSO) contributed a data set based on the Swiss Household Budget Survey 1998. An overview of this survey will be given by presenting the sampling design, the survey process, the weighting model and the form of the estimators. Special attention will be given to unit and item nonresponse in the survey. The nonresponse mechanisms provided for the simulation study will also be presented.


Resampling-Based Variance Estimation in DACSEIS with Application to the Swiss Household Budget Survey by Sylvain Sardy, Anthony Davison, Swiss Federal Institute of Technology Lausanne

We consider the Horvitz-Thomson estimator adapted to calibration (raking) and imputation in survey sampling. Our goal is to estimate its variance. Neglecting the imputation mechanism tends to severely underestimate its variance. We consider adaptation of resampling-based techniques such as block-jackknife, balanced repeated replication, bootstrap, multiple imputation and linearization, and show with a simulation based on the Swiss Household Budget Survey how the various estimators compare in term of relative bias and variance, as well as cpu time. Contact: [email protected]


- 93 -


Variance Estimation under Multiple Imputation by Susanne Rässler, University of Erlangen-Nürnberg, Germany and Ralf Münnich, University of Tübingen, Germany

In this paper we discuss nonresponse and imputation issues of large-scale data sets with different scaled variable. The problem of finding a suitable multivariate imputation model is reduced to univariate specifications which are much easier to perform. A regression-switching variable-by-variable Gibbs sampler is described useful for complex high survey data with missing values at different positions. Possible theoretical shortcomings of this approach are addressed as well as data problems. A simple simulation study is performed comparing different approaches to illuminate the advantages and disadvantages of different imputation techniques with the focus on variance estimation. Results are presented from further simulation studies of the DACSEIS project concerning MI in comparison to other resampling-based variance estimation methods.


Non-Bayesian Multiple Imputation by Jan Bjørnstad, Statistics Norway

In national statistical institutes (NSI's) the methods used for imputing for nonresponse very seldom, if ever, satisfy the requirement of being "proper" that is needed in order to use Rubin's combination formula. However, the idea of creating multiple imputations to measure the imputation uncertainty and use it for variance estimation and for computing confidence intervals is still of interest. The problem is then that Rubin's combination formula is no longer valid with the usual nonproper imputations used by NSI's. The reason being that the variability in nonproper imputations is too little and the within imputation component must be given a larger weight in the variance estimate. This paper suggests an approach for dealing with the problem of determining what this weight should be to give valid statistical inference, and also for what kind of nonresponse mechanisms and estimation problems it is possible to determine a simple combination formula not dependent on unknown parameters.



- 94 -


Session 11: Metadata II Spiegelsaal Chair: Thomas Burg, Statistics Austria

Standardising, Evaluating and Documenting Quality: The Implemen-tation of ISTAT Information System for Survey Documentation – SIDI by Giovanna Brancato, Concetta Pellegrini, Marina Signore and Giorgia Simeoni, ISTAT, Italy

The paper describes the main features of the Istat Information System for Survey Documentation, SIDI, as well as the implementation strategies adopted for its internal dissemination and use. In addition to the management environment, a navigation subsystem has been developed on a web-platform and is available on the Istat intranet. The system is aimed at supporting the survey managers in the quality control activity. Indeed, SIDI manages both quality indicators and metadata in an integrated way. A set of standard quality indicators has been defined for each relevant phase of the survey production process; the quality indicators are collected for each survey occasion. The system is equipped with several functionalities for graphical and tabular representations. Therefore it allows the survey managers to: i) monitor their production processes; ii) evaluate the quality over time; iii) compare the indicators with average values and iv) compare the quality of different surveys. The metadata managed in SIDI concern both the information content and the process characteristics of the survey. In particular, the observed phenomena and statistical units are documented together with the survey operations (e.g. data collection technique) and quality control actions aimed at preventing, reducing, and evaluating non-sampling errors. The integrated management of qualitative and quantitative information makes it possible to properly analyse the quality indicators by relating them to the survey context, i.e. the survey methodologies and the quality control procedures. The implementation of SIDI started in 2001. For what concerns metadata SIDI is populated for almost all Istat surveys, while the quality indicators are computed for the most important ones. Many NSIs have faced the difficulties related to the implementation and the updating of information systems. Therefore, a strategy for the implementation of SIDI has been defined. It includes:

- The creation of a net of quality facilitators in Istat. Up to now, about 50 people have been trained.

- The development of generalised software to support survey managers in com-puting standard quality indicators by integrating the quality activity into statistical production processes, speeding up the work and avoiding errors.

- The integration between SIDI and other local information systems or databases storing absolute values for calculating quality indicators.


- 95 -


- The support to survey managers and quality facilitators by the centralised unit in charge of the system.

Recently, the system has been extended in order to produce, in an automatic way, standard and validated methodological notes for the Italian Statistical Yearbook. To this purpose the documentation managed in SIDI has been enlarged to encompass metadata related to secondary studies (i.e. studies using statistical information already released as for instance National Accounts). In conclusion, the SIDI system supports the standardisation of the survey method-ologies and the quality control activities used in the Institute. At the same time, it is an important tool in order to achieve a common starting level of quality for Istat surveys. Finally, it provides Istat top management with useful information for taking decisions for improving quality.

Contact: [email protected], [email protected], [email protected], [email protected]

Metadata and Survey Documentation – Portuguese NSI Experience by Isabel Morgado, National Institute of Statistics, Portugal

In the Portuguese National Statistical Institute there are some standard procedures related to the life cycle of the surveys:

- During the design phase, a document called “Methodological Document” must be produced, describing the survey, its administrative and methodological aspects, as well as concepts and classifications used;

- During the quality evaluation phase, a “Quality Report” will be produced;

- Questionnaires have to be centrally registered and have a unique identifier number.

Since the beginning of 2002, an integrated statistical metadata system is being designed and implemented. One of the components of this system is about surveys and related metadata. Based on the principle that metadata should be collected once and where it is produced, we integrated the metadata capture with the survey life cycle. For that it was created a standard template to the “Methodological Document”, and a set of input screens were developed, based on that template, to populate the metadata repository with surveys metadata. This input screens are used by survey managers in the design phase of the surveys.

Having Methodological Documents conform to this template, has also the advantage of harmonize survey documentation. Subject matter units, in a conjunct work with statistical methods technicians, have been reformulating former existing documents. This task must be done in a phased way, and has some organizational issues that must be taken on oneself.


- 96 -


The Surveys component of the system is composed by:

- a central repository, where metadata is stored;

- The Methodological Document with a word template an input module; a presentation module, where metadata is presented as an XML document; a management module; workflow functionalities, that controls the workflow of the documents through the life cycle of the surveys, according to the standard procedures.

- Collection Instruments;

- The Quality Report;

The integrated statistical metadata system being implemented has other components, beyond the Surveys component, like Concepts and Classifications components. As surveys use concepts, classifications, methodologies, questionnaires and variables, the Survey component, through the Methodological Document, is the one where the usage of those co-ordination instruments is shown, in a production point of view, beyond it constitutes also metadata to support data dissemination.

Quality Report is the other document that will integrate the Surveys component. Some metadata will be created in the system, during the Execution phase of each survey instance, manually or automatically generated from the production systems, and will contribute to the content of this report.

Another important issue related to the contents of the Survey Documentation is its quality. Metadata systems are useful if they are complete and up to date, but not less important is that metadata stored have great quality.


Managing Statistical Metadata at OECD by Russell Penlington, OECD

The OECD is developing a new system, called MetaStore, for internal access and management of metadata in a decentralized environment. The system sits on top of a large number of production databases to enable centralized management and sharing of metadata.

MetaStore is one component of a wider "Statistical Information System" initiative to streamline data and metadata collection, processing and dissemination within the organization. MetaStore makes it possible to manage metadata attached to all OECD databases and to any subset of these databases

MetaStore and other components of the Statistical Information System have had impetus from the recent introduction and application of Quality Framework and


- 97 -


Guidelines for OECD Statistical Activities. As such, MetaStore has been designed to address many aspects in the quality of accessing and managing metadata.


Study Documentation in Cross-National Surveys by Peter Ph. Mohler, ZUMA, Germany and Beth-Ellen Pennell, University of Michigan, U.S.A.

Documentation does not seem to be anyone's favorite pastime in research projects. Considering that science needs to rely on inter-subjectivity and replicability, this is somewhat counterintuitive. At the same time, if we compare the tools available for documentation -say toolboxes such as the DDI nomenclature (Data Documentation Initiative)- with the user-friendly and sophisticated tools available for statistical analyses, it is easy to see why research teams shy away from addressing documen-tation needs. To date, good documentation seems to be largely left to data archives or large infra-structures. They and their sponsors invest heavily in professional data documentation (often in the form of codebooks). At the same time, internationally and cross-nationally, researchers lack a standard tool for process documentation, one that allows research teams to attend to the note-taking needs of documentation.

Process documentation is even more pressing for cross-cultural studies. Here almost all documents and data are multiples of a single culture study. These multiple sets of documentation units cannot be handled by simple hierarchical categorical systems. They require intelligent database management systems (relational databases). Moreover, surveys always are embedded into several layers of contextual information, including data from other surveys, institutional and legal contexts, and socio-cultural contexts. This information has to become part of the study documentation. Knowledge Management provides a key to managing the multiple sets of information necessary for cross-national studies. Now widely used in industry and IT, knowledge management focuses on the usefulness of and need for the information collected (knowledge needed to produce goods). The paper discusses and illustrates new developments in study documentation using a data base management systems approach.



- 98 -


Metadata Common Vocabulary: An SDMX Standard for the Exchange of Statistical Information by Marco Pellegrino, Eurostat

There are several types of metadata covering the statistical production from data collection to dissemination; but there is no universally accepted model or any standard definition of all the different types of metadata. Given the diverse requirements of national and international offices, reaching agreement on a common metadata model is an impossible task: although there is some commonality, each model has a slightly different objective and each describes different methodological issues and stages of the statistical production. The focus of the SDMX initiative on data and metadata exchange, launched by six international organisations (BIS, ECB, Eurostat, IMF, OECD and UNSC, see http://www.sdmx.org) is the exchange of statistical information by taking advantage of existing exchange protocols, dissemination formats and emerging e-standards such as XML. All this implies a common understanding of the nature of what we should exchange.

The need of a common terminology on metadata is evident if we consider how many times the same information items are referred to by different names or, conversely, how many times the same name actually refers to different concepts. Statisticians and subject-matter experts often use common terms with different meanings. This implies that what is understood by one user can be easily misunderstood by another, in the absence of a standard. The focus of the SDMX project of a Metadata Common Vocabulary, jointly conducted by Eurostat and OECD, aims at establishing a core set of standard definitions of metadata items for: a) improving the standardisation of metadata for the purposes of data exchange; b) promoting the preparation of a standard documentation describing the data, thus helping international comparability.

The Metadata Common Vocabulary (MCV) is concerned with the elaboration of terminological building blocks based on statistical international standards or best practices, easily understandable and re-usable. Agreement on a common set of terms that can be used to describe the collection, processing and dissemination of data would still provide the flexibility for each organisation to manipulate these elements to derive a variety of metadata models and documentation outputs. An agreed-to list of basic metadata elements (with associated definitions) would simply provide a common unambiguous vocabulary; this would facilitate the information interchange between organisations and between the metadata models embedded in the information systems of different national and international organisations.

The MCV Glossary presents the following "fields": term, definition, source, related terms and context. The authors used the "context" field extensively throughout the glossary, sometimes for providing additional explanations, other times for highlighting peculiarities in how a certain definition is applied within a certain domain or


- 99 -


geographical context. Of course, the MCV is linked to other subject-specific glossaries (on classifications, on data editing, on quality assessment, on subject-matter statistical areas) and to more universal glossaries of statistical terms already available on the web, such as Eurostat CODED or the OECD Glossary of Statistical Terms. The insertion of some definitions derived from other glossaries should not be seen as a redundancy, but as a means of resolving the complex and interdisciplinary nature of metadata


Ontological Ambiguity of Statistical Metadata as a Restrictive Framework for Improving Quality in Statistics by Heikki Rouhuvirta, Statistics Finland

The working hypothesis is that comprehensive and fully described statistical infor-mation could not exist without detailed and relevant statistical metadata. This means that all kinds of ontological ambiguity in statistical metadata concept become a problem to the management of quality in Statistics.

In practice the conceptualization of statistical metadata is determined by existing technology used for data management, or even in some cases by software solutions being on hand. Similarly the model of statistical metadata has been developed in the framework of specific modelling tools and its implementation methods.

The first issue of the paper is to examine how statistical metadata is conceptualized using different modelling methods. The second issue is to evaluate the impact of used methods to the ontology of statistical metadata.

It is obvious that implementation of technology to statistical information managing is forced to conceptualize statistical metadata in the specific, technology determined way, which is not necessary optimal in the context of statistical information. After the evaluation results the paper will describe a approach to determine the statistical metadata ontology based on the theory of empirical knowledge and its epistemology as well as the methodology to perform in empirical research.

The last issue of the paper is to examine what kind of technology we need to implement the defined statistical metadata ontology into statistical information system and how this is possible to carry out. The paper will present the current state of implementation work at Statistics Finland and some example instances from statistical information with rich, e.g. comprehensive and fully detailed, statistical metadata.



- 100 -


Session 12: Questionnaire Design and Testing II Backstage Chair: Frank Nolan, Office for National Statistics, United Kingdom

What’s Behind an Answer? by Ing-Mari Boynton, Statistics Sweden

This study is an example of measuring quality in an already existing survey.

In a yearly survey on Household Economy conducted by Statistics Sweden, The Swedish National Institute of Health wanted to find out more about the background to the answers of one of the questions in the survey. The subject of the question was the economic aspect of the consumption of medical products in the population. The National Institute of Health asked Statistics Sweden to look further into the matter.

Design and method

To test the question a qualitative testing method was suggested. By interviewing respondents who had

- answered the question in a certain way, and

- agreed to another contact

we would obtain insights into the background of the answers. After detailed planning of the study a topic guide – based on the customer’s questions – was made. The topic guide was the basis for the interviews.

A selection of suitable respondents – representing different types of households within the population – was made. Semi-structured in-depth interviews were used to raise issues for detailed exploration, to identify the range and diversity of attitudes, behaviour and motivation. Interviews were scheduled, carried out and recorded on tape. The recorded interviews were transcribed and the material was put together. Textual analysis was made and the results were presented.

Results

The results of the study gave insight on how and why people in different situations think and act concerning consumption of medical products. The study concludes that economic situation is one reason to refrain from purchasing medicine. There are also other reasons.

Economic situation It is possible to distinguish between those having a temporary lack of money and those more or less always living on a low budget. We can also see those who are more or less seriously handicapped/ill and constantly in need of medicine – and those who are usually healthy and normally not in need.


- 101 -


Trust in health care Contacts with health care include visits to a health care centre or hospital, telephone inquiries, etc., and are mainly with doctors or nurses (who are responsible for prescribing medicine). Trust in health care is an important issue. Does the patient trust the doctor/nurse? Do doctors/nurses listen to the patient? Does the patient feel secure? On the other hand, does the patient listen to the doctor/nurse? Does the doctor/nurse feel confident in his/her role and in the existing situation? Is there sufficient time – time for trust to be established? Much of what is revealed in the interviews indicates that there is generally little trust in contacts between the citizen and health care. In nearly all of the interviews, a lack of trust from both sides is expressed.

Other reasons mentioned More specific reasons for refraining from purchasing medicine are mentioned, as well as alternative forms of medicine and health care.

As a result of the study the original question were re-formulated and new questions on the subject were added to the yearly survey. contact: [email protected]

The Development of Quality Indicators for Establishment Question-naires Using Process and Survey Data by Deirdre Giesen, Statistics Netherlands

At Statistics Netherlands there is an increasing interest in how data collection for establishment surveys can be improved to reduce response burden and increase data quality. For 2004 an evaluation and revision of the questionnaires for the Production Survey is planned. The Production Survey data are collected with an integrated set of questionnaires, consisting of a uniform core part and 179 unique parts, containing questions specific for type of industry and size class.

A pilot study (Hak & van Sebille, 2002) has shown that in-depth interviews with respondents of business surveys, focusing on all steps of the response model for establishment surveys (Sudman et al. 2000) can yield useful insights in the quality of questions and answers in Production Surveys. This kind of research however is extremely time intensive and it would be impossible to evaluate all questions in the relevant subgroups of respondents. Therefore we have decided to use a revision strategy that mixes quantitative and qualitative methods and where less expensive methods are used to prioritize the use of the more expensive and informative quali-tative methods.

We are currently developing quality indicators for the Production Survey question-naires, using as much as possible information that is already available. These include both quantitative measures, such as unit-response, item non-response and plausibility


- 102 -


indexes as well as more qualitative information such as spontaneous comments on the questionnaires or information collected by field staff. We will use these indicators to decide on which questionnaires, questions and groups of respondents we should focus in our qualitative fieldwork. In this fieldwork we will collect on site data on the response process in order to find out what makes questions problematic for respondents and how we can improve the questionnaires.

The paper will discuss the development of these indicators and first results from using them.

Hak, Tony & Martin van Sebille (2002). Het respons proces bij bedrijfsenquêtes. Verslag van een pilot studie. [The response process in establishment surveys. Report of a pilot study]. Unpublished report. Erasmus Research Institute of Management & Statistics Netherlands.

Sudman, Seymour, Diane K. Willimack, Elizabeth Nichols & Thomas Mesenbourg (2000) Exploratory Research at the U.S. Census Bureau on the Survey Response Process in Large Companies. In : ICES II, Proceedings of the Second International Conference on Establishment Surveys. Alexandria, VA: American Statistical Association, pp. 327-337.


VISICAI A Tool for Improving the Quality of Computer Based Questionnaires by Thomas Bachmann, TNS Infratest, Germany

VISICAI is a tool for the visualisation of CAPI, CATI or Online questionnaires and their question routing. It is a proprietary development of NFO Infratest. Before the survey starts every questionnaire must be checked to avoid problems in wording, filtering and question routing. Compared to a paper based questionnaire an electronic questionnaire includes programmed controls and routings, predefined sets of variables, random generators, pictures and so on is completely different to a printed version. Therefore it is almost impossible to check all possible branches within the questionnaire via the interviewer’s interface.

VISICAI helps in solving this problem. This tool is developed to support the quality controls during the preparation phase of a survey’s fieldwork. It is used by the programmers and by the survey coordination units for debugging and checking the complex structure of electronic questionnaires. It is also used for visualizing the questionnaire for inhouse-consultants and last but not least (with different restrictions) also for our clients. It helps to improve the survey’s and data quality by the fact that the user has a completely new interface for understanding and controlling the questionnaire and the data structure.


- 103 -


Some additional features included are: counts for the numbers of different question types, length estimations, various search functions, shortest / longest way through the questionnaire.

This paper will include:

- the presentation of VISICAI and of some of the features

- operation conditions for the tool, programmers guidelines for folder structure and standardized data interfaces

- some examples for improvement of data and survey quality


Advantages in Terms of Quality of the In-House Strategy for CATI Surveys by S. Macchia, M. Murgia, ISTAT, Italy

The usage of CATI technique is growing in Istat surveys: Istat researchers plan the survey architecture, while private companies are charged with the entire data capturing procedure. This organisation has always succeeded in concluding the data capturing phase, but often presented problems, due to the fact that private companies were often very experienced in telemarketing or opinion polls, but:

- had never faced in advance the development of electronic questionnaires so complicated in terms of skipping and consistency rules between variables

- had never put in practice strategies to prevent and reduce non response errors

- had not at their disposal a robust set of indicators to monitor the interviewing phase.

That is why the so called ‘in house strategy’, was tested for a very important survey (the Birth Sample Survey) and planned for other surveys to be held in 2004: it consists in relying on a private company for the call centre, the selection of interviewers and to carry out the interviews, but in giving it all the software procedure, developed in Istat, to manage the data capturing phase, concerning: the calls scheduler, the electronic questionnaire and the indicators to monitor the interviewing phase. The main advantages of this strategy are that:

- the responsible of the survey has the full control on the enumeration process

- a strict collaboration between the survey responsible and the CATI experts in defining the characteristics the package should have and in improving it according to the results obtained step by step during the developing phase, in order to guarantee the complete correspondence between the planned requirements and the developed procedure


- 104 -


- it is not necessary, for continuous surveys, to face all the planning and developing activities every time a different company is charged with this job.

Following this strategy, all the software system is planned having the quality as the main purpose to fulfil.

The electronic questionnaire, which is the core of the application, is developed so as:

- to prevent from data capturing errors ! through the range and consistency controls management

- to make the interview agreeable for respondents (to reduce respondents burden) ! making question wording friendly by its customisation according to respondent characteristics that can be already known or captured with previous questions

- to make the interviewer’s job easier ! minimising the segmentation effect, defining standard for the screen management, providing help on-line and making the consistency controls easy to be managed.

In addition, the CATI calls management system is set and adapted by Istat researchers during the interviewing phase so as to minimise total non-responses and to avoid sample distortions by giving all units the same possibilities to be contacted.

Finally, the interviewing phase is monitored through the production of a conspicuous set of indicators to keep daily under control different aspects, like:

- the productivity of interviewers

- the proceeding of contact results

- the number of sample units still at disposal and their movements from a stratum to another.

In this document, this strategy will be described in details, analysing thoroughly the standards adopted for each function and the main results obtained in the Birth Sample Survey.



- 105 -


A Comparison of CAPI and PAPI in a Nationally Representative Danish Health Survey by Ekholm O., Hesse U., Nørlev J., Davidsen M., National Institute of Public Health, Denmark.

Introduction

The Danish Health Interview Surveys have been conducted between 1987 and 2000 and, so far, data have been collected using Paper and Pencil Interviewing (PAPI). However, due to well-known benefits of Computer-Assisted Personal Interviewing (CAPI), a conversion from PAPI to CAPI has been discussed. Since one of the aims with the programme is to describe changes in health and morbidity over time, a prerequisite for this conversion is that the results from the two data collection methods are comparable. It is well known that a respondent does not reply to a question without thinking about why it was asked and for what purpose it will be used. The use of a computer to record answers instead of paper and pencil may have an influence on how the individual responds to a specific question. Several studies have investigated the response rates, the interview durations and the responses to questions, but no study (to our knowledge) have investigated these indicators according to gender and age. Thus, the purpose of this study is to investigate the differences between the two modes with regard to response rates, interview durations and responses to questions concerning health, lifestyle and morbidity.

Methods

Data were derived from the Health and Morbidity Survey in Denmark 2003. Four random samples of each 1,000 Danish citizens (age 18 or more) were drawn from the Civil Person Register. One of the samples was interviewed using PAPI and the other three samples were interviewed using CAPI. The interviews were carried out in the respondents’ home and were performed by a trained interviewer.

Results

The response rate in the CAPI group was 64,4% and in the PAPI group the corre-sponding rate was 67,5%. The most obvious differences in response rates between the two modes were found among women and elderly. The average interview duration among CAPI cases was significantly (p<0.05) shorter than the average interview duration among PAPI cases (mean: 58.5 and 60.3 minutes, respectively). Differences in responses to questions were found in questions that could be apprehended as sensitive (questions concerning alcohol habits, weight and height). Furthermore, the analyses showed that CAPI responders were as likely as PAPI responders to answer “don’t know” to questions.


- 106 -


Discussion

The results from this study indicate that women and elderly are less willing to par-ticipate when the data collection method is CAPI. These findings might be understood as an outcome of different habits and familiarity with computers. However, those who actually participated in the study seemed to apprehend the CAPI mode as more confident and anonymous than the PAPI mode and, thus, made them more willing to report “socially improper behaviour”.

This study gives us no explicit answer if the CAPI or the PAPI mode is preferable in the forthcoming surveys. But the results show that a conversion from PAPI to CAPI concerning response rates and responses to questions is not unproblematic and should be one of several very important considerations (other examples are costs and a quicker turnaround) in the decision process.


Changing from PAPI to CAPI: A Longitudinal Study Dealing with Mode-Effects in the German Socio-Economic Panel (SOEP) Using an Experimental Design by Jörg-Peter Schräpler, Ruhr-University Bochum and DIW Berlin, Jürgen Schupp and Gert G. Wagner, DIW Berlin, Germany

This paper describes the conversion from PAPI to CAPI of a subsample of the German Socio-economic Panel (SOEP) and examines the implication of the move to CAPI for data quality.

A representative gross-sample E contains 2000 German households and is split into two additional subsamples E1 and E2 with the same structure using twin – sample points. Each of the 125 sample point contains 16 addresses (8 for E1 and 8 for E2) and had to be realized in the first wave alternately with PAPI and CAPI mode per interviewer. In the subsequent waves the PAPI mode is partly replaced by CAPI. With this experimental longitudinal design we are able to control for possible interviewer effects in the analysis of mode effects.

The paper assesses whether any mode effects are apparent within the response rate and within the data, we examines monetary amounts like gross income, item and unit nonresponse rates, satisfaction items and the consistency of some key measures.

We could find some minor effects but our main results show that we have made the shift without introducing strong mode effects.



- 107 -


Session 13: Data Processing I (Capture&Coding) Museum Chair: Mats Bergdahl, Statistics Sweden

Generalized Quality Control Approach for ICR Data Capture in Statistics Canada’s Centralized Operations by Walter Mudryk and Hansheng Xie, Statistics Canada

Statistics Canada is embarking on a program of increasing its use of imaging tech-nology to make its data processing operations as paperless as possible. The intention is to use Intelligent Character Recognition (ICR) to capture most questionnaires, documents and faxes being received in the central operations area and then to use these captured images for further down line processing and editing.

Following our successful experience with the ICR capture of the 2001 Canadian Census of Agriculture, it became evident that quality control procedures played an important role in ensuring the success of this operation and that they should become an integral part of any future ICR process. This is because quality control is important in the pre-processing steps leading up to scanning and imaging (i.e., document preparation and scanner calibration) as well as during the actual scanning production.

Accordingly, Statistics Canada’s central processing area is in the process of developing a generic quality control approach for its scanning operations. The challenge here is to develop procedures and systems that can be easily applied to any type of survey, of any size, over any time-frame, in a user friendly manner. Furthermore, the procedures should involve statistical measurements upon which objective process decisions for quality improvement can be made.

The purpose of this paper is to describe this generic quality control approach for Statistics Canada’s central operations and to show its relevance to other similar ICR situations. The approach will focus on the statistical methods and procedures that are being used (or proposed) for ensuring quality at the document preparation and scanner calibration stages, as well as the controls required during production. The positive impacts of this approach on various operations (i.e., how it helps to control and improve quality of the entire scanning process) will also be discussed.


Computer Assisted Coding by Interviewers John Michiels and Wim Hacking, Statistics Netherlands

In many stages of statistical production process variation can arise. One of these stages is the coding of open text answers. At Statistics Netherlands codes need to be established for economic activity of businesses, occupation, and education. The coding


- 108 -


is executed by coding experts and this process is known to have a considerable random component. At Statistics Netherlands new techniques for coding open texts answers have been developed, which remove a large part of this process variation. The new coding techniques can be categorized in two approaches: the first approach is to code open text answers during the interview by assisting interviewers in assigning codes. An advantage of this approach is that feedback from respondents can give more detailed information than is contained in regular open text answers. The second approach is the automatic coding of open text answers at the statistical office using dedicated algorithms. In collaboration with the IKAT institute at Maastricht University learning machines have been developed for classifying open text answers: TDIDF, Naïve-Bayesian, Nearest Neighbour and Support Vector machines. In addition to these learning machines expert systems can be developed to improve upon the coding process. Results on both approaches are being presented, including an analysis of the accuracy of these approaches.


The French Inter-Enterprise Relations Survey by Olivier Haag and Emmanuel Raulin, Ministry of Industry, France

A survey has been conducted in several EU Member States in order to get information on the nature, extent and motivations of relationships between enterprises. In addition to the core activity of the enterprise, four other enterprise functions supporting the core activity have been analysed: purchasing, sales and marketing, auxiliary services and R&D. The enterprises were first asked about the number of relations by function and the part taken by these relations within the function (the amount of turnover, of R&D expenses, etc.). Secondly, they had to provide characteristics relating to each of the three most strategic relationships; these characteristics were on the one hand about the relation itself (like type, duration, reason for engaging into it, intensity, etc.) and on the other hand about the partner (like kind of partner, criterion of selection, location, etc.). The French enterprises were requested to describe separately three relationships and not to get “average” information about a “middle” relation like in the EU questionnaire.

Usually, the enterprise surveys in France are essentially quantitative surveys. That is why this survey was particular and consequently survey techniques have been to be adapted, notably as regards non-response or measurement errors treatment.

While item non-responses were essentially treated by donor methods (stratified hot-decks), two other cases of non-sampling errors led to use original methods for dealing with them: the first case arose when an enterprise has described separately not so many relations as expected according to the declared number of its relationships; the second case occurred when enterprises have declared on the questionnaire that they have no inter-enterprise relation whereas it was well known through other statistical


- 109 -


sources (like the Annual Structural Survey) these enterprises work or seem to as subcontractor!

Finally, the response rate of this compulsory survey reached 85% (4226 responses of 5220). 12000 relations were described separately on the questionnaires. 5200 relations have been imputed, 2800 of them because of a lack of separate descriptions (case 1) and 2400 of them because of measurement errors (case 2). The variance resulting of these two treatments has been estimated and will be presented.

The presentation will focus firstly on the close link that has been established for this survey between the management of the data collection and the foreseen methods for treating item non-responses, measurement errors or lack of separate descriptions: recalling several sub-samples of enterprises, trying to obtain the description of at least one relation for each surveyed enterprise, use of auxiliary information, etc.

Secondly, the presentation will insist on the techniques of data analysis, especially clustering techniques, that have been used: the aim was, first of all, to display a limited number of clusters (for each enterprise function) in order to regroup the relations described by the enterprises according to their profiles (characteristics of each relation) and, after that, to calculate the joint probability for observing simultaneously, and for a given enterprise, relations from cluster i

and cluster j and cluster k.


Simultaneous Detection of Outliers in Business Statistics in the Hungarian Central Statistical Office by Zoltán Csereháti, Hungarian Central Statistical Office

The detection and treatment of outliers are among the most important tasks to do before processing survey data. Nearly every survey is potentially affected by outlying values. This kind of extreme values can emerge mainly in two different ways. Error may occur somewhere in the data recording stage causing outlying observations. The other way is that there can exist a correct data, which seems to be inconsistent with the remainder. Both of these cases are very important.

The distributions in business statistics are typically very skew. That is why the detection and treatment of outliers are very important tasks. One of the most important limiting factors is timeliness. In the case of the monthly surveys there are only a few hours for the task of outlier detection. In the area of business statistics there is one more thing complicating the problem. We are interested in a good estimation considering the whole population and considering the single strata at the same time. That is why we need to handle the outlying values not just in the whole population but also in all the strata simultaneously. There are many well-known outlier tests for the first task, but not many for the simultaneous case. It is not easy to choose from the possible tests because there is a trade off considering how important is the accuracy of the estimate


- 110 -


for the whole population as compared to the one for the single strata. This poses the need of a simultaneous outlier detection algorithm. This can be done by a modified Grubbs-type method.

However we must not accept the result of any outlier-test automatically without any criticism. There are several reasons to say that. For example it is a serious problem that our time series are strongly affected by the chosen outlier treatment strategy. One has to avoid the occurrence of any fractions. This can easily occur in a longitudinal survey. Another problem is that a sample estimate is with high probability slightly under-estimating, and with low probability much more over-estimating. We have the chance to detect some of the over-estimating cases, but no chance to detect any under-estimating case. This means – if we have many strata – that we can improve our estimates in some of the strongly over-estimated cases, reducing the amount of over-estimation, but we cannot correct any estimation of the slightly under-estimated strata. Another problem is the issue of variance estimation. One can calculate no more exact variance estimations when outliers occur.

Consequently it seems to be that outliers cannot be detected fully automatically. There are several important factors to take into account, which cannot be easily presented in the form of numbers. If we detect too many outliers, it can disturb our calculations. It seems to be that “the eye of an expert” is always as important as a good detection-algorithm.


An Alternative Approach for the Implementation of Data Editing: the INSPECTOR Project by Gregory Farmakis, George Petrakos and Photis Stavropoulos, Liaison Systems SA, Greece; José Figueiredo, Daniel Mota and Daniel Santos, National Institute of Statistics, Portugal

Data editing is indispensable for national statistical offices and for any other producer of statistical information. Firstly, it ensures that the statistical production process uses data of acceptable quality. Moreover, it assists in the identification of weaknesses in survey design, data collection and data entry and leads to subsequent improvements. Statistical offices expend large amounts of effort and money on the design of editing and on software for it; reductions would be to their benefit. Moreover, with the increase of cooperation between statistical offices, especially within the European Statistical System, the harmonisation of data editing activities has become a pressing issue.

The INSPECTOR project has devised an alternative approach to data editing and has developed a software prototype which implements it. The novelty of the approach lies in the treatment of edits as definitions of the domains of variables and not as Boolean statements of relationships between variables. Users of the INSPECTOR approach


- 111 -


simply declare the variables and their domains required in order to implement a set of data edits instead of writing computer code. A suitable database stores these definitions for future use. An important point is that the variables are a superset of the survey variables. The additional variables are defined, as functions of survey, auxiliary or historical variables or as vectors of other variables, in order to cover all required edits.

This approach is implemented in the INSPECTOR software prototype as follows: the user declares all the variables and their domains required for the editing of a given survey’s data. The declaration also involves the statement of how to derive any variable which is not an actual survey variable. The system uses these declarations in order to (a) upload the required data from the survey file and any historical and auxiliary data files and (b) to derive all required variables. The result of this process is an “enhanced dataset”, a superset of the original survey dataset. Editing is applied by checking whether the data values of the enhanced dataset belong to the domains of the variables and a report presents the records and variables with “errors”. The subsequent steps of error localisation and imputation were not within the scope of the INSPECTOR project and are not implemented in the system.

The prototype has been designed in such a way as to enable the implementation of several installations which share edits (i.e. variables and domains) between them. Such an installation in Eurostat and national statistical offices can promote the harmonisation of editing practices.

In Q2004 we are going to explain in detail the approach and its benefits for the European Statistical System. We will present the design of the software prototype and we will make a live demonstration. Finally, we will report the results of its application in real conditions by the national statistical office of Portugal and will compare them with the results of currently used editing practices in the same setting.


Population and Housing Censuses Data Processing: 2001 Italian Experience and Future Prospects by A. Ferruzza, D. Zindato, ISTAT, Italy Population and Housing census is the most important opportunity to collect demo-graphic and sociological data at enumeration areas geographic level. Italian experience is described to speak about product and process innovations implemented to improve quality. New units of enumeration and new units of analysis had been used in the 2001 Italian census. Optical reading had been used for data capturing and the most innovative equipment for data processing. Internet had been used to disseminate information during the census operations and to disseminate produced data at the end of the operations.


- 112 -


Population and Housing Census Data Processing System is presented. This system to validate and to product census data is splitted in different sub-systems: Reception System, Units Structure Definition System, Structure Definition Linkage System for Households and Institutional-Households, Structure Definition Linkage System for Dwellings and Buildings, Values Definition System for Households and Institutional Households, Values Definition System for Population, Values Definition System for Dwellings and Buildings. The processing system is supported by a process control system and it is based also on a datawarehouse for validation and diffusion of the data.

The realized production process has been based on this system: it is organized on the execution of several steps of the different subsystems and, also, on the revisers and supervisors interactive analyses and correction of the obtained data.

Census data had been acquired by OCR system; this is a innovative element of the production system also because it has strongly conditioned analysis choices in the validation process and because optical read data has been used during the validation process. Moreover automatic coding process had been used in order to classify some textual variables.

Another innovative element of this system is the possibility to analyze, in different production steps, the census data through simil-dissemination tables; this has been obtained thanks to the datawarehouse system that support the production.

This Data Processing Systems improve accuracy, coherence completenesses of statistics together to the possibility to disseminate data step by step and not only at the end of the process of validation.

That is at the end of the Units Structure Definition System the Legal Population was disseminated using Internet, at the end of the Structure Definition Linkage System for Households and Institutional-Households and of the Values Definition System for Households and Institutional Households, demographic variables of population were ready to be disseminated.

Finally Units Structure Definition System is described and analysed.



- 113 -


Interviewer Effect and Data Checking by Alessandra Federici, Agostina Loconte, Alessandro Martini, Manuela Michelini and Giovanna Tagliacozzo, ISTAT, Italy

Istat carries out the citizen’s safety survey every 5 years, and year 2002 it was its last edition. The survey is aimed at collecting information on the crimes against people and against property as well as the perception of safety in the environment and the safety measures adopted. A random sample of 60,000 households has been selected. In each sampled household only one person aged 14 and over has been randomly selected and interviewed. A CATI technique has been adopted and the interviews have been carried out by 200 female interviewers specially trained. The survey has been planned and carried out with a great awareness of the inter-viewers’ role for assuring the quality of survey results, training courses and have been set up. Nevertheless, the interviewers might have had an impact on the observed data. It is worth noting that this survey is characterized by an increasing degree of sensitiveness and difficulties as it investigates different aspects of life and safety of citizens. The aim of this paper is to estimate the interviewer effect on data quality by an ex-post point of view. As soon as the collecting data phase was completed, a complex phase of checking and correcting data started. CATI technique avoids some kind of errors, like out of range data, but data entry error could still occur due to interviewer lack of attention; a limited awareness of the questionnaire could produce an incorrect use of the open answer. On the other side, in this particular survey interviewers knew exactly the correct definition of crimes under investigation, because sometimes respondents are not able to do it. They have to be careful in understanding if the crime they are talking about is just attempted or committed. In the CATI questionnaire specific items have been studied to identify the real nature of the crime: whether it’ s attempted or committed. Even time is a problem due to the telescoping effect: it is not so easy to remember when a crime has happened and the survey aim is to record crimes thet took place in the previous twelve months. These four identified kind of errors have different levels of severity in terms of inter-viewer professionality: tireness, lack of attention, lack of training and burn-out risk. As regards the two first kind of errors an “Interviewer Effect” has been analyzed through a multivariate analysis on error occurrences by interviewers. Then other two sources of error are also related to socio-demographic interviewee characteristics and has been analyzed by a multilevel model and a subsequent cluster analysis on residuals. These are the more common errors we found in this survey, and it can be said that behaviour of interviewers in any step of the interview has some consequence on quality. The ex-post point of view based on the above control strategy unable first of all to


- 114 -


evaluate interviewers performance but gives also important feedbacks to improve interviewers training and recruitment strategies.

Contact: [email protected], [email protected], [email protected], [email protected], [email protected]

The Automatic Coding Process in the 2001 Italian General Population Census: Efficacy and Quality by S. Macchia and S. Mastroluca, ISTAT, Italy

For the first time in Italy, on the occasion of the 2001 Population Census, an automatic coding system has been adopted. The first purpose was to exonerate the 8100 Municipalities from the task of coding all the textual answers enumerated by census questionnaires and to centralise this activity through a computerised procedure which would have guaranteed savings of time, human resources and the enhancement of the quality of data. The encouraging results in terms of recall (percentage of texts coded automatically) and precision (percentage of correct codes assigned on the total of automated coded texts), obtained in a certain number of tests, like those on data of the quarterly labour force survey and of the two population census pilot surveys, lead us to plan a procedure based on the usage of the automatic coding system ACTR (Automatic Coding by Text Recognition). Considered the huge amount of census data, the coding activity was partially managed directly in Istat and partially by a private company. In detail, automatic coding of less complex textual variables (Municipality, Country and Education Level) captured through OCR/ICR was carried out in outsourcing, while the subset managed in house concerned all the textual variables (including Occupation and Industry) pertinent to residents in Institutional Households captured with a traditional data entry procedure and the residual part of answers optically read that the private company did not code, because it had to guarantee a recall rate coherent with results obtained in preceding experiences. Afterwards, Istat is going to code also a sample of answers regarding Occupation and Industry enumerated in Household forms. ACTR (developed by Statistics Canada) is a generalised software, independent from language and classification, which makes a texts analysis aimed at measuring the similarity between texts to be coded and descriptions of its internal ‘dictionaries’ (reference files). That is why, for each variables, a dictionary has been implemented, containing official descriptions from classification manual (often restructured in order to be processed), synonymous and pre-coded textual answers (empirical descriptions) given in other previous sample surveys. This last item has been really determining for Population Census data, because the wording used by respondents was closer to empirical descriptions than to official ones as the questionnaires were self-filled. The private company was supplied with these dictionaries in order to use them in its coding procedure.


- 115 -


The computerised procedure used for the in house application consists of two steps: a batch one and an assisted one, having the purpose of assigning codes to the responses which the batch step did not succeed in coding. The online procedure gives support to coders in navigating inside dictionaries in order to select the correct code and implements a function to enrich these dictionaries with very frequent empirical answers, so as to assure ever higher recall rates. In this document the results of the coding activity in terms of recall and precision will be presented. The precision of automatic coding is measured by submitting to expert coders samples of texts extracted for each variable, stratified at provincial level (NUTS3). In addition, the costs faced for the assisted coding activity, in terms of time and resources will be evaluated. Finally, an attempt will be made to evaluate the impact of ACTR on texts, captured by OCR/ICR, that the private company did not code, considering its results both from the quantitative and qualitative point of view.


Session 14: Improving Surveys Forstersaal Chair: David Marker, Westat, U.S.A.

Social Changes and Social Surveys Techniques: New Strategies and Challenges for the Italian Statistical Institute by Linda Laura Sabbadini, ISTAT, Italy

The nineties have been years of big transformation on the field of the official statistical production of the population social surveys. It has been a real jump of quality; it allowed to build a heritage of social information never reached before. The hidden crime, the violence and abuses within the family, the living standard, the disability prevention, the time use, the measurement of the informal support networks: the most diversified themes are considered and measured asking to the Italian National Institute for providing measuring strategies and adequate methodologies. The survey techniques have been heavily renewed form a technological point of view. CATI; CAPI, mixed techniques are more widespread but they force the Italian National Institute to act in a flexible way in order to optimise the reached standard of quality. The use of new technologies asks for continuos innovation and challenges on the field of quality. The problem for the future work of the Italian National Institute is how to win new challenges in the measurement of new social aspects using adequate surveys techniques.



- 116 -


Reducing Nonresponse, Enhancing Flexibility, Improving Quality in Household Surveys. The New Access Panel in the German Statistical System by Anja Nimmergut and Thomas Körner, Federal Statistical Office Germany

One of the most important problems for household and person surveys is the generally decreasing willingness to participate in surveys of official statistics, a tendency leading to increased non-response rates in many survey areas. In consequence, the achievement of an adequate sample size and a tolerable non-response bias requires an increased effort in recruiting households and persons. At the same time user requirements – especially those for data based on random samples – also increase. In order to cope with this problem the German statistical offices started implementing an Access Panel for official statistics in November 2003.

In Germany, with the exception of the Mikrozensus, all household and person surveys are carried out on a voluntary basis. As a large compulsory survey with a sample size of 370.000 households, the Mikrozensus constitutes a robust basis for the recruitment of the Access Panel: By contacting respondents of the Mikrozensus a permanent sample of households willing to participate in surveys of official statistics is established. The Access Panel will be used as a frame for voluntary household and person surveys in Germany. Its possible advantages include

- increased response rates, as the sample is drawn exclusively from households willing to participate in surveys,

- the possibility to draw sub-sample strata without any expensive previous screening, as up-to-date information is available from the Access Panel data base,

- the possibility for an effective stratification, as the information in the data base allows a stratification with characteristics that highly correlate with the variables of interest and

- the possibility for a sophisticated correction of the non-response bias, as the Mikrozensus information of the respondents could be compared with the Mikrozensus information of the non-respondents.

The decision for the establishment of the Access Panel was taken on the basis of the result of a large scale pilot study carried out by the Federal Statistical Office Germany together with five Regional Statistical Offices and the Centre for Survey Research and Methodology (ZUMA). The study started in December 2000 and was finished in September 2003. In the context of the study, the following questions have been analysed:

- methods for the recruitment of households,

- response and participation patterns of the persons and households,

- sampling procedures for stratified random samples from the Access Panel,


- 117 -


- pilot surveys with relatively high and low response burden in order to analyse the extent of response rates,

- development of appropriate estimation models,

- methods helping to minimise panel attrition,

- development and test of a suitable database and

- estimating costs and benefits of implementing an Access Panel.

The experiences of the pilot study have shown that an Access Panel could help to save time and resources in carrying out surveys. It can yield improvements in the design and the collection stage of a survey. The preparation of data collection can be realised more quickly and response rates are considerably higher than in comparable voluntary surveys (see table 1).

Table 1: Comparison of response rates

Maximal response rates of conven-tionally conducted voluntary household surveys

Response rates achieved during the pilot study for the Access Panel

Surveys with low response burden

50% 80%

Surveys with high response burden

9-15% 40-50%

According to these results, the Access Panel has great potential not only to solve some of the problems mentioned above but also to enhance the flexibility of official statistics with respect to voluntary ad-hoc-surveys. The first sample to be drawn from the Access Panel is the European Community Statistics on Income and Living Conditions (EU-SILC) which starts in 2005.


Nonresponse and Double Sampling by Ineke Stoop, Social and Cultural Planning Office of the Netherlands

Low nonresponse rates in surveys are considered to be a problem as they may reduce accuracy and cause bias. Efforts at increasing response rates are generally motivated by the assumptions that 1) higher response rates lead to higher data quality, and 2) obtaining the participation of respondents that are hard to contact or reluctant to cooperate will not only increase response rates but also reduce bias. This reduction of


- 118 -


bias will occur if late contacts are similar to noncontacts, and reluctant respondents are similar to refusers.

As high response rates are increasingly difficult and expensive to obtain, many organisations seem willing to abandon the principle of random sampling, and switch to nonprobability sampling. By using quota sampling or (web) access pools, costs can be kept down and the issue of nonresponse turns irrelevant. Many others, among which most national statistical institutes, stick to the principles of statistical theory and probability sampling, and keep aiming at high response rates.

Recent studies have cast doubt on the effectiveness of extended interview efforts and high response rates in reducing nonresponse bias (Keeter et al., 2000; Curtin et al., 2000; Teitler et al. 2003). One reason for these disappointing results might be that the extra respondents obtained through these extra efforts, differ from the general group of nonrespondents. Results of Stoop (2003) show that converted refusers are rather similar to easy respondents, and quite dissimilar from persistent refusers.

An alternative approach would be to aim directly for reducing nonresponse bias, rather than via increasing response rates. This could be done by double sampling. In this approach extensive fieldwork efforts will not be deployed in increasing the general response rate at very high costs by so many percentage points by getting the cooperation of the most willing part of the refusers, but by accepting moderate response rates at moderate costs and obtaining the cooperation of a very high percentage of a small sample of refusers, among which very non-willing ones (Hansen and Hurvitz, 1946; Groves, 1989; Elliot, 1991; Bradburn, 1992). Stoop (2003) shows that maximum efforts can indeed succeed in obtaining the cooperation of the majority of persistent refusers. The information on this second, small sample can be used to estimate the bias in the first, large sample. In this way, the double sampling approach can be a way to fight nonresponse, not by attacking high nonresponse rates but by attacking nonresponse bias.


Improving the Accuracy of Regional Extensions of National Household Surveys Using Additional Sampling. by Marc Christine and Laurent Wilms, INSEE, France

The French national household surveys are based on a sample of dwellings referred to as “master sample” (MS). This sample constitutes an initial selection stage with primary units (UP). This MS has been built in order to ensure good features for the national surveys and reasonable data collection expenses by concentrating the sample in given UPs.


- 119 -


In some cases, regional users are interested in regional estimates, with a given level of accuracy. These estimates provide useful information to the local policy makers and allow comparisons between the regional and the national levels.

However, both the way in which the MS is selected and the size of the regional fractions of the national dwelling samples resulting from it may prevent us from obtaining the accuracy desired at the regional level.

Therefore, a complementary regional sample has to be selected. It allows the statis-tician to build regional extensions of a given survey and to increase the regional accuracy of a national survey. The French statistical institute, INSEE, has developed a method to select this additional sample of dwellings in this framework.

This paper presents the way it has been implemented and the theoretical and practical difficulties that have been met.

This sample will itself be selected from a specific “reserve” divided in UPs, separate from those of the MS. This new reserve forms the “Master Sample for Regional Extensions” (EMEX).

The accuracy of the estimators resulting from the joint use of the MS and the EMEX can be improved by using the balanced sampling technique.

Nevertheless, the MS was selected first and only afterwards was the EMEX selected conditionally to the MS. Consequently it is not generally possible to obtain balancing properties from the joint sample with respect to the reference regional population.

In this context of sequential and conditional selections, the paper first focuses on theoretical issues. It presents a few specific situations where regional balancing can be achieved by the global sample. Then, when it is not the case, the paper describes the method called inverse balancing which provides an approximation of the solution of the problem.

From a practical point of view, we have made simulations for some regions in order to compare the sampling properties developed in this latter framework with the traditional sampling without balancing (using age and income variables as balancing properties).

The final choice between the two methods involves an estimation of the accuracy but also practical constraints such as the number of secondary units drawn from the primary units in the second stage of sampling and the result of this on the interviewers’ work.



- 120 -


Improving the Quality of ISTAT Consumer Price Index: The Case of the "Centralised Surveys" on Prices by Carlo de Gregorio, Stefania Fatello, Stefano Mosca, Francesca Rossetti, ISTAT, Italy

The authors are all part of Istat task force working, within the Consumer Price unit, at the re-design of the so called "centralised price surveys". This paper reports the fruits of the activities introduced in last months. The authors wish to thank Carla Sciullo, Maria Teresa Santoro and Norina Salamone, working in other units at Istat, for their helpful comments and aid in building the weighting structure. The authors remain the only responsible for the views expressed in the paper.

The growing demand of affordable price indexes, also at two or three-digit level (or even more) of the COICOP classification, has induced in recent years NSIs to make further efforts in order to improve the quality of estimates of the Consumer Price Index (CPI). The annual renewal of CPI basket and the activities yearly made to determine the new base period, represent a major opportunity to introduce innovations and to develop the necessary improvements in the production processes of the elementary indexes.

The Italian National Statistical Institute (ISTAT) actually produces CPI estimates relying on a mix of local and centralised monthly surveys. Local surveys are performed by the statistical offices of larger municipalities, under ISTAT supervision. Centralised surveys are instead directly performed by ISTAT: they concern products with a uniform price across the country or with peculiar features that make it preferable a centralised treatment.

This paper focuses on centralised surveys: their number is nearly forty, representing about 20% of year 2003 CPI weights. In particular, the main features of the innovations that will be introduced with the new December 2003 base are analysed. They mainly concern methodological aspects of the surveys, such as sampling design, sample size, data collection, seasonal products treatment, imputation techniques, weighting. Other aspects are also analysed, such as those relating to the organisation of the production process and to the tracking of its phases, from data collection to the release of the elementary indexes. The new year 2004 surveys will represent anyway only a first step of a longer term renewal process which will be accomplished in years 2005-2006.

Given the strong heterogeneity of the products and services that constitute the object of our surveys, a few case studies are presented in order to outline the general approach which is being followed.

As an example, the case of the "Other lodging places services" index (one of the components of "Accomodation services" grouping) is thoroughly examined. In year 2004 will be surveyed the prices of a sample ten times larger than year 2003's, stratified by region and by kind of tourist site (e.g.: sea, lake, country, mountain, city). Weighting will be introduced to aggregate strata, using ISTAT Tourism statistics.


- 121 -


Imputation of seasonal outlets' missing observation, data collection method and seasonal pricing treatment are also described.

The paper examines also the case of "Passenger transport by sea services" index, one of the components of "Transportation services" index. Also in this case weighting has been introduced using ISTAT Transportation statistics data. The sample has been significantly enlarged, and stratified by length of the travel, region and company. The choices made to treat seasonal price patterns are also illustrated.

The concluding remarks of the paper try to evaluate costs and benefits of the inno-vations introduced.


Resources vs Coverage - How to Minimize One and Maximize the Other: A Case Study from an Education Survey by Nancy R. Johnson, Meagan M. Wilson and Karen D. Deaver, U.S. Census Bureau, Stephen P. Broughman, National Center for Education Statistics

We've all been there ... You design the optimal approach for survey coverage. Then the final budget comes through, or the schedule is tight, or other work demands your time, and you just can't do what you want. How do you proceed? What do you sacrifice? Can you actually plan for this?

This paper will articulate the assumptions and describe the methodology and criteria for targeting specific state sources to maintain or improve the frame coverage for the Private School Survey (PSS). These criteria take into consideration the quality and timing of the state information as well as the net coverage improvement to the overall PSS at the national and sub-national levels.

As part of its frame development, the PSS collects data from Early Childhood Centers (ECCs). However, due to budget, scheduling, or other work competing for time or money it's not always feasible to update the frame from every state source. When this occurs, the challenge is to prioritize the processing of the state sources that take into consideration costs, schedule, and frame coverage requirements. This initiative is essential to ensuring that the quality of the estimates and total variance meets both the Census Bureau and customer expectations.



- 122 -


Creation of a Neuron Net for Sample Household Surveys in Urban and Rural Areas by Saleh Movlamov, State Statistical Committee of Azerbaijan Republic

Observation units are characterised by quantitative and qualitative features. Simple random sampling is completely justifiable when values of these variables are unknown. As a rule, classical methods of simple random sampling use the univariate method. By traditional ways of sampling even the benefit of a multivariate sample is very small. On the other hand, conducting a representative sample survey by using methods of clustering, optimisation requires much more financial and time resources.

In sample surveys, two types of errors occur: registration mistakes and errors of representativeness. If observation is organised and conducted correctly, it is possible to avoid registration mistakes. The error of representativeness, however, is unavoidable and depends on sample design, so that its minimisation significantly determines the quality of a survey.

The paper describes a new approach of sampling used for the 2003 labour market survey of Azerbaijan which takes into consideration the peculiarities of the country: inhomogeneous population distribution (one fourth is concentrated in the Capital which accounts for two third of economic potential), nine climatic zones, mountainous regions with villages far away from the district centre, big differences of working and living conditions between towns and villages.

According to economic resources availability and the sample error level, one per cent of the 1.7 m. households were selected separately in urban and rural areas on the base of information of the statistical population register set up on 1999 census results. A neuron net was created by clustering of population census districts (separately for each region):

- Census districts in urban areas were clustered by the number of households. Simple random sampling was applied to select districts and respondents.

- In rural areas, additionally to the number of households, the distance from the regional centre as a main characteristic of the geographical and ecological situation, financial and transport difficulties was taken into account. Then rep-resentative villages and households were selected by simple random sampling.

Up-to-date information on selected census districts (number of households and their addresses) was collected in advance: in cities and urban settlements by interviewers, in rural areas census register data were updated with “household registration books”, conducted in villages.

Weighting was carried out using a step-by-step procedure. Each step was applied separately, so that its contribution to the final weights could be identified. The weights


- 123 -


computed at each step were “normalised”. At each step, the weights were computed from sample values already weighted according to the results of preceding steps.

In conditions of democratisation of society statistics play a greater role. The creation of an information society without realignment of official statistics is impossible. At all levels of hierarchy (micro, intermediate and macro) is a need for statistical information. New approaches of data collection, processing and estimation shall allow organising a distributed information system in order to improve quality of statistical information and confidence in official statistics. Contact: [email protected]

Session 15: Quality Components Mozartsaal Chair: Lucie Laliberté, IMF

Value as a Component of Quality by Carol C. House, National Agricultural Statistics Service, U.S.A.

This paper discusses the concept of the “value” of statistical information as a critical component of overall quality. Value is often neglected in quality assessments of statistics because it is very difficult to measure. In lieu of measuring value, statisticians rely on related concepts of relevance and efficiency. More recently, value has become an increasing important part of the performance assessment of statistical agencies. In this paper, the author explores the concept of the value of statistics to the data users, and examines some potential measures than could be included in a broader assessment of the quality of official statistics.


The Concept of “Accuracy” in the Imputation Process by Carsten Kuchler, Federal Statistical Office, Germany and Martin Spiess, German Institute for Economic Research (DIW Berlin)

Since imputed values are substitutions of unobserved and regularly unknown values, it is rather unclear how to apply the quality concept of “accuracy” to such completed data sets. In a common approach “accuracy” is used in terms of minimising the distances between the imputed and their corresponding missing values, that are assumed to be known like in simulation studies or from administrative sources in cases studies. However, statistical analysis is normally focussed on findings related to the distribution of variables. Hence from a statistical point of view referring to “true values” may be neither required nor desirable.


- 124 -


The first part of this paper examines “true value” concepts of accuracy within the scope of the imputation process. It gives reasons for an opposite approach modelling the distribution of random variables rather than approximating deterministic functions. According to the multiple imputation theory proposed by Rubin (1987) this approach refers to the statistical properties of estimators based on the completed data set. Finally the evaluation concept of a simulation study is presented, that illustrates the distribution based criteria by applying them to data sets with imputed values generated with Artificial Neural Networks.


Use of Proximity Data for the Assessment of Geographical Comparability by Håkan Lindén, Eurostat Michalis Petrakos and Photis Stavropoulos, Liaison Systems, Greece

Geographical comparability of statistical information is an important aspect of statistical data quality. The production of harmonised and therefore comparable statistics is an explicit requirement in most legal texts that relate to the European Statistical System. The process of globalisation in recent years has also given prominence to requests for comparable statistics at the international level. Yet assessing comparability at an even quasi-quantitative way has been a very difficult task.

In this paper we propose a methodology for assessing the geographical comparability of statistical data based on standard statistical techniques for the analysis of proximity data.

More specifically, a common list of metadata items related to survey procedures is selected, which includes survey concepts and practices that affect comparability. Different countries are compared in a pair wise manner in terms of the presence or absence and degree of similarity of each of these items and results are combined using a scoring scheme. This yields a two dimensional symmetric matrix of dissimilarities. Standard statistical techniques such as Multidimensional Scaling and Cluster analysis can then be applied on the matrix. The results from this analysis are either a map or a dendrogram of countries, which can be used to visually identify which country statistics comparisons are more reliable. Moreover, the relative degree of comparability for each country’s statistics can be assessed. The results can also be used to identify common methodological practices that can serve as universal standards in future implementations.

The same kind of analysis can be applied on statistical data on flows. Flows (of goods, tourists, migrants etc.) have the unique property of being measured twice, at the dispatching and receiving ends of the flow. Discrepancies often occur and are


- 125 -


documented with the use of mirror statistics. Mirror statistics can be transformed into dissimilarity matrices and analysed with the same methodology, identifying clusters of countries that have small differences in their statistics. This analysis can help identify methodological aspects which are responsible for discrepancies, thus making it possible to alleviate them. This will not only make statistics in flows more coherent but it might also help to reduce costs by eliminating the need for double data gathering.

Case studies involving ICT usage in enterprises and Foreign trade statistics are presented.


The Impact of the Auxiliary Information in the Estimation of Unem-ployment Rate at Sub-Regional Level: Further Investigations on the Italian Results in the EURAREA Project by Michele D’Alò, Loredana Di Consiglio, Stefano Falorsi and Fabrizio Solari, ISTAT, Italy

This paper focuses on the Italian contribution in the EU-funded EURAREA project and some further analyses originated from the results in EURAREA will be presented.

The EURAREA project intended first to assess the performances of some small area standard estimators such as: direct, generalized regression, regression-synthetic and EBLUP and then to improve the previous estimators along different lines of research.

The assessment context was a simulation study based on replicated samples drawn from census or register database of six NSIs of the countries involved into the project. The purpose was to study the properties of some design-based, model-assisted and model-based small area estimators applied to specific real target variables such as household income, unemployment rate at NUTS3 and finer disaggregation levels. In absence of a common set of auxiliary variables, similar variables have been selected for all the countries in order to compare the simulation results.

As far as concern the development of the standard methods, in this paper we con-centrate on model-based methods that make use of a spatial autocorrelation structure utilizing the distance among areas. A framework similar to that described for assessing the standard estimators has been set up for the evaluation of the enhanced method. The comparison of the two set of results allows us to check out whether the introduction of a random spatial component produced any improvement in the estimation.

The mentioned methods have been applied considering as target variable the un-employment rate of the Local Labour Market Areas (LLMAs). These areas are unplanned domains in the Italian Labour Force Survey (LFS) and are constituted by a set of


- 126 -


municipalities defined on the basis of daily commuting. Two different set of auxiliary variables are compared: the EURAREA common set of covariates and, as an alternative, the information actually available in Italy. More precisely, the auxiliary variables used in this analysis are the covariates used in the Italian LFS and known values at the previous census.


Outlier Detection in Ukrainian Business Statistics by Michel Grun-Réhomme, University of Paris 2, Olga A. Vasechko, Research Institute of Statistics, Ukraine

Generally the national offices of statistics make structural business survey with a stratified sample design. Outlier detection in the sample, before the data analysis, is necessary for theoretical and practical reasons : to verify the hypothesis of homo-genous strata, to have a good estimation and to have the possibility to make an exhaustive survey on these enterprises in the next survey.

Outliers are, by definition, small enterprises that are far from the other enterprises as for as the value of the turnover is concerned. We find different methods (algebraic, graphic and probability).

After a brief summary of these methods, we present here new similar approaches, based on the relative variance, on outliers detection problem in structural business statistics.

We consider the application of this idea to the detection of outliers in Ukrainian structural survey and we compare these approaches with the other methods. A SAS program is used to outliers detection.



- 127 -


Comparing Demographic Variables Across Nations: The Way from Concept to Indicator by Juergen H.P. Hoffmeyer-Zlotnik, ZUMA, Germany

In attitude measurement functional equivalence of the measurement can be controlled by standardized procedures.

In the case of socio-demographic variables the procedure of translation and a con-trolling of translation is not promising because these variables depend on societal systems but not on language.

Behind socio-demographic variables often one can find a set of different national or specific cultural concepts. These concepts depends on national structure and on national institutions and organizations as well as on national law. For the measurement of socio-demographic variables one need knowledge about these different concepts, the cultural and national structures behind the variables and the national law. Basing on national concepts and structures indicators of measurement were formulated.

With this knowledge functional equivalence of the measurement of socio-demographic variables can be reached by harmonizing the variables. In the presentation the way from national concepts to indicators useable in interna-tional comparison is discussed for different variables: income, education, household and race.


Problems of Data Reliability of Sample Surveys of Population Economic Activity in Ukraine by Volodymyr Sarioglo, Research Institute of Statistics, Ukraine

Now in Ukraine state statistics more and more attention is given to the problem of reliable estimation of employment and unemployment, incomes and expenditures, population poverty indicators for regional and subregional (areas, separate cities) levels. State sample surveys, by results of which these indicators are measured, provide mainly the possibility of their reliable estimation at national level. The estimations received for smaller territorial units in many cases are insufficient reliable and demand use of special approaches to specification.

Accuracy of the received estimates of population economic activity level for regions is satisfactory. Approximately same picture is observed at the analysis of the received estimates for employment level. The special attention is given to the indicator of unemployment level for regions. It is the least reliable: there is no region where the coefficient of variation is less than 5%, i.e. the reliability of this indicator is insufficient


- 128 -


as a whole. It is explained first of all by considerably smaller sample size for every separate region.

Methodical approaches to specification of indicators estimations of unemployment level for Ukrainian’s regions on the results of sample survey of population economic activity, which quarterly is carried by State Statistics Committee of Ukraine from 1999, are dealt with.

The main features of existing approaches for surveys that are carried out on the constant basis (monthly, quarterly, annual) are usage of the information from addi-tional data sources on the level of small area or large areas that include it as well as data of previous surveys. Most efficiently additional information is used by the way of application of so-called small area methods. One of approaches to estimator construction for small area indicators estimation is the method of composite esti-mation.

For today a traditional approach to construction of indirect estimator under certain conditions is usage of synthetic estimation method – the estimates are constructed for larger areas that include small areas, under condition that indicators estimation for these large areas have sufficient degree of reliability. At such approach actually it is considered, that the characteristics of large and small areas are the same. At construction of unemployment level estimation in Ukraine as such large areas can be considered, for example, economic rayons – groups of regions, that integrated by proximity of such characteristics as production of industrial output, concentration of industrial potential and labor-power.

At use of small area methods the important problem can become the necessity of estimates coordination, because the estimates can be calculated on the base of survey data (direct estimates) and of some estimator specially constructed for small area.

The investigations in the field of reliability increasing of regional unemployment level estimation are in Ukraine in an initial stage. The development of methods of employment and unemployment statistics estimation on regional and subregional level is beginning more and more actual problem.



- 129 -


Session 16: Corporate Quality Programmes II Großer Saal Chair: Werner Münzenmaier, Baden-Württemberg Ministry of Finance, Germany

Quality Management in Official Statistics in Switzerland since Stockholm 2001 by Markus Braun and Chantal Donzé, Swiss Federal Statistical Office

The purpose of the paper is: first to present the quality management system imple-mented in the Swiss Federal Statistical Office (SFSO) and then to summarize some experiences.

The quality management system presently operational in the SFSO has been planned and developed in 2000, in narrow accordance with the recommendations of the Leadership Group on Quality (LEG). Its implementation followed in 2001; the next two years were devoted to the productive phase.

The system lies mainly on seven characteristics:

- Products: What is meant with product quality? Our definitions refer to international standards (e.g. relevance, accuracy, timeliness, accessibility, comparability, coherence and completeness).

- Processes: Good products stem from good processes. We defined relevant processes (e.g. team coaching, customer relationship, documentation of all relevant production steps, data controlling) and criteria to measure the quality expected by the products.

- Guidelines: A quality handbook is yearly produced to communicate the above mentioned definitions of product quality and process quality standards.

- Advices: We train all teams for quality, on the basis of the quality handbook.

- Audits: We organise annual audits in each organisational unit, in order to evaluate all relevant processes and to recognize critical points which have to be improved.

- Reporting: We summarize and publish the results of these evaluations in an annual quality report.

- Controls: Essential corrections to be done after the audits are again controlled during the following year.

What are the positive effects? In each organisational unit critical points could be recognized, improved and again controlled. This is one of the real and concrete effects resulting from the established quality management system. On the level of the office, it brought transparency and competition, and allows the Director General to implement high level measures.


- 130 -


In 2003, this quality management system was tested upon some other offices, which are producers of statistical data in the field of the environmental protection. These tests showed that it is possible to adapt the system of the SFSO to other organisational units of the Swiss Statistical System.

One important point is to evaluate and continually improve the quality management system itself, while staying in line with the fundamental principles of the ESS. This continual improvement is the main driving force in our quality management system.


Implementation of Quality Programs at the National Statistical Institute of Spain: Some Progress in Improving Relationships with Reporting Industrial Enterprises by Carmen Arribas and Pedro Revilla, National Statistical Institute of Spain

Quality has always been a constant worry at the National Statistical Institute of Spain. Nevertheless, a more systematic approach has been implemented since the Stockholm International Conference on Quality in Official Statistics. The Eurostat framework on quality (LEG on Quality recommendations, Task Force on Quality Indicators, etc.) is being taken into account. A Quality Group has been settled and a “Manual on Good Practices in the Production Process” has been produced and distributed to the staff. Checklist for self-assessment and internal auditing on different statistical projects are been carried out. Some projects on quality, performed for different units, fits into the corporate general Quality Program. A TQM approach used to produce industrial surveys is also presented in this paper. Some progress in improving relationships with reporting industrial enterprises is being achieving, by offering individual tailored data in compensation for answering questionnaires.


Quality in Romanian Official Statistics by Clementina Ivan Ungureanu, Ilie Dumitrescu, Daniela Stefanescu and Sandica Neagu, National Statistical Institute, Romania

Statistics, in general, and particularly official statistics have always placed into the center of their concerns the quality concept and data quality, though its object was limited to data production based on specific methods, always placed in estimation area. As consequence, quality as statistics attribute, was determined by the plausibility degree of these estimates. In other words, an estimates level closer to the real state of art entail the perception of higher results quality.


- 131 -


Statistical quality, or more properly said statistical data quality, was often judged in the light of applying the statistical and mathematical methods able to confer to survey results the guarantee/trust in the achieved statistical product. Nevertheless, in a broader sense covering both the data produced as result of exhaustive and sampling surveys, there were and are still used analysis elements based on the comparability over time and space, assessments through sequential testing of methodological tools reliability, accuracy of data collection, processing, analysis and dissemination of results.

Practically, in the beginning, the whole qualitative elements for each separate stage of the statistical process were not gathered into a synthetic coherent form of quantitative nature, but rather into descriptive statements included in a number of final reports presenting the statistical process and assessing its results. The increasing complexity of the statistical process, together with the process of harmonizing the methods and methodological standards at international level, as well as the IT impact, entailed the need of making available global assessment methods, able to certify the quality of the statistical process, both as a whole and for its various segments and to reveal the gaps, disfunctionalities and errors occurring in different stages, that could affect the final results quality. This desideratum, further becoming imperative, was determined both by the faced challenges and the exigencies of information supplier to continuously improve their quality, respectively by the user requirements, more and more sophisticated and exigent in relation with data timeliness and quality.

A series of central statistical offices, placed in the avant-garde of the demarche towards the assessment of total quality and as a corollary of total quality management, resorted to the easiest, hence the most effective solution. This one consisted of importing or taking over concepts dealing with quality assessments and applying them in economic and business area, further adapted more and more successfully to statistical field. The extension at international level and the proliferation of practically all statistical offices interest towards concerns, approaches and demarches meant to ensure the conceptualization of quality in statistics were favoured by certain well-known cir-cumstances, namely:

- setting up and adoption at large scale of the fundamental principles of official statistics functioning;

- approaching within the UN Statistical Division, the Conference of European Statisticians and Eurostat some major issues envisaging the philosophy of the statistical process in the field of official statistics, such as: ways of meeting user needs, quality versus timeliness, strategic planning, etc., as well as the intense concerns and actions launched by Eurostat in view to apply the quality concept at European level.

As candidate country, Romania could not be left aside the outlined framework, especially because it envisaged the passage from aspirations to concrete applicability


- 132 -


concerns. In this context, within the National Phare Programme 2001 was launched a large-scale project meant to define and implement the methods envisaging the assessment of statistical quality, inspired by the performances reached by a series of statistical offices advanced in this area. The objectives of this project are the following:

- setting up the methodological tools leading to an improved quality of Romanian statistics;

- reviewing the whole national methodologies in view to reach a high compliance level and to create a solid basis for permanently updating the meta-information system;

- setting up a wide framework for communication and transparency of the whole system of statistical indicators;

- ensuring statistics accessibility to all user categories and a wide perception of statistical culture.

The expertise insofar acquired by the National Institute of Statistics and especially the prospective programme envisaging the setting up of an adequate framework for the organization and functioning of the system for applying total quality in Romanian statistics are subject of the detailed paper presented by authors.


Quality Aspects in the Polish Official Statistics by Agnieszka Prochot and Ewa Walburg, Central Statistical Office, Poland

Quality of statistics is a very important issue in the Polish Official Statistics. In connection to adjustment of the Polish Official Statistics to requirements of European Union as well as other international organizations the range of work related to quality has been constantly extending.

Users of Polish Official Statistics need information on quality of statistical data for complete use of such data. Statisticians use information on quality for an analysis of statistical surveys process and for introducing of possible modifications. Measurement and presentation of quality issues is not an easy task.

At present, various forms of reports, describing the surveys process and quality, as well as various ways of reports dissemination are applied in the Polish Official Statistics system.

The Statistical Information System on quality needs a description and a documentation of surveys methodology and statistical processes and a solution of quality


- 133 -


measurement problem. In relation to this, work on quality in the Polish Official Statistics is aimed at:

- Implementation of the quality definition established in the European Statistical System. Attainment of international comparability in the scope of description and dissemination of information on quality.

- Building of the information system on statistical data quality, adaptation of modern methods and standards of collection, storage and dissemination of metadata.

- Providing users with information on main parameters of data quality with regard to international standards and best methodological practice.


Implementation of Quality Management Model – Case of Serbian Statistics by Predrag Jovanovic and Zoran Davidovic, Republic Statistical Office of Serbia

Contribution presents outline of the implementation of quality management in the national statistical institute in the environment burdened with risks and uncertainness, in the country, which is latecomer in the process of the social and economic of transformation.

Reorganization of the statistical system as whole, reconceptualization of basic ideas and notions, as much as radical changes of fundamental statistical methodologies, programs and approach to the statistical surveys needs to be effective and rationally carried out a stabile framework, in other words approach permanently inspired and lead by an integrated idea.

In the Statistical Office of Republic of Serbia (SORS) this integrated idea is understood as the implementation of the Eurostat version of the Total Quality management model - EFQM model. Radical reorientation from production oriented to customer oriented statistical system needs continual monitoring of the needs and requests of the users. On the other hand, quality of the statistical products and services, statistical processes is under significant influence of the quality and satisfaction of the producers and the kind of leading of the processes, which belongs to the domain of “enablers”. These aspects of the quality management are gradually performed in the Statistical Office of Republic of Serbia with help of a number of researches, which have their outcomes, namely quality indicators by separate segments of the EFQM matrix.



- 134 -


Quality of Statistical System in Serbia – Diagnosis and Recommen-dations for Betterment by Predrag Jovanovic, Republic Statistical Office of Serbia

The paper contains analysis of the present state of affaires in the system of the official statistics in the Republic of Serbia. Comprehensive changes of the statistical system in the unstable and dynamic environment with permanently diminishing resources and growing needs for better quality of the statistical products and services, are short descriptions of the current situation of the statistical system and the processes inside of it. Introducing the changes has multidimensional targets:

- reconceptualization and standardization (adoption of standards and method-ologies, of Eurostat, and other systems of statistical standards from international organizations, for example GDDS and SDDS):

- securing relevance and impartiality;

- leveling up of the professionalism;

- enabling of simple access and education of users. Developing of the all modern methods of dissemination;

- permanent diminishing of cost, diminishing of the response burden, and growing “recycling” of the public information;

- securing of complete confidentiality and secrecy of individual data;

- establishment of the close collaboration and coordination in the official statistical system;

- developing and maintaining of international cooperation.

On the basis of a set of indicators for the first time autonomous assessment of the quality of the statistical system as a whole will be performed. That is the way for opening possibilities for monitoring changes of the system more realistically and for more effective intervention in the course of its development.


Quality Management in the Family Income and Expenditure (FIES) of Japan by Katsutoshi Nagashima, Statistics Bureau of Japan

Recently, quality of statistics has been discussed in various meetings and conferences. Eurostat and some countries have proposed the quality dimensions of statistics and some countries also have published quality reports or evaluations of statistical surveys. But the official statistics in Japan may rather in the developing stage in the sense of


- 135 -


current quality. We have adopted quality management process in the execution of statistical surveys but not yet evaluated from the viewpoint of the total quality concept based on the proposed quality dimensions. In the paper, next two contents are reported, an introduction of quality management in official statistical survey in Japan and a comment on the quality of economic statistics.

As for the first point, the quality management process in the execution of the Monthly Family Income and Expenditure Survey (FIES) is reported as the example of official statistical surveys in Japan. The quality management process of FIES consists of three stages, planning and field operation stage, data processing stage and publication stage. The quality management process in the planning and field operation stage aims how to obtain good cooperation between the Statistics Bureau, local governments and enumerators in order to increase respondents rates and to decrease blank fields rates or error rates in the questionnaires. Traditional quality management methods are applicable to the quality in the tabulation process stage. And correspondence to user needs and publication arrangement may be important parts of publication and dissemination stage.

Second is a comment to the quality of economic statistics. The quality of economic statistics, especially basic statistics for GDP, may be evaluated not only based on the quality dimensions of their characteristic purpose but also based on the effect to the improvement of GDP statistics. Background is as follows: Recently, Japanese economy has experienced low-growing economy. And the judgement to the current economic trend (rising or lowering) has become more difficult. This situation has requested the official statistics organizations to produce more accurate statistics. Especially GDP Quarterly Estimation (QE) has been criticized because of its instability, (. i.e. deviance of early estimations from final figures was rather large) and GDP statistics sometimes have not coincided with judgements by economists or business leaders. The FIES originally aims to supply consumption structure data on household level and has been appreciated as the important data source for micro economic analysis. But the FIES also has been used as the primary data for compilation of Personal Consumption estimate in GDP, so the FIES also criticized as the main factor for the instability of QE. The Statistics Bureau started new survey that supplements the stability of FIES results in order to improve the quality of the FIES.



- 136 -


Special Session Forstersaal

Early Bird Invited Session on Multiple Imputation Chair: John Kovar, Statistics Canada

Valid Survey Inference via Imputation Requires Multiple Imputation by Donald B. Rubin, Harvard University, U.S.A.

Valid survey inference means valid randomization-based frequentist inference, in the sense of Neyman -- that is, estimates of population estimands that are approximately unbiased, tests of true null hypotheses that reject at most at their nominal levels, and confidence intervals that cover the true population estimands at least at their nominal levels. Valid survey inference via imputation means that these properties must hold when analyzing an imputed data set as if it were complete. This implies, in general, that the available tools of analysis must be limited to those tools that have been designed for complete-data analysis, supplemented with completely generic tools to "correct" the results of those complete-data analyses on the imputed data. Simple examples will be used to show that if these implications of "valid survey inference via imputation" are accepted, the imputation must be multiple imputation. This conclusion does not necessarily suggest that multiple imputation must be implemented following the guidelines in Rubin (1987, etc.) nor that imputation must be used to address all problems of missing data in surveys. However, I now believe there is ever increasing evidence that these assertions are essentially accurate, especially given the flexibility of modern computing and the constraints of real world survey practice.



- 137 -


Session 18: Mozartsaal Further Topics on Variance Estimation (DACSEIS II) Chair: Ralf Münnich, University of Tübingen, Germany

Variance Estimators for Change: An Evaluation Based on the 2000 Finish Labour Force Survey by Yves Berger, University of Southampton, United Kingdom

There is considerable interest in estimation of changes between two waves of a survey, for example the change in the number of unemployed or in the unemployment rate. We consider three variance estimators of change: the Kish variance estimator (Kish, 1965, Wiley), the Tam variance estimator (Tam, 1984, American Statistician) and a novel variance estimator (Berger, 2004, Southampton). We compare these estimators by a simulation-based approaches based on the 2000 Finish Labour Force survey. Variance estimation would be relatively straightforward if the sample remained the same from one wave to the next. Unfortunately, this is rarely the case, as samples at different waves are usually overlapping sets of units selected with rotation schemes. Three rotations will be considered: rotation with simple random sampling, rotation group sampling and rotation with systematic sampling.


On Combining Registers and Surveys: Repeated Weighting Estimation by Harm-Jan Boonstra., J.A. van den Brakel, Paul Knottnerus, Nico Nieuwenbrooek, and R.H. Renssen, Statistics Netherlands

To combine data from surveys and registers in a consistent way, such that estimated multidimensional tables are mutually consistent concerning their common margins, a strategy of repeated weighting is employed. It amounts to repeated application of the general regression estimator or some other calibration type estimator. For the resulting estimators variance formulas are given, taking the repeated weighting and uncertainty of estimated calibration totals into account. A simulation study with data from a pseudo universe based on a Dutch Labour Force Survey shows satisfactory results for point and variance estimators.



- 138 -


Small Area Estimation Problems in DACSEIS by Kaja Sostra, Statistics Finland, Kersten Magg, and Ralf Münnich, University of Tübingen, Germany

Small area estimation is becoming increasingly important in survey sampling due to a growing demand for reliable small area statistics. Several techniques have been introduced for small area estimation. The estimators to be considered in this study include direct estimators, such as the Horvitz-Thompson (HT) and generalized re-gression (GREG) estimators, as well as indirect estimators, based on synthetic and composite estimators. These estimators are the 8 standard estimators proposed by the EURAREA project (http://www.statistics.gov.uk/methods_quality/eurarea/default.asp). In order to allow for a close to practical application, a large scale Monte-Carlo simulation was conducted on synthetic and real universe data to comparatively investigate the properties of the estimators. The first part of the presentation will demonstrate the effectiveness of the different estimators in the context of different kinds of small areas on synthetic German data from complex household and individual surveys. This will comprise different sizes and compositions of small areas and its influence on the accuracy of the estimators. The second part will pay special attention on models for Finnish unit-level register data.


Software Evaluation with DACSEIS Data Sets by Seppo Laaksonen and Pauli Ollila, Statistics Finland

The final part of the DACSEIS workpackage for software evaluation concentrates on testing based on three samples selected from the DACSEIS universes reflecting the Finnish Labour Force Survey (FLFS), Swiss Household Budget Survey (SHBS) and German Microcensus (GMC, 16 federal states), following the original sampling design. The methodology (in coherence with the DACSEIS simulations) included calculations of the Horvitz-Thompson estimator for all, the GREG estimator (SHBS, GMC) and the calibration estimator (FLFS) with auxiliary variables reflecting the real estimation situation. In addition to full-sample calculations, for each sample one or two of the different nonresponse mechanisms created into the universes were selected. For studying domain results, the levels of study were defined reflecting the reality of each survey. The software to be evaluated in this phase were Bascula, Clan, Poulpe, SAS, SPSS, Stata, Sudaan and WesVar. The user's viewpoint was considered to be important: theoretical and practical aspects and problems occurring in the flow of processing were documented. Technical comparisons were focused on speed and memory with data sets of varying sizes (including a variety of federal states of GMC) in various computers.



- 139 -


Session 19: Quality Indicators I Forstersaal Chair: Michael Colledge, Australia

Standard Quality Indicators Håkan Lindén, Eurostat and Haralambos Papageorgiou, University of Athens, Greece

In October 2002, Eurostat set up a Task Force with interested Member States for the development of quality indicators. The general objective of the Task Force was to identify indicators to measure and follow over time the quality of the data produced in the European Statistics System (ESS). The goal is to have a set of indicators that is representative, easily produced and well understandable. It was also decided that the Eurostat internal quality reports should be the basis for the quality indicators. The Task Force decided to initially focus on producer-oriented quality indicators, and in a second step develop user-oriented indicators.

When developing the set of indicators the special features of the ESS had to be taken into account. The indicators had to be relevant and related to the statistics produced by the Member States, and the statistics compiled by Eurostat for the entire EU. Therefore, the indicators have to be based on mutually agreed terminology, formulae, variables under consideration, statistical reference domains and level of analysis.

In addition, it is proposed to produce the indicators in regular time periods (i.e. each year). This assurance can be achieved, on one hand, by the consideration of the same variables and formulae each year, as already mentioned, but also with the provision with related metadata recording any breaks in time series occurred. De facto, metadata seem indispensable in order to identify the discrepancies of the concepts and measurement of related statistics from different surveys and sources. The existence of quality reports will assist in this task.

This paper presents the set of standard quality indicators developed for assessing the quality of the various statistics produced according to each quality component from the point of view of the producers of statistics. The definitions and the calculation formulae of the most representative ones are provided and related metadata requirements are also depicted.



- 140 -


Continuous Measurement of Quality in the American Community Survey by Deborah H. Griffin, U.S. Census Bureau

The U.S. Census Bureau is committed to the production of high quality survey data to meet the demographic data needs of America’s communities. The American Community Survey (ACS) has been designed to produce demographic, social, economic, and housing data on a continuous basis. The survey uses rolling samples and a mixed mode data collection strategy to produce single and multi-year estimates for areas as large as the nation and as small as census block groups.

Due to the continuous nature of the survey, it is possible to continuously measure quality and make refinements to the program on an ongoing basis. A set of quality and performance indicators have been identified that describe the accuracy of the ACS data and the efficiency of ACS operations. These quality and performance indicators are used by managers to monitor and evaluate the ACS and to identify areas requiring special attention. A subset of these indicators accompany annual data releases, allowing data users to better understand the accuracy of the survey data they will be using.

This paper summarizes the key quality and performance measures used by the ACS, their definitions and interpretations. It further explains the many ways these data are used to monitor the accuracy of data for population subgroups and geographic areas. Examples are provided of how these data have been used to identify and correct real problems. The paper includes documentation of the website that has been created to allow data users to obtain key measures of survey accuracy for the data set that they are using. The website includes tips on how these measures should be used and interpreted by users and provides guidelines to help educate users on the potential limitations of the survey data.

The ACS has been in a demonstration period since 1996, most recently sampling about 800,000 addresses annually. In full implementation the survey will grow to sample 3 million addresses annually. It is critical that the transition to this full sample be monitored to maintain the high levels of quality recognized to date. The established quality and performance indicators will allow us to monitor all operations and promptly identify areas that could be suffering growing pains. This continuous review and attention to quality is expected to enable the ACS to produce high quality data to support the 21st Century’s demographic data needs.



- 141 -


Quality Profile for Structural Indicators by Martina Hahn, Eurostat

Eurostat in co-operation with the European Statistical System (ESS) has developed a quality profile to be attached to structural indicators. The quality profile fully builds upon existing quality reporting in the various domains of European statistics as well as upon the annual quality assessment of structural indicators carried out by Eurostat. Whereas ESS quality reporting so far mainly focuses on the producer perspective and on data sources, the quality profile pursues a different course: It is directed towards the main users of structural indicators, i.e. the Commission services using the indicators in its analysis in the annual Spring Report to the European Council as well as the Council itself. In addition, the profiling of single indicators is supplemented by an analysis of the “set”. As far as possible and relevant a development perspective for single indicators is outlined including a qualitative estimation of the related costs and burden for the respondents. It is used as an input to the political selection process of new indicators as well as to guide the use and interpretation of structural indicators.

The quality profile analyses structural indicators on the basis of user-oriented quality criteria derived from the joint Eurostat and ESS definition of quality. It includes as well an assessment whether an indicator qualifies for integrated policy analysis. For each indicator a grade is awarded reflecting in how far it meets the principal needs of structural indicators with regard to their role in the implementation of the Lisbon strategy. The following user-oriented quality criteria are covered:

- Objective and relevance of an indicator - Data availability (coverage/timeliness) - Comparability across countries - Comparability over time - Contribution to quality of the set/potential to qualify for integrated policy analysis - Other restrictions limiting the use of an indicator in the Spring Report - Overall assessment (quality grade) - Development perspective (including an estimation of the related costs/burden)

The quality profile is issued by Eurostat in co-operation with the European Statistical System making use of existing working structures and quality information. At the same time the quality profile launches a new process of ESS consultation during the political decision making on structural indicators. It is disseminated together with the indicators on the Eurostat structural indicators website, thus supplementing Eurostat metadata by information on the indicators’ “fitness for use”.



- 142 -


Indicators to Define a Territorial Quality Profile for the Italian Consumer Price Survey by Cristina Cecconi, Federico Polidoro and Rosabel Ricci, ISTAT, Italy

The growing attention to the quality of official statistical information requires an effort to guarantee data quality for the users. In particular National Statistical Institutes have to focus not only on the completeness, timeliness, accuracy and clarity, but also on the quality of data collection process. Taking into account the increasing importance of the inflation measures, for the consumer prices survey the aim of improving quality is crucial in order to fulfil policy makers, experts and media requirements. The Italian survey on consumer prices is carried out by 81 chief towns. Such a strong dispersion of data collection, enhances the need of tools to monitor the quality of the process. A part from data check and correction, a relevant issue consists of managing synthetic measures of the accuracy of the survey in different towns in order to compare different geographical areas. The use of quantitative indicators provides a comparable evaluation of the quality of territorial data collection. Focusing the attention on the Italian survey, this paper refers to methodological approach that allows the synthetic treatment of these indicators. The starting point is building basic indicators (absolute number of outlets and monthly observations, numbers of collectors, relative frequency of outlets with respect to population, relative frequency of monthly observations with respect to the outlets, frequency of data errors for different categories, number of monthly substitutions for 12 months, etc.). Then indicators are normalized and standardized to be comparable, taking into account also possible different unit of measure. As third step, normalized and standardized indicators are grouped to provide a synthetic measure of the quality of the data collection for each town. The final aim is pointing out the level of territorial homogeneity ranking the chief towns through different statistical indices identifying the critical areas where intervening to improve the quality of the survey.


Quality in Territorial Statistics: An Experience of ISTAT Regional Office for Sicily by F. Abate, A. Militello and A. P. M. Mirto, ISTAT, Italy

The aim of this paper is to illustrate the main features of quality concept applied in territorial official statistics. The study will be divided in three parts talking about general concept of quality especially used in business survey, secondly investigation of different quality indicators concern social surveys carried out in Sicily, finally quality components in the dissemination activity of the regional office.


- 143 -


The first part of the study will discuss the quality assessment through reports detailing the business surveys and in particular using tourism statistics as an operative example of application. On this subject the paper will discuss seven main components of quality for european statistics (relevance, accuracy, timelines and punctuality, comparability, coherence, completeness) and their application in economic surveys. Moreover the relevance is one of the component strictly linked to territorial dimension considered in a double side: the regional or local partitions and the functional dominia. In the direct statistical surveys in which families and individuals are as reference unit of survey, some indicators able to measure non sampling errors are used and recommended by Istat. These indicators should, above all, indicate causes and avoid replications of non sampling errors during the different phases. An excursus of different methods to monitor the quality in social surveys will be described talking about Labour Forces, Multipurpose Survey and Family Consumption; finally last experience in “Time use Survey” will be exploited. For the first time during the Multipurpose Survey on “Use of time” carried out since april 2002 to march 2003, during training and/or debriefing phase two new monitoring forms have been tested referring to: key role of the interviewer and exam of collective problems faced by municipalities and winning strategies to solve these problems. More in detail some indicators for monitoring family fall and replacement and others related to methodology accomplishment during the interview are performed. These values calculated by interviewer are available in Italy at regional (NUTS 2), provincial (NUTS 3) and municipality (NUTS 5 level) level. Changing and monitoring of user’s need for territorial statistics have addressed dissemination policy by Istat regional office. Data and indicator requirement on small space shares are aimed to face some administrative fulfilments, to optimize action plans for evaluating spin-off on employment and income. Innovations for development of weak areas and local actor acknowledgement in planning and processing have strengthen data availability at small area level. Databanks are an effective tool to face and solve quickly user’s informative problems. Main available Istat databases on thematic areas will be described: population and housing census, industry and service census, agriculture census, population and demographic statistics, foreign trade, short-term economic indicators, social provincial indicator system, cultural statistics site. Moreover current regional office experiences in planning and realising Sicilian small areas databanks will be drawn. To ensure reliable and accurate territorial information specific metadata should address the local user in evaluating sources and methodologies and drawing strategies for planning political actions at territorial level.



- 144 -


Implicit Quality Indices for the Italian Consumer Price Survey by Cristina Cecconi, Ersilia Di Pietro and Rosabel Ricci, ISTA, Italy

As commonly known, the Consumer Price Index (CPI) is one of the most important socio-economic indicators that a national statistical institute can produce. Its pro-duction process is very complex indeed and involves a lot of methodological and operational decisions. One of the most important and delicate among them is the treatment of items substitutions.

Even if the Italian CPI is an index that refers to a fixed basket, different factors make often indispensable to review the sample during the year using quality adjustment solutions, in fact, for example, technological changes, consumers behaviour and market transformations have a great influence in the decisions taken in the compu-tation of a price index.

Since quality adjustments are considered sources of bias and non-comparability in the price indices, Implicit Quality Indices (IQI) seems to be a useful and potentially important instrument for assessing the impact of quality adjustment decisions on a CPI. In fact the IQIs are a measure of the effect of operational adjustment that have been made to the raw price data in order to obtain a price index.

The aim of this paper is applying the current methodology existing on this topic to the Italian consumer price index and analyse the aggregate effect of adjustment to remove non-price effects and obtain the true price change.


An Evaluation of Quality Indicators for two Establishment Surveys by H. Witvliet, Statistics Netherlands

When the different process steps in a statistical process are carried out by different groups, it becomes very important to make measurable agreements about the quality of the products that are delivered. The agreements should contain information about timeliness, the product description and the desired quality.

In search for useful quality indicators for the accuracy, we started with two surveys, for which we obtained data-sets from most process steps. Using the Eurostat dimensions for the accuracy we analyzed the different errors that could occur in every process step. Finally we tried to find quality indicators that could best describe the effects we observed. The most suitable indicators may become a uniform and integral part of new agreements.

Contact: [email protected] / [email protected]


- 145 -


Session 20: Non-Response Studies Spiegelsaal Chair: Peter Ph. Mohler, ZUMA, Germany

Exploring Survey Nonresponse in the UK: The Census-Survey Nonresponse Link Study by Roeland Beerten and Stephanie Freeth, Office for National Statistics, United Kingdom

Like other survey organisations, the UK Office for National Statistics(ONS) has seen a decline in response rates on major UK government surveys during the last few decades. In order to investigate the factors that affect nonresponse ONS has been carrying out research on survey nonresponse following the three last decennial Censuses of the population in Britain (1971, 1981 and 1991). In 2001 the most ambitious study in connection with the 2001 Census was set up.

Addresses sampled for surveys taking place around the time of the Census have been linked with census records at both the household and the individual respondent level. In addition, and for the first time, observations made by the interviewer at the time of the interview, interviewer attitudes, area characteristics and survey design features have also been linked. Thus, a large number of variables are available to investigate the factors that affect both contact and co-operation. This work builds on previous research and extends the work of Groves and Couper (1998) by including individual respondent-level information.

This paper presents an overview of the study and the methods being used by ONS to investigate the relative contribution of different factors affecting nonresponse. We will provide an overview of the various data sources, the linking process, the analysis strategy, and we will present the first results from our analysis.


Analysing Unit Nonresponse Using Matched Census-Survey Records Experiences from the Hungarian Labour Force Survey by Gergely Fraller, Erika György and Roland Horváth, Hungarian Central Statistical Office

The purpose of the presentation is to give a short briefing about the results of the analysis of nonresponse features in the 2003 1st quarter’s Hungarian labour force survey with help of the Census from year 2001. This analysis can be a tool for us to measure and improve the quality of the LFS.

The Hungarian LFS

The target population of the LFS is the resident population in private households. The Hungarian LFS is a multistage, stratified probability sample, consists of 41795


- 146 -


addresses in 2003 1st quarter. According to the rotation system, a selected address remains in the sample through six consecutive quarters, then leaves it. This means that each quarter one sixth of the sample is rotated out. The demographical questions relate to every person living at the selected address, the economic activity questions (labour market features) relate only to persons aged between 15 and 74 years. If the interviewer can’t contact with any respondents at an address, flags the address with one of the six different nonresponse codes. These nonresponse codes relate to the households / addresses.

Matching with Census Data

For the analysis of nonresponse we used the former LFS data (if existed) to the correspondents’ addresses or we imputed data from Census of year of 2001. Matching the two databases has been done on the level of addresses with numerical address codes.

- Problems occurred during matching: There’s no information about the eventual move of the people living at the address in the time of the Census.

- The identifying of the households is problematic. (The Census and the LFS use different household sequence numbers.)

- It is possible, that the Census has only dwelling related data about addresses, doesn’t have person related, because during the interview period of the Census nobody provided data, due to nonresponse.

Analysing Nonresponse

- Used variables / characteristics: Settlement characteristics

- Dwelling characteristics

- Household description - with the characteristics of the head of the household - with the composition of the household

- Characteristics of the interviewer

In case of nonresponse and in lack of former LFS data we impute variables related with persons and dwellings from the Census. Problems: The census and the LFS use different typologies and definitions. To the analysis of the nonresponse the variables originating from the two databases should be recoded, to be comparable to each other.

Methods and Software used

We compare the respondents LFS data with the nonrespondents’ census data, with multivariate analysis methods: regression analysis, factor analysis, cluster analysis,


- 147 -


neural networks, etc. We use to the analysis the Sas System for Windows and the Sas Enterprise Miner software.


Reducing Item Nonresponse: An Experiment to Assess Sample and Questionnaire Design Effects by J.W.S Kappelhof, Social and Cultural Planning Office of the Netherlands

One of the major surveys commissioned by the SCP is the Amenities and Services Utilization Survey (AVO), a longitudinal household survey held every four years, since 1979. The households are asked to answer one face-to-face interview per household and an extensive self completion (SC) drop-off questionnaire for each of the household members above the age of 5. Since response rates of the AVO-waves were gradually getting lower, with an absolute low of 45% in 1991, new field strategies were developed and implemented in the next wave (1995) to improve the response rate and minimize the chances of nonresponse error. These were quite successful because response rates in the subsequent waves were 69% (1995) and 66% (1999). Besides our efforts in getting a higher response rate, questions in the SC-questionnaire needed to be improved, partly because some questions were getting backdated but mostly to reduce the item nonresponse in the SC-questionnaire. After a study of possible reasons for item nonresponse and misinterpretations of questions in previous AVO’s the SC questionnaire has been redesigned and improved in 2003. Furthermore, an experiment was developed to measure the extent of questionnaire design effect, because the AVO is a longitudinal survey and we didn’t want to loose continuity. The experiment was conducted among an access pool because it involved repeated measures and it’s difficult (and very expensive) to reach the same respondents in the field twice and getting them to do a questionnaire again. The experiment consisted of testing two versions (old and new) of the SC questionnaire in an access pool. The ‘old’ version is an exact copy of the 1999 SC-questionnaire, except for minor but necessary changes, for example currency. The ‘new’ improved version differs in lay out, skip patterns, ordering of questions, etc. The respondents in the access pool were randomly assigned to one of three groups. The first group got the ‘new’ questionnaire first and after a period of three months the ‘old’ version. The second group was given the ‘old’ version first and the ‘new’ after three months. A third group was given the ‘new’ version of the SC-questionnaire on both occasions. This design enabled us to assess the effect of the changes in the questionnaire after controlling for learning-, time-, sampling effects and the reliability.



- 148 -


The Impact of Questionnaire Design on Item-Nonresponse Concerning Income Data by Ulrike Timm, Federal Statistical Office, Germany

The collection of income data generally comes along with several severe response and measurement problems. Particularly in the case of self-administered questionnaires official statistics have to deal with an increasing rate of item-nonresponse for income data and thus a serious loss of data quality.

Item-Nonresponse for income questions has several reasons. As frequently noted income questions are part of the „sensitive questions“ respondents are often reluctant to answer. However, the respondents‘ unwillingness to answer such „sensitive questions“ is not the only reason for item nonresponse. Another problem can rather be found in the fact that respondents, although willing to report correct income information, cannot recall the information or unintentionally forget to report income components. This leads to the conclusion that one has to distinguish between intended and non-intended item nonresponse.

This paper mainly focuses on the impact of the second reason of item nonresponse. If respondents are willing to participate in a survey but fail to report complete in-formation, a solution can be an optimized questionnaire design that may help to improve completeness of data, in particular income data.

This paper is based on the results of the German pilot survey for the Community Statistics on Income and Living Conditions (EU-SILC). The survey carried out in 2002 included a multitude of different aspects: Besides a number of conceptional questions such as the integration of different national household budget surveys questions relating to survey design and questionnaire design were analysed in order to achieve an optimal collection method for income data. The evaluation of the questionnaires showed that each of the two different questionnaire designs can have a strong impact on the extent of item non response in income data.

Two different methods of data collection – the data collection in tabular form versus a „classical“ question-and-answer-design – have been compared regarding item nonresponse.

In the tabular questionnaire design, the advantage of a very clear layout goes along with high rates of item nonresponse, whereas question-and-answer-design leads to a much more complicated layout, but at the same time its results show an increasing completeness of income information. The results show clearly that questionnaire design has a strong impact on data quality and consequently should always be in focus.



- 149 -


Analysis of Non-Response in a Regional Survey on the Willingness to Pay for Public or Private Long-Term Coverage by Daniela Cocchi, Enrico Fabrizi and Carlo Trivisano, University of Bologna, Italy

The regional health agency of Emilia Romagna (Italy) is evaluating the opportunity of either introducing a compulsory public scheme to extend long-term coverage or to support the development of private programs for the same purpose.

A sample survey on this topic has been carried out in 2002 in cooperation with the University of Bologna by means of about 1415 personal interviews, by a two-stage design. At the first stage, the largest municipalities have been included in the sample, while the other municipalities have been stratified according to a) two size classes and b) a demand indicator such as the incidence of elderly people on the overall population and an indicator of service availability such as the number of nursing home beds for the municipality health district. Municipalities have been selected with probability proportional to their size. Within the selected municipalities, at the second stage households have been classified in 9 strata according to size and head of the household; simple random samples have been extracted within strata. The assistance of municipalities has been requested for obtaining the list of households stratified according to variables related to the survey topic.

The peculiarity purpose of the survey is such that a high total non-response was expected. For this reason, sampling was performed in each stratum by an inverse sampling scheme until the obtainment of the requested number of interviews in each stratum. The properties of missing at random and selected at random are analyzed according to this sampling plan. A model is built for the analysis of the total non-response and the reasons for non responding.

Contact [email protected]

Research Into the Response Process in Establishment Surveys: A Pilot Study by Tony Hak, Erasmus University, Rotterdam, The Netherlands and Martin Van Sebille, Statistics Netherlands

At four medium size companies that were known as rapid and correct reporters to Statistics Netherlands programs, financial controllers who had completed the annual Production Questionnaire were interviewed in detail about how they had proceeded. These reporters explained for each item of the questionnaire how they had retrieved the reported data, or rather how they in many cases had not sought or retrieved these data in their records and had either estimated or guessed the reported amount, or had not responded to that item at all. A considerable amount of data error was found. Causes of data error ranged from incorrect definitions of concepts and categories, through


- 150 -


computing errors and recording mistakes, to lack of time and / or interest and outright decline to spend time and cost on finding the correct data.

This pilot study was not only successful in demonstrating the relevance of a more in-depth study of the actual response process in a larger number of companies, it also showed that two different methods of data collection would be useful and feasible in such a larger study. Firstly, the pilot study itself showed the feasibility and productivity of in-depth focused interviewing. Financial controllers who had spent 2 to 14 hours themselves on the completion of a questionnaire were willing and able to spend another 2 to 3 hours on discussing in detail and per questionnaire item how they had collected or guestimated the data they had reported. Second, the feasibility of participant observation was discussed. It appeared that only one of the interviewed reporters had completed the questionnaire in more than one sitting. The other three had done this in one sitting of 2 to 6 hours. It was agreed that it would be possible to arrange the attendance of a researcher at such a response sitting to observe the details of the response process.

The results of this pilot study, and the recommendations based on it, will lead to a larger study on the response process in establishment surveys in which data will be collected by means of (prospective) participant observation where feasible and by means of (retrospective) focused interviewing when participant observation cannot be arranged or fails to generate the desired data.


Data Analysis of Recreational Fishing Survey in Finland: Do Non-Respondents go Fishing? by Pentti Moilanen and Anssi Ahvonen, Finnish Game and Fisheries Institute

Recreational fishing is a very common hobby in Finland. Nearly 40% of the population, which means nearly 2 million people, go fishing at least once a year. The total catch amounts to some 40 million kg. It means 27 % of the total catch in Finland.

Recreational fishing is a very multiform phenomenon in Finland, because of many possibilities for fishing. There are close to 30 000 lakes larger than 4 hectares in Finland, many rivers, large coastal waters and an excellent archipelago in south-western Finland for recreational fishing. Also fishing licence system is multiform including landowner’s licence system, licences for right to fish with spinning rod in any county regardless of the landowners and also everyman’s right for ice fishing and to fish with rod and line.

The data for recreational fishing statistics are collected every other year by postal questionnaire using a sample drawn from the population register maintained by the Population Register Centre. The sample comprises about 6000 household-dwelling


- 151 -


units. The sample design is stratified sampling. For the estimation, a weighting factor is formed for each statistical unit. The weighting factor is formed from the inverses of the inclusion probability and response probability of the sampling unit and from the calibration weight. The bias caused by non-response has been corrected using the homogeneous response group model. The non-response rate has been about 35%.

When compiling the 2002 statistics post-sampling was conducted for those who did not respond to the postal questionnaire using telephone interviews. The results of the non-response analysis is described closer in the paper.


Income-Nonresponse in Household Surveys – An International Comparison Using the SOEP and the BHPS by Jörg-Peter Schräpler, Ruhr-University Bochum, Germany

Many validation studies deal with item nonresponse and measurement error in earnings data. In this paper we explore motives of respondents for the failure to reveal earnings using the German Socio-Economic Panel (SOEP) and the British Household Panel Study (BHPS). These Long-On-Going Panel studies collect socio-economic information of private households. We explain the evolution of income nonresponse and demonstrate the importance of a discrimination between refusing the income-statement or don't know in both surveys. A primary finding in this analysis is that in both surveys refusals and don't knows relate to different respondent characteristics. The description and the estimates in the multivariate analysis presents evidence for our first hypothesis, that the respondent characteristics for refusal and don't know responses differ markedly. Respondents who do not know their answer are in both surveys mainly females and in low or middle occupational states and respondents who work irregularly. Respondents who refuse their answer are mainly in high occupational positions in the SOEP and males in the BHPS. This finding is important for the interpretation and reduction of item nonresponse. Many studies try to predict item nonresponse with respondent characteristics but fail to use response categories such as refuse and don't know. The resultant conclusions are ambiguous and hard to interpret, which may be the reason for the inconsistency in this area.

The second relevant finding is that survey respondents fall on a cooperation continuum (Bollinger/David 2001; Burton et al. 1999; Loosveldt 1999b). Respondents who refuse to answer the sensitive questions (e.g. income) because of privacy concerns are often notwhole-hearted survey co-operators and have misgivings about the whole process. They drift to the end of the cooperation continuum and are harder to persuade to participate in the following wave.


- 152 -


The third relevant finding is that interviewer/area has a strong effect on a respondent's propensity to give refusals and don't know responses. We find differences between both surveys. The identifiable systematic effect in the SOEP: female interviewers get noticeably more don't knows and refusals than males. For the BHPS we are able to use an experimental subsample. For a quarter of the full sample in wave 2 is an interpenetrating sample design experiment implemented. The estimates of cross-classified multilevel models suggest predominantly interviewer cluster effects rather than area cluster effects.

A fourth relevant finding in this study is the influence of the presence of respondent's partner during the interview situation. We find empirical evidence that a present partner reduce the "don't knows". One explanation is that a partner may help in the case of retrieval problems.

Overall we find in this study many similarities regarding to nonreporting income in both surveys. We can reasonable assume that our findings and the explanation of respondent behaviour might be relevant for most other household panel surveys.


Session 21: Data Processing II (Editing & Imputation) Museum

Session in Honour of John Charlton Chair: Svein Nordbotten, Svein Nordbotten and Associated, Norway

Automatic Edit and Imputation for Business Surveys by Ton de Waal, Statistics Netherlands

Statistics Netherlands participated in the EUREDIT project, a large international research and development project on statistical data editing and imputation that lasted from March 2000 till February 2003. In the EUREDIT project, 12 institutes from 7 different countries were involved. The main goals of this project were the development and evaluation of new and currently used methods for automatic data editing and imputation. In this paper we describe the two business surveys used in the EUREDIT project and the evaluation experiment that was carried out. We sketch the general approach applied by Statistics Netherlands on the two data sets. A prominent role in our approach is played by so-called edits rules. Correct data satisfy all edit rules (with a very high probability). Edit rules are specified by subject-matter specialists based on statistical analyses and their experience with these kinds of data. Our approach is based on the subdivision of automatic edit and imputation into three steps, namely the error localisation step, the imputation step, and the consistent imputation step. The error localisation step entails determining which values in a data set are erroneous or suspicious, given a set of edit rules for that data set. In our approach we apply different methods for systematic errors, such as errors where the respondent reported in incorrect units, and for random errors, such as errors where the respondent by accident


- 153 -


typed in a digit too many. The systematic errors are easy to correct once identified. The random errors are detected by applying the so-called Fellegi-Holt paradigm of minimum change that says that the data of a record should be made to satisfy all edit rules by changing as few fields as possible. Detected random errors are usually difficult to correct. They are therefore first set to missing. Subsequently, these missing values are, together with the originally missing values, imputed during the imputation step. The imputation step entails imputing the missing data in a data set in the best possible way. In the EUREDIT project we have used a combination of deductive imputation, multivariate regression imputation using the EM-algorithm, and (ratio) hot-deck imputa-tion for the imputation step. Finally, the consistent imputation step entails slightly adjusting the imputed values for an imputed data set such that all edit rules for that data set become satisfied. The consistent imputation step is carried out by solving a linear programming problem. We provide results of our approach on the two evaluation data sets, and draw some conclusions.


Improvement of Data Editing Processes by Elmar Wein, Federal Statistical Office, Germany

Statistics Germany has investigated a lot of effort in the development of a new data editing concept during the last years. The new concept will be implemented in the second quarter of 2004. So the first part of the contribution will be a mix of meth-odological aspects of the concept and their realisation. An overview will be given on the main characteristics of the planning process of data editing, e.g. considerations on the transmission of user demands on statistical data or collection of important preconditions of data editing processes and the respective IT-tools. Dependent on the available time of the contribution's presentation and it's permissible length a second example of the data editing planning process may be a description of the methodological considerations on the specification of edits which will be supple-mented by an overview of a developed IT-tool.

The first part will be continued by the description of the main aspects of the per-formance of data editing, e.g. the development of standardised process chains, the use of selective editing and imputation methods. A short report will be given on simulation studies of selective data editing methods.

Opposed to the first part a second part of the contribution will deal about current research on the development of a selective data editing method. The contents of the second part depend strongly on current methodological work. So possible aspects of this part may be the computation of indicators on the basis of comparisons between raw and plausible data, and data editing specifications or achieved improvements of a selective editing methods.



- 154 -


Comparing Model-Based and Traditional Methods to Deal with Systematic Unity Measure Errors by Marco Di Zio, Ugo Guarnera and Orietta Luzi, ISTAT, Italy

One important problem in Official Statistics is the treatment of non-sampling errors in the Editing and Imputation (E&I) phase. The complexity of the investigated phenomena and the existence of several types of non-sampling errors make the E&I phase a very complex and costs consuming task. In the E&I context a common error classification leads to define two different error typologies: systematic errors and random errors.

In the family of systematic errors, one that has a high impact on the final estimates and that frequently affects data in statistical surveys measuring quantitative characteristics (e.g. business surveys) is the unity measure error times a constant factor (e.g. 1,000). This error is due to the erroneous choice, by some respondents, of the unity measure in reporting the amount of questionnaire items. Relating to this type of systematic error, the critical point is the localisation of items in error rather than their treatment. In fact, once an item is classified as erroneous, the best action (treatment) is uniquely determined and consists in a deterministic action recovering the original value through an inverse action (e.g. division by 1,000) neutralising the error effect.

The unity measure error is generally tackled through ad hoc procedures using mainly graphical representations of marginal or bivariate distributions, and ratio edits (i.e. bounds on ratios between variables). With traditional approaches, the error localisation problem is not only complex, but also costs and resources consuming.

In terms of complexity, two elements are to be considered: identifying errors is difficult when the actual variable values have a high variability that makes part of the "true" values distribution overlapping to part of the erroneous values distribution; furthermore, traditional techniques allow no more than pair-wise comparisons between variables. The result of these elements is the risk of misclassifying a certain amount of units.

Relating to costs, the elements to be taken into account are: 1) the complexity of designing and implementing ad hoc deterministic procedures aiming at automatically identifying such types of errors; 2) the resources spent in manually checking observations, including also those having low probabilities of being in error (over-editing).

In this paper we discuss an alternative approach based on mixture modelling. The main advantages are that it allows a multivariate analysis reducing the risk of mis-classification, and that it provides elements that can be used to optimise the trade off between the automatic and interactive treatment of this kind of errors, i.e. between costs and accuracy.


- 155 -


This method has been evaluated through an experimental application on the 1999 Italian Water Survey System (WSS), a periodic total survey collecting information on water abstraction, supply and use. The results obtained by the mixture model approach have been compared with those obtained through the current editing procedure.


Imputation and Improvement of the Quality of Provisional Figures by Christian Harhoff, Statistics Denmark

Parts of the education statistics compiled by Statistics Denmark are collected from the educational institutions, which report data relating to a specific date on courses of education over the year. The information on basic education cannot be regarded as final data, as a real picture of the status of students is not known to the educational institutions. A real picture will not be provided until data are reported again one year later. Statistics Denmark is presently making attempts to redress this situation by imputing values for those students whose status is not known. This is performed by applying Hot-Deck, with donors from the corresponding students of the previous year. When the real status of the students is known in connection with the survey conducted one year later, it is possible to evaluate the quality of the imputation. In the discussion paper, the results achieved during the imputation and weighting are compared with each other and with the results from some final statistics.


Statistical Editing and Imputation for Social Data by Frank van den Eijkhof, Utrecht University, The Netherlands

In statistical analyses, improving the quality of datasets by cleaning raw data is mostly aimed at identifying missing values (item nonresponse) and finding univariate outliers, after which some imputation technique is performed. A dataset consists of records; each record contains all values of a respondent. Official statistical bureaus do have more information; they investigate the possible and impossible combinations of values in records. The resulting constraints are called edits. An example of an edit for censuses is, that a person cannot be married and be of an age lower than 15. Statistical editing now consists of changing values in records such that no more univariate outliers exist and that all records are made consistent with the edits. Combining statistical editing and imputation is difficult. Statistics Netherlands currently uses an approach based on three consecutive steps. In the first step, the error localisation step, the implausible fields are identified using the Fellegi-Holt paradigm. This paradigm says that the set of values of a record to be changed by statistical editing must be minimal. In the second step the found minimal set and the missing data are imputed. Finally, in the third step the imputed values are changed such that the record is consistent with


- 156 -


the edits. However, the changes in this third step may be large and the resulting dataset may thus have a distorted joint distribution. We will propose an alternative approach that combines the second and third step. This new approach imputes values that are consistent with the edits. Since the resulting imputation does not need adjustment to the edits afterwards, the joint data structure is probably better preserved than in the current three-step approach. We will compare the results of the new approach with those of CherryPi.


Assessing Editing and Imputation Effects on Statistical Survey Data by Giorgio Della Rocca, Marco Di Zio and Orietta Luzi, ISTAT, Italy

In large-scale surveys conducted by National Statistical Offices (NSOs), the treatment of non-sampling errors represents a critical problem because of its impact on final results. Measuring and documenting the quality and the effects on data of data processing activities performed during the survey process, including editing and imputation (E&I), has become a mandatory requirement in many NSOs.

When evaluating E&I procedures, several aspects are usually taken into account: accuracy, timeliness, cost, respondent burden, and so on. In this context, accessible information can be acquired in different ways at the different stages of the E&I life-cycle: before the application of the E&I procedure, for assessing the quality of the procedure itself, i.e. its capability of correctly deal with errors and meet the specific survey quality requirements; during E&I, for monitoring performance aspects and tuning the E&I procedure, as well as to assess the quality of input data (error characteristics and possible error sources); after the data processing, for documentation and survey management purposes (monitoring over time the changes in the E&I impact on data, collect information to be used in future survey repetitions for improving data and survey quality).

In the area of assessing E&I activities, the EUREDIT project1 established a general framework for the comparative evaluation of E&I methods. Exploiting and integrating the EUREDIT research activity, in the paper we propose a set of quality criteria and the corresponding statistical measures for the assessment of E&I procedures in different contexts. In particular, these measures have been then implemented in the generalized tool IDEA - Indices for Data Editing Assessment, in order to better support survey managers in the evaluation task by simplifying and standardizing it.

1 The EUREDIT Project was funded under EU Fifth Framework research program (www.cs.york.ac.uk/euredit/).


- 157 -


Depending on the evaluation aim, IDEA indicators are computed by comparing different sets of data: when assessing the effects of E&I on original survey data (e.g. for monitoring, tuning or documenting the E&I activities), the raw and the corresponding final or clean (edited and imputed) survey data are taken into account, while when evaluating the quality of E&I methods, the final data are compared to the corresponding true ones (i.e. actual variable values for sampling units, either real or simulated).

IDEA indicators allow the evaluation of E&I processes at two different levels: at “high” level, i.e. by considering all variables and units subject to E&I, and at “low” level, i.e. by considering single items or subsets of items and units. At high level, IDEA produces the standard quality indicators required by the Istat Information System for Survey Documentation (SIDI) relating to the E&I phase. Al low level, the proposed indicators allow the evaluation of E&I procedures with respect to three performance criteria: i) effects on elementary data; ii) effects on marginal and joint distributions; iii) effects on data relations. Different indicators have been defined for each one of the above mentioned performance criteria, taking into account the type of surveyed variables (nominal, ordinal, continuous).



- 158 -


Quality Control System for the Editing and Imputation of the 2001 Italian Population Census Data by Fernanda Panizon, ISTAT, Italy

Italian Population Census which has been carried out by ISTAT in 2001: after the data collection phase, after the quantitative controls on coverage of collected data, after determining the exact amount of census units (resident population living both in private households or in institutional ones, number of households, dwellings and buildings) the issue of data quality process shifts to the editing and imputation step.

To perform this crucial phase a comprehensive informative system for data processing has been projected in order to satisfy the informative needs of the data quality control system. Integrity and consistency of data and procedures has been assured by preparing a large relational database. In order to easily interrogate data during computer processing a parallel data warehouse in which data are stored at different phases and a series of instruments have been organised in a system of control.

The process of data validation begins with preliminary analysis on gathered data, to evaluate the kind and distribution of errors, to decide about the best edit and imputation strategy (stochastic and/or deterministic) and about the most convenient ranking of imputation step to assure coherent final data. Before starting to draw the plan of correction it is crucial to prepare a system of checks, queries and reports on data to be examined and to train a staff of revisers to perform this task. Comparisons with external information and statistics at the macro-level, identification of incompatibility at the micro level, definition of special rare sub-population to be monitored during data processing are all taken into consideration to separate sys-tematic errors from random ones and to choose the appropriate action for validating data.

On the basis of the preliminary analysis a first design of the correction plan is defined for each kind of units or for groups of variables. To test validation procedures another set of controls has to be organised, (enlarging the previous system) to monitor their execution (length, computational performances) and to analyse statistical outcomes in terms of: lacking of inconsistency, validity of cleaned macro data, frequencies of imputations, crossed distributions of modifications on variables. During the test phase some sets of data are submitted to the procedures which are modified and tried until they seem to be adequate to run on every subset of data.

The system of data control which works during the effective validation procedures can re-use indicators and reports already prepared for previous test analysis, but it is also improved by new checks on the final look of statistical tables and indicators prepared for dissemination.


- 159 -


Each phase of data processing is controlled and these controls will feed a specific archive that iscalled the “register of quality” in which the main quality indicators are selected and stored, such as the results of checks on row data (amount of detected errors), the results of controls and of the corrective actions (percentage of modified data, differences in original and final distributions of variables) and other synthetic evaluations on final outcomes. The measure of data quality by means of the indicators from this register is useful not only during data validation phase, but is important to obtain the whole set of quality indicators of census data to be provided in final issues and to record data quality documentation.


Imputation of Economic Data under Several Linear Restrictions by Caren Tempelman, Statistics Netherlands

Missing data is a prevalent problem in survey research. Several methods have been developed to deal with missing data. With regard to item nonrespons, imputation is mostly used. Some well- known imputation methods are (random) regression impu-tation and hot deck imputation. However, when we are faced with economic data standard imputation techniques will not suffice, due to the fact that economic data consist of many logical linear constraints, such as the fact that profit has to equal total revenue minus total expenses. If two or more of these items are missing standard imputation techniques will be able to provide imputations that preserve the distribution of the data, but will most likely not be able to provide imputations that also satisfy the linear restrictions on the data. This means that the imputed values need to be adjusted in order to satisfy these linear restrictions and this will distort the distribution of the imputed values, thereby distorting the distribution of the data. We were therefore looking for an imputation method that immediately satisfies the linear constraints while preserving the distribution of the data.

An additional issue to this problem is the fact that the structure of the restrictions is rather complex. Data items can be present in several linear restrictions. For example, the different reported expenses have to add up to the total expenses, but the latter also has to equal total revenue minus profit. This means that all missing items need to be imputed simultaneously immediately satisfying all linear restrictions on the data. We will discuss an imputation method we have developed that can handle this type of data.



- 160 -


Session 22: Fieldwork Backstage

Chair: Denise Lievesly, UNESCO

Setting the Standard for Quality Control in the United States Decennial Census by Jennifer W. Reichert and Rodrick J. Marquette, U.S. Census Bureau

This paper presents progress to date on development of a Quality Control Standard for U.S. Census Field Operations. In past U.S. censuses, development of quality control for census field operations has been a somewhat ad hoc process. As new production operations began development, the associated quality control was custom-developed for each particular operation. This method of managing quality can cause critical aspects of quality control to be overlooked. Development sometimes can focus too much on implementation issues specific to an operation, and not enough on what a comprehensive quality control program should include. This problem can be exacerbated by the aggressive schedule always associated with censuses. The Quality Control Standard will provide structure for development and implementation of quality control measures for all data collection methods and technology. The Standard would ensure that all aspects of quality control are incorporated into the plan for the operations, covering such things as: standard requirements and documentation for field quality control, setting quality expectations and goals, developing field materials for quality control operations, and monitoring and reporting quality control results. Having a standard in place will ensure that quality control development follows an established framework which will allow timely planning and provide reliable methods for measuring quality.

For example, the 2006 Census Test is planned to include all census operations that will be used in the 2010 Census. Having a quality control standard in place for the planning of the 2006 Census Test will allow us to use an established set of guidelines and templates to develop and implement the quality control procedures and materials. Using guidelines and templates will reduce the amount of time and resources necessary to develop the quality control operations for the 2006 Census Test because the development wouldn’t be „from scratch.“ Rather, we will be able to start with pre-existing documentation, and our efforts can focus on refining and improving process quality.



- 161 -


Quality Information at a Territorial Level: Towards an Integrated Approach by Marco Ricci, ISTAT, Italy

The network of regional offices (RO) in Istat (the Italian NSI) provides – together with other institutional functions – the coordination of the data-collection phase in many surveys, such as general censuses of population, economic activities and agriculture, sample surveys on households and enterprises, administrative sources on demography and building.

This coordination includes tasks of training the local employees and surveyors, providing technical and organizational assistance, monitoring the field-operation work progress, carrying out quality controls, making institutional interventions towards local authorities.

Putting into effect such different actions, RO acquire a lot of information about the level of cooperation of the actors involved in the field operations. At the same time, in order to performing the mission of territorial coordination as best as possible, they receive support information from the central structures of Istat about the output quality of previous surveys. The flows from both the centre and the “field” represent a considerable basis of quality information, that each RO responsible make use for. The aim is to plan the better use of resources for ensuring the quality of the field operations, giving a priority to the critical points of the survey network.

The standardization of the activities linked to the management of quality information as above referred is a target to which the RO pay attention now. That requires a systematic analysis of such information according different aspects, such as ways of getting it, consistency, informative value, actions to be implemented and so on. Afterwards, an arrangement as territorial quality information system should be planned, as an integration to the quality information systems available at a central level, mainly the so-called SIDI (Information system about surveys’ documentation).

Based upon such premises, this paper provides a preliminary analysis of territorial quality information available in the Istat RO, with a transversal approach throughout the surveys. A basic conviction in this approach is that quality information referred to a survey can often deserve a value for coordinating the field operations in other surveys, yet absolutely different. Information will be classified in four main groups of indicators, concerning: reference lists, general accuracy of field operations, raw data accuracy, field operations’ timeliness. Examples of indicators for each group should allow pointing out the main topics in getting, organising, using and further developing quality information for coordinating the data-collection phase.



- 162 -


Monitoring and Evaluating Fieldwork Efforts in the European Social Survey by Michel Philippens and Jaak Billiet, Catholic University of Leuven, Belgium

The European Social Survey (ESS) is a high-profile academically-driven multi-country survey covering over 20 nations. Given that large country-variation in nonresponse rates can have detrimental effects on the comparability of survey estimates, the ESS has tried to strive for the achievement of a target response rate of 70% in all countries. In order to reach this goal, the ESS insisted that all countries would adopt a minimum set of fieldwork standards. In addition, countries were recommended to use a range of response-enhancing measures such as respondent incentives and refusal conversion procedures. Setting appropriate standards and recommending increased efforts will however not guarantee that these are actually implemented by the appointed survey organisations. This requires a further stage of monitoring, evaluating and giving feedback. The ESS has tried to facilitate these three steps by building the gathering of nonresponse and fieldwork data into the design phase of the survey. This was done by developing a uniform contact description form that resulted in a standardized call record dataset in each of the participating countries. In addition, a short questionnaire on fieldwork procedures was administered from each of the national coordinators. The collection of nonresponse and fieldwork data in the ESS will shortly be discussed in the first section of the paper. In a second part of the paper we will exemplify how these call record data can be used to monitor fieldwork efforts and to assess whether these efforts were effective in increasing response rates and reducing nonresponse bias. In our examples we will focus on monitoring and evaluating contact patterns and refusal conversion strategies. Based on the results of our analysis we will try to formulate some suggestions that are designed to maintain and improve fieldwork standards where appropriate.


Characteristics and Identification of Faked Interviews in Surveys - An Analysis by Means of Genuine Fakes in the SOEP Survey by Joerg-Peter Schraepler, Ruhr University Bochum and DIW Berlin, Germany and Gert G. Wagner, DIW Berlin and Berlin University of Technology (TUB), Germany

To the best of our knowledge, most of the few methodological studies that analyze the impact of faked/falsified interviews on survey results are based on “artificial fakes” generated by project students in a “laboratory environment”. In contrast, panel data provide a unique opportunity to identify data which are actually faked by interviewers. By comparing data of two waves, most fakes are easily identifiable. Moreover, fakes can be discovered in the SOEP survey because interviewed persons receive a gift by mail after the interview has taken place. If a household receives a “thank-you” letter


- 163 -


and gift for having supposedly given an interview, the intended respondent is likely to contact the fieldwork organization, which then becomes aware of the faked interviews.

Because interviewers know that panel respondents will be interviewed again over the course of time, clever interviewers will not fake these interviews. Thus the share of faked interviews will be lower than for survey research consisting only of cross-sectional “snap-shots”.

The SOEP provides a rich source of faked interviews because it is built on several sub-samples, all of which provide the opportunity to identify fakes. In fact, over time, falsified data on 90 households and about 184 respondents have been detected. Almost all of them occurred in the first wave of a subsample; some in a second wave. The share is less than one percent of all interviews. These data are used to analyse the impact of fakes on survey results. The major result is that one percent of faked interviews has no impact on the mean and the proportions. But in very rare, exceptional cases there may be a serious bias in estimates of correlations and regression coefficients.

Besides the “conventional” statistical tests of stability and consistence used to detect fabricated data, we also apply an unconventional benchmark called Benford’s Law, which is used by several accountants to discover frauds. It turns out that it is difficult to detect fabricated survey data. However Benford’s Law could become a new instrument for quality control of interviewers.


Preparation of an EU Training Package on Certification of Causes of Death by Monica Pace, Silvia Bruzzone, Stefano Marchetti, Francesco Grippo, Simona Cinque, Gennaro di Fraia, Marilena Pappagallo, Simona Pennazza, Giuseppe Sindoni and Luisa Frova, ISTAT, Italy

Quality in official statistics may be improved at several different steps of data pro-duction; however in case of causes of death it is of primary importance to obtain good certificate completion by physicians or other professional in charge of such task. These data are the basis of mortality statistics and the improvement of data quality at its source is an effort in which ISTAT is now involved. The project on the preparation of a European training package on certification of causes of death is the result of a call for tender launched by EUROSTAT and acknowledged to ISTAT. This project, 18 months in duration, is included in Community statistical Programme 2003-2007 and it is part of the work on establishing EU-wide comparable data on public health and on its determinants. The project includes the participation of 40 countries-regions (EU, EEA/EFTA, CEEC) and the precious collaboration of experts from seven different European countries.


- 164 -


The objective of this project is to provide a common training background for European physicians in the completion of the medical part of death certificate so to improve the overall certification quality and achieve better comparability on mortality data among Europe. The philosophy of such an ambitious project is to provide a set of generalized products, satisfying international standards such as WHO guidelines and certificate, as well as EUROSTAT recommendations. An easy customization step to adapt the generalized product to national needs will be required for each product after its release. The products developed, or still under development, are:

a) a technical report where information on death certification practices and existing training tools have been collected by means either of a survey sent to 40 countries-regions or review, respectively;

b) a reference manual with information on mortality data and their importance for statistical purposes, definitions used in death certification practice, instructions on how to complete the medical part of the death certificate, a collection of case histories with examples of completed death certificates, a glossary, a list with suggestions on how to improve the specificity of the reported diseases or conditions;

c) a web-based training tool based on e-learning methodologies and useful for train on death certificate completion. This tool should allow to practice on multiple choice tests and death certificates completion based on case histories too in a tutorial section; moreover it provides a presentation text section, and some tools such as glossary, links and a list with suggestions on how to improve the specificity of the reported diseases or conditions. Each page of this tool is dynamic, based on an Oracle database, thus to reduce the necessary IT skills during the national adaptation phase;

d) a ready-to-use leaflet reporting the necessary information and examples of good certification practices both for natural and external causes of death.

The main features of these products, the status of the project, and some ideas to facilitate the next national adaptation steps will be shown.

Contact: [email protected], bruzzone@istat


- 165 -


New Methods for Measuring Quality Indicators of ISTAT’s New CAPI/CATI Labour Force Survey by G. Giuliani, M. G. Grassia, L. Quattrociocchi and R. Ranaldi, ISTAT, Italy

In the last two years, the Italian Institute of Statistics has invested huge resources to transform the labour force survey in compliance with EUROSTAT standards while revolutionising the survey technique. From a PAPI technique in the current labour force survey, with interviewers of the municipalities which entered the sample, we have passed to a mixed technique: CAPI for the first family interviews and CATI for the confirmation interviews following the first, with the direct management of interviewers in the new survey on the part of ISTAT.

The internal survey network has made it necessary to evaluate interviewer performance in time, as well as their impact on survey quality. But what indicators should be used to monitor the continuous data collection process? The first aim has been to construct an indicator plan comprising indicators useful to monitor the data collection process and changes in time of the network of interviewers, as well as the panel survey in its entirety, and therefore the impact of its course on sampling error. This indicator plan has been constructed in line with international standards for official face-to-face and telephone surveys (AAPOR1 Standard Definitions). The second aim has been to analyse these indicators using multidimensional analysis techniques, in order to ensure the comprehension of the “quality” phenomenon, a complex issue in itself.

In particular for the network of interviewers, the issue was to evaluate and represent the work of n interviews working in different geographical locations and with different personal characteristics, observing p outcomes of interviews in t time intervals. The aim was therefore to evaluate if and to what extent differences in the survey’s quality indicators can be ascribed to personal and/or geographical characteristics.

To this purpose we applied STATIS method and we proposed new developments of it, which, on the basis of the main quality indicators, allows us to estimate the latent factors that measure structural differences and dynamics through time. The results show the formation of clusters of units with common personal and geo-graphical characteristics which progressively diverge.



- 166 -


The Electronic Questionnaire in ISTAT’s New CAPI/CATI Labour Force Survey by M. G. Grassia, F. Pintaldi and L. Quattrociocchi, ISTAT, Italy

With the aim of complying to EUROSTAT standards, two years ago ISTAT began to design a new labour force survey. After an experimental phase, the survey is now being run in parallel with the current one and will substitute it completely during 2004. Innovations in the new survey concerned both contents and technical aspects. In particular, concerning the survey technique, we passed from a PAPI technique handled by interviewers of the municipalities which entered the sample, to a mixed technique: CAPI for the first family interviews and CATI for the confirmation interviews following the first.

The questionnaire for the new survey is particularly complex and consists in a general opening part which collects information on the family’s demographic characteristics and 12 sections of which 10 are repeated for each family member. The use of an electronic questionnaire has allowed for automatic branching, the activation of online help upon request of the interviewer, the interactive coding of open items using a search engine for certain key variables (such as economic activity and profession), the introduction of soft and hard control mechanisms of range and coherence (with the possibility of conciliating on the spot), managing confirmation items for waves following the first. All of this has helped simplify the interview and therefore significantly improve data quality. To evaluate such improvement, we have compared the off-line check programme of the old survey with the on-line check programme of the new survey, analysing the number of partial non-responses, the number of edit activations, the number of conciliations in the new survey and the number of soft controls purposefully not corrected (which shouldn’t therefore be imputed automatically). The significantly better results for the new survey have confirmed the correctness of the choice of an electronic questionnaire as data collection instrument. Comparisons between the qualitative standards of the two surveys will be developed in order to analyse longitudinal aspects of the sample, examining in detail the effects of a confirmation questionnaire in the waves following the first and the use of a CATI technique.



- 167 -


Session 23: Weighting and Calibration Leibnizsaal Chair: Carl-Erik Särndal, University of Montreal,Canada

Non-Response Adjustment of Household Surveys by Jelke Bethlehem and Bary Schouten, Statistics Netherlands

Non-response is a recurring problem in household surveys in many countries. Response rates of surveys of Statistics Netherlands often vary between 50% and 60%. Part of the sampled households or persons end up being non-respondents due to a variety of reasons. The majority of non-respondents are refusers. However, in order to be able to refuse one needs to be contacted first. Non-response also arises from the lack of contact. Contact is more likely to fail for persons or households that are difficult to reach in general, but a household may also have moved or there may be a lack of sufficient interviewer capacity in certain districts.

Research shows that non-response is usually selective. Respondents and non-re-spondents differ at various demographic characteristics. When non-respondents answer differently to important survey questions, or even worse non-response is a result of a lack of interest in survey topics, then estimators for population charac-teristics may be seriously biased. To avoid a substantial negative impact on the quality of survey results, often a non-response adjustment procedure must be carried out.

For non-response adjustment purposes Statistics Netherlands has a large amount of background information available. This information originates from registers and other administrative sources. These background variables can be linked directly to both respondents and non-respondents. Current research is aimed at using as much of this auxiliary information as possible, and finding the proper models to do that.

In our presentation we will describe current practices at Statistics Netherlands in the area of non-response adjustment, the state of the art with respect to non-response research, and point at future plans for research.



- 168 -


Participation in CATI Surveys: Traditional Nonresponse Adjustments Versus Propensity Score Matching in Reducing the Nonresponse Bias by Emilia Rocco and Nicola Salvati, University of Florence, Italia and Monica Pratesi, University of Pisa, Italia

In CATI surveys, several stages of the survey process can be identified. Participation at each stage can be analysed as dependent on the outcome of the previous stage. For a specific type of CATI surveys - e.g., a survey on general population with sample selected from the population register and telephone numbers of the selected units obtained from a merge with a telephone directory - the following stages can be identified: pre-recruitment (merge with the telephone directory), contact (call scheduling and call backs), interview.

The initial sample is often modified as a result of the pre-recruitment stage. The identification of the telephone number of the selected individual is not always successful: when the number is not identified the individual is substituted with another with the same characteristics and a known telephone number. This updated sample is contacted by telephone following the rules of the call scheduling. The units not contacted after a fixed number of tentative telephone calls (missed contacts) and the units who refuse the interview (refusals) are often substituted in order to maintain the planned sample size. The underlying assumption is that survey substitutes do not differ systematically from the individual they replace. However, since substitutes are, by definition, telephone subscribers contacted by phone and respondents they may differ from the nonrespondents and may bias survey estimates.

This paper uses propensity score matching methods to find out possible substitutes without requiring that substitutes be perfect replacement for nonrespondents and to compensate for the self-selection bias occurred in the above stages of a CATI survey on individuals. In the paper, the pre-recruitment, contact and interview stages are distinguished in a CATI survey targeting the general population aged 20-74 of the Municipality of Florence. Using a parallel population register of the Municipality of Florence, the probability of being in each stage of the CATI survey is estimated, based on a vector of covariates measured both in the register and in the survey. The aim of the research is to illustrate for each of the previous stage the performance of the propensity score matching adjustment relative to other commonly used nonresponse methods (individual substitutions and weighting by estimated individual probabilities of response) in order to evaluate survey results and their accuracy, develop adequate weighting procedures and strategies for reducing nonresponse.



- 169 -


Mixed Mode Survey a s a Means of Improving Response: Lessons from the Panel of Ex-Participants in French Employment Policy Targeted Programs by Karl Even, French Ministry of Labour, France

The Employment Policies Unit has conducted three surveys in 2002 and 2003 focussed on participants in 13 big targeted programs implemented by the Ministry (sample of 60 000 people). These populations are famed for the difficulty to locate : many of them are young, movable, vulnerable, have mobile phone rather than fix phone, and are quite reluctant to answer a survey which aimed to get information on the problems they have encountered (are encountering) on the labour market. This leads to a generally high non-response rate. Moreover, the survey occurred quite a long time after the event for which they have been sampled.

This convinced us to use a mixed mode data collection in order to minimise non-response. The scheme was based on phone interviews (CATI) as a first step, and on face-to-face interviews carried out by an interviewer and self-completion postal questionnaire as resorts in a second and third step. The paper will focus on the advantages of such a strategy : raised quality by improving the accuracy of response through two aspects :

- The volume : the strategy has significantly increased the response rate and led to reduce the confidence limits.

- The quality properly: it reduces the non response error in the sense that the people we reached in the second and third rows are homogeneous and quite different as they tend to respond differently to the major interest variables (in catching people who are reluctant to phone survey or for who it is impossible to conduct the interview in a limited time (non French speaking people)).

The paper will analyse the impact on accuracy figuring out what would have been the main estimators without the two resort modes, and measuring how far switching to two additional modes of data collection is, or is not, an effective means of reducing non-response error based on some variables (to be chosen).

The communication will answer these questions through a description of the popu-lations who responded to face-to-face and self-completion modes (logistic regression including demographic variables and variables of interest asked in the questionnaire). It could also show the response patterns by type of response mode (if some stylised facts appear) and analyse the differences among the modes. The paper will include a study of an extra survey we have conducted with the face-to-face interviewers on usefulness and relevance of face-to-face interviews and pre-letters sending. The purpose was to get information from interviewers on how the interviewed people took in the pre-letters, and on the reasons they did not answer to telephone.


- 170 -


The communication will address a balanced message by presenting problems gen-erated by using mixed mode survey :

- Statistical problem : the calibration procedure becomes more complex with a more-than-two-steps data collection mode.

- Cost problem : the face-to-face step brought 10% of the total number of re-spondents but represents 27% of total cost. In counterpart the mail survey step brought 9% of total number of respondents but represents 3% of total cost.


On Adjustment for Coverage Problems in Short-Term Business Surveys by Annika Lindblom and Lennart Nordberg, Statistics Sweden

Coverage problems, due to differences between sampling frames and target popula-tions, may arise in all types of surveys and for many different reasons. This paper will focus on some common types of coverage problems in short-term business surveys.

The following approach is common for short-term business surveys at many statistical agencies, including Statistics Sweden. A sample is drawn from a frame at time 0, say, and for practical reasons this sample is kept over a number of production rounds.

Let be U (

t)the target population at time t and suppose that the sample is representative of the current population U

(0). Since business populations are very changeable, the sample may be less representative for the current target population U(τ)at time τ, τ > o. The objective of this paper is to explore the consequences that this approach may have in terms of bias in estimates, and also to discuss possible means to reduce such bias. Results from empirical studies of several surveys at Statistics Sweden will be included for illustration.



- 171 -


Improving Timeliness of Short-Term Business Statistics through State-Space Modelling of Preliminary Survey Data by A. Lamberti, A. Naccarato and A. Pallara, ISTAT, Italy

Timeliness of statistical information, i.e. the length of time occurring between its availability and the reference point of the event or phenomenon it describes, is a key issue for improving survey process quality. Timeliness is a particularly critical component of quality for producing short-term business statistics at NSI’s of European Community, as each Member State has to meet the standard quality requirements of the Regulation No. 1165/98 about terms for transmission of the results and detail of the information provided on short-term statistics (STS). The usual practice in repeated surveys conducted by NSI is to report advance (preliminary) estimates of the variable of the interest for the current reference period, obtained through either a special sampling design or a sub-sample of survey respondents, and final revised estimates for the previous period. It is then important to consider the problem of constructing improved preliminary estimates for the current reference period, that will, in turn, lead to improved estimates of changes. There may be, hence, an interplay and/or a trade-off between timeliness and other dimensions of quality, such as accuracy and coherence, and finding a balance among these factors is a critical aspect for yielding STS of good quality.

Let us consider a repeated survey, with a sample of tN observations at time t on the

variable of interest tiy , to each of them a vector of known auxiliary variables 1p

tix×

is

associated. The estimation method proposed in this paper views the problem of constructing improved preliminary estimates at time t as a prediction problem. It uses dynamic linear models (DLM) for repeated surveys (Patterson, 1950; Harvey, 1984; Bell and Hillmer, 1990) using only the information available at a time t* – with t* between t-1 and t – and considering all the respondents beyond t* as non response. Assuming that t* is a fixed time point between two consecutive reference period in a panel survey, it is then possible to distinguish between two subsets of respondents in each survey and subsequently re-formulate the DLM, taking into account information from preliminary respondents at time t (respondents up to t* ) and using available data on all respondents from previous survey occasions.

Let ( )TtN1tTt t

y,,yy K= be a realization of a random vector

tN1

TtY×

related to the

known matrix ( )TtN1tt tx,,xX K= in the following way:


- 172 -


1Nt

1pt

pNt

1Nt

ttt

uXY××××

+= β (1)

where 1pt×β is an unknown coefficient vector,

1Nt

t

u×

is the error vector with

( ) ( ) ( ) K,2,1,;0,,0 =≠===××

ltltuuEVuuEuEtttt

NN

Tlt

NNt

Tttt

In the case of panel surveys, it can be assumed that the coefficient vector 1pt×β in model

(1) is not fixed, but rather follows a Markovian representation

ttt T εββ += −1 (2)

where T is a (p×p) matrix of known coefficients and tε is a random vector with

( ) ( ) ( ) ( ) lt0uE,LuE,0E,QE Tlt

Npt

Tttpp

Tlt

ppt

Ttt

t

≠====×××

εεεεεε

Equations (1) and (2) represents the state-space formulation of DLM (Harvey, 1984).

Let us also consider a partition of the tN observations into two subsets: the first one,

s, includes the preliminary respondents while the second, r, includes the late

respondents. Considering these two subsets the variance-covariance matrix tV can be

partitioned into:

=

−×−×−

−××

×)()()(

)(

ttttttt

ttttt

ttnNnN

trrnnN

trs

nNntsr

nntss

NNt VV

VVV

In the proposed approach an MLE estimation of 1pt×β is obtained through the Kalman

filter and then estimates of Yt and of the sub-matrices

)( ttt nNntsrV−×

, ttt nnN

trsV×− )(

,

)()( tttt nNnNtrrV

−×−for late respondents are obtained using all available data for surveys

previous to t.


- 173 -


We will present an empirical application of the proposed method to monthly data on the index of industrial turnover in Italy.


Composite Estimators for the Labour Force Survey by Francesca Inglese and Monica Russo, ISTAT, Italy

Each quarter, ISTAT develops Labour Force Survey (LFS) based on a two stage sample (municipalities, households). Sample households at quarter t , divided in four rotation groups, has in common: 2, 1, 2 and 1 rotation groups with samples households

respectively at quarters: 1t − , 1t + , 3t + , 4t + , 5t + .

We studied the performances of a class of AK composite estimators (AKCE) in com-parison to those of post-stratified ratio estimator (PSRE) adopted in the LFS. Com-parative analysis is referred to quarterly level and quarter-to-quarter change population parameters, concerning the employed (EMP) and people looking for a job (PLJ) of the region Piemonte.

Let tY , the number of persons of the population having a certain attribute at the

quarter t . AKCE of tY (Gurney, Daly, 1965), is

( ) ( )( ) ( )( ) ( )1,1

4

3,

2

1,

��21�2141�−−

==

++

−−++−= ∑∑ tttAK

ggt

ggttAK YKYAKYAKY δ

where g,tY is the PSRE of tY based on the rotation group g ( )4321 ,,,g =

observed at the quarter t , 1−tAKY is the AKCE of 1−tY and

( ) ( )∑=

−−− −=4

3111 21

gg,tg,tt,t YYδ

is the estimator of quarter-to-quarter change ( )1−− tt YY based on the two rotation

groups common to quarters t and 1−t , besides, A,K [ ]1,0∈ are constants: if 0=A

AKCE turns into the simple composite estimator (SCE), tKY ; if 0=A and 0=K ,

tAKY identifies itself with the PSRE, tY , based on all the four rotation groups.


- 174 -


The assumptions are: (i) ( ) ( ) constant�Var�Var .., == YY gt ∀ t and

4321 ,,,g = ; ( ) 0�,�Cov ,, =ftgt YY ∀ t and fg ≠ ; ( )gdtgt YY ,,�,�Cov + referred

to g to “ d ” quarters of distance is equal to

( ) ( )( ) ( )( ) ( )..21

,

21

,,,�Var�Var�Var�,�Cov YYYYY dgdtgtdgdtgt ρρ == ++

( 521 ,,,d K= , with 2≠d ), where dρ is the coefficient of correlation between

values observed on the persons common to the surveys t and dt + . Under the above assumptions, it is possible to demonstrate that:

( ) ( ) [ ]γβα�Var�Var 2 ++= AAYY ttAK

( ) ( ) ( ) ( )( )[ ]55

44

33

112 221411 ρρρρ KKKK −+−−−=α−

( ) ( ) ( )( )( ) ( )( )[ ]424

554

112 211221 ρρρ KKKKKKKKK ++−++++−−=β−

( ) ( ) ( ) ( )( )[ +−+−+++−++−=γ−

3233

123212 23121211 ρρ KKKKKKKKK

( ) ( )( ) ]525

425 211 ρρ KKKKK +−+++ ;

( ) ( ) γβα�Var�Var 2 ++= AAYY ttAK ;

( ) ( ) γ�Var�Var =ttK YY ;

( ) ( ) ( ) γγβα�Var�Var 2 ++= AAYY tKtAK ;

( ) ( ) ( ) WYYY AKttAKtAKttAK ⋅=−=∆ −−�Var��Var�Var 1,1

( )( ) ( ) ( )( )[ ] ++−+++−+= KAKAAKKWAK2222 2112 γβα

( )( )AK −+− 221 1ρ ;

( ) ( ) ( ) ( ) ( ) WYYYY tttttt ⋅=−=−=∆ −−�Var2�Var��Var�Var 11,1 ρ ;


- 175 -


( ) ( ) WWAKttttAK =∆∆ −− ,1,1�Var�Var ;

( ) ( ) WWKttttK =∆∆ −− ,1,1�Var�Var

where WK is derivable by WAK putting 0=A ;

( ) ( ) WW KAKttKttAK =∆∆ −− ,1,1�Var�Var .

dρ ( )5431 ,,,d = , expressed like mean values of all those calculable in the period

April 1995-ottobre 1998 (Casciano, Falorsi, 1999), are equal to: 0,94; 0,91; 0,89; 0,88, for EMP and 0,61; 0,52; 0,45; 0,41, for PLJ.

In the table below are indicated the values of the efficiency of the AKCE and SCE as regards to the PSRE, related to optimal values of A and K, determined on the basis of

every possible combination of the values 9020100 ,;;,;,;K K= and

120100 ;;,;,;A K= : the section A is referred to quarterly level estimators; the

section B is referred to quarter-to-quarter change estimators.

AKCE SCE

Optimal values of Optimal values of Variab-les

A K Efficiency

K Efficiency

A. Quarterly level estimator

EMP 0,8 0,7 0,54 0,6 0,60

PLJ 0,9 0,4 0,84 0,2 0,91

B. Quarter-to-quarter change estimator

EMP 0,2 0,4 0,93 0,5 0,94

PLJ 0,1 0,1 1,05 0,1 1,05

For optimal values of A and K for the AKCE and of K for the SCE, we observe different values, because of they are some constant whose values depend on the different character of the variables of interest. Therefore, if different values of A and K were used or of K, the additive law of the estimates would not be guaranteed; with the purpose to assure such law it is opportune to individualize values of A and K or of K common both to the variables of interest considered, both to the type of estimate, of quarterly level and of quarter-to-quarter change.

Contact: [email protected] , [email protected]


- 176 -


How to Make Representative Samples – Microdata Calibration by the Minimum Information Loss Principle: Concept and Practice by the ADJUST Computer Program by Joachim Merz and Henning Stolze, University of Lüneburg, Germany

Microdata as samples have become increasingly important for economic and social analyses. One striking problem with almost any practical analysis of microdata, microdata as a cross or longitudinal sample from any kind of a survey or within microsimulation modelling, is to achieve representative results.

In this paper a consistent solution of the microdata adjustment (calibration, grossing-up) problem - that is to achieve representative results by re-weighting a sample of microdata to fit aggregate control data - is presented based on the Minimum Information Loss (MIL) principle. Based on information theory this principle satisfies the desired positivity constraint on the weighting factors to be computed. For the consistent solution which simultaneously adjusts hierarchical microdata (e.g. household and personal information), a fast numerical solution by a specific modified Newton-Raphson (MN) procedure with a global exponential approximation is proposed.

In addition to the theoretical concept and its solution, the paper presents the easy to use but also powerful (with unconstrained mass data processing) computer program ADJUST (http://ffb.uni-lueneburg.de/adjust).

Practical experiences for large microdata sets in a pension reform analysis with e.g. more than 60.000 households and 240 restrictions simultaneously to be achieved within the Sfb 3 microsimulation model show that this MN procedure was able to rather largely reduce the computional expenses by 75%. The available efficient PC-computer program ADJUST is also successfully applied in a described microsimulation analyses of the recent 1990 German tax reform investigating the impacts on market and non-market labour supply within the formal and informal economy, and in a recent firm microsimulation analysis on explaining factors of successful firms in the German engineering industry. Last but not least, the adjustment of the recent Time Budget Survey of the German Statistical Office and a refined adjustment of the German Socio-Economic Panel for professions and self-employed will illustrate the adjustment importance in the survey business within statistical offices and in recent socio-economic research.



- 177 -


On Calibration Estimators of a Finite Population Variance by Parimal Mukhopadhyay, University of Qatar

Deville and Sarndal (1992) proposed a class of estimators of a finite population total from a re-weighting approach that addresses the issue of reduction of variance. By calibrating sample weights so that the value of the estimate is equal to the population total for each auxiliary variable $x_1, \ldots, x_p$, they obtained a class of estimators - calibration estimators of a finite population total. Generalized regression estimator of Sarndal (1980) is found to be a member of this class. In this paper, following their approach, calibration estimator of a finite population variance has been derived. It is found that this estimator is a member of the class of generalized predictors of a finite population variance derived by Mukhopadhyay (1990). Asymptotic properties of this estimator have been investigated.


Nonresponse in Multi-Phase Survey by Paavo Väisänen, Statistics Finland

Multi-phase surveys are a type of complex survey which are rather commonly con-ducted by national statistical institutes. The use of auxiliary information to increase accuracy by using, for instance, calibration techniques still adds to the complexity of the survey, and often the need for several weights makes the data difficult to utilise. Nonresponse may arise at all survey phases, and the design of survey phases affects the nonresponse rates. Large nonresponse is harmful in two different ways: it reduces sample size and thus increases the standard error of estimates and it causes bias because respondents may differ significantly from nonrespondents.

In multi-phase surveys, different data collection methods may be used or data collection may take place in sequential time points, where data can be collected by telephone or face to-face interviews, mail questionnaires, diaries or internet inquiries. The Harmonised European Time Use Survey (HETUS) is an example of a multi-phase survey. Both households and individuals serve as survey units and data are collected by face-to-face interviews concerning households and individuals, and in addition, the HETUS includes diaries kept for two days and week diaries for employed individuals. In the HETUS, four types of nonresponse can be observed. First, cluster nonresponse or household nonresponse, where all persons in a household are missing; second, unit nonresponse, where some persons in a household are missing but the household is, nevertheless, taken into the survey. The third type of nonresponse is missing diaries, where one or both diaries or the week diary are missing. The fourth type is item nonresponse, where answers to one or more variables are missing at the household or individual levels, or a diary is only partially filled in.


- 178 -


The Finnish Time Use Survey was implemented according to the Eurostat guidelines for HETUS. The same person had to answer to the household interview, personal interview and keep two one-day diaries and a week diary. Response burden was rather high which was seen in nonresponse rates which were higher than in other surveys and increasing in all of the phases. The nonresponse rate at the household level was 36%, at the individual level 39%, and finally, diary nonresponse was 48%. The week diaries were kept only by persons who were working during the survey week. The nonresponse rate of week diaries was 17%. If the unit nonresponse rate of employed individuals is assumed to be the same as the whole sample, the total nonresponse of the week diaries would be 65%.

The estimation procedure exploited information from statistical sources and ad-ministrative files, such as the Population Register and the Jobseeker Register. Cali-bration techniques were used to balance the estimates to correspond with the auxiliary data. Weights were calculated in several steps which took into account the sampling design, survey phases, the household level information on the population, the household level nonresponse adjustment, the calibration of demographic data, the allocation of the sample to diary days, and the balancing of the diary weights.


Session 24: Improving Processes Großer Saal Chair: Carmen Arribas, National Statistical Institute, Spain

Improving Efficiency and Process Quality - Statistics Austria’s Experience by Peter Findl, Statistics Austria

In order to improve efficiency and process quality Statistics Austria carried out comprehensive process analysis. The paper informs about a project realized within the Directorate Population Statistics to elaborate basic information on specific statistical processes with a view to replace traditional organisation schemes by a more flexible and efficient process- and project-orientated organisation, as well as to improve process quality.

All statistical projects of the Directorate Population Statistics were analysed in detail to obtain information on personal resources needed for the individual statistical processes and sub-processes on a monthly basis. Simultaneously, areas of weaknesses concerning efficiency and quality of data collection, data processing and publication processes were identified and finally, recommendations for organizational, technical and methodical improvements were suggested.


- 179 -


The results of this project set the basis for re-design of statistical production processes and for a new planning and management of all statistical projects of the directorate. Improvements, e.g. concerning the data editing processes or the acceleration of the publication processes, have already been realized. The experiences and the results gained from this pilot project were used for reorganisation projects/process analysis within the other directorates of Statistics Austria.

Contact: [email protected].

Developing a Handbook on Improving Quality by Analysis of Process Variables by Nia Jones, Welsh Assembly Government, United Kingdom and Daniel Lewis, Office for National Statistics, United Kingdom

Improving process quality has a direct effect on product quality. Key process variables are factors that can vary with each repetition of the process, and have a large effect on critical product characteristics. Analysis of these variables should lead to adjustments of the process, whose effects on product quality should be monitored.

Statistical processes fall into two categories. For some processes, general quality improvement tools such as process flow maps and cause-and-effect diagrams are suitable, but the task of identifying variables is difficult and limits the approach. For the other category of processes, it is relatively straightforward to identify measurable process variables. Some areas of National Statistical Institutes are already measuring and analysing variables for their processes, and have experienced quality improvement. However their analyses are often ad-hoc, without emphasis on such concepts as process stability and capability.

A project team (consisting of members from the Greek, Portuguese, Swedish and United Kingdom National Statistical Institutes) has developed a handbook on improving process quality by analysis of process variables. The handbook consists of guidance on how to identify, measure and analyse process variables, following the plan for continuous quality improvement from Morganstein, D., and Marker, D.A. (1997). Several examples of applying this approach to statistical processes are described. The work was supported by a Eurostat grant following a 'Leadership and Expert Group on Quality' recommendation.



- 180 -


Using Real-Time Process Measures to Improve Data Collection by David A. Marker, Westat, U.S.A.

Many statistical organizations have come to realize that measuring process variables is key to improving quality. This involves identifying key process variables, measuring them, and acting on them in a timely fashion. The goal is not to analyse them after a survey is completed, but to get feedback on these variables when decisions could still be made that would improve the quality of the current round of data collection. The importance of process measures is seen by the current effort of the European Statistical System to develop a “Handbook on improving quality by analysis of process variables” as part of the LEG Recommendation #3.

This presentation will provide examples from surveys that collected real-time process measures and used them to improve quality of the existing data collection. Examples will come from a variety of surveys and organizations, including a new longitudinal survey of businesses in the United States. The presentation will also discuss the questions that survey managers must ask at the beginning of a survey to increase the opportunities for getting real-time process measures.


Benchmarking and Quality Management Processes Relating to the Production of Consumer Price Indices: The UK Experience by David Fenwick, Office for National Statistics, United Kingdom

The UK Office for National Statistics took part in a benchmarking exercise with the Australian Bureau of Statistics that covered the production of the UK Retail Prices Index and the equivalent index in Australia. The exercise covered all stages of production from the collection of prices to the compilation of the index itself. The paper describes the benchmarking process and identifies the critical factors that contributed to its success. It also considers the lessons that were learnt in the process and provides practical guidance on how to maximise the benefits of benchmarking for the organisations involved. The process of benchmarking was an incremental process. Critical to the success of the exercise were the presence of agreed operational guidelines and an agreed set of objectives that were to be measured by a pre-de-termined set of performance indicators. Lessons learnt during the day to day conduct of the exercise included the need in benchmarking to focus resources on critical areas where discussion and further investigation will be most productive. These critical success factors are described in detail and the general themes that emerge for successful benchmarking are presented.


- 181 -


The paper describes the benchmarking process within the context of the overall quality management system for the Retail Prices Index. ISO 9001 has provided a key element for a formal quality management system for the monthly production process of the index and is an integral part of an ongoing programme designed to maintain and improve quality. The paper also describes some of the benefits that have accrued more generally from using a formal Quality Management System and conversely addresses some of the disadvantages. Business performance is considered in the context of the ultimate quality of the consumer price index.


How the U.S. Census Bureau Uses the Capability Maturity Model as a Roadmap to Process Improvement by John M. Bushery, U.S. Census Bureau

Quality has always been an important component of the U.S. Census Bureau’s products and services. The Census Bureau’s Quality Program works to support and advance initiatives to improve the Bureau’s processes and the quality of the Bureau’s products and services. A key strategy of the Quality Program is to partner with areas of the Bureau in their process improvement efforts.

One example of that partnership is between the Quality Program and the Census Software Process Improvement (CSPI) Program, which works to help the Census Bureau improve its software development processes. The CSPI Program bases its process improvement approach on the Capability Maturity Model (CMM). The CMM is an industry standard (similar to ISO 9000) developed by the Software Engineering Institute (SEI) of Carnegie Mellon University, used to assess and improve software development processes.

The SEI developed the CMM by taking the best ideas of the experts in Total Quality Management (TQM), including Deming, Juran, and Crosby, and applying them to the software process. The main strength of the CMM is that it describes an evolutionary improvement path from an ad hoc, immature process to a mature disciplined one. The characteristics and structure of these levels show organizations what they must do to move from one level to the next. And although the CMM originally was developed to improve software processes, the CMM methodology applies to all types of processes.

This paper will:

- Briefly describe two of the Census Bureau’s programs that promote process improvement and better quality: the Quality Program and the Census Software Process Improvement (CSPI) Program.


- 182 -


- Describe the Capability Maturity Model (CMM), summarize its history, and explain how it relates to process improvement and Total Quality Management.

- Describe the characteristics of Level 1 and Level 2 of the CMM.

- Describe in detail the Quality Assurance Key Process Area of the CMM and how the Census Bureau implements quality assurance in some of its processes and discuss how these methods can be used for all types of processes.


Reorganising the Process of Producing Data to Improve the Quality of the Surveys: The Work and Study Experiences of Upper Secondary School Graduates Survey by Andrea Cutillo, Simona Pace, Valentino Parisi and Fabio Massimo Rottino, ISTAT, Italy

This paper illustrates the actions undertaken on specific phases to improve the quality of the information coming from the survey on the study and work decisions of the qualified in the upper secondary school. This survey is carried out by the Italian National Institute of Statistics (ISTAT) to analyse the choices taken by the people who got a diploma in the upper secondary school three years before.

This survey is carried out every three years concerning a single cohort of students, interviewed with the CATI technique three years after the diploma. The students who got the diploma of the upper secondary school in 1998 have been interviewed in 2001. The survey is a two stages sample survey, stratified in the primary units. The schools are the primary units, the students who got a diploma in a certain year are the secondary units. The survey is divided in two periods: in the first ISTAT applies to the schools to get information on the list of names and telephone numbers of the students. Once extracted the sample, in the second period ISTAT applies to the students to interview them.

The main innovations introduced in the survey concern the following points.

1. Obtaining the list of the names to extract the sample. In the past surveys ISTAT asked for the list of the names through paper forms. This involved long times to revise and to digit the data. To improve the process of collecting data we have introduced a way to obtain data from the schools on the web. This fact has also permitted a retraining of the employees assigned to this duty: they have changed from an activity of revision of paper forms to an activity of monitoring the phases of the survey using multimedia instruments.

2. Harmonisation of the questionnaire on transition The questionnaire of the survey has been modified considering the other ques-


- 183 -


tionnaires on transition (regarding university graduates and university qualified) to permit an integrated view of the information by the three surveys.

3. Realisation and integration of the processes of training and interviewing The interviews are carried out by an external firm. Training the interviewers is a crucial moment for the quality of the information. For this reason we take particular care of the procedures and of the materials for the training of the interviewers (e.g. training on the course of studies, on the codification of the professions, briefing, debriefing) and for the monitoring during the survey (producing daily reports and summarising reports).

4. Treatment of the data We have realised probabilistic integrated and generalised check strategies ex-perimenting with several strategies of correcting data (cyclic, contextual and hierarchical), also carrying out a valuation system of the performances of the check strategy in connection with the reconstructed information. We have also started an experiment to use an automatic classification of the courses of university studies instead of the manual classification.

5. The diffusion of the results To permit a faster diffusion of the results, we have replaced the single volume with thematic fascicles. To have a larger diffusion of the information, we also publish the fascicles on the web.


Development of a Handbook for Quality Management under a Decentralized National Statistical System in Korea by Sulhee Kim, National Statistical Office, Korea

The Korean statistical system can be considered to be decentralized. At present November 1, 2003, 453 statistics has been approved in accordance with the Statistical Act, including 336 statistics compiled by 61 central and local governmental agencies and 117 statistics compiled by 71 non-governmental agencies. Under these circumstances, Korea National Statistical Office (NSO), which is the central government authority in charge of statistics, has the responsibility of coordinating the national statistical services as well as compiling 52 fundamental statistics and disseminating integrated statistical information. Under a decentralized system, redundancy and omission of statistics and inconsistency between statistics compiled by various agencies may occur. Moreover, since survey managers in most statistical organizations except several major ones may have not been adequately trained in survey methods, they may cause quality difference in each statistics.

As prescribed in the Statistical Act, the Commissioner of NSO could request a statistical agency to implement, suspend, change or improve the statistical matters. Recently, the


- 184 -


National assembly standing Committee has consistently demanded for quality improvement of the official statistics compiled by non-NSO agencies rather than those produced by NSO. Accordingly, there is a need of a type of handbook, which can be used for checking quality elements in production processes. As known to national statistical institutes, Statistics Canada Quality Guidelines (1998) is a well-structured and substantial work. However, there is a need of a handbook, which takes the Korean circumstances that may be considered different from Canada as well as European countries. The handbook should be understandable by the staff that may be non-professionals in statistics.

The NSO Korea has been conducting several preparations prior to introduction of a national quality management system for official statistics. As a part of these prepa-rations, in 2003, a handbook for quality management of official statistics has been developed. The handbook can be regarded as the first version of the handbook that would be revised via repeated examinations in the future. It would take magnificent time to make a handbook of each statistics. Therefore, as official statistics can be classified into several categories according to compiling methods, the handbook is divided into four volumes; complete survey (census), sample survey, analysis statistics and administrative statistics.

The development of the handbook has been conducted in partnership with a research team, which was belonging to the Korean Statistical Society. First of all, several quality guidelines of national statistical institutes from other countries such as Canada, UK and New Zealand were examined from the angles of structure, contents, reader’s knowledge level, degree of details and so on. The readers of the handbook were determined as all the staff in charge of producing official statistics in other agencies as well as those in NSO. For example, the volume of sample survey consists of the sections for eight processes including survey planning, sample design, questionnaire design, training interviewers and data collection, result analysis, user service, post-maintenance and quality maintenance. Each section is made up descriptions, guidelines and checklist for each topic.



- 185 -


Managing Data Quality Through Effective Quality Assurance Project Plans by John Warren, U.S. Environmental Protection Agency

A Quality Assurance Project Plan is an approved written document that describes the quality assurance procedures, quality control specifications, technical activities, and data management procedures that should be implemented to ensure that the final results will meet the overall project=s goals. The Quality Assurance Project Plan takes the overall goals established through a project=s planning phase and translates these into a set of specifications that assures the final data meets the intended outcome. Deviations from this plan can be noted and assessed as to the potential impact on the quality of the resulting data.

An important aspect of the plan is the establishment of a meaningful specification of quality assurance procedures and associated quality control techniques. The requirement of a certain method to achieve specific technical specifications should be closely related to the type, quantity, and quality of the samples demanded by the planning phase. One way to do this is to consider aspects of the project such as sample design, sample collection methods, and sample analytical methods, and how they impact the standard operating procedures of the facility accepting samples for analysis.

This paper discusses some of the techniques found by the Agency to ensure the data collected meets the standards of quality contained in the Quality Assurance Project Plan.



- 186 -


Session 25: Variance Estimation Forstersaal Chair: Claes Andersson, Statistics Sweden

Predicting and Monitoring Design Effects: A Tool for Survey Quality Improvement by Peter Lynn, University of Essex, United Kingdom, Siegfried Gabler and Sabine Häder, ZUMA, Germany

A central objective of the European Social Survey (ESS) is to improve the quality of cross-national survey design and implementation. This requires advances in the standards of survey methodology in many countries and advances in the conceptu-alisation and implementation of ideas of standardisation and comparability.

In this paper, we will describe and evaluate a novel aspect of the specification and implementation of the sample design for the ESS. The national sample size was specified in terms of the effective sample size of interviews. Guidance was provided to national teams on how to predict the design effect due to variable selection

probabilities ( PDEFF ) and the design effect due to clustering ( CDEFF ), and how

to use these predictions to determine the necessary sample size. Additionally, a 4-person "sampling panel" was created, to work closely with the national teams in developing an appropriate sample design.

The method used to predict PDEFF involved applying a standard assumption of

equal variance across weighting classes, i.e. issi ∀= 22, where i denotes weighting

class and )(2 yVars = . We will present the predictions obtained using this method

for each of the 22 countries that took part in the first round of the ESS in 2002-03. We will then compare these predictions with estimates of the realised design effects from the round 1 data and will decompose the differences into a part due to imprecision in estimation of the frequency distribution of the design weights and a part due to violation of the assumption of equal variance.

To predict the design effect due to clustering, participating countries were encouraged where possible to refer to estimates of intra-cluster correlation, ρ , obtained from other

national surveys. Where no such estimates or other relevant evidence were available, it

was suggested that they should assume 02.0=ρ and then apply the formula

( )ρ11 −+= bDEFFC to their proposed design, where b denotes the average

cluster size in the sample. We will present the predictions thus obtained for each of the participating countries and will compare them with realised design effect estimates. We


- 187 -


will assess the accuracy of the predictions of both b and ρ . Additionally, we will

present some alternative methods of estimating ρ based on data from surveys with

complex designs. We will investigate the sensitivity of these methods to variation in b

and in b and will discuss the implications for assessments of this sort.

We will draw conclusions regarding methods of predicting design effects as a tool for quality control and, in the context of regular or repeating surveys, methods of estimating design effects as a tool for quality maintenance and improvement. We will highlight implications for sample specification, for the provision of guidance on prediction, and for the choice of estimation methods in the context of a wide range of sample designs.

Contact: [email protected]; [email protected], [email protected]

Variance Estimation in the French “Labour Forces Survey” by Pascal Ardilly and Guillaume Osier, INSEE, France

The French “Labour forces survey” is a quarterly household survey concerning some 54 000 dwellings. It is based on a rotating scheme, consequently a given dwelling remains in the sample during six quarters. The sampling design is a stratified design, with several stages and unequal probabilities. A “home made” software called POULPE, implemented with SAS-V8 to produce variance estimation for any kind of survey, has been used for this particular survey. This software pretends to be “universal”, since it can handle the traditional complex sampling designs and takes into account the total non-response and any possible calibration. It is based on an analytical approach, such that adapted formulas of variance estimation are considered at each stage. It can be supplied with any linearized variable, so that (almost) any complex estimation can be considered.

In the “Labour forces survey”, stratification crosses administrative regions and urban degrees. Primary units are considered, representing different administrative and demographic areas (communes, groups of districts, districts). Some of them are selected with a proportional to size sampling (the whole sample contains 2 554 primary units). We split these primary units in secondary units called “sectors” (a sector contains between 120 and 240 dwellings), and a unique sector is selected in each primary unit, with proportional to size sampling. Each sector is then divided into groups of about 20 dwellings called “areas”, and for a given quarter a unique area is selected in each sector, with a simple random sampling. Finally, in each sampled area, all the dwellings built before the census 1999 are interviewed, and some new built dwellings are selected with a simple random sampling.


- 188 -


In the case of this survey, the existence of samples which size is equal to one in a particular stage of the sampling is a problem to estimate the accuracy, therefore we had to make some “collapses” and to simplify the design.

The operational phase of POULPE relies on three SAS files : the first one describes precisely the sampling design, the second one gives the whole information to calculate the probabilities of selection at each stage, and the third one contains the essential micro data, namely identifiers, variable of interest, auxiliary variables and an estimation of the individual probabilities of response. In fact, the practical difficulties are encountered during the fastidious preliminary work necessary to prepare these tables. Once this is done, the software is very easy to use.

We will present the main features of the software POULPE, the sampling design of the “Labour forces survey”, and how this design has been simplified to be integrated in the software. Then, we will present some standard errors and design effects for cross sectional estimators.


Variance Estimation for Household Surveys with Different Level Variables by Ariane Neethling, University of Stellenbosch, South Africa

A household survey design is often conducted by selecting a sample of households and then including a single or all household members in the sample. Many household surveys generally use separate weighting procedures and thus different sets of weights for estimating individual and household characteristics. Person weights are usually adjusted by using post-stratification adjustment or calibration techniques on available person level auxiliary information. These weights do not take into consideration the fact that some persons belong to the same household. Hence, the household has to be seen as a cluster. Household weights cannot be adjusted in a similar way, as reliable population information is generally not available. These weights are often derived from person weights of the household members. This method causes its own problems.

To obtain an acceptable set of weights for individual and household estimators, an extension of the calibration method, which is usually based on only person auxiliary information, is desired. The integrated linear weighting procedure has been developed to achieve a single set of weights, which would be appropriate for both person and household estimation, and to overcome the problem of inconsistence between person and household based estimates.

Another estimator, the cosmetic estimator, is proposed with the aim of getting the best of the design-based approach as well as model-based approach. The integrated


- 189 -


weighting method can be applied on the cosmetic estimator to form a new cosmetic calibration-integrated estimator, which is defined in both approaches.

When survey results are presented, it is also good practice to provide the variance estimates for the estimators used in the survey, since measuring data quality is strongly connected to variance estimation. Furthermore, measuring precision and accuracy of an estimator in complex surveys is usually discussed in terms of the variance/mean square error of the estimator under consideration. Considerable literature on variance estimation for common survey estimates is available, but is often computationally complex. Further, even for widely used estimators, such as the ratio and simple regression estimator, the issue of the ‘best’ variance estimator has not been finally resolved. Given this background, variance estimators for integrated weighting estimators is still not solved.

Since integrated methods for weighting characteristics of persons and households, are based on linear regression estimation, those variance estimators could be modified for integrated weighting estimators. In this case, the household has to be taken as a cluster. Other variance estimation formulae, for example for the cosmetic estimator based on the model-based approach, could also be extended to integrated weighting estimators.

In this paper the integrated weighting techniques with specific focus on variance estimation as a tool for the assessment and management of data quality, will be discussed from a practical perspective.


Implementing Linearisation Variance Estimation: An Application to the French „Taxable Income Survey“ by Fabien Dell, Xavier D'Haultfeuille, Philippe Février and Emmanuel Massé, INSEE, France

A statistic is of no value without its standard deviation. On the basis of this obser-vation, this paper reviews the methods for estimating the variance of statistics estimated through sampling. The first part examines the classical case of a total. After reviewing variance formulae in the general context, we show which estimator may be implemented. The second part shows how the linearisation method put forward by Deville ([6] and [8]) can be used to estimate the variance of complex (non-linear) estimators. Our applications mainly deal with inequality indices (Theil, Gini, Atkinson, etc.) Linearisation is also compared to the bootstrap variance estimation method for complex estimators. The third part of this paper provides details about the SAS linearisation macro code.


- 190 -


After presenting the necessary tools, our main focus is on a practical application to a specific survey to gain precise knowledge of the estimated poverty rates based on the French „Taxable Income Survey“. Therefore, we shall attempt to estimate the variance of the poverty rate estimator used. Three problems face a statistician undertaking this task:

• The poverty rate is a non-linear statistic. We need to linearise the statistic in order to estimate ist variance.

• The Taxable Income Survey accompanies the Labour Force Survey, which is conducted according to a special sample design that cannot be considered to be a simple random sampling. The area sample design and the computation of the poverty rate on the basis of individual income distribution give rise to clustering.

• In addition to the non-response to the Labour Force Survey per se, the Taxable Income Survey is not based on the full sample drawn for the Labour Force Survey. Adjustments are made for total nonresponses on the basis of the Labour Force Survey before matching, and on the basis of the Taxable Income Survey after matching. These adjustments need to be taken into account when estimating the precision of the estimations made.

We sequentially address these three problems and give detailed confidence intervals for poverty rates computed on the Taxable Income Survey.


Generalized Variance Function: Theory and Empirics by A. Pavone, ISTAT, Italy

Concise presentation of sampling errors in survey reports is a challenge when com-puting and printing standard errors for each estimate could be problematic, including time-consuming and costly effort. In this case provide sampling error models from which the analyst can derive sampling errors is one solution. These models are often declared in the form of generalized variance function (GVF). As a rule, they are obtained in two steps: (i) for each domain of interest, it is directly estimated survey-specific and corresponding variances, and then (ii) the relationship between estimates with its sampling errors is modelled.

This contribution attempts to extend the traditional GVF models (Wolter 1985) in the case of economic-estimated totals that are typically disseminated in business surveys.

Let Y denote an estimator of the population total Y. GVF models are typically created for the relative variance of the estimated total:


- 191 -


( ) ( )Y

YVarYε

ˆˆ = (1)

where: ( )YVar ˆ is the variance of Y .

Graphical inspection shows that the relative variance is a decreasing function of the

magnitude of the mean (expectation) of the estimator ( )YE ˆ . A shape that exhibits this

propriety is:

( ) eYYε expˆˆ 21

ββ −= (2)

where: e is a residual term serially uncorrelated, with zero mean and constant variance.

The parameters 21,ββ are unknown and have to be estimated. They depend upon the

population, the sampling design, the estimator, and the Y attribute itself.

However, this model is not linear in the variables and it is necessary to make some suitable transformation. By log-linear transformation the exponential model can be rewritten as:

( )( ) ( ) ( ) eYYε +−= ˆlnlnˆln 21 ββ (3)

If the assumptions of the classical linear regression model are fulfilled, the parameters can be estimated by OLS method.

An attractive of this log-linear model is that the slope coefficient 2β measures the

elasticity of ( )Yε ˆ with respect to Y , that is, the percentage change in relative variance

for a given (small) percentage change in the expectation of the total estimator.

This model can be also reformulated. The parameters of the equivalent model are in biunivocal correspondence with the original model. In fact, if we decompose the relative variance into its two components, the model (3) can be written in the following way:

( ) ( ) ( ) ( ) eYYVar +−+=

ˆln1lnˆln 21 ββ (4)

It is critical to note that, if the design effects vary greatly among domains, this general GVF prediction model is not exhaustive. In this case it is required to integrate the right-hand side of the GVF model with some proxy for the design effect.


- 192 -


This approach was adopted to approximate sampling errors for the Italian enterprises’ accounts estimates. The data were collected by the National Statistics Institute of Italy, merging the exhaustive survey SCI (on large enterprises � 100 employees) with the sample survey PMI (on small and medium enterprises, up to 99 employees) to overlapping domains. The applied work has showed a high number of domains with a small sampling error, compared to other domains with same total level of survey characteristics. We consider ways in which to address this problem based on an appropriate treatment of the design effect.


Session 26: Quality Indicators II Mozartsaal Chair: Raoul Depoutot, INSEE, France

The Prototype of the Quality Indicators System at the Brazilian Insti-tute of Geography and Statistics by Zélia Magalhães Bianchini and Sonia Albieri, National Institute for Geography and Statistics (IBGE), Brazil

Strategies for assessing and improving statistical quality have been developing and implementing at IBGE, the Brazilian Institute of Geography and Statistics. IBGE is starting the implementation of quality indicators for surveys and administrative registers based on a methodological study of quality indicators conducted in 2002 within the Program of Statistical Co-operation of the European Union and the Mercosur.

The idea of this paper is to focus on short summaries of certain aspects of good statistical practice and quality management initiatives that are underway at IBGE that address each of the dimensions of quality: relevance, accuracy, timeliness, punctuality, accessibility, consistence, coherence, exhaustiveness, transparency and integrity. A description for the prototype of the quality indicators system we are implementing is presented in a summary way as well as the plans for their update and for data releases issues. This system is planned to be implemented as part of the Metadata Project.

Although significant advances have been achieved at IBGE, we have a lot of examples of good statistical practice or initiatives to improve and we are pointing out some of them. We use methodologies comparable with international best practices, and try to adopt modern, cost effective and appropriate methods and technology during collection, capture, coding, editing, imputation, estimation and dissemination operations. But, IBGE does not have a comprehensive project-based internal cost-accounting system, that could assist decision making regarding starting terminating specific survey activities, while taking full account of the corresponding budgetary implications. This is a serious limitation towards improving cost-efficiency that we are trying to overcome.


- 193 -


In order to conduct or to be in the lead of the activities related with the quality issues (product and process of production), we are forming an internal committee at IBGE.


Effective Use of Data Quality Indicators by John Warren, U.S. Environmental Protection Agency, U.S.A.

Quality is a very general term. In its broadest sense, Data Quality is a measure of the degree of acceptability or utility of the data for a specific purpose. For most environmentalists involved in the collection of data, the term Data Quality Indicators brings to mind Precision, Bias (Accuracy), Representativeness, Comparability, Completeness, and Sensitivity. These, often called the PBRCCS parameters, can be considered the principal indicators of data quality. Defining these indicators has not proved to be a straight-forward because different interpretations and formal definitions can be made for each indicator.

The U.S. Environmental Protection Agency has defined these indicators using a mixture of qualitative and quantitative terms:

• Precision: a quantitative term and is a measure of agreement among repeated measurements of the same property under identical or substantially similar condi-tions.

• Bias: a quantitative measure of a systematic or persistent distortion of a measure-ment process that causes errors in one direction. Accuracy is composed of both precision and bias.

• Representativeness: mostly a qualitative term but one that can have some quantitative aspects. It is a measure of the degree to which data describe a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition.

• Comparability: a qualitative term that expresses the measure of the confidence that several data sets can contribute to a common analysis.

• Completeness: a quantitative expression of the amount of valid data obtained from a measurement system.

• Sensitivity: a quantitative measure of the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest.


- 194 -


This paper discusses these indicators in more depth and describes some of the background behind the recent release of the guidance document Guidance on Data Quality Indicators.


Quality Measures for Economic Indicators by J.M. Museux, G.L. Mazzi and G. Savio, Eurostat

Eurostat compiles European Union and EU-Zone infra-annual economic statistics relevant for short-term economic analysis. The time series are mainly released through the Euroindicators website. Much effort is devoted to render the statistics promptly available, reliable and harmonised for a sufficiently long period. These statistical outputs have to be accompanied with user oriented quality measures.

Eurostat has defined a quality framework focusing on the quality of the output. Data quality is made up of six dimensions: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, and coherence. The starting point is the user and how he/she sees the product. Standard quality reports have been defined together with a list of quality measures.

For time series, given their particular nature, not all of these measures are appropriate or possible to calculate. The current proposal aims to make to bridge between the need for a practical framework to assess the quality of economic indicators and the general quality framework developed by the ESS. It provides an alternative understanding of the basic dimensions. For instance, accuracy is handled focusing on revision analysis. The comparison of the length of the series contributes to the assessment of comparability. If relevant, seasonal adjustment process is taken into account and assessed equally.

A concrete example for the Euroindicators series will be provided.


Measurement of Data Quality in the Case of Short Term Statistics by Rudi Seljak and Metka Zaletel, Statistical Office of the Republic of Slovenia

There is a constantly growing demand for the fast data delivery especially in the case of business surveys. Monthly or quarterly surveys which should provide these data are usually based on samples and are focused just on few variables (i.e. turnover, number of employees). The main goal is to get results of sufficient quality as quickly as possible. Since the deadlines for the publishing of results are becoming shorter and shorter there is a temptation of publishing data of poor quality just for the sake of


- 195 -


timeliness. Therefore there is of great importance to constantly measure different aspects of quality of these data in order to avoid publishing results which would not satisfy agreed quality standards.

In the paper we present the system for quality control of statistical results in the case of the short term surveys measuring turnover of enterprises from 3 different areas: retail trade, hotels and restaurants, services. The published results of the surveys are different time based indices. Quality is controlled by the set of indicators defined on the base of the Standard Quality Report and other methodological documents prepared by Eurostat. Since most of the statistical process is automated the aim was to incorporate into this process also the calculation of the majority of defined quality indicators so that consequently we would have the indicators at disposal at the same time as the estimated indices. We describe which indicators are calculated automatically, which were the problems that we faced and what are our plans for the future work in this area.


A Set of Quality Indicators About the Domestic Output Price Survey by Valerio De Santis and Maria Moscufo, ISTAT, Italy

The aims of this paper are to show: 1) a set of quality indicators calculated for the production process of producer price index (hereafter PPId) on domestic market and the different proposals of global, and at same time synthetic, presentation of quality; 2) some indicators which measure the effects determined by use of different computation formulae at different levels of aggregation of index.

The quality indicators implemented concern the survey and the inputting, checks and imputation of data (prices). They are calculated both after production of provisional index and after production of final index, both for total and division indices (2-digit level of classification of economic activities NACE Rev. 1.1). The analysis of these quality indicators permits to identify actions (which regard the reporting units - enterprises and/or statistical reviser) to improve the quality of production process and statistics results; on the other hand, their temporal analysis permits to check the effectiveness of the actions made to improve the production process.

In the quality analysis, are also calculated some indicators which measure the effects determined by use of arithmetic and geometric mean on the estimates of elementary aggregate micro indices (price indices in absence of weights) and the effects determined by use of chained Laspeyres formula on estimates of higher levels indices.

These indicators are part of a study conducted in view of adopting a chained PPId index.



- 196 -


Session 27: Non-Sampling Errors I Leibnizsaal Chair: Paul Biemer, RTI International, U.S.A.

Measuring Micro-Level Measurement Error by Clyde Tucker, U.S. Bureau of Labor Statistics, U.S.A.

Some important indicators of social and economic conditions come from asking questions of individual respondents in large national sample surveys. Because these indicators are reported at the aggregate level, error in them also has been reported at that level. Known as mean squared error, it encompasses both variance from sampling and bias. While it may be reasonable to view sampling error as an aggregate concept, to do so for bias will limit the ability to determine its sources. Except for coverage error, bias is largely the result of errors in the measurement of respondent characteristics. Thus, to understand this bias, the size of individual-level errors must be ascertained. The conceptual foundation for measuring errors at the micro level is outlined, and several methods for measuring these errors are reviewed. Discussion also covers the treatment of nonresponse in the context of micro-level measurement error.

Clyde Tucker is Senior Survey Methodologist at BLS. He may be contacted at 2 Massachusetts Ave., NE, Washington, DC 20212. The ideas expressed in this paper are the author's and do not necessarily represent the views of BLS.


Quality at the Item Level: Decomposing Item and Concept Response Rates by James M. Dahlhamer, National Center for Health Statistics, U.S.A., John Dixon, Bureau of Labor Statistics, U.S.A, Pat Doyle, Judy Eargle, Deborah Griffin and Pamela McGovern, U.S. Census Bureau

Survey quality assessment spans many areas but is heavily focused on measurable outcomes (such as sampling error and unit nonresponse rates). As more aspects of survey quality become quantifiable, more measures can be brought into the discussion of how well surveys perform. This is the second in a series of papers that take the first tentative steps toward expanding quantifiable measures of survey quality by focusing on quality at the item level in such a way that it can be compared across surveys with different designs, objectives, and outcomes. The first paper in the series (prepared for presentation at the Federal Committee on Statistical Methodology (FCSM) in November 2003) addressed methods for computing comparable item nonresponse rates across surveys that measure similar concepts with vastly different instruments and sample designs. That paper recommended a number of topics for future research, two of which are addressed in this paper.


- 197 -


The first topic addressed in this paper is developing an understanding of the com-ponents of the differences between rates from the initial and final data. Initial data refer to information as it is collected in the field where the source of nonresponse is the respondent’s reply of “don't know” or “refused” (or, in the case of mail-out mail-back surveys, blanks in items that should have been answered), based on the characteristics of the respondent. Final data refer to the data after all post-collection processing procedures have been completed. The sources of nonresponse differ between the two because the editing and imputation procedures can change the universe for an item, which can generate missing data not observed in the initial data. The sources also differ because the editing and imputation procedures can identify out-of-range and inconsistent information (beyond that which might have been addressed during the interview using an automated instrument). The third source of differences is in the treatment of person noninterviews in interviewed households (which is only relevant to surveys with individual-level questions where proxy interviews are not allowed or not encouraged by design). We discuss the variation in these differences across multiple surveys with different sample designs, interview rates, modes, and post-processing procedures.

The second topic focuses on the separate components of item nonresponse (refusals and don't knows) in the initial data, comparing each of the two rates across surveys according to the concept rate methodology developed for the FCSM paper. Item or concept nonresponse error is a function of the balance between types such as don't knows and refusals. The literature, however, has generally failed to make this distinction, possibly leading to ambiguous conclusions and erroneous solutions for nonresponse reduction. Variation in rates across surveys may be attributable to differences in question format, survey mode and content, and the interplay of these factors on the question-response process. Cross-survey comparisons permit the exploration of survey and item design influences on nonresponse types and therefore on error. The implications for nonresponse adjustment and reduction strategies are discussed.


Response Error Estimation in Presence of Record Linkage Errors: The Case of the Italian Population Census Brancato G., D'Orazio M and Fortini M., ISTAT, Italy

Traditionally, response error estimation is performed by means of reinterview studies with or without reconciliation, allowing the evaluation respectively of the response bias and the simple response variance. The need for lowering the costs in NSIs addresses toward the use of alternative methodologies, such as, for example, methods based on the integration between survey data and administrative sources.


- 198 -


In order to lower the costs and gain efficiency, in the quality evaluation program for the 14° Italian Population and Households Census it was decided to carry out a single control survey instead of the two planned surveys (coverage and quality). To this aim, the questionnaire of the coverage survey was extended to include a subset of the questions of the census form, for which it was intended to estimate the simple response variance (SRV). From an operational point of view, since the evaluation of the response error requires the comparison of the two measurements (census and coverage survey) on the same subjects, the analyses to estimate the response variance will be performed after the execution of a record linkage (RL) between the census data and the census post-enumeration (PES) survey data.

The RL was applied on groups of units enumerated into the same enumeration areas (EA) or, in other words, no tentative match was performed for records pertaining to different EAs, considered a priori as belonging to different units.

To enforce the RL, the names, the surnames and the addresses of the units, appro-priately encoded, were included among the key variables; to the same extent, in-formation on households structures was taken into account in linkage procedure. Consequently, the whole process of linkage was rather complex and implied several automated steps each one followed by manual recovery actions.

Although such an approach provides advantages in terms of costs, it introduces a series of difficulties even when acceptable levels of accuracy are achieved in executing the RL. In detail, the following errors may occur in the data collected through the coverage survey:

• False link (FL): erroneously linked units;

• False non link (FNL): units observed in both occasions, which the linkage pro-cedure is not able to link;

• Units observed in the census but not observed in the coverage survey (NP2).

The fist type of error is known to cause an overestimate of the response error, whereas the second induces underestimate. The third type of error can produce an underestimate of the true response error if the non observed units have a higher inclination to the response error.

The present work presents some strategies that can be adopted to take into account for the above errors when estimating simple response variance from the data of a coverage survey used as re-interview.



- 199 -


Mail Surveys in Different National Contexts by Sabine Klein, ZUMA, Germany

When Don A. Dillman first published his book on optimising mail survey outcome rates by applying the Total Design Method (TDM) in 1978, he broke new ground in mail survey methodology. At the time, he predicted potential outcome rates of about 75 percent for general population surveys. Although TDM had been developed and tested only in the American context, it soon became recommended standard practice everywhere.

Research has shown that applying TDM undoubtedly increases outcome rates in mail surveys.

Why is it, then, that we still find huge differences in outcome rates between countries even within Europe? It seems to be no problem for a Scandinavian country to achieve about 65 percent, for example, whereas countries like France may struggle to obtain even 20 percent response in mail surveys.

This paper aims to identify national “peculiarities” which contribute to a climate that is either favourable or unfavourable for mail surveys and which may help account for differences in outcome rates for one and the same survey. Such peculiarities include legal restrictions, geographic dispersion of population, sample design options, and what is seen as overall “survey climate”.

The discussion is based on data from six countries (Canada, Denmark, France, Finland, Norway and New Zealand) that fielded the International Social Survey Programme (ISSP) 2001 module on Social Networks as a mail survey.


Measuring Response Errors in Censuses and Surveys Through Rein-terview Data Having a Different Measurement Error Variance by Antonia Boggia, Piero Demetrio Falorsi and Alessandro Pallara, ISTAT, Italy; Aldo Russo, Università Roma Tre, Italy

Reinterviews have been extensively used as a tool for estimating and reducing response errors in censuses and sample surveys. By response error it is meant any error occurring at the data collection stage for a variety of reasons. Errors may be due to the respondent, to the interviewer, or to both. In the practice of censuses and large surveys some different techniques have been proposed (Forsman and Schreiner, 1991) in order to measure response error components. One such method relies on replicated measurements obtained through reinterview of a subsample of units from the original survey on a set of questions from the original interview. When the responses given by the same units during the reinterview differ from those given in the original interview


- 200 -


the differences can be evaluated through reconciliation, normally by asking the respondent to determine what is the correct information between the two interviews.

When using the approach based on reinterviews to measure response error compo-nents, it is usually assumed that: (i) the two measures are modelled as random variables; (ii) the repeated measurements on the same unit are independent; (iii) the measurement conditions for the two occasions are identical or as close to identical as possible; this imply that the random variables which are associated to the two measurements are subject to the same measurement error variance. Under these assumptions, the standard response error model based on reinterview data, developed at the U.S. Bureau of Census (Hansen et al., 1964), yields unbiased estimates of simple and correlated measurement variances. Furthermore, an estimate of the measurement bias can be obtained, assuming that the reconciled reinterview process yields the true value. While theoretically sound, the standard survey error model seems to be not realistic in many practical situations, because some of the above assumptions are difficult to be met. Frequently, e.g., the reinterview program, because of budget and operational restrictions, are carried out using a different data collection technique with respect to the original interview. In this case, the two measurements, although still independent, cannot be considered as obtained under identical measurement conditions and, therefore, they can be deemed as having different measurement error variances.

In this paper it is shown how to obtain an unbiased estimate of the response variance when the assumption of identical second-order moment of the two measurements fails to hold. Under the standard response error model the variance of the standard estimator of the population total of the true values is decomposed in the sum of the sampling variance and measurement variance and the measurement variance can be split in two terms, both depending from the (unknown) variances and covariance between measurements, but only one depending also from the sampling design. Indeed, in presence of measurement errors the estimator of the total variance is biased and it can be shown that the bias is equal to the component of the measurement variance not depending on the sampling design. A method of moments estimation of the simple and correlated measurement variance components under the standard response error model is then proposed, assuming different error variances of the repeated measurements in different surveys.

The proposed methodology is illustrated with data from a reinterview survey carried out to evaluate accuracy of the 5th Italian Census of Agriculture.

Contact: [email protected]; [email protected]; [email protected]


- 201 -


Seasonal Differences in a Danish Health Interview Survey by Michael Davidsen, Ola Ekholm and Mette Kjøller, National Institute of Public Health, Denmark

Introduction

When measuring the prevalence of health and illness in a population, certain indicators may be expected to vary according to the time of year while others may not. This may have bearings on the design of surveys, e.g. by ensuring evenly spread or focused time of interview (depending on the choice of health indicators). In the Danish Health Interview Survey program invitation traditionally has been conducted in three waves in order to reduce seasonal variation but in designing a new survey in 2004 this strategy has been called into question.

The purpose of this study is to investigate seasonal differences among a broad range of public health topics with focus on illness, symptoms, activity restrictions, falls and life style indicators.

Methods

In year 2000 the Danish National Institute of Public Health conducted a nationally representative survey of adult Danish citizens. All persons were selected by a hybrid of county-stratified random sampling and supplemented by an extra random samples from one county. All persons accepting an interview had one in their home by a trained interviewer. People were selected in three waves during spring, autumn and winter but not necessarily interviewed at that time - the date of interview is used. At the end of interview a self-administered questionnaire was handed out to be returned at convenience of the interviewee.

Results

A total of 17139 persons (74%) had an interview. Of these 18% were interview in the winter, 24% spring, 16% in summer and 42% in the autumn. The age distribution was not equal across the four seasons in that more persons aged 25-44 were interviewed during summer that in other age groups and that persons aged 25-44 were interviewed more often during autumn. In most cases patterns were as expected were found. Visit to own doctor and short-time number of sick days were lowest during summer and highest in the winter. Asthma and allergic symptoms were highest during summer and people had most allergic complaints during summer. Alcohol intake was highest during summer while smoking and BMI did not differ. More persons rated their health as good or very good during summer than in any other season. Colds were most common during winter. Quality of life (SF36) was viewed most optimistically in the summer. Long-standing illness is most frequent in summer.


- 202 -


Discussion

In general the differences in prevalence in the four seasons considered Is not over-whelming. However a general pattern was seen indicating seasonal differences in important variables used in many European health surveys. Thus seasonal differences can be expected not to be negligible and should be considered in the design of health surveys and in some cases in design of analysis.


Evaluating Interviewer Effect on Istat Victimization Survey Using Multilevel Models by Isabella Corazziari, Maria Giuseppina Muratore and Giorgia Simeoni, ISTAT, Italy

The survey on the citizen’s safety is carried out by Istat every five years. The first edition took place is 1997-1998 and the second one in 2002. The survey collects information on the crimes against people and properties from the crimes’ victims point of view. Furthermore the perception of safety in the environment and the adopted safety measures are investigated. Therefore the sensitiveness of the observed phenomena is considerable.

In the first edition a random sample of 50000 households has been selected; in 2002, the sample size was of 60000 households. In each sampled household only one person aged 14 or more was randomly selected and interviewed. The survey has been conducted using a CATI technique for data collection in both occasions. When using a CATI technique the interviewers’ behaviour during data collection is very important in order to achieve good quality survey results. In this specific survey, due to the sensitiveness of the information contents, the quality of their work is even more important. In fact, the interviewers’ skills could affect deeply the quality of the collected information. Furthermore the importance of creating a trustworthy interviewer/interviewee relationship to obtain a higher quality interview has been highlighted in many victimization surveys.

In order to evaluate the interviewer effect on sensitive data a multilevel model has been applied to the 2002 citizen’s safety survey data. The multilevel model can be considered an appropriate methodology to evaluate this kind of effect. Indeed, the structure of the data is clearly hierarchical, since respondents are nested within interviewers; furthermore the interviews are randomly allocated to the interviewers and the interviewers can be considered as a random sample from an unobserved population of possible interviewers.

In the first survey a similar analysis has been carried out and an interviewer effect was detected. It is interesting to assess its importance in 2002 survey, as efforts have been


- 203 -


made to improve both the questionnaire and the data collection process. In particular, the wording of the questionnaire has been reviewed. Furthermore, on the basis of a daily monitoring activity performed using control charts applied on the main data collection quality indicators, specific debriefings were organised and focused on the arising problems.

In 1997-1998 survey no information on the interviewers socio-demographic charac-teristics was available, such information has been collected in the second edition, allowing a better application of the multilevel model methodology and consequently permitting a more effective interpretation and explanation of the interviewer effect on data.

The results of the application of the multilevel model will be presented in the paper, as well as, the effectiveness of the efforts spent in the design and implementation of the second edition of the survey. This experience can be viewed as an application of the Deming cycle for quality improvement (Plan, Do, Check, Act) where this analysis can be considered the “check” step of the cycle and the starting point for improvements of next surveys.


Applying Methods of Record Linkage for Census Validation in the Basque Statistics Office by Leire Legarreta Antuñano and Marina Ayestarán Arregi, Basque Statistics Office, Spain

The aim of this paper is to describe the experience of the Basque Statistics Office in applying novel methods of record linkage for census validation.

The probabilistic methods of record linkage, pioneered by Fellegi and Sunter, make a feasible and efficient comparison of large data bases in a statistically justifiable way, when unique identifiers are not available. Despite their relative complexity, they provide numerous advantages compared to other ad hoc procedures used for this purpose. Initial experiences in Eustat in applying this new methodology and the results thus obtained, with linkage percentages raising for some files from 48% to 93%, clearly showed the need to carry on with this approach.

Based on the theory introduced by Fellegi and Sunter, we have implemented an automatic linkage methodology. One of the improvements we have performed refers to new methods for creating standardised lists for the configurations that variables can adopt. As pairs of strings often show typographical variations, new techniques to compare strings have been developed. These techniques take into account the frequency of the multiple configurations that can be found in the files to be linked, as well as the specific spelling and phonetics of our area. Linkage, integrated as an


- 204 -


automatic system, allows the calculation of all the parameters implied in the process from the data files. It uses classic methods to calculate weights, allows the development of each stage of linkage with control over tolerable errors and makes use of different blocking criteria to minimise running time.

These linkage techniques are being used to validate the 2001 Population and Housing Census in the Basque Country using data obtained from two different sources: the Census Validation Survey itself, and the quarterly labour force survey of the Basque Country. Applying our improved linkage techniques to these two different sources allows us to compare two different validation techniques.


Multivariate Models for Estimating Coverage Error in the Italian 5° Census of Agriculture by Matteo Mazziotta, Monica Russo, ISTAT, Italy

The ISTAT carried out an areal sample survey (Falorsi P.D., Russo M., 2001) to supply coverage rate estimates of the 5° Agriculture Census, where the final sampling units were made up of cadastral maps of the Land Registry Office (Falorsi P.D., Pallara A., Russo M., 2001). During the survey phase, it was necessary to fix the farms having lands in land parcels of sample cadastral maps, up to the date of October 22nd 2000 (Mazziotta M., 2001). The survey design is made of two stages with stratification of the first stage units; the survey implicates 131 cadastral maps (Mazziotta M., Russo M., 2002). In this paper we have considered only model variability (the sample design is fixed). The results shown have to be considered not weighted.

We intend to underline that the formulation of the expected value and variance of N in Petersen model has to be ascribed to Wolter (JASA, 1986), while all the other formulas are the result of our research.

After the collection of data and record linkage between list A (Census) and list B (coverage survey), the results are:

List B

In Out

List A In 5.839 1.646 7.485

Out 720 22N +2N

6.559 2+N N


- 205 -


The purpose is to estimate the coverage rate of the 5° General Agriculture Census (τ); to reach this aim it has been estimated N; so it is possible to consider three alternative

multivariate models: the homogeneity within lists model or Petersen model ( aM ), the

equal catchability model ( bM ) and the behavioral response model ( cM ) (for their

hypothesis see Wolter, JASA, 1986).

The estimates based on Petersen model ( aM ) are: 408.8ˆ =aN , 89,0ˆ1 =+p and

78,0ˆ 1 =+p .

The expected value is: ( ) ( ) ( ) 408.8ˆ22

111 =+≅ ++

−++ ppppNNE a , and the

variance is: ( ) ( ) ( ) 11,293ˆ22

111 =≅ ++

−++ ppppNNV a . Then the coverage rate

estimate is: 8902,0ˆˆ 11111

1 === −++

− NNNN aaτ and its variance is

( ) ( ) 000015,0ˆ 211

1 == ++−

+ ppNpV aτ .

The estimates based on bM model are: 445.8ˆ =bN and 83,0ˆ =p .

The expected value is: ( ) ( ) 445.81ˆ 22 =−+≅ − ppNNE b , and the variance is:

( ) ( ) 71,3531ˆ 22 =−≅ − ppNNV b . Then the coverage rate estimate is:

( ) ( ) 8863,042ˆˆ 1111

1121

211

1=++== +

−

+++++

−NNNNNNNNbbτ and its

variance is: ( ) ( ) 00002,01ˆ 1 =−= − pNV bτ .

The estimates based on cM model are: 282.8ˆ =cN , 9,0ˆ =p and 78,0ˆ =c .

The expected value is:

( ) ( ) ( ) 282.81ˆ2

12

321 =−−++≅ +

−+

− ppcppcppNNE c , and the variance

is: ( )≅cNV ˆ ( ) ( ) 2821 21

23

21 =−−+ +−

+− ppcppcppN . Then the coverage

rate estimate is: ( ) ( ) 9038,0ˆˆ 221

21

1211

111

1 =−+== +

−

+−++

− NNNNNNNccτ

and its variance is

( ) ( )( ) ( )( ) ( )[ ] 000015,011311ˆ 21 =+−−−−−= − pppppNppV cτ .


- 206 -


In conclusion, the values of N estimate are approximately equal among models. Besides all estimators are almost unbiased.

Table 1: Estimate, bias, variance and variation coefficient of N and coverage rate

estimate in aM , bM and cM models

Model N

( )NB ˆ

( ) ( )NVNMSE ˆˆ ≅

( )NCV ˆ

τ ( )τV ( )τCV

aM 8.408 0,035 293,11 0,20% 89,02% 0,000015 0,44%

bM 8.445 0,042 353,71 0,22% 88,63% 0,000020 0,51%

cM 8.282 0,034 282,00 0,20% 90,38% 0,000015 0,43%

An important consideration regards the estimates accuracy:

( ) ( ) ( )bac NMSENMSENMSE ˆˆˆ << .

The variation coefficients of N are approximately equal and very small, then it is possible to affirm that the precision level of estimates is quite good. The coverage rate estimates are rather convergent, also the variances of the coverage rate estimates are

similar and close to zero. The variation coefficients of τ are small and ( )cCV τ is

smaller than the others. Finally, we can affirm that the differences among the models are very slight (although model hypothesis are different).


Some Aspects Related to Quality in the Post Enumeration Survey of the Italian Census of Population 2001 by Giancarlo Carbonetti, Nadia Mignolli, ISTAT, Italy

In Italy, the post enumeration survey (PES) was carried out a month after the 14th general census of population (on November 2001), with the main purpose of showing coverage errors, in order to give a quality measurement of the census itself.

As well known, the census count may be greater than the true N size of the population (over-count) but usually the error mechanism is such that the census count is less than


- 207 -


the true N size (under-count). However, PES as a survey is subject to some quality constraints.

In this paper, attention is focused on the aspects related to the main phases of survey planning, decision on the sampling design, questionnaire building, and the monitoring phase during data collection.

Regarding the survey design, a three-stage sampling was adopted: in the first two stages a stratified random design was realized, in the third all the units were collected (cluster sampling design). In the first stage a sample of 98 municipalities was selected (sample of first-level units); in the second stage, a sample of enumeration areas was selected from each first level unit (sample of second-level units), sampling about 1,100 areas; in the third stage all households in each enumeration area and all their components were observed, collecting more than 68,000 households and about 180,000 individual members.

The main reason for this strategy was to have a representative sample for the planned domains (first stage) and to collect data with reference to the same area frame of the census.

The PES data collection method was the same of the census, with home-delivered and self-completion questionnaires. In this case the enumerators, selected among the ones involved in the prior survey, had not to cover the same enumeration area during PES. This approach contributed to guarantee independence between PES and census and any possible influence on responses was kept under control.

All the information was collected through a questionnaire made up of four parts referred to the census day:

1. Demographic information;

2. Educational degree and training;

3. Working activity;

4. Information on housing.

Therefore, a hard effort was requested from the respondents who had to remember events after a long time. For this reason, particular accuracy was put in the proper wording and formatting of the PES questions to improve response quality and rates. For example, in the most important section of the questionnaire (the forth), which represented one of the sources to value if all the persons were counted during the census and if the household form was completed correctly, a specimen was inserted. It reproduced the part of the census form containing the list of the household members and it helped respondents to remember which of the two groups they belonged to: the usually resident persons or the temporarily present ones.


- 208 -


All data collection phases were supported by a centralized monitoring activity based on quality indicators, with useful opportunities of comparison between the researchers and the persons responsible for PES in the Municipalities. This strong control represented a real innovation and appeared very positive during the individuation of the second level units of the sample.


How the Under-Reporting Affects the Income Data from a Household Survey by Claudio Quintano, Rosalia Castellano and Andrea Regoli, University of Naples, Italy

The purpose of this work is to study the quality of the economic information from a household survey, with special reference to the accuracy of income data. The well known difficulty of collecting good-quality income data may derive on the one hand from the survey instruments, on the other hand from the respondents. The former may not be able to ensure the exhaustive registration of the various items that make up the total individual or household income. The latter may be reticent in providing the requested information or, at least, in providing accurate information. For this reason, the observed income levels often underestimate the actual income: the degree of the under-reporting is especially marked for income from self-employment as well as from financial assets.

In this study we propose a two-stage method for evaluating and correcting the un-derestimation in income data, on the basis of the rich informative content of the Survey of Household Income and Wealth conducted in year 2000 by the Bank of Italy.

At the first stage only the information from the sample survey is used in order to identify the income values that, being too low, are strongly suspected of underestimation: a total household income is considered too low if its corresponding equivalised income (once the household size and composition are taken into account) is not greater than a threshold which is defined as function of the first quartile of the distribution. Low income values are deleted and then replaced with newly estimated values resulting from a random regression imputation procedure: the imputed income is the predicted value from the linear model based on a set of explanatory variables that are correlated with income, including consumption expenditure, number of earners, characteristics of the head of household. Such a procedure works within imputation classes: in order to take account of the heterogeneity with reference to their economic status, the households are grouped on the basis of their sources of income, the labour market position of their members, the presence of spouse and children.


- 209 -


The second stage consists in comparing the average incomes from the survey after the imputation with the average incomes from an external source, the National Accounts estimates. For this purpose we refer to the classification of income by source: wages and salaries, income from self-employment, pensions and other transfers, property income. In order to make a suitable comparison between the two sources, the definitions and concepts must be made homogeneous; in particular, the net wages and salaries from the survey are to be converted in gross values, through the estimation of tax and welfare contributions. From this comparison a correction coefficient is derived that increases the incomes from the survey and makes them consistent with the aggregate estimates from the National Accounts.

The improvement in the data quality at the end of the procedure is evaluated through the changes at each stage in the main statistics of income distribution as well as in the main inequality indicators.



- 210 -


Session 28: Confidentiality I Spiegelsaal Chair: Luisa Franconi, ISTAT, Italy

Resolving Confidentiality and Data Quality Issues for Tabular Data by Lawrence H. Cox, National Center for Health Statistics, U.S.A.

Traditional methods for statistical disclosure limitation in tabular data are cell sup-pression, data rounding and data perturbation. All three methods are applicable to count data whereas only cell suppression is effective for magnitude data. Cell sup-pression eliminates otherwise useful data, thwarting straightforward data analysis. Because the suppression mechanism is not describable in probabilistic terms, suppressed tables are not readily treated by more sophisticated statistical techniques, viz., imputation. Data quality characteristics of suppressed tables are consequently poor. Rounded and perturbed data are easily analyzed, but little formal attention has been paid to the degree to which these methods degrade or distort analytical outcomes as compared with original data. Recently, an alternative to cell suppression, controlled tabular adjustment (CTA), was developed. CTA replaces tabular cells that represent disclosure by safe values, uses linear programming CTA to adjust remaining values to rebalance tabular equations, and does so optimally with respect to any of a range of global and local measures of data “closeness”. A forthcoming Monographs of Official Statistics paper examines how distributional properties of original data can be preserved after CTA in the univariate case, e.g., ensuring that adjusted data preserve approximately means and variances exhibited by original data and that original and adjusted data exhibit high positive correlation. In this paper, we extend that examination to multivariate distributional properties, e.g., ensuring that correlations and regressions between two variables exhibited by original data are preserved approximately by adjusted data. In addition, we examine the degree to which data rounding and perturbation methods preserve data quality and demonstrate how to ensure important determinants of quality can be preserved under rounding or perturbation. All of these methods are based on linear programming formulations that are easy to implement or enhance.


Statistical Disclosure Control vs Data Quality: Getting the Balance Right by Ole Black and Marta Haworth, Office for National Statistics, United Kingdom

Users are demanding more detailed and better quality data. At the same time, suppliers are increasingly more concerned about confidentiality protection. Effective Statistical Disclosure Control is a high priority for National Statistical Institutes. To some extent, these two objectives are in conflict.


- 211 -


This paper will present some case studies showing how the two sides have been taken forward in the past and suggest possible solutions which might be developed in future. The paper will cover a range of social and economic applications and emerging methodological research findings.

contact: [email protected], [email protected]

Do Disclosure Controls to Protect Confidentiality Degrade the Quality of the Data? United States Data from the 1999 Trends in Mathematics and Science Study by Marilyn Seastrom, Steve Kaufman and Patrick Gonzales, National Center for Edu-cation Statistics, U.S.A. and Shep Roey, WESTAT, U.S.A.

The National Center for Education Statistics (NCES) in the United States Department of Education has had a formal process for disclosure control to protect the confidentiality of potentially identifiable data in public use microdata files in place since 1987. This paper will explore the evolution of disclosure control techniques used at NCES over the last 15 years, with a focus on the potential impact of each technique on data quality. Historically NCES has used the removal of all direct identifiers; coarsening techniques, such as collapsing categories and top and bottom coding to avoid outliers; and a distance measure analysis of the sample institutions against available institution frames to identify further potential disclosures that require more coarsening of the data. In a 2002 revision of the NCES Statistical Standards, an additional disclosure edit was introduced involving either blanking and imputing or data swapping on a sample of cases. Each additional alteration to the data has the potential for negatively impacting the quality and usability of the data. Techniques currently in use at NCES will be demonstrated with United States data from the 1999 Trends in International Mathematics and Science Study and the impact on the statistical properties, quality, and usability of the data will be evaluated.


Simulation of a Database Cross Match – as Applied to the German Structure of Costs Survey by Rainer Lenz and Daniel Vorgrimler, Federal Statistical Office of Germany, Thorsten Doherr, Centre for European Economic Research, Germany

Many of the survey data are subject to confidentiality restrictions. For this reason, the data are anonymized using a specific method. Within a database cross match, the data intruder matches an external database (additional knowledge) with the whole of


- 212 -


confidential (or target) data. The aim is to bring together information on objects which are believed to belong to the same entity. In [2] an upper bound for the disclosure risk of enterprises of the German structure of costs survey has been estimated using knowledge which is in general not available to the data intruder. This paper studies a realistic scenario where the intruder uses a commercial database as additional knowledge to re-identify as many records as possible of the survey. The result obtained by matching the original survey data (which can be considered as the weakest form of anonymization where anonymization essentially consists of deleting direct identifiers) and the result obtained by matching very strongly anonymized data are compared and discussed.

References

[1] Elliot, M., Dale, A., “Scenarios of attack: the data intruder’s perspective on statistical disclosure risk”, Netherlands Official Statistics, pp. 6-10, 1999.

[2] Lenz, R., “Disclosure of confidential information by means of multi objec-tive optimisation”, Comparative Analysis of Enterprise (micro) Data Confe-rence, London 2003.


The Normative Framework for Maximizing Third-Party Access to Statistical Information while Ensuring Confidentiality: The Experience of a Small and Young Statistical Agency by Louise Bourque, Statistical Institute of Québec, Canada

The Institut de la statistique du Québec (ISQ), created in 1999, is the Québec gov-ernment’s official statistics agency. It works in collaboration with Statistics Canada to produce complementary information. Like all national official statistics agencies, the ISQ is lawfully bound to statistical secrecy. Furthermore, its strategic orientation is to maximize the use of its statistical products by third parties, thus making accessibility to its statistical information as wide-ranging and easy as possible, while ensuring confidentiality.

On its inception, the ISQ, therefore, had to devise a normative framework on acces-sibility, putting emphasis on access to its microdata files. The context in which this framework was developed and implemented as well as the underlying philosophy will be touched on first: the ISQ is a relatively small and young statistical agency with some 250 employees and limited financial means. Despite this, the ISQ has set up a normative framework which aims at combining rigour and flexibility. On the one hand, it provides an interesting range of choices in order to give third parties secure access to microdata files. On the other hand, it contains a series of guidelines to ensure the confidentiality of tabular data. The measures in the normative framework apply both before the constitution of statistics (methods of obtaining consent from the persons


- 213 -


involved) and after (microdata files protected by means of disclosure protection techniques accompanied by contractual terms and conditions of varying degrees of severity; remote access to data; guidelines on the confidentiality of results tables).

The pitfalls and stakes resulting from such a range of measures will be presented next, together with the policy decisions the ISQ has taken in order to avoid as much as possible any ensuing problems. The advantages of the ISQ’s normative framework on quality will be presented and will target the following three dimensions of quality: accessibility, timeliness, and accuracy. Finally, in concluding, I will review what has been accomplished up until now and give you a brief overview of our future direction, as regards both projects and the positioning of the agency.

Contact: [email protected], http://www.stat.gouv.qc.ca

Session 29: Computer Assisted Surveys Backstage

Chair: Jelke Bethlehem, Statistics Netherlands

New Strategies to Improve Quality of ISTAT new CAPI/CATI Labour Force Survey by S. Bergamasco, S. Gazzelloni, L. Quattrociocchi, R. Ranaldi and A. Toma, ISTAT, Italy

During the last two years, Italian National Institute of Statistics (ISTAT) has undertaken a whole project aimed at redesigning the Labour Force Survey (LFS) in order to fulfil the European Union (EU) regulation. The main project objectives has been not only achieving in fulfilling the EU requirements but also deploying a survey system based on high quality levels in every task composing the process survey.

The major enhancements implemented can be itemized in 1. The adoption of a Mixed Mode survey technique (CAPI/CATI survey); 2. The implementation of a detailed monitoring system and 3. The deployment of an information system supporting the “whole survey process” from the sample definition to the data collection and elaboration.

The adoption of a Mixed Mode survey has been due to the need of interviewing the households belonging to the sample for four waves: the first interview is generally carried out by CAPI technique, the last three interviews are carried out by a CATI technique or by a CAPI interview depending on the household willingness to be contacted by telephone. Moreover, firstly the CAPI interviews are conducted by a professional interviewers network managed directly by ISTAT and regularly managed and monitored by a local intermediate level represented by ISTAT regional office (UURR) and the secondly the CATI interviews are carried out by a specialised CATI supplier.


- 214 -


To sum up all the needs briefly described above has requested the design and deployment of a complex information system whose core is a database able to manage all the tasks assigned to ISTAT interviewers, to co-ordinate the Mixed Mode technique and to contain both the survey data and the management information (the data process).

From a technological perspective the information system grants that all the interviewer activities are achieved by reliable hardware and software tools assuring all the requirements of a secure information system (data privacy, data availability and data integrity) and simultaneously, from an organisational perspective, provides a monitoring system based on a series of indicators rigorously defined in order to plan and control the methodological choices, the implemented survey process and the quality of information collected.

The goal of this paper is to focus on the solutions ISTAT has deployed to succeed in a so demanding process survey renovation.

Contact: [email protected], [email protected], [email protected], [email protected], [email protected]

Quality Indicators for Data Collection via Computer Assisted Personal Interviewing by C.A.M. van Berkel, J.A. van den Brakel and M. Vosmer, Statistics Netherlands

The data collection process on behalf of computer assisted personal interviewing starts with receiving samples and questionnaires, it ends with delivering the collected data accompanied with fieldwork reports. Indicators are presented by which the quality of the data collection process and of samples, questionnaires, collected data and fieldwork reports can be measured and evaluated. The indicators can be used to improve the survey process. All indicators are mathematical functions and follow the same standard format.


Catalysts for Change: The Rationale for Mixed Mode Data Collection in the UK Jacqui Jones, Helen Borgerson , Gill Williams, Linda Scott and Jennet Woolford Office for National Statistics, United Kingdom

The UK Office for National Statistics (ONS) is responsible for conducting business surveys, social surveys and the decennial census. The outputs from these sources are used for a variety of purposes ranging from government policy making to academic research. As in other national statistics institutions ONS has gradually introduced additional modes of data collection into its survey designs. For example telephone data


- 215 -


entry for business surveys and computer assisted interviewing (CAPI and CATI) for social surveys. For the future the use of web data collection is being considered for business surveys, social surveys and the census. The catalysts for introducing additional modes of data collection have generally centred on rationales such as providing more choice and flexibility for respondents, at least maintaining response rates, minimising non-response bias, moving validation to the point of data capture and improving timeliness. A further catalyst has been the pursuit of operational efficiency savings.

This paper provides an overview of the introduction and planned introduction of additional modes of data collection in ONS and identifies the rationale for their introduction. Case studies will be described in the paper from business surveys, social surveys and the census. The case studies will outline the impact of mixed mode data collection and map these to the data quality dimensions defined for the European Statistical System (relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, coherence and completeness).


The New Survey on Producer Prices on Non-Domestic Market: A Mixed Mode Data Collection by Valerio De Santis and Tiziana Iacobacci, ISTAT, Italy

In this paper we show some methodological aspects and preliminary results concerning the comparison between data collected by web questionnaire and those collected by papery form, in the context of the new monthly survey on producer prices on non-domestic market (hereafter PPIND).

In the PPIND survey a new mode of data collection has been adopted, together with the usual procedure based on papery form: the new one was based on electronic questionnaire, available on an internet web site suitably designed and implemented for the PPIND survey.

The web procedure was set up both for the construction of the base of the index period and for the monthly data collection.

The reporting units, selected for the sample, have had to choose either of the two drawing up modes for recording and sending the price data to ISTAT. In the light of the wild spread of the internet tool on the business field, our goal, largely achieved, was lead the most of reporting units to adopt the electronic questionnaire, unless some clear difficulties (i.e. the internet connection wasn’t available), for yielding some significant quality improvements on the data production process, in comparison with


- 216 -


the use of the papery form; the improvements will be measured by appropriate quality indicators, about which we will focus our attention.

The adoption of the electronic data collection on web has allowed as follow:

• a real time data communication: the information flow is recorded and/or updated and saved on web form by the reporting units; quotations are promptly captured by the Oracle database set up for this survey;

• a smaller number of employees involved in the data recording operations: the reporting units, in fact, have filled in the electronic form;

• a lower sending costs: since the most of reporting units have adopted the web form, ISTAT sends by postal service a reduced number of papery questionnaires and it makes the survey less expensive;

• using the web questionnaire, reporting units have been able to directly solve some errors occurring while filling in the form. In fact, behind the web questionnaire a system of data control – realised by warning messages or by constrained variables – has been designed for reducing both the typing errors and the incoherence errors, the last ones due to the complex structure of the form (two kinds of prices, f.o.b. and ex-work, monthly data for two-year period, information about the country destination within the euro-zone/non euro-zone).

Reporting units have answered positively to the web questionnaire. By analysing the first results – at the beginning of the data collection operations – about the 50 per cent of reporting units decided to choose the web questionnaire. Further, about 20, 25 per cent of the reporting units, that firstly chose the papery questionnaire for the two-year period, have been available to adopt the web questionnaire.


An Experience of Internet Data Collection for a National Survey on Graduates by Francesca Brait, Maria Pia Sorvillo and Marina Venturi, ISTAT, Italy

The development of new methods of data collections is one of the main tasks of national statistical offices, in order to improve the quality of collected data and, at the same time, to minimize the statistical burden on respondents. Under this regard, new technologies and the spreading of Internet can help, both from the collector and the supplier side.

In the field of analysis of transition from school to work, Istat (Italian national statistical office) carries out a set of 3 surveys. One of these is addressed to those who have completed upper secondary school (diplomati). The sample design includes a first step


- 217 -


(sample of clusters of schools) and a second step (extraction of the sample of graduates to interview).

The schools identified in the first step are requested to send the complete list of those who graduated three years before, extracting information from their administrative registers. In the previous editions of the survey, each school in the sample received a paper form to fill in. This solution presented some drawbacks: the consistency of information in every paper form had to be checked ex post, and some corrections had to be made calling back the school.

Starting from this experience, a different strategy was decided for the new edition of the survey, planned in 2003/2004: to ask each school for the complete list of graduates, and to organize an electronic exchange of data from the schools to Istat via Internet.

The new collection system is realized on a web platform. Each school received a user-id and a password to enter an application in a web site at the address http://diplomati.istat.it. Once entered the application, the school had the opportunity to fill in an electronic questionnaire or to upload data from a standard format file. For both choices, data were checked on-line and, if completely consistent and correct, they were entered into a data base at Istat service centre.

The application architecture is client/server, which allows lower occupation of the internet connection, speeding up the collection operation and the sending of the form.

A three-layer architecture has been created, as follows:

1. The client, which can be a simple PC with an Internet connection and which connects via a Netscape or Explorer browser to the ad hoc site.

2. The web server, which is Apache, where the HTML and JAVA servlets are located.

3. The DBMS ORACLE server, which is used to store data.

This innovation in the process of data collection has guaranteed a diminution of statistical burden for respondents; furthermore, an improvement in data quality has been achieved, as shown by a comparison with data sent on paper form, because of on line checking procedures.



- 218 -


Session 30: Quality of National Accounts Museum

Chair: José Mata, National Statistical Institute, Portugal

The Reliability of Quarterly General Government Data Compiled in the Frame of National Accounts by Albert Braakmann, Federal Statistical Office Germany

Quarterly receipts, expenditure and financial deficit/surplus of general government according to ESA-95 are not published at present. Instead, the European Pact on Stability and Growth is based on half-yearly data. However, there is a growing pressure to publish quarterly general government data and at the European level a dis-semination of such quarterly figures is intended in 2005, if the reliability is sufficient.

The paper focusses on the reliability of quarterly data on receipts and expenditure of general government in Germany. The first part of the paper will briefly present the more theoretical background of quality measurement in statistics in general and the meaning of reliability in particular. The second and central part will show reliabilty indicators based on figures for the German general government sector. Conclusions will be drawn at the end.


How the Judge the Reliability of the Dutch Provisional National Account by Remko Hijman, Henk Nijmeijer and Brugt Kazemier, Statistics Netherlands

It is not only important to produce good quality statistics, but also that the users of these statistics believe that they are of good quality. Therefore, it is necessary that the subsequent provisional estimates of the national accounts show a similar picture of economic performance. I.e. the subsequent estimates have to be sufficiently reliable.

In this paper, it is analysed to what extent this requirement is met, taking into account that this requirement seriously conflicts with timeliness. A computer program was developed to enable a quick reliability-check to a large number of economic variables, derived from the national accounts. To conduct this test, a minimum reliability standard is defined, with respect to the difference between an estimate and the final estimates.

In many cases these checks will signal one-time events, but sometimes they will signal latent problems in either the basic statistics, underlying the national accounts, or the compilation (process) of the national accounts itself. Assuming that these problems will be solved, these checks may lead to more reliable provisional estimates, and in some cases perhaps more accurate final accounts.



- 219 -


Positive Effects of Measuring Revisions in Balance of Payments Statistics by Jean-Marc Israël, Carmen Picon-Aguilar and Violetta Damia, European Central Bank, Germany

Quality is a subjective concept and encompasses all aspects of how well goods or services meet users’ needs. When applied to statistics, quality is linked to users’ expectations about the information content of the disseminated data, e.g. sufficiently reliable data delivered in a timely manner that are also revised, when needed, with the aim of fostering economic and financial analyses and reducing noise and uncertainties in modelling, in particular forecasting exercises.

This study is based in two pillars; (1) the cascading structure offered in the IMF Data Quality Assessment Framework (DQAF) for balance of payments (b.o.p.) statistics, which has been used also as reference background of quality, and (2) the main findings of the Task Force on (Output) Quality2, which was co-ordinated by the ECB (DG-Statistics). The purpose of the paper is to measure output quality, using some quality indicators which have been selected to be operational for b.o.p./i.i.p. outputs at the European level.

Measuring quality means that the various trade-offs among the numerous dimensions and/or elements of quality are taken into account, e.g. between accuracy and timeliness. Another example is the trade-off between accuracy and reliability, i.e. later estimates incorporating more information and statistical analysis, as well as involving more effort, generally increase accuracy, while they affect the reliability of the early estimates. Against this background, the intention of this paper is not to present a unique measure to indicate the level of quality of b.o.p. data, but to identify the different aspects of data quality that could be enhanced.

In the European context, the selected indicators to operationally assess the quality of b.o.p. data are expected to reflect the revision studies as an element of the “accuracy and reliability” dimension3 and the “consistency” element within the “serv-iceability” dimension. The related quantitative indicators may be simple descriptive statistics or more sophisticated statistical/econometric calculations. Some of these indicators could be externally disseminated, and be the basis of a quality report for users of b.o.p. statistics. Such a report should read easily and address the main concerns expressed by the users; it should also clarify that a qualitative assessment is

2 Task force mandated by the CMFB whose participants are Banca d´Italia, Banco de España, Banco de Portugal, Bank of Greece, Banque de France, Central Statistics Office of Ireland, Deutsche Bundesbank, De Nederlandsche Bank, ECB (DG-Statistics), Commission (Eurostat), Office for National Statistics, Sveriges Riksbank, Suomen Panki and Ufficio Italiano dei Cambi. 3 Integrity and accessibility, although also considered to be important, are in general well covered by the EU countries in their b.o.p./international investment position (i.i.p.) statistics.


- 220 -


as important as quantitative indicators and that enhancing quality is a stepwise and costly process.


Measuring the Quality of Balance of Payments Statistics by Sabine Guschwa, Deutsche Bundesbank, Germany

4

During the previous years, a lot of work has been dedicated to quality issues in the statistical community - at international as well as at national level. Many activities have been carried out to assess and measure quality of various kinds of statistics.

However, measuring and reporting on the quality of statistics is a complex issue since quality is a multidimensional concept. While some dimensions can easily be measured, some of them are difficult to capture. The dimensions of quality are interrelated with existing conflicts between them and quality does do not only focus on the output side, but also on the input side of statistics. It is therefore rather difficult for the statistics compilers to find an optimal mixture of quality aspects. Consequently, the challenge is to find a balance between the variety of quality dimensions and to appropriately communicate a balanced quality assessment to the users. Moreover, the problem is enhanced by the fact that users´ needs for quality information vary depending on their needs for statistics.

The paper at hand sketches the latest developments of the work on quality in the field of balance of payments statistics at the European level by taking into consideration the work carried out by the joint ECB/Eurostat Task Force on quality. Starting from internationally accepted quality guidelines, the IMF data quality assessment framework (DQAF), some indicators on output quality were selected and further developed which were regarded as most important to users. The paper shortly pre-sents several quantitative and qualitative quality indicators and further concentrates on some of them. The modalities of the implementation of the indicators are discussed and the paper also attempts to exemplify several limitations. After elaborating on the indicators, it is considered how they could be communicated best to users. Possible approaches to describe and communicate quality indicators are discussed by assessing these approaches against the perceived needs of users and experiences made. Finally, the current work on the national level will be briefly outlined as well as further considerations for the future.


4 The views expressed in this paper are those of the author and do not necessarily represent those of the Deutsche Bundesbank or the Deutsche Bundesbank´s policy.


- 221 -


Fiscal Data as a Tool for Improving Information Quality in the Trade Satellite Account by Silvia Biffignandi and Stefano Pisani, University of Bergamo, Italy

This paper focuses on the use of administrative data (in the specific case, fiscal data) for improving statistical information quality, mainly relevance and accuracy.

To be precise: a) the use of fiscal data has demonstrated that it is possible to improve the information on the structural characteristics of the internal trade sector, especially regarding the gain in more detailed information; b) the accuracy which may be obtained by working on fiscal data is discussed.

In this study information coming from two different types of taxes is compared. The first tax is VAT (value added tax), which is applied in a standardized way at European level. The second tax is the Regional Tax on Production (in Italian IRAP), which is a tax applied in Italy. Each source of data is constructed with reference to the motives for which the tax is applied, therefore each gives a partial description of internal trade. In order to use the above-mentioned sources for economic analysis, statistical work is required.

Firstly, this paper deals with the problem of transforming administrative data into statistical economic information. With this aim, a conceptual framework which links IRAP and VAT databases is proposed. The framework formalizes the logical link between fiscal definitions and the definitions of the satellite account for trade (or more generally national accounting definitions). The conceptual framework proposed allows analyses on administrative data to be carried out which are similar to that usually carried out on macroeconomic aggregates.

Secondly, the authors give a critical evaluation of administrative data quality and discuss how an appropriate application of fiscal data requires that the limitations that these intrinsically have be taken into consideration, i.e. mainly under-declared revenues or over-declared costs. Moreover, the quality of information is discussed, taking into account the fact that fiscal data is a powerful tool for statistical analysis. Given the censorial nature of the data used, structural analyses can be performed which are highly detailed both from the sectorial point of view as well as from the territorial point of view.

In conclusion, the authors, using the detailed information outlined above, apply data analysis techniques to VAT and IRAP information, aimed at identifying clusters of enterprises which are not classified a priori. Clusters are based on the homogeneous structural characteristics of the enterprises themselves. Special attention is paid to the level of efficiency in delivering a service as well as on the role of commerce in consumer price formation. Relevance of the obtained information is underlined.


- 222 -


Bibliography

Biffignandi S. Pisani S. (2000) “Satellite Accounts For Trade: How To Build Them And How To Set Up Suitable Data”, paper presented at Second International Conference on Establishment Surveys, Buffalo , N. Y. USA, June 17-21 2000.

Falorsi P. D., Pallara A., Russo A. (2003) Temi di ricerca ed esperienze sull’utilizzo a fini statistici di dati di fonte amministrativa, Milan, Italy.

Viviani A. (2002) “Statistics for Government in Local Public Administration” paper presented at International Conference on Economic and Social Statistics, Canton, 17-18 December.


Consistent Micro Data for Efficient Compilation of Quarterly and Annual National Accounts by Karl-Gustav Hansson, Statistics Sweden

Quarterly and annual national accounts in fixed and current prices are neither con-sistent with each other, nor internally. The same goes for their preliminary values and different revisions. The discrepancies are due to how basic data actually are compiled and the national accounts are consolidated and balanced. It can be shown that the use of conventional produced price indices imply that the accounted quantities of goods and services are not the same in the current a fixed accounts, as well as impose residuals between production and expenditure approaches. It can also be shown that conventionally produced price indices can create misleading quarterly distributions on the micro level. All this implies that the outcome of economic analysis and forecasting are dependent of which version is used. A solution to these problems is to consider a new set of desired properties for the national accounts, and using IT techniques based on a structure of straight forward data administration tables for continuous consolidation of monthly, quarterly and annual national accounts. This would lead to a considerable decrease in the differences between preliminary and revised figures of national accounts and in the residuals between the production and expenditure approaches.



- 223 -


Ensuring High Quality Financial Markets Statistics: A Methodological Framework by Francesco Mandalá, European Central Bank

This paper introduces a methodological framework for ensuring high quality financial market data. The framework consists of a formal definition of high quality financial market data and of necessary conditions defining a robust filtering algorithm. In this setting, a simple rule for the detection of outliers in high frequency financial data is proposed.


Revisions of National Accounts 1980-1998. Swedish Expenditure Accounts and GDP in Some Countries by Lars-Erik Öller, Statistics Sweden and Stockholm University and Karl-Gustav Hansson, Statistics Sweden

A revision generally raises a preliminary growth rate. The revision distributions are skew, often with fat tails of outliers. For many variables, revisions are correlated with the business cycle, i.e. the size of a change is underestimated. This results in a tendency toward bimodality in the frequency distributions. Systematic differences in the seasonal patterns further emphasize the disparity between preliminary and final data. We identify where in the accounts the greatest benefits from increased reliability may be achieved. In the international comparison Statistics Canada has the smallest revisions.


Modelling the Construction of a Social Accounting Matrix in the Context of Statistical Matching by Marcello D'Orazio, Marco Di Zio and Mauro Scanu, ISTAT, Italy;

In the last years, interest on Statistical Matching problems has increased. This is due to the large amount of data-sets available and, at the same time, to the need of timely and not costly information. Actually, Statistical Matching techniques aim at combining information from different sources. In particular, it is assumed that the two sources (e.g. two samples) do not observe the same set of units, so that neither merging nor record linkage techniques can be applied. In this paper we present the results of a real Statistical Matching problem related to the estimation of the Social Accounting Matrix in ISTAT. A Social Accounting Matrix is a system of statistical information containing economic and social variables in a matrix formatted data framework. The matrix includes economic indicators such as per capita income and economic growth. In Italy,


- 224 -


as well as in other countries, an archive containing all the necessary variables is not available. Hence, the Social Accounting Matrix is built by means of the fusion of the Household Balance Survey conducted by the Bank of Italy and the Household Expenditure Survey conducted by the Italian National Statistical Institute. In this paper it will be outlined how this application can be modelled in the Statistical Matching context. Once clarified the statistical issues and the hypotheses that cannot be avoided, the emphasis will be posed on the consequences of the Conditional Independence Assumption, i.e. the assumption of independence between the variables observed distinctly in the two sources conditionally on the commonly observed variables. Finally, it will be outlined how this assumption has been overcome in the application under consideration.


Session 31: Corporate Quality Programmes I Großer Saal Chair: Heli Jeskanen-Sundström, Statistics Finland

Quality Management at Statistics Sweden: History, Current Work and the Future by Mats Bergdahl and Lars Lyberg, Statistics Sweden

Statistics Sweden started systematic quality work, based on a TQM approach, about a decade ago. Early work included training of quality facilitators (pilots) and managers on different levels, assessment of the organization’s quality level according to the Swedish SIQ model, and the conduct of numerous improvement projects. A lot of good results were achieved and the experiences proved fruitful. However, it was clear that the approach needed some adjustment and further development.

Internal work now includes development of a new strategy and value system for the agency, identifying our agency’s weak and strong points as a basis for improvements, refining our customer satisfaction surveys, continued work on current best methods documents for our most important processes, a rolling system for quick audits of all our statistical products, and the continuation of individual process improvement projects. The agency has recently decided to adjust parts of the organization to better accommodate data collection needs of establishment surveys. This reorganization is inspired by process views adopted by other agencies such as ONS, ABS, Statistics Netherlands, and Statistics Canada.

The agency also initiated the launching of a Leadership Group on Quality in 1999 aiming at providing guidelines for quality work within the European Statistical System (ESS). A set of recommendations for quality work within the ESS was presented at a conference in Stockholm 2001. Statistics Sweden now participates in projects on response burden, customer satisfaction surveys, self-assessment and the use of


- 225 -


process data in statistics production under the umbrella of the LEG implementation group. Statistics Sweden furthermore participates in international quality work such as Eurostat’s working group on quality assessments and in refining the ESS quality framework.

This paper will review these efforts in some detail with a view towards the future.


Data Quality in the Context of International Organisations: OECD Experience in the Implementation of a Corporate Quality Framework by Enrico Giovannini and Denis Ward, OECD

For international organizations, the quality of statistics disseminated depends on two aspects, the quality of national statistics received, and the quality of internal processes for collection, processing, analysis and dissemination of data and metadata. In several fields, national statistics are developed closely in accordance with international standards. On the other hand, statistical processes at the international level are often derived from best practice developed at a national level. Thus, there is a clear inter-dependence between the two aspects.

In this context, the improvement in the quality of OECD statistics is one of the main objectives of the current OECD Statistics Strategy (OSS). Several projects have been launched in the framework of the OSS and considerable improvements have been achieved over the period 2001-2003. These initiatives will enhance quality, though the need was felt for a common framework that could be used across the Organisation to systematically assess, compare and further improve OECD statistics. This need led to the development of the “Quality Framework and Guidelines for OECD Statistical Activities” presented in this paper.

The OECD Quality Framework focuses on improving the quality of data collected, compiled and disseminated by the OECD through an improvement in the Organisation’s internal statistical processes and management. Obviously there will also be a positive spillover effect on the quality of data compiled at the national level.

This paper describes the main elements of the OECD Framework, focusing on the internal review processes embodied, such as the use of checklists and self evaluation forms for new and existing statistical activities. It also describes the results of the first year of implementation of the framework (in 2003) and presents some preliminary views on future adjustments to the framework arising from our experience this year. Finally, it includes some initial thoughts on the possible application of such a framework by other international organizations.



- 226 -


Restructuring the Office for National Statistics to Improve Quality by Frank Nolan, Office for National Statistics, United Kingdom

Business continues to restructure in a drive to gain increased profits and efficiency. This trend has also been present in national statistical offices. Traditionally the statistical office has been organised in a vertical structure which different departments have responsibility for the end-to-end processing of different outputs. This organisational structure has been challenged in the last decade as national statisti-cians see advantages in moving to horizontal structures. The move to restructure has been brought about by several developments: the need for greater focus on customers (both supplying and receiving data); better understanding of the commonality of statistical processes through a system of value added; the need for investment in new technical infrastructure; and the opportunities for standardisation.

The paper will look at the experience of statistical modernisation taking place in the United Kingdom's Office for National Statistics. The elements of quality improvement will be highlighted, along with the challenges.


The role of Information Technology and Quality in Statistics: A new Approach and Perspectives at ISTAT, by Vittoria Buratta, ISTAT, Italy

Quality of statistics must be approached today taking into consideration the new methodological and technological framework.

The recent ICT innovations have deeply changed the world of statistics first of all in terms of accessibility. In a context of growing information the key values for users and producers are quality and orientation ability.

To reconcile these needs, the NSIs are more and more often directing their information processes towards the logic of an “information system”. That has required - and requires - a change in the way of thinking of statisticians and technical experts with an important impact on methodologies, technologies, organisation and, above all, the levels the information is exchanged at.

The development of a logic of statistical information system (SIS) leads to a more efficient and effective statistical activity, rationalising information capturing by types of respondents and allowing for an ex-ante integration of the information coming from different sources as well as increasing the quantity and quality of the information made available to the various users. Yet, it is necessary for this process to be accompanied by the development of tools and methods that could allow for reading and dissemination of the information available. These developments “user oriented” require a new


- 227 -


transparent approach to quality management based on standards, guidelines, measurement and evaluation.

The need for a renewed approach to quality of statistics in Italy has been one of the basis of the restructuration process. Due to the new organization now in Istat there is a “Direction for Methodology and IT. This direction is in charge of the definition of the strategy for quality in statistics.

The first step of this strategy will be the identification of “standards” for all the information processes. Therefore these standards will be applied to statistical surveys and statistical information systems.

The main objective of the “quality standards policy” will be the achievement of a good common level of quality performance for these processes and, at the same time, a good common level of documentation. In other words the standards will regard data and metadata.

The second step will be the release of statistical guidelines for the national statistical institute and for the national statistical system.

The main results will be:

A common vision of quality for national statistics

An operational guide for the training of researchers and technicians of the national statistical institute

The opportunity of reengineering of the oldest statistical processes.

It is necessary to outline that the definition of quality standards is a pre-requisite for other positive actions in the field of quality. I refer to monitoring and evaluation of processes and products as qualifying functions in the organization.

The paper will provide a presentation of the new “system oriented” strategy for quality adopted in the Italian national statistical institute.



- 228 -


Session 32: Small Area Estimation Großer Saal Chair: Marta Haworth, Office for National Statistics, United Kingdom

Evidence-Based Guidance on the Applicability of Small Area Estimation Techniques by Patrick Heady, Office for National Statistics, United Kingdom

The theoretical development of small area estimation techniques over the past two to three decades has been very rapid. But practising statisticians are still faced with something of an information gap when it comes to choosing methods in practical situations. There is a plethora of advice on modelling methods and their associated diagnostics which ensure that a sufficiently dedicated and knowledgeable statistician is likely to come up with a reasonable method “in the end”. Equally important in practice, however, is knowing where to start. The search for the ideal method is likely to be considerably shorter if the statistician can draw on a stock of methods and models that have been shown to be effective for estimating similar variables, in countries with similar societies and similar statistical system to her own.

One of the major objectives of the EU-sponsored Eurarea project was to provide information of this type. Taking advantage of NSIs access to 100-percent data from censuses and population registers, the project used simulation methods to assess the performance of alternative estimation techniques to estimate the characteristics of real local populations (rather than the model-generated data sets more commonly used to assess the properties of such methods) in six different European countries. In this paper we examine both the commonalities and the differences between countries in the relative performance of different estimators (Direct, GREG, regression-synthetic, and EBLUP) when applied to estimate income, household population and unemployment at NUTS3-level and below.

The paper concludes with a discussion of the impact of differences in national sta-tistical systems for the applicability of these methods.


Some Aspects of Small Area Statistics and Data Quality by Jan Kordos, Central Statistical Office, Poland

The author discusses some aspects of data quality of small area statistics obtained from different sources of statistical data. The following sources of statistical data are considered: a) censuses, b) sample surveys, and c) registers. In the beginning the Eurostat’s (2000) concept of data quality related to small area statistics is considered. The following components of data quality are deliberated: (i) relevance, (ii) accuracy of


- 229 -


estimates, (iii) timeliness and punctuality in disseminating results, (iv) accessibility and clarity of the information, (v) comparability, (vi) coherence, and (vii) completeness. The author starts with a census of population, which plays the central role in the provision of small area data about people, families and households. A post-enumeration survey is considered as a tool for data quality assessment and for correction of some characteristics for small area (e.g. counties in USA, and poviats in Poland). Some examples from Polish experiments are quoted. Since censuses of population are co-ordinated and integrated internationally, so other components of data quality are satisfied.

Sample surveys are much more complicated for small area estimates and assessment of their data quality. There are problems with accuracy, timeliness, comparability and completeness. The author reviews the methods that are used to produce direct and indirect estimates for small areas. To solve these problems the author suggests designing a sample survey from the outset for small area estimates. Marker’s (2001) approach is considered as one of the solution.

Registers are used both for direct small area estimates and as auxiliary information for small area estimation. It is emphasised that in some countries with well-developed civil registration systems, census and survey data can be successfully used together with data from administrative records. Advantages and disadvantages of registers for small area statistics are considered. Some concluding remarks are presented.

References

EUROSTAT (2000). Definition of Quality in Statistics. Eurostat Working Group on Assessment of Quality in Statistics, Luxembourg, 4-5 April.

KORDOS, J. and PARADYSZ, J. (2000), Some Experiments in Small Area Estimation in Poland, Statistics in Transition, Vol. 4, Number 4, pp. 679-697.

MARKER, D. (2001), Producing Small Area Estimates From National Surveys: Methods for Minimizing use of Indirect Estimators, Survey Methodology, Vol. 27, No. 2,pp. 183-188.

PARADYSZ, J. (1998), Small Area Statistics in Poland - First Experiences and Application Possibilities, Statistics in Transition, Vol. 3, Number 5, pp. 1003-1015.

PLATEK, R. and SÄRNDAL, C.-E. (20 01). Can A Statistician Deliver? Journal of Official Statistics, Vol. 17, pp. 1-127 (with discussions and rejoinder).

RAO, J.N.K. (2003), Small Area Estimation, John Wiley & Sons, New Jersey.



- 230 -


Does the Model Matter? A Comparison of the Accuracy of Model-Assisted and Model-Dependent Estimators of Domain Totals by Risto Lehtonen, University of Jyväskylä, Finland, Carl-Erik Särndal University of Montreal, Canada and Ari Veijanen, Statistics Finland

In this paper we examine the estimation of totals for domains and small areas, for a sampled finite population. We argue that that the choice of the model, including its mathematical form, parameters and effects, is an important determinant of the quality of the resulting estimates of domain totals. We compare the effect of model choice on the accuracy of two different estimator types of domain totals, model-dependent estimators and model-assisted estimators. Synthetic estimators and generalized regression (GREG) estimators have been chosen as special cases of these estimator types. We examine selected models of the family of generalized linear mixed models. For a continuous response variable, fixed-effects and mixed linear models are used, and for a polydomous response, we use the corresponding logistic models. We show that model improvement (the transition from a weaker to a stronger model) has very different effects on the different estimator types. The aspects of model improvement to be discussed are (a) the structure of the fixed-effects part of the model, (b) the structure of the random-effects part, and (c) the mathematical form of the model. We also show that the difference in accuracy bet-ween the different estimator types depends on the specification of these aspects of the model. For a well-specified model the difference in accuracy between Synthetic and GREG is negligible, but it can be substantial if the model is misspecified. Synthetic then tends to be highly inaccurate. By contrast, GREG continues to work well, because it is much less sensitive to the misspecification. The results are based on simulations with repeated samples drawn from two finite populations, one artificially constructed, the other constructed from real data from the Finnish Labour Force Survey. The paper is an extension of Lehtonen, Särndal and Veijanen (2003).

Reference:

Lehtonen, R., Särndal, C.-E. and Veijanen, A. (2003). The effect of model choice in estimation for domains, including small domains. Survey Methodology 29, 33-44.


Small Area Estimation in the Israel Labour Force Survey by Natalie Shlomo, Central Bureau of Statistics, Israel

There is a great demand for providing estimates on labour force characteristics in small localities and geographic areas, particularly where resources are low. In general, direct estimates on small localities from a sample that was not originally intended to take these estimates into account will have high variances and low credibility. The Israel Labour Force Survey (LFS) is designed as a rotating quarterly panel survey based on


- 231 -


approximately 10,000 households per quarter. The sample was designed to provide labour force estimates for large localities and other geographical regions in the country by main population groups, sex and age groups. The survey also provides comparative estimates between consecutive quarters and years. The sample is not large enough, nor is the current method for calibrating sampling weights applicable, for providing valid direct small area labour force estimates.

To calculate reliable labour force estimates for smaller localities, we will integrate administrative sources with the survey data from the Israel LFS. The main adminis-trative source that will be used is the National Unemployment Office File which provides data on all claims for benefits for the unemployed persons in the localities of Israel. Additional administrative sources include demographic data (age, sex, ethnicity) for each locality as well as labour force data from the previous census carried out in 1995.

Integrating the administrative data with the survey data can provide efficient estimates for the small localities and geographical areas, by offsetting the large variances of the direct sample estimates due to their small sample sizes, provided of course that the biases in the administrative data are small. A special weighting technique will be carried out on the LFS data to obtain more efficient direct estimates.

In this paper, we will look at model-based methods for small area estimation, including the well known Fay and Herriot (1979) model for cross-sectional data, as well as an extension for time-series data presented in Rao and Yu (1994) in which the rotating panel design of the LFS will be exploited.

This paper will give an account of the results obtained based on the small development towns scattered in the north and south of Israel, where unemployment is high and there is a demand for reliable labour force estimates from the policy makers.

Contact: [email protected], web site: www.cbs.gov.il

Estimating Accurate Weights of Consumer Price Indexes for Small Reference Populations by Rosa Bernardini Papalia, University of Perugia, Italy

The task of estimate consumer price index (CPI) weights for small sub groups of the reference population cannot be achieved by the design-based procedure which use survey data only from the sub group of interest because of the availability of a smaller sample relative to the national sample. Similar situations arise when estimates are needed for many domains obtained by classifying the population according to various characteristics (i.e. low/high income households). In this view, the price movement measurement is weighted by the importance of the item in the spending patterns of the appropriate population group. If used with caution, each index can illustrate and explain the impact of local economic conditions on consumers' experience with price


- 232 -


change. For subpopulations comprising only a small fraction of the total population, there is a small number of sampled households and, the direct estimation method could yield CPIs which are not accurate enough.

In this study we refer to special group indices which are computed using the same methodology as for the reference population index and which have the same item coverage of the reference population. The prices used for the subgroups are then based on data collected for the whole CPI. Each of the indices is computed using a set of expenditure weights representing different basket, but all are drawn from the same reference period. The limitation of the experimental indices depend on the following problems: surveys used to adjust weights for the special groups population are sensitive to sampling and response errors; the amount of sample units used in determining the weights was significantly reduced; outlets selected are identical for all indices and they do not necessarily reflect the consumption habits of each subgroup; the basket of goods and services is the one consumed by the general population, items with zero expenditure are removed but items that may specifically represent the consumption of each sub group are not considered; the prices used are identical to the prices for the general population. Potential dissimilarities between the indices for the special groups and the index for the reference population should result solely from different sets of weights.

The study is aimed to illustrate the use of alternative model-based estimators of the expenditure shares and to propose an alternative estimator based on the maximum entropy principle. The proposed estimator is derived by the estimation of a demand system which consists of a set of budget share equations. Using this estimation procedure: (i) goods results are produced in the case of small-sized samples; (ii) restrictions expressed in terms of inequality can be introduced; (iii) it is possible to estimate systems with a large number of demand equations; (iv) it is not necessary to introduce hypotheses regarding the form of the error term distribution; (v) it is possible to develop demand systems with are coherent with the empirical evidence and with the available knowledge while displaying consistency with consumer-theory assumptions. Estimates are efficient, consistent and robust when the error term distribution is not normal and the explanatory variables are highly correlated.


Local Area Macroeconomic Aggregates Estimation by Alessandro Faramondi, Flavio Foschi, Claudio Pascarella and Augusto Puggioni, ISTAT, Italy

This paper describes methods used by Istat for the estimation of the main economic aggregates about “small territorial areas”, specifically referring to value added (VA) related to Provinces and Local Labour Systems (LLS). LLSs do not follow usual ad-


- 233 -


ministrative standards, they are defined on the basis of commuting flows, whereby job demand and offer coincide the most. Since the sources used are different, the analysis is divided by enterprise size ac-cording to offer estimations.

Enterprises with at least 100 employees undergo census-like surveys. It is possible to gross up data by integrating missing answers through civil registers and disaggregating by Local Unit according to the information supplied by the Statistical Register of Active Enterprises (ASIA).

Enterprises with less than 100 employees undergo instead a sample survey that is tailored to supply accurate estimations by regional territorial areas. Resorting to estimators for small areas requires survey data integration with ASIA, social security data (INPS) and civil registers data. Estimation processes are diversified starting from the quality analysis of the available data obtained through sources integration.

The described methodologies merge into a more general process that is sub-regional estimation. This last allowed to supply - within Eurostat time-table - final estimations with a t+24 months lag that are consistent with information on regional economic accounts.

Finally, the paper shows the results of simulations aiming at evaluating the im-provement in data quality obtained through a composite estimator for small and medium-size enterprises.


Population Statistics and Re-Estimations of Old Age Populations by D.A. Jdanov, R.D. Scholz and V.M. Shkolnikov, Max Planck Institute for Demographic Research, Germany

Increasing growing old-age populations is one of the most challenging socioeconomic problems in Europe. It is connected with pension schemes and the medical care systems. This results in increasing attention to precise estimation of numbers of the elderly people. During the last decades registration systems of European countries provide reliable data on deaths at old ages. However, population estimates at old ages do not have equally reliable quality in some countries. Therefore it is important to develop methods for re-estimating populations up to high ages.

Detailed mortality data (deaths and population by age, year of births, and calendar year) for 19 countries are available in the "Human Mortality Database" (HMD, www.mortality.org or www.humanmortality.de). Populations at ages over 80 in the HMD are re-estimated using extinct and almost-extinct cohort methods. If information on


- 234 -


deaths is reliable and migration is negligible, these methods return precise population estimates.

In some countries data on deaths include an open-ended age interval. The most useful methods for splitting death counts in open age interval into Lexis triangles (i.e. by age and year of births) can be divided for two big groups. The first group based on the different kinds of regression and data for other countries/years. The second group uses model approach, e.g. fit data using stable population model.

Our study presents a comparative analysis of the HMD population estimates with the official population estimates for a range of European countries such as Sweden, Denmark, Germany, France, the Netherlands, Switzerland, Hungary, and Lithuania. This is an indirect way for evaluation of the quality of official population statistics in different countries. The results clearly show that the deviations between the HMD estimates and the official statistics largerly vary across time and by country being generally lower in census years and in countries with the population registers.

Contact: [email protected], Scholz@ demogr.mpg.de, [email protected]

Session 33: Quality Reporting Mozartsaal Chair: Hans Viggo Sæbø, Statistics Norway

Quality Reporting and Quality Indicators in International Trade Statistics by Veronica Corsini, Eurostat

International trade statistics, as produced by Eurostat, measure value and quantity of goods traded between Member States of the European Union and value and quantity of goods traded by Member States of the EU with third countries. They are the official source of information about import, export and trade balance of each Member State, of the Euro zone and of the EU. Community legislation in the field of international trade statistics ensures that the statistics provided by Member States are based on precise legal texts and on definitions and procedures which, to a large extent, have been harmonised.

However, quality of international trade statistics is far from being completely satis-factory and harmonised. Many factors contribute to discrepancies between the Member States, among others the Intrastat system, that is the way trade between the Member States is collected after the Internal Market came into force in 1993, causing the disappearance of the borders and of the Customs formalities for imports and exports. One of the main problems of data quality is given here by the so-called mirror discrepancies: in theory Intra-EU trade statistics should be fully comparable, since it is based on legal texts and harmonised concepts, therefore export from Member State A


- 235 -


to B, as declared by A, should be equal to import of Member State B from A, as declared by B. However bilateral comparisons have revealed major and persistent discrepancies in the various Member States Intra-EU bilateral flows. The main reasons are known but remedies are not easily found.

In order to analyse data quality and to propose way to improve it, the Eurostat unit in charge of International Trade Statistics started at the end of 2000 the yearly production of a Quality Report. This QR covers, in agreement with the official definition of quality in the European Statistical System, the following quality dimensions: timeliness, accuracy, accessibility and clarity, comparability, coherence. Comparable national QRs on trade statistics have been also produced by most of the Member States. Moreover, in 2001, a database containing Quality Indicators for international trade statistics at Member States level has been set up. It stores and updates the information provided by the Member States in their national QRs or contained in other documents related to quality of trade statistics. The QR and this database can be used jointly to monitor and to improve data quality of international trade in the EU.


Quality Report Template for the CBS of Croatia by Lidija Gligorova, Mladen Krulik and Marinela Kustura, Central Bureau of Statistics, Croatia

A three-day workshop entitled the 'Development of the template on quality reporting from the CBS' was held in the CBS of Croatia in November 2003. During the workshop we produced a draft version of the quality report template for the CBS, with the assistance of Johanna Laiho from the Statistics Finland. We are planning to produce a final version of the quality report template by the end of February 2004. The paper will present the experience in developing the quality report template for the CBS.


The Quality Assurance Committee at Statistics Austria by Peter Hackl, Economics University of Wien, Austria

In 2000, in Austria a quality assurance committee (Ausschuss Qualitätssicherung) was established as a working group of the Statistical Council (Statistikrat). The Statistical Council is a legal body, defined by the Austrian Federal Statistics Act 2000 (Bundesstatistikgesetz 2000) as board of 15 independent members appointed by the Federal Chancellor, by ministries and public institutions, giving support to Statistics Austria by comments and recommendations. Among other tasks the Statistical Council gives his assessment of the output of Statistics Austria with respect to quality aspects such as the compliance with scientific standards, and comments on activities of continuous quality improvement. For quality related issues, in spring 2000 the Statistical Council established a special working group, called quality assurance committee.


- 236 -


The presentation reports on the activities and experiences of the quality assurance committee. The main issues have been (i) standards for the documentation of products, (ii) regular feedback talks, and (iii) the publication policy. In all issues, the committee could contribute to the implementation of new devices and the establishment of new standards. In all issues these development happened in collaboration and agreement with the leading executives of Statistics Austria. The next issue of interest will be (iv) the development of human resources within Statistics Austria; the committee thinks that sound training standards, strengthening of collaboration and networking with national and international research institutions including universities, and participation in international research activities are essential for the development towards an efficiently working producer of excellent statistical products.

Contact: [email protected], http://eeyore.wu-wien.ac.at/stat4/hackl/home.html

Taking into Account Quality in the French Business Surveys by Philippe Brion, INSEE, France

French business surveys are conducted by INSEE and statistical offices of ministries. The most important survey is the annual enterprise survey, with a sample of about 200 000 units. Quality aspects are tackled at different steps of the surveys. This paper will focus on three stages.

First, the National Council for Statistical Information plays an important part before carrying out the surveys, at two stages. First with a “usefulness opinion” given about the projected survey, and then with the examination of a report presenting different aspects of the survey by a specialized committee (“label committee”). The questions of burden (taking into account the sampling aspects, and the methods of coordination of samples), the test of the questionnaire and the existence of a users committee are specifically considered by the label committee, which then issues a quality label and decides if the survey will be compulsory or not.

The second stage is relative to the execution of the surveys. Data editing is an im-portant step. For the annual enterprise surveys, a specific software has been developed in the nineties, which is based on the study of internal consistency between the different variables of the questionnaire. Macro-editing is also used, for example by looking at the impact of one unit on aggregates. Internal metadata are produced (for example “rough”, modified and final values) and might be used to check the quality of the parameters of the data editing process.

The surveys being made by mail, the question of timeliness is also important : in order to avoid waiting for all returns of questionnaires, we are now thinking about stopping criteria to decide at what stage it is possible to produce results with the returns, at least at aggregated levels.


- 237 -


Business surveys in France use legal units as collecting units. More and more, problems are raised by complex structures within big enterprise groups. For example, internal flows generated by the creation of new legal units may cause an apparent increase of variables such as turnover. To deal with these situations, we have undertaken the use of a specific (tailor-made) approach of these groups, for some business surveys.

Third, after carrying out the surveys, the survey departments are asked to fill a standard quality report, which is composed of a quality sheet (or four pages) containing indicators on different topics, and of a “free” part giving comments, and descriptions of the methods that have been used.

The return of the results to enterprises, which is mandatory, is also an important element concerning quality : in some cases, the enterprises get their results compared to those of the class (defined by the activity and the size) they belong. This procedure is particularly interesting for repetitive surveys.


Quality Reports at Statistic Austria by Thomas Burg, Statistics Austria

Since the legal status of official statistics in Austria changed, quality is a hot topic at Statistics Austria. Due to the fact that product quality is one of the piles of TQM, quality reporting is an indispensable ingredient for this system.

The paper starts with an overview about the development of quality reporting in Austria throughout the last years.

In the second part a description of the current Quality Report system is given and elucidated with some examples. Along with this it is shown what problems and obstacles an organization like a statistical office is faced when establishing quality reporting.

In the last part the paper deals with the possible consequences of quality reporting. Since the middle of 2003 we started a series of feedback talks in co-operation with the Statistical Council, which are based directly on quality reports of certain statistical products. The feedback talks not only incorporate internal but also important external user groups and as a result of this discussion process possible fields of improvements concerning various quality dimensions are located. Finally the role of quality reports as a source of metadata is brought up which leads directly to the critical question of dissemination and the relation between quality report and standard documentation of statistical products.

Contact: [email protected].


- 238 -


Session 34: Non-Sampling Errors II Spiegelsaal Chair: Peter Lynn, University of Essex, United Kingdom

A Test of the Item Count Methodology for Estimating Cocaine Use by Paul Biemer, RTI International, U.S.A., Douglas Wright, U.S. Substance Abuse and Mental Health Agency The item count method allows survey respondents to remain anonymous when reporting a sensitive behavior such as cocaine use. This is accomplished by including the sensitive behavior of interest in a list of other relatively non-stigmatizing behaviors. The respondent reports only number of items in the list in which he/she has engaged, not which behaviors. If the average number of non-stigmatizing behaviors is known for the population, one can estimate the rate of the sensitive behavior for the population by the difference between the average number of behaviors reported for the population including and excluding the stigmatized behavior. A very large test of this methodology (n=70,000 persons) was conducted for the U.S. National Survey of Drug Use and Health (NSDUH) in 2001 in order to estimate past year cocaine use. The project involved cognitive laboratory research to determine the number of item count questions that should be included and the topics to be covered by those questions (other than cocaine use). In addition, replications of item count questions were embedded in the questionnaire in order to estimate the response variance associated with the item count methodology. This paper reports on the results of the cognitive testing and the estimates of cocaine use that were obtained from the item count methodology. We also report on the results of response error analysis which helps to explain the poor results obtained by the method.


Evaluation of Quality of Mortality Data Using Textual Analysis Ap-proach by Alessandra Battisti, Luisa Frova and Marilena Pappagallo, ISTAT, Italy

The evaluation of the quality of causes of death statistics is an important issue for any country. This study aims to evaluate the quality of death certificates by AIDS in Italy. It uses a method based on textual analysis.

Textual analysis is a qualitative instrument that may be considered a merge of qualitative and quantitative statistical approach. The opportunity of using textual data sources in social research is not a new matter in statistics, but it is an approach not often used, as textual data sources are complex and not immediately ready to be statistically processed.


- 239 -


In this paper a first attempt of textual analysis is conducted on mortality data. In particular, attention is focused on patient died for AIDS from 1997 to 1999 in Italy. Data are examined by geographical area of death.

The way of collecting mortality data in Italy is based on completion of a death certificate in which the physician report all the information on mortality causes of death, that is all the diseases and conditions that contributed to death. A basic concept in classifying causes of death is the underlying cause of death. The underlying cause, defined as “(A) the disease or injury which initiated the train of morbid events leading directly to death, or (B) the circumstances of the accident or violence which produced the fatal injury”, it is selected, among all diseases or conditions reported on the death certificate, by means of a set of International rules defined by World Health Organization.

In this paper the qualitative evaluation of a death certificate is measured through the frequency of “useless” words used for describing the morbidity process that has determined the death. Useless words are defined as those terms not necessary to describe the morbidity process and consequently not influential to the identification of the underlying cause of death. Therefore, the presence of “useless” words in a death certificate can be considered a proxy of bad quality of certification.

The textual analysis has highlighted a geographical distribution of the different groups of “useless” words.

In the North of Italy redundant words are wider spread than in the rest of the country, while in the Center and South of Italy there are more generic and tautological words. Another important result is a very poor association between the use of “useless” words and the year of death.

The application of textual analysis to mortality data can be considered an important tool able to monitor the medical certification and to evaluate periodically the impact and the efficacy of actions undertaken to improve quality of certification.

Contact: [email protected], pappagal@istat

Assessing Cross-National Construct Equivalence in the ESS by Jaak Billiet and Jerry Welkenhuysen-Gybels, Catholic University of Leuven, Belgium

One of the prime objectives of cross-national survey research is to compare concepts across countries or cultures. It is therefore important that these concepts are measured adequately in all of the countries involved in the survey. Moreover; in order that country-scores on items or scales can be compared in a valid way, concepts have to be measured in a sufficiently equivalent way. This paper tries to assess this equivalence by checking whether the indicators of a number of latent traits such as social trust, ethnocentrism and political efficacy measure the same construct or trait in all of the


- 240 -


countries. This will be done by testing for the factorial invariance of factor loadings across countries. Whenever possible, response style effects are taken into account in this analysis. Different methods are compared. The analysis shows that it is possible to obtain comparable measurement elements for a number of concepts that are measured in ESS round 1.


Use of Vignettes in Business Surveys in Implementing the International Definition of Innovation by Dominique Francoz, Ministry of Youth, Education and Research, France

Innovation is internationally defined in the OECD Oslo Manual; this manual is currently in revision. The definition of innovation needs to be broadened to encompass new forms of innovation and also innovation in the service industries. Even in the framework of the Oslo Manual, innovation remains subject to a high degree of interpretation, as key concepts need to be translated to be meaningful in different language and cultural contexts.

To assess cross-country differences in the interpretation and use of the innovation definition, and to better understand countries’ perspectives regarding a broader definition, we have devised a vignettes questionnaire. Vignettes present the re-spondents with brief, ambiguous examples of specific situations or activities that may or may not represent innovation.

The vignettes use non-technical language and ask the respondent to select among a series of possible choices. They reveal how the respondents conceive innovation. We are conducting a two-phase pilot survey using the vignettes: Experts from OECD member states who are participating in the Oslo Manual revision; and firms in the different countries in every industry.

The paper will present the results of this process. It will assess the usefulness of vignettes in business surveys. We also expect that the different points of view of the statisticians of innovation will begin to shed light on response disparities among firms, and especially between those in manufacturing and services industries. Finally, our results may be helpful in analysing the results of the 3 rd Community Innovation Survey and their cross-country comparability.



- 241 -


The Coverage Rate Estimation for the Fifth Agriculture Census in the Presence of Matching Error by Patrizia Cella, Nicoletta Cibella and Tiziana Tuoto, ISTAT, Italy

As “every census misses some units”, the coverage evaluation in the Census has become a very relevant problem for assessing completeness and accuracy of data. ISTAT produces a quality survey (PES, Post Enumeration Survey) to evaluate the Fifth Census of Agriculture undercount using one of the most common Coverage Error Models, the Petersen one (Wolter, 1987). The Petersen Model requests to match Census and PES units, assuming no errors in matching operations. This is a really strong assumption when a computer matching is implemented: automated operations are not avoided from errors which introduce some bias in the coverage rate estimation. This work wants to evaluate the matching errors probabilities and correct the coverage rate estimation from the bias due to those errors.

To obtain a one-to-one match between PES and Census (afterwards the Match), we rely on unique identifying information when available; otherwise we perform a re-cord linkage procedure, based on classical methodology originally proposed by Fellegi & Sunter (1969) which takes in account name and address variables (Jaro, 1989; Winkler, 1995). This automated approach requires parsing and standardization techniques and specified discriminatory decision rules, both may generate errors in matching results.

To evaluate the matching errors, we employ an accurate clerical Rematch Study (the so-called Rematch), assumed to be perfect. Comparing Match and Rematch results we can built a Matching Error Model (Ding & Fienberg, 1994) which allows us to estimate the probability of false nonmatches of matching cases, γ , and the probability of false

matches of nonmatching cases, β . Using estimated matching errors probabilities, we

are able to correct the Petersen model results for the bias due to these errors.

The Post-Enumeration Sample Survey of the 2001 Italian Census of Agriculture collects 6,559 farms whereas the farms observed at the Census in the sampled area are 7,485. The Match procedures link 5,399 records. Applying the Rematch we link 5,839, out of which 5,334 was already matched in the Match procedures. The estimates of the

matching errors probabilities are: 0865.0ˆ =γ and 0903.0ˆ =β . Taking into account

these probabilities, we can correct the estimates of coverage rate, obtained with the Petersen model, for the bias deriving from the matching procedures. The coverage rate estimation increases to about 4%, raising from 0.8231 up to 0.8678.

From our experiences, we recommend to consider the matching errors especially when the match requires automated procedures. When clerical manual rematch is too expensive because of the large number of farms collected, the errors probabilities can be estimated rerunning Match and Rematch procedures on a subsample. Otherwise we


- 242 -


are analysing the possibility of modelling errors taking into account the linkage weights, as e.g. Belin & Rubin (1995).

References

Belin T.R., Rubin B. (1995), “A method for calibrating false-match rates in record linkage”, Journal of the American Statistical Association, vol.90, n.430.

Ding Y., Fienberg S.E. (1994) “Dual system estimation of Census undercount in the presence of matching error”, Survey Methodology, 20, 149-158.

Fellegi I.P., Sunter A.B. (1969) “A Theory for record linkage”, Journal of the American Statistical Association, 64, 1183-1210.

Jaro M.A. (1989) “Advances in record linkage methodology as applied to matching the 1985 Census of Tampa, Florida”, Journal of the American Statistical Association, 89, 414-420.

Winkler W.E. (1995), “Matching and Record Linkage”, in Cox, Binder, Chinnappa, Christianson, Colledge, Kott (a cura di), Businness Survey Methods, Wiley & Sons, pp. 355-384.

Wolter K. (1986) “Some coverage error models for Census data”, Journal of the American Statistical Association, 81, 338-346.

Contact: [email protected] [email protected]

Data Collection Using Mixed Effects Models Tiberiu Postelnicu, Romanian Academy Within quality programs, such as statistical process control (SPC) and Six-Sigma , it is important to evaluate the measurement system or method. When a good measurement system is in place, the measurements of a quality characteristic are precise and therefore the characteristic may be controlled and the variation may be reduced. The usual way to investigate the variability or precision of the measurement system is to conduct a well-described experiment. The variability is often divided into two components: the first caused by observers, and the other by the measurement device itself. Since these components are called “Reproductibility” and “Repeatability”, respectively, the experiment is also known as an “R&R study”.

It is presented that the variances of the different components are estimated with linear combinations of the mean squares from the analysis of variance (ANOVA) table. This combined standard deviation of all components is the variability of the measurement system, which is compared to the width of the lower (LSL) and the upper (USL) specification limit for the quality characteristic.

If the effects of products and observers, the interaction of the two, and the repeatability are modeled with a normal distribution, then the model is called the “random effects


- 243 -


model” (REM). In fact, the REM is the basis on which standard R&R studies are built. But this model makes sense only if results and observers are drawn from a large population, and if the underlying distributions of the populations are approximately normal.

In official statistics the assumption of random observers is often not true. In such situations, it is obviously wrong to model the observer effect as a random component representing a larger population. The assumption that each observer measures a fixed amount from the true value makes more sense. The modification of the model in this way is called the “mixed effects model” (MEM), since it involves both random and fixed effects. The relation between both models s that the MEM converges to the REM , in probability, if the number of observers is large, and the fixed observer deviations stem from a normal distribution.

This paper deals also with the analysis of measurement systems when only few observers perform the measurements. The standard model with random observer effects is not correct and may give misleading results, especially when the number of observers is small. An alternative measure for the gauge R&R, based on the width of a 99% prediction interval is proposed.

References Dolezal,K.K. et al., J.Qual. Technol.,1998, 30 , 163-170. Vardeman,S.B., Technometrics, 1999, 41, 201-211. Lon-Mu Liu et al., Comput.Statistics & Data Analysis, 2001, 37, 455-476.



- 244 -


Session 35: Confidentiality II Leibnizsaal Chair: Lawrence H. Cox, National Center for Health and Statistics, U.S.A.

Sample Design in the Context of Data Security by Manuel Nunez, Robert Garfinkel and Ram Gopal, University of Connecticut, U.S.A.

Consider a statistical database in which there is a per record access fee and one or more confidential fields. A DB user would like to compute statistics based on the confidential data, but given the access fee, the user would settle for a random sample of records and computing a sample estimate of the population statistics. There are two main sources of estimation error: one from using a random sample and another from using a protection mechanism to maintain the data confidentiality. We propose mathematical models to study what are reasonable protection mechanisms that would yield usable data and a reduction of the estimation error. We consider issues resulting from inference threats arising from the combination of the sampling and protection mechanisms. Given the tradeoff between access fees and quality of the estimation, we also provide algorithms to determine the ideal sample size and to resolve other related sample design issues.

Contact: [email protected], Internet: www.cdc.gov

Estimation of the Probit Model Using Anonymized Micro Data by Martin Rosemann, Institut für Angewandte Wirtschaftsforschung, Germany, Gerd Ronning and Harald Strotmann, University of Tübingen, Germany

During the last decade statistical disclosure control has become an important topic in offcial statistics. In particular (sufficient) masking of micro data in order to provide scientists with scientific use files has been a demanding project. However the effects on the statistical analysis of these masked data has been discussed only rarely. In this paper we consider the effect of some widely used anonymization procedures on the estimation of the (binary) probit model. Since the dependent variable is polytomous (dichotomous), we apply post randomization (PRAM) to this variable whereas the continuous explanatory variables are masked alternatively by noise addition and microaggregation. We show that post randomization can be integrated into the estimation of the probit model if the masking procedure is known to the scientist. Moreover estimation results are still reliable (see Ronning 2003) whereas naive use of the probit model leads to severely biased estimates. Masking of the explanatory variable can be seen as incorporating ’errors in variables’ into the model. We present both analytical and simulation results regarding the e_ects of data protection on parameter estimation for this model which is a standard tool in biometric and microeconometric research.


- 245 -


References

Ronning, G. (2003): ”Randomized Response and the Binary Probit Model”. IAW Diskussionspapiere 10 (August 2003).


Protecting Data Quality and Confidentiality by MASSC: A Survey Sampling Based Method by A.C. Singh and F. Yu, RTI International, NC, U.S.A. We propose a method termed 'MASSC' for statistical disclosure limitation (SDL) of categorical or continuous micro data, while limiting the information loss in the treated database defined in a suitable sense. The new SDL methodology exploits the analogy between (1) taking a sample (instead of a census,) along with some adjustments via imputation for missing information, and (2) releasing a subset, instead of the original data set, along with some adjustments via perturbation for records still at disclosure risk. Survey sampling reduces monetary cost in comparison to a census, but entails some loss of information. Similarly, releasing a subset reduces disclosure cost in comparison to the full database, but entails some loss of information. Thus, optimal survey sampling methods for minimizing cost subject to bias and precision constraints can be used for SDL in providing simultaneous control on disclosure cost and information loss. The method consists of steps of Micro Agglomeration for partitioning the database into risk strata, optimal probabilistic Substitution for perturbation, optimal probabilistic Subsampling for suppression, and optimal sampling weight Calibration for preserving estimates for key outcomes in the treated database.

The proposed method uses a new framework in the practice of disclosure limitation in that the original database itself is viewed as the population and the problem of disclosure by inside intruders is considered. (Inside intruders know the presence of their targets in the database in contrast to outside intruders.) This framework has two main features: one, it focuses on the more difficult problem of protecting from inside intruders and as a result also protects against outside intruders, and second, it provides in a suitable sense model-free measures of both information loss and disclosure risk when disclosure treatment is performed by employing known random selection mechanisms for substitution and subsampling. Empirical results are presented to illustrate computation of measures of information loss and the associated disclosure risk for a small data set.



- 246 -


Confidentiality and Linking Administrative and Survey Data: Views from the Public. Main findings and Discussion from a Qualitative Study Employing Focus Groups by Jack Eldridge, Office for National Statistics, United Kingdom

This paper describes the results from a qualitative research project to investigate the range of public attitudes, beliefs and concerns about the sharing and linking of data collected by Government.

Six focus groups were held, with the discussants recruited directly from the general public by Office for National Statistics interviewers. The groups discussed:

- what they thought happened to data collected by government;

- what confidentiality and the term "data used for research and statistical purposes" mean;

- data sharing between government organisations;

- linking administrative and survey data;

- who should have access to and be able to link government data sources;

- concerns about sharing and linking data; and

- how these concerns could be allayed, including suggestions for methods of redress and penalties.

The paper describes the outcomes from the groups and the implications for data collection and use.


Microdata Disclosure by Resampling by Sandra Gottschalk, Centre for European Economic Research (ZEW), Germany

A problem which statistical offices and research institutes are faced with by releasing microdata is the preservation of confidentiality. Official statistics are not allowed to pass on to external users outside the office, unless disclosure limitation is guaranteed. The same holds for survey data, conducted by private or public research institutes, if confidentiality is promised to the respondents. Traditional methods to avoid disclosure often destroy the structure of data. In this paper, I discuss an alternative technique of creating scientific-use-files, which reproduce the characteristics of the original data quite well. It is based on an idea of Fienberg (1997 und 1994) [2], [3] to estimate and resample from the empirical multivariate cumulative distribution function of the data to get synthetic data. The procedure should create datasets which have the same charateristics as the original survey data. The typical user of microdata is interested in


- 247 -


causal relationships between variables. E.g., econometric parameter estimates can be reproduced quite exactly with resamples, though econometricians have no disadvantages in using only “estimations” of the original datasets. As the elements of the resample are drawn from the cumulative distribution function and do not necessarily correspond to any of those individuals in the original sample survey, an identification of true values is not possible. Nevertheless one cannot rule out the possibility of disclosure, as synthetic datasets could be very similar to real observations. Especially, extreme values are at risk.

To estimate the empirical cumulative distribution function is computationally difficult. The exact reproduction of multivariate relationships are not possible until now. For that reason, I pick up a procedure demonstrated by Devroye and Györfi (1995) [1] and Silverman (1986) [5], which avoid the necessity of estimating the distribution function, but directly generate a resample. Even higher dimensional algorithms can be constructed by using directional information in the data, such as the covariance or the correlation matrix. Therefore, the multivariate distribution can nearly be performed. In this paper, I present some applications of this method with (a) simulated data and (b) innovation survey data, the Mannheim Innovation Panel, which is conducted by the Centre for European Economic Research (ZEW) within the German economy. The performance of resamples is demonstrated. In a second step, I compare resampling with a customary method of disclosure control, disturbance with multiplicative error (see e.g. Hwang, 1986 [4]), concerning confidentiality on the one hand and the applicability of the anonymized data for different kind of analyses on the other.

The experiments show that univariate distributions can be best reproduced by univariate resampling, in comparison. Linear regression results can be reproduced quite well if the resampling procedure implements directional information in the data in the form of the correlation structure. If multiplicative disturbance is controlled for in the estimation approach, parameter estimates will also remain. With regard to disclosure avoidance anonymized data with multiplicative perturbed variables performs better on the average. Even though resamples consist of synthetic values, confidentiality problems remain.

References

[1] Devroye, L. and L. Györfi (1985), Nonparametric Density Estimation, New York.

[2] Fienberg, S.E. (1997), Confidentiality and Disclosure Limitation Methodology: Challenges for National Statistics and Statistical Research, Technical Report No. 161, Carnegie Mellon University, Pittsburgh.

[3] Fienberg, S.E. (1994), A Radical Proposal for the Provision of Micro-Data Samples and the Preservation of Confidentiality, Technical Report No. 611, Carnegie Mellon University, Pittsburgh.


- 248 -


[4] Hwang, J.T. (1986), Multiplicative Errors-in-Variables Models with Application to Recent Data Released by U.S. Department of Energy, Journal of the American Statistical Association, 81, 395, 680-688.

[5] Silverman, B.W. (1986), Density Estimation for Statistics and Data Analysis, Monographs on Statistics and Applied Probability 26, London.


Session 36: Survey and Census Design II Forstersaal Chair: Ib Olar Thomsen, EFTA Statistical Advisor’s Office

A Random Sample as a Key for a Better German Register-Based Census by Josef Schäfer, Statistical Office of North Rhine-Westfalia, Germany

In order to reduce the costs and to improve the public acceptance, a register-based census is supposed to replace in Germany the traditional interviewer-based census.

Two main sources will provide the required information: the address-based communal population registers and a buildings and dwellings survey, where the buildings owners are questioned by mail.

The link between the registered persons and the corresponding dwellings will be established by the so-called procedure of “generating households”, which make use of different information delivered from both sources: family-names, birth-names, gender, age, family state, moving in-dates, connection marks for spouses and children with their parents from the register, number and size of dwellings, “main tenants” and numbers of persons of/in each dwelling from the buildings and dwellings survey.

The quality of the (communal) population registers had been tested by the federal statistical office and the statistical offices of the Länder (federal states) in the test survey “Zensustest”. This test survey also allowed an improving of the “households-generating” procedure. The “Zensustest” showed that the total amount of nominal members and shortages in the communal population registers and the corresponding variance between the communities are definitely too large for being acceptable for a census. The “households-generating” procedure has been proven effective in reproducing household-structures and in detecting possible nominal members in small buildings, but not in larger ones. Also in small buildings shortages can partly be found by a comparison of the number of inhabitants reported from the buildings-owner to the number of registered persons, but in larger ones it had been very difficult to identify probable shortages.

A random sample of the addresses listed in the population registers could be a solution to this problem. Such a sample will allow the estimation of the size of the balance


- 249 -


between nominal members and shortages and therefore the population of each community. This additional sample to a register-based census could also provide some kind of information on variables usually not available in the register, such as the personal level of education.

A specific sample-design has therefore been developed with two main purposes:

• Achieving a good estimation of the population within each single community or – taking into account the large sampling-fractions and the high proportion of small buildings in small communities– for communities with more than 10 000 inhabitants

• Providing an accurate additional information– at least for large cities and “Kreise” (administrative districts).


The Dutch Virtual Census 2001, A New Approach by Combining Different Sources by Eric Schulte Nordholt, Statistics Netherlands

In 2003 data from many different sources have been combined to produce the Census 2001 tables for the Netherlands. The Netherlands did not interview inhabitants as in a traditional Census. The last traditional population census - based on a complete enumeration - that has been conducted in the Netherlands has reference year 1971. Since then the willingness of citizens to participate in such a census has declined further, partly because of privacy considerations. However, Census information is still of interest for policy and research purposes. Therefore, Statistics Netherlands found an alternative in the Virtual Census, based on a combination of registers and surveys that Statistics Netherlands already had available. This alternative has the advantage of considerably lower costs and less privacy concerns. In applying this method the Netherlands takes up a unique position as participant in the European Census Round. The table results can be compared with those of earlier Censuses in the Netherlands and with those of the other countries in the 2001 Census Round.

In producing tables for the 1981 and 1991 Census Round the focus in the Netherlands was on combining population register data with results form the Labour Force and Housing Surveys. Less care was given to overall consistency of the table estimates. For the 2001 Census Eurostat and other international organisations required more detailed information than for earlier Census Rounds. Moreover, in the last decade Statistics Netherlands has acquired more and more experience in dealing with data of various administrative registers for statistical use. This enabled the development of a Social Statistical Database (SSD), which contains coherent and detailed demographic and socio-economic statistical information on persons and households. The Population Register forms the backbone of the SSD. Sample surveys are still needed for


- 250 -


information that is not available from registers. Examples of variables that are not available in the Dutch registers are level of education and occupation. However, these two variables are available in the Dutch Labour Force Survey.

Overall numerical consistency among all tables in the Census tables set is required. This need stimulated methodologists at Statistics Netherlands to develop a new estimation method that ensures numerically consistent table sets if the data are obtained from different data sources. The method is called repeated weighting, and is based on the repeated application of the regression method to eliminate numerical inconsistencies among table estimates from different sources. The newly developed method of repeated weighting guarantees that combining survey and register information leads to consistent estimates in the tables of the Census Programme. The Dutch Census tables 2001 have been estimated by making use of all available information and therefore the best possible quality of the estimates has been obtained.


Strategies in Dealing with New Data Needs and their Consequences on Quality – Considerations Based on the Integration of the European Statistics on Income and Living Conditions (EU-SILC) in the German Household Budget Surveys by Iris Meyer, Federal Statistical Office Germany

When federal statistics have to supply new statistical information, it is a general question whether these should be obtained by introducing additional questions into an existing survey or whether a new survey should be conducted. The German federal statistics faced exactly this kind of question due to the regulation (ec) No. 1177/2003 of the European Parliament and of the Council concerning EU-SILC. The regulation accompanied by different Commission Regulations gives a rather strict framework for producing comparable data within the European Community. EU-SILC is supposed to be one of the data sources within the community programme to combat social exclusion. In order to reach this goal, current high quality data on the distribution of income as well as on the extent and composition of poverty and social exclusion need to be produced by the Member States. The German Federal Statistical Office had to decide either to introduce monetary and non-monetary target variables needed for EU-SILC to the existing Household Budget Survey or to realise EU-SILC as a new, separate survey.

The introduction of legally fixed and closely defined survey characteristics into an existing statistical system usually causes problems, especially when the purpose of the existing survey strongly differs from the new one. Due to different purposes, there might be differences in sampling, measurement concepts or in recording units. In this case, it should be considered to which extent measurement concepts can be changed, adopted or enlarged or whether data needs to be converted. These problems are avoided when a


- 251 -


new, separate survey is developed. For the new survey, the entire data processing has to be established which takes additional resources. Each of these two strategies on integrating new survey characteristics into an existing statistical system has consequences on survey quality. The presentation focuses on how these strategies influence selected aspects of quality like punctuality, comparability and coherence. A special focus will be put on the respondent burden, which indirectly determines survey quality.


The Revision of TANKAN Survey in March 2004 Shin Yamada, Bank of Japan

The Short-term Economic Survey of Enterprises in Japan (Tankan) is an economic survey compiled by the Bank of Japan. Tankan is a nationwide business survey conducted on a quarterly basis (in March, June, September, and December) which covers overall corporate activities to provide an accurate picture of business trends of enterprises in Japan. It consists of “judgment survey”, which covers the qualitative aspects of business confidence, and “quantitative survey” which covers quarterly and annual figures and projections.

We are planning to revise Tankan in March 2004, with the aim of reflecting the recent structural changes of the industry as well as future enhancing the statistical accuracy. The five main points of the revision are as follows:

1. The category of industries will be re-classified to reflect the recent structural changes of industries.

2. The surveyed items in the questionnaire will be revised to reflect the changes of economic background, to avoid duplication of statistics and to reduce the burden on respondents.

3. The new criterion to determine the scale of each enterprise will be based on the size of capital while the current criterion is the number of employees. This revision is to reflect the rapid diversification of employment style among enterprises such as the emergence of holding companies and internet-related companies with relatively small number of employees and large capital. In consequence, the population of Tankan will increase from approximately 160,000 to 215,000 and the number of sample enterprises will increase from approximately 8,000 to 11,000.

4. The financial institutions survey will be revised comprehensively to cope with the increasing weight of financial institutions in fixed investment. We will expand the scope of surveyed industries, the method of sample design and aggregation, and surveyed items.


- 252 -


5. “Missing-value Supplement” method to decrease non-sampling errors will be changed to improve statistical accuracy. When corresponding enterprise gives missing-values in annual projections, we have supplemented them with the average figures of the strata. After the revision, the supplementing figures will be the figures of the current or previous fiscal year.

We hope that this revision of Tankan will help to provide more accurate picture of business trends of enterprises in Japan. We will continue to make every effort to improve Tankan, including the introduction of on-line electronic reporting system to reduce the burden on respondents.


Session 37: Perception Surveys Museum Chair: Rainer Stäglin, German Statistical Society and DIW Berlin

Planning and Carrying out User/Customer Satisfaction Surveys by NSIs in EU and EEA Countries by Werner Holzer, Statistics Austria

The User/customer satisfaction survey is an important tool to detect user needs, and potentially user feedback could be integrated into the planning process of official statistics. National Statistical Institutes (NSIs) are concerned with a great number of different customers/users, all having different expectations concerning the quality of products and services. In order to get information on the expectations and satisfaction of the different users/customers, diverse types of user/customer statisfaction surveys can be carried out.

The objective of this paper is to inform about the current practice of NSIs in all EU and EEA countries with regard to user/customer satisfaction surveys, as well as activities planned.

The first chapter presents an overall view of types and periodicity of user/customer satisfaction surveys carried out or planned by NSIs and points to experiences and problems regarding conception, performance and analysis of user/customer satis-faction surveys. It also deals with aspects like the definition and classification of users/customers of statistical products and services, the identification of key users, users/customers register and organizational aspects.

The second chapter provides more detailed information on the different types of user/customer satisfaction surveys that the NSIs carry out, in terms of target groups, survey methods, number of respondents, non-response rate, quality dimensions, scope, customer background information, scales, comparability, methods of analysis,


- 253 -


action plans, required resources etc. Moreover, current good practice is discussed by presenting examples of currently used questionnaires.


Perception Surveys 2003 in the Czech Statistical Office by Petra Kuncova and Vladimír Masat, Czech Statistical Office

In 2002 the Czech Statistical Office (CZSO) accepted Total Quality Management (TQM) principles. The implementation of TQM principles was interrupted by the consequences of enormous flooding that had damaged the central office buildings in Prague seriously. Despite the fact, that since September 2002 the central office units has been scattered to ten places in Prague with natural communication difficulties, the new CZSO top management decided to continue the application of TQM principles in summer 2003. There was also a strong need to improve public perception of the CZSO after problems with external trade figures. One of the most visible, quick and effective ways was to change Internet presentation of the office. It can be done very quickly but it is necessary to do it according to user’s needs. As one of the first steps two surveys were realised in 2003 – User Survey and Staff Opinion Survey.

Questionnaire for the User Survey contained 13 questions plus identification questions. These questions concerned not only an Internet presentation but other statistical sources as well. It was to decide whether to ask open questions or to offer some possibilities to choice. Asking answers without any limitation (open question) is very difficult for data processing but gives more space for personal opinion and for the purpose of the survey it was more suitable. Finally, both approaches were combined. In some cases there was left room for describing ideas, and in other cases (when suitable) were offered some possibilities for answer (e.g. Yes, No). Questionnaires were sent by several ways – via e-mail, Internet, on paper via information services offices. The answers covered users from state administrative, students, analysts, academics, journalists, users from enterprises and others. Users taking part in the survey could use the opportunity to express their opinion on: if they can find needed information on web site of the Czech Statistical Office, if they use printed publications as main source, what statistical information they miss, data from which area of statistics they prefer etc. Least but not last they commented graphic look of www pages - if they prefer simple look to coloured one, more pictures, better graphics etc. Final results were different according to main group of users – students and analytics differed the most. Survey findings were used for CZSO website changes – for example we added international comparisons, we focused on time series.

In its second part this contribution describes the preparation, organisation, achieved results and evaluation of the Staff Opinion Survey taking into account some Czech specialities. All employees taking part in the survey (the response rate was over 60 %) could use the opportunity to express their opinion regarding: satisfaction with their


- 254 -


work, possibilities of their career growth and training, evaluation of their work, problems of equal treatment, co-ordination and working relationships, organisational structure, quality of superior managers, working climate, image of CZSO, intention to change job and reasons for it. It was very important to guarantee the anonymity of respondents, therefore the possibility to classify the achieved results in detail was limited. For the evaluation of the results the percentage of each possible answer was stated and the average mark (as the weighted average of five marks scale) for each question computed. The final assessment includes - besides total results – the differences in opinion of three age groups, two sex groups and eighteen organisational groupings. The last part of the contribution presents the measures accepted by the CZSO top management after being acquainted with the results of this survey.


Conceptualising Total Business Survey Burden by Jacqui Jones and James Rushbrooke, Office for National Statistics, U.K., Gustav Haraldsen and Trine Dale, Statistics Norway, Dan Hedlin, Statistics Sweden

The UK Office for National Statistics, Statistics Norway and Statistics Sweden are currently working together on the LEG project: Developing methods for assessing perceived response burden. The first two stages of this project have now been com-pleted. These stages involved a literature review and primary data collection, through focus groups and cognitive interviews, of respondent's perceptions of response burden for paper self-completion questionnaires. Outputs from these two stages have enabled project members to build on previous work undertaken by persons such as Bradburn (1979), Fisher (2001) & Haraldsen (2002) and conceptualise not just perceived respondent burden but total business survey burden. In doing this it is envisaged that a holistic approach to business survey burden can be identified. This approach should then have benefits for stakeholders, users, survey organisations, businesses and respondents.

This paper will provide an overview of the conceptualisation of Total Business Survey Burden (TBSB). TBSB identifies five main areas of business survey burden:

• Stakeholder/user burden

• Survey organisation burden

• Survey design burden

• Business burden

• Respondent burden


- 255 -


It looks at the interaction between these areas and burden factors that are found in each area.

References

Bradburn, N. (1979) Respondent burden Health Survey Research Methods: Second Biennial Conference, Williamsburg Virginia, Reeder, L. ed. U.S. Government Printing Office, Washington D.C.

Fisher, S. & Kydonifs, L. (2001) Using a Theoretical Model of Response Burden (RB) to Identify Sources of Burden in Surveys Paper presented at the 12th International Workshop on Household Survey Nonresponse, Oslo

Ford, B., Hocking, R. & Coleman, A. (1978) Reducing Respondent Burden on an Agriculture Survey, Proc. Survey Res. Methods Sect., American Statistical Association, Washington D.C. pp. 341 - 345

Haraldsen, G. (2002) Identifying and Reducing the Response Burden in Internet Business Surveys Paper presented at the International Conference on Questionnaire Development Evaluation and Testing Methods (QDET), Charleston , South Carolina


Response Burden Barometer of Statistics Austria by Norbert Rainer, Statistics Austria

In the last few years Statistics Austria developed a system of response burden measurement covering all compulsory surveys on businesses (“Response burden barometer”). This indicator system shows the total response burden for each of the survey over time and by the factors influencing a change of the response burden. The response burden is measured as the time necessary for filling in the respective questionnaires, whether on paper or electronically. Data are currently available for the years 2001 – 2003.

It is one of the requirements of the Federal Statistics law 2000 to keep the response burden as low as possible. The law stipulates for example that sample surveys should be carried out instead of full coverage surveys, that administrative data should be used instead of asking the respondents, that means of electronic data deliveries should be provided free of charge, that questionnaires should be tailored to the specific economic branches or institutions, etc. Therefore, the development of the response burden is of key interest to the management and as well as to the political decision makers, the respondents and their representatives.

The paper consists of three main parts: the first part explains the background of the development of the response burden barometer. The second part describes the methodology applied and the third part will focus on the challenges and limits of such


- 256 -


an indicator. Topics are the relation between the administrative and statistical burden, changes in the institutional background that transforms administrative burden to statistical burden, the discrepancy between actual and perceived burden, and similar aspects.


OECD Statistician Satisfaction Survey by Ann Nichols and Erica Matthews, OECD

In early 2001 the OECD launched its Statistics Strategy within which several initiatives have been undertaken to raise the profile of statistics vis-à-vis OECD stakeholders and top managers, to improve the quality of OECD statistics and the efficiency of internal statistical processes. A key objective of the Strategy is to “improve the skills and the quality of work of those involved in statistical activities” and several initiatives have been undertaken to achieve this target, from both technical and organisational perspectives.

The “human factor” is absolutely crucial when implementing broad organizational and technical reform and the launch of a corporate strategy for OECD statistics requires a great deal of effort to increase the participation in various elements of the Strategy of all statisticians working in different Directorates across the Organisation. A Statistical Policy Group (the internal body where all Directorates are represented) was relaunched as the centre of the consultation process across the Organisation. In addition, new instruments to improve internal co-ordination and communication have been developed and focused training initiatives have been organized for OECD statisticians.

December 2003 marked the end of the first phase of the OECD Statistics Strategy with the implementation of the new statistical information system. The second phase will commence with the implementation of the “OECD Quality Framework” for statistics. It was therefore considered appropriate to review the current working conditions of OECD statisticians and to evaluate the degree to which they have been affected by the Strategy over the last two years.

The aim of the OECD Statistician Satisfaction Survey is to provide subjective measures on the extent to which OECD statisticians and analysts dealing with statistical products are satisfied with their current working conditions and to identify areas that require more attention and investment. The results of the survey will be summarised in a report and the main findings referred back to managers in charge of statistical activities. The results will also be used to identify new priorities and refine existing activities in future elaborations of the OECD Statistics Strategy

The paper will present a brief overview of the OECD Statistics Strategy, together with an equally brief overview of similar staff surveys undertaken in national and other international organizations. It will touch on a management survey conducted by the OECD Human Resource Management Directorate in 2001. The focus of the paper will be


- 257 -


the human resource aspects of implementing change in the context of an organization where quality statistics are a high priority. It will provide an outline of the OECD Statistician Satisfaction Survey, present key results and recommendations resulting from the survey and their implications for the future evolution of the OECD Statistics Strategy. Contact: [email protected]

The User Satisfaction Survey as an Integral Part of the Marketing Information System of the Basque Statistics Office (EUSTAT). by Jose Jabier Zurikarai, Basque Statistical Office, Spain

In the Basque Statistics Office (EUSTAT), the Marketing System consists of three main elements:

- Market Information sub-system

- Product and service design sub-system

- Commercialisation sub-system

The Market Information system is formed by the group of people, activities and processes that ensure the organised flow of information needed to know how Eustat products and services are working in the marketplace, as well as the evolution of the market, its competitors, their characteristics and dimensions.

Its function is market research, in order to provide more and better information to answer the questions: What are the market’s needs and demands? Who is making them? What is currently being done to satisfy them? Who are the competitors and what are their products and services?

Actions was previously taken in the information system to improve existing information regarding the demand for products and services. Regular analyses were made of statistics showing visits to the website, requests for information, as well as processing interviews carried out with referential clients.

However, all this was found to be insufficient, given that the information available through the above sources gives no indication as to what the users feel about our activity. Furthermore, they were appropriate for more partial purposes.

Through the Satisfaction Survey, the user becomes a truly active informant, valid to measure the various parameters related to our work. The Satisfaction Survey is a tool that will allow us to better our knowledge of our clients’ needs and priorities, their opinions about our own performance, thus completing our internal analyses and evaluations.


- 258 -


In this survey, the first aim is to obtain a general evaluation of the Basque Statistics Office. The questionnaire has been designed to provide that general evaluation. But it's also valid to check which are those fields of our activity where we're doing it well, and where we should improve.

The methodology used will be presented in the Eustat Satisfaction Survey. Self-weighted items that refer to crucial aspects related to our products and services. Those who complete the survey not only give a score according to their satisfaction with the items on offer; they also evaluate the importance that they give to each item. The scores thus give us the self-weighting. All this is done with the intention of handing the user the task of defining what they really believe to be important.


Session 38: Recommended Practices Backstage Chair: Werner Grünewald, Eurostat

The Road to Quality - The Implementation of the Recommendations of the LEG on Quality in the ESS by Martin Karlberg, Eurostat and Laurent Probst

1. Background

In 1999, Statistics Sweden submitted a proposal to the Statistical Programme Committee (SPC) regarding the formation of a Leadership Group (LEG) on Quality in Statistics, with the general aim to deal with a wide range of quality topics and to give a series of recommendations concerning e.g.

• current best practices

• minimum survey standards

• systematic quality improvements, at National Statistical Institutes (NSIs) as well as in the European Statistical System (ESS) as a whole.

The proposal was unanimously supported by the SPC, and consequently, the LEG on Quality group was formed. The group delivered its final report in 2001; this report included

• a proposal for a Quality Declaration of the ESS

• 21 recommendations to the SPC

• a 22nd recommendation to form an Implementation Group with the task of coordinating the implementation of the 21 LEG recommendations.

The SPC unanimously


- 259 -


• adopted the quality declaration

• accepted the 21 recommendations of the report

• accepted the proposal of setting up an Implementation Group.

The LEG on Quality Implementation Group has now been operating since 2002. It is co-ordinated by Eurostat, with representation from 10 ESS NSIs (Austria, Finland, France, Germany, Italy, the Netherlands, Norway, Portugal, Sweden and the United Kingdom) and the Berlin State Statistical Office. Its tasks are to

• closely follow the implementation of all recommendations

• co-ordinate joint development activities.

2. Current situation

Currently, two years after the recommendations were issued, the LEG recommenda-tions have been disseminated, annual surveys on implementation have been dis-tributed to ESS NSIs and five multinational development projects have been initi-ated on

• analysis and measurement of key statistical process variables

• audit methods in statistics

• customer satisfaction surveys

• measurement of perceived response burden

• a self-assessment programme for surveys.

Around half of the 21 recommendations have now been implemented, or will be implemented fairly soon. To summarise:

• 5 recommendations have been met

• 4(5) recommendations are likely to be met by the end of 2004(2008)

• 9 recommendations will require more action if they are to be met

• 2 recommendations have been down-prioritised.

It is important to note that, while many of the LEG recommendations have been met by the vast majority of EES NSIs, there has been no systematic inventory of the compliance with the recommendations in the 10 countries acceding to the EU in May 2004. Recommendations that currently seem more or less obsolete could therefore get renewed actuality after May 2004.

3. Future work

The LEG implementation group has now entered its final year. Upcoming activities will include


- 260 -


• monitoring of the implementation work, covering recommendations likely to be met and the state of affairs in the 10 acceding countries

• implementation and dissemination of the results achieved by the four devel-opment projects already concluded

• initiation of a few more development projects regarding some of the recom-mendations currently not met (on CBM development and best practices).


LEG Quality Recommendations and their Implementation in Statistics Lithuania and Statistics Finland by Daiva Norkeviciene, Department of Statistics to the Government of the Republic of Lithuania and Johanna Laiho, Statistics Finland

The Leadership Group (LEG) recommendations were given by Eurostat’s LEG Working Group to provide the first joint approach and framework towards quality improvement within NSIs and in the ESS. The work of the LEG follows the Eurostat quality dimensions, which enables the NSIs to establish a link between data quality and TQM in development work. The recommendations demonstrate how the improvements could be made and can be seen as mile stones for the NSIs in the long run.

We consider this work very valuable, as well as having joint quality targets for the ESS to become a more coherent system. However, each NSI must work towards the aims within their national operational frameworks, which may sometimes differ significantly.

In this paper we present and put side by side implementations of LEG recommenda-tions in two NSIs, Statistics Finland and Statistics Lithuania. The histories of these organisations are rather different from each other. In addition, Lithuania is among the new member countries of the European Union. On the other hand, both NSIs are the co-ordinate organisations of official statistics in relatively small European countries and their quality improvement development work is following the EFQM excellence model.

We will first give a short overview of quality management in both organisations and then compare the TQM approach adopted by these organisations. We will subsequently aim to contrast the approaches adopted in both organisations in relation to the LEG recommendations. Our goal is to find out whether differences exist in the operational framework or organisational culture, and whether this has an impact on the quality path that the NSI has to follow for further improvements. We are also interested in comparing the orientation of quality management in both organisations and seeing whether differences in priorities of the LEG recommendations are significant. We will also evaluate the experiences and needs foreseen and guided by the LEG recommendations.



- 261 -


Recommendations for Revising Standards for Statistical Surveys for U. S. Federal Agencies by Al Tupek, Jennifer Madans, William Arends, Jay Casselberry, Kevin Cecco, Steve Cohen, Pat Flanagan, Brian Harris-Kojetin, Karen O’Connor and Marilyn McMillen Seastrom, National Center for Health Statistics, U.S.A

The United States statistical system is highly decentralized, consisting of more than seventy agencies and units that engage in some statistical activities with ten agencies that have statistical activities as their principal mission. While this structure provides substantial benefits in ensuring responsiveness to program needs, it makes it more difficult to ensure that agencies are uniformly applying sound statistical techniques and best practices. The U.S. Office of Management and Budget (OMB) is charged with developing and overseeing the implementation of government-wide policies, standards, and guidelines for Federal statistical programs.

The current standards for surveys were issued by OMB more than 20 years ago. As part of ongoing efforts to update and revise a number of statistical standards, OMB formed an interagency committee consisting of members from eight agencies that have statistical activities as their principal mission under the auspices of the Federal Committee on Statistical Methodology (FCSM), to make recommendations for updating and revising standards for statistical surveys. The agencies represented on the committee included those that produce a wide variety of economic and demographic statistics including economic activity, employment, health, education, agriculture, and transportation. Some of the agencies represented on the committee had their own detailed written standards for their agency’s statistical surveys, while others did not have formal documentation or standards that applied across programs within the agency.

The existing agency written standards formed the initial basis for the committee to recommend standards for all statistical surveys. A common framework for the stan-dards was first chosen based topic areas that had been endorsed by all of the principal statistical agencies as part of the Information Quality Guidelines all Federal agencies were required to issue in 2002. The framework covered:

1. Development of Concepts and Methods

2. Planning and Design

3. Collection of Data

4. Processing and Editing of Data

5. Production of Estimates and Projections

6. Data Analysis

7. Review Procedures

8. Data Dissemination


- 262 -


Separate agency standards dealing with each topic area were reviewed and discussed.

In this paper we will discuss the recommendations for standards the interagency committee provided to OMB as well as the process of developing standards taking into account diverse perspectives from a number of agencies that conduct and sponsor very different data collections. The subcommittee worked diligently to create a common set of standards that will assure the quality of surveys conducted and sponsored by U.S. Federal agencies while recognizing and accommodating some differences in practices among different organizations. The use of interagency teams to reach agreement in the highly decentralized structure of the U.S. Federal statistical system may also provide some analogies to international efforts to agree upon standards among national statistical institutes.


Formalizing the Quality Program at the U.S. Census Bureau by Robert Tinari, Cheryl Landman and Deborah Stempowski, U.S. Census Bureau

The Census Bureau has always been committed to quality. But in order to fully commit itself, it is clear that a formal quality program was necessary. Not only does Bureau management have a vested interest in creating quality programs and products, but the Office of Management and Budget and the Department of Commerce are also demonstrating that they are becoming increasingly interested in tracking the quality of government-wide processes and products.

The quality management program is vital and necessary for many reasons. A successful program, at the very least, can:

• Reduce re-work and cycle time,

• Give program managers the quality assurance/quality control resources they need to build in quality up front as well as implement meaningful quality enhancements and improvements,

• Educate staff of the importance of quality in their day-to-day work and assist them in doing so.

Over the past few years, the Methodology and Standards Directorate at the U.S. Census Bureau has been working to design such a program. This paper documents the work that was done and the work that continues to be done to create a formal quality program. The formal program has transitioned to a planning, research and evaluation division within the U.S. Census Bureau. For the most part, the quality program will work closely with the programs to utilize existing resources at the Bureau instead of creating all new positions. The Quality Program will continue and enhance functions that have been initiated in the development period. The staff functions include:


- 263 -


• Develop, disseminate, train, and review Quality Framework documents (principals, standards, guidelines). Documents will be developed by cross-directorate working groups and disseminated on the Quality Management Repository. Staff would have responsibility for overseeing the first two efforts, training in the application of standards, and periodic review of the documents for currency.

• Assist divisions in building quality into their processes and programs working on problems identified by the divisions,

• Develop training on quality procedures for survey operations (e.g. Quality module in Project Management),

• Coordinate different efforts related to quality management including Census Bureau response to OMB Information Quality Guidelines,

• Conduct research in applications of statistics to measuring quality and per-formance.

This paper provides an overview of how the U.S. Census Bureau is implementing its quality management program, highlights the steps of the quality program transition, and shares lessons learned through this process.


Guidelines for Data Release: General Rules for the Sample Surveys in the National Statistical Institute of Portugal (INE) by Luís Correia, Instituto Nacional de Estatística, Portugal

Official statistics play an important role in our society giving to official institutions, economic sectors and public in general, information about economic, demographic and social indicators. Hence, all the organizations dealing with the production of official statistics should be aware of such importance and should guarantee a high level of data quality. Nowadays, quality in statistics represents one of the major concerns in the National Statistical Institutes. According to Eurostat, quality can be divided into six components: relevance; accuracy; timeliness and punctuality; accessibility and clarity; comparability; and coherence. The purpose of this article is related with accuracy. It aims to establish general rules to redefine the guidelines for data release based on the reliability of the estimates.



- 264 -


Introduction of a Unified Seasonal Adjustment Procedure Within the Hungarian Central Statistical Office by Erika Földesi and Péter Bauer, Hungarian Central Statistical Office

A unified seasonal adjustment procedure was introduced in the Hungarian Central Statistical Office (HCSO) in the first quarter of 2002. We developed it as a recom-mended practice for all departments dealing with seasonal adjustment within the HCSO. This procedure applies the TRAMO-SEATS method and the Demetra software. The idea behind the procedure is that the best results can be achieved if only experts of time series theory and experts of actual time series co-operate during the seasonal adjustment process.

In the presentation we present the former situation and the considerations taken into account for the choice of the method. We give an overview of the adjustment process: how the data producer departments and the methodology section co-operate, what kind of practical information needed for an apropriate adjustment. Some experiences we have gained and problems we have encountered since the introduction are discussed (aggregational problem, outlier at the end of the series, uncertainty of the end of the trend, working-day and easter effect, revision).


Session 40: Associated Session on Data Warehouses

Datawarehousing Population Census Data: Reporting on Data During Processing Phase and Validation of Macrodata by Angela Chieppa and Antonio Laureti Palma, ISTAT, Italy

At the inside of the Informative System to manage data of the latest Italian Population Census, there is a dedicated sub-system for people responsible for the validation of information to disseminate. This system has three main requirements:

1. to guarantee reporting on data during the processing period, together with metadata of the processing steps – Controlling the quality of the output data determines the need of many information: first of all, controllers need reports with the results of the checks they’ve designed; they also need crossing data in different ways to find errors not foreseen ; finally, they need to control data about the execution of every automated procedure (execution time, imputations made...), to avoid introducing distortion and to get information for “continuously” improve the processing phase. In other words, there is a demand for “datawarehousing” data of the processing system, to get all the reports needed. These reporting


- 265 -


activity support the decision to to validate a processing step and to turn to the next one.

2. to produce macrodata among which the organization may choose the ones for dissemination – Though a dissemination plan is ready before the processing phase start, analizing data could determine many changes to it. Confidentiality issues, for example, have to be managed; but also analizers may found some interesting new evidence coming out from the data enumerated. That is to say that many more macrodata for dissemination have to been produced than the ones descending from the first plan.

3. to manage the validation of these macrodata, supplying the different systems for dissemination (web system, generator of printable layouts) - The validated macrodata are the only macrodata allowed to “feed” the system designed for disseminate information.

The system was realized through a datawarehouse with a three-level scheme: source / reconciliation/ dissemination (reporting).

At the source level of this DW there are:

• Census microdata at the end every step of imputation or any other kind of modification (coding, linkages...)

• information about automated procedures

• data coming from other systems: other surveys of ISTAT, administrative sources, past Censuses; all these data are useful for comparisons.

The reconciliation level is crucial in this system, since, unlike other datawarehouses, in this case there have to be managed data not completely “cleaned”: missing or out of range values are not only possible but very frequent at different steps. So that the reconciliation consists in:

• managing appropriately these’not cleaned’ cases

• calculate dimensions needed to produce reports

• integrating the different sources, building “common” dimensions to allow joining data

These “reconciliated” data could finally be used to build different data mart that “fasten” the reporting activity and the analysis. At this level, the users studying the reports, can validate the macrodata they want to disseminate. This operation of “validation” is stored as an attribute in the “macrodata catalogue”, which is very useful in determining if data could supply another system or not.



- 266 -


Information Quality of Dissemination Data Warehouses: The Experi-ence of the National Statistical Institute of Italy (ISTAT) by Ennio Fortunato, Virginia Lacquaniti and Nadia Mignolli, ISTAT, Italy

Among data warehouses available on the web site of the National Statistical Institute of Italy (Istat), DemoS represents a useful example of a multidimensional collection of the most important indicators to analyze the Country from a social, demographic and economic point of view.

DemoS is a territorial data warehouse, it gathers information after a deep quality control, a real monitoring phase and following the whole research activity carried out by Istat.

The about 2000 indicators collected in DemoS have been either defined from the study of specific literature (national and international) or from the analysis of those that are now more widespread. In addition to these, some others are compiled autonomously through studies and analysis.

As Italy has got so many territorial differences, Demos was designed and implemented for the dissemination of statistical data following a three-coordinate-based logic: Time, Space, Events.

This triplet defines content and structure of dynamic output tables. As the above mentioned parameters vary, it allows a content analysis of basic indicators according to the selections related to the time-space plan. Therefore two report types are produced:

• The first report type shows the trend of a specific indicator for a number of years on a set territory to be chosen according to one item exclusive selection at a certain territorial level within: total Italy, partial Italy, regions (20), provinces (103), main municipalities (103). This report type shows the territory on the side and the years on the top. The indicator identifies the table: it is possible to have more reports of the same type in the answering page of the system.

• The second report type allows the comparative analysis of several indicators given a reference year; as by the previous report type, it is possible to select the territory. This report type shows the territory on the side, the indicators on top and the year identifies the table: it is only possible to have one report of this type in the answering page of the system.

Reports can be accessed through a tree-shaped analysis: this starts from the selection of a specific field of interest; it then goes on to a page where it is possible to select one or more indicators for the chosen area, the interest years and the territorial level preferred for the analysis.

The analysis is carried out in an exclusive way by choosing a set territorial item for the chosen level.


- 267 -


It is important to underline a helpful innovation of DemoS, represented by a user-friendly metadata system specific for each indicator, that offers a complete docu-mentation concerning all variables used and deep information on their sources.

The system also allows users to see maps: for each indicator it can produce a map of Italy in which provinces are shown with different shading to highlight the different intensity of the event.

Through this approach, this data warehouse provides the users with access to con-sistent and harmonized statistical information in order to use data in a skilled way and to perform data analysis.

Contact [email protected], [email protected], [email protected]

A Data Warehouse for Physical Flows by Erik Löfving and Anders Grimvall, Linköping University, Sweden

Data on flows of materials and substances through the economy and the environment can greatly improve the understanding of environmental problems. At present, such data are collected by a large number of organizations and are stored in different systems and formats. To facilitate exchange of this information and to enable joint analyses of data from different sources, it is important that we harmonize the systems that are used and also define a conceptual data model that may persist for a long time.

The proposed structure of a data warehouse for physical flow data includes a relational database in which flow data are stored along with information about the underlying node system and relationships between nodes in different systems. Moreover, links to classifications used in official statistics are incorporated into the database. Inclusion of information about the underlying systems makes it possible to assess the comparability of different studies and to extract new combined studies by using aggregation and disaggregation operations.

Data cube representations are tied to the database to facilitate statistical analysis of collected data. We also show how currently used matrix operations on ordinary two-dimensional flow matrices can be generalized to operations on data cubes of arbitrary dimension. These operations, together with the database thinking introduced in this paper, can help clarify relationships between different tools or approaches in industrial ecology.



- 268 -


A Statistical Web Warehouse System by Giuseppe Sindoni and Leonardo Tininini

6, ISTAT, Italy

A Web warehouse is a data warehouse deployed on the Web, thus exploiting both the efficiency and flexibility of data warehouses, and the accessibility and usability of the Web.

When building a statistical Web warehouse: • micro data can be organized in fact tables;

• macro data can be organized in data cubes;

• dimensions and hierarchies correspond to category attributes and classification hierarchies;

• measures correspond to summary attributes;

• roll-up operation corresponds to summarization, i.e. shifting from a more detailed to a less detailed aggregation level;

• drill-down operation corresponds to shifting from a less detailed to a more detailed aggregation level.

This almost straightforward mapping between data warehouse and statistical concepts might imply that data warehouses completely solve all the problems which statisticians have in disseminating their information. Unfortunately this is not the case, as some peculiarities of statistical data with respect to operational data make it necessary to extend data warehouse techniques with more specific models and structures.

The first peculiarity regards sample surveys. Sample aggregates, i.e. aggregates obtained from samples, must be significant: a collection of sample aggregates is significant only if, for each group, the number of sample units is above a given threshold.

The second regards privacy and secondary disclosure. Most statistical data for publication are subject to national and international laws protecting the privacy of citizens. With respect to aggregates, secondary disclosure control implies that the organisation responsible for the published tables must be reasonably sure that disclosure rules are not violated.

Finally, sparse tables should be avoided. These are statistical tables containing a high percentage of null data, due to the use of meaningless combinations of classifications.

These requirements imply that commercial data warehouse systems can not be directly used without further customisation, because they allow applications to navigate arbitrarily all the dimensions available for a fact, without providing functions to check:

6 The author was partly supported by CNR-IASI, viale Manzoni 30, 00185 ROMA (ITALY)


- 269 -


i) sample aggregate quality, ii) if the resulting table violates disclosure rules, iii) if a given combination of classifications is meaningful.

The paper describes the design principle and innovative model of a statistical Web warehouse system which has been developed at Istat (the Italian National Institute of Statistics) to meet the above requirements.

The system is based on a representation model for statistical tables, tailored for Web-based data access and navigation. The model is based on the decomposition of the information space in simple statistical tables, represented by four components:

• time (When): the data’s reference time-stamp.

• territorial context (Where): the data’s reference territory.

• object (What): the measure (summary attribute) of the statistical table.

• classification (How): the dimensions used to classify the table measure.

A very relevant issue which had to be faced in designing the system was to keep track of the navigation state, as navigation is based on a sequence of selection steps, each depending on those coming previously. Each selection step is an incremental and interdependent definition of each of the four table components. A query to the system consists of the specification of a quadruple <t, s, o, c>, for example <2001, {“provinces”, “Italian islands”}, “Resident population”, {“ sex”, “marital status”}> (c is a (possibly empty) combination of classifications, and s is the combination of territorial detail d and territorial area a).

The same table representation is also used when storing spatio-temporal availability information in the system’s meta-database. This storage of a <t, d, o, c> quadruple means that the simple table defined by the <o, c> couple is available for the year t down to territorial detail level d.
