Survey mode effects on income inequality

measurement1

Pirmin Fessler2, Maximilian Kasy3, and Peter Lindner4

November 20, 2012

Abstract

Exploiting a quasi-experiment we use non-parametric re-weighting and regres-

sion approaches to estimate the causal effect of the interview method on item-

nonresponse and the unconditional observed income distribution. The minor

change of interviewing households via the telephone instead of personally leads

to major changes in item non-response, which increases by roughly 20% − 30%.

The observed distribution of income is compressed translating to a decrease in the

Gini coefficient by about 10%. Commonly used rankings of countries by Gini co-

efficients of the income distribution might largely be an artifact of different survey

techniques as the interview mode than true differences in income inequality.

JEL Classification: B41, C01, C14, C64, C81, C83, D31

Key Words: EU - Statistics on Income and Living Conditions, Computer Assisted Per-

sonal Interviewing, Computer Assisted Telephone Interviewing, Survey Methodology,

Unconditional Distribution, Non-Parametric Re-Weighting

1

1 Introduction

Intertemporal and international comparisons of economic inequality are typically based

on inequality measures calculated from survey data; see for instance a recent report

by OECD (2011). The surveys most widely used for such calculations include the

Survey of Consumer Finances (SCF), the Panel Study of Income Dynamics (PSID),

and the EU - Statistics on Income and Living Conditions (EU-SILC). Across different

surveys and across different waves of the same survey different interview modes are

used. In particular, some of these surveys are conducted in person (computer assisted

personal interview, CAPI), while others are conducted via telephone (computer assisted

telephone interview, CATI). For a comparison of the different EU-SILC surveys see

Table 1.

Existing evidence (e.g. de Leeuw, 1992; Lohmann, 2011) suggests that in such sur-

veys nonresponse and misreporting of income data is a concern. This is of particular

importance in the upper and lower tails of the distribution, and both of these prob-

lems might be even more severe in telephone interviews. Nonresponse and misreporting

are important problems in the context of inequality measurement because many com-

mon inequality measures, such as the Gini coefficient, are not robust5 (c.f. Cowell and

Victoria-Feser, 1996), which implies that minor data contaminations can have a large

impact on measured inequality. The possibly large influence of minor mismeasurement

stands in contrast to comparably small differences in inequality measures across coun-

tries and time, see Table 1, which suggests that comparisons across different surveys

might be quite problematic.

In this paper, we use quasi-experimental variation of interview methodology in the

Austrian EU-SILC 2008 survey to provide causal estimates of the effect of the interview

mode (CAPI vs. CATI) on the estimated income distribution measured in several ways

as e.g. the Gini coefficient and the 90/10 percentile ratio. Our estimates exploit the

2

panel structure of the EU-SILC survey conducted in Austria in 2007 and 2008 and

control for a rich set of covariates from the baseline survey. We find that a switch

from CAPI to CATI leads to major changes in response behaviour, which imply large

differences of estimated inequality measures. Selective item-nonresponse in the tails is

significantly higher for CATI (on average by about 20%-30%), and incomes for CATI

tend to be closer to the mean income. A switch from CAPI to CATI in particular de-

creases the Gini coefficient of household income by roughly 10%, and has a statistically

insignificant effect on the 90/10 percentile ratio (see Table 2). The smaller effect on

the latter statistic is likely due to the fact that it is robust, whereas the Gini coefficient

is not. A “Back of the Envelope” calculation in the last column of Table 1 illustrates

these possible effects and leads to severe re-ranking of several countries. These calcu-

lations suggest that countries with considerable proportions of CATI interviews would

typically rank higher in rankings of income inequality (e.g. 7 positions higher in the

case of Denmark and 5 positions higher in the case of Finland).

We draw the following general conclusions from these findings. First, it seems that

CAPI yields more reliable measures of inequality, and should be used where possible.

Second, when making international or inter-temporal comparisons of inequality, we

should make sure to “compare apples with apples”. Comparisons of inequality measures

based on surveys using different methodologies might be quite misleading. Third, given

the issues with survey data in general and surveys using CATI in particular, it might

be advisable to focus on robust inequality measures such as quantile contrasts when

conducting inequality comparisons. We view our estimates as the lower bound of the

effect as coverage effects of the interview mode cannot be covered.

In our analysis of the effects of modes on the distribution of measured incomes we are

using reweighting and regression methods developed in the literature on distributional

decompositions, in particular by DiNardo et al. (1996) and Firpo et al. (2009). Our

3

paper deviates from this literature in studying effects on measured, rather than actual,

incomes. Survey methodology is not part of the standard economics curriculum, but

economists become increasingly aware of the importance of careful data collection.

This is partly due to the rise of experimental methods, and the consequent collection of

primary data by development economists (see Duflo et al., 2008) and labor economists

(see List and Rasul, 2011), but also true for economists using “subjective” data from

large pre-existing surveys, see for instance Gabriella and Pudney (2011). This paper is

intended to contribute to the rising awareness of data issues among economists.

The rest of this paper is structured as follows. In the next section we introduce the

hypotheses in the background of the analysis and some more literature for it. It helps to

clarify the mechanism that are at play in this specific part of the data production process

and embeds the analysis in a wider context. The underlying data as well as the quasi-

experimental set-up are laid out in the following part of the paper. It gives a detailed

account on the switch of the interviewing mode and the measures that are used. Section

4 is dedicated to giving a short overview of the identification strategy. Additionally,

we introduce the specific implementation of the estimation strategy. The main part of

the paper, section 5, discusses the results reached from the investigation. We split it

into the discussion of the effect of the interviewing mode on item-nonresponse, reported

income level, and the distribution of income. Finally, section 6 sums up the results,

draws some conclusions and concludes the paper.

2 Interview Modes

Evidence on the impact of the interview mode is mixed. In a meta-study de Leeuw

(1992) compared face-to-face interviews, telephone surveys, and self-administered mail

questionnaires. By employing 67 mode-comparison-papers she showed that the main

4

difference lies between self-administrated and interviewer-based survey modes, the dif-

ferences between interviewer-based modes, and therefore also between CATI and CAPI

seem less pronounced. While there was no significant difference in response validity and

social desirability bias, Face-to-Face interviews lead to slightly less item non-response.

CATI is the oldest of the computer assisted interviewing methods and is heavily used

especially in market research. CAPI is widely used especially for complex surveys of

governmental statistic agencies and universities (de Leeuw (2008)).

”Because of the increased availability of other survey modes, face-to-face

interviews are typically reserved for the most difficult and longest surveys

that place the greatest burden on respondents. These are all kinds of surveys

for which the other modes are not so likely to perform well. Face-to-face

surveys also tend to be reserved for surveys that are most important to

society, for which sponsors are willing to pay the cost.” (de Leeuw et al.

(2008), p. 164)

The main advantage of face-to-face interviews is that they are more flexible than

telephone interviews. The interviewer can use response cards, visual scales etc. but

also explain things better by being physically present which allows for a broader range

of communication and interaction between the interviewer and the respondent. Via

telephone the respondent can only rely on his/her memory when answering questions

with multiple answer possibilities.

Also interview length is an important determinant of the quality of the interviews given

a certain mode. While CAPI and face to face in general is usually used for longer inter-

views (>30min) telephone interviewing is less suitable for longer interviews. Major data

collection organizations in the US are refusing to conduct telephone interviews which

are expected to last longer than 18 minutes (de Leeuw et al. (2008)).

5

We hypothesize that there are three different mechanisms at work how the mode

influences measured income inequality.

First, on the level of unit non-response. We find that even though higher income house-

holds tend to be selected more often towards a CATI interview the highest values are

found for households interviewed via CAPI. The same is also true for the lower end of

the distribution. Therefore it seems that via CATI the households at the tails of the

income distribution can not be reached as good as via CAPI (see Table 4).

Second, on the level of item nonresponse. In addition to the effect of leading to gen-

erally higher item nonresponse the effect of CATI is especially pronounced at the tails

of the distribution. Especially at the top this is very worrisome as the share of total

income held by the top income earners is a multiple of their population share.

Third, on the level of the income values. We find that CATI leads to a positive effect

on income values reported which is relatively higher for lower income groups than for

very high income groups and leads to significantly lower measures of income inequality.

This might be due to two mechanisms. On the one hand, the questionnaires in these

surveys are rather complicated and so especially financially less literate (low end of the

income distribution) households as well as households with very complicated income

structures (high end of the income distribution) might be more more likely to inhibit

measurement error over the telephone than in a personal interview. On the other hand,

it might be simply easier to lie over the phone than in a personal conversation.

We can only quantify the third effect in terms of measures of income inequality

and the second in terms of a causal estimate of the interviewing mode on the item

non-response. However, we also find strong evidence that the first effect exist and is

non-negligible. Therefore it is very likely that we underestimate the total effect of the

interview mode on observed inequality measures as both the first and second effect also

6

seem to compress the observed income distribution. In general people with very low

income as well as people with very high income tend to report values biased towards

the mean. Overall three effects this measurement error seems to be larger when CATI

instead of CAPI is used as an interview mode. The decision to not report at all (unit

nonresponse), selectively not report (item non-response), or report values closer to the

mean might be easier via the phone than in a face to face situation.

3 Quasi-Experiment and Data

For this empirical analysis we use Austrian EU-SILC data6 of the waves 2007 and 2008.

The documentation on the data is provided by Statistik Austria.7 Additionally to the

data available in the standard user dataset we obtained directly from Statistik Austria

an indicator for both years about whether a specific household was interviewed using

the CAPI or the CATI interviewing mode. We use both waves since the CATI option

was first (after a test period for 2007 with households that are excluded as is described

below) introduced in 2008. Only households that already were interviewed in 2007

could choose between a personal or telephone interview. The first choice of the data

producers was to use the cheaper CATI option and first contact was - wherever possible

- established via phone. We model the selection (see Section 4.1) in order to control

for it and identify the remaining effect due to the interview mode.8 Table 3 reports

how many households can be used for the identification of the effect of the interview

mode. There are 5,711 households interviewed in 2008 of which about 30% (i.e. 1,710

households) are surveyed via a CATI interview. Only two thirds are panel households

in the sense that they were also interviewed in the previous wave. Thus the effective

sample size is reduced to 3,377 households which could be (self-) selected to one of both

interview modes. 42.6% of these actually were interviewed via telephone while the rest

7

opted for a personal survey.9

In EU-SILC 2008 selection towards CAPI is characterized by the following rules.10

Households that re-participate, i.e. already took part in a previous wave, in the 2008

EU-SILC-Survey were generally intended to answer the questionnaire via the telephone.

This tendency was implemented through a first contact (after the sending of an infor-

mation letter) by calling the respondent and asking the household to participate via

telephone. The change to the CAPI-mode was available for households that indicated

that they refuse to take part in the survey over the phone, could not respond on the

phone due to illness or age related reasons, or could either not be reached over the phone

(i.e. wrong number, not available on the phone, etc.) or set dates were postponed by

the respondent for several times.11 All CATI-interviews were conducted during the

weekdays (Monday to Friday) from 4pm to 8pm and additionally on Tuesdays between

9am and 1pm.12 In general there seems to be a tendency to push households to con-

duct interviews on the phone while no unit-non-response is allowed due to the wish of

a respondent to conduct a CAPI-interview.

To sum up the quasi-experiment at hand: All households were interviewed in 2007

via CAPI. All of them were targeted to be reinterviewed via CATI in 2008. Due to

accessibility problems via phone and the possibility to opt for the CAPI mode roughly

40% of them were interviewed via CATI and 60% via CAPI. We are able to control

for the non-random part in this treatment assignment with the information which was

gathered in 2007 for all households with the same (CAPI) mode, including income and

item nonresponse information, to estimate the effects of the treatment - the interview

mode - on the observed income distribution in 2008.

Data from the 2007 wave (instead of the 2008 wave) is used because it is pre-

determined and hence by construction not exposed to a mode effect. Thus the controls

8

are no outcome of the selected mode and can be considered strictly exogenous to the

mode. Descriptive statistics of the controls by sub-samples are given in Table 4.

3.1 Measures

The main focus is laid on disposable income. Household disposable income is con-

structed by summing up all of the households income sources. EU-SILC provides flag

variables to distinguish three categories with regard to necessary imputation of this

variable, (i) not imputed, (ii) partly imputed, (iii) completely imputed. Whereas “not

imputed” implies that all items the aggregate household level variable consists of, were

provided by the respondents, “partly imputed” means that one or more items where

missing and had to be imputed and “completely imputed” means that all items where

missing and had to be imputed.

Using this information we construct a household income item non-response dummy

(HINR) being 1 if the household disposable income flag is indicating that household

disposable income is “partly imputed” or “completely imputed” and 0 otherwise. The

dummy therefore indicates if there was missing information with respect to the main

household income variable.13

Disposable income14 is taken as it is provided in the data. Table 415 reports in the

upper part the mean of the variables in the panel households and the CAPI and CATI

sub-samples. Already here becomes clear that the average income is lower for house-

holds surveyed in a CAPI interview and the item non-response is 20%-30% lower for

these households. Furthermore, we can see that most of the income-poor households,

i.e. 75% of the households below an income of 10.000e , and all of the income rich

households (above 200.000e ) are interviewed via CAPI. This fact already indicates the

9

problems originating from the telephone interview mode in terms of a possible com-

pression of the income distribution via unit- and item nonresponse. CATI seems to

increase coverage problems espcially at the tails of the distribution. Obviously none of

the very high as well as very few low income households could be reached via CATI.

Furthermore, it can be seen that both disposable and gross income increased from 2007

to 2008 on average 6% and 9% respectively. The difference in the change between the

two groups, however, seems small compared to the difference of 2008 income, i.e. 2

percentage points relative to about 20% (non log).

Table 4 also provides the mean for all household level16 and personal level con-

trol variables from the EU-SILC wave of 2007. There are some albeit generally small

differences in the averages between the two groups. Most noticeable, more affluent

households in terms of income (see the difference of log income in 2007) as well as

wealth (see as two indicators the percentage of owning the primary residence and the

size of the primary residence) seem to be (self-) selected to CATI. Additionally, phone

coverage (see land line and mobile phone coverage) and education seem to be lower

for households with a CAPI-interview and women answer the questionnaire more often

over the telephone. These differences indicate the possibility of households with certain

characteristics to be (self-) selected into one or the other interviewing mode, and hence

we model this process and control for it in the estimations (see Sections 4, and 4.1).

Another insight into how strong the selection into the different interviewing modes is

can be gained from the statistics in Table 5. According to income deciles in 2008 we

see in the second column that the participation rate in a CAPI interview is decreasing

with income. Stated otherwise households with higher income tend to opt at a higher

rate for CATI-interviews. Also it is reported that item nonresponse is higher for higher

income households. However, there is a staggering difference of more than 10 percent-

10

age points in the two top deciles. At the lowest end, in the first decile the difference is

almost as high, and in between income from households with CATI-interviews always

have higher item nonresponse, although the difference is not stable over the deciles.

As can be seen in the last two columns, the percentage of households with missing

income data does not follow the same trend in 2007. Here the two groups are defined

according to the 2008 interview mode choice and we report item nonresponse in 2007

when both groups (CAPI as well as CATI) were still interviewed via CAPI. Again the

percent of households with missing income data increases with income level. The dif-

ference between the groups however is not stable already suggesting that there might

be indeed a interview mode effect at play. In the top decile the income of households

that were interviewed via CAPI in 2008 was missing more often than for the other group.

4 Identification and Estimation Strategy

We approach the problem of estimating the effect of the interview mode (CAPI versus

CATI) on the income distribution by referring to recent microeconometric literature on

causal inference and program evaluation as it best fits the quasi experimental setting

of our data. The background of this approach can be found in the books of Morgan

and Winship (2007), Angrist and Pischke (2008), Abbring and Heckman (2007), or

the papers of Angrist and Krueger (1998), Blundell and Dias (2002), and Imbens and

Wooldridge (2009).

As we cannot assume random assignment of the interview mode in the case of EU-

SILC 2008 we need to control for selection in order to justify a causal interpretation of

the estimated effects. In order to check the validity of the Conditional Independence

Assumption and robustness of the results in general we use various ways (OLS, OLS

11

with controls, fully interacted model, propensity score matching, and coarsened exact

matching techniques) to control for the selection into the different interview modes

when estimating the average treatment effects (ATE) on item nonresponse and income

levels.

Note that even small effects on different parts of the income distribution might accumu-

late to relevant overall effects on income inequality measures. That is why we turn in a

second step towards an analysis of the mode effect on income inequality measures. After

evaluating in a naive way simply the differences of various statistics about the income

distribution, we follow the approach suggested by Firpo et al. (2009) and use recentered

influence functions (RIF) to estimate the ATE of a a bivariate variable (i.e. CAPI vs.

CATI) on any distributional statistic ν(F (Y )), where F (Y ) is the unconditional dis-

tribution of household income. The results concerning the effect of the interview mode

on the distribution reported below are based on these calculation of the RIF17 and use

appropriate weights in order to estimate the effect on the population estimate of the

distribution. For the standard errors the delta method is used. A robustness check

with bootstrapped standard errors, however, yields very similar results.

For the readers convenience the analytical tools are summed up in Appendix B.

4.1 (Self-) Selection with relation to the mode

In order to model the selection into interview mode we apply a logit model where the

dependent variable is the interview mode (CATI=0, CAPI=1) and the explanatory

variables are the controls.18 Table 6 shows the average marginal effects calculated from

this model. Additionally, Figure 1 displays the common support of the propensity

scores given by the logit model. One can see that almost the whole support of both,

CATI- and CAPI-households’ propensity scores, is included in the common support.

Selection bias can hence be expected to be relatively small. However, the chance of (self-

12

) selection to CAPI decreases (statistically) significant on a 5% level with household

size, household disposable income, the availability of a telephone line and a mobile

phone in the household as well as the main income earner being female, being married

and living together as well as having higher educational attainment.

4.2 Methods used to control for selection bias

As a first step we estimate logit regressions using HINR as dependent variables and the

the CAPI (mode) dummy as independent variable as well as two sets of controls, i.e.

only household-level and household- and personal-level controls. Our next step in terms

of flexibility is to estimate a fully interacted linear model (FILM),19 which allows the

effect of the mode to vary over all controls. A further step in terms of flexibility is to

estimate treatment effects based on propensity score matching (PSM).20 While FILM

allows the mode effects to vary over all controls, propensity score matching additionally

allows (i) to impose common support based on the overlapping regions of the propensity

scores (see Figure 1) and (ii) does - due to its semi-parametric nature - not impose as

strong linearity assumptions as logit and FILM are imposing. On the imposed common

support we match the nearest - in terms of the propensity score - CATI-neighbour to

every CAPI-household (1 to 1 matching) in order to balance the joint distribution of

the covariates (controls).21

We use coarsened exact matching (CEM) as a further robustness check. Iacus et al.

(2008) developed a method to temporarily coarse data based on ex-ante user choice and

then run the analysis on the common support of the uncoarsened data.22

While PSM still uses a parametric model to match CAPI and CATI observations and

therefore extrapolates outside the common support23 in order to calculate treatment

effects, CEM imposes a user input based non-parametric matching strategy to balance

the joint distributions of covariates among CAPI- and CATI-observations. This reduces

13

the necessary extrapolation outside the common support but of course comes with a

decrease in sample-size.

As matching variables we use a subset of our household and personal level covari-

ates. For categorical variables (household size and being female as the household head)

we impose an exact matching strategy and for the two continuous variables we use

a coarsening strategy for matching on certain parts of the distributions by imposing

cut-points (household level: household disposable income 20000, 30000, 40000, 50000,

60000; and personal level: age 30, 40, 50, 60, 70). This matching strategy leads to a

perfectly balanced dataset in terms of the joint distribution of the categorical variables.

As household disposable income is allowed to be matched in approximately 10,000 Euro

brackets (and below 20,000 Euro as well as over 70,000 Euro) and age is allowed to be

matched in about 10 year age brackets (and below 30 as well as older than 70) some

imbalances remain with respect to those two variables. Out of the 343 covariate com-

binations defined we find 224 which define the common support, i.e. where at least

one CAPI and one CATI observation can be found. In terms of the sample the total of

3,377 observations collapses to 3,190 observations which lie inside the common support.

63 CATI- and 124 CAPI-observations lie outside the common support. The weights for

the matched observations are chosen in a way that CAPI and CATI observations are

balanced for each covariate combination. To control for the remaining imbalances in

the matched dataset we still use age and household disposable income as controls when

we calculate the treatment effects.

14

5 Results

5.1 Mode effects on income item non-response

Table 7 shows the estimated effects of CAPI on item non-response. All estimated ef-

fects resulting from the logit regressions are significant at a 1% significance level, but

are not significantly different from each other indicating again that selection bias is

rather small. The estimate using the largest set of controls, i.e. all household- and

personal-level controls given in Table 4, is −0.071 which indicates that the probability

of item non-response when interviewed via CAPI is 7.1 percentage points (the average

item non-response is 21.8%) lower than in the case of a CATI interview. This implies

a reduction of item non-response of about 30% given the EU-SILC estimates.

The estimates resulting from the FILM and PSM estimations closely resemble the

logit estimates. The FILM and PSM estimates are −0.076 and −0.072 respectively

and significant at the 5% level. The estimate using 3,190 CAPI households after the

coarsened exact matching and reweighting to balance the covariate distributions is

−0.068 and significant at the 5% level. Thus these more flexible estimation approaches

confirm the previous finding of the interview mode on the item non-response using the

somewhat naive linear approach.

5.2 Mode effects on the income level

Analogous to the estimates of the mode effects on item nonresponse we estimate the

effect of the mode on household income again with increasing flexibility and less re-

strictive assumptions. All estimates shown in Table 8 are produced analogously to the

estimates in Table 7, but with the logarithm of household income of 2008 being the

dependent variable instead of HINR. Overall we find a significant negative effect of the

15

CAPI mode on average (log) household income. The estimate ranges between −0.02

(propensity score matching) and −0.2 (OLS without controls). This estimate (taking

as an example 0.04) implies that interviewing via CAPI leads on average about 1,000

Euro lower reported income values than interviewing via CATI. As the income distri-

bution is very skewed and most households have income below the mean income this

result already implies that CATI leads on average to income values closer to the mean

income. Again the more flexible methods FILM, PSM and CEM confirm the negative

effect, however the PSM estimate is not significant.

The average effect, however, is of no help if one wants to analyse the impact on the

entire distribution and resulting inequality measures. In fact it might be quite mislead-

ing since the effect on inequality measures is positive instead of negative. Furthermore,

even many small and insignificant effects over the distribution could accumulate to

relevant effects on inequality measures.

5.3 Mode effects on the income distribution

Differences

Starting from a simple idea in order to compare the whole distribution in more detail

we calculate the difference of a given statistic for both sub-samples,24 i.e. for house-

holds interviewed with CAPI and CATI respectively, and provide standard errors of

this difference using a bootstrap with 500 replicates.25 We apply this procedure to the

GINI-coefficient; the poverty rate, i.e. the proportion of population with lower income

than 60% of the median; and the 90/10 percentile ratio. Additionally, we provide the es-

timates of the difference between the CAPI- and the CATI sample statistics (i) without

any adjustments and (ii) on the common support resulting from the CEM-Procedure,

which controls partly for selection bias but at the cost of a reduction of the sample.

16

Table 9 reports first the measures of the Gini-Coefficient, the Poverty Rate, and the

90/10 percentile ratio for both the CAPI and CATI sub-samples in EU-SILC 2008. We

find that the difference of the Gini-Coefficient is 0.026 for the full sample and 0.024 using

only the common support sample implying a 8.1% and 7.2% lower Gini-coefficient using

CATI instead of CAPI. While the estimate of the whole sample is significant at the 5%

level, the difference for the balanced sample is only significant at the 10% level.26 Table

9 additionally reports a significantly higher poverty rate for the CAPI sample (i.e. 3.9

percentage points in the whole and 3.1 percentage points in the balanced sample) and

an (statistically) insignificant difference of the 90/10-percentile ratio between the two

sub-samples. To control rigorously for selection we employ RIF-regressions to estimate

the effect of interview mode on the distributional statistics of income.

Overall Effect

We regress the RIF of the Gini, the Poverty Rate, and the 90/10 percentile ratio of the

income distribution of 2008 on the interview mode and (i) linear, squared and cubed

income from 2007 as well as (ii) all our controls for households characteristics (including

income). In both cases also all interactions of the control variables with the interview

modes are included in the model. For the standard errors of the reported ATE we

use the delta method.27 Using CATI instead of CAPI as interview modes reduces the

observed Gini-Coefficients significantly (see Table 2). According to our estimates of

0.033 with the limited set of controls and 0.027 using the full set of controls, inequality

measured by the Gini Coefficient drops by around 10% if the interview mode is switched

from CAPI to CATI. This is a severe effect on the most prevalent inequality measure.

Furthermore, we see that the estimate on the poverty rate decreases greatly and looses

its significance and the 90/10-percentile ratio switches signs but remains insignificant.

17

The result suggests that using the latter two measures when comparing inequality over

time or between countries might be more robust.

Heterogeneity of the effect

The estimates for the difference in percentiles are reported in Figure 2. Panel (a) to (c)

show that the effect of the interview mode on the percentiles28 follows a u-shape. These

graphs are based on the full sample in Panel a), the balanced (using the coarsened exact

matching procedure outlined above) observations in Panel b), and the re-weighted k to

k matching (same matching procedure as before) in Panel c). The u-shape means that

the percentiles are lower for households interviewed with CAPI but less so (and in the

more controlled version even higher) at the extremes of the distribution. This implies

a higher spread of income in households interviewed with CAPI. Our understanding is,

that there are two mechanism that play an important role here. First, it is easier to

lie over the phone, hence biasing reported income at the extremes of the distribution

towards the mean; and second, households at the tail of the distribution are especially

hard to interview, and thus are mostly interviewed via CAPI (whilst in a CATI survey

these households, however important they are for a correct estimate, do not take part

at all).

Panel d) of Figure 2 shows the ATE on the percentiles using RIF-regression estimates

with the full set of control variables. We see again that it follows a u-shape. The effect

of the interview mode on the percentiles is closer to zero, since the model controls for a

wide range of characteristics, and is negative in most parts of the income distribution.

In a distribution that is skewed to the right this coupled with the positive effect for the

highest income percentiles translates once again to a more compressed income distri-

bution for households interviewed with CATI. Note that the coverage effect, i.e. that

18

households at the extremes of the distribution are only covered with CAPI interviews,

is not accounted for in this estimation.

6 Conclusion

In this paper we exploited the rare situation of having panel data in combination with

a change of the interview mode, which occurred in the 2008 wave of the Austrian EU-

SILC data for some - but not all - panel households. The quasi-experimental nature of

these data allowed us to estimate causal effects of this change of the interview mode.

We have studied the effect of the interview mode (CAPI versus CATI) on item non-

response and the level as well as the distribution of household income.

First, we find descriptive evidence that CATI compresses the income distribution by

leading to less coverage in the final sample via higher unit-nonresponse especially at

the tails of the distribution. Second, controlling for a rich set of covariates from the

baseline survey, we find that the change from CAPI to CATI has increased item non-

response significantly in statistical as well as real terms. This result is robust over all

the parametric, semi-parametric and non-parametric methods - which allow for a huge

degree of flexibility - we applied. Every researcher pursuing answers to economic ques-

tions with the evaluation of survey data should thus be concerned with the interviewing

mode of data collection and the follow up imputations. One has to keep in mind that

all missing values are usually imputed in various ways or, even more severe, dropped

from the analysis altogether. Again CATI tends to compress the observed income dis-

tribution - in this case via item nonresponse especially at the tails. Third, we find that

overall households which are interviewed by CATI on average report higher income

values. This effect is stronger for lower income than for higher income. In general the

19

income distribution is skewed implying that most part of the distribution lies below

the mean. Compared to CAPI the effect of CATI therefore is a mean-reverting one,

again compressing the income distribution. Ultimately we find that this level effect

accumulates to a severe bias with regard to income inequality measured by the Gini

coefficient. We conduct RIF-Regressions to compare the effect of the introduction of

CATI on the unconditional distribution of income and find a highly significant effect

which reduces the Gini-Coefficient by around 10%.

Given these results in terms of the effects of the mode of on income distribution we

advocate the careful use of survey data taking into account different interview modes

and being aware of the possibility that differences between countries and within coun-

tries over time might be due to yet another candidate, namely the interview mode.

Commonly used rankings of countries by Gini coefficients (see OECD (2011) or Table

1) of the income distribution might largely be an artifact of different survey techniques

such as the interview mode than true differences in income inequality.

20

Notes

1We’d like to thank Markus Knell for valuable comments and discussion. Additional to the usual

disclaimer, the opinions expressed in this work are those of the authors and do not necessarily reflect

the ones of the OeNB.

2Economic Analysis Division, Oesterreichische Nationalbank.

3Assistant Professor, Department of Economics, Harvard University.

4Economic Analysis Division, Oesterreichische Nationalbank.

5A statistic is called robust if it has a bounded influence function, see Huber (2003).

6See http://epp.eurostat.ec.europa.eu/portal/page/portal/microdata/eu_silc for further

informations [accessed on 13th of November 2012].

7See Statistik Austria (2010) for a version in German.

8According to the handbook (see Statistik Austria (2010)) there were about 160 and 40 interviewer

respectively for the CAPI and CATI part of the survey. Information on which household was inter-

viewed by which interviewer, to control for possible interviewer effects, however, is unfortunately not

part of the EU-SILC data. There seem to be enough interviewers, however, for the possible interviewer

bias being rather small.

9There is a minor technical issue with 15 households that were not part of the 2007 wave, were,

however, interviewed over the telephone in 2008. From the information provided by Statistik Austria

in a personal correspondence this is due to the fact the households that were already part of the panel

component in 2006, but could not be reached in 2007 (wave non-response), were possibly interviewed

in the CAPI mode. Since this is only the case for the 2008-wave and was changed in subsequent waves

to completely drop those households, we leave out these 15 households in the analysis. Furthermore as

there exists no 2007 information on these 15 households we can not use them in our empirical exercise.

10This information is based both on the official documentation (see Statistik Austria (2010)) and

personally transmitted material from Statistik Austria.

11Information from Statistik Austria: ”CAPI interview mode was offered only if there is a refusal of

the phone interview, there are health related difficulties, or age related reasons that make a telephone

interview impractical.”

12No general rules were set for interviewing time of CAPI-interviews.

13Furthermore, as a robustness check, we repeated the analysis (results can be provided upon re-

quest) using a differently constructed measure of item non-response. The construction is based on the

21

individual income components and is calculated as the percentage of components imputed (i.e. missing

in the first place) for the total household income. The results are qualitatively very similar and left

out due to space constraints.

14As a robustness check we also worked with gross income.

15These estimates are not weighted since we are not interested in population but only sample averages

at this stage of the analysis.

16We also included regional indicators for of households as controls, but do not report them in Table

4 due to space constraints. The partitioning is approximately as follows: Burgenland 4%; Kaernten

8%; Niederoesterreich 20%; Oberoesterreich 20%; Salzburg 7%; Steiermark 14%; Tirol 8%; Vorarlberg

5%; and Wien 17%.

17For the calculation of the RIF the reader is also referred to Essama-Nssah and Lambert (2011).

18We also control for regional differences (not reported). The estimates of which are not significant.

19We use film, a STATA program provided by Edwin Leuven and Barbara Sianesi, see http://www.

ifs.org.uk/publications/2712 [accessed on 19th of November 2012].

20We use psmatch2, a STATA program provided by Edwin Leuven and Barbara Sianesi, see http:

//www.ifs.org.uk/publications/2684 [accessed on 19th of November 2012].

21Note that propensity score matching is not exact matching. That means, that the common support

imposed by the propensity score differs from the common support in the strict sense which would

include only cells of covariate combinations which include both CAPI- and CATI-households (the later

is a subset of the former.). As exact matching is not feasible given finite data and many (and including

continous) covariates, the resulting model still extrapolates also for cells which do not include both

CAPI- and CATI-households

22We use cem, a STATA program provided by Matthew Blackwell, Stefano Iacus, Gary King and

Giuseppe Porro, see http://ideas.repec.org/c/boc/bocode/s457127.html [accessed on 19th of

November 2012].

23With relation to realizations of CAPI- and CATI-observations in cells defined by covariate combi-

nations and not with relation to the common support of the propensity scores.

24Appropriate weights are used in the calculation of the statistics, since we are now interested in the

effect on the population and not, as before, in the sample.

25For the estimations the whole procedure is bootstrapped in the sense that a bootstrap sub-sample

is drawn, and then the difference of the statistic calculated. With these 500 estimates of the differences

22

we are able to estimate standard errors.

26To check for the differences at the top we also used a General Entropy Class index with α = 2 which

shows huge differences (0.18 for the CATI versus 0.49 for the CAPI common support samples). This

is due to the fact that it is very sensitive to top income and as we saw the highest income observations

are all from CAPI interviews. This sensitivity, however, also renders a high variability of replicates in

this bootstrapping procedure and thus generates high standard errors yielding insignificant results.

27Thus the standard errors are based on the whole sample of the 2008 wave of EU-SILC and not

only on the panel component. Bootstrapped standard errors using 500 replicates were also calculated

as a robustness check and yield very similar results.

28The effect is estimated for 20 quantiles.

23

Appendix A Figures and tables

24

Table 1: Overview of EU-SILC surveys and their mode of data collection

PAPI CAPI CATI Self-administrated Gini 2007 adjusted GiniBelgium 0.0 100.0 0.0 0.0 26.3 -

Czech Republic 99.7 0.0 0.0 0.3 25.3 -Denmark 0.0 0.0 94.2 5.8 25.2 27.6Germany 0.0 0.0 0.0 100.0 30.4 -

Estonia 2.2 97.6 0.2 0.0 33.4 33.4Ireland 0.0 100.0 0.0 0.0 31.3 -Greece 80.8 14.9 2.1 2.3 34.3 34.4Spain 0.0 92.9 7.1 0.0 31.2 31.4

France 0.0 100.0 0.0 0.0 26.6 -Italy 100.0 0.0 0.0 0.0 32.3 -

Cyprus 0.0 100.0 0.0 0.0 29.8 -Latvia 11.3 81.2 7.5 0.1 35.7 36.0

Lithuania 95.3 0.0 3.8 0.9 33.8 33.9Luxembourg 100.0 0.0 0.0 0.0 27.4 -

Hungary 100.0 0.0 0.0 0.0 25.6 -Malta 0.0 100.0 0.0 0.0 26.3 -

The Netherlands 0.0 0.0 100.0 0.0 27.6 30.4Austria 0.0 94.0 6.0 0.0 26.2 26.4Poland 100.0 0.0 0.0 0.0 32.2 -

Portugal 8.0 92.0 0.0 0.0 36.8 -Slovenia 0.0 44.5 55.5 0.0 23.2 24.5Slovakia 99.4 0.0 0.0 0.7 24.5 -Finland 0.0 3.4 96.6 0.0 26.2 28.7Sweden 0.0 0.0 100.0 0.0 23.4 25.7

United Kingdom 0.0 100.0 0.0 0.0 32.6 -Iceland 0.0 0.0 100.0 0.0 28.0 30.8Norway 0.0 0.6 99.4 0.0 23.7 26.1

Notes:(i) This table shows percent shares of EU-SILC surveys in 2007 conducted by paper assisted personalinterview (PAPI), computer assisted personal interview (CAPI), computer assisted telephone interview(CATI), and self administered.(ii) Gini-Coefficients are based on household disposable equivalence income.(iii) The adjusted Gini-Coefficients is the “Back of the Envelope” calculation accounting for the effect(decrease of 10%) of the CATI interviewing technique from the RIF-regression.(iv) Source: Eurostat: Comparative Intermediate EU Quality Report 2007. Version 5, and Eurostatwebsite for Gini Coefficients.

25

Table 2: Effect of Interview Mode an Inequality

Gini Coeff. Poverty rate 90/10 Percentile RatioRIF-regression 0.0329 0.0274 0.0068 0.0019 -0.2344 -0.1829

(0.0107) (0.0101) (0.0127) (0.0134) (0.2469) (0.2571)Controlling forpast income X X X X X Xother covariates X X X

Notes:(i) This table shows the effect of the interview mode on aggregate measures of inequality. We reportthe inequality statistic (Gini coefficient, poverty rate, and the percentile ratio), the difference betweenthe sub-samples and the effect of the interview mode using RIF-regressions.(ii) Standard errors are reported using delta methods.(iii) Source: EU-SILC 07/08.

26

Table 3: Number of Households in different modes and waves

Number %-ShareTotal Number of Households 2008 5,711 100%CATI-Interview-Mode 1,710 29.94%CAPI-Interview-Mode 4,001 70.06%Panel Households 2007/2008 3,772 66.05%CATI-Test Households 2007 395 6.92%Effective Sample Size 3,377 59,13%From the Effective Sample Mode was:

CATI 1,438 42.58%CAPI 1,939 57.42%

Notes:(i) This table reports the household sample size for the 2007/2008 waves of EU-SILC.(ii) Effective sample size is ”Panel Households” minus ”Test Household 2007”.(iii)Source: EU-SILC 07/08.

27

Table 4: Mean of Dependent and Explanatory Variables

Panel CAPI CATIDependent variables: 2008Mean household disposable income 10.24 10.16 10.37Mean household gross income 10.51 10.42 10.66Household Item nonresponse 21.84 18.08 27.53Share of interviews below 10k 16.52Interviewed by 74.80 25.20Share of interviews above 200k 0.07Interviewed by 100.00 0.00Mean difference disposable income to 2007 0.06 0.07 0.05Mean difference gross income to 2007 0.09 0.10 0.08

Control variables at the household level: 2007Household size 2.35 2.28 2.45Households with kids 31.06 29.27 33.65Single family home 39.34 37.16 42.48Home-owners 53.51 49.40 59.43Size of Flat in sqm 97.96 92.60 105.70Land line coverage 66.16 58.14 77.74Mobile phone coverage 87.30 85.18 90.36Mean household disposable income 10.18 10.09 10.32Mean household gross income 10.42 10.32 10.58City 38.44 38.18 38.81Urban 26.17 26.95 25.04Rural 36.03 35.73 36.46

Control variables at the level of the household head: 2007Female 45.87 44.03 48.52Blue collar 43.44 42.46 44.85Self-employment 6.21 5.77 6.83Jobless 50.36 51.77 48.32Weekly working hours 19.91 19.84 20.02Married 49.69 45.52 55.71Education: secondary school 21.35 25.91 14.76Education: apprenticeship 52.54 52.68 52.34Education: higher secondary school 16.04 13.13 20.25Education: university 10.07 8.28 12.66Age 52.39 51.86 53.14

Notes:(i) This table shows the mean of the variables on the consideration as well as the full set of controlvariables.(ii) Each statistic is provided for the panel component of the 2008 wave, and the CAPI and CATIsub-samples of the panel component.(iii) All income variables are reported after taking the natural logarithm.(iv) The percentage of households in each region is left out due to space constraints, but the partitioningis approximately as follows: Burgenland 4%; Kaernten 8%; Niederoesterreich 20%; Oberoesterreich20%; Salzburg 7%; Steiermark 14%; Tirol 8%; Vorarlberg 5%; and Wien 17%.(vi) Source: EU-SILC 07/08.

28

Table 5: Structure of Missingness of Income over Deciles Conditionalon Interview Mode

2008 2007Interview mode CAPI or CATI CAPIInterview mode in 2008 % CAPI Int CATI CAPI CATI CAPITotal . 0.271 0.181 0.271 0.234First Decile 0.691 0.233 0.135 0.223 0.157Second Decile 0.715 0.124 0.108 0.124 0.188Thrid Decile 0.629 0.174 0.141 0.215 0.161Fourth Decile 0.596 0.241 0.143 0.158 0.173Fifth Decile 0.557 0.203 0.118 0.250 0.237Sixth Decile 0.592 0.223 0.214 0.304 0.260Seventh Decile 0.556 0.278 0.190 0.311 0.243Eigth Decile 0.520 0.295 0.267 0.301 0.233Ninth Decile 0.467 0.341 0.236 0.358 0.344Tenth Decile 0.441 0.435 0.316 0.330 0.424

Notes:(i) This table shows the average use of the CAPI interviewing mode over deciles (column 2).(ii) Additionally, one can see the average rate of missing data on the income variable for 2008 and2007 over the deciles (column 3 and 4, as well as 5 and 6 respectively).(iii) Source: EU-SILC 07/08.

29

Table 6: Logit-regression of the (Self-)Selection of the Mode on Con-trol Variables

Selection towards CAPIMode Selection

household characteristicsHousehold size 0.031

(0.012)Household with kids -0.041

(0.028)Single family home 0.023

(0.022)Owner occupier -0.028

(0.021)Living space in sqm -0.001

(0.000)Land line -0.178

(0.020)Mobile phone -0.097

(0.029)Log-disposable income -0.052

(0.018)Personal characteristics of household headFemale -0.072

(0.018)Self-employed 0.009

(0.037)Jobless 0.079

(0.037)Weekly working hours 0.002

(0.001)Married and living together -0.068

(0.022)Education: apprenticeship -0.090

(0.023)Education: higher sec. school -0.191

(0.030)Education: university -0.131

(0.036)Age -0.001

(0.001)N 3376

Notes:(i)This table shows average marginal effects (AME) of household characteristics of the 2007 EU-SILCwave on being interviewed by CAPI in the 2008 EU-SILC wave. All average marginal effects arecalculated from a logistic regression using an CATI(0)/CAPI(1) as dependent variable. Furthermorewe controlled for federal states and regional population density. The coefficents of both are notsignificant.(ii) Standard errors calculated by the delta method are given in parentheses.(iii) Source: EU-SILC 07/08.

30

Figure 1: Propensity score density of CAPI and CATI

Notes:

(i) This graph shows estimated propensity score densities resulting from the logit model presented in

table 6.

(ii) Source: EU-SILC 07/08.

31

Table 7: Interview Mode Effect on Income Item Non-Response

I II IIILogit Model -0.088 -0.075 -0.071

(0.014) (0.014) (0.015)Fully Interacted Model -0.076

(0.016)Propensity Score Matching -0.072

(0.022)Coarsened Exact Matching -0.068

(0.014)N 3291 3290Household Controls X XPersonal Controls X

Notes:(i)This table shows average partial effects (APE) of being interviewed by CAPI on household andpersonal income item non-response. Results are reported from a logistic (using an item non-responsedummy for household income [at least one item non-response in an income question] as dependentvariable) as well as a fully interacted model and various matching techniques.(ii) Standard errors are given in parentheses (for the standard errors of the marginal effects that deltamethod is applied).(iii) Source: EU-SILC 07/08.

32

Table 8: Interview Mode Effect on Household Income

I II IIIAverage effect of CAPI on log household incomeOLS-Regression -0.205 -0.052 -0.040

(0.022) (0.016) (0.015)Fully Interacted Model -0.043

(0.016)Propensity Score Matching -0.017

(0.032)Coarsened Exact Matching -0.060

(0.017)N 3377 3376Household Controls X XPersonal Controls X

Notes:(i) This table shows the effect (regression coefficient, as well as matching estimators) of being inter-viewed by CAPI on the logarithm of household disposable income.(ii) Standard are given in parentheses.(iii) Source: EU-SILC 07/08.

33

Table 9: Differences of Inequality Measures between Interview Modes

Gini Coeff. Poverty rate 90/10 Percentile RatioIa IIa Ib IIb Ic IIc

Inequality Measure CAPI 0.3341 0.3332 0.2280 0.2226 4.801 4.791Inequality Measure CATI 0.3076 0.3096 0.1885 0.1920 4.523 4.527

Difference (Bootstrap) 0.026 0.024 0.039 0.031 0.278 0.263Bootstrapped Std. Err. (0.012) (0.013) (0.014) (0.013) (0.218) (0.267)

Notes:(i) This table shows the effect of the interview mode on aggregate measures of inequality. We report theinequality statistic (Gini Coefficient, the Poverty Rate, and the Percentile Ratio), and the differencebetween the sub-samples.(ii) Standard errors are reported using bootstrapping method.(iii) Columns Ia-c show the results using the full (panel) sample, i.e. 3.377 observation; and columnsIIa-c only use the matched observation from the above coarsened exact matching procedure.(iv) Source: EU-SILC 07/08.

34

Figure 2: Average treatment effect on quantiles

(a) Difference in percentiles (b) Common support

(c) Re-weighting (d) RIF-regression on percentiles

Notes:

(i) This graphs show the effect of the interview mode on the percentiles (20 percentiles were used)

over the whole income distribution. Panel a) displays the simple difference of the percentiles, Panel b)

shows the differences for the balanced sample using the coarsened exact matching technique explained

above, Panel c) shows the results for the exact k to k matching within a bin of the matching procedure,

and Panel d) shows the effect on the percentiles using RIF-regressions.

(ii) 95%-confidence intervals are provided using bootstrapping standard errors (Panel a) to c)) and the

delta method (Panel d)).

(iii) Source: EU-SILC 07/08.

35

Appendix B Technical notes

Appendix B.1 Causality of the interview mode effect

Let us assume N individuals which are indexed by i = 1, ...,N and randomly drawn

from a large population. Each individual can be characterized by a pair of potential

responses, R0i for the outcome without treatment and R1i for the outcome under the

treatment. In addition, each individual has a vector of characteristics, referred to as

covariates, Xi. The problem at hand can be interpreted as a quasi-experiment,

E [Ri∣Mi = 1] −E [Ri∣Mi = 0]´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶

Difference in response

= E [R1i∣Mi = 1] −E [R0i∣Mi = 1]´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶

Average effect of CAPI on CAPI-Responses

+ E [R0i∣Mi = 1] −E [R0i∣Mi = 0]´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶

Selection bias due to mode assignment

, (1)

where Ri is the response behaviour, i.e. some outcome variable (in this analysis it

is the item non-response behaviour and the reported income), and Mi is the mode the

respondent is confronted with, i.e. CAPI or CATI. Equation 1 shows that the difference

in the means of some outcome variable between the CAPI- and CATI-Respondents

consists of an average treatment effect on the treated, i.e. the effect of CAPI (1 signifies

CAPI in the subscripts whereas 0 signifies CATI) on those interviewed with mode CAPI.

But what is added to the treatment effect is a selection bias resulting from the fact that

respondents who are selected to CAPI might differ from those who are selected to CATI.

If the mode would be assigned randomly, the selection bias would be zero. Formally we

can interchange E [R0i∣Mi = 1] and E [R0i∣Mi = 0] and could rearrange equation 1 to

E [Ri∣Mi = 1] −E [Ri∣Mi = 0] = E [R1i∣Mi = 1] −E [R0i∣Mi = 1] = E [R1i −R0i] .

36

As we can not assume random assignment of the mode we need to control for se-

lection in order to to justify a causal interpretation of estimated effects. Under the

Conditional Independence Assumption (CIA) also refered to as unconfoundedness as-

sumption, given as

{R0i,R1i} áMi∣Xi, (2)

selection bias disappears. Under this assumption differences in averages across mode

groups have a causal interpretation, because the assignment of the mode conditional

on selection variables Xi is at random. The remaining question is how to select the

vector of covariates Xi. This set of controls needs to be chosen in a certain way to

reasonably ensure the CIA but without introducing bad control bias, which is another

form of selection bias (see e.g. Angrist and Pischke (2008)). Good controls are in our

case variables which are themselves not a (possible) outcome of the mode. A safe way to

prevent the danger of using controls which could induce selection bias is to use controls

which are themselves already fixed at the time of mode selection or strictly exogenous

variables in relation to the mode (Imbens (2004)). In our case all characteristics of a

certain household i gathered already before the interview (in 2008) are natural candi-

dates (2007 wave variables).

Appendix B.2 Effect on the income distribution - Recentered

Influence Functions

The above summarized approach, however, yields only a causal effect on the mean of

the outcome variable. Especially, since we are interested in the effect on the income

distribution, we turn to another approach in the literature. After evaluating in a naive

way simply the differences of various statistics of the income distribution, we follow the

37

approach suggested by Firpo et al. (2009). To summarize, the Influence Function can

be used in order to estimate the ATE of a a bivariate variable (i.e. CAPI vs. CATI)

on any distributional statistic ν(F (Y )), where F (Y ) is the unconditional distribution

of household income. A Linear approximation for ν(F (Y )) is given by

ν(F ) ≈ ν(F 0) + ∫ IF (Y ;ν,F 0)dF (Y ),

where F 0 is the baseline distribution, which is in our case the distribution given CATI

interviews. This approximation exists for “smooth” statistics ν, by Freechet differenti-

ation and the Riesz representation theorem. Given the CIA (see 2) and our treatment

indicator Mi we have

E[IF (Y i)∣X] = E[IF (Y )∣X,Mi = i]

, which justifies estimation of the following linear model, with appropriate interactions

and powers of the covariate vector X:

E[IF (Y )∣X,Mi] =X ⋅ β0 +Mi ⋅X ⋅ β1 (3)

The effect of the the mode Mi on the population ν is thus given by

ν(F 1) − ν(F 0) = E[X] ⋅ β1. (4)

With this procedure we estimate the effect on the Gini-Coefficient as well as the

poverty rate (two of the most widely published estimates from EU-SILC), i.e. the share

of the household population with less than 60% of the median income. The Recentered

Influence Function (RIF) for the Gini-Coefficient (see e.g. Firpo et al. (2007) page 24f.)

can be written as,

38

RIF (y, νGI , F 0) = 1 + 2

µ2R(Fy)y −

2

µ[y(1 − p(y)) +GL(p(y), Fy)]

,where R(Fy) is the integral of the Generalized Lorenz curve with the ordinates

GL(●), and p(y) is the density. Additionally the RIF for the poverty rate νPR(F ) =

F (0.6 ⋅ F −1(0.5)), where med ∶= F −1(0.5) is the median for the distribution given by F

and 0.6 ⋅ F −1(0.5) = 0.6 ⋅med is the poverty ceiling, is given by

RIF (y;νPR, F 0) = 1(y ≤ 0.6 ⋅med) − 0.6 ⋅ f(0.6 ⋅med)f(med) ⋅ 1(y ≤med).

Finally, we can calculate the RIF of the the P90/P10-percentile ratio given as

νRA = Q90

Q10 where Qi is the i’s percentile of the income distribution by

RIF (y;νRA, F 0) = ν + ν ∗ [ 1(y ≤ Q10)Q10 ∗ f(Q10) −

1(y ≤ Q90)Q90 ∗ f(Q90)] .

39

References

Abbring, Jaap H. and James J. Heckman, “Econometric Evaluation of Social

Programs, Part III: Distributional Treatment Effects, Dynamic Treatment Effects,

Dynamic Discrete Choice, and General Equilibrium Policy Evaluation,” in James J.

Heckman and Edward E. Leamer, eds., Handbook of Econometrics, 1 ed., Vol. 6, Part

B, Elsevier, 2007, chapter 72.

Angrist, Joshua and Alan B. Krueger, “Empirical Strategies in Labor Economics,”

Working Papers 98-7, Massachusetts Institute of Technology (MIT), Department of

Economics 1998.

and Jorn-Steffen Pischke, Mostly harmless econometrics, Princeton University

Press, Princeton New Jersey, 2008.

Blundell, Richard and Monica Costa Dias, “Alternative approaches to evaluation

in empirical microeconomics,” CeMMAP Working Papers CWP10/02, Centre for

Microdata Methods and Practice, Institute for Fiscal Studies October 2002.

Cowell, Frank A. and Maria-Pia Victoria-Feser, “Robustness Properties of In-

equality Measures,” Econometrica, January 1996, 64 (1), 77–101.

de Leeuw, Edith D., Data Quality in Mail, Telephone and Face to Face Surveys,

Amsterdam: TT-Publikaties, Amsterdam, 1992.

, “The Effect of Computer-Assisted Interviewing on Data Quality: A Review of the

Evidence,” MethodikA/Department of Methodology and Statistics, Utrecht Univer-

sity 2008.

40

, Joop J. Hox, and Don A. Dillman, International Handbook of Survey Method-

ology, European Association of Methodology: Psychology Press, Taylor & Francis,

New York, 2008.

DiNardo, John, Nicole Fortin, and Thomas Lemieux, “Labor market institutions

and the distribution of wages, 1973-1992: A semiparametric approach,” Economet-

rica, 1996, 64, 1001–1044.

Duflo, Esther, Rachel Glennerster, and Michael Kremer, “Using Randomiza-

tion in Development Economics Research: A Toolkit,” Handbook of Development

Economics, June 2008, 4, 3895–3962.

Essama-Nssah, B. and Peter J. Lambert, “Influence functions for distributional

statistics,” Working Papers 236, ECINEQ, Society for the Study of Economic In-

equality 2011.

Firpo, Sergio, Nicole Fortin, and Thomas Lemieux, “Decomposing Wage Dis-

tributions using Recentered Influence Function Regressions,” Unpublished Working

Paper, University of British Columbia June 2007.

, Nicole M. Fortin, and Thomas Lemieux, “Unconditional Quantile Regression,”

Econometrica, 2009, 77 (3), 953–973.

Gabriella, Conti and Stephen Pudney, “Survey design and the analysis of satis-

faction,” Review of Economics and Statistics, 2011, 93 (3), 1087–1093.

Huber, Peter J., Robust Statistics Wiley Series in Probability and Statistics, John

Wiley & Sons, 2003.

Iacus, Stefano Maria, Gary King, and Giuseppe Porro, “Matching for Causal

Inference Without Balance Checking,” Working Paper Series, Harvard 2008.

41

Imbens, Guido W., “Nonparametric Estimation of Average Treatment Effects Under

Exogeneity: A Review,” The Review of Economics and Statistics, February 2004, 86

(1), 4–29.

and Jeffrey M. Wooldridge, “Recent Developments in the Econometrics of Pro-

gram Evaluation,” Journal of Economic Literature, March 2009, 47 (1), 5–86.

List, John A. and Imran Rasul, “Field Experiments in Labor Economics,” Handbook

of Labor Economics, 2011, 4, 103–228.

Lohmann, Henning, “Comparability of EU-SILC survey and register data: The re-

lationship among employment, earnings and poverty,” Journal of European Social

Policy, 2011, 21 (1), 1–18.

Morgan, Stephen L. and Christopher Winship, Counterfactuals and Causal In-

ference: Methods and Principles for Social Research Analytical Methods for Social

Research, Cambridge University Press, 2007.

OECD, Divided We Stand - Why Inequality Keeps Rising, OECD, 2011.

Statistik Austria, Standard Dokumentation Metainformationen - Definitionen, Er-

laeuterungen, Methoden, Qualitaet, Statistik Austria, 2010.

42