An ounce of planning is worth of a pound of weighting: Measuring cash holdings from the 2013 Bank of Canada Method-of-Payments Survey

Heng Chen, Bank of CanadaChris Henry, Bank of CanadaKim P. Huynh, Bank of CanadaQ. Rallye Shen, Bank of CanadaKyle Vincent, Bank of Canada

AbstractThis article details the methodology used in the Bank of Canada 2013 Method-of-Payments Survey to measure cash and non-cash payments. Measuring cash holdings is difficult due to the anonymous nature of cash and that some populations are hard-to-reach. To ensure that the survey is a representative sample, a variety of methods in survey design, weighting, and variance estimation are used to estimate a measure of cash holdings in the Canadian population. Overall, we find that Canadians hold on average about 84 dollars.

KeywordsSample weighting, non-probability samples, calibration, raking, variance estimation.

Acknowledgements: We thank Ben Fung, Geoff Gerdes, Catherine Haggerty, Arthur Kennickell, Marcos Sanches, and numerous colleagues at the Bank of Canada for their useful comments and encouragement in undertaking this survey. Maren Hansen provided excellent editorial assistance. We acknowledge the tremendous collaboration and support from Shelley Edwards, Jessica Wu, and Ipsos Reid for their dedication to this study. Finally, we thank Statistics Canada for providing access to the 2011 National Household Survey and the 2012 Canadian Internet Usage Survey. The views of this paper are those of the authors and do not represent the views of the Bank of Canada.

Section 1: Introduction

The Bank of Canada has an interest in understanding the levels of cash holding as it is the sole issuer of Canadian bank notes. Measuring the amount of cash holdings is difficult, however, as cash is an anonymous and untraceable payment method. Therefore in 2009, the Bank of Canada undertook a Method-of-Payments Survey, which found that demographic factors are strongly correlated with cash usage. Arango, Huynh, and Sabetti (2011), in their analysis of these factors, find that an increase in the availability and usage of non-cash payment methods such as debit, credit, and even mobile payments make holding cash seem quaint among younger, educated, and wealthier demographics.

The 2009 MOP introduced a three-day payment diary, which served as a memory aid to record all payments, including those in cash. This payment diary methodology has been successfully used in six other countries: Austria, Australia, France, Germany, the Netherlands, and the United States; more details are available at Bagnall et al (2014).In 2013, the Bank of Canada conducted the Method-of-Payments Survey (MOP) to measure cash usage in Canada. The 2013 MOP is a mixed-mode survey involving both

paper and online collection methods. An Online sample was selected from a market access panel accessible by email and an Offline sample from a panel accessible by regular mail. In addition, a subsample was taken from another comprehensive annual household survey, the Canadian Financial Monitor (CFM), which collects data on household finances from approximately 12,000 Canadian households per annum. The CFM survey instrument collects information that complements and overlaps with the MOP, providing a complete picture of household finances. In total, 3600 surveys were collected for the 2013 MOP from across the country and then weighted to ensure that the sample is representative of the Canadian population.

Surveys of the aforementioned type require post-stratification (sample weighting) as they are likely to be based on non-probability or, at best, highly unequal probability samples. For example, in the 2009 MOP the 18-24 year-old males were the most difficult to recruit and had the lowest response rates. Hence, as there is a large degree of heterogeneity in the recruitment procedure, this type of study benefits from sample calibration. The objective of this paper is to describe the methods undertaken to ensure that the 2013 MOP is a representative sample.

The chief methods used consist of:

1) Revamping and redesigning the survey to ensure that it is user-friendly and easy to fill out. Users report a satisfaction rating of about 90 percent on the survey.

2) Engaging respondents using a mixture of financial incentives and appeals to civic duty, via an official letter from the Governor of the Bank of Canada.

3) Subsampling the CFM, whose data collection is conducted on a rolling monthly basis. This provides our own survey the advantage of periodic updates.

4) Collaborating with the survey company to ensure that the field work achieves preset population targets. The survey company hit all the targets with the help of an additional boost wave, in which additional invitations were sent out promising high incentives for response.

5) Cleaning and editing the data using external data sources such as the market access panel demographic profile, and verifying the subsample with the CFM. As a result, the level of missing data is quite low, at only about 1-3 percent, and so only light imputation is required.

6) Using raking methodology to conduct post-stratification weighting, using external administrative and large-scale survey data from Statistics Canada. The weights from the post-stratification are invariant to initial weights.

7) Using resampling methods with bootstrap replicate survey weights (BRSW) for the variance estimation. Usage of BRSW results in a decrease of 20-50 percent in the variance of the estimate.

The rest of this paper describes in detail these procedures. Section 2 discusses points 1-5, Section 3 discusses the post-stratification methods, and Section 4 highlights the resampling methods for variance estimation. Finally, Section 5 concludes.

Section 2: The 2013 Method-of-Payments

The 2013 Method-of-Payments Survey is an update to the 2009 MOP. For further information on the 2009 MOP survey, refer to Arango and Welte (2012); for the 2013 MOP, see Henry, Huynh, and Shen (2014). Indeed, planning for the survey included incorporating lessons learned from both the 2009 MOP and the CFM.

One important lesson from the 2009 MOP is that certain hard-to-reach populations were under-represented in the final sample. This led to zero/low cell counts, which in turn gave rise to extreme weights. While sampling targets were achieved for marginal demographic counts, missing cell counts at a nested level caused difficulties for the weighting process.

Several measures were implemented to ensure that the sampling procedure in the 2013 MOP would avoid this problem. First, established sampling targets were nested by region, age, and gender. Thus we knew beforehand, for example, how many males aged 18-24 from the Prairies region of Canada were required for the sample to reflect the Canadian population. These pre-defined targets, built into the statement of services for the survey company, facilitated the ongoing monitoring of returns during data collection. Frequent updates allowed us to project which cells were likely to have an excess or shortage. Finally, various levels and types of incentives were randomly offered to potential respondents, which allowed us to determine the most effective combination. Financial incentives ranged from $5 to $20; other types of incentives included an advanced letter signed by the Governor of the Bank of Canada requesting participation, and a token $2 coin included in the survey package regardless of whether or not the respondent participated.

Collaboration with the survey company, Ipsos Reid, was important to ensure that these tools were effectively employed to hit the nested sampling targets. Ipsos Reid provided almost daily updates to establish up-to-date projections for the final returns. During data collection, certain cells were identified as in danger of being under represented in the final sample. Through timely collaboration with Ipsos Reid and senior management, an additional sampling wave was added for the offline recruitment, which utilized the full spectrum of high-level incentives – the $20 completion incentive, advanced letter, and token $2 incentive. As a result, we were able to hit (and exceed) all nested targets, and ensure that no zero cells would impede the calibration.

The other main innovation of the 2013 MOP was to leverage the existing CFM survey, via the survey instrument and the method of data collection. By comparing the 2009 MOP with the CFM we discovered that the surveys contain some overlapping content. This led to a sub-sampling of past CFM respondents (with the sampling frame provided by Ipsos Reid) in addition to maintaining the Online and Offline panels employed for the 2009 MOP. For certain aspects of the questionnaire – specifically/for example, information on a respondent’s bank account(s) and credit card(s) – the respondent had already provided the desired information in the CFM; this allowed us to shorten the 2013 MOP questionnaire, which likely had a positive impact on response rates.

Other topics in the 2009 survey, such as cash usage, were maintained in the 2013 MOP but made directly comparable to questions in the CFM. This sub-sampling approach proved very successful in recruiting respondents, with a response rate of over 50

percent. Furthermore it provides an external benchmark with which to compare measures of consumer cash holdings/usage.

Section 3: Survey Weighting

Data collection methods for large scale surveys are almost always biased. Two common explanations are that the sampler does not have access to a full sampling frame, and that the sampler does not have full control over the sample selection procedure. However, sample calibration/balancing is a post-stratification procedure that can be used to re-weight sample data so that it is more representative of the target population (Sarndal, 2007). Such sample weighting can facilitate more accurate estimates of population unknowns.

The calibration procedure leverages the availability of national-level counts of auxiliary/demographic information, so as to balance the sample to these counts. For example, in our calibration analysis we make use of national-level counts based on the 2011 National Household Survey (NHS) and the 2012 Canadian Internet Use Survey (CIUS) for a variety of demographic variables. Calibration over such information offers users the ability to reduce non-response and non-coverage bias effects. Consequently, the resulting estimators should be less biased than those based on the un-weighted data (Kish, 1992) when calibration is used.

In our calibration analysis we follow a series of steps to arrive at a suitable set of calibration weights. We summarize the process with the flowchart in Figure 1 and provide a detailed breakdown below. We then conclude the section with a reflection on non-response weights.

Subsection 3.1 Outline of calibration analysis.

Stage A: We first consider a set of potential calibration variables that include both demographic and technological-oriented variables. These are chosen based on their conjectured relationship with important survey questions. A round of data editing and cleaning is undertaken, and imputation of missing values in the calibration variables is achieved with the aid R package called “mice” package (van Buuren, 2012).

As recruitment came from three separate market access panels, a comparative analysis is performed using the Epps-Singleton test for homogeneity on the demographic variables. The three subsamples are found to be fairly homogenous in composition and we therefore concatenate them into one final sample for the calibration analysis.

A correlation analysis over the calibration variables is conducted, in order to determine any potential collinear variables to eliminate from the analysis. As the calibration variables are classified as categorical or ordinal variables, we use the polychoric correlation measure (Drasgow, 2004) with the aid of the R package “polycor” (Fox, 2010) to compute the correlations.

Stage B: The calibration variables are found to be only mildly to moderately correlated, and hence no variables are eliminated from the analysis. With respect to nesting/pairing calibration variables, we pair the two most correlated variables with each other, as well

as the gender variable with several other calibration variables. Nesting allows us to avoid sparse cells. For example, the gender variable is often paired with other calibration variables because it is a binary variable. This pairing can avoid small cell counts while accounting for disagreements between the joint sample and national distributions.

A range of calibration techniques can be used; see Deville et al. (1993) for a mathematically detailed account of some commonly used procedures. We select/choose the raking and generalized regression (GREG) procedures as these are popular methods among both national statistical agencies and academics; see Sarndal (2007) and the R Survey package by Lumley (2010).

Two sets of proposed initial weights for the raking method are based on both a simple random design and a stratified sampling design based on several key demographics. The correlation of the two sets of generated weights, when using the raking algorithm and the full list of calibration variables, is high. Hence, we conclude that the final weights are likely to be insensitive to the choice of the initial weights, and therefore base them on the simple random sampling design.

Stage C: The raking and GREG procedures are evaluated based on the range of weights they generate. With numerous combinations of calibration variables, the GREG procedure gives rise to a number of negative weights. Furthermore, since raking is a popular post-stratification method that is widely used in the statistics profession, we choose to use the raking method.

Subsection 3.2 Non-response weights.

The issue of non-response is a common concern among survey practitioners; see Kish, (1992) for more details. Typically, non-response will bias and inflate the variance of the estimates and increase survey costs (as follow-ups can be expensive). However, several procedures can be used to account for such non-random, non-response issues.

In some cases, typically when the quantity of non-response is small, imputation can be prescribed to resolve such issues. However, determining a suitable model and imputation strategy can be resource intensive and computationally expensive, as surveys are comprised of many questions. Instead, the original calibration weights can be adjusted to compensate for the non-response issues.

Further, when responses are missing completely at random (MCAR) (Rubin, 1976) and non-response counts are small, one approach is to base estimation on a rescaling of the original calibration weights for those who have responded. However, the non-response pattern will usually be such that it cannot be viewed as MCAR. A common approach in such a case is to assume that the responses are missing at random (MAR) (Rubin, 1976). In other words, the responses are MCAR within strata/classes of the survey respondents. This assumption allows the computing of estimates for the response probabilities, possibly based on a logistic regression or propensity scoring model, and appending them to the original calibration weights. Essentially, a respondent i will receive an original calibration weight 1/Pi and an estimated response weight rh

depending on their stratum membership h. Their corresponding weight, for the survey question under consideration, would then bew i=1/(Pi rh).

The aforementioned approach comes with precautions. As noted by Sarndal (2007), an inherited bias in the procedure is likely when estimating the actual response probabilities. Hence, a high level of caution should be exercised when positing/exploring response model(s). However, rigorous methods have been developed to approximate potential non-response bias as a function of the responses and the national-level covariate information (Sarndal and Lundstrom, 2008); with such methods, suitable calibration variables can be chosen to reduce the non-response bias.

Subsection 3.3 Non-probability samples

In an empirical setting, non-probability sampling is a common concern, as latent factors may influence selection probabilities. (Consider the discussion above on nonresponse). When estimation is based on non-probability samples, the population attributes are typically assumed to be distributed somewhat evenly so that the sample weighting (through a posited probability sampling design and sample calibration) can still provide accurate results.

Web-based studies are gaining popularity due to their convenience and efficiency in obtaining data. Almost all of these surveys rely on convenience or volunteer sampling, both of which are non-probability sampling designs. Apart from legally mandated surveys such as national census most surveys rely on volunteer sampling. Such limitations have been acknowledged in the literature, and the demand is increasing for calibration methods to improve the accuracy of results based on such samples; see Schonlau, et al., (2007) and Disogra et al. (2011).

Internet and mobile usage information from nationwide surveys presents much potential for use in calibration analyses. We (therefore?) exploit such demographic information from the 2012 CIUS for our calibration analysis, namely ownership of a mobile device and online payment activity, as some of the MOP survey questions are oriented to/relate to mobile means-of-payment. More details about the/this weighting procedure are available in Vincent (2014).

Section 4: Variance Estimation

Since the 2013 MOP employs stratified random sampling (Section 2), and survey weights are applied to ensure a representative sample (Section 3), variance estimation should take both the sampling design and calibration procedure into account. Heuristically speaking, the variance depends on the weighting procedure, not just on the numerical values of the weights. However in most payment surveys, variances are usually calculated by taking the calibrated weights as fixed values, thereby biasing the variance estimates. In order to capture the randomness from both sampling design and weight calibration, we propose a resampling method, specifically the bootstrap replicate survey weights (BRSW).

For example, if the weight calibration is raked over the external variables x, then the estimated variance of the population-based weighted y variable is:

V̂ar ( y )= ∑k , l∈S

πkl−π k π lπkl

yk−xk bπk

yl−x lbπ l

where πkis the k-th unit’s inclusion probability in the sample, πkl is the pairwise inclusion probability of the k-th and l-th units, and the parameter b is the OLS estimate from regressing y on x. The first term is proportional to the sampled covariance for the k-th and l-th units. The second and third terms form the fitted residuals.

Our reasons for choosing resampling over the linearization method are as follows. First, most software packages use linearization as if the weights were fixed, which does not allow for model-based information (b), but rather uses w k or the calibrated weight in the denominator. Although the correct linearization method is suggested by Lu and Gelman (2003), their method can be difficult to implement when sample weights are complicated functions of sample sizes within strata. (Only under simple random sampling does w kl have a straightforward formula depending on w k and w l.) Second, the linearization estimators use the initial weights. Hence this requires the survey dataset to include both the calibrated weights and the base weights, which may confuse users. Third, the end user of the data must be given the set of strata variables, which may not be possible if confidential variables are used in calibration. These complications cause resampling to be the more popular method for estimating variances when calibrated weights are used (Shao 1996, Kolenikov 2010).

Among the many resampling methods for a complex survey, we choose to use BRSW. We prefer the bootstrap to either the jackknife or balanced repeated resampling (BRR), because the jackknife is inconsistent for non-smooth functions (e.g. the median estimate), and BRR is more suitable for a stratified clustered sampling design, which was not used in our 2013 MOP Survey. As for choosing to use replicate survey weights, we do so to protect the privacy of survey respondents (no strata information will be provided), and because replicate weights incorporate information about corrections for non-response bias (weights are re-adjusted for each replicate).

The construction of the BRSW involves first re-creating the sample in each replicate and then adjusting the associated calibrated weights. For example, if a unit from a replicate is not sampled, a zero weight is assigned to it, and then the weights of the other units in the same stratum are expanded to compensate. In the next step, the weight calibration is applied to each of these replicate sets of weights. These two steps generate the bootstrap replicate weights. We use the bsweights package in Stata to implement this method (Section 3.2 in Chen and Shen, 2014).

Table 1 shows mean and variance computations from the 2013 MOP and 2013 CFM data. As a basis of comparison, we decompose the 2013 MOP Survey sample statistics into online, offline and CFM subsamples. Overall, the cash holdings from the weighted 2013 MOP Total mean of 83.68 is lower than 2013 CFM survey of 94.67. Part of this finding is driven by the MOP Online mean cash holding which is about 80.06 and constitutes about about one-third of the sample. The MOP offline mean of 91.10 is closer to the 2013 CFM mean. Part of this reason is that it is drawn from the offline access panel. The CFM subsample respondents provide a point of comparison as they are participants of both the 2013 MOP and 2013 CFM. The average cash holdings for this overlap set of respondents is 83.41 (2013 MOP) and 89.82 (2013 CFM). Overall, the 2013 MOP

estimate of cash holdings is lower than the 2013 CFM. Part of this discrepancy maybe due to the timing of the survey as the 2013 MOP was conducted in October-November 2013 while the 2013 CFM was sampled from a subset of January-August 2013 participants.

We compute the variance based on linearization and the BRSW methodology. In the second row, the variances are calculated by linearization without considering the weighting procedure, while in the third row the variances are calculated by BRSW. The BRSW variances are much smaller than those calculated by linearization, because the resampling method takes into account the weight calibration procedure, which is applied after the sample is collected.

Note that the paper-based (MOP CFM and MOP offline) BRSW variances improve about 20-25 percent, while the MOP online BRSW variance improves by over 50 percent. A plausible explanation would be three-fold. First, the cash-on-hand variable is more correlated to the raking variables (e.g. online payment) for the online respondents than for the paper-based ones. By computing the R squared for different subsamples, we find that the R squared from the online panel is highest at 0.0179, compared to 0.0048 and 0.0087 for the offline and CFM panels, respectively. Second, since the sample variance also depends on the sample size, the relatively larger sample size for the online subsample could also drive the sizable improvement.

Table 1: Mean and Variance Estimates for Average Cash on Hand

2013 MOP 2013 CFMRaw Weighted Weighted

CFM CFMTota

lTota

lOnlin

eOfflin

eSubsampl

eSubsampl

e Total

Mean87.6

283.6

8 80.06 91.10 83.41 89.82 94.67

VarLin15.8

413.2

7 48.51 50.66 25.81 7.60 4.40VarBRSW N/A 7.76 17.90 41.10 20.09 N/A N/A

R-squared N/A0.00

9 0.018 0.005 0.009 N/A N/AObservations

3413

3413 1294 679 1440 1440

12280

Notes: Average cash on hand is measured in Canadian dollars. Statistics are based on respondents reporting positive figures (i.e. excluding zero responses). VarLin is the linearized variance and VarBRSW is the variance based on bootstrap replicate survey weights. R-squared is the goodness of fit from the raking variables regression.

This exercise demonstrates that if we ignore the random fluctuations due to the weight calibration in the variance estimation, the resulting variances will tend to be conservative (too large) and the confidence intervals too long, their coverage exceeding the pre-specified nominal level.

Section 5: Conclusion/Discussion

Our experience from the 2013 Method-of-Payments Survey dictates/highlights that survey weighting should not be viewed as a panacea to the ills of a poorly designed survey. Rather, a well-thought-out survey design and preparation will go a long way to ensuring that the survey is representative. Namely, we suggest that survey teams:

1) Use mixed-mode survey methods so that each mode can be used to validate and verify the others. In addition, various trusted external data should be brought in to help calibrate the survey collection methods.

2) Use the methods espoused by Dillman (2007) to induce higher response rates.3) Work closely with the data collection agency to ensure that the objectives are laid

out in advance. Closely monitor all fieldwork and collaborate to head off any difficulties.

4) Conduct post-stratification using a variety of methods, but ensure that the methods are robust and make sense. Again, use a variety of external data to conduct post-stratification.

5) Compute variance estimation using resampling methods, as they result in variance reduction as well as provide a way to anonymize the sampling design.

Overall, we found that the 2013 MOP the average cash holdings is about 84 dollars which is about 10 dollars less than what was found in the 2013 CFM. Understanding the source of difference in estimates is left for future work.

References

Carlos Arango, Kim Huynh, Leonard Sabetti, 2011. "How Do You Pay? The Role of Incentives at the Point-of-Sale," Working Papers 11-23, Bank of Canada.

Carlos Arango, Angelika Welte, 2012."The Bank of Canada’s 2009 Method-of-Payments Survey: Methodology and Key Results," Discussion Papers 12-6, Bank of Canada.

Chen, H. and Shen, R. (2014), Variance Estimation for Survey-Weighted Data using Resampling Methods: 2013 Method-of-Payment Survey Questionnaire. Technical Report.

Deville, J. C., Sarndal, C. E., and Sautory, O. (1993). Generalized Raking Procedures inSurvey Sampling. Journal of the American Statistical Association 88, 1013-1020.

Dillman, D. Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method, 3rd ed. (2009)

Disgora, C., Cobb, C., Chan, E., and Dennis, M.J. (2011). Calibrating non-probability internet samples with probability samples using early adopter characteristics. Proceedings of the Section on Survey Research Methods, JSM.

Drasgow, F. (2004). Polychoric and Polyserial Correlation. John Wiley and Sons, Inc.

Fox, J. (2010). Polycor: Polychoric and Polyserial Correlations. R package version 0.7-8.

Henry, C, Huynh, C. and Shen, Q. Rallye (2014). 2013 Method-of-Payments Survey Report.

Kish, L. (1992). Weighting for unequal pi. Journal of Official Statistics 8, 183-200.

Kolenikov, S. (2010). Resampling variance estimation for complex surveydata. The Stata journal 10 (2):165–199.

Lu, H. and Gelman, A. (2003). A method for estimating design-based sampling variances for surveys with weighting, post stratification, and raking. Journal of Official Statistics 19 (2):133–151.

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. John Wiley & Sons, Ltd.

Rubin, D. (1976). Inference and missing data. Biometrika, 63 p. 581-592.

Sarndal, C.-E. (2007). The calibration approach in survey theory and practice. Survey Methodology 33, 99-119.

Sarndal, C.-E and Lundstrom, S. (2008). Assessing auxiliary vectors for control of non-response bias in the calibration estimator. Journal of Official Statistics, 24 (2), p. 167-191.

Schonlau, M., Van Soest, A., & Kapteyn, A. (2007). Are 'webographic' or attitudinal questions useful for adjusting estimates from web surveys using propensity scoring. RAND Corporation.

Shao, J. (1996). Resampling methods in sample surveys (with discussion). Statistics37:203–254.

van Buuren, S. (2012). Flexible Imputation of Missing Data. Chapman and Hall/CRC Press.

Vincent, Kyle (2014). 2013 Method-Of-Payments Survey Calibration Manual. Technical Report.

Figure 1: Survey Weighting Workflow