Non-response weighting at NatCen/ScotCen

- a brief (and biased) history

Susan Purdon

Survey Methods Unit, NatCen

The case against

• 10 years ago non-response weighting not the norm

Reasons:

• Response rates pretty high on govt sponsored surveys

• Beyond age and sex, not many control totals around

The case against

• Many surveys had long histories: weighting would introduce discontinuities

• Non-response is subjective and cosmetic!

• No two statisticians would create the same set of non-response weights. Unscientific.

• Weighting makes analysis more complex and error prone.

The counter-arguments

• Non-response rates have now grown (c. 1%/year)• The under-representation of certain groups is a

constant and clearly biasing• Trends in response rates undermine the argument

against introducing discontinuities• ONS have concluded that all national statistics

should be calculated on a consistent basis (same age-sex-region distn)

The implications

• Most surveys now come with non-response weights

• New industry of calculating weights for old surveys

• Not too comfortable a position (what if new method around the corner?)

How it’s done on gen pop surveys

• Approach kept as simple as possible. Adjust (standardise) for age-sex; no major attempt to eliminate other biases.

• Usual approach = calibration weighting where

(a) adjust to national age-sex totals; but

(b) give all household members same weight.

Calibration weighting

• Biases observed for individuals, but non-response is a household process (yes/no determined by whoever answers the door);

• Implies that probability of response depends upon who you live with (e.g. young men living with parents v. other young men)

• So only under-represent young men in certain types of households

Calibration weighting

• Calibration weighting attaches a probability of selection per household that ‘explains’ difference between survey and population age/sex.

• Assumes that non-response is not an independent process

More elaborate approaches

• Non-response hierarchies internal to the survey

• Panel attrition

• Studies with informative sampling frames

Suggestions for analysts

• Non-response weights are subjective. You don’t have to trust them.

• Check that observed differences are not attributable to weighting

• If you think the survey organisation has missed a trick then tell them!