WERS the Validity: What’s The Current State of Play?Dr Andrew R Timming

Reader in Management

University of St Andrews

Preface

• WERS is the single greatest source of data on work and employment in the history of not just the United Kingdom, but also the world

• No other dataset comes close

• But scientific advancement depends on critique

Background

• From WIRS (1980, 1984, 1990) to WERS (1998, 2004, 2011)

• WERS 2011: (i) management questionnaire, (ii) financial performance questionnaire, (iii) worker representative questionnaire, (iv) survey of employees and (v) employee profile questionnaire

• 2,680 face-to-face interviews with managers; 1,002 interviews with employee reps; 21,981 respondents completed the survey of employees

Structure of Presentation

• Look at my top 10 “best practices” for survey design

• Evaluate how the WERS instruments stand in relation to those top 10 practices

• Focus mainly on survey of employees

• Summarize key points to draw from the presentation

Tip 1

• Surveys should be relevant and timely

• Relevant according to whose judgement?

• Evidence of timeliness is clear: workplaces in the shadow of recession

• But we still have to ask how much value is added by some of the questions.

Timeliness

Relevance

Tip 2

• The layout should be clear, not cluttered

• If the survey is too spread out, it reduces response rate; too cluttered, it reduces the quality of the data

• On this point, the Survey of Employees gets an A+

• Nice aesthetics, clear instructions, inviting, good boundaries and delineation between items, clear flow

Clear Layout

Tip 3

• Strike a balance on length of the survey

• MQ response rate=46.3%; WR response rate=63.9%; Survey of Employees=54.3%; financial performance questionnaire= 31.8%

• Face-to-face administration is usually much higher than postal, but in WERS 2011 we see the opposite

• Was the length of the instruments driving away respondents?

Tip 4

• Items should be short and simple

• Principle of parsimony

• Express what you need to express in as few words as possible and as clearly as possible

• On this point, WERS does a pretty good job, but not perfect . . .

Tip 5

• Avoid leading questions

• ‘Do you agree that Scotland should be an independent country?’ vs ‘Should Scotland be an independent country?’

• How we ask a question can result an an expected response

Overall, how satisfied or dissatisfied are you with …

Tip 6

• Response categories should be exhaustive and mutually exclusive

• Exhaustive: all possibly choices are listed

• Mutually exclusive: no two choices overlap

• WERS is generally good on this point

Mutually exclusive?

Tip 7

• Respondents should be knowledgeable about the topic or topics.

• Albanian foreign policy towards China . . .

• No problem with survey of employees, but there may be concerns with management and employee rep questionnaires

Are workers’ reps qualified to speculate?

Can a manager really know the answer to this question?

Tip 8

• Items must not be double-barreled

• ‘Double-barreled’ implies that a single item is asking two questions in one

• Double-barreled items lead to serious questions about whether we’re measuring what we say we’re measuring

• On this point, WERS could be improved

Double-barreled items

Tip 9

• Complex constructs require multiple items

• Multiple items allow for the construction of latent variables

• On this point, WERS does a really good job

Tip 10

• Survey instruments should always be piloted and reformed according to results

• Obviously, the fact that there are different iterations of the survey implies sufficient piloting

• But on the question of reform, we need to continue to refine the instrument, even though this may lead to a break in continuity from previous versions

Summary

• On the whole, WERS 2011 is a fantastic resource and a very well designed set of instruments . . . but it’s not perfect

• What is included and excluded from surveys is a political choice and influenced by power

• Do we tolerate imperfection for the sake of continuity?

Counter-Arguments

• ‘We shouldn’t change or reduce the amount of data gathered because they are always relevant to someone’.

• ‘Other surveys have used these same measures, so why shouldn’t we as well’?

• ‘Our response rates are, in fact, much better than other comparable surveys’.

• ‘Effecting changes now will make it difficult to do comparable longitudinal analyses’.

• ‘You’ve cherry picked a few examples to make the whole survey look bad’.

• ‘Making changes would cost too much money without an effective return’.