Nudge Meets RCT: Evidence-based

Manipulation for the Public Good?

 Martyn Hammersley

The Open University

Workshop on:

Bio-Social Methods for a Vitalist Social Science

July 2013

The Nudge Unit‘The Behavioural Insights Team, often called

the “Nudge Unit”, applies insights from academic research in behavioural economics and psychology to public policy and services.’

One of its ‘responsibilities’ is listed as:

‘championing scientific methodology to bring greater rigour to policy evaluation.’

This, of course, is where randomised controlled trials (RCTs) come in.

(https://www.gov.uk/government/organisations/behavioural-insights-team)

Nudge Nudge, Wink Wink

‘The team's work has already led to an extra £200m in income tax being collected by telling late payers on tax forms that most people in their towns had already paid their tax. This increased payment rates by 15 percentage points. Now the team is applying lessons learned from increasing voter turnout in US elections to get Essex jobseekers into work 15%-20% quicker than traditional methods’ (Benjamin 5.2.13)

The appliance of behavioural science?

• What is proposed here is the application of academic knowledge, both theoretical and methodological, to policymaking.

• But what is being applied is only a narrow range of academic knowledge. And it aims at challenging the already established approaches of other disciplines: political science, sociology, social policy research, and evaluation research.

• It forms part of a broader process of territorial colonisation by psychology and economics.

Assumptions about the relationship between research and policymaking• ‘Libertarian paternalism’: A focus on nudging

policies, rather than on coercion, rational persuasion, rhetoric and spin, or laissez-faire.

• Policymaking is or should be ‘evidence-based’

• The task of research is to determine ‘what works’, this indicating relatively conclusively what policies should, and should not, be pursued.

• It is claimed that the only reliable means for determining this is the RCT.

The opening pitch of ‘Test, Learn, Adapt’

• ‘Randomised controlled trials (RCTs) are the best way of determining whether a policy is working. […]’

• ‘RCTs are not routinely used to test the effectiveness of public policy interventions in the UK. We think that they should be’ (Haynes et al 2012:4)

Focus on ‘What Works?’Effectiveness and efficiency are important, but

they are not the whole of policymaking. Equally, if not more, important, both in principle and in practice, are:

• The identification, prioritising, and diagnosis of policy problems

• Evaluating policy proposals as regards ethics, side-effects, and the distribution of benefits and costs among those involved.

• Politics in the narrow sense: how will this play with the Daily Mail? Will this wrong-foot (nudge?) the Opposition?

Nudging people’s choices• Origins in techniques of ‘demand management’

and HR employed by commercial organisations, often informed by psychology.

• ‘We argue for self-conscious efforts, by institutions in the private sector and also by government, to steer people’s choices in directions that will improve their lives’. The aim is ‘to influence choices in a way that will make choosers better off, as judged by themselves’ (Thaler and Sunstein 2009:5)

• Some Nudge Unit projects aim at the common good, rather than that of the nudged individuals.

Problems with ‘the common good’

• Is there any such thing: common to whom, and good by what criteria? Some sense can be made of this notion, but it is always going to be a contentious matter.

• Does the state currently operate so as to serve the ‘common good’? Are there preconditions that must be met if it is even to approximate this mode of functioning?

• Can democracy deliver this? Do we live in a democracy?

Research-Based Policymaking?• Evidence is not all that is required in

policymaking: it needs to be a process of phronesis, in other words wise judgment.

• Research is not the only source of relevant evidence.

• RCTs are not the only source of valuable research evidence

• The provision of evidence about effectiveness or efficiency is not the only contribution that social science can make to policymaking (Hammersley 2013)

Making the case for RCTs

‘What makes RCTs different from other types of evaluation is the introduction of a randomly assigned control group, which enables you to compare the effectiveness of a new intervention against what would have happened if you had changed nothing’ (Haynes et al 2012:4).

As I will show, this is at best only partly true. Moreover, RCTs are not applicable to all policies: these must involve specific and standardisable ‘treatments’.

A germ of truth‘The introduction of a control group eliminates

[…] biases that normally complicate the evaluation process […]: if you introduce a new “back to work” scheme, how will you know whether those receiving […] extra support might not have found a job anyway?’ (p4)

By randomly allocating units to treatment and control groups, the chances of background differences between the groups affecting the outcome are minimised, and we can estimate statistically the chances that these background factors produced any outcome difference.

But• Randomisation does not eliminate all possibility

of background factors affecting the outcome, nor does it deal with all potential sources of error: measurement errors, expectation effects.

• While the internal validity of RCTs is generally relatively high, there are problems with external validity: the ‘treatment’ rolled out is not always the same as in the trial, and the people and situations often differ in causally significant ways (Cartwright and Hardie 2012).

• The results of RCTs will often have a short shelf-life, as may the effects of policy nudges.

The positioning of the ‘subject’

• An(other) attack on the rational model: homo economicus is a myth. Perhaps we knew that?

• The behaviour of subjects is largely a product of ‘the automatic system’ (Thaler and Sunstein 2009): they know what is good for them but routinely fail to act in ways that are effective in achieving it.

• The task is to nudge the defective citizen into behaving in ways that are best for everyone.

The positioning of policymakers

• They know what is in the best interests of ‘subjects’.

• They are motivated to ‘nudge’ subjects in ways that serve those interests.

• They can be provided with technical means that have a demonstrable capacity to achieve this.

The positioning of the behavioural scientist

‘The unit tries to nudge people into leading better lives, and save the rest of us a fortune. It is politics done like science’ (Benedictus 2013)

Ellieban: ‘I confess to being a bit of a fangirl, but the behavioural insights team is non-ideological and non-partisan, and exists purely to help government (any government) make policy that actually works. […] They are scientists, not politicians or propagandists.’

Conclusion

• The Behavioural Insights Unit combines commitment to government-by-nudge with a commitment to evidence-based policymaking. It promotes itself by missionary rhetoric that exaggerates what both nudging and RCTs can do.

• It misleadingly positions both itself and policymakers as politically neutral, and it promotes a narrow and overly direct model of the general relationship between research and policymaking.

Conclusion concluded• There is undoubtedly a role for nudging

policies, and for RCTs to test them, even if this is quite limited.

• Some of us may want to believe that there is a clearly defined and easily identifiable common good, and that we are ruled by a benevolent and competent regime that can realise this good simply by nudging us to do what we know is in our own best interests. However, unfortunately, this fairy tale, like others, is false.

References• Benedictus, L. (2013) ‘The nudge unit: has it worked so far?’, The

Guardian, Thurs 2 May. Available at: http://www.guardian.co.uk/politics/2013/may/02/nudge-unit-has-it-worked

• Benjamin, A. (2013) ‘David Halpern: We try to avoid legislation and ordering’, The Guardian, 5.2.13. Available at: http://www.guardian.co.uk/society/2013/feb/05/david-halpern-government-nudge-unit

• Ellieban ‘Comment on Benjamin’, see above.

• Cartwright, N. and Hardie, J. (2012) Evidence-Based Policy, Oxford, Oxford University Press.

• Hammersley, M. (2013) The Myth of Research-Based Policy and Practice, London, Sage.

• Haynes, L., Service, O., Goldacre, B., and Torgerson, D. (2012) Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials, London, Behavioural Insights Team, Cabinet Office, UK Government.

• Thaler, R. and Sunstein, C. (2009) Nudge, London, Penguin.