what is evidence-based policy?
TRANSCRIPT
Lari HokkanenBreakfast Seminar
2.3.2017
What isevidence-based policy?
(I don’t know.)
Not that simple, sorry
A brief survey of the field• Technocrats: will save the world (somehow)• Political theorists: technocrats do not understand
what politics is (power! values! interests! disagreements!)
• Politicians: well yeah.. umm.. kinda good idea but…• Civil servants: We have a lot to do already• Economists (“randomistas”): let’s do RCTs and all is
fine• Philosophers of science: Umm.. The metaphysics of
causality...• Scientists in general: we don’t have time to
communicate our results and twitter is so hard
They all get something right!• Politics will get better with the use of evidence• Politics is mostly about interests and values• Politicians do their job fast and with limited
information• Civil servants are busy • RCTs are superb evidence (if interpreted right)• Talk about causality is important• It’s hard to communicate the results of studies to
political actors
UtopiaEvidence-based policy: Scientists produce quality evidence → Politicians use it wisely as a basis for public policy
Reality?Policy-based evidence: Scientists produce quality evidence → Politicians cherry pick results to support their political beliefs
So we get something like this
How policymakers think• Politics isn’t science: evidence is only one thing that affects
political decisions – several other principles of good policymaking: consensus, public
opinion• Cognitive shortcuts: rational (prioritizing certain kinds of information) and
irrational (gut feelings, deeply held beliefs, feelings)• Have to do decisions fastly; have to try to evaluate things without deep
understanding• Evidence can be against values → values win• Evidence is always interpreted through one’s own ideological lens• Policy process is complex: there is not only one point where to inject
evidence
What counts as good evidence?Hierarchy of evidence
- Comes from evidence-based medicine: idea is that RCTs are best to produce causal claims
- But not that simple: RCTs are good but most studies in social sciences observational studies which are also good
Getting evidence: why do we need experiments (RCTs)?
- World is becoming harder place to understand without experimenting- Social problems = Wicked problems- Civil servants might not know how reality works- (Even comparative) evidence on policies usually context dependent→ it’s good to try them in our own context
The golden standard(?): randomized controlled trial
A methodological word of caution
- RCTs are a bit overhyped but still really good- The results from RCTs are context dependent
- contextual differences will shape intervention outcomes - It’s a big discussion about what we can infer from them RCTs: need to think
about the mechanisms behind results (not a problem)- Other sources of evidence are also needed because they are black box -explanations (mechanisms)
- RCTs good at internal validity and generating causal claims, but generalizing is not easy when we talk about complex issues (as opposed to examples from medicine)
“[In education] ‘what works?’, which is what politicians would love to know about, is the not the right question, because [in education] everything works somewhere and nothing works everywhere. The interesting question is ‘under what conditions does this work?’“
The nature of evidence• Evidence is evidence of something for someone: pluralism• Evidence is uncertain and context-dependent• There are no common standards for evaluating evidence - what counts as
good evidence? And for whom?• Evidence-based policy need (mostly) quantative causal knowledge;
unfortunately it’s hard to infer (universal) causal mechanisms even from the best studies
• Aaann.. Evidence-producing science is also value-based (to some extent) → plurality of views within disciplines
• If scientists cannot decide what counts as good evidence, how can politicians?
Beyond methods: different disciplines offer conflicting evidence
- Most policies are still designed based on assumptions about rational action (economics)
- e.g. taxing, monetary incentives in social policy- Incentives are good to keep in mind but we are also quite irrational - What if we would look empirically how people act and design policies based on that?- Problem: many disciplines with good ideas contradicting with each other → how to do policy recommendations?
My (very preliminary) thoughts
• Evidence-based policy is more ex tempore testing of policies (experimental governance) than trying to get meta-analyses into political processes
– Like this: There is emerging evidence on the effects of policy x (might work, or not) → Let’s try with RCT & other methods in our context to see if it works here
– Limited testing is better than not testing (finnish basic income experiment)– Fostering the culture of experimentation is thus important– Evidence should be fitted to politicians’ mindsets: workshops, scenario
building, policy briefs