Qualitative methodology in Behavioural Studies for

EU Policy-Making

Workshop-reportJune 2016

European Commission

Joint Research Centre

Institute for Prospective Technological Studies

Contact information

Rene van Bavel

Address: JRC-IPTS, Edificio Expo Calle Inca Garcilaso 3, 41092 Seville, Spain

E-mail: Rene.VAN-BAVEL@ec.europa.eu

Legal Notice

The opinion expressed in this report does not imply a policy position of the European Commission.

Neither the European Commission nor any person acting on behalf of the Commission is responsible

for the use which might be made of this report

1

QUALITATIVE METHODOLOGY IN BEHAVIOURAL STUDIES FOR EU POLICY-MAKING

Main points from the workshop

1. The twin objectives of the workshop were to (a) get to know each other and address

issues relating to the Framework Contract for the Provision of Behavioural Studies

and (b) discuss the role of qualitative methodologies for EU policy-making. These

minutes present a structured summary of the main points relating to the latter

objective.

2. These points must be set against the backdrop of research for policy support in the

EU. In other words, what are the relevant issues to consider given that EU policy-

makers expect robust research based on diverse perspectives (which decreases

biased perceptions, provides a reality check, and promotes buy-in from participating

stakeholders)? [SP]1

Why conduct qualitative research?

3. The research industry is experiencing profound change, driven by technology and

digital innovations. Two of the major trends in the research consultancy world are

the rise of big data and the accompanying rise in the importance of strategic

qualitative research. These two trends are highly complementary – we need

qualitative depth to turn ‘big data’ into ‘smart data’. [MF]

4. Good qualitative research has a role at all stages of the research process, given the

importance of context and its major impact on behaviour. The context needs to be

investigated, but it is difficult to assess direct with quantitative methods. To

properly understand the role of context requires a mix of qualitative techniques

such as observation, ethnography (in its various forms), cognitive interviewing, even

desk research. [MF]

5. Qualitative methods can be used to guide quantitative research, especially among

researchers whose main area of competence is quantitative. They see the process as

a pyramid, with qualitative at the bottom and methods like experiments at the top.

[TM] Qualitative methods help mirror the real environment, stripping out the

irrelevant stuff. This makes them particularly relevant for EU behavioural studies,

which are quite bespoke. [CD]

1 Initials in square brackets refer to those participants who made, contributed to, or supported the point (broadly

speaking). See list of participants at the end of the document.

2

6. However, there is also a role for qualitative methods after quantitative methods. It

can help understand patterns in the data that are unexpected or ambiguous. In one

study on social media, for example, odd quantitative findings emerged about people

clicking on a particular link on a webpage. Qualitative research subsequently

uncovered that the patterns of behaviour that had been observed were in fact due to

Facebook's algorithms, not necessarily to people's preferences. [GV] Qualitative

research can also be used to help explain the why – the reasons behind peoples’

views and opinions as expressed in quantitative surveys. It can also be used as a risk

hedge (e.g. "we didn't pick something up in the experiment, but we found in

subsequent qualitative research that…"). [CD]

7. Qualitative methods are also excellent as a communications strategy, and can help

engage the audience and bring issues to life. For example, a quote from qualitative

research can capture the policy-maker's attention in a way that statistics cannot.

These verbatims can be used as a complement to cold numbers or complex models

and help make sense of them. [TM]

8. Qualitative methods may be useful when focusing on specific populations that would

otherwise be hard to target in a study, such as marginalised sectors of the

population, or when researching particularly sensitive or controversial topics. With

quantitative surveys you might get politically correct or socially desirable responses

but won’t find out what people really think or how they really behave… with

qualitative you’ve at least got a chance to get under the surface. [CD]

9. They can also help recognise who the participant identifies with and pinpoint their

motivations (using e.g. collage techniques), leading to more effective policy

interventions. Take the example of campaign against drunk driving: a teenager calls

her dad who is at home and asks him to pick her up because she missed last bus. But

the father has had a few beers at home. The message reads: “Would you like to be

dead with your girl or would you rather have your daughter mad at you because you

weren’t able to give her a ride?” Such a message is likely to hit home more than one

picturing young, reckless drivers. [CRS]

10. 'Immature' topics of research may indeed be good candidates for qualitative

research. For example, one study on illegal downloads was quite explorative in

nature; the contractor was not looking for anything particular. They started with

qualitative research, with one focus group in each EU Member State. They did this

first because there was not a lot of available research to guide the study. This

allowed them to understand the drivers and barriers of the investigated behaviour,

i.e., what motivates and what could discourage young people to access illegal content

online and buy counterfeit products online. [AD]

3

Which qualitative methods are used in behavioural studies?

11. Qualitative research has been very much affected by technological changes (e.g.,

mobile technology, social media, online communities, wearable devices and new

touchpoints) which open new options for researchers and participants to interact.

Qualitative data is no longer limited to focus groups, interviews, document analysis

and participant observation, although these traditional methods are still important.

[MF]

12. One of the methods used is to have a "shop-along", where researchers follow

consumers through a supermarket and listen to their thoughts aloud. Thinking aloud

is different to self-reporting (addressed further on) because participants are not put

on the spot. [TM] They use it to characterise a phenomenon as much as possible, to

"cover all the bases" or to "seek the variability", not to reach conclusions. It is about

being open to surprises. [GV]

13. Journey maps are also qualitative alternatives. In a study on illegal downloading,

participants drew a tree of their decision-making process when searching for

information on the internet. Journey maps can vary quite significantly depending on

the product (e.g. films vs. music) and on participants' own preferences on whether

or not to use illegal content. They can also vary from country to country depending

on the culture regarding illegal downloads. [AD]

14. The Zaltman metaphor elicitation technique2 can be used to tap into what goes on in

people's minds, but which they might be unaware of. [CRS]

15. The question emerged: is qualitative vs. quantitative a useful distinction? Some

workshop participants felt it was outdated, thanks to the changes brought about by

the digital revolution [GV]. Take the example of eye-tracking, which allows exploring

what parts of a letter of website people pay attention to, or mystery shopping. Are

these qualitative methods?

16. For example, insights from decision-making are used to perfect online algorithms

(the kind used by Google of AirBNB). Qualitative research has helped develop these

hypotheses, which then feed into the system to produce large amounts of data,

which can be explored quantitatively. This analysis might suggest that these

algorithms be improved, resulting in a kind of feedback loop between qualitative

and quantitative methodology. [GV]

2 Zaltman, G. (2001). U.S. Patent No. 6,315,569. Washington, DC: U.S. Patent and Trademark Office.

4

Criticism of qualitative methods

17. Probably the biggest problem of qualitative methods, in policy-makers' eyes, is their

inability to answer questions such as: to what extent are the results generalizable?

What is the prevalence of the phenomenon being observed? A political party facing

an election, for example, would be foolish to rely solely on focus groups as

representative of the whole population. They might offer a good overview of the

breadth of opinions among the electorate, but their results should not be generalized

to the general public. [RvB]

18. From the point of view of inferential statistics, the problem of generalisation of

qualitative methodology boils down to the problem of representativeness: samples

sizes are too small. Even if purposeful sampling is used, there is some risk: how to

decide who the right participant is? [DdR]

Rebuttal – measuring attitudes and opinions and estimating their prevalence in a

given population is not the aim of qualitative methodology. Quantitative surveys

are probably best suited for that, as they allow giving an equivalent weight to each

participant. [FD]

19. Another problem of qualitative methodologies is that data analysis heavily relies on

the researcher's subjective interpretation.

Rebuttal – there is also subjectivity in quantitative behavioural research (e.g., in

designing experiments and data analysis), as well as in the way survey results are

analysed and interpreted. [BF, NRP]

20. Qualitative research is criticised for not being replicable. Each observation of a social

reality is unique to that place, point in time, and observer.

Rebuttal – replicability is not only an issue for qualitative research. It is also a big

issue for experimental work [GV]. A recent study has showed that only 1/3 of

quantitative psychological experiments had been replicated3. This being said,

triangulation can contribute to addressing this issue (see point 32).

21. Another problem with focus groups is reaching consensus or finding shared ideas.

For one, participants who speak the most are usually people with the highest

education level. Researchers may incorrectly consider their opinion as the

consensus. [FD, CD] And what is the threshold to determine that an opinion is

generally shared? How do you deal with different opinions? [DdR] The moderator

3 Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science,

349(6251), aac4716.

5

may spend some time building consensus, but this does not entirely solve the

problem. [AD]

Rebuttal - in focus groups, the idea is to go for maximum variation, not seek for a

consensus. Two focus groups dealing with simple issues may lead to similar results.

But the objective is the opposite: to obtain as diverse results as possible. So you try

to recruit different people for different focus groups. The more obscure and

sensitive the issue, the more diverse the results will be. [MF, GV]

22. Often, qualitative methodologies rely on self-reporting, which has its disadvantages.

This applies equally to quantitative surveys, whereby respondents are also generally

asked to self-report. When asked, people will respond practically anything4. They

will even talk about things they don't know, because it is embarrassing not to have

an answer. People confabulate: they can come up with a lot of reasons to rationalize

their opinions and behaviours, at times not based on real experience. For example,

children may talk about condoms when they haven't even had any sexual

intercourse. So this is a concern – but it is a concern equally for quantitative surveys

as for qualitative research. According to this view, the focus of empirical research

should be on behaviour, not introspection. [BF, TM]

23. When self-reporting is required, qualitative methods may be good for thoughtful,

reflective choice, but not so good for habitual, automatic, non-conscious choice. They

may also not be so good for reporting sensitive issues or issues where social

desirability biases are at play, as people will not express themselves freely. And

what is considered 'sensitive' varies from country to country (e.g. food is a sensitive

issue for Finns). [BF, CRS]

Rebuttal – not all qualitative methods rely on self-reporting. Moreover, sometimes

it’s the self-reporting itself that is of interest, as it discloses people's narratives and

representations (i.e. whatever they have in their head is considered important,

regardless of its relation to actual behaviour). [GV, RvB]

24. Recommended methods for complementing or overcoming the shortcomings of self-

reporting

- "Concept mapping procedure": compared to focus groups, it allows for less social

desirability, more structure and larger sample size. It was developed originally in

sociology for policy issues. People have opinions on certain issues; the goal is to

explore these and their understanding. How? By eliciting concepts through clear

questions. The concepts (or terms or nodes) are then pooled and sorted

4 Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental

processes. Psychological Review, 84(3), 231.

6

according to content. Participants subsequently rate their importance, establish

links between them and discuss the results5. [BF]

- Another technique is to ask people about their experience, instead of relying on

their possible fabrications of experience. In other words, instead of asking

general or hypothetical questions, ask people to recall a particular event or

situation in which they were involved. [BF]

- "Thought listing technique": people are exposed to a message or an event and are

asked to list the thoughts that go through their head. It is a fully unstructured

method. Participants get three minutes to write down what they want. They then

check the difference in salience of one thought between the treatment and the

control group. This could be used as a manipulation check or as a proof of

principle. This technique had been criticised heavily because words were not

equally distributed. [GV] However, it is about relative frequency, i.e. compared

between one treatment and the other. [BF]

- "Projective techniques" could be useful, as they would not rely on people having

to voice what they were thinking. It allows participants a way out. Some

examples include collage, sentence completion, personification and laddering in

means-chains theory6. [BF, FD, MCS]

Some contemporary problems in the use of qualitative methods for policy-making

25. Commodification of qualitative research (i.e. sold by the bulk). Contracting

authorities are often concerned with the number of focus groups and the number of

participants in a focus group. This shouldn't be given too much importance. The

number of participants has little relevance, as long as some minimum standards are

met. Similarly, there is no ideal length of a focus group: they can last half an hour or

half a day. There is no way of saying a priori how much time was needed. The quality

of focus groups is more important. Quality is determined not by the number, size or

duration of the groups, but more so by factors such as how participants will be

selected and recruited, the design of the discussion guide, and the way in which the

qualitative information is analysed. Award criteria for tenders based on the number

of focus groups and participants show a lack of understanding of what qualitative

research tries to achieve. [MF]

5 See, for example: Yin, Y., Vanides, J., Ruiz‐Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison

of two concept‐mapping techniques: Implications for scoring, interpretation, and use. Journal of Research in Science Teaching, 42(2), 166-184. 6 Saadatfard, E. (2014). Means-End Chain Methods Comparison: An Application to Consumer Goods in Iran. International Journal of Economics, Commerce and Management United Kingdom Vol. II, Issue 7, 2014. Available at SSRN: http://ssrn.com/abstract=2481777

7

26. Time constraints. Ideally, qualitative and quantitative methods should be used

sequentially, with qualitative methods used to fine-tune quantitative experiments.

But time constraints were a problem. There is often not enough time for qualitative

results to feed into the design of quantitative research leg. This means they are

often run in parallel because of high time constraints put on contractors. More time

is required, especially at the very beginning of a project – need a proper inception

phase. In behavioural studies for EU policy-making, it would be advisable to allow

contractors more time for these tasks and to have EU officials available during these

phases to quickly respond to the contractors' doubts. Time is thus the biggest

constraint in using qualitative research effectively for these studies. [CD]

27. Timing: qualitative research is often used too late in the process. Because of this, its

impact on policy-making is low, and the opportunity to get full leverage of

qualitative findings is missed. It should be incorporated from the outset to allow for

the depth and richness of qualitative findings to have maximum power to inform

policy development. [MF]

28. Budgetary pressures. These lead to poorer outcomes, because qualitative methods

and analysis involve an iterative process that cannot be done quickly. Budgetary

pressures imply less importance being given to analysis and interpretation, because

they translate to work being done by junior researchers. The real value comes when

translating data (which is now easy to collect) into meaning (which costs money).

[MF]

29. Increased tendency for reportage: Reportage is the tendency in qualitative reporting

to simply feedback what participants say – numerous direct quotes, linked together

with a bit of descriptive text, with minimal analysis, interpretation or attempt to link

the information to the ‘bigger picture’. This makes it harder for policy-makers to

understand and to use for policy, since it is of little value. [MF]

30. There was support to the idea of providing training in behavioural sciences to EU

officials across the European Commission. Designing experiments takes a lot of time

and implies a lot of interactions between contractors and EU officials. It is a very

intensive process during which policy-officers are asked to make decisions, but they

are often hesitation as many of them are not very knowledgeable about behavioural

sciences and the JRC cannot be behind every project. Training could help policy-

makers gaining more confidence to make these decisions. [CD]

8

Good practices for qualitative behavioural studies for EU policy-making7 31. When considering qualitative research…

Be receptive to research proposals that include qualitative methodology.

Move beyond the sterile quantitative-qualitative dichotomy. Simply ask why a

specific method is being proposed. The value added of the qualitative research be

clear (e.g. novel methods shouldn't be chosen just because they are novel)

Remember that the purpose of qualitative methodology is to identify issues, to

see what is out there. Don't expect quantification of what you find in a focus

group (i.e. "most participants thought…"). That is not the purpose of a focus

group.

Do not always expect qualitative research to be followed up with a quantitative

study. Qualitative research has its own place. It is not just a pilot. Depending on

the question it can be a standalone study. Alternatively, qualitative research can

follow the quantitative phase, rather than precede it.

If qualitative work is conducted prior to quantitative research, ask for the

context to be mapped out well. Make sure people have been listened to. Do

expect the qualitative findings to create the environment where the quantitative

research can lead to good results.

32. To ensure quality…

Give emphasis to how the research will be designed and analysed, instead of

buying by the bulk. Ask the bidder to justify why they chose the method they

chose, and how they link to the idea being studied and the objective of the study.

Ask for details in the design of the qualitative methodology used. Ask: "let us

know all the quality standards you intend to apply to guarantee a good result".

Ask for success stories in order to appreciate the value of qualitative research for

policy-makers.

Ask for details on how researchers expect to analyse findings and how they will

translate them into clear policy recommendations.

Consider the benefits of triangulation (of data or of method). Triangulation is

good for the robustness of a study. It allows you to rule out odd effects which

might appear occasionally, but which should not be given importance. Finding

7 This part of the discussion was too dynamic and fast-paced to accurately record who made what claim.

9

the same result in different countries and using different methodologies is a sign

of robustness.

Do ask about inter-coder reliability for increased quality in qualitative studies.

An inter-coder reliability test assesses the degree to which two or more

independent researchers agree on the how to code the same body of content.

This is often not brought into qualitative studies, and could be introduced in the

terms of reference of a study as a guarantee of quality.

33. About interviews and focus groups…

Keep interviews loosely structured – don’t design interview guides as a

restrictive questionnaire, for this will impose words and categories on

participants which they might not have used on their own.

Ask for a good discussion guide, arrived at in a good, well-thought out inception

phase. Allow for that inception phase.

Ensure the right participants for focus groups are selected, setting clearly what

are the selection and exclusion criteria. Focus groups that are well-targeted will

yield more informative results.

Consider the role of the moderator of different focus groups in a study.

Qualitative research recognizes the impact of a researcher's presence on the

object being studied (in this case the focus group interaction). If the results of

similar focus groups are going to be compared, the profile of the moderators in

these groups should be comparable too.

Acknowledgments

This document captures the main points of the discussion at the workshop "Qualitative

Methodology in Behavioural Studies for EU Policy-Making" (Seville, 27 July 2016), and

includes insights from all participants. Drafted by RvB with input from FD, JSL and SRA.

Thanks to MF for comments on an earlier draft.

Participants

Name Surname Institution Initials

Jean-Jacques AMITY CHAFEA J-JA

Femke DE KEULENAER Ipsos FdK

10

Denise DE RIDDER University of Utrecht DdR

Franςois DESSART JRC, European Commission FD

Charlotte DUKE London Economics CD

Allison DUNNE GfK AD

Bob FENNIS University of Groningen BF

Mark FRANCAS TNS MF

Ioannis MAGHIROS JRC, European Commission IM

Tim MITCHELL Decision Technology TM

Silvia PELLA DG JUST, European Commission SP

Sara RAFAEL ALMEIDA JRC, European Commission SRA

M. Carmen RODRIGUEZ SANTOS University of Leon CRS

Nuria RODRIGUEZ-PRIEGO JRC, European Commission NRP

Joana SOUSA LORENҪO JRC, European Commission JSL

René VAN BAVEL JRC, European Commission RvB

Giuseppe VELTRI University of Leicester GV

Agenda

09:15 Arrival at JRC-IPTS and registration

Session 1: Welcome and introduction

09:30 Welcome and tour de table

Ioannis Maghiros, JRC-IPTS

10:15 Objectives of the meeting & setting the scene

Rene van Bavel – Nuria Rodríguez, JRC-IPTS

10:30 Foresight and Behavioural Insights Unit's current and future work

Joana Sousa Lourenco – François Dessart – Sara Rafael Almeida, JRC.DDG.02

11:00 Discussion

11:15 Coffee Break

Session 2: Presentations by research teams: key considerations based on past

11

experience

11:45 BEACONS (Denise de Ridder – Bob Fennis)

12:05 Decision Technology (Tim Mitchell)

12:25 GfK Belgium (Allison Dunne)

12:45 University of Leon (M. Carmen R. Santos)

13:05 London Economics (Charlotte Duke – Femke De Keulenaer)

13:25 London School of Economics (Giuseppe Veltri – Mark Francas)

13:45 Lunch

Session 3: Application to EU policy

14:45 Qualitative methods for EU policy-making: the view from Brussels

Silvia Pella, DG JUST

15:15 Discussion: What should be the role of qualitative methodology in

behavioural studies for EU policy?

a) How can qualitative research reliably contribute to behavioural

studies?

b) How to overcome concerns about lack of representativeness?

c) When to use qualitative methods, when not?

d) What are best practices to ensure successful outcomes?

e) Strengths and weaknesses of interviews, focus groups, participant

observations and analysis of documents

f) Generally, how can qualitative methodology inform the policy-making

process?

g) How can policy-makers be made more aware of the value of qualitative

methodology?

16:15 Coffee break

16:45 Discussion: What should be the role of qualitative methodology in behavioural studies for EU policy? (continuation)

17:15 Closing remarks; any other business

Rene van Bavel, JRC-IPTS

17:30 End of the workshop