Qualitative Methods as an approach to policy development

and EvaluationUsing Qualitative Comparative Analysis in comparative policy

research

David Byrne – Durham Universitydave.byrne@durham.ac.uk

Presentation at: Using qualitative research to inform policy and practice – Newport 4th April 2006

What Works?

• For Whom? – Cui bono? – the political actors’ question.

• How? – what implemented processes achieve any sort of outcome? – the engineers’ question

• Why? What underlying generative mechanism (s) produces any given future condition? – the scientists’ question.

‘ … policy researchers, especially those concerned with social as opposed to economic policy, are often more interested in different kinds of cases and their different fates than they are in the extent of the net causal effect of a variable across a large encompassing population of observations. After all, a common goal of social policy is to make decisive interventions, not to move average levels or rates up or down by some miniscule fraction.’ (Rihoux and Ragin 2004 18)

‘ … an idea widely shared by social scientists – that it is possible to derive useful empirical generalizations from the examination of multiple instances of social phenomena (i.e. from comparable cases).’ (Ragin Fuzzy-Set Social Science 2000 332)

Configuration

The idea of configuration is associated with the work of Ragin but can be found earlier in Norbert Elias’ proposal of a figurational sociology. For us what matters is that the condition of a case at a given point in time – in complexity language its co-ordinates in multi-dimensional state space – is the product of a whole set of factors acting together in interaction.

Each logical combination of values on the Each logical combination of values on the independent variable is represented by one row of a independent variable is represented by one row of a truth table. Once this part of the truth table is truth table. Once this part of the truth table is constructed, each row is assigned an output value constructed, each row is assigned an output value (a score of 1 or 0 on the dependent variable) based (a score of 1 or 0 on the dependent variable) based on the scores of the cases which share that on the scores of the cases which share that combination of scores on the independent combination of scores on the independent variables. Thus, both the different combinations of variables. Thus, both the different combinations of input values (independent variables) and their input values (independent variables) and their associated output values (the dependent variable) associated output values (the dependent variable) are summarized in a truth table. ….. each row is not are summarized in a truth table. ….. each row is not a single case but a summary of all the cases with a a single case but a summary of all the cases with a certain combination of input values. In this respect, certain combination of input values. In this respect, a truth table is like a cell from a multi-way cross-a truth table is like a cell from a multi-way cross-classification of several categorical independent classification of several categorical independent variables. (C.C. Ragin – variables. (C.C. Ragin – The Comparative The Comparative MethodMethod1987 87)1987 87)

mixed

sixform

religious

comp

Highdep highpcsp

number yconsist

1 1 0 1 0 0 26 0.769231

1 0 0 1 1 0 18 0.611111

1 1 0 1 1 0 17 0.588235

1 0 0 1 1 1 14 0

1 0 0 1 0 0 11 0.909091

1 1 1 1 0 0 9 0.888889

1 0 0 1 0 1 8 0.5

1 1 0 1 0 1 7 0.428571

1 1 0 0 0 0 3 1

1 1 0 1 1 1 3 0

1 1 1 1 1 0 3 0.333333

0 1 1 1 0 0 2 1

1 0 1 1 0 0 2 0.5

0 1 0 1 0 0 1 1

0 1 1 1 1 0 1 1

1 1 1 1 0 1 1 1

More than one way to skin a cat

A crucial implication of the idea of configuration is that a given condition may be the product of DIFFERENT combinations of causal factors. Multiple and complex causation is possible.

‘If … we live in a world of great causal complexity, then a common pattern will be for outcomes to result from different combinations of causal conditions.’ (C.C.Ragin Fuzzy Set Social Science 2000 103)

BUT IS IT?

• Configurational accounts do allow for complex causation

• However, they are still in some ways reductionist.

• They still reify components over the properties of the system as a whole

• Still – a step in the right direction.

‘For causation, the main contrast is between the conventional view of causation as a contest between individual variables to explain variation in an outcome and the diversity-oriented view that causation is both conjunctural and multiple. In the conventional view, each single causal condition, conceived as an analytically distinct variable, has an independent impact on the outcome. In the diversity-oriented view, causes combine in different and sometimes contradictory ways to produce the same outcome, revealing different paths.’ (2000 15)

‘Fuzzy sets offer researchers an interpretive algebra, a language that is half verbal-conceptual and half mathematical-analytical …. Most theoretical arguments, as verbal formulations, deal with set-theoretic relationships, so they offer the opportunity for creating a close correspondence between theory and data analysis.’ (Ragin 2000 4)

The Nature of Our Materials

• Cases about which we have both measures and descriptions

• The measures are Numbers.– Some of the Numbers are continuous

measures.– Some of the Numbers are categorical

specifications.

• Our descriptions take the form of texts.• The texts are available as online reports

We are dealing with Cases

• In this kind of research we have cases which, typically are institutions or political / quasi-political entities.

• The cases can be identified as case nodes.

• We assign attributes to case nodes.

Using NVIVO with Our Cases

With Measures

1. Categorical Measures can be entered directly as text attributes.

2. Continuous Measures can be banded into ordinal values and entered as numbers.

With Textual Materials

1. We can read through our texts to establish new attributes ‘in vivo’.

2. We can code these new attributes as text attributes to our case nodes.

The Flexibility of NVIVO

1. We can change the range specifications of pre-existing attributes in the light of interpretive engagement with our materials. This usually means extending the rang of values.

2. We can create new attributes which emerge from our interpretive engagement with our materials.

An Example – English Secondary Schools – Pre-existing attributes

Continuous Variables

1. School Performance

2. Truancy Rates

3. Size information

Categorical Variables

1. School Type

2. Gender

3. Selective or not

4. Catchment area type (can be ordinal)

Data Sources

Ofsted Reports – available online and can be downloaded as text documents.

Ofsted reports are reports of an inspection. They include descriptive accounts of the school across a range of parameters and specification of performance in relation to objectives. There is a good deal of descriptive text which takes essentially the same form as an ethnographic report.

Procedure in NVIVO

• Create a set of case nodes corresponding to cases.

• Create or import attribute table based on existing measurements.

• Work through an interpretive process which establishes new attributes and varies existing attributes.

• Construct final attribute table.• Export in .dat form.

Using SPSS

• Import into SPSS

• Recode using SPSS facilities

• Especially the Visual Bander for specifying ranges so as to convert continuous variables into ordinal variables.

• Cluster appropriately as a data reduction exercise to construct both cause conditions and effect outcomes.

Using Tosmana or QCA

• Input .dat file exported from SPSS• Specify variable type etc. using Tosmana

facilities if using ordinal variables• Establish configurations• Look at ‘contradictory configurations’.• Extend range of causal specifications to

reduce proportion of contradictory configurations.

• Return to materials in NVIVO to look for new causal elements.

References Rihoux, B. and Ragin, C. 2004

‘Qualitative Comparative Analysis (QCA): State of the Art and Prospects’, paper presented at : APSA 2004 Annual Meeting, Panel 47-9, Chicago,

Ragin, C. (1987) The Comparative Method Berkely: University of California Press

Ragin, C.C. (2000) Fuzzy-Set Social Science Chicago: Chicago University Press