writing with quantitative and mixed methods data

The University of Manchester

Writing with Quantitative andWriting with Quantitative andMixed Methods DataMixed Methods Data

James Nazroo

Cathy Marsh Centre for Census and Survey Research and Sociology, School of Social Sciences

[email protected]


Overview

Starting points: Who is the audience? Telling a story

Working with/in a structure and style

Showing data Volume Good table manners Visualising data

Writing a discussion: summary of, or theorising with, data

Multivariate analysis, steps in a story or introducing complexity

Mixing data and mixing writing?


Examiner: comprehensive and detailed

Peer-reviewed journal (fellow academics): robust and innovative

Policy and practitioners: clear and evidenced

Conference: performance, visual

Media: novel, controversial and easy

Read examples before you start

Who is the audience?And what kind of story do they want?


What are the key messages?

How do these dictate the literature reviewed (the foundations for the story)?

Which data are needed to tell the story, and in what order do they need to be presented (a linear narrative?)?

Concluding the story Summary? Interesting and simple? or Interesting and complex?

Example: explaining gender differences in depression

Telling a story: the data-theory dialogue


Nazroo (1997) Gender inequalities in depression

Introduction, literature review Consistency of findings on gender differences in depression Challenge some explanations: artefact, alternative disorders and biological difference Value of a focus on gender roles: role strain, cost of caring and identity relevant stressors Key hypotheses

Methods: why a study of couples, why a focus on life events, sample, measures

Results Women more likely to have an onset of depression Difference entirely a consequence of events in the ‘domestic’ arena Clear gender differences in domestic roles Men under-report the significance of events in the ‘domestic’ arena, and women are more

likely to blame themselves for the occurrence of such events For couples with minimal differences in roles there are no differences in the impact of

events in the ‘domestic’ arena

Discussion Summarise findings Theoretical implications: caring work and role identity, not a generic greater vulnerability Consistency with other literature Unique contribution of this paper, ability to precisely test relevant explanations Possible limitations of this study


Science, positivism and social science: what does your audience expect?

A classic structure, the IMRaD style: Introduction (with questions/hypotheses) Methods

Sample Measures Analytical approach

Results Discussion and conclusion

Example, guidance from the British Medical Journal

Writing within sections, and writing certain types of (precise) sentences

Writing QuantitativelyWorking with/in a structure and style


How much data can there be?

Example: tables in the ELSA wave 1 report

Presenting data parsimoniously Which bits of data are needed for the story? Which bits of data are redundant in tables and figures?

Volume of data


For clarity we prefer visual displays, and we leave out extraneous detail to focus attention on the story line

To allow others to inspect and possibly reinterpret the result we want to leave as much of the original data as possible in numerical form

Data must be explained, do not assume that the reader understands the methods used and how the presented data relate to this

Showing data (Marsh 1988)


The title should be the first thing the reader looks at: Summarise contents When, where and who (date, geographical unit, and unit of analysis)

Source of the data and unit of measurement

Labels for rows and columns (do not use variable mnemonics)

Missing data (cases, responses, items from a scale)

It should always be possible to convert a percentage table back to raw cell frequencies – need base numbers (and maybe weighted bases)

Show which way proportions/percents run

Layout, order columns/rows to make comparison easier Which do you want the reader to compare? Which are theoretically next to each other? or Order according to size of effect

Showing data: good table manners (Marsh 1988)


Household incomes of mixed and non-mixed children

White Indian Pakistani Bangladeshi Black Caribbean

Mixed ethnicity? No Yes No Yes No Yes No Yes No Yes

Household income, column %

> £52,000 6.9 15.8 7.9 25.4 1.7 0.5 0.2 16.8 0.6 2.1

£31,201-52,000 17.0 22.6 12.1 22.8 3.3 13.3 3.4 5.1 7.4 15.9

£20,801-£31,200 22.0 24.3 15.2 10.4 6.1 12.9 3.6 0.0 11.2 20.0

£10,401-£20,800 29.4 22.6 31.9 17.1 35.6 17.3 28.4 15.3 23.2 20.1

£0-£10,400 18.3 6.8 17.8 13.4 36.3 35.6 37.5 30.6 46.4 34.8

Mean equivalised income (£) 12389 16966 10573 19079 5717 8352 4844 12927 6934 9753

Income unknown/refused, % 6.4 8.0 15.1 10.9 17.1 20.4 26.8 32.2 11.2 7.1

Base 13,604 544 371 68 715 47 268 17 217 225

Millennium Cohort Study, sweep 1 (cohort members aged on average 9 months)


Household incomes for children in different ethnic groups: the implications of mixed ethnicity

Not mixed Mixed

White Indian Pakist-ani

Bangla-deshi

Caribb-ian

White Indian Pakist-ani

Bangla-deshi

Caribb-ean

Household income, column %

> £52,000 6.9 7.9 1.7 0.2 0.6 15.8 25.4 0.5 16.8 2.1

£31,201-52,000 17 12.1 3.3 3.4 7.4 22.6 22.8 13.3 5.1 15.9

£20,801-£31,200 22 15.2 6.1 3.6 11.2 24.3 10.4 12.9 0 20

£10,401-£20,800 29.4 31.9 35.6 28.4 23.2 22.6 17.1 17.3 15.3 20.1

£0-£10,400 18.3 17.8 36.3 37.5 46.4 6.8 13.4 35.6 30.6 34.8

Mean equivalised income, £ 12389 10573 5717 4844 6934 16966 19079 8352 12927 9753

Income unknown % 6.4 15.1 17.1 26.8 11.2 8 10.9 20.4 32.2 7.1

Base 13,604 371 715 268 217 544 68 47 17 225

Millennium Cohort Study, sweep 1 (cohort members aged on average 9 months)


Why show graphs when they take more space for less information?

Using the correct type of graph (scatterplot, pie chart, bar chart, stacked bar, line, etc)

It is good practice to begin numerical axis at zero, otherwise you should clearly label the axis

Use simple subdivisions of the scale, single units, twos, or (multiples of) fives

Importance of layout, make comparisons easy, as for tables

Examples

Showing data: visualising data

Combining the strengths of UMIST andThe Victoria University of Manchester

Ethnic differences in equivalised household income

31% 27%

41%48% 45%

69%

90%

0%

20%

40%

60%

80%

100%

WhiteEnglish

Whiteminority

Chinese Caribbean Indian Pakistani Bangladeshi

Bottom tertile Middle tertile Top tertile

1999 Health Survey for England


The ethnic make-up of the UK Muslim population2001 Census

Indian8%

Bangladeshi17%

Pakistani44%

Black African6%

Other Asian6%

Mixed4%

White British4%

Other White7%

Other4%


Visualising data: the importance of layout

Mortality in 1981-1992 by social class and employment status

0

20

40

60

80

100

120

140

160

180

I II IIINM IIIM IV V I II IIINM IIIM IV VRG Class

Employed in 1981 Unemployed in 1981

Fair Society, Healthy Lives: The Marmot Review (2010)




0

20

40

60

80

100

120

140

160

180

I II IIINM IIIM IV V I II IIINM IIIM IV VRG Class


‘The Figure shows the social gradient in the subsequent mortality of those that experienced unemployment in the early 1980s. For each occupational class, the unemployed have higher mortality than the employed’





0

20

40

60

80

100

120

140

160

180

I II IIINM IIIM IV VRG Class


‘The Figure shows the social gradient in the subsequent mortality of those that experienced unemployment in the early 1980s. For each occupational class, the unemployed have higher mortality than the employed’



Clear links between methods, data and conclusions

How important is it to theorise, rather than summarise? Pushing the data beyond description to explanation

Use the introduction to set the discussion up

Make connections with existing empirical literature to create the space to explore similarities and differences in findings, which then require explanation and allows for theoretical development

Example: How publically provided healthcare (the NHS) minimises ethnic inequalities

Discussion: summarising, or theorising with, data


Summary of findings: few ethnic differences in access to and outcomes of primary health care in the UK

Similarity with findings in other studies, and why there are differences with others

Contrast between these UK findings and findings in the US

Limitations: conditions covered, sample representativeness Strengths: uniqueness of this work in methods and coverage

Concluding comments Few ethnic differences in the UK But marked differences in the US Role of insurance in the US Likelihood that differences between the UK and US are healthcare

system driven

Nazroo et al. (2009) Ethnic inequalities in access to and outcomes of healthcare in the UK


Story-telling

Writing within a structure / working on a structure

Selecting data necessary to evidence the narrative

Displaying data

Writing a discussion

Do data speak for themselves? Discuss the contrasting experiences of data exploration and hypothesis testing; and what this means for writing – identifying stories, or building stories?

Summary and a question for discussion


Multivariate modelling is inevitably complex Typically examining the influence of a variable when other variables are

held constant But in real life other variables are not constant (an issue of variable

rather than case analysis?) And many variables to consider, which are useful for theoretical

development (part of the story) and which just need ‘controlling’ for?

Translating coefficients into meaningful values, tell your reader what a ß coefficient or an odds ratio means. The significance of standard errors or confidence intervals.

Build your model in a way that follows your intended narrative

Example: The impact of employment transitions on mental wellbeing for older people. (Route out of work is not randomly distributed, so need for multivariate analysis.)

Multivariate Analysis and Storytelling


ß coefficient

Remain working 0

Start working 0.02 (-0.40, 0.44)

Remain not working 0.44 (0.30, 0.60)

Become unemployed -0.05 (-0.71, 0.62)

Stop working, sick 1.16 (0.66, 1.67)

Start looking after the home -0.60 (-1.18, -0.03)

Retire 0.04 (-22, 0.31)

Retirement and depressionA transition model for those ≤ state pension age

Model adjusted for gender, age and depression score at wave 1

How to write (tell) the story?

Comparison is with those who remain working

ß coefficient is the change in points on the depression scale that the transition is associated with, compared with those who remain working

Theoretically uninformative factors (age and gender) are included, but not shown

Those who remain unemployed and who retire sick are more likely to have a rise in depression score, compared with ...

Those who stop working to look after the home are less likely to have a rise in depression score, compared with ...

Those who retire have the same level of change as ...

CESD score at Wave 3 of ELSA


Undifferentiated model

Remain working 0

Start working 0.02 (-0.40, 0.44)

Remain not working 0.44 (0.30, 0.60)

Become unemployed -0.05 (-0.71, 0.62)

Stop working, sick 1.16 (0.66, 1.67)

Start looking after the home -0.60 (-1.18, -0.03)

Retire 0.04 (-22, 0.31)

Retire wealthy -

Retire not wealthy -


Same story as before, but additional rows suggest that retirement may be complex



Undifferentiated model Differentiated model

Remain working 0 0

Start working 0.02 (-0.40, 0.44) 0.02 (-0.40, 0.44)

Remain not working 0.44 (0.30, 0.60) 0.46 (0.30, 0.61)

Become unemployed -0.05 (-0.71, 0.62) -0.04 (-0.71, 0.62)

Stop working, sick 1.16 (0.66, 1.67) 1.17 (0.66, 1.68)

Start looking after the home -0.60 (-1.18, -0.03) -0.60 (-1.17, 0.03)

Retire 0.04 (-22, 0.31) -

Retire wealthy - -0.41 (-0.82, 0.01)

Retire not wealthy - 0.37 (0.03, 0.70)


Differentiating retirement reveals contradictory effects, or the importance of context



Multivariate analysis, but can only show one variable/tell one story

Scales, meaningful visual display

Showing statistical significance, and/or standard errors, and/or 95% confidence intervals

Example: Explaining the relationship between age and depression in later life

Multivariate Analysis and Visualising Data


Age and depression : multivariate analysis (CES-D score > 4)

50-54 55-59 60-64 65-69 70-74 75-79 80+

Ln

od

ds

ra

tio

Adjustment for gender only



50-54 55-59 60-64 65-69 70-74 75-79 80+

Ln

od

ds

rati

o

Adjustment for gender only + Retirement status



50-54 55-59 60-64 65-69 70-74 75-79 80+

Ln

od

ds

rati

o

Adjustment for gender only + Retirement status + Marital status



50-54 55-59 60-64 65-69 70-74 75-79 80+

Ln

od

ds

rati

o

Adjustment for gender only + Retirement status + Marital status + Economics



50-54 55-59 60-64 65-69 70-74 75-79 80+

Ln

od

ds

ra

tio

Adjustment for gender only + Retirement status + Marital status+ Economics + Activities of daily living


Adjusted for gender

And retirement status

And marital status

And economics

And Activities of daily living

Age

50-54 1 1 1 1 1

55-59 1.02 0.86 0.87 0.91 0.88

60-64 0.96 0.77 0.78 0.79 0.75

65-69 0.75 0.67 0.64 0.62 0.54

70-74 0.97 0.84 0.75 0.70 0.58

75-79 1.30 1.18 0.98 0.89 0.71

80 + 1.44 1.33 0.93 0.81 0.49

Age and depression: odds ratio for CES-D > 4

Darker shading p < 0.05Paler shading p < 0.1


Why do mixed methods research? Pragmatics of funding and positioning, or something with an intrinsic value?

The threat to our position as methodologists: from expert to novice (also for multidisciplinary research)

Politics, conflicting epistemological (and ontological) positions, and contrasting explanatory logics: quantifying the qualitative or qualitatively driven?

Teamworking and power: Where in the team do particular skills and experience lie? Methodological disrespect Authorship Need for seniority/mentorship in all ‘arms’ of the research

All of this requires a clear and negotiated starting point

The benefits and risks of doing mixed methods research


One question or complementary questions

Triangulating or revealing different dimensions

How to deal with contrasting data, theorising and addressing complexity

Analysis and writing: How to handle so much (relevant) data? Write separately (an illustrative table or quote), with attempt to integrate in

a conclusion/or challenge substantive or epistemological orthodoxies Do unique insights only emerge when we demonstrably integrate data and

findings / or are the insights produced in more tacit ways (implicitly drawing on alternative orientations and data) / or by creatively drawing out ‘tensions’ in alternative uses of and orientations to the ‘mixed’ data?

Example: is marital violence a problem of male violence against women, or of violence in relationships and violent spouses?

Writing with mixed methods dataIntegrated and/or contrasting stories


Nazroo (1995) Gender and Marital Violence

Introduction, literature review Feminist literature on marital violence Survey evidence suggesting women are as (or more) violent as men Positivist critique of feminist methodology, then critique of positivistic survey methods Different theoretical orientations to gender-relations, patriarchy and power

Methods: combining within one study sample qualitative approaches, positivistic survey methods, and quantitative coding of meaning and context

Results Survey methods: women more aggressive than men Quantitative coding of meaning: men's aggression is much more dangerous,

intimidating and harmful than women’s Qualitative findings: detailed, credible and evocative accounts of abusive relationships

that resonate with feminist analysis

Discussion Summarise findings and explain contradictions Implications for methodology Implications for theoretical orientations

Conclusion Highly misleading results from quantitative surveys Critique of positivistic orientation to data


Decisions made very early in the research process will influence the kinds of questions you can tackle/stories you can tell. Plan early.

Particularly important when working across methods/disciplines/teams.

So think how evidence will be ‘integrated’ early on. How the narrative will draw on different elements/types of evidence.

Is the study qualitatively or quantitatively lead, if either? How do the different forms of data / different questions complement each other? What does this mean for the kind of narrative that can be told?

One person to do the final write up?

And do not assume readers have strong knowledge of methods used, or share your epistemological position, particularly when using mixed methods.

Some Concluding Thoughts


Mixed methods, discuss experiences of writing up apparently contrasting data

Mixed methods, experiences of writing in teams with contrasting orientations to data

More questions for discussion

writing with quantitative and mixed methods data

Documents