1

Qualitative Research: how to do good work, get it published and

have an impactProfessor Stewart Clegg

UTS

2

Social science

• No need for physics envy! Social science that apes physics will have little useful to contribute

• Follow Popper’s idea of critical rationalism – we must try to falsify our initial hunches about the relations between phenomena in our data

• Only then, if we cannot falsify, can we speak about ‘objective’ knowledge – but it is always provisional on the next study ….

3

Validity enhancing approaches

• Analytic induction• Constant comparative method• Deviant case analysis • Comprehensive data treatment• Tabulations

4

Generalizability

How to generalize in qualitative research

Deductive inference Choosing a critical or deviant case to refute an existing theory

Comparative inference Maximize variation in cases – if findings hold across cases then this is a good sign of generalizability

Exemplary or emblematic inference

The case stands as an exemplar of the phenomenon under consideration

5

Flyvbjerg: Five misunderstandings about case study research

1. General, theoretical (context-independent) knowledge is more valuable than concrete, practical (context-dependent) knowledge

But: take us close to data, build skills in data analysis

2. One cannot generalize on the basis of an individual case

But: single cases are crucial in refuting initial hypotheses – the ‘black swan’ argument

3. Case studies can generate initial hypotheses but other methods have to be used for hypotheses testing and theory building

But: exemplary or atypical cases can can reveal more information than the randomly sampled average

4. Case studies contain a bias towards verification – to confirm preconceived prejudices

On the contrary, the bias is towards falsification, because of constant comparison

5. Difficult to summarize and develop general propositions and theories on the basis of specific case studies

But: complex narratives represent complex situations

6

Theoretical sampling

• Clarity about unit of analysis – could be one organization but many samples of meetings, encounters, etc.

• Sample chosen not to prove but disprove the case

• Extend sample as new insights emerge to test them out – maybe search for new deviant cases

7

Writing it up

• Write clear, grammatically elegant language.• It should not be necessary to be a sub-disciplinary, or

even disciplinary, specialist, to be able to comprehend

• Have you made it clear what new insights flow from the research?

• Why these are significant, innovative and valuable?• Is it evident‘who benefits?’from the research?

8

Introduction

• The background to the research• The context of the research, particularly the major ideas

(or theoretical perspective) from which the research is derived.

• The reasons for doing the research• The aims and purpose, setting them in the context that is

relevant• The rationale for the design of the inquiry• Why the research is important, valuable or significant• Give clear directions by the end of the 3rd or 4th paragraph:

– In this paper I will, first…, second…, third… .

9

Literature review

• The literature review indicates gaps by pointing out: – Aspects of the field which have not previously been

researched – Limitations or shortcomings of previous research– Areas which other researchers have indicated are in

need of further examination• The literature review indicates that the researcher:

– Is transparent and reflexive about conduct, theoretical perspectives and values

– Understands the theoretical contexts within which the literatures reviewed have been generated

10

Literature Review

• Check out the International Journal of Management Reviews

As the first reviews journal in the field of business and management, the International Journal of Management Reviews (IJMR) is an essential reference tool for business academics and doctoral students alike. The journal covers all the main management sub-disciplines including, for example, HRM, OB, International & Strategic Management, Operations Management, Management Sciences, Information Systems & Technology Management, Accounting & Finance, and Marketing. Each issue includes state-of-the-art literature review articles/surveys which examine the relevant literature published on a specific aspect of the sub-discipline, for example, HRM: Appraisal Systems.

Has someone already saved you a lot of work?

11

Methodology

• This section outlines the methods used to achieve the research aims. – It should describe in as much detail as possible the data

collection procedures used, whether they be experiments, surveys, questionnaires, observations, participatory methods, case studies, document collection or other methods.

– Reference should be made to the research methods literature or literature specific to the field of study to justify the choice of methods.

12

Methodology• The methods must be shown to be reliable (that is, they can

be in principle, replicated) as well as internally and externally valid. – Internal validity means that the conclusions drawn from the study

actually reflect the situation under consideration. – External validity means that the results are generalizable to a wide

range of situations. – It is often necessary to present evidence in this section that the

research is actually achievable, which may involve describing sources of funding, the previous experience and attributes of the researcher, or the extent to which preliminary enquiries into the feasibility of the research have been made.

13

Methodology

• The methodology, or a separate section, should also describe the methods of data analysis to be used. – As with the data collection methods, the analysis methods should be

justified by reference to the relevant literature. – A methodology section can contain a flow chart which summarizes

the way in which the various processes involved in the project fit together.

14

Checkpoints

• How do you represent data or evidence faithfully?• How do you convey the depth, diversity, subtlety and

complexity that the research will generate?• How do you demonstrate that the data or evidence

that you are dealing with will be critically interrogated?

• Remember that for every claim you raise you must support this with evidence, usually a page specific citation.

15

Checkpoint• How do you demonstrate that the study, although qualitative,

is not merely going to be a piece of subjective impressionism?– How do you guard against just re-telling what the subjects

already know?– How do you guard against ‘taking sides’ or being ‘romantic’

about subjects?– Why is this suite of qualitative methods the best for this

research?– What is the sampling frame? What is its rationale?

16

References • It is important to ensure that all the key journals and books in

the field have been referred to. • You will be judged on the quality of the journals and authors

that you reference: make sure the majority of journals are top-tier and the majority of authors are first-rate

17

Writing as a practice

• For most academics, the first substantial writing that they will do is probably a thesis

• Several important lessons:– Never throw anything away – ideas that don’t work where and

when they were first drafted may come in useful elsewhere– Always file drafts with an easy version/date retrieval system– Don't assume that you begin at the beginning and work

systematically through the middle to the end:• Often, it is easier to begin with the middle, with some data

analysis, or research design issues, than the beginning.

18

Writing as a practice

• Usually, I write the first chapter or the abstract last. – It makes more sense – how do you know what you are going to say until

you have said it? (Karl Weick said that!)

• One thing I often find myself doing is trying to sort out the literature into a dialectical debate– If you can establish a thesis and an anti-thesis in the literature then you

create the space in which you can provide a new synthesis that does not just reproduce the existing terms of debate but transcends and repositions them. I’ve done this a few times.

– Tables can help here. Sometimes they are used in the paper/book; other times they are just useful in sorting out thinking, and don’t appear.

19

Practicing

• Irrespective of anything else that you become as an effective researcher, you must become an effective writer.

• These things help:– Read widely– Choose exemplars and model your style on their sentence

structures, punctuation, grammar, vocabulary, etc.– Choose the target journal carefully – does the paper

contribute to the debates that occur within it?

20

Good writing

• Good writing has rhythm through:– The balance of long and short sentences– Clever use of punctuation, to break up complex ideas– A strong narrative that unfolded directionally – from beginning through middle

to end– Structure – appropriate use of sub-heads

• Good writing doesn’t try too hard – it guides the reader – its form makes its function more effortless

• Good writing amuses, pleases, informs, impresses• Bad writing dulls, irritates, bores, and depresses

21

Rules for writing

• One paper = one idea• One sub-headed set of material = one theme• One paragraph = the development of one aspect of

one theme• One sentence = one subject

The importance of a good editor

22

More rules

• Don’t start paragraphs with “indexical pronouns” or “conditional phrases”– Indexical pronouns

• E.g.: It, This, – Conditional phrases

• E.g.: However, Because, Thus, Yet,

• Don’t start sentences with them either!• Avoid e.g., i.e., etc. in prose

23

Visual checks

If it looks wrong it probably is:Quick visual checks:

• Length of sentences – problem is that they are usually too long• Length of paragraphs – here the problem can either be that

they are too long or too short: a rough rule – about three paragraphs a page.

• Microsoft green lines – they are probably telling you something you need to know

• Spell checkers – they do the job – so how come so few people seem to use them … or use them lazily?

• Over-repetition in form: stock phrases, such as:– “It is argued that …”, “Smith argues …”, “According to …”

24

Further visual checks

Structural levelsFirst level sub-heads

Second level subheadsThird level subheads

Don’t go any further than third level – too messy and indicates too much going on and author

not in control of material – also it buggers up the fonts.

• Spelling; if it’s a US journal use US spelling; Australian, Australian spelling; British, UK spelling – and so on.

• Does the journal have a length rule? If not, establish the norm – and follow it.

25

Where to publish

Which journal?– Make sure that you cite debates in the journal in question– Always check the journal style requirements and conform

to them– Nothing irritates a journal editor more than the thought

that because the paper is written to a competitors style guidelines it is a reject

– Always check the Editorial Team and Board:• It is to one or more of these people that the paper will

go – have you cited the likely suspects?

26

Publishing

Publish in journals! This is what national metric systems tend to count.Go for the best journals you can, realistically– The worst they can do is reject– The best they can do is offer excellent feedback from which

you can learn– Always resubmit – either to the original journal if invited or

elsewhere if not.– Always respond to advice received– Always write a clear letter to the editor explaining what you

have done and not done in response to the suggestions received.

27

Why that journal?

Sensemaking– You know the ranking orders for the journals; you’ve

checked their citation impact factor – That journal publishes the right kind of stuff– Key references were published there– It’s a favourite journal– Anyone would kill to be published there!

28

Drafting • Look at six or eight articles from the journal decided

on• Draw up a a series of columns and in these identify:

– The number and identity of sub-headings used– The hierarchy of sub-headings used

• Model the paper’s structure on what seems to work there

• Try to have a sense of audience – do people that write in that journal seem like the kind of scholars you want to address and whom you respect?

29

Test the paper

• Draft early, draft often– I ordinarily draft anything from 30 – 100 times; I

have drafted even more.• Send it to friendly others to read– That’s how I come to draft so much – they offer

good advice• Once you have done ten or twenty drafts and it is getting

in shape send it to some significant others– Always keep old drafts and always number

sequentially so that one knows where one is in the series

30

Finalizing the paper

• Use Google to track down references– http://www.scholar.google.com/ – Someone will have cited that hard to find reference

somewhere for which they don’t have all the details• Have you done:– An abstract?– Keywords?– Checked the bibliographic items?– Sent an anonymous text and a separate author bio?

31

Final check

Checklist– Have you exceeded the word count?– Have you got the format and style right?– Have you sent the additional material that they need, such

as a brief biographical note?– Have you checked the spelling, grammar and bibliography

thoroughly and at least several times?

32

Letter from the editor

• Chill out!– No one gets accepted first time– Plenty of people get rejected– The important thing is to learn from the Editorial letter– What one wants, realistically, is a Revise and Resubmit

• Keep cool!– Maybe leave returning to the paper for some weeks

after getting the Editorial letter– Try and deconstruct the paper from the point of view

of the reviewers– Try and respond to them by reconstructing the paper

33

Responding to reviews

Surviving the review processA R&R is a good thing!

Read letters of review very diligently

Write letters to editor extremely carefully

34

After acceptance

• Getting something accepted is hard enough

• Now you have to demonstrate ‘impact’

• And that is more than being published

35

Research impact

Impact: “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia”. (Higher Education Funding Council For England 2011: 26;)“The demonstrable contribution that excellent research makes to society and the economy …[including] … all the diverse ways that research-related skills benefit individual, organisations and nations: (ESRC 2012: 1)

36

Types of research impact

• Instrumental impact, influencing changes in policy, practices and behaviour though knowledge mobilization

• Conceptual impact, changing people’s knowledge, understanding and attitudes towards social issues

• Capacity building where involvement in research developed the skills of those involved.

37

Impact through Knowledge Mobilization

• Knowledge Mobilization: – Getting the best evidence to the appropriate decision

makers in both an accessible format and in a timely fashion so as to influence decision making

• Two elements:– Evidence-based policy– Need to demonstrate economic impact

38

Knowledge Mobilization

• The movement for evidence-based policy began in health research in the 1990s

• It was rapidly joined to a political mandate demanding demonstration of economic impact from research council funding in terms of measured outcomes rather than just research outputs.– Question: What’s the difference between a research

outcome and a research impact?

39

Measuring research outcomes

• Peer review is the traditional way of measuring research quality so that research is judged by the quality of the journals it is published in: impact factors then become a de facto measure of research outcomes

• Peer review is a good guide to perceived research excellence but not infallible:– Strongly weighted towards US journals because their

impact is situated in a large domestic market and benchmarked globally

40

Moving away from peer review

• Citation analysis: measures such as the h-index (Harzing’s Publish or Perish).

• Case studies from research demonstrating its contribution to economic, social and/or public policy as well as cultural/quality of life impacts with some supporting quantitative indicators

• Pathways to impact statements

41

Pathways to instrumental impact statements

• In the UK the ESRC now expects research applications to include a Pathway to Impact document which outlines the possible pathways through which the proposed research will make an impact

• As well to follow the fashion – but how?– ESRC developed an Impact toolkit advising• How to achieve maximum impact• How to develop an impact strategy• How to promote knowledge exchange, public engagement

and effective communication with stakeholders

42

Measuring instrumental impact

• Payback frameworks• Tracking Forwards• Tracking Backwards

43

Payback Framework

• Payback Framework methodology adapted from health research field and applied to major research centres seeking demonstration of:– Knowledge production– Impacts on future research– Policy impacts– Practice impacts – Wider social and economic impacts

44

Tracking Forward

• This method seeks to track forward from the research outputs to assess the ways in which they have been incorporate into practice.

• Uses surveys of and interviews with researchers and users • Looks at relationships established between researchers

and recipient communities as key to knowledge mobilization

• Looks for involvement of users at all stages of the research and for well-planned user engagement strategies – these require good infrastructure and management support

45

Tracking Backwards

• ESRC Evaluation Committee has sought to establish how and in what ways quantified economic assessment of the value of major program investments could be made

• Mixed success – the measures produced can only ever be approximations of value

• This the default position of most politicians and commentators unfortunately – instrumentalism rules!

46

Issues in demonstrating impact

• Knowledge transfer is a reflexive rather than linear process – sustained by dialogue and conversation rather than one-shot publication or advice

• Have to know:– When to look for research impacts – how long before

effects materialize?– How to assess that the specific contribution made by

research was the key factor – in a multi-causal world establishing causal paths is extremely difficult.

47

Conceptual Impact

• Conceptual impact occurs when research makes significant changes to practice-based thinking, debate, culture and direction.– Conceptual impact is increased by• Effective media training for communication on the part

of researchers • Translation of research findings by skilled knowledge

brokers for different audiences

48

Capacity Building

• Developing the training and capacity of next generation researchers

• Two studies have investigated the impact of PhDs who have moved into practice on practice:– Johnson & Williams (2011) Evaluating the impact of social

scientists, ESRC publication• PhDs’ high-quality skills in interpreting and evaluating research

findings have more impact than specific substantive knowledge

49

Demonstrating impact

• The Social Sciences and Humanities Research Council of Canada ask applicants for research grants to outline the following:– Expected Outcomes• Elaborate on the potential benefits and/or outcomes of your proposed research

and/or related activities.• Indicate and rank up to 3 scholarly benefits relevant to your proposal.

– Social Benefits• Indicate and rank up to 3 social benefits relevant to your proposal.

– Audiences• Indicate and rank up to 5 potential target audiences relevant to your proposal.

– Expected Outcomes Summary• Describe the potential benefits/outcomes (e.g., evolution, effects, potential

learning, implications) that could emerge from the proposed research and/or other partnership activities.

50

Penultimate words• Being a successful researcher has never been easier in some

respects – the range of acceptable investigative techniques and the affordances of the digital ecology

• Being a successful researcher, however, has never been harder in some important respects – particularly the need to demonstrate ‘value for money’ and ‘impact’

• The time for ‘pure’ scholarship is greatly reduced these days

51

Select references

Alvesson, M. (2011) Interpreting Interviews, London: Sage.Alvesson, M. & Karreman, D. (2011) Qualitative Research and Theory Development: Mystery as Method, London: Sage.Alvesson, M. and Skoldberg, K. (2009) Reflexive Methodology: New Vistas for Qualitative Research, London: Sage.Buchanan, A. (2013) ‘impact and knowledge mobilisation: what I have learnt as Chair of the Economic and Social research Council Evaluation Committee’, Contemporary Social Science (Special issue on Knowledge mobilization and the Social Sciences: Research Impact and Engagement, 8(3): 176-190.Chamaz, K. (2006) Constructing Grounded Theory: A Practical Guide through Qualitative Analysis, London: Sage.Denzin, N. and Lincoln, Y. (2005) The Handbook of Qualitative Research, London: Sage.Hennink, M., Hutter, I., and Bailey, A. (2011) Qualitative Research Methods, London: Sage.Silverman, D. (2011) Interpreting Qualitative Data, London: Sage.Silverman, D. (2010) Doing Qualitative Research, London: Sage.Silverman, D. (2007) A Very Short, Fairly Interesting, Reasonably Cheap Book about Qualitative Research. London: Sage.