handa. validity in qual research

7/27/2019 Handa. Validity in Qual Research

1/8

Reliability in Qualitative Research

External reliability

Would independent researchers discover the same phenomena or generate similar

constructs in similar settings?

Internal reliability

Given a set of previously generated constructs, would other researchers match data in the

same way as the original researcher?

Five ways to reduce threats to internal reliability:

Mechanical recording Low inference descriptions (verbatim quotes and detailed accounts) Multiple researchers Participant researchers Peer examination

Goetz, J. & LeCompte, M. D. (1984).Ethnography and qualitative design in educational

research. Orlando, FL: Academic Press.

Reliability (pp. 205-207)

Can the findings be replicated? Or more fitting with qualitative research, are the resultsconsistent with data collected?

Making studies dependable:

The investigators position with regard to assumptions & frameworks and in relationto those studied is made explicit. Triangulation (multiple methods of data collection and analysis) Audit trail

Merriam, S. (1998). Qualitative research and case study applications in education (2nd

ed.). San Francisco: Jossey-Bass.

Reliability

refers to the degree of consistency with which instances are assigned to the same

category by different observers or by the same observer on different occasions(Hammersley, 1992, 67). For reliability to be calculated, it is incumbent on the

investigator to document his or her procedure and to demonstrate that categories havebeen used consistently (Silverman, 2000, p. 188).

Reliability and observation

Short notes made at the time Expanded notes made as soon as possible after each field session

1


2/8

A fieldwork journal to record problems and ideas that arise during each state of

fieldwork A provisional running record or analysis and interpretation (Spradley, 1979)

Reliability and texts

Categories should be applied in astandardizedway. Inter-rater reliability: involves giving the same data to a number of analysts (or raters)

and asking them to analyze it according to an agreed set of categories. Reports are

examined and any differences are discussed.

Reliability and transcripts

Standardization of transcript conventions

Group data analysis (with audio- or video-data)

Silverman, D. (1993).Interpreting qualitative data: Methods for analysing talk, text, andinteraction. London & Thousand Oaks, CA: Sage.

2


3/8

Validity in Qualitative Research

Internal validity

The extent to which scientific observations and measurements are authentic

representations of some reality

External validity

The degree to which such representations can be legitimately compared across groups

typicality of phenomenaValidity increases by: Increased length of time in site reduces artificial responses Triangulation of data through multiple sources

Goetz, J. & LeCompte, M. D. (1984).Ethnography and qualitative design in educational

research. Orlando, FL: Academic Press.

Internal validity (pp. 201-205)

How closely do research findings match reality? (Reality for qualitative researchers isholistic, multidimensional and ever changing.) Are investigators observing and

measuring what they think they are measuring? Triangulation (with multiple researchers, sources of data, or methods of confirmation) Member checks Long-term observation Peer examination

Participatory or collaborative modes of research Researchers biases/subjectivities clarifying assumptions, worldview, theoretical

framework of study

External validity (pp. 207-212)

What is the extent to which the findings can be applied to other situations? How

generalizable are the results? Rich, thick description Typical or modal category (eg. describes how typical the program, event, sample is

compared with the majority of others in the same class so that others can make

comparisons with their own situations)

Multi-site designs Reader or user generalizability

Merriam, S. (1998). Qualitative research and case study applications in education (2nd

ed.). San Francisco: Jossey-Bass.

3


4/8

Validity

By validity, I mean truth: interpreted as to the extent to which an account accurately

represents the social phenomena to which it refers (Hammersley, 1990, 57). Hammersleysuggests that researchers can address issues of validity by adopting a subtle form of

realism, ie: Validity is identified with confidence in our knowledge but not certainty. Reality is assumed to be independent of the claims that researchers make about it.

Reality is always viewed through particular perspectives; hence our accountsrepresent reality they do not reproduce it (Hammersley, 1992, 50-51).

Silverman (2000, p. 188) argues that we cannot say that the claims of a research study arevalid when:

Only a few exemplary instances are reported The criteria or grounds for including certain instances and not others are not provided The original form of the materials is unavailable

Silverman (1993) rejects triangulation and members validation as appropriate methods

for validating qualitative studies. He suggests instead:1. Methods of generalising to a larger population

2. Methods of testing hypotheses

3. The use of simple counting procedures (Silverman, 1993, p. 166).

Silverman, D. (1993).Interpreting qualitative data: Methods for analysing talk, text, and

interaction. London & Thousand Oaks, CA: Sage.

4


5/8

Techniques of analysis for enhancing the quality and validity of qualitative data

Testing rival explanations

Negative cases Triangulation (methods, sources, analyst, theory/perspective)

Keep data in context

Techniques for enhancing the credibility of the researcher

Report any personal and professional information that may have affected data

collection, analysis, and interpretation either positively or negatively Intellectual rigor

Patton, M. (1990). Qualitative evaluation methods. Thousand Oaks: Sage.

Wolcott describes getting it right or trying not to get it all wrong by the following

activities: Talk little, listen a lot

Record accurately, in their words, immediately after or during events Begin writing early, share drafts with others knowledgeable about the setting

Let readers see for themselves, include primary data in final accounts Report fully, deal with discrepant cases Be candid, sees subjectivity as a strength of qualitative research Seek feedback Try to achieve a balance return to site or field notes to reread the data, then reread

the draft Write accurately write for technical accuracy, internal consistency with

generalizations grounded in what is seen/heard

Wolcott, H. (1994). On seeking and rejecting validity in qualitative research. In H.

Wolcott, Transforming qualitative data: Descriptions, analysis, interpretation. (pp. 337-373). Thousand Oaks, CA: Sage.

Different positions taken by scholars on issue of validity:

Fallibilistic validity

validity is understood as a test of whether an account accurately represents the social

phenomena to which it refersDefenders of this view hold that one can have good

reasons for accepting an account as true or false, yet an account is always fallible (p.

169).

Relativized validity

validity of an account is relative to the standards of a particular community at a

particular place and time. The validity of an account or interpretation is judged in termsof the consensus about words, concepts, standards and so on in a given community of

interpreters (p. 169).

5


6/8

No validity

The most radical of postmodernists would argue that it is meaningless to talk of a true

account of the work there are only different linguistically mediated social constructions.No single account can be judged superior to others a needless interplay of differentinterpretations.

Nonepistemic validity

Validity may be interpreted as a criterion of good communication or dialogue; also can be

valid if it leads to change, empowerment a criterion of action (Kvale, Guba & Lincoln,Lather as described by Schwandt, p. 170).

Schwandt, T. (1997). Qualitative inquiry: A dictionary of terms. Thousand Oaks, CA:

Sage.

Silverman (2000) suggests 5 ways of thinking critically about qualitative data analysis inorder to aim at more valid findings:

The refutability principleSeek to refute assumed relations between phenomena (Poppers critical rationalism)

The constant comparative method

Attempt to find another case through which to test out a provisional hypothesis.

Comprehensive data treatment

All cases of data are incorporated in the analysis

Deviant-case analysis

Actively seek out and address deviant cases

Using appropriate tabulations

Use of simple counting techniques, theoretically derived and based on members own

categories can offer a means to survey the whole data corpus.

Silverman, D. (1993).Interpreting qualitative data: Methods for analysing talk, text, and

interaction. London & Thousand Oaks, CA: Sage.

6


7/8

Other criteria for establishing quality in qualitative research

Trustworthiness

How can an inquirer persuade his/her audiences that findings of an inquiry are worth

paying attention to, worth taking into account?Trustworthiness is established by the following criteria/activities: Credibility (qualitative research equivalent to internal validity)

Prolonged engagement Persistent observation

Triangulation (sources, methods, investigators, theories; Denzin, 1978) Peer debriefing Negative case analysis

Referential adequacy (archived recorded data for others to reference) Member checks

Transferability(qualitative research equivalent to external validity) Thick description Adequate data base for others to make transferability judgements possible; to make

applications to other situations

Dependability

(qualitative research equivalent to reliability) Triangulation Multi-site research design/teams of researchers in close communication Audit trail

Confirmability

(qualitative research equivalent to objectivity) Reflexive journal Audit trail Triangulation

From a constructivist perspective:

Trustworthiness: credibility, transferability, dependability, confirmabilityAuthenticity: fairness, enlarges personal constructions, leads to improved understanding

of constructions of others, stimulates to action, empowers to action

Lincoln, Y. & Guba, E. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.

7


8/8

Triangulation & Member Validation

These techniques were originally proposed to assist in converging on a single trueversion. They may also generate multiple perspectives.

Triangulation:

1. diverse sources of data (find phenomenon in several different settings; differentpoints in space & time)

2. investigator triangulation: multiple observers in the field; team research; engagementin continued discussion

3. theory triangulation: approach data with several hypotheses in mind to see whathappens

4. methodological triangulation: (eg ethnography + interviews; mixed methods)

Member validation:

1. Use researchers concepts to predict members descriptions

2. Show that the researchers account can lead to successful passing as a member.3. Ask members to judge the adequacy of the researchers account:

Strong version: members evaluate the final report Weak version: members comment on the accuracy of some interim

document (eg. interview transcript).

4. Regard successful action research as a form of member validation.

Seale, C. (1999). The quality of qualitative research. London: Sage.

8

handa. validity in qual research

Documents