chapter 018

1Copyright © 2013, 2009, 2005, 2001, 1997 by Saunders, an imprint of Elsevier Inc.

Chapter 18

Critical Appraisal of Nursing Studies


Critical Appraisal of Research

An examination of all aspects of a research report Systematic Unbiased Careful


Critical Appraisal of Research (Cont’d)

Purpose is to judge the study’s Strengths Weaknesses Meaning Significance


Evolution of Critical Appraisal of Research in Nursing

Originally intended to focus on limitations Followed both publication and oral presentation

Rare after 1960s in nursing, but still in existence in multidisciplinary publications

Skills in critical appraisal now are introduced at baccalaureate level of nursing education


Critical Appraisal of Research in Nursing

Critical appraisal of studies is essential In quantitative research, issues are credibility

and generalization Construct validity Internal validity Statistical conclusion validity External validity


Critical Appraisal of Research in Nursing (Cont’d)

Important questions: Methodology: was research sound enough to

produce credible findings? Believability: do findings reflect reality? What are the study’s strengths? Nursing’s knowledge base: are findings consistent

with those of previous studies?


Critical Appraisal of Research in Nursing (Cont’d)

Important questions (Cont’d) : Are there implications for

• Increasing nursing’s knowledge base?

• Generating theory (qualitative)?

• Application to practice?

What are the limitations and flaws?• How do they affect how the study is understood,

believed, applied?


When are Critical Appraisals of Research Implemented in Nursing?

As a nurse When reading journal articles (informal process)

In undergraduate and graduate coursework As free-standing assignments In order to perform research synthesis As part of a thesis or dissertation—literature

review


When are Critical Appraisals of Research Implemented in Nursing? (Cont’d)

At research presentations After a study is presented

In journals The commentary pages following new research

(some journals) Peer review

For conferences (abstract review) For publication (manuscript review) For potential funding (proposal review)


Critical Appraisal of Studies by Students

Three levels Identifying elements or steps of study Determining study strengths and limitations Evaluating credibility and meaning of study

findings


Critical Appraisal of Research by Nurses

Performed in conjunction with reading research articles in professional journals (must be critical appraisal, not merely acceptance of the printed word)

Will enable practicing nurses to synthesize most credible, significant, and appropriate evidence for use in their practice


Critical Appraisal of Research by Educators

Part of curricular content update Role modeling for students Collaboration with other faculty in research Mentoring of students who are conducting

research (usually graduate students)


Critical Appraisal of Studies By Researchers

Self-critique related to research trajectory Critique of literature in one’s area, to update

one’s literature review


Critical Appraisal of Research Presentations and Publications

Useful to the author Generates ideas for subsequent research

Commentaries following published papers Letters to the editor


Critical Appraisal of Abstracts for Conference Presentations

Performed by a committee Several criteria

Is the presentation suitable for this conference? Is research still in progress, or has it been

completed? Are the study problem, purpose, methodology, and

results logical?


Critical Appraisal of Abstracts for Conference Presentations (Cont’d) Several criteria (Cont’d)

What are the study’s implications?• For nursing’s knowledge base

• For theory development (qualitative)

• For practice

Is the writing clear and concise? Are proper parts of the abstract present?


Critical Appraisal of Research Articles For Publication

Performed by nurse researchers and faculty who have conducted research

Is research well designed, and does it contribute to the body of knowledge?

Two or three reviewers; anonymous process


Critical Appraisal of Research Proposals

For approval of proposed research Academic Institutional

To permit data collection in an institution For funding of proposed research

Intramural (either university or hospital) Private Federal


NURSES’ Expertise in Critical Appraisal

Educational Level Expected Level of Expertise in Critical Appraisal of Research

Baccalaureate Identify the steps of the quantitative research process in a study.

Identify the elements of a qualitative study.

Master's Determine study strengths and weaknesses in quantitative and qualitative studies.

Evaluate the credibility and meaning of a study and its contribution to nursing knowledge and practice.

Doctorate or Post-doctorate

Synthesize multiple studies in systematic reviews, meta-analyses, meta-syntheses, and mixed-methods systematic reviews.


Critical Appraisal Process for Quantitative Research

Three steps: performed one at a time, or simultaneously Step I: Identifying the steps of the research

process in studies Step II: Determining study strengths and

weaknesses Step III: Evaluating the credibility and meaning of

a study to nursing knowledge and practice


Step I: Identifying the Steps of the Research Process in Studies

Understanding terms and concepts in report Identifying study elements and grasping

nature, significance, and meaning of these elements


Guidelines for Identifying the Steps of the Research Process

Review the abstract Read the entire study Review these questions

Does title clearly identify the focus of the study by including the major study variables and the population?

Does title indicate type of study conducted?


Guidelines for Identifying the Steps of the Research Process (Cont’d) Review these questions (Cont’d)

Was abstract clear? Was writing style of report clear and concise? Were parts of research report plainly identified? Were relevant terms defined?


Guidelines for Writing a Critical Appraisal

Introduction Describe qualifications of authors Discuss clarity of article title Discuss quality of abstract

State the problem Significance of the problem Background of the problem Problem statement


Guidelines for Writing a Critical Appraisal (Cont’d)

State the purpose Examine the literature review

Are relevant previous studies and theories described?

Are the references current? Are the studies described, critically appraised, and

synthesized? Is a summary provided of the current knowledge

(what is known and not known) about research problem?



Examine the study framework or theoretical perspective Is the framework explicitly expressed or must

reviewer extract the framework from implicit statements in introduction or literature review?

Is the framework based on tentative, substantive, or scientific theory?

Does the framework identify, define, and describe relationships among concepts of interest? Provide examples.



Examine the study framework or theoretical perspective (Cont’d) Is a model of framework provided for clarity? If

model is not presented, develop one and describe it

Link study variables to relevant concepts in the map

How is framework related to nursing's body of knowledge?



List any research objectives, questions, or hypotheses

Identify and define (conceptually and operationally) study variables or concepts

Indicate the study’s variables: Independent variables Dependent variables Research variables or concepts



Identify attribute or demographic variables and other relevant terms

Identify the research design Identify the specific design of the study. Draw a

model. Does the study include a treatment or

intervention? If so, is the treatment clearly described with a protocol and consistently implemented?



Identify the research design (Cont’d) If study has more than one group, how were

subjects assigned to groups? Were the extraneous variables identified and

controlled for? Were pilot study findings used to design study? If

so, briefly discuss pilot and changes made in study based on pilot.



Describe sample and setting Identify inclusion or exclusion sample or eligibility

criteria Identify the specific type of probability or

nonprobability sampling method used to obtain the sample. Did the researchers identify a sampling frame for the study?



Describe sample and setting (Cont’d) Identify the sample size. Discuss the refusal rate

and/or percentage, and include the rationale for refusal if presented in the article. Discuss the power analysis if this process was used to determine sample size.

Identify sample attrition (number and percentage) for the study.



Describe sample and setting (Cont’d) Identify characteristics of the sample. Discuss IRB approval. Describe the informed

consent process used in study. Identify the study setting and indicate if it is

appropriate for study purpose.



Identify/describe each measurement strategy used in study Identify each study variable that was measured Identify name and author of each measurement

strategy Identify type of each measurement strategy Identify validity of each scale and accuracy of

physiologic measures



Identify/describe each measurement strategy used in study (Cont’d) Identify level of measurement (nominal, ordinal,

interval, ratio) achieved by each measurement method used in study.

Describe reliability of each scale for previous studies and this study. Identify the precision of each physiological measure.



Describe procedures for data collection Describe statistical techniques conducted to

analyze study data List statistical procedures conducted to describe

sample Was level of significance or alpha identified? If so,

indicate what it was (0.05, 0.01, 0.001, and so forth)



Describe the statistical techniques conducted to analyze study data (Cont’d) Identify focus (description, relationships, or

differences) for each analysis technique List statistical analysis technique performed Provide specific results Identify probability (p) of statistical significance

achieved by the result



Describe researcher’s interpretation of findings Are findings related back to study framework? If

so, do findings support study framework? Which findings are consistent with those

expected? Which findings were not expected? Are the findings consistent with previous research

findings?



What study limitations did researcher identify?

How did researcher generalize the findings? What were the implications of findings for

nursing practice? What suggestions for further study were

identified? Is description of the study sufficiently clear for

replication?


Step II: Determining Study Strengths and Weaknesses

Comparing study with ideal research report Formulate some idea of whether the

compared difference Is none or slight (a strength) Is substantial

If substantial, is this poor writing or poor research design?

Are the logical links in the study substandard? Do the parts match?


Guidelines for Determining study Strengths and Weaknesses

Read the following questions and make judgments about elements or steps in study

Provide a rationale for your decisions Document from relevant research sources


Research Problem and Purpose

Is problem sufficiently delimited in scope that it is researchable but not trivial?

Is problem significant to nursing and clinical practice?

Does purpose narrow and clarify aim of study?

Was study feasible to conduct in terms of funding; researchers’ expertise; availability of subjects, facilities, and equipment; and ethical considerations?


Review of Literature

Is literature review organized to demonstrate progressive development of evidence from previous research?

Is a theoretical knowledge base developed for the problem and purpose?


Review of Literature (Cont’d)

Is a clear, concise summary presented of current empirical and theoretical knowledge in area of the study?

Does literature review summary identify what is known and not known about research problem and provide direction for formation of the research purpose?


Study Framework

Is framework presented with clarity? Is model/conceptual map of framework adequate to explain phenomenon of concern?

Is framework linked to the research purpose? If not, would another framework fit more logically?


Study Framework (Cont’d)

Is framework related to body of knowledge in nursing and clinical practice?

Do framework elements really reflect study variables?

If a proposition or relationship from a theory is to be tested, is the proposition clearly identified and linked to study hypotheses?


Research Objectives, Questions, or Hypotheses

Are objectives, questions, or hypotheses expressed clearly?

Are objectives, questions, or hypotheses logically linked to the research purpose?


Research Objectives, Questions, or Hypotheses (Cont’d)

Are hypotheses stated to direct the conduct of quasi-experimental and experimental research?

Are objectives, questions, or hypotheses logically linked to concepts and relationships (propositions) in framework?


Variables

Are variables reflective of concepts identified in framework?

Are variables clearly defined (conceptually and operationally) and based on previous research or theories?

Is the conceptual definition of each variable consistent with the operational definition?


Design

Is design used in study the most appropriate design to obtain needed data?

Does design provide a means to examine all objectives, questions, or hypotheses?


Design (Cont’d)

Is the treatment clearly described? Is the treatment appropriate for examining the study purpose and hypotheses? Does study framework explain the links between the treatment (independent variable) and the proposed outcomes (dependent variables)?


Design (Cont’d)

Was a protocol developed to promote consistent implementation of treatment to ensure intervention fidelity? Did researcher monitor implementation of treatment to ensure consistency? If treatment was not consistently implemented, what might be the impact on the findings?


Design (Cont’d)

Did researcher identify threats to design validity and minimize them as much as possible?

Is design logically linked to sampling method and statistical analyses?


Design (Cont’d)

If more than one group is used, do groups appear equivalent?

If a treatment was implemented, were subjects randomly assigned to treatment group or were treatment and comparison groups matched? Were the treatment and comparison groups assignments appropriate for purpose of the study?


Sample, Population, and Setting

Is sampling method adequate to produce a representative sample?

What are the potential biases in the sampling method? Are any subjects excluded from study because of age, socioeconomic status, or ethnicity without a sound rationale?

Did sample include an understudied population, such as young, elderly, or a minority group?


Sample, Population, and Setting (Cont’d)

Were sampling criteria (inclusion and exclusion) appropriate for type of study conducted?

Is sample size sufficient to avoid a type Ii error? Was a power analysis conducted to determine sample size? If a power analysis was conducted, were the results of the analysis clearly described and used to determine final sample size? Was attrition rate projected in determining final sample size?


Sample, Population, and Setting (Cont’d)

Were the rights of human subjects protected? If so, in what ways?

Is setting typical of clinical settings? Was refusal rate a problem? If so, how might

this weakness influence findings?


Sample, Population, and Setting

Was sample attrition a problem? Did researchers provide a rationale for

attrition of study participants? How did attrition influence final sample and

study results and findings? (Did a random sample become a convenience sample due to excessive attrition or refusal?)


Measurements

Do measurement methods selected for study adequately measure study variables?

Are measurement methods sufficiently sensitive to detect small differences between subjects? Should additional measurement methods have been used to improve quality of study outcomes?


Measurements (Cont’d)

Do the chosen measurement methods have adequate validity and reliability? What additional reliability or validity testing is needed to improve quality of measurement methods?



Scales and questionnaires Are the instruments clearly described? Are techniques to complete and score the

instruments provided? Are validity and reliability of instruments

described? Did researcher reexamine validity and reliability of

instruments? If an instrument was developed for study, is

instrument development process described?



Observation Is what is to be observed clearly identified and

defined? Is interrater reliability described? Are techniques for recording observations

described?



Interviews Do interview questions address concerns

expressed in research problem? Are interview questions relevant for research

purpose and objectives, questions, hypotheses? Does design of questions tend to bias subjects’

responses? Does sequence of questions tend to bias subjects’

responses?



Physiological measures Are physiological measures/instruments clearly

described? If appropriate, are brand names, of instruments identified?

Are accuracy, precision, and error of physiological instruments discussed?



Physiological measures (Cont’d) Are physiological measures appropriate for the

research purpose and objectives, questions, or hypotheses?

Are methods for recording data from physiological measures clearly described? Is the data recording consistent?


Data Collection

Is data collection process clearly described? Are forms used to collect data organized to

facilitate computerizing the data? Is training of data collectors clearly described

and adequate?


Data Collection (Cont’d)

Is data collection process conducted in a consistent manner?

Are data collection methods ethical? Do data collected address research

objectives, questions, or hypotheses? Did any adverse events occur during data

collection, and were these appropriately managed?


Data Analysis

Are data analysis procedures appropriate for type of data collected and clearly described?

Did researcher address missing data? How was missing data managed?

Do data analysis techniques address study purpose? Research objectives, questions, or hypotheses?

Are results presented in an understandable way by narrative, tables, or figures, or a combination of methods?


Data Analysis (Cont’d)

Are statistical analyses logically linked to design?

Was sample size sufficient to detect significant differences if they had been present?

Was a power analysis conducted for non-significant results?

Are results interpreted appropriately?


Interpretation of Findings

Are findings discussed in relation to each objective, question, hypothesis?

Are various explanations for significant and non-significant findings examined?

Are findings clinically significant? Are findings linked to study framework?


Interpretation of Findings (Cont’d)

Are study findings an accurate reflection of reality, and valid for use in clinical practice?

Do conclusions fit results from data analyses? Are conclusions based on statistically significant and clinically important results?

Does study have limitations not identified by researcher?


Interpretation of Findings (Cont’d)

Did researcher generalize findings appropriately?

Were identified implications for practice appropriate, based on study findings and findings from previous research?

Were quality suggestions made for future research?


Step III: Evaluating a Study

Determination of validity, credibility, significance, and meaning of a study

Accomplished by examining links To those within study process, relative to findings To those that depend on previous studies in same

area Did author take research in a direction

supported by previous research?


Step III: Evaluating a Study (Cont’d)

Are strengths of the study sufficient? How do weaknesses of study undermine

author’s discussion? Principal findings Conclusions Limitations Implications Suggestions for subsequent research


Guidelines for Evaluating a Study

Questions to ask: Are there any other hypotheses that explain the

findings? Did author identify these? Do findings seem valid? Believable? To what populations can findings be generalized? What remains unclear? Did author identify these

points?


Guidelines for Evaluating a Study (Cont’d)

Questions to ask (Cont’d) : What would have decreased/eliminated study

limitations? Based on this, what subsequent research is

indicated: what comes next? What is the state of the body of knowledge, now,

with these findings?



Based on previous literature, did author use previous work to Generate up-to-date problem statement? Derive a purpose that advances enquiry in this

area? Improve on previous studies’ designs? Select a sample that was more representative?

More inclusive? Select or devise better measurement strategies?



Based on previous literature, did author use previous work to Select appropriate statistical tests and apply them

correctly Analyze findings so that they build on those of

previous studies? Identify current knowledge in this area? State implications for practice, body of nursing

knowledge, or theory?



Include a final summary of quality of the research report Does study contribute to the body of knowledge,

and in what way? Current state of knowledge in this area: are

practice applications appropriate now?


Critical Appraisal Process: Qualitative Studies

Requires a different approach than steps used for quantitative study

Three prerequisites in applying rigorous appraisal standards An appreciation for philosophical foundation of

qualitative research Basic knowledge of major qualitative approaches Empathy for participant’s perspective


Critical Appraisal Process: Qualitative Studies (Cont’d)

Problem Statement Is research problem explicit? Why was this study needed? Why did researcher care about this topic? Do we

know?



Purpose and Research Questions Does purpose make sense, in light of the

problem? What are explicit or implied research questions? Are those questions related to problem and

purpose? Does qualitative inquiry make sense for these

questions?



Literature Review Were both quantitative and qualitative studies

cited? Were other types of literature reviewed? Are references current? Does collective authorship of referenced articles

transcend nursing? Did author evaluate available literature? Does synthesized information from literature

review reveal a gap?



Philosophical Foundation of the Study Method/submethod stated or implied? Is source for philosophical foundation primary or

secondary? Were methods of study congruent with this

philosophy and research tradition?



Sampling and Sample How were study participants selected? Where were they recruited? Did sites fit sampling

needs of study? Inclusion and exclusion criteria What is the “n”? What were refusal rate and the attrition rate?



Data Collection How did researchers collect data? More than one method of data collection? Any rationale for data collection choices? Span of time for data collection? One interview or more? Participant review of typed transcripts? What changes in data collection methods were

made in response to context and early information obtained?



Protection of Human Study Participants Informed consent Planned action, should the interviews reveal

disturbing content—counselor availability? Contact numbers?

Benefits and risks of participation ID’d by authors Any adjustments to recruitment, consent, data

collection, and analysis to prevent participant distress?



Data Management and Analysis How were data managed and analyzed? How was rigor of process maintained? Was data management/analysis method

consistent with purpose and data? Measures to minimize or allow for effects of

researcher bias, such as dual coding (not used in all methods), expert review, participant validation, bracketing



Findings Did findings address purpose? Did researcher go Beyond mere reporting of

quotations and actually interpret? Were themes or essences identified? Data congruency with interpretations:

• Did quotations match themes/essences?

• Did researcher attempt to address variations in findings?



Discussion Any new information produced? Findings discussed, in comparison with other

studies or literature? Implications of findings addressed:

• Body of nursing knowledge?

• Clinical application?

• Policy application?

• Theory generation?



Logic and Form of Findings Were readers able to “hear the voice” of

participants and gain understanding of phenomenon studied?

Were elements of research report clearly identifiable by reader?

Were study findings, purpose, and method congruent with article’s presentation?

Was there logic, and organization, in the way findings were presented?



Evaluation Summary Philosophical congruence: Were development and

implementation of study congruent with philosophical foundation of study?

Methodological coherence: Did data collection, analysis, and interpretation processes fit together to form a coherent approach to address the research problem?



Evaluation Summary (Cont’d) Intuitive comprehension: Do findings provide a

credible reflection of reality and expand reader’s comprehension of the study topic? If so, how can findings be used in nursing practice?

Intellectual contribution: What do findings contribute to current body of knowledge?

chapter 018

Documents

imprint of elsevier

new research

quantitative research

research sound

research synthesis

research presentations

study findings

study strengths