Developing a Review Protocol, Searching and Data Extraction

Trudy Bekkering, PhDCenter of Evidence-Based Medicine &

Belgian Branch of the Dutch Cochrane CenterCentre for Methodology of Educational Research

Katholieke Universiteit Leuven, Belgium

This presentation is partly based on previous presentations of Oliver Wendt and Chad Nye

Phases Campbell Review Process

1. Register title2. Develop protocol (project plan)3. Undertake systematic review4. Publish systematic review

http://www.campbellcollaboration.org/systematic_reviews/index.php

The Review Protocol

• A priori statement of aims and methods of the review

• Research question(s), aims, methods are considered in advance of identifying the relevant literature– Conduct review with minimal bias– Access to peer review– Greater efficiency in review process

(Torgerson, 2003)

Review Protocol Advantages

• Clear research question before the reviewAvoid retrieving irrelevant papers• A priori inclusion and exclusion criteria avoid changing

criteria as review progresses or studies may be included on basis of their results

• If decisions are explicit it enables them to be justified• Develop protocol as independently as possible from the

literature avoid influence by one or two major studies, less bias (Torgerson, 2003)

Review Protocol Construction

• “The review protocol should ideally be conceived by a group of reviewers with expertise both in the area and the science of research synthesis” (Egger & Davey-Smith, 2001)

• Prior knowledge of an area can aid the review process, but can also introduce bias to the review can be reduced by clear, consistent and logical justifications for inclusion criteria

Contents Protocol• Title and authors• Background• Objectives• Methods

– Searching for studies– Selecting studies– Data extraction (including assessment study quality)– Data analysis

• Acknowledgements / conflicts of interests• References• Tables / Figures

Background

• Describes the context of the review: why is it important?– Description of the problem– Description of intervention– How the intervention might work– Description potential subgroups

• Clarify conceptual issues central to the review

Objective(s)

• Precise statement of objective, using PICO(S)– Which population?– Which intervention(s)?– Which comparison(s)?– Which outcome(s)? – (Which study design?)

A well formulated objective focuses the review

Example objective

• To assess the effects of • home-based programmes aimed at • improving developmental outcomes for • pre-school children from disadvantaged

families

(Miller & Eakin, 2011)

Methods

• How to select studies?• How to search for studies?• Assessment quality of included studies• How to extract data?• How to analyse results?

Selecting studies

• Define inclusion and exclusion criteria• Key features:

– Established a priori– Explicit (explain all terms)– Based on PICO– Include study design, if applicable

• All retrieved studies are listed in the review either under included or excluded together with a justification

Example inclusion criteria

Types of studies• Randomised controlled trials. The control group will

either receive no intervention or standard care. Studies comparing two different types of home based programme without a control group will be excluded.

• This is an area where RCTs are technically and ethically possible. As such this review is confined to RCTs as they provide the best evidence of effectiveness.

(Miller & Eakin, 2011)

Types of participants• Home based child development interventions for pre-school

children from socially disadvantaged families• Parents with children up to the age of school entry and who

are socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of school entry can vary between countries (4 to 7 years) and so the upper age range for this review will be the school entry age for the country in which the trial took place.

• Studies will be excluded if they aim to recruit particular clinical subgroups of parents.

Example (cont)

Example (cont)

Types of interventions• Home based interventions designed to improve child

intellectual and socio-emotional development through the provision of relevant knowledge and skills to the parent.

• The intervention is delivered by trained lay or professional family visitors.

Example (cont)

Types of outcome measuresPrimary outcomesThe following child outcomes will be included:• Cognitive development (including language

development and attention)• Socio-emotional development (including self

regulation and behavioural development)Adverse outcomes:• Parents feel disempowered

Methods

• How to select studies?• How to search for studies?• Assessment quality of included studies• How to extract data?• How to analyse results?

Searching for studies– Preliminary Searches

• Supports beginning steps: Definition of key concepts & research question

• Use of standard reference tools and broad searches for review articles and key primary studies

– Main Searches• Identification of primary studies through searches of databases,

Web, branching, manual searches– Final Searches

• Refine search terms and update original searches

Examples of Databases

– Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Full-text Collection; Education Full text, Linguistics and Language Behavior Abstracts.

– Psychology: PsycINFO, PubMed (Medline), Ageline, Psychology: A SAGE Full-Text Collection, Criminology: A SAGE Full-Text Collection

– Sociology: Sociological Abstracts, Contemporary Women’s Issues. Sociology: A SAGE Full-text Collection

– Multidisciplinary: Academic Search Premier, ProQuest Dissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science

Main Searches: Decisions

– Preparation of Search Strategies• What are the key concepts to be searched?• How are these represented in each discipline?• What are their related terms?• How are these key concepts represented in the

controlled vocabulary within each database to be searched?

Sample Research Question

Approaches to Parent Involvement for

Improving the Academic Performance of

Elementary School Age Children

Chad Nye, Herb Turner, Jamie Schwartz

The purpose of this review is to determine the effectiveness of parental involvement in improving the academic performance of school age children in grades K-6.

Using a Thesaurus

1. From the research question, determine the main concepts to be searched (usually there are three):

– Intervention: Parental involvement– Outcome: Academic performance– Population: Kindergarten or Elementary students

2. Consult the main database to be searched.3. Look up each concept in the thesaurus for this

database.– A thesaurus is an alphabetical listing of the controlled vocabulary (or

descriptors) used within a subject database– A hierarchical arrangement is used so that Broader, Narrower and

Related headings may be discovered– The user will be sent from invalid headings to valid headings.

Example: ERIC

• Selecting the ERIC Descriptors– Descriptors: Parental Involvement See: Parent participation– Related descriptors: Family involvement, Parent-school

relationship, Parent role, Parents as teachers– Related keywords: parent* involvement, parent* effectiveness,

parent* support, family support

– Descriptors: Academic Performance See: Academic achievement

– Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains

– Descriptors: Elementary School Children See: Elementary school students

– Related descriptors: Elementary education, Primary education, Kindergarten

Intervention: Parental involvement

Outcome: Academic performance

Population: Kindergarten or Elementary students

Main Searches: Decisions

• Construction of the Search Statements– What terms should be searched as descriptors or as

“keywords”? – What Boolean operators should be used? – Where should truncation characters be used? (e.g.

parent* will retrieve parent, parents, parental)– What limiting features are available to narrow results?

(e.g. use of Publication Type codes, time period, language)?

Boolean Operators

AND: Both terms must be present in order for a record to be retrieved. Used to combine different concepts.

e.g parent participation AND achievement

OR: Either term may be present in order for a record to be retrieved. Used to search for related terms or synonyms.

e.g. parent OR family

NOT: Used between two terms to ensure that the second term will not appear in any of the results.

e.g. literacy NOT adult(Parental involvement OR parent participation) AND

academic achievement AND (elementary OR primary education)

Example: ERIC, cont’d

• Combining Keywords/Descriptors using Boolean operators:

1. DE=(Parent participation OR Family involvement OR Parent role OR Parent-school relationship OR Parents as teachers)

2. “Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support”

3. #1 OR #2 = 28,958 records4. DE=(Academic achievement OR Science achievement OR

Reading achievement OR Achievement gains) – 46,574 records

5. DE=(Elementary school students OR elementary education OR elementary schools OR primary education OR kindergarten) - 291997 recs

6. #3 AND #4 AND #5 = 1,669 records

Limiting Your Results

Using the Limiting Commands:• Limiting fields contain information that is common to a large

number of records within a database. These include language, document type, publication year and so.

• Some limiting fields will vary across databases (e.g. Classification Code, Age Group)

• Decisions about whether you are going to restrict:• The language of the document

• The search to a certain time period

• The results to empirical studies only

Example: ERIC, cont’d

Using Limits in ERIC:

1. Search by DT= research -reports2. Combine this search with the previous one

– 835/1669 recs• Review Date Range• Review Language

One final comment…• Locating ERIC documents: http://www.eric.ed.gov/

ERIC recordAUTHOR : Polovina,-Nada; Stanisic,-JelenaTITLE: A Study on Family-School Cooperation Based on an Analysis of School DocumentationPUBLICATION YEAR: 2007SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p115-133ERIC DOCUMENT LINK: http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED499159DOCUMENT TYPE: Journal-Articles; Reports-ResearchLANGUAGE: EnglishDESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent-

Teacher-Cooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic-Achievement; Parent-Influence; Parent-Child-Relationship

IDENTIFIERS: Serbia-ABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of

different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, the relationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No. 149001 (2006-2010), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)

Next Steps

Repeat these steps for each database to be searched.

Additional Retrieval Methods– The web…to locate grey literature

• Use Advanced Search screens on large engines (eg. Google, Altavista, AlltheWeb, MSN Live)

• Consult specific sites

– Branching (Pearl Growing)• Scan the bibliographies of previous reviews on your topic

– Manual searches• Browse the Table of Contents of key journals for current years

Managing Your Results

• Export the results– Save as a Text file

• Import into a bibliographic management software:– RefWorks, – Reference Manager, – EndNotes

• Edit your inhouse database– Add Source code for each database searched (e.g ERIC1,

PsycINFO1…)– Add notes to the records (e.g.includes vs excludes)

• Compile a Search History document listing the original search strategies

– Use of IRMG/ECG Database worksheet

Information Retrieval: Wrap Up

• Importance of information retrieval process– Not a “one-shot”deal– Requires expertise in the planning and implementation of

searches– Consulting with the Trials Search Advisor or an Information

Specialist is highly recommended

• Use a bibliographic management software– Store, manage and organize results

• Must have ability to replicate review– Documentation of entire process, including search strategies used

for each database, decisions taken, etc.

Methods

• How to select studies?• How to search for studies?• Assessment quality of included studies• How to extract data?• How to analyse results?

Data Extraction

What is data extraction?• Process of reading through a study and extracting the

relevant information from each study• The reviewer fills out an data extraction sheet with

the appropriate information taken from that study • Data extraction sheet may be on paper or

electronically

Extraction Form

• Use close-ended items as much as possible (end goal may be to build a database)

• Design for easy use– Cluster items that deal with similar themes and

format so that they are easy to complete(Lipsey & Wilson, 2001)

Extraction Form (cont.)

• Extraction form consists of two different parts1. characteristics that apply to the entire study2. results: studies typically have multiple outcomes

and therefore different effect sizes

Extraction Manual

• Essential for data extraction process • Specifies and describes what study features should

be extracted• Helps to minimize error and bias in the judgments of

the extraction process• Facilitate the consistency of extraction• Should be pilot-tested with several articles and

revised accordingly before actual use

Extraction Manual (cont.)

• Have a full definition of each item and guidelines for extraction– Assists the coder in handling ambiguous, unusual

cases• Bottom line: spell everything out in detail, give

full definitions of the various response options, and provide guidelines for dealing with borderline cases

Assessment study quality

• Study quality may affect conclusions• A systematic review always includes an

assessment of individual study quality• Different tools available• Describe how to use this information

– Description methodological quality– Exclude studies of poor quality?– To include in conclusions (strong versus weak

evidence)

Example data extraction form

Example. Study characteristics

1. Research design• Non-randomized• Randomized study

• Pseudo-randomized• Cluster-randomized

2. Setting• Public school• Private school

3. Etc.

Example. Population characteristics

For each group:1. Number of participants that were enrolled2. Number of participants that were randomized3. Number of participants that completed study4. Age5. Gender6. Ethnicity7. Baseline reading score 8. Etc.

Example. Intervention characteristics

1. Intervention offered to intervention group• Intensity• Duration• Implementation integrity

2. Intervention offered to control group• Intensity• Duration• Implementation integrity

Extraction study results

• Descriptive or other statistical information contained in studies needs to be converted into a standard metric by which studies can be compared

• Continuous outcomes: mean and standard deviation in the groups

• Dichotomous outcomes: numbers in the groups having the outcome versus the numbers in the groups not having the outcome

Extraction Form Example

• See example in your package

A. Publication Source1. Journal 2. Dissertation 3.Thesis 4. Unpublished Paper 5. Book 6. Other: ________

B. Subject Characteristics (page(s): ____________)Group Pretest Posttest Attrition Age Grade(s) Males (n) (n) (n) (n) (n) (n)

Tx: _22___ _22__ ______ ______ ______ _____ Cp: ______ _____ ______ ______ ______ _____

Co: _22___ __18_ __ 4___ ______ ______ _____

C. Socioeconomic Status (page(s):___________

1. Low 2. Middle 3. Upper

4. Low-Middle 5. Middle Upper 6. Labeled Mixed

7. Unclear 8. ___________ 9. NR

Comments:__________________________________

Etc., etc.

Bias in data-extraction• “Data extraction bias”: reviewer introduces bias into the

review– Studies in accord with own view– Different judgments of quality or methodological adequacy to

different studies– Reviewer’s awareness of the study authors, the journal, the results

blinding the reviewers– Using multiple reviewers– Having detailed extraction manual– Validating the data extraction

• Checked by another person• Two people working independently (much less common but likely to be more

reliable) (Petticrew & Roberts, 2006)

Reliability of extraction• Reliability has two dimensions:1. Consistency of a single reviewer2. Consistency between different reviewers• To check draw subsample of studies, have reviewers extract

them again and compare the results• For single reviewer reliability: this person needs to extract

studies again after sufficient time has passed so that no memory exists of original extraction

• For between reviewer reliability: different reviewers extract the same sample without reference to what the other has done (Lipsey & Wilson, 2001)

Reliability of extraction (cont.)• Once the appropriate double extraction has been done on a

sample, the two sets of results are compared item by item• Calculate reliability coefficients:

– For continuous variables use Pearson’s r (e.g., participant’s age, effect size values)

– For categorical variables use percentage agreement procedure or Cohen’s kappa (e.g., type of intervention, outcome measure, design)

• Report actual reliability, then resolve discrepancies (Lipsey & Wilson, 2001)

Training the reviewers

• Effective reviewer must– Understand extraction protocol in detail– Have the knowledge and skills to properly read and

interpret research report– Have background in methodology– Be familiar with specific research domain

• Pilot review is recommended– Carefully read the manual– Extract pilot studies and compare results– Discuss disagreements and improve manual

Protocol on methods

Protocol contains • Data extraction form, including manual • Method of extraction

– Paper or electronically (which software)• How many reviewers• How are reviewers trained• How to resolve disagreements

Summary

• Review protocol important first step when undertaking systematic review

• Helps to focus and structure the review• Limits the scope for bias• Use standard format – C2 format is recommended

http://www.campbellcollaboration.org/education_articles/ECG_Resources_for_Reviewers.php

Examples of C2 Review Protocols are available on:http://www.campbellcollaboration.org/library.php

References / more reading• Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic

reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2nd ed., pp. 23-42 ). London, UK: BMJ Books.

• Frost LA & Bondy AS. (2002). The Picture Exchange Communication System. Newark, DE: Pyramid Educational Products.

• Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.• Miller S & Eakin A. Home based child development interventions for pre-school

children from socially disadvantaged families (Protocol). The Campbell Library, 2011.

• Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing.

• Torgerson C. (2003). Systematic Reviews. London, UK: Continuum.• Resources Campbell Collaboration: http://www.campbellcollaboration.org/• education_articles/ECG_Resources_for_Reviewers.php• Resources Cochrane Collaboration: http://

www.cochrane.org/training/cochrane-handbook