15 de abril de 2008

15 de Abril de 200815 de Abril de 2008

A Meta-Analysis is a review in which bias has been reduced by the systematic identification, appraisal, synthesis and statistical aggregation of all relevant studies on a specific topic according to a predetermined and explicit method.

A Meta-Analysis is a review in which bias has been reduced by the systematic identification, appraisal, synthesis and statistical aggregation of all relevant studies on a specific topic according to a predetermined and explicit method.

OverviewOverviewSystematic ReviewSystematic Review Meta-AnalysisMeta-Analysis

In 1987, a survey showed that only 24, out of 86 English-language meta-analyses, reported all the six areas considered important to be part of a meta-analysis:

In 1987, a survey showed that only 24, out of 86 English-language meta-analyses, reported all the six areas considered important to be part of a meta-analysis:

Study DesignStudy DesignStudy DesignStudy Design

Control of BiasControl of BiasControl of BiasControl of Bias

Statistical AnalysisStatistical AnalysisStatistical AnalysisStatistical Analysis

Application of ResultsApplication of ResultsApplication of ResultsApplication of ResultsSensitivity AnalysisSensitivity AnalysisSensitivity AnalysisSensitivity Analysis

CombinalityCombinalityCombinalityCombinality

In 1992 this survey was updated with 78 meta-analyses and the researchers noted that methodology has definitely improved since their first survey;

However it needed better searches of the:

Literature;Quality evaluations of trials;Synthesis of the results.

In 1992 this survey was updated with 78 meta-analyses and the researchers noted that methodology has definitely improved since their first survey;

However it needed better searches of the:

Literature;Quality evaluations of trials;Synthesis of the results.

So, in 1999, several researchers suggested and created the Quality of Reporting of Meta-Analyses (QUOROM) Statement to improve and standardise reporting.

The QUOROM Statement – that includes a checklist and a trial flow diagram – describes the preferred way to present the different sections of a report of a Meta-Analysis;

It is organized into 21 headings and subheadings.

So, in 1999, several researchers suggested and created the Quality of Reporting of Meta-Analyses (QUOROM) Statement to improve and standardise reporting.

The QUOROM Statement – that includes a checklist and a trial flow diagram – describes the preferred way to present the different sections of a report of a Meta-Analysis;

It is organized into 21 headings and subheadings.

The number of published meta-analyses has increased over time.

According to a study, after the QUOROM statement the estimated mean quality score of the reports increased from 2.8 (95% CI; 2.3–3.2) to 3.7 (95% CI; 3.3–4.1), that represented an estimated improvement of 0.96 (95% CI; 0.4–1.6, p = 0.0018 two

sided t-test).

However, the QUOROM group admits itself that this checklist requires continuous research in order to improve the quality of a meta-analysis.

The number of published meta-analyses has increased over time.

According to a study, after the QUOROM statement the estimated mean quality score of the reports increased from 2.8 (95% CI; 2.3–3.2) to 3.7 (95% CI; 3.3–4.1), that represented an estimated improvement of 0.96 (95% CI; 0.4–1.6, p = 0.0018 two

sided t-test).

However, the QUOROM group admits itself that this checklist requires continuous research in order to improve the quality of a meta-analysis.

But what is Reproducibility?

Why is it so important?

Reproducibility is one of the main principles of the scientific method, which refers to the ability of a

test or experiment to be accurately reproduced by someone else working independently.

But what is Reproducibility?

Why is it so important?

Reproducibility is one of the main principles of the scientific method, which refers to the ability of a

test or experiment to be accurately reproduced by someone else working independently.

The lack of reproducibility can lead to major consequences:

•a failure in the reproducibility will most probably end in results' heterogeneity;

•at a clinical level, if a diagnostic test is not reproducible there is the risk of a patient being wrongly diagnosed;

•non-reproducible items of a checklist can lead to a decrease on its credibility and, consequently, of the meta-analyses that used it as a model.

The lack of reproducibility can lead to major consequences:

•a failure in the reproducibility will most probably end in results' heterogeneity;

•at a clinical level, if a diagnostic test is not reproducible there is the risk of a patient being wrongly diagnosed;

•non-reproducible items of a checklist can lead to a decrease on its credibility and, consequently, of the meta-analyses that used it as a model.

The question we want to answer is if the QUOROM Checklist is a reproducible method

in the evaluation of Meta-Analysis.

Primary Aim:

Evaluate the reproducibility degree of the QUOROM Checklist

The question we want to answer is if the QUOROM Checklist is a reproducible method

in the evaluation of Meta-Analysis.

Primary Aim:

Evaluate the reproducibility degree of the QUOROM Checklist

Secondary Aims:

Specify which points of the QUOROM Checklist are less reproducible;

Verify if there are differences in the reproducibility between the evaluation of meta-

analysis from Low Impact Factor journals and from High Impact Factor ones.

Secondary Aims:

Specify which points of the QUOROM Checklist are less reproducible;

Verify if there are differences in the reproducibility between the evaluation of meta-

analysis from Low Impact Factor journals and from High Impact Factor ones.

Our target population was the meta-analyses. We had to select a considerable sample of meta-analyses, so we decided to select a total of 52.

Our inclusion criteria were:The article being published in a medicine subjects’

journal;The article being published in a journal with impact

factor ≤2 or ≥8;The article reporting a meta-analysis;The article being published in the last three years (2005-

2008);Having access to online full text.

Our target population was the meta-analyses. We had to select a considerable sample of meta-analyses, so we decided to select a total of 52.

Our inclusion criteria were:The article being published in a medicine subjects’

journal;The article being published in a journal with impact

factor ≤2 or ≥8;The article reporting a meta-analysis;The article being published in the last three years (2005-

2008);Having access to online full text.

First, we separated 40 journals using a Stratified Sampling Method.First, we separated 40 journals using a Stratified Sampling Method.

From Journals of ISI Web of Knowledge that fit our criteria (n=1234), we selected:From Journals of ISI Web of Knowledge

that fit our criteria (n=1234), we selected:

20 Journals20 Journals20 Journals20 Journals0 < IF ≤ 2(1234 journals)

0 < IF ≤ 2(1234 journals)

IF ≥ 8(82 journals)

IF ≥ 8(82 journals)

IF – Impact FactorIF – Impact Factor

Low IF JournalsLow IF Journals High IF JournalsHigh IF Journals

Low IF Journals: 48 meta-analyses

Low IF Journals: 48 meta-analyses

2626

Pool n.1Pool n.1Pool n.1Pool n.1

Low IF Meta-Analyses


After this, we proceeded to the selection of the Meta-Analyses.

For that, we used a Multi-Stage Sampling Method.

The totality of the Journals’ articles were removed from each stratum, following the inclusion criteria previously described.

After this, we proceeded to the selection of the Meta-Analyses.

For that, we used a Multi-Stage Sampling Method.

The totality of the Journals’ articles were removed from each stratum, following the inclusion criteria previously described.

2626


High IF Meta-Analyses


High IF Journals: 219 meta-analysesHigh IF Journals: 219 meta-analyses

We repeated the whole process of selection of the articles until we had

enough meta-analyses.

We repeated the whole process of selection of the articles until we had

enough meta-analyses.





The impact factor of the journal from where each Meta-Analysis came, the name of the journal, the authors and the year of publication were recorded in a database, which was kept secret until the evaluation of the checklist was concluded. It was used only at the end to find out if Reproducibility and Impact Factor were related.

The impact factor of the journal from where each Meta-Analysis came, the name of the journal, the authors and the year of publication were recorded in a database, which was kept secret until the evaluation of the checklist was concluded. It was used only at the end to find out if Reproducibility and Impact Factor were related.

Pool n.1Pool n.1Pool n.1Pool n.1 Pool n.2Pool n.2Pool n.2Pool n.2



Pool n.1Pool n.1Pool n.1Pool n.1 Pool n.2Pool n.2Pool n.2Pool n.2




52 Meta-Analyses52 Meta-Analyses

2626 2626

5252

Finally, we mixed all the articles in a single pool, occulting the strata from each one came.

Finally, we mixed all the articles in a single pool, occulting the strata from each one came.

Before analyzing we established some rules that helped us understanding each item of the checklist:

•If a certain item was present in the meta-analysis, but not in the place the checklist determines, we would not consider the item present;

•When a item had more than one point, we would only consider it present if the meta-analysis answered to more than half of the points;

Before analyzing we established some rules that helped us understanding each item of the checklist:

•If a certain item was present in the meta-analysis, but not in the place the checklist determines, we would not consider the item present;

•When a item had more than one point, we would only consider it present if the meta-analysis answered to more than half of the points;

•At the item (e), we would give more importance to the point that ensures the replication of the methods;

•At the item (o), the meta-analysis had to have a diagram describing trial flow, so that the item could be considered.

•At the item (e), we would give more importance to the point that ensures the replication of the methods;

•At the item (o), the meta-analysis had to have a diagram describing trial flow, so that the item could be considered.

Each student/investigator analysed a group of 4 articles and submitted them to the QUOROM Checklist.

Each student/investigator analysed a group of 4 articles and submitted them to the QUOROM Checklist.

After the students’ analysis, the articles were mixed again.

After the students’ analysis, the articles were mixed again.

Then, each student analysed another 4 articles, randomly selected from the 48 articles previously analysed by the rest of the group. This way, each student/investigator analysed different articles.

Then, each student analysed another 4 articles, randomly selected from the 48 articles previously analysed by the rest of the group. This way, each student/investigator analysed different articles.

By analyzing a meta-analysis, each student had to insert the data in the SPSS program. For each item, number 1 was attributed to those which are covered in the meta-analyses, and number 0 to those which aren’t.

By analyzing a meta-analysis, each student had to insert the data in the SPSS program. For each item, number 1 was attributed to those which are covered in the meta-analyses, and number 0 to those which aren’t.

Thus, our study can be classified as an observational, cross sectional study, whose methods are characteristic of a survey study, and whose purpose is to study the reproducibility.

Our variables are:

The actual Impact Factor of the journals from which we randomly selected the articles;

The year of publication of the articles;

The Impact Factor of the journals from which we randomly selected the articles at the year of publication;

The classification of each item of the checklist: we considered thirty-six categorical variables, which can have two numerical codes: 1 or 0. These are our expected outcome of research.

Our variables are:

The actual Impact Factor of the journals from which we randomly selected the articles;

The year of publication of the articles;

The Impact Factor of the journals from which we randomly selected the articles at the year of publication;

The classification of each item of the checklist: we considered thirty-six categorical variables, which can have two numerical codes: 1 or 0. These are our expected outcome of research.

From the classification of the items we had other variables:

Summation of the present items by observer 1;


Average of the two summations;

Difference between the summations;

Number of concordances between the two observers by article.

From the classification of the items we had other variables:



Average of the two summations;

Difference between the summations;

Number of concordances between the two observers by article.

Concordance in each Item of the Checklist(reproducibility of each Item)

We made eighteen concordance tables to calculate:•The proportion of concordance and 95% confidence

intervals*;•Positive proportion of concordance;•Negative proportion of concordance;•Kappa Factor.

* we used a normal distribution but with those whose limit of confidence intervals was over one, we used a binomial distribution.

Concordance in each Item of the Checklist(reproducibility of each Item)

We made eighteen concordance tables to calculate:•The proportion of concordance and 95% confidence

intervals*;•Positive proportion of concordance;•Negative proportion of concordance;•Kappa Factor.

* we used a normal distribution but with those whose limit of confidence intervals was over one, we used a binomial distribution.

Table 1.

Lower Upperq 1,00 0,90 1,00 1,00 1,00 1,00a 0,98 0,90 1,00 0,98 0,92 0,95h 0,98 0,90 0,99 0,98 0,00 NAr 0,98 0,90 1,00 0,98 0,00 NAb 0,96 0,87 0,99 0,96 0,78 0,85g 0,94 0,84 1,00 0,96 0,40 0,55o 0,90 0,82 0,98 0,77 0,86 0,80c 0,83 0,73 0,93 0,81 0,36 0,42i 0,83 0,73 0,93 0,86 0,22 0,30j 0,83 0,73 0,93 0,82 0,18 0,22d 0,79 0,68 0,90 0,68 0,62 0,57p 0,77 0,66 0,88 0,78 0,14 0,14l 0,75 0,63 0,87 0,69 0,43 0,42f 0,73 0,61 0,85 0,68 0,36 0,35n 0,73 0,61 0,85 0,70 0,26 0,24e 0,69 0,56 0,82 0,53 0,53 0,39m 0,69 0,56 0,82 0,66 0,24 0,18k 0,60 0,47 0,73 0,55 0,19 0,05

NA = Not Applicable (one of the variables is a constant: the observer 1 found these items in all articles)

P.C. - Kappa

Proportions of concordance of each item of the QUOROM checklist *

* items sorted by decrescent proportion of concordance

Item P.C.95% confidence interval

P.C. +

Global Reproducibility

•The comparison of the summation of each observer was done using the ICC method (Intraclass Correlation Coefficient).

•Then we represented the concordance limits of the “difference between the summations” in a scatterplot:

For that, we had to be sure that this variable followed a normal distribution and, if so, to calculate the mean and the standard deviation, all this by making an histogram.

Global Reproducibility

•The comparison of the summation of each observer was done using the ICC method (Intraclass Correlation Coefficient).

•Then we represented the concordance limits of the “difference between the summations” in a scatterplot:

For that, we had to be sure that this variable followed a normal distribution and, if so, to calculate the mean and the standard deviation, all this by making an histogram.

ICC = 0,729

The ICC method revealed that 72,9% of the total variance is explained by the variance between the articles.

ICC = 0,729

The ICC method revealed that 72,9% of the total variance is explained by the variance between the articles.

Histogram: differences between the summations

Following a normal distribution, it would be expected that the mean was 0, so it may have occurred a systematic error in the study.

The concordance limits were [- 4,934 ; 4,434]

This means that 95% of the differences between the summations are in this interval.

Histogram: differences between the summations

Following a normal distribution, it would be expected that the mean was 0, so it may have occurred a systematic error in the study.

The concordance limits were [- 4,934 ; 4,434]

This means that 95% of the differences between the summations are in this interval.

Relation between IF and Reproducibility

For this analysis we didn’t use the actual impact factor, but the one at the year of publication of the articles*.

We made two scatterplots, to see if there were correlation between:

•The “difference between the summations” and impact factor;

•The “number of concordances between the two observers by article” and the impact factor.

* As the ISI Web of Knowledge database wasn’t updated with the impact factors of 2007, in the articles published in that year we used the impact factor of 2006.

Relation between IF and Reproducibility

For this analysis we didn’t use the actual impact factor, but the one at the year of publication of the articles*.

We made two scatterplots, to see if there were correlation between:

•The “difference between the summations” and impact factor;

•The “number of concordances between the two observers by article” and the impact factor.

* As the ISI Web of Knowledge database wasn’t updated with the impact factors of 2007, in the articles published in that year we used the impact factor of 2006.

No correlation between “impact factor” and “difference between the summations” were found, nor between impact factor and “number of concordances between the two observers by article”, because in both scatterplots [figures 4, 5] there was not any preferential orientation of the points

No correlation between “impact factor” and “difference between the summations” were found, nor between impact factor and “number of concordances between the two observers by article”, because in both scatterplots [figures 4, 5] there was not any preferential orientation of the points

15 de abril de 2008

Documents