This topic and the next one deal with quality and quantity of evidence, respectively. In a certain way, they go hand-in-hand. The quantity of evidence that remains in the end on a search, depends on the number of things that are discarded as not being of sufficient quality. You might find 5 papers, but discount them because of the errors or low quality that are part of their design. We start with considerations of QUALITY so that we can use these principles next to narrow the quantity uncovered in a search.

22

Let’s use the EVIDENCE MAP to discuss options for quality. [CLICK] In different zones (low level, mid level, and high level) you will have differing quality for differing reasons. You already know that clinical trials will be highest quality due to their much more scientific nature and high level of peer review. However, you can still get high quality things that are not peer-reviewed. Mid level things can be technically very good quality as long as they follow certain rules. Low level things can have good quality if the person or organization responsible for the content follows good procedures in collecting or creating the evidence. Now let’s look at each of the three segments. Then we will focus in much more detail on the high level things.

33

Look at examples of some quality at low or no level of peer review. [CLICK] For any treatment, you want to know about SAFETY and EFFICACY. A new product must have some safety information, but may have little or no scientific information about its long-term efficacy.

Everything requires technically proper usage. Manufacturers produce INSTRUCTIONS for use. Most individuals learn better by seeing a demonstration and having a tutorial. Webinars are online case demonstrations and available from all companies, many academies, and some CE centers. They are usually produced in real time and demonstrate a procedure once or twice. While they are not peer-reviewed in the sense that someone reviews the scientific foundation for the procedure, they are technically very good. They are clear and professionally organized tutorials. Often, information on SAFETY and known facts about short-term EFFICACY are included in these demonstrations. The quality of these sources of evidence is based on the TRUSTWORTHINESS of the agent or source producing them. Here is a simple rule for assessing the value or quality of this information. If the agent producing them is a large dental manufacturer or supplier whose profit depends on your long-term TRUST and they do not charge you for this information (e.g., 3M ESPE), then most likely, they are TRUSTWORTHY. If the agent charges you access to the information or represents a small company that is often a single product, you should be much more careful with your TRUST.

44

Manufacturers of pharmaceuticals, restorative dental materials, or other dental products [CLICK] are very interested in meeting all the required standards and providing accurate information about the properties of their materials. While they have a strong desire to be competitive in the market place, they also know that any distortions or inaccurate reporting of their products will undermine the trust of their clients. In addition, there are AGGREGATORS of information (such as NIH, review groups, Delphi groups) who try to filter the good from the bad. They also depend on your trust.

Let’s look at a simple example of a potentially good quality TECHNICAL REPORT for a relatively new composite restorative material from DENTSPLY. The company will provide a technical brochure that includes physical, chemical, mechanical, and biological properties for the product versus other products they are competing against. Their product will NOT be the best in every category. They will run the tests following published guidelines or standards (e.g., ADA/ANSI or FDI/ISO) in their own laboratories but often also contract to universities or other independent centers to run the same tests for them as well.

We have not talked yet about standards and guidelines in depth. The quality of these mid level sources of evidence depends on their conformance with published standards or guidelines for testing. What is different from the low level reports is that treatments are being compared to something here – to other treatments or to target values for acceptance. Most of these tests are not clinical trials but do provide a first major screen for SAFETY and EFFICACY.

For just a moment, let’s look at the world of GUIDELINES and STANDARDS that could be involved.

Most of the world runs because organizations (a) voluntarily adopt standards or guidelines for production, testing, and reporting information, or else (b) the government produces regulations to accomplish the same thing. Invariably people prefer the former. In the figure, [CLICK] the upper half of the flowchart represents STANDARDS and [CLICK] the lower part represents REGULATIONS. [CLICK] The left-hand side represents situations in the UNITED STATES and [CLICK] the right-hand part summarizes the rest of the WORLD.

The American Dental Association (ADA) generates STANDARDS that are registered with the American National Standards Institute (ANSI) that collect standards for all industries. A parallel thing happens globally as well. The FDI represents the group of all national dental associations (and the ADA is a member). It works to develop dental standards and registers them through the International Standards Organization (ISO). In technical brochures you will see statements that tests were performed in compliance with ADA/ANSI or ISO standards. Since the ADA started this process first, most of its standards are later adopted by the FDI/ISO and are almost identical.

In cases where it is either complicated politically or there are complex political boundaries involved, the government will step in and define REGULATIONS. In the US, most of the products for dentistry are regulated under the FDA. These regulations apply to drugs and devices. The rules require pre-clearance, continual testing, evidence of good practices, and annual approval. The advantage of national regulation rather than ones for states is that companies have a single authority for responsibility rather than 50 individual ones. Globally it is more complicated. The World Health Organization (WHO), the United Nations Environmental Programme (UNEP), and other groups are trying to organize global regulations. At the moment there are strong EU regulations but not much other agreement in other regions around the world.

5

66

Now, let’s look at good-to-high quality peer-reviewed publications. [CLICK] We are going to evaluate the quality at several levels – (1) the journal reputation (and its review process), (2) the design of the clinical trial, and (3) reported information in the article that is published. Let’s quickly look at each of these.

First, be cautious. (1) Not all articles published in journals with good reputations are good publications. Journals choose to publish 20-60% of the material submitted based on their priorities for new and interesting content, scientific excellence, and core focus of the journal. Good papers may be published in poor journals and poorer articles may appear in good journals. We will look at impact factors in Module 9 to clarify this further. (2) Any experiment may have flaws in its design – due to intentional omissions, planning problems, or funding limitations. These compromise the quality. (3) The article that is published may not have the level of detail reported, the level of analysis that is appropriate, or sufficient critique to point out limitations. All of these are problems for its quality rating.

Reputation and impact are both important in judging quality of a journal. We will defer impact factor comments to Module 9.

The New York Times [CLICK] is generally considered a high quality source of news information. However, on occasion its reporters have fabricated information or distorted information during reporting. [CLICK] When that is discovered, retractions are printed. [CLICK] However, even good scientific publications do not receive this sort of correction. Bad papers almost always stand as published -- and the reader must beware.

Examine papers that already have been published and imagine that you were the reviewer before they are accepted. [CLICK] Use your skills as both an author or reviewer to see if the core information is present to make a publication a good one. Does it contain the requisite information? Is the introduction fairly and concisely presented? Is the objective stated in detail as a PICO question? Does a complete presentation of the materials and methods exist? Are the results presented in carefully organized detail with proper statistical analysis? Does the discussion dissect the process and include a strong scientific analysis of the value and meaning of the results. Are the conclusions statistically valid answers to the PICO question?

8

One of the great advocates for good clinical trial DESIGNS has been JOHAN KARLBERG. He lives and works in Hong Kong at the University. For almost 3 decades he has pushed for good trial designs and ones based on good science.

There are several very common errors that plague most clinical trials and these diminish their value. Trials are designed to follow treatment, rather than all the potentially important variables. Trials test products, rather than being hypothesis driven. Quality of trial data is poor. Statistical analyses are limited. Biases are not eliminated, and blinding is not included. All of these effects distort the results significantly and limit value the real value.

We need clinical research. However, differences in methodologies and documentation have always made combining information difficult across trials. Therefore, there are now 2 relatively new RULES for clinical research which dictate how it should be conducted in order to be published. New publications must follow the CONSORT and PRISMA rules. What are these two things?

CONSORT = Consolidated Standards of Reporting Trials

These rules require that randomized controlled trials (RCTs) follow very strict requirements for reporting all of their parameters. [CLICK] There are actually 37 different steps involved. [CLICK] Journals now are requiring that these rules be followed for consideration for review and for publication.

[CLICK] PRISMA = Preferred reporting items for systematic reviews and meta-analyses

[CLICK] These new rules try to provide a framework for combining information in reviews and meta-analyses and [CLICK] involve 27 specific steps.

While they may ultimately help the process of collecting and reporting clinical research, at the moment they seem to be having the opposite effect because of increased costs to clinical research trials.

In an ideal world we should be able to come up with a single rating for the quality of any publication. EBM and EBD have struggled with this for several years now. Here is the reason that this is not yet practical. EBM or EBD is a process that includes several steps. [CLICK] We have not yet agreed on the rating scale for individual steps – that would allow us to put all of the values together into a single master value. Let’s look at the steps – research design, reporting, recommendations, and extrapolations.

Design is probably the easiest to critique. [CLICK] The US Preventive Services Task Force designed a system more than 10 years ago to rate designs in terms of 3 major levels (RCTs, other clinical trials, and expert opinions). [CLICK] As shown in the figure, its second level can be subdivided into 3 other ones. This is discussed in more detail at the following site: http://en.wikipedia.org/wiki/Evidence-based_medicine [CLICK] The UK’s National Health Service came up with a similar ranking system [CLICK] based on 4 levels that are parallel in emphasis.

Research reporting we have already discussed. It is crucial that different clinical trial types consistently report all of the same list of information about their trial designs and outcomes. [CLICK]CONSORT guidelines are for clinical research trials. PRISMA guidelines are for reviews and meta-analyses.

Recommendations (based on conclusions) from any trial involve some balance of benefit-to-risk. [CLICK] The US PSTF has rated quality at 5 levels based on the strength of this ratio. These are shown in the figure.

Finally, one might rate the overall quality based on the strength of the steps and the strength of extrapolation of the evidence. This process approaches what we discussed at the beginning in terms of being an overall rating. [CLICK] A task force, abbreviated GRADE, is now working on this type of scale. Others have suggested modifications of some of these individual scales. Yet, everything is complicated by the fact that not all evidence in the world can be ranked in terms of classic RCTs.

In your screening of the quality of evidence, you will come across many reasons to reject items. If you do not find what you want, what do you do? You will notice that most scales at their ends fall back on “expert opinions or expert groups” for recommendations. That is what you should use.

12

Here is a quick review of the concepts from this module.

[CLICK] (1) What is KEY to judging quality of low-level peer-reviewed evidence?

[CLICK]

[CLICK] (2) What is KEY to judging quality of mid-level peer-reviewed evidence?

[CLICK]

[CLICK] (3) What are KEYS to judging quality of high-level peer-reviewed evidence?

[CLICK]

[CLICK] (4) What is the difference between a STANDARD and REGULATION?

[CLICK]

[CLICK] (5) How would you RATE the vast majority of published clinical trials?

[CLICK]

[CLICK] (6) What is the GOAL for CONSORT and PRISMA guidelines?

[CLICK]

[CLICK] (7) What are the general assessment levels for QUALLITY SCALES?

[CLICK]

13

THANK YOU.