three-dimensional evidence network plot system: covariate … · topic are inputted via a microsoft...

Journal of Clinical Epidemiology - (2017) -

ORIGINAL ARTICLE

Three-dimensional evidence network plot system: covariate imbalancesand effects in network meta-analysis explored using a new software tool

Sarah Batsona,*, Robert Scorea, Alex J. SuttonbaSystematic Review Team, DRG Abacus, 6 Talisman Business Centre, Talisman Road, Bicester OX26 6HR, UK

bDepartment of Health Sciences, University of Leicester, Centre for Medicine, University Road, Leicester LE1 7RH, UK

Accepted 17 March 2017; Published online xxxx

Abstract

Objectives: The aim of the study was to develop the three-dimensional (3D) evidence network plot systemda novel web-based inter-active 3D tool to facilitate the visualization and exploration of covariate distributions and imbalances across evidence networks for networkmeta-analysis (NMA).

Study Design and Setting: We developed the 3D evidence network plot system within an AngularJS environment using a third partyJavaScript library (Three.js) to create the 3D element of the application. Data used to enable the creation of the 3D element for a particulartopic are inputted via a Microsoft Excel template spreadsheet that has been specifically formatted to hold these data. We display and discussthe findings of applying the tool to two NMA examples considering multiple covariates. These two examples have been previously iden-tified as having potentially important covariate effects and allow us to document the various features of the tool while illustrating how it canbe used.

Results: The 3D evidence network plot system provides an immediate, intuitive, and accessible way to assess the similarity and dif-ferences between the values of covariates for individual studies within and between each treatment contrast in an evidence network. In thisway, differences between the studies, which may invalidate the usual assumptions of an NMA, can be identified for further scrutiny. Hence,the tool facilitates NMA feasibility/validity assessments and aids in the interpretation of NMA results.

Conclusion: The 3D evidence network plot system is the first tool designed specifically to visualize covariate distributions and imbal-ances across evidence networks in 3D. This will be of primary interest to systematic review and meta-analysis researchers and, more gener-ally, those assessing the validity and robustness of an NMA to inform reimbursement decisions. � 2017 Elsevier Inc. All rights reserved.

Keywords: Covariate; Evidence networks; Heterogeneity; Meta-analysis feasibility; Network meta-analysis; Novel graphical tool; Three dimensional

1. Introduction

Network meta-analysis (NMA) is an increasingly popu-lar statistical method used for estimating the comparativeefficacy of all treatments of interest for a given condition,by simultaneously synthesizing data from all relevant ran-domized controlled trials (RCTs) [1]. Such analyses arecommonly used to identify the most effective treatmentsand inform economic decision models to estimate the rela-tive cost-effectiveness of the treatment options.

Like all statistical modeling, NMAmakes a number of as-sumptions that, if not satisfied by the data being synthesized,

Funding: This research did not receive any specific grant from funding

agencies in the public, commercial, or not for profit sectors.

Conflicts of interest: None.

* Corresponding author. Tel.: þ44-(0)-1869-241281; fax: þ44-(0)-

1869-220152.

E-mail address: [email protected] (S. Batson).

http://dx.doi.org/10.1016/j.jclinepi.2017.03.008

0895-4356/� 2017 Elsevier Inc. All rights reserved.

can lead to erroneous results and misleading conclusions [2].The first assumption that needs satisfying is that the networkis connectedwhich can be checked by constructing a networkdiagram [3]. More generally, evidence network diagrams arecommonly used for visualizing the available evidence basefor the purpose of assessing the feasibility of the meta-analysis and understanding the strength and diversity of theevidence available.A conventional network diagram consistsof ‘‘nodes’’ representing the treatments of interest and edgesrepresenting available direct comparisons between pairs ofinterventions and is a key component of global NMA report-ing checklists [4]. All nodes should be connected to form asingle network via edges and any nodes which are not con-nected should be excluded. The amount of available evidencecan also be presented in network diagrams by ‘‘weighting’’the nodes and edges using different node sizes and linethicknesses [5].

Delta:1_given name

Delta:1_surname

Delta:1_given name

mailto:[email protected]






2 S. Batson et al. / Journal of Clinical Epidemiology - (2017) -

What is new?

Key findings� Visually assimilating, exploring, and interpreting

the distribution of covariate values across trials ina network meta-analysis (NMA) is challengingdue to the complexities of representing the networkstructure simultaneously alongside study-level co-variate values.

� This article describes a three-dimensional (3D) ev-idence network plot systemda novel, freely acces-sible, web-based package to facilitate theexploration of covariate distributions and imbal-ances across evidence networks in NMA.

What this adds to what was known?� The primary innovation which allows for the ex-

tensions to evidence networks and improvementsis the use of a 3D graphical environment, incorpo-rating the graphical representation of covariates ona third ‘‘z’’-axis.

� We believe this work to be the first application of a3D graphical environment to evidence networks inNMA.

What is the implication and what should changenow?� We propose that the 3D evidence network plot sys-

tem will facilitate the exploration of covariate dis-tributions and imbalances across evidencenetworks and be of most value in the context ofsupporting NMA feasibility/validity assessmentsand to aid in the interpretation of NMA results toa wide audience.

Further assumptions of NMA relate to the comparabilityof the studies being combined. As for pairwise meta-analysis, differences in the results of studies (beyond thatexpected by chance) within each (pairwise) treatment com-parison are described as between-study heterogeneity. Suchvariability in study results can lead to inconsistency in treat-ment estimates across different comparisons in an NMA,where estimates of comparative effectiveness differ betweenthose from direct comparisons and those derived from indi-rect comparison routes through the network [6]. Althoughheterogeneity and inconsistency random-effect (RE) termscan be included in NMAmodels to allow for them [7], resultscan become increasingly difficult to interpret as the numberand magnitude of such terms increase. This can lead to chal-lenging issues, in terms of limiting the ability to generalizefrom the results [8], for both decision makers [9] and for de-signers of further studies that are intended to update the

evidence base in the future [10,11]. Therefore, it is highlydesirable to explain the causes andmagnitude of heterogene-ity and inconsistency rather than simply accommodate them.

Heterogeneity and inconsistency can frequently be ex-plained by the differences in trial design and the conductof the individual trials included in the NMA. Assumingsummary information from published trial results is beingused for the NMA rather than individual patient data[12], it may be possible to identify causes of heterogeneityand inconsistency by extracting information on study- andaggregate patient-level characteristics (e.g., duration oftreatment or duration of condition before randomization).Such variables are often described as potential effect mod-ifiers, and if these impact the effectiveness of the interven-tions of interest, treatment by covariate interactions can beincluded in the NMA model [7]. Treatment-covariate inter-actions can be used to explain and reduce heterogeneity andinconsistency in the same way as they are used in meta-regression for pairwise meta-analysis [13]. In addition,when treatment by covariate interactions relating to patientcharacteristics are identified, it implies that treatment effi-cacy varies between patients. Therefore, optimal treatmentdecisions could vary across patient groups depending ontheir characteristics. In the current paper, we focus on po-tential effect modifiers which are expressed on a continuousscale (including dichotomous patient-level covariatesaggregated at the study level, e.g., % of males), althoughwe note that categorical variables (e.g., individual indica-tors of study quality) can also be considered using a regres-sion framework and plots including these have beenconsidered elsewhere [14]. Regression modeling is gener-ally superior to subgroup analyses as it allows a holisticanalysis, exploring the impact of covariates on all of thedata, and allows the simultaneous consideration of multiple(continuous and categorical) covariates [8]. However, itshould not be forgotten that regressing study-level sum-mary covariate information on study-level average treat-ment effects is potentially susceptible to ecological bias.

A recent publication [14], outlining a process for assess-ing the feasibility of conducting a valid NMA, highlightedthe importance of assessing whether there are differences intreatment, patient, and outcome characteristics across com-parisons that may affect the summary measures of treat-ment effects relative to an overall reference treatment.These potential effect modifiers may be known or suspecteda priori or identified post hoc. Visually assimilating,exploring, and interpreting the distribution of covariatevalues across trials in an NMA is challenging due to thecomplexities of representing the network structure simulta-neously alongside study-level covariate values. Althoughmultiple plots could more easily be constructed for individ-ual comparisons within the network, these are of limiteduse because many will be sparse and uninformative, andeach plot only provides a subset of the required informa-tion. A holistic approach is required to assess the distribu-tion of covariate values across the whole evidence network.

3S. Batson et al. / Journal of Clinical Epidemiology - (2017) -

Cope et al. [14] present what we believe to be the firstpublished attempt to present such information graphicallyfor categorical and aggregate patient-level dichotomous co-variates (i.e., expressed as a %). This was achieved byincluding pie-charts for each study close to the relevantcomparison edge on a network diagram (see Cope et al.Figs. 4 and 5). Although we acknowledge the utility of suchan approach, we find assimilating the information fromsuch displays challenging, particularly as it is not alwaysclear which comparisons certain study pie-charts relate to.Furthermore, no method of displaying information fromcontinuous covariates is proposed.

In this paper, we present a novel, interactive, freelyaccessible, web-based package to present continuous andaggregate patient-level dichotomous covariate informationsuperimposed on an evidence network. We see our workas complementary to that previously discussed [14], butwe believe our method is easier to interpret for aggregatepatient-level dichotomous covariate information (thecontext in which both approaches could be used) as it cir-cumvents the issues highlighted above. The primary inno-vation underlying the suggested improvements is the useof an interactive three-dimensional (3D) graphical environ-ment. Such 3D models are easier to interpret when they canbe rotated rather than the reliance on static images. Thereaders of this publication are encouraged to explore thetool in conjunction with this paper (available at http://3dnma.com/plot).

In the current paper, Section 2 outlines the rationalefor and features of the software tool we have developed;Section 3 introduces the first illustrative NMA examplefrom rheumatoid arthritis (RA), which has a relativelysimple star-shaped evidence network and is presentedas an introduction to the concept of the 3D network plotsystem using disease duration and baseline risk as twocontinuous covariates of interest [8]; Section 4 intro-duces a second example in advanced breast cancer, takenfrom the paper by Cope et al. [14]. Here, we graphicallypresent one of the continuous covariates considered byCope et al. (not originally presented graphically) andan aggregate dichotomous patient-level covariate-visceral metastases. The latter was presented visuallyby Cope et al., and we report a comparison with our3D network plot system. Section 5 introduces advancedapplications of the system whereby we demonstrate thereduction of network data to a single summary covariateper treatment contrast and incorporate data relating to theprecision of the outcome data.

2. Features of the 3D network plot system

The 3D network plot system allows the incorporation ofa third ‘‘z’’-axis to display covariate bars for each trialincluded within a network. The covariate bars for eachstudy are placed on the edges of the relevant comparisonsand projected on the third dimension.

First, a data set is uploaded to the system in the form of aMicrosoft Excel sheet; a template can be downloaded fromthe system. When a covariate is not selected, the systemproduces a conventional two-dimensional (2D) network di-agram. This allows for examination of the network struc-ture, instantaneous removal of treatment nodes from thedata set, and for conducting sensitivity analyses or produc-ing focused views on a proportion of a complex network.When a covariate is selected, the system incorporates athird z-axis and ‘‘bars’’ to represent the covariate measurefor each trial. This produces a 3D representation which al-lows for the distribution of covariate effects for each com-parison to be simultaneously displayed. The system canhold multiple covariates, and each covariate can be selectedfrom a drop-down menu. The system also features zoomfunctionality and an automatic rotation function (whichcan be used simultaneously with other manual controls)to facilitate the exploration of data.

Following the initial input of data, many aspects can bemanually configured. This includes the position of thetreatment nodes on the xey axis and the coloring of thenodes; the principal view can be centered on any treatmentnode of interest. The treatment nodes and treatment con-trasts can be weighted to reflect the volume of availableevidence. The covariate name, value, scaling, and unitscan be amended, in addition to the width of the covariatebars. The distance and positioning of the covariate barsacross the treatment contrasts can also be modified, andthe color can be changed to represent positive and nega-tive values (relative to a reference value). Finally, thebackground grid feature and covariate labeling elementscan be toggled on and off. All manual changes withinthe system can be saved and reloaded (as ‘‘raw data’’),and the system is able to generate figures in PortableNetwork Graphics format.

The software can be accessed at http://3dnma.com/plot andrequires a user to register with their email address. The datasets for the examples used within this publication can beselected and loaded from within the system via a drop-downmenu (example data sets 1e6). A full instruction manual isalso available and can be downloaded fromwithin the system.

3. Example 1: rheumatoid arthritis

The first example data set includes trials from a review oftheNICE technology appraisal for certolizumab pegol (CZP)for the treatment of RA in patients who had failed disease-modifying antirheumatic drugs (DMARDs) [15]. The dataset includes 12 methotrexate (MTX) controlled trialscomparing seven different treatments (placebo þ MTX,adalimumab þ MTX, CZP þ MTX, etanercept þ MTX,infliximab þ MTX, rituximab þ MTX, and tocilizumab þMTX), forming a ‘‘star’’-shaped network where all compar-isons are relative to placebo.

The data set was used as an example in the NICE guid-ance on methods for meta-regression for the outcome of

http://3dnma.com/plot




American College of Rheumatology 50% (ACR50) at6 months [8]. Two potential sources of effect modificationare explored in the NICE guidanceedisease duration andthe baseline risk of ACR50. Baseline risk is defined asthe risk of ACR50 in the placebo arm of each trial.

3.1. Disease duration

Screenshots from the covariate visualizer of the evidencenetwork and an evidence network incorporating informationon the average disease duration from each study are shown inFigures 1 and 2. A 2D view from the covariate visualizer (thexey plane with no third z-axis displayed) is shown in Fig. 1and essentially represents a standard 2D network plot; thegrid function is not displayed in this example. Fig. 2 showsthe inclusion of the third (z) axis and the addition of covariate‘‘bars’’dthe grid is also added on this example to aid theinterpretation of the image in 3D. A single bar on the z-axisrepresents the covariate level of a single trial on the relevantedges connecting the treatment nodes compared in that trial.For example, the RAPID 1 and RAPID 2 trials comparedplacebo þ MTX with CZP þ MTX. Therefore, thetwo bars representing each of these trials lie on the edge con-necting the placeboþMTXnodewith CZPþMTX.The dis-ease duration for each trial has been centered on the meandisease duration across all trials of the network (8.21 yearsor 98.5 months). Consequently, in this example, the valueson the z-axis may be positive or negative. Negative covariatevalues (representing trials in which disease duration is lowerthan themean across all trials of the network) are presented asred bars, and positive values (representing trials in which dis-ease duration is higher than the mean across the network) arepresented as green bars. This further aids the interpretation ofthe distribution of the covariates across the network.

Fig. 1. Evidence network in 2D. 2D, two dimensional; MTX,methotrexate.

Disease duration was identified as a potential source ofeffect modification [8]. Previous studies in RA havedemonstrated that shorter disease duration is associatedwith smaller treatment effects compared with a longer dis-ease duration [16].

In Fig. 2, it is evident that there is considerable variabilityin the mean disease duration of patients enrolled in the RCTsincluded in the network. The studies with the most extremevalues are immediately identifiable as Weinblatt 1999 (highvalue) and the CHARISMA trial (low value). Importantly,it can be clearly seen that this variability is systematic. Forexample, the RAPID 1 and 2 trials, which comparedCZP þ MTX with placebo þ MTX, were both conductedin patients with shorter disease duration compared with themean disease duration of the network. Thus, in this network,there is likely to be an unfavorable bias against CZPþMTXas a shorter disease duration is associated with smaller treat-ment effects [16]. A similar scenario can also be observed fortocilizumabþMTX. The single trial (Weinblatt 1999) whichcompares etanerceptþMTXwith placeboþMTXwas con-ducted in patients with the longest disease duration of all thetrials in the network. Thus, there is likely to be a favorablebias for etanercept þ MTX in this network compared withother DMARDs as longer disease duration is associated withgreater treatment effects [8,16].

Although the meta-regression models presented by Diaset al. were not strongly supported by the trial data, the es-timate of the interaction coefficient from the RE model didnot include the null value within its 95% credible interval(CrI) [0.14 (95% CrI: 0.01, 0.26)] [8]. Thus, consistent withthe findings from larger studies in RA, the interaction coef-ficient from the RE adjusted model suggests that for each 1-year unit increase in disease duration, the log odds ratio(OR) of each comparator vs. placebo þ MTX increasesby 0.14 (i.e., larger treatment effects are associated withlonger disease duration). Therefore, in this example, theimplications of disease duration interaction with treatmenteffects should be considered for the decision model.

3.2. Baseline risk

As themeta-regressionmodels adjusting for disease dura-tion were not strongly supported by the RA data set from thetechnology appraisal, Dias et al. [8] recommended thatfurther explanations of the causes of heterogeneity shouldbe sought. Baseline risk is the underlying risk of the outcomeof interest within a study population and represents a sum-mary of both known and unknown risk factors. Baseline riskis a potentially important source of heterogeneity, particu-larly among studies where the baseline risk varies.

The screenshots from the data set visualized in the covar-iate visualizer are shown in Figures 3 and 4. Fig. 3 demon-strates the view along the xez plane, and Fig. 4 incorporatesall three axes. Together, these figures highlight considerablevariability in baseline risk across the trials of the network.Note, this particular visualization is only possible when all

Fig. 2. Evidence network displaying the disease duration covariate values for each trial. MTX, methotrexate.


trials contain a common comparator treatment arm, such asthat seen in ‘‘star’’-shaped networks. In this example, twotrials (Abe 2006 and CHARISMA) exhibit higher baseline

Fig. 3. Evidence network displaying the baseline risk of placebo þ MTXmethotrexate.

risks compared with the other trials of the network. Thiscould be a potential cause for concern highlighting unfavor-able bias toward the active treatments of these trials.

arm of each trial view of the network along the xez planes. MTX,

Fig. 4. Evidence network displaying the baseline risk of placebo þ MTX arm of each trial. MTX, methotrexate.


The baseline risk models explored by Dias et al. usingthe RA data set suggested strong interaction effects be-tween the baseline risk of ACR50 and relative treatment ef-fects; this relationship should be incorporated in cost-effectiveness analyses [8]. The estimate of the interactioncoefficient from the RE model suggested that for eachone unit (the difference from 100% to 0%) increase inthe baseline log odds of ACR50, the log OR of eachcomparator vs. placebo þ MTX decreased by 0.95 (i.e.,the greater the ACR50 response in the placebo þ MTXarm the smaller the treatment effect).

4. Example 2: advanced breast cancer

The second data set is the case study presented by Copeet al. which allows for the comparison of everolimus incombination with hormonal therapy with alternative thera-pies in terms of progression-free survival (PFS) in womenwith advanced breast cancer [14]. The data set consists of24 RCTs which form a connected evidence network forthe PFS outcome (which include hormonal and chemo-therapy comparators). Data for several potential effectmodifiers for each trial within the network are tabulated


in Cope et al. [14]. This approach is consistent with thecommon format of reporting potential treatment effectmodifiers in NMA publications which can be difficult tointerpret. In addition, Cope et al. visually presented twoof the aggregate dichotomous patient-level covariates (hor-mone receptor status and visceral metastases) as a pie-chartfor each study, close to the relevant comparison edge on anetwork diagram (see Cope et al. Figs. 4 and 5).

Cope et al. reported that differences within and across thetreatment comparisons were observed in terms of estrogenreceptor positive status, exposure to prior hormonal thera-pies, exposure to prior chemotherapies, and visceral metasta-ses [14]. However, Cope et al. also reported that differences

Fig. 5. Evidence network of the Cope et al. data set disp

in performance status and age were less prominent, althoughdifferences in postmenopausal status, human epidermalgrowth factor 2 status, and types of hormonal therapies werenoted [14]. In the following sections, we present one of thecontinuous covariates considered by Cope et al. [14] and anaggregate dichotomous patient-level covariate-visceral me-tastases. The latter was presented visually by Cope et al.[14], and we contrast our approach with theirs.

4.1. Continuous covariate-median age

We have explored all the potential treatment effectmodifiers presented in Cope et al. [14] as 3D network

laying the median age covariate. TAM, tamoxifen.

Fig. 6. Evidence network of the Cope et al. data set displaying the proportion of patients with visceral metastases view 1.


plots. Indeed, as suggested by Cope et al. [14], the differ-ences in terms of age across the trials appear to be lessprominent than those observed with the other potentialtreatment effect modifiers. On display of the median agecovariate within the 3D network plot system, it is imme-diately evident that there is relatively little variability inmedian age across the trials of the evidence network; inthis example, the median age values across the trials havenot been centered to a reference value in the 3D networkplot system, and therefore, all bars are represented ingreen as the default color for positive values (Fig. 5).However, it is easy to identify the treatment comparisons

associated with the most extreme median age values(Fig. 5). For example, the median age of patients in theO’Shaughnessy 2001 trial (bar has been colored red tohighlight the trial in Fig. 5) is 70 years. This is the highestacross the network and is 10 years greater than themean of median ages (60 years) across the trials of thenetwork. The O’Shaughnessy 2001 trial is the only studythat contributes to the direct comparison ofcyclophosphamide þ MTX þ fluorouracil vs. capecitabine(CAP). This could lead to a potential bias in comparisonsof CAP with other treatments of interest depending onthe potential direction of effect modification by age. The

Fig. 7. Evidence network of the Cope et al. data set displaying the proportion of patients with visceral metastases view 2. MA, megestrol acetate;TAM, tamoxifen.


median age of patients in the Ingle 1982 trial (bar has beencolored yellow to highlight the trial in Fig. 5) is 49 years,which is the lowest across the network and is 11 yearsbelow the mean of median ages. However, Ingle 1982 isone of three trials contributing evidence for the direct com-parison of tamoxifen (TAM) with megestrol acetate (MA).Although it would be challenging to detect any covariateeffect(s) of age with such little variability across moststudies, conducting a sensitivity analysis removing individ-ual trials for which covariate values are potentially aconcern is possible.

4.2. Aggregate dichotomous patient-level covariate-visceral metastases

Cope et al. reported visceral metastases data (present,absent, or not reported) by presenting a network diagramtogether with pie-charts for each study next to the relevantcomparison edge (see Cope et al. Fig. 5) [14]. We havevisualized the visceral metastases covariate data via the3D network plot system as the percentage of patients withthe outcome. Where a proportion of patients were reportedto have visceral metastases, the remaining proportion of

Fig. 8. Evidence network displaying the weighted mean level of disease duration covariate per treatment contrast. MTX, methotrexate.


patients had no visceral metastases. Furthermore, any trialnot reporting the covariate of interest is clearly highlightedwith a black circle on the 2D axis and labeled ‘‘N/R.’’ Asdisplayed in Figures 6 and 7, it is evident that there is vari-ability in the proportion of patients with visceral metastasesacross the trials of the evidence network. The treatmentcomparisons that the trials with the most extreme valuescontribute to are also easily distinguishable (Figs. 6 and7). For example, the trials contributing to the comparisonsof docetaxel with nab-paclitaxel, liposomal doxorubicin, orvinorelbine enrolled patients with the highest proportion ofvisceral metastases (87e91%). In comparison, all trials

which included a treatment of TAM or MA included lessthan 50% of patients with visceral metastases.

5. Advanced application and features of the 3Dnetwork plot system

5.1. Reduction of data to a single covariate bar pertreatment contrast

As an alternative to presenting covariate values for eachtrial, the system can be used to present an overall summaryof the average covariate value across all trials for each

Fig. 9. Evidence network displaying the disease duration covariate values for each trialdcovariate bar widths reflect the inverse variance weightingof each trial. MTX, methotrexate.


treatment contrast. This allows for an overall summary of thelevel of the covariates for each treatment comparison. Theexact influence a trial’s covariate value has on estimation ofNMA parameters varies by parameter and thus is difficultto represent by a single value [17]. Therefore, we suggestexamining both theweighted and unweightedmean covariatevalue (weighting can be based on 1/variance of each trial’streatment effect).We have presented this summary for the co-variate of disease duration in RA (example 3 data set). Here,the weighted mean of the covariate values (centered to thenetworkweightedmean) is presented for all studies in a giventreatment comparison (Fig. 8). Fig. 8 can be compared with

Fig. 2 to facilitate the comparison between presenting sum-mary information per treatment contrast vs. all trial level ev-idence per contrast, respectively (although an unweightedmean was used to center the covariate values in Fig. 2). Notethat there are three trials for the treatment contrast ofplacebo þ MTX compared with adalimumab þ MTX, twoof which have a higher disease duration than the mean ofthe network, whereas a single trial has a lower disease dura-tion than the mean of the network (Fig. 2). However, for thetreatment contrast of placebo þ MTX compared withadalimumab þ MTX, the summary of the covariate for thiscontrast is higher than the weighted mean of the network.

Fig. 10. Evidence network displaying the weighted mean level of disease duration covariate per treatment contrastdcovariate bar widths reflect theinverse variance weighting of the pooled data available for each treatment contrast. MTX, methotrexate.


5.2. Incorporation of data relating to the precision ofoutcome data

The width of the covariate bars is adjustable within thesystem. A good use for this functionality, to add morerelevant information into the plot, as to further aid theinterpretation of the data, is to make the bars proportionalto the precision (1/variance) of the primary outcome foreach of the trials. In this way, the precise trials, providingmost information and thus being most influential in thesynthesis, are more visually dominant (in the same wayas having the central estimate of effect plotting symbol

being proportional to the precision of the study in a con-ventional forest plot). Here, we have applied this approachto the RA example for the covariate of disease duration ina network which presents all trial-level covariate data(Fig. 9) (example 4 data set) and a network which presentssummary data for each contrast (Fig. 10) (example 5 dataset). These plots illustrate the influence of each of the tri-als within the networks. For example, Abe 2006 is one ofthe smallest trials of the network, and it is clear fromFig. 9 that it has a small influence on the relative efficacyof infliximab þ MTX vs. placebo þ MTX compared withthe two other trials contributing to this comparison


(ATTEST and START trials). More generally, in Figures 9and 10, respectively, it is evident that the outcome data forthe single trials contributing to the treatment contrasts ofrituximab þ MTX and etanercept þ MTX vs.placebo þ MTX are associated with large variances (smallprecisions). Therefore, it is likely that the results fromcontrasts involving these treatment nodes will be associ-ated with greater levels of uncertainty compared withother treatment nodes of the network.

6. Discussion

We have developed a novel interactive tool which pro-duces a 3D version of a network diagram, incorporatingthe graphical representation of covariates on a third z-axis.The tool is available as an open access resource (availableat http://3dnma.com/plot). We propose that this novelgraphical approach can be used to facilitate the explorationof covariate distributions and imbalances across evidencenetworks. We believe it will be most valuable in the contextof supporting NMA feasibility/validity assessments andaiding in the interpretation of NMA results.

In terms of NMA feasibility/validity assessments, the 3Dnetwork plot system can support this by aiding in the iden-tification of (1) imbalances in covariates considered to bepotential effect modifiers and (2) appropriate analyses tomodel and adjust for potential effect modifiers, identifiedas being imbalanced across treatment comparisons.Furthermore, the tool can aid in the interpretation ofNMA results through (1) constructing the plots for usingdata from published NMAs as an appraisal/assessment toolto check that important covariate imbalances do not exist,in the presence of heterogeneity, for covariates not adjustedfor in the original analysis and (2) reconciling the results ofa meta-regression analysis through the impact of covariateadjustment vs. unadjusted model results via the visualiza-tion of covariate values across the whole network.

The proposed 3D plotting system was designed to pro-vide a high level of flexibility and functionality to the user,be that data driven or visual. Most of the development wasconducted within an AngularJS environment using a thirdparty JavaScript library (Three.js) to create the 3D elementof the application. By using Three.js, it was possible torapidly develop the application as Three.js supported alarge proportion of the processing required for 3D develop-ment. The 3D element of the plot was developed using dataheld within a specially formatted Microsoft Excel spread-sheet. The use of Microsoft Excel to develop the systemmeans that the application is both intuitive and userfriendly. The system is also able to accommodate themanual input of multi-arm trials although the examples pre-sented in this publication include two-arm trials only (seeexample 6 data set from within the system).

In a pairwise meta-analysis context, exploratory plots ofthe data to explore potential associations between treatmenteffects and continuous study-level covariates are relatively

straightforward. Such plots can also be overlaid with the re-sults of a meta-regression analysis. Software is available toplot the effect size of individual studies against the covari-ate of interest [18]. This software incorporates a plottingsymbol, where the size of the symbol is proportional tothe precision of the study. A fitted regression line of bestfit can then be superimposed on top of this together withassociated confidence and/or Crlsdan example of whichcan be found in Ibrahim et al. 2015, Fig. 3 [19]. Work isunderway to develop an approach to incorporate both studyoutcome/modeling results and study covariate valuesclearly for an NMA.

Future developments proposed for this applicationinclude the ability to produce stacked bar charts to visualizecategorical covariates. In addition, integrating the systemwith an algorithm to optimally locate treatment nodesshould help to make the network appear less cluttered[20]. One interesting option would be to further developthe 3D element of the system, such as allowing the systemto be viewed in stereoscopic 3D (as found in virtual realitytechnologies) which may assist with interpretation [21].The ability to print the networks using a 3D printer to pro-duce real-life sculptures to further aid in the understandingand interpretation of evidence networks is now a possibil-ity. As digital media replaces paper, some journals arenow starting to permit publication of 3D figures and anima-tions [22]. We would fully support this initiative in appliedstatistics and in medical research generally.

7. Conclusion

The 3D evidence network plot system is the first tool de-signed specifically to visualize covariate distributions andimbalances across evidence networks in 3D. This will beof primary interest to systematic review and meta-analysis researchers and, more generally, those assessingthe validity and robustness of an NMA to inform reim-bursement decisions. It will facilitate the exploration of co-variate distributions and imbalances across evidencenetworks and be of most value in the context of supportingNMA feasibility/validity assessments and aid in the inter-pretation of NMA results. Given its free availability onthe web and simple spreadsheet interface, it has the poten-tial to make this powerful tool available to a wide audience.

References

[1] Bafeta A, Trinquart L, Seror R, Ravaud P. Analysis of the systematic

reviews process in reports of network meta-analyses: methodological

systematic review. BMJ 2013;347:f3675.

[2] Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence synthesis for de-

cision making 1: introduction. Med Decis Making 2013;33:597e606.

[3] Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for de-

cision making 2: a generalized linear modeling framework for pair-

wise and network meta-analysis of randomized controlled trials.

Med Decis Making 2013;33:607e17.

[4] Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH,

Cameron C, et al. The PRISMA extension statement for reporting


http://refhub.elsevier.com/S0895-4356(16)30845-9/sref1














of systematic reviews incorporating network meta-analyses of health

care interventions: checklist and explanations. Ann Intern Med 2015;

162:777e84.

[5] Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graph-

ical tools for network meta-analysis in STATA. PLoS One 2013;8:

e76654.

[6] Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N,

et al. Interpreting indirect treatment comparisons and network

meta-analysis for health-care decision making: report of the ISPOR

Task Force on Indirect Treatment Comparisons Good Research Prac-

tices: part 1. Value Health 2011;14(4):417e28.

[7] Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing

between-study heterogeneity and inconsistency in mixed treatment

comparisons: application to stroke prevention treatments in individ-

uals with non-rheumatic atrial fibrillation. Stat Med 2009;28:

1861e81.

[8] Dias S, Sutton AJ, Welton NJ, Ades AE. Evidence synthesis for de-

cision making 3: heterogeneityesubgroups, meta-regression, bias,

and bias-adjustment. Med Decis Making 2013;33:618e40.

[9] Ades AE, Lu G, Higgins JP. The interpretation of random-effects

meta-analysis in decision models. Med Decis Making 2005;25:

646e54.

[10] Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR,

Abrams KR. Evidence-based sample size calculations based upon up-

dated meta-analysis. Stat Med 2007;26:2479e500.

[11] Welton N, Ades AE. Research decisions in the face of heterogeneity:

what can a new study tell us? Health Econ 2012;21:1196e200.[12] Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual

participant data: rationale, conduct, and reporting. BMJ 2010;340:

c221.

[13] Thompson SG, Higgins JP. How should meta-regression analyses be

undertaken and interpreted? Stat Med 2002;21:1559e73.

[14] Cope S, Zhang J, Saletan S, Smiechowski B, Jansen JP, Schmid P. A

process for assessing the feasibility of a network meta-analysis: a

case study of everolimus in combination with hormonal therapy

versus chemotherapy for advanced breast cancer. BMC Med 2014;

12:93.

[15] National Institute for Health and Clinical Excellence. Certolizumab

pegol for the treatment of rheumatoid arthritis. NICE technology

appraisal guidance, 2010, TA186.

[16] Nixon RM, Bansback N, Brennan A. Using mixed treatment compar-

isons and meta-regression to perform indirect comparisons to esti-

mate the efficacy of biologic treatments in rheumatoid arthritis.

Stat Med 2007;26:1237e54.

[17] Donegan S, Dias S, Tudur Smith C, Marinho V, Welton N. Graphs of

study contributions and covariate distributions for network meta-

regression. Res Synth Methods 2017.

[18] Harboard RM, Higgins JP. Metaeregression in Stata. Stata J 2008;

8(4):493e519.

[19] Ibrahimi P, Jashari F, Bajraktari G, Wester P, Henein MY. Ultrasound

assessment of carotid plaque echogenicity response to statin therapy:

a systematic review and meta-analysis. Int J Mol Sci 2015;16(5):

10734e47.[20] Rucker G, Schwarzer G. Automated drawing of network plots in

network meta-analysis. Res Synth Methods 2016;7(1):94e107.

[21] Ware C, Franck G, editors. Viewing a graph in a virtual reality

display is three times as good as a 2D diagram. IEEE Conference

on Visual Languages; 1994 4e7 Oct 1994: IEEE.

[22] Grossman T, Chevalier F, Kazi RH. Bringing research articles to life

with animated figures. Interactions 2016;23(4):52e7.






































































three-dimensional evidence network plot system: covariate … · topic are inputted via a microsoft...

Documents