comparative effectiveness research through a ... - aafp

Comparative Effectiveness ResearchThrough a Collaborative ElectronicReporting ConsortiumAlexander G. Fiks, MD, MSCEa,b,c,d,e, Robert W. Grundmeier, MDb, Jennifer Steffes, MSWe, William G. Adamsf,David C. Kaelber, MD, PhD, MPHg, Wilson D. Pace, MDh, Richard C. Wasserman, MD, MPHe,i,for the Comparative Effectiveness Research Through Collaborative Electronic Reporting (CER2) Consortium

abstractThe United States lacks a system to use routinely collected electronic healthrecord (EHR) clinical data to conduct comparative effectiveness research(CER) on pediatric drug therapeutics and other child health topics. ThisSpecial Article describes the creation and details of a network of EHRnetworks devised to use clinical data in EHRs for conducting CER, led by theAmerican Academy of Pediatrics Pediatric Research in Office Settings (PROS).To achieve this goal, PROS has linked data from its own EHR-based “ePROS”network with data from independent practices and health systems across theUnited States. Beginning with 4 of proof-of-concept retrospective CER studieson psychotropic and asthma medication use and side effects with a plannedfull-scale prospective CER study on treatment of pediatric hypertension, theComparative Effectiveness Research Through Collaborative ElectronicReporting (CER2) collaborators are developing a platform to advance themethodology of pediatric pharmacoepidemiology. CER2 will providea resource for future CER studies in pediatric drug therapeutics and otherchild health topics. This article outlines the vision for and present compositionof this network, governance, and challenges and opportunities for using thenetwork to advance child health and health care. The goal of this network is toengage child health researchers from around the United States in participatingin collaborative research using the CER2 database.

A 2010 Institute of Medicine reportnotes that clinical data, “because oftheir potential to enable thedevelopment of new knowledge and toguide the development of bestpractices from the growing sum ofindividual clinical experiences, . . .represent the resource most central tohealthcare progress.”1 TheComparative Effectiveness ResearchThrough Collaborative ElectronicReporting (CER2) Consortium (http://www2.aap.org/pros/CER2.htm) wascreated to develop a pediatric“learning healthcare system”2

infrastructure to harvest routinelycollected clinical data from electronichealth records (EHRs) and other

sources to generate new knowledgeand improve care. Such approaches areincreasingly valuable as more childrenreceive care using EHRs.3 Althoughmany of the individual sitesrepresented on this consortium’sresearch team had begun separate andlimited efforts in this direction,a system using clinical data froma broad array of practices providingpediatric primary care to conductcomparative effectiveness research(CER), here defined as “the directcomparison of existing health careinterventions to determine whichwork best for which patients andwhich pose the greatest benefits andharms,”4 had not yet been created.

aThe Pediatric Research Consortium, bDepartment ofBiomedical and Health Informatics, cCenter for PediatricClinical Effectiveness, and dPolicyLab at The Children’sHospital of Philadelphia, Philadelphia, Pennsylvania;ePediatric Research in Office Settings, American Academy ofPediatrics, Elk Grove Village, Illinois; fBoston UniversitySchool of Medicine, Boston, Massachusetts; gDepartmentsof Internal Medicine, Pediatrics, and Epidemiology andBiostatistics and the Center for Clinical InformaticsResearch and Education, The MetroHealth System, CaseWestern Reserve University, Cleveland, Ohio; hAmericanAcademy of Family Physicians National Research Network,Leawood, Kansas; and iDepartment of Pediatrics, Universityof Vermont College of Medicine, Burlington, Vermont

Dr Fiks contributed to the conception and design ofthe study and the acquisition and analysis of data,and drafted the manuscript; Drs Grundmeier,Adams, Kaelber, Pace, and Wasserman and MsSteffes contributed to the conception and design ofthe study and the acquisition and analysis of dataand critically reviewed the manuscript; and allauthors approved the final manuscript assubmitted.

www.pediatrics.org/cgi/doi/10.1542/peds.2015-0673

DOI: 10.1542/peds.2015-0673

Accepted for publication Apr 22, 2015

Address correspondence to Alexander G. Fiks, MD,MSCE, The Children’s Hospital of Philadelphia, 3535Market Street, Room 1546, Philadelphia, PA 19104.E-mail: [email protected]

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online,1098-4275).

Copyright © 2015 by the American Academy ofPediatrics

PEDIATRICS Volume 136, number 1, July 2015 SPECIAL ARTICLE at University of Pennsylvania Library on June 23, 2015pediatrics.aappublications.orgDownloaded from

http://www2.aap.org/pros/CER2.htm

http://www2.aap.org/pros/CER2.htm

mailto:[email protected]

http://pediatrics.aappublications.org/

Although such a system may benefitpediatric care in many ways, the needfor such an infrastructure isespecially acute with regard to drugtherapeutics in pediatric populations.It has been estimated repeatedly overthe past 40 years that 75% to 80% ofpharmaceuticals possess insufficientlabeling information for dosing,safety, or efficacy in children.5–10

Even when research is available toguide use, the impact of thesemedications on children in real-worldpractice settings is often poorlyunderstood. To address theseknowledge gaps, large EHR databasesfrom diverse practice settings area particularly helpful resourcebecause they can link prescribingdata with clinical outcomes, canidentify cohorts of children for moredetailed study, and can drive decisionsupport to improve care based onmedical evidence.11 Because mostchildren are healthy, drug use andadverse events in most settings areuncommon. Therefore, largepopulations are needed to study rarepediatric conditions, and manyquestions cannot be readily answeredwithout data pooled across healthsystems.12 In addition, we focus ondata from the primary care medicalhome, because EHRs from this settingprovide a longitudinal record of carethroughout childhood, and althoughonly a subset of children receivespecialty or hospital care, nearly allreceive primary care.

This article describes the creation ofan EHR research network sufficientlylarge to conduct CER, the CER2

Consortium (Fig 1). We detail thenetwork structure, initial researchfocus, governance, and future vision.The consortium was created througha partnership between the AmericanAcademy of Pediatrics (AAP), HealthResources and Service AdministrationMaternal and Child Health Bureau,Eunice Kennedy Shriver NationalInstitute of Child Health & HumanDevelopment and independentpractices and health systems. Thegoal is to enable widespread

collaboration to generate newknowledge to improve child healthand health care. The collaboration ofthe 2 federal agencies that focus onresearch in child populations isespecially significant, becausealthough each has invested resourcesin developing and maintainingresearch networks, neither had basedsuch a network on clinical data fromEHRs.

A SCHOLARLY FOCUS ONPHARMACOEPIDEMIOLOGY COUPLEDWITH THE ABILITY TO SUPPORTA BROAD RESEARCH AGENDA

Given the evidence that moreinformation is needed aboutmedication safety and effectivenessin children, the short-term focusof the CER2 Consortium ispharmacoepidemiologic researchdirected at those issues. As reportedrecently, the Best Pharmaceuticals forChildren Act and the PediatricResearch Equity Act have collectivelyresulted in .500 labeling changes forpediatric medications, providingevidence to support well-informedprescribing.13 However, additionalinformation is needed to guidemedication decisions because, as theAAP Committee on Drugs writes, an

overwhelming number of drugs stillhave no information in the labelingfor use in pediatrics.13 Even whenpediatric labeling does exist,medications are often used for off-label indications and among off-labelage groups. Although off-labelprescribing does not imply that use isharmful or unwarranted,13 expandingthe evidence base is likely to increasethe confidence of both clinicians andfamilies that medications are likely toimprove health without detrimentalside effects.

With this background, the CER2

Consortium was funded with a focuson developing a national resource toevaluate medication safety andeffectiveness in high-priority areas.Specifically, 4 retrospective,observational proof of conceptstudies are being conducted, and 1full-scale prospective trial is plannedto demonstrate the utility of CER2 asa tool to improve medication safety,effectiveness, and prescribing. Thesestudies will assess the prevalence ofuse of specific atypical antipsychoticsin children and adolescents (ages3–18 years) and concomitantmedications; evaluate the effects ofexposure to atypical antipsychoticsand metabolic effects of usage of

FIGURE 1Map of CER2 participating sites. Two hundred and twenty-two practice sites in 27 states representing1.2 million covered lives.

e216 FIKS et al at University of Pennsylvania Library on June 23, 2015pediatrics.aappublications.orgDownloaded from


these medications by basicdemographic factors including self-reported race and ethnicity, child age,and gender; assess the prevalence ofon- and off-label use of frequentlyprescribed asthma medication inchildren aged 0 to 5 years throughEHR data; and examine rates ofpsychotropic polypharmacy inchildren and adolescents (ages 3–18years) and acquire information ondrug safety, efficacy, and side effects.This combination of prospectivestudies guided by EHR data,combined with the secondary dataanalyses, will develop methodsapplicable to future work in diverseareas.

In addition to these proof of conceptstudies, the initial grant that fundsthe CER2 Consortium also supportsa larger-scale evaluation of pediatrichypertension. The estimatedprevalence of hypertension isincreasing, and it now affects 2% to5% of children,14–18 which makes itone of the top 10 chronic diseases inchildhood. Pediatric hypertensionpredisposes children to hypertensionin adulthood. In fact, althoughadditional evidence is needed tosubstantiate the direct link betweenroutine blood pressure measurementand the identification of children atincreased risk of adult cardiovasculardisease,19 multiple early markers ofcardiovascular disease such as leftventricular hypertrophy, arterialintima-media thickness, alteredarterial compliance, atherosclerosis,and diastolic dysfunction have beenrecognized in childhood.20–27

Additionally, secondary hypertensionis more common in children thanadults, so prompt recognition may beparticularly important to potentiallydiagnose and treat underlyingdiseases.28 Despite clear criteria forthe diagnosis and well-definedrecommendations for the evaluationand management of hypertensivechildren (the National Heart, Lung,and Blood Institute recommendsscreening starting at 3 years ofage),29 this condition often remains

undiagnosed. Therefore, theopportunity exists to greatly increasethe diagnosis and improve theevaluation and management ofhypertension in the study’s targetpopulation (children and adolescents3–18 years old). The consortiumseeks to better understand theepidemiology and effectiveness oftreatment of this condition and, byusing advanced clinical decisionsupport (CDS) tools in EHRs, toautomatically alert physicians at thepoint of care to improve recognition,evaluation, and management ofpediatric hypertension.

The goal of CER2 is to engage scholarsfrom around the country in a diverserange of research projects focused onpediatric medication use, safety, andeffectiveness, as well as other areasincluding preventive care, treatmentof acute conditions, and chronicdisease management. For example,PROS and the Children’s Hospital ofPhiladelphia (CHOP) PediatricResearch Consortium (PeRC) haveused data from CER2 sites to studythe impact of patient portals ondecision making for pediatricasthma.30 In the future, we intend toengage epidemiologists and healthservice researchers in generalist andspecialist fields in a range of researchtopics by using this database.

A NETWORK STRUCTURE TO SUPPORTGENERALIZABLE RESEARCH

The CER2 Consortium was designedwith the explicit goal of includinghealth systems and independentpractices that, taken together, wouldreflect the broader range of settingswhere children in the United Statesreceive care. The use of this approachgreatly increases the likelihood thatfindings from the network will begeneralizable. Toward this end,PROS’s own network of largelyindependent pediatric practices31

partnered with CHOP and its PeRCnetwork,31 the American Academy ofFamily Physicians Electronic NationalQuality Improvement and Research

Network (eNQUIRENet), theMetroHealth System/Case WesternReserve School of Medicine, andBoston Medical Center/BostonHealthNet i2b2 database.32 Inaddition, the Allied Physicians group,a unified practice group of 25suburban practices in the New Yorkarea sharing a common EHR, andEskenazi Health, a multisite healthsystem in the Indianapolis area, havealso joined the consortium and begunto contribute data. Details on thecharacteristics of these networks areshown in Table 1. Together, the CER2

sites have the capacity to performprimary care drug therapeutics CERon a racially, ethnically, andeconomically diverse group of .1.2million children seen in urban,suburban, and rural private andpublic sector practices and clinics bypediatricians, family physicians, nursepractitioners, and physicianassistants.

NETWORK GOVERNANCE

A governance document for CER2 wascreated by the consortium to clarifythe goals, responsibilities, andprotections afforded to thosecontributing data and performingresearch. Specific details of thegovernance document address datastorage and stewardship (includingadministrative, technical, and physicalsafeguards) as well as requiredtraining for researchers using thedata. Overall, protections addressrisks to data privacy and, to avoidpublic comparisons between healthsystems, emphasize that datapresented publicly will not single outhealth systems or practices by name.As an independent organizationrepresenting pediatricians nationally,the AAP owns the data contributed tothe consortium and, through theexisting AAP Department of Researchand PROS network, acts as an honestbroker to oversee research throughCER2. Personal identifyinginformation has been limited to birthdates, dates of service, and 5-digit zip

PEDIATRICS Volume 136, number 1, July 2015 e217 at University of Pennsylvania Library on June 23, 2015pediatrics.aappublications.orgDownloaded from


codes of residence (a limited dataset). Within the governancedocument, research allowable by theconsortium was broadly defined toinclude studies that compare variousinterventions, including therapeutics,diagnostics, and practice-leveldelivery system characteristics;

identify best practices; and informinnovations in care delivery andpayment mechanisms. Thegovernance structure allows 2 levelsof participation for institutions:Partners of the consortium, whocontribute data, and Affiliates, whomay or may not provide data. In

addition, permission may be grantedfor individual researchers to use thedata even if their institution has notjoined the consortium. This flexiblestructure was created to encourageparticipation by researchers fromvaried organizations while alsorecognizing the particular

TABLE 1 CER2 Practice Characteristics

Organization Sites Number ofChildren

EHR Medicaid, % Number ofProviders

Race, % Ethnicity, % Practice Setting

PeRC 31 412 320 Epic 4.2% 209 White: 61.0% Hispanic: 5.5% 4 urbanBlack: 19.4% 27 suburbanAsian: 3.2%Other or

multiplea: 16.4%MetroHealth 33 135 215 Epic 81.7% 138 White: 31.2% Hispanic: 17.9% 26 urban

Black: 47.5% 7 suburbanAsian: 1.7%Other or

multiplea: 12.2%Unknown or

declined: 7.4%Boston Medical

Center18 53 202 Centricity 79.0% 59 White: 11.0% Hispanic: 11.0% 18 urban

Black: 45.0%Asian: 6.0%Other or


declined: 16.8%eNQUIRENet 65 158 773 Centricity, NextGen,

Allscripts, eCW, eMDs16.8% 1467 White: 33.7% Hispanic: 27.8% Not availableb

Black: 4.8%Asian: 1.6%Unknown or

declined: 59.9%ePROS 39 291 051 Allscripts, Connexin, EHR

Scope (EHS), eMDs, GE,Greenway Medical, PCC

35.0% 102 White: 68.0% Hispanic 11.0% 9 urbanBlack: 22.5% 23 suburbanAsian: 5.0% 7 ruralOther or


declined: 2.0%Eskenazi 10 103 681 Regenstrief 77.6% 42 White: 23.1% Hispanic: 11.2% 10 urban

Black: 42.0%Hispanic: 11.2%Other or


declined: 19.8%Allied Physicians 26 91 129 Centricity Not availablec 102 Not availablec Not availablec 2 urban

24 suburban

Totals 222 1 245 371 30.9% 2119 practitioners White: 43.8% Hispanic: 12.6% 69 urbanBlack: 21.9% 81 suburbanAsian: 3.7% 7 ruralOther or

multiplea: 7.9%65 unknownb

Unknown ordeclined: 22.7%

eCW, eClinicalWorks; GE, General Electric; PCC, Physician’s Computer Company.a “Other race” includes Native American/Alaskan Native, Native Hawaiian/Pacific Islander, and multiple races.b The number of practices in each type of setting was not available.c This information was not routinely captured by practices.



contributions of organizations thatshare data. To ensure that theinterests of institutions sharing dataare protected, an executive committeemade up of the Partners decideswhich studies should proceed usingthe consortium data. A framework isprovided for settling any disputesthat arise. The guiding principle ofthis group is to support the conductof studies by Partners and Affiliatesdesigned to advance health andhealth care for children. Therefore,the governance document explicitlyemphasizes the importance ofpublishing studies conducted withconsortium data.

DATA ORGANIZATION AND ACCESS

Researchers seeking to assembledata from disparate health systemscan use either a federated ora centralized database model.A federated database structurefacilitates sharing and interchangesof data between autonomousdatabases, such as EHRs locatedwithin different practice or clinicsites or organizations, unitingindependent database systems toshare and exchange information.33

Authorized users access these datathrough Web portals that query datathat is aggregated virtually foranalyses. A primary advantage ofthis data model is that data neverleave the host institution, avoidingchallenges such as slow transferspeeds and potential privacy threatsthat may arise when data aremoved between institutions.Prominent networks that have useda federated model include theHealth Maintenance OrganizationResearch Network,34 the ScalableArchitecture for TherapeuticInquiries Network,35 and theShared Health Research InformationNetwork, a Web-based approach ledby the Harvard Clinical andTranslational Science Centerand involving Harvard-affiliatedacademic teaching hospitals thatallows researchers to query

aggregate data about demographics,diagnoses, medications, andselected laboratory values acrosshospitals.

The CER2 Consortium has chosen tocentralize data within a contracteddata coordinating center at CHOP,which is also a data partner (Fig 2).The consortium provides methodsfor collaborators to directly andsecurely access the data where theyreside at CHOP. This model isparticularly well suited to theconsortium, and potentially verybroadly generalizable, because dataare aggregated from sites that maylack sufficient staff (in number,training, or both) to locally extractand transform data intoa prespecified format, validate thedata, and adequately maintain thedatabase to ensure that researcherscan access it. With this centralizedmodel, sites assemble data inwhatever structure is mostconvenient and then review the datafor compliance with the HealthInsurance Portability andAccountability Act and applicablelocal institutional review board(IRB) protocols. Then, after anadditional round of review toconfirm consistency with IRBprotocols and avoid breaches ofprivacy, clinical informaticians,statisticians, and data analystsworking with the data at CHOPsystematically review the data forquality. This process involvesa thorough review of data tables toflag missing data, outliers, andimplausible values. As needed,CHOP works with sites contributingdata to validate data elements byinspection and chart review. Todate, this iterative review hasproven particularly helpful toensure that needed data elementsare extracted, that elements outsidethe scope of approved projects andregulatory approvals are excluded,and that the strengths andlimitations of data from differentsites are well characterized. Thisprocess helps to identify networks

with data that are relevant tospecific study questions and mayalso lead to additional work toaddress data gaps. For example,1 contributing network structureddata at the patient level, withoutproviding links to specificencounters. Review ultimatelytriggered revisions to the datastructure, facilitating the extractionof visit-level diagnostic and billingcodes needed to achieve researchgoals.

The lack of agreed-upon EHR datastandards complicates efforts toaggregate and validate the data. Forthe CER2 Consortium, adoption ofa common final data model for allEHR data was necessary. Thosecontributing data to the CER2

Consortium use a mix of dataformats including the Clarity modelused for Epic (Verona, WI), the i2b2model36 used by Boston MedicalCenter/Boston HealthNet, theObservational Medical OutcomesPartnership (OMOP)37 used byeNQUIRENet and ePROS, and theRegenstrief Medical Record Systemused in the Eskenazi system.Because of its history as a tool forpharmacoepidemiologic research,the presence of a data conversiontool already developed by dataanalysts in CHOP’s Department ofBiomedical and Health Informatics,the inclusion of both patient- andencounter-level data, and the abilityto handle claim data, the teamultimately chose by consensus touse the OMOP data model andstandardize all data using readilyavailable standard-basedvocabularies from OMOP. Given thatnot all participating sites andinvestigators have expertise inOMOP, the Department ofBiomedical and Health Informaticshas assumed the role of convertingdata from OMOP as needed tosupport data analyses for particularprojects.

In the process of data extraction,additional steps were taken to



facilitate future growth of theconsortium. Specifically, when weextracted data from sites witha common data model (eg, Epic andi2b2), we developed reusable scripts.These scripts were packaged tofacilitate easy installation atparticipating sites. The CHOP datacoordinating center maintains thesescripts and provides assistance asneeded to the sites. This approachenables us both to pull new data fromcurrent collaborators and to morereadily extract data fromcollaborators who join the

consortium and use the same EHRs asexisting members.

INNOVATIONS IN CER2 TO ADDRESS THELIMITATIONS OF EHR DATA FOR CER

Standardized EHR data, pooled acrosshealth systems in the CER2 database,provide readily accessible, detailedinformation on children’sdemographic characteristics, medicalproblems, growth, medicationsprescribed, allergies, and test results,including laboratory studies.However, specialized approaches areneeded to address certain limitations

of EHR data. Examples of theseapproaches follow.

Central to assessing child health isthe monitoring of growth, yetgrowth data in EHRs are vulnerableto quality problems. Common errorsin growth data include weights ofclothed children, incorrectplacement of decimal points,missing digits due to keystrokeerrors, and unintentional miscodingof metric and English system values.Although these errors are common,it is not feasible to detect them withmanual chart review because of thevolume of data. As a result, ourgroup is working with an expert ingrowth statistics to develop andvalidate algorithms to facilitate theautomated cleaning of growth data.Preliminary data indicate that thesestrategies, which use anexponentially weighted movingaverage of all other SD scores fora particular growth parameter todetermine whether a growth valueshould be retained, will be highlyeffective in ensuring the validity ofdata points used in CER studies withgrowth as a main outcome,exposure, or covariate.38 For futureand ongoing studies, we expect toimplement validated algorithmsbefore data analyses focused ongrowth to ensure that erroneousvalues are excluded and trueoutlier values retained. This workprovides a model for developingrigorous, automated methods toclean large EHR data sets. Suchapproaches will be increasinglyneeded to support valid researchinvolving these data sets and fulfillthe promise of “clinical data” tomeaningfully advance health andhealth care.

Evaluating health disparities isanother high priority area for theconsortium. However, despite theprioritization of capturinginformation on race and ethnicity inEHRs as part of the FederalMeaningful Use Program,39 data onrace and ethnicity are often missing.

FIGURE 2Data flow for the CER2 Consortium. Data from each primary care practice are initially extracted atthe local level from the on-site EHR to the local clinical data repository, which is located behind eachpractice or organization’s firewall. Study data are then deidentified and securely transferred to theData Coordinating Center at CHOP, where they are stored on a secure offsite server and aggregatedwith data from other participating practices and networks.



To address this knowledge gap,consortium researchers areevaluating algorithms, validated inadults, that use data on surname andaddress in addition to standardcovariates to impute data on ethnicityand race. Our results suggest thatthese methods are effective forchildren as well as adults, even giventhe challenges of a growing numberof children living in multiracial,multiethnic families.40 With studiessuch as these, the consortium hopesto advance both the fields of growthmeasurement and disparitiesresearch and the broader goal ofaddressing missing or erroneous datain EHRs.

Although EHR data offer substantialadvantages over administrative datain providing detailed clinicalinformation that extends beyondbilling codes, insights into clinicaldecision making often entaila review of free text, especiallyclinician notes. Therefore,consortium members are working toobtain funding that will both testand advance the ability ofcommercially available text parsersto make sense of text elements suchas medical histories (including thedocumentation of side effects) andassessments that are found in notes.Although this work is at an earlystage, the application ofmathematical and engineeringapproaches will increasingly make itpossible to extract information innotes and reports across millions ofencounters and diverse EHRenvironments. In addition, with theanalysis of free text, a majorconcern is privacy, calling for thedevelopment of workflows thatallow individual sites to codenotes by using standardizedalgorithms that remove identifiersbefore the aggregation of thesedata.41

In addition to developing methods toimprove the utility of secondary EHRdata for CER, the research team plansto develop and test strategies by

using secondary data to identifyparticipants from whomsupplementary prospective datacollection is warranted. Thesestrategies, already tested atindividual sites within theconsortium, require additionaldevelopment to function effectivelyacross the network. Initialevaluations of these approaches willbe part of the first set of plannedconsortium studies and will focuson identifying adverse effects ofmedication on children 6 to 18 yearsold receiving psychotropicpolypharmacy. For these studies, theEHR database will be used toidentify children meeting inclusioncriteria and their families who, withapproval from applicable IRBs, willbe contacted for prospective datacollection.

To proactively improve care,consortium studies also plan toimplement CDS. Such studies willaddress a key barrier in the CDSfield: the difficulty in replicating thebenefits of CDS approaches fordiverse practices that use multipleEHRs.42 Our hypertension CDS willfocus on helping clinicians improvethe identification, evaluation, andmanagement of the many childrenwho have unrecognizedhypertension by promptingclinicians to recognize and addressabnormal values and use evidence-based guidelines.43 For example,the planned decision supports willencourage clinicians to recheckabnormal blood pressure valuesusing appropriate-sized bloodpressure cuffs to help distinguish“white coat” hypertension fromtruly elevated blood pressures. Forthis study, EHR modifications willbe implemented at multiple sites,and the impact of the modificationswill be monitored. This work willbuild on the success of CDSapproaches in one of theconsortium sites, the MetroHealthSystem.44 Ultimately, our effortswill demonstrate our ability togeneralize CDS strategies

developed at a single site across thenetwork.

Even with these innovations, certainlimitations of EHR data derived frompediatric practices will persist.Although data will reflect the detailsof clinical care provided in highlyvaried practice settings and thenetwork includes sites caring formany underserved communities,children in CER2 sites are nota random sample meant to reflectregional or overall US populations.As in any resource that uses datafrom medical records, variation inphysician documentation mayintroduce bias in the data that arecaptured. Prospective datacollection, as well as manual chartreview, may be needed to preventor address these biases in certaincircumstances. After a detailedreview of the data, consortiumresearchers will adopt thesestrategies as needed. Furthermore,it is not possible to link childrenacross health systems in CER2.However, given the geographicdiversity of study sites, weexpect few children will be cared foracross multiple studysites, mitigating this potentialproblem.

CONCLUSIONS AND FUTUREDIRECTIONS

The CER2 Consortium leveragesthe clinical data in EHRs, primarilyin primary care settings, for .1.2million children (∼1.8% of the USpopulation ,18 years of age)with the short-term goal ofpromoting medication safety andeffectiveness for children and thelonger-term objective of derivinginsights from real-world data tobroadly improve child health andhealth care. To achieve these goals,the consortium has establisheda flexible governance structurethat prioritizes both patient privacyand data availability to researchers.Data are available within asecure research environment and



stored in the standard OMOPdata model, a format increasinglyfamiliar to health serviceresearchers around the UnitedStates. Advancing the network’smission, multiple studies areunder way that will detailresearch methods to improve theanalysis of secondary data,supplement routinely collected EHRdata with prospective datacollection, and use CDS to supporthealth care decision-making. Tomeet long-term goals in specificareas, data may need to be linkedfrom subspecialty clinics,emergency departments, andhospitals, following children acrossthe continuum of care to betterextract new knowledge. As theconsortium evolves, thelongitudinal record in the primarycare EHR will provide a foundationfor understanding the diagnosis,treatment, and evaluation ofchildren over time for a broad

population of children withvarying degrees of medicalcomplexity.

ACKNOWLEDGMENTS

We thank Stephanie Mayne,Banita McCarn, and Weiwei Liufor help with background researchand manuscript preparation. Inaddition, we recognize BenyaminMargolis, PhD, Stella Yu, ScD,MPH, Romuladus Azuine, DrPH,Anne Zajicek, MD, PharmD,Perdita Taylor-Zapata, MD, andGary Mirkin, MD, for theircontributions to the network.Finally, we recognize the rest ofthe CER2 team: Russell Localio,PhD, Michelle Ross, PhD, EliasBrandt, Janeen Leon, ElenaDeBartolo, Carolina Freuder,Rob Valuck, PhD, Laurel Leslie,MD, Laura Shone, DrPH, CarrieDaymont, MD, and AngelaCzaja, MD.

ABBREVIATIONS

AAP: American Academy ofPediatrics

CDS: clinical decision supportCER: comparative effectiveness

researchCER2: Comparative Effectiveness

Research ThroughCollaborative ElectronicReporting

CHOP: The Children’s Hospital ofPhiladelphia

EHR: electronic health recordeNQUIRENet: Electronic National

QualityImprovement andResearch Network

IRB: institutional review boardOMOP: Observational Medical

Outcomes PartnershipPeRC: Pediatric Research

ConsortiumPROS: Pediatric Research in Office

Settings

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

FUNDING: Supported by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) under grant

R40MC24943 and title “Primary Care Drug Therapeutics CER in a Pediatric EHR Network,” number UB5MC20286 and title “Pediatric Primary Care EHR Network for

CER,” and number UA6MC15585 and title “National Research Network to Improve Child Health Care.” This information or content and conclusions are those of the

authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by, HRSA, HHS, or the US government. Funding

was provided, in part, by a grant from the Eunice Kennedy Shriver National Institute of Child Health and Human Development under the Best Pharmaceuticals for

Children Act.

POTENTIAL CONFLICT OF INTEREST: Alexander G. Fiks is the associate director and Richard C. Wasserman is the director of Pediatric Research in Office Settings.

Jennifer Steffes is an employee of the American Academy of Pediatrics who works for Pediatric Research in Office Settings. All other authors collaborate with

Pediatric Research in Office Settings.

REFERENCES

1. Institute of Medicine (IOM). Clinical Dataas the Basic Staple of Health Learning:Creating and Protecting a Public Good:Workshop Summary. Washington, DC:National Academies Press; 2010

2. Institute of Medicine (IOM). LearningHealthcare System: Workshop Summary.Washington, DC: National AcademiesPress; 2007

3. Lehmann CU, O’Connor KG, Shorte VA,Johnson TD. Use of electronic healthrecord systems by office-basedpediatricians. Pediatrics. 2015;135(1).Available at: www.pediatrics.org/cgi/content/full/135/1/e7

4. Slutsky JR, Clancy CM. AHRQ’s EffectiveHealth Care Program: why comparativeeffectiveness matters. Am J Med Qual.2009;24(1):67–70

5. Benjamin DK Jr, Smith PB, Murphy MD,et al. Peer-reviewed publication of clinicaltrials completed for pediatric exclusivity.JAMA. 2006;296(10):1266–1273

6. BlumerJL. Off-label usesof drugs in children.Pediatrics. 1999;104(3 pt 2):598–602

7. US Department of Health and HumanServices, Public Health Service, Food andDrug Administration, Center for DrugEvaluation and Research. Offices of Drug

Evaluation Statistical Report. Rockville,MD: US Department of Health and HumanServices; 1989

8. American Academy of Pediatrics Committeeon Drugs. Unapproved uses of approveddrugs: thephysician, thepackage insert, andthe Food and Drug Administration: subjectreview. Pediatrics. 1996;98(1):143–145

9. Gilman JT, Gal P. Pharmacokinetic andpharmacodynamic data collection inchildren and neonates. A quiet frontier.Clin Pharmacokinet. 1992;23(1):1–9

10. Wilson J. Pragmatic assessment ofmedicines available for young children


http://www.pediatrics.org/cgi/content/full/135/1/e7

http://www.pediatrics.org/cgi/content/full/135/1/e7


and pregnant or breast-feeding women.In: Morselli PLGS, Sereni F, eds. Basic andTherapeutic Aspects of PerinatalPharmacology. New York, NY: RavenPress; 1975:411–421

11. Shojania KG, Jennings A, Mayhew A,Ramsay CR, Eccles MP, Grimshaw J. Theeffects of on-screen, point of carecomputer reminders on processes andoutcomes of care. Cochrane DatabaseSyst Rev. 2009;(3):CD001096

12. Lannon CM, Peterson LE. Pediatriccollaborative networks for qualityimprovement and research. AcadPediatr. 2013;13(6 suppl):S69–S74

13. Frattarelli DA, Galinkin JL, Green TP, et al;American Academy of PediatricsCommittee on Drugs. Off-label use ofdrugs in children. Pediatrics. 2014;133(3):563–567

14. Sorof JM, Lai D, Turner J, Poffenbarger T,Portman RJ. Overweight, ethnicity, andthe prevalence of hypertension inschool-aged children. Pediatrics. 2004;113(3 pt 1):475–482

15. Moore WE, Stephens A, Wilson T, WilsonW, Eichner JE. Body mass index andblood pressure screening in a ruralpublic school system: the Healthy KidsProject. Prev Chronic Dis. 2006;3(4):A114

16. Ford ES, Mokdad AH, Ajani UA. Trends inrisk factors for cardiovascular diseaseamong children and adolescents in theUnited States. Pediatrics. 2004;114(6):1534–1544

17. Luepker RV, Jacobs DR, Prineas RJ,Sinaiko AR. Secular trends of bloodpressure and body size in a multi-ethnicadolescent population: 1986 to 1996.J Pediatr. 1999;134(6):668–674

18. Morrison JA, James FW, Sprecher DL,Khoury PR, Daniels SR. Sex and racedifferences in cardiovascular diseaserisk factor changes in schoolchildren,1975–1990: the Princeton School Study.Am J Public Health. 1999;89(11):1708–1714

19. Thompson M, Dana T, Bougatsos C,Blazina I, Norris S. Screening forHypertension in Children andAdolescents to Prevent CardiovascularDisease: Systematic Review for the U.S.Preventive Services Task Force. Rockville,MD: Agency for Healthcare Research andQuality; 2013

20. Arnett DK, Glasser SP, McVeigh G, et al.Blood pressure and arterial compliancein young adults: the Minnesota Children’sBlood Pressure Study. Am J Hypertens.2001;14(3):200–205

21. Berenson GS, Srinivasan SR, Bao W,Newman WP III, Tracy RE, Wattigney WA.Association between multiplecardiovascular risk factors andatherosclerosis in children and youngadults. The Bogalusa Heart Study. N EnglJ Med. 1998;338(23):1650–1656

22. Burke GL, Arcilla RA, Culpepper WS,Webber LS, Chiang YK, Berenson GS.Blood pressure and echocardiographicmeasures in children: the BogalusaHeart Study. Circulation. 1987;75(1):106–114

23. Johnson MC, Bergersen LJ, Beck A, DickG, Cole BR. Diastolic function andtachycardia in hypertensive children. AmJ Hypertens. 1999;12(10 pt 1):1009–1014

24. Knoflach M, Kiechl S, Kind M, et al.Cardiovascular risk factors andatherosclerosis in young males: ARMYstudy (Atherosclerosis Risk-Factors inMale Youngsters). Circulation. 2003;108(9):1064–1069

25. Lauer RM, Clarke WR. Childhood riskfactors for high adult blood pressure:the Muscatine Study. Pediatrics. 1989;84(4):633–641

26. Malcolm DD, Burns TL, Mahoney LT, LauerRM. Factors affecting left ventricularmass in childhood: the Muscatine Study.Pediatrics. 1993;92(5):703–709

27. Sanchez A, Barth JD, Zhang L. The carotidartery wall thickness in teenagers isrelated to their diet and the typical riskfactors of heart disease among adults.Atherosclerosis. 2000;152(1):265–266

28. Sinaiko AR. Hypertension in children.N Engl J Med. 1996;335(26):1968–1973

29. National High Blood Pressure EducationProgram Work Group on High BloodPressure in Children and Adolescents.The fourth report on the diagnosis,evaluation, and treatment of high bloodpressure in children and adolescents.Pediatrics. 2004;114(2 suppl 4th report):555–576

30. Fiks AG, duRivage N, Finch S, et al.Determinants and outcomes of providerand family participation in a multi-stateasthma portal implementation. Posterpresentation at the 2015 Pediatric

Academic Societies Annual Meeting;2015; San Diego, CA

31. Fiks AG, Grundmeier RW, Margolis B,et al. Comparative effectivenessresearch using the electronic medicalrecord: an emerging area ofinvestigation in pediatric primary care.J Pediatr. 2012;160(5):719–724

32. Adams WG, Anderson N, Berner ES, et alDistributed health outcome monitoringand evaluation using i2b2. AMIA JtSummits Transl Sci Proc. 2013;2013:2

33. Maro JC, Platt R, Holmes JH, et al. Designof a national distributed health datanetwork. Ann Intern Med. 2009;151(5):341–344

34. Thompson EE, Steiner JF. Embeddedresearch to improve health: the 20thannual HMO Research Networkconference, March 31–April 3, 2014,Phoenix, Arizona. Clin Med Res. 2014;12(1–2):73–76

35. Schilling LM, Kwan BM, Drolshagen CT,et al Scalable Architecture for FederatedTranslational Inquiries Network(SAFTINet) technology infrastructure fora distributed data network. eGEMS. 2013;1(1):11

36. Murphy S, Churchill S, Bry L, et al.Instrumenting the health care enterprisefor discovery research in the genomicera. Genome Res. 2009;19(9):1675–1681

37. Madigan D, Stang PE, Berlin JA, et al.A systematic statistical approach toevaluating evidence from observationalstudies. Annu Rev Stat Appl. 2014;1:11–39

38. Daymont C, Fiks AG, Grundmeier RW. Anautomated method for cleaning growthmeasurements in large data sets. Posterpresentation at the 2014 PediatricAcademic Societies Annual Meeting;2014; Vancouver, BC

39. Centers for Medicare & MedicaidServices (CMS), HHS. Medicare andMedicaid programs; electronic healthrecord incentive program—stage 2.Final rule. Fed Regist. 2012;77(171):53967–54162

40. Grundmeier RW, Song L, Ramos MJ, et alImputing missing race/ethnicity inpediatric electronic health records:reducing bias with use of US censuslocation and surname data [publishedonline ahead of print March 11, 2015].Health Serv Res. 2015. doi:10.1111/1475-6773.12295



41. Friedlin FJ, McDonald CJ. A software toolfor removing patient identifyinginformation from clinical documents.J Am Med Inform Assoc. 2008;15(5):601–610

42. Osheroff JA, Pifer EA, Teich JM, Sittig DF,Jenders RA. Improving Outcomes WithClinical Decision Support: AnImplementer’s Guide. Chicago, IL:

Healthcare Information and ManagementSystems Society; 2005

43. Gunn PW, Hansen ML, Kaelber DC.Underdiagnosis of pediatrichypertension: an example of a new eraof clinical research enabled byelectronic medical records. AMIA AnnuSymp Proc. October 2007:966

44. Bar-shain D, Palcisco K, Greco PJ,Kaelber DC. Using advanced electronicclinical decision support to improve thequality and recognition of abnormalblood pressure values in children. Oralpresentation at the 2013 PediatricAcademic Societies Annual Meeting;2013; Washington, DC



DOI: 10.1542/peds.2015-0673; originally published online June 22, 2015;Pediatrics

ConsortiumEffectiveness Research Through Collaborative Electronic Reporting (CER 2 )

David C. Kaelber, Wilson D. Pace, Richard C. Wasserman and for the Comparative Alexander G. Fiks, Robert W. Grundmeier, Jennifer Steffes, William G. Adams,

Reporting ConsortiumComparative Effectiveness Research Through a Collaborative Electronic

ServicesUpdated Information &

/peds.2015-0673http://pediatrics.aappublications.org/content/early/2015/06/16including high resolution figures, can be found at:

Permissions & Licensing

tmlhttp://pediatrics.aappublications.org/site/misc/Permissions.xhtables) or in its entirety can be found online at: Information about reproducing this article in parts (figures,

Reprints http://pediatrics.aappublications.org/site/misc/reprints.xhtml

Information about ordering reprints can be found online:

rights reserved. Print ISSN: 0031-4005. Online ISSN: 1098-4275.Grove Village, Illinois, 60007. Copyright © 2015 by the American Academy of Pediatrics. All and trademarked by the American Academy of Pediatrics, 141 Northwest Point Boulevard, Elkpublication, it has been published continuously since 1948. PEDIATRICS is owned, published, PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly

at University of Pennsylvania Library on June 23, 2015pediatrics.aappublications.orgDownloaded from

http://pediatrics.aappublications.org/content/early/2015/06/16/peds.2015-0673


http://pediatrics.aappublications.org/site/misc/Permissions.xhtml

http://pediatrics.aappublications.org/site/misc/Permissions.xhtml

http://pediatrics.aappublications.org/site/misc/reprints.xhtml


DOI: 10.1542/peds.2015-0673; originally published online June 22, 2015;Pediatrics

ConsortiumEffectiveness Research Through Collaborative Electronic Reporting (CER 2 )

David C. Kaelber, Wilson D. Pace, Richard C. Wasserman and for the Comparative Alexander G. Fiks, Robert W. Grundmeier, Jennifer Steffes, William G. Adams,

Reporting ConsortiumComparative Effectiveness Research Through a Collaborative Electronic


located on the World Wide Web at: The online version of this article, along with updated information and services, is

of Pediatrics. All rights reserved. Print ISSN: 0031-4005. Online ISSN: 1098-4275.Boulevard, Elk Grove Village, Illinois, 60007. Copyright © 2015 by the American Academy published, and trademarked by the American Academy of Pediatrics, 141 Northwest Pointpublication, it has been published continuously since 1948. PEDIATRICS is owned, PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly

at University of Pennsylvania Library on June 23, 2015pediatrics.aappublications.orgDownloaded from



comparative effectiveness research through a ... - aafp

Documents