The HMO Research Network (HMORN) is a well established alliance of 18 research departments in the United States and Israel. Since 1994, the HMORN has conducted numerous multi-institutional studies encompassing a wide range of methodologies and scientific disciplines. As part of these studies, HMORN-based scientists have published papers that describe their approaches to solving many of the problems facing multi-institutional research.

We present here a review of these papers in order to provide guidance to newer networks and to identify the challenges that continue to confront multi-institutional research. Careful attention to such organizational themes can help the HMORN and newer multi-institutional consortia develop into sustainable “learning research networks”.

BACKGROUNDBACKGROUNDData model, quality and validityData model, quality and validity • Range of data elements available, benefits of a research data

warehouse, examples of studies supported by a data warehouse.• HMORN virtual data warehouse (VDW) and its approach to

standardizing variable names, labels, definitions and coding.• Approaches adopted by HMORN studies and networks to

standardize assessment of data quality. • Proposed conceptual model for multisite data quality assessment.• Site-level variation in variables; validity of critical data elements;

performance of alternative algorithms for establishing diagnoses.• Issues in research design (e.g., calculating associations; sample

attrition; date accuracy; data completeness; representativeness).

Governance and access to the network and its dataGovernance and access to the network and its data • Governance structure of the HMORN. • Governance structure of HMORN scientific networks.• Distributed research network data governance models. • Progressively more sophisticated distributed query software.

Protection of the privacy and safety of human subjectsProtection of the privacy and safety of human subjects • Delays due to variability in IRB review for multisite studies. • Desire of investigators for facilitated multisite IRB review.• HMORN strategy for IRB review of multisite project protocols.

Administrative processes in researchAdministrative processes in research • Need to continually improve the efficiency of administrative

processes (e.g., contracting and financial closeout).• Need to develop standardized approaches for establishment of

data use and business associate agreements. • Cancer Research Network assessment of administrative

processes, and staff and PI satisfaction with those processes.

Management of knowledge• Needs for standardized tools to help conduct studies efficiently. • Description of www.researchtoolkit.org resource compendium.

Relationships with host organizations • Interest of HMO leaders, clinicians, and members in participating

in traditional or cluster randomized trials. • Faster translation of research into practice as a potential

advantage of HMORN-based research.

Describing and Optimizing Research Collaborations: A Review of Describing and Optimizing Research Collaborations: A Review of the HMORN Literaturethe HMORN Literature John F. Steiner, MD, MPH and Andrea Paolino, MA (Kaiser Permanente Institute for Health Research - Denver, CO)John F. Steiner, MD, MPH and Andrea Paolino, MA (Kaiser Permanente Institute for Health Research - Denver, CO)

Eric B. Larson, MD, MPH and Ella E. Thompson, BS (Group Health Research Institute - Seattle, WA)Eric B. Larson, MD, MPH and Ella E. Thompson, BS (Group Health Research Institute - Seattle, WA)

Four papers were purely conceptual, 22 were descriptive, and 26 included an analytic component to assess the performance of some aspect of the HMORN. Of the 52 items, the primary themes were organization, quality, or the validity of data elements (30), data and network governance (11), human subjects privacy and safety (4), the relationship between the research enterprise and the “host” health care system (4), knowledge management (2), and business processes (1).

RESULTSRESULTS

COMMON THEMESCOMMON THEMES

Commissioned by the AcademyHealth Electronic Data Methods (EDM) Forum, a project supported by the Agency for Healthcare Research and Quality (AHRQ) through the American Recovery & Reinvestment Act of 2009, Grant U13 HS19564-01

ACKNOWLEDGEMENTSACKNOWLEDGEMENTS

We reviewed 52 publications by HMORN investigators through December, 2012 that described general issues conducting research within the Network. Two reviewers independently reviewed and assigned to each paper a single purpose (conceptual, descriptive, or analytic), and one or more of six primary and secondary themes: (a) organization, quality, or the validity of data elements; (b) data and network governance; (c) human subjects privacy and safety; (d) the relationship between the research enterprise and the “host” health care system; (e) knowledge management; and (f) business processes. Reviewers resolved differences in their ratings by consensus.

METHODSMETHODS The HMORN must evaluate its processes by developing standard metrics to compare performance across sites, and demonstrate the ability to conduct research better, cheaper and faster than competitors.

Work already underway to address gaps in the literature and document HMORN research efficiencies include:

• Developing targeted publications to close key gaps.• Developing and capturing IRB and contracting efficiency metrics.• Improving knowledge management tools and approaches.• Collecting operational benchmarks for research center productivity

and efficiency to support site-level quality improvement efforts.

A LEARNING RESEARCH NETWORKA LEARNING RESEARCH NETWORK

THE STATE OF THE LITERATURETHE STATE OF THE LITERATURE

Our review identified numerous gaps in the published literature. Such gaps in the literature exist for numerous reasons. • Incentive is to publish scientific content, not research processes.• Few forums or opportunities exist to publish research efficiencies.• Increasingly limited funding exists for infrastructure related work.• HMORN’s informal nature and site variation make standardization challenging.

GAPS IN THE LITERATUREGAPS IN THE LITERATURE

ANALYTIC PAPERSANALYTIC PAPERS

DESCRIPTIVE PAPERSDESCRIPTIVE PAPERS

CONCEPTUAL PAPERSCONCEPTUAL PAPERS

FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee Chen,1997 Data models, quality, or validity Somkin, 2005 Relationship with health system Field, 2004 Data models, quality, or validity Mazor, 2007 Relationship with health system Raebel, 2005 Data models, quality, or validity Somkin, 2008 Relationship with health system Davis, 2005 Data models, quality, or validity Mazor, 2009 Relationship with health system Herrinton, 2007 Data models, quality, or validity Bowles, 2009 Data models, quality, or validity Greene, 2005 (JNCI) Administrative processes in research Arteburn, 2010 Data models, quality, or validity Smith, 2010 Data models, quality, or validity Greene, 2006 Human subjects privacy and safety Davis, 2010 Data models, quality, or validity Andrade, 2011 Data models, quality, or validity Brown, 2010 Network / data governance and access Scholes, 2011 Data models, quality, or validity Curtis, 2012 Data models, quality, or validity Dolor, 2011 Knowledge management Koebnick, 2012 Data models, quality, or validity Ritzwoller, 2012 Data models, quality, or validity Nichols, 2012 Data models, quality, or validity Allen, 2012 Data models, quality, or validity Desai, 2012 Data models, quality, or validity Delate, 2012 Data models, quality, or validity

FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee Selby,1997 Data models, quality, or validity Durham,1998 Network / data governance and access DeStefano, 2001 Data models, quality, or validity Platt, 2001 Network / data governance and access Vogt, 2004 Data models, quality, or validity Platt, 2003 Network / data governance and access Hornbrook, 2005 Data models, quality, or validity Wagner, 2005 Network / data governance and access Wallace, 2007 Data models, quality, or validity Williams, 2008 Network / data governance and access Geiger, 2008 Data models, quality, or validity Go, 2008 Network / data governance and access Magid, 2008 Data models, quality, or validity Toh, 2011 Network / data governance and access Andrade, 2011 Data models, quality, or validity Lieu, 2011 Network / data governance and access Sittig, 2012 Data models, quality, or validity Andrade, 2012 Data models, quality, or validity Lazarus, 2006 Human subjects privacy and safety Greene, 2010 Human subjects privacy and safety Greene, 2011 Knowledge management Marsolo, 2012 Human subjects privacy and safety

FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee FFiirrsstt aauutthhoorr,, yyeeaarr PPrriimmaarryy ppuurrppoossee Ford, 2002 Data models, quality, or validity Greene, 2005 Network / data governance and access Kahn, 2012 Data models, quality, or validity Maro, 2009 Network / data governance and access

The HMORN has made major contributions to the literature on the development of common data models and strategies for data governance. In some areas the HMORN has conducted substantial but unpublished work (e.g., contracting and IRB efficiencies, costs to build and develop a site VDW, data quality improvement processes). In other areas little has been done (e.g., management of knowledge about successful strategies for conducting research, documentation of successful research translation locally or across sites). No multi-institutional networks have systematically described their culture or assessed the complex effects of culture on scientific productivity or efficiency.