Eugene Washington PCORI Engagement Award – OHDSI Symposium Summary Report

Background The Observational Health Data Sciences and Informatics (OHDSI) collaborative is a multi-stakeholder, interdisciplinary collaborative that is striving to bring out the value of observational health data through large-scale analytics. Our research community enables active engagement across multiple disciplines (clinical medicine, informatics, biostatistics, computer science, epidemiology, life sciences) and spans multiple stakeholder groups (researchers, patients, providers, payers, product manufacturers, regulators). Our mission is to transform medical decision making by generating reliable scientific evidence about disease natural history, healthcare delivery and the effects of medical interventions through large-scale analysis of observational health databases for population-level estimation and patient level prediction. While the OHDSI collaborative is a growing community, many potential stakeholders remain unaware of how OHDSI’s standardized analytic tools could support their research and decision making, or how they could get involved and contribute to future research and development within the community. To help raise awareness of observational research and the OHDSI project, the first annual OHDSI symposium was held on October 20th 2015 in Washington DC. The OHDSI symposium had two primary objectives: increasing community engagement and improving our ability to receive feedback from a broad range of stakeholder groups; the significance of which is cyclical. As our engagement improves, our international community will continue to grow in number and diversity. Bringing together diverse skill sets and perspectives will allow us to optimize our technical infrastructure and streamline collaborative processes. With regular feedback from stakeholder groups, we can ensure future research is conducted to answer relevant health care questions. Through these efforts, our ability to facilitate open science will increase exponentially and greatly accelerate the generation of high quality, meaningful evidence.

Conference Summary The symposium was a full day event with morning and afternoon sessions (see Appendix A). The morning comprised of three presentations which gave an overview of the OHDSI program, a presentation of the first OHDSI network research study and the unveiling of OHDSI’s latest open-source software tools. The aim of these presentations was to demonstrate the potential of observational research and to inform participants about OHDSI’s history, mission and future goals. Each presentation included a Q&A session to engage the broader community and ask participants how their own research or policy questions could be answered through collaboration with OHDSI. The afternoon was more hands on. It included demonstrations of OHDSI software tools, a poster session and two panel discussions designed to facilitate audience feedback. The demonstrations aimed for maximum impact by providing tangible examples of how OHDSI conducts collaborative research. The poster session gave OHDSI collaborators a chance to showcase their individual work and highlight the diverse range of research topics covered across the community.

While all sessions were relevant to PCORI, the last panel, “The Value and Challenges of Evidence from Observational Data: A Multi-Stakeholder Perspective,” included Maryan Zirkle from PCORI. This panel took the broadest view of observational research and OHDSI and discussed the value proposition for a range of stakeholders.

Key Findings The major outcomes resulting from the OHDSI symposium were:

Increased community engagement - Since the symposium, we have seen an increase in the number of new collaborators, higher attendance at community meetings, more activity in the online OHDSI forums, the formation of several new working groups, and an increased demand for OHDSI network research

Improved ability to receive feedback from a broad range of stakeholder groups - The symposium offered a unique opportunity to collect feedback about the value OHDSI’s open source software and network research can offer the healthcare community. Thanks to questions raised during the symposium Q&A sessions and feedback from the evaluation questionnaire, the OHDSI community can ensure our work results in meaningful research reflecting the needs of patients

Stakeholder Engagement The symposium was an excellent opportunity to foster further collaboration between academia, industry, and government stakeholders. An example of this was the formation of the symposium planning committee. To aide with planning, an organizing committee was formed and included representatives from government, academia, and industry. Members of the committee gave valuable insights on how to increase participation from each sector and aided in developing the symposium agenda to ensure each presentation was relevant and valuable to the range of stakeholders present in the audience. With the committee’s help, we were able bring in speakers for our closing panel from the FDA, Takeda Pharmaceuticals, John Hopkins and PCORI. In addition, the symposium attracted almost 300 participants from academia, government, industry, and patient groups. Having this range of stakeholder perspectives allowed the symposium to facilitate cross-sector discussions and provide insights on how OHDSI can achieve its aim of generating reliable evidence to inform medical decision making. Evaluation results and findings Overall, the symposium was well received with participants giving each presentation positive feedback. Based on responses in the evaluation questionnaire, it is clear the broader healthcare community sees OHDSI’s open source tools as a powerful platform to facilitate clinical research. However, some participants felt the symposium was too focused on software development and would have liked to see more OHDSI research. In addition, while many attendees enjoyed the multi-stakeholder perspectives shared throughout the day, some would have liked more time for in-depth discussion on specific focus areas.

Finally, many participants enjoyed the OHDSI software demonstrations and mentioned they would like to learn more about how to use the suite of OHDSI tools. Some even suggested providing how-to tutorials during the next symposium. Lessons learned The 2015 OHDSI symposium provided a wonderful opportunity to showcase OHDSI’s achievements to-date and improve our engagement efforts both across the OHDSI collaborative and broader healthcare community. To build upon this year’s success, the next symposium should include:

More network research – Based on participant feedback, there is a clear demand for OHDSI network research. To fulfill this demand, the OHDSI community should streamline collaborative research processes and create a central infrastructure to facilitate result sharing.

Break-out sessions and tutorials – To allow for more in-depth discussions on specific focus areas, the next symposium should include more structured time for break-out sessions and tutorials on how-to use OHDSI tools. This could be accomplished by offering workshops the day before or after the symposium. Alternatively, the symposium agenda could be broken down to include workshop sessions.

The symposium informed not only the next symposium, but also the actual OHDSI research direction. More emphasis will be placed on carrying out studies, rather than so much focus software development. We believe that this is a natural progression, where early work has to build the infrastructure to get started, but the collaborative must quickly emphasize concrete results to provide value and remain focused on the ultimate goal.

Dissemination Plans

Following the symposium, a thank-you email and evaluation questionnaire was circulated to all participants. Links to the conference materials were circulated as well. This included presentation slides and poster abstracts which had been uploaded onto the OHDSI.org website. In addition, videos of each presentation were recorded during the symposium. In the upcoming months, these videos will be edited and made available on the OHDSI.org website. Once available, these presentations will be an excellent tool for further sharing OHDSI’s vision and highlighting our continued progress towards our goals. The OHDSI collaborative aims to hold a symposium each year. As the time comes to plan the 2016 OHDSI Symposium, links to the 2015 presentations will be circulated to potentially attendees, including patient groups, to encourage their participation both at the symposium, and the OHDSI community. One potential barrier to this dissemination plan is obtaining approval from all presenters to make their presentations publically available. Since each presenter will be asked to review and approve the content uploaded onto OHDSI.org, it is possible some will decline to share their work.

Appendix A- Conference Agenda

2015 OHDSI Symposium

Time Description

7:30 - 8:00am

Registration

8:00 - 8:30am

Introductions

8:30 - 10:00am

Welcome to the journey: Overview of OHDSI : past, present, future Speaker: Patrick Ryan, PhD, Sr. Director and Head, Epidemiology

Analytics, Janssen Research & Development

10:00 - 10:15am

Break

10:15 - 11:15am

OHDSI in action: Real-world evidence for clinical characterization Speaker: George Hripcsak, MD, Chair of the Department of Biomedical

Informatics at Columbia University Medical Center

11:15 - 12:15 am

OHDSI in action: Open-source analytics for patient-centered evidence Speaker: Jon Duke, MD, Senior Scientist, Regenstrief Institute

12:15 - 2:45 OHDSI collaborator showcase Poster session of OHDSI research Software demonstrations of OHDSI open-source tools

During this time, lunch will be provided

2:45 - 3:45pm

Panel Discussion – Experiences from the OHDSI international data network Moderator: Christian Reich, MD, PhD, Vice President of Real World

Evidence Systems, IMS Health, USA Rae Woong Park, MD, PhD, Professor, Ajou University School of

Medicine, South Korea Peter Rijnbeek, PhD Assistant Professor, Erasmus Medical Center,

Netherlands Parsa Mirhaji, MD, PhD, Director of Clinical Research Informatics at

Montefiore Healthcare System, Albert Einstein College of Medicine, USA

3:45 - 4:00pm

Break

4:00 - 5:30pm

Panel Discussion – The Value and Challenges of Evidence from Observational Data: A Multi-Stakeholder Perspective

Moderator: David Madigan, PhD, Executive Vice President and Dean of the Faculty of Arts and Sciences at Columbia University

Robert Ball, MD, MPH, ScM, Deputy Director – Office of Surveillance and Epidemiology, CDER, US Food and Drug Administration

Nareesa Mohammed-Rajput, MD, Medical Director of Clinical Informatics, Suburban Hospital part of Johns Hopkins Medicine

Maryan Zirkle MD, MS, MA, Program Officer – CER Methods and Infrastructure Program, PCORI

Lesley Wise, Vice President of PV Risk Management and Pharmacoepidemiology, Takeda Pharmaceuticals

5:30pm Closing remarks

Appendix B - List of Attendees

Full Name Affiliation

Hamed Abedtash Indiana University

Solomon Adjei Optum

Tanvir Ahmed National Institute of Health

Mohammad Al-Ansari Oracle

Demissie Alemayehu Pfizer

Crystal Allard US Food & Drug Administration

Katie Allen Regenstrief Institute

Steven Anderson US Food & Drug Administration

Andrew Williams Maine Medical Center

Alex Asiimwe Bayer

Charles Bailey Children's Hospital of Philadelphia

Robert Ball US Food & Drug Administration

Juan Banda Stanford University

Raj Bandaru Informatics Consulting Services

Elande Baro US Food & Drug Administration

Maura Beaton OHDSI

Lorne Becker The Cochrane Collaboraton

Rimma Belenkaya Montefiore Medical Center

Michael Benigno Pfizer

Tomas Bergvall WHO Uppsala Monitoring Centre

Jesse Berlin Johnson & Johnson

Jiang Bian University of Florida

Paul Biondich Regenstrief Institute

Clair Blacketer Johnson & Johnson

Joaquin Blaya Lumiata

Olivier Bodenreider National Institute of Health

Mary Regina Boland Columbia University

Richard D Boyce University of Pittsburgh

Brian Bradbury Amgen Inc

Roselie Bright US Food & Drug Administration

Gunnar Brobert Bayer

Ned Brody

Jason Brown Signet Accel

Martin Brown PCORI

Keith Burkhart US Food & Drug Administration

Evanette Burrows Children's Hospital of Philadelphia

Zahid Butt MD, FACG Medisolv

Dr. Karim Calis US Food & Drug Administration

Larry Callahan US Food & Drug Administration

Christian Cao US Food & Drug Administration

Soledad Cepeda Johnson & Johnson

Kinnera Chada US Food & Drug Administration

Shreya Chakrabarti Columbia University

Aloka Chakravarty US Food & Drug Administration

Song Chen Optum

Yingkai Cheng Bayer

Minnie Chou Amgen Inc

Jim Cimino University of Alabama at Birmingham

Douglas Clark Biogen

Milton Corn National Institute of Health

Trinka Coster US Army

Dennis Cotter Medical Technology and Practice Patterns Institute

Ryan Crawford

Victor Crentsil US Food & Drug Administration

Mary-Jo Curran NewYork–Presbyterian Hospital

Mark Danese Outcome Insights

Oanh Dang US Food & Drug Administration

Chintan Dave University of Florida

Kourtney J. Davis GlaxoSmithKline

Suranjan De US Food & Drug Administration

Frank DeFalco Johnson & Johnson

Mitesh Parekh AstraZeneca

Rachael DiSantostefano Johnson & Johnson

Fan Du University of Maryland

Jon Duke Regenstrief Institute

Ian Duling AstraZeneca

Ryan Duryea Johnson & Johnson

Scott DuVall University of Utah

Noemie Elhadad Columbia University

David Epstein Deloitte

James Eudicone AstraZeneca

Lee Evans LTS Computing LLC

Kaye Evans-Lutterodt

Kristin Feeney Deloitte

Joseph Finkelstein Columbia University

David Fram Commonwealth Informatics, Inc.

Henry Francis US Food & Drug Administration

Henry Francis US Food & Drug Administration

Lewis Frey Medical University of South Carolina

Leif Friberg Karolinska Institutet

Gregory Fusco Takeda Pharmaceuticals

Aaron Galaznik Pfizer

Dina Gifkins Johnson & Johnson

Sigfried Gold InfoViz Consulting

Danny Gonzalez

Michael Goodman AstraZeneca

Geoffrey Gordon Commonwealth Informatics, Inc.

Susan Gruber Readan Udall Foundation

Shahrukh Haider US Food & Drug Administration

Taylor Hamilton Johnson & Johnson

Marianne Hamilton Lopez National Academy of Sciences

Ben Hamlin National Committee for Quality Assurance

Tarek Hammad Merck & Co

Rave Harpaz Oracle

Abraham G. Hartzema University of Florida

Kevin Haynes HealthCore

Raymond Heatherly SHYFT Analytics

Jonathan Herz AstraZeneca

james hill US Department of Veteran Affairs

Ingeborg Holt Commonwealth Informatics, Inc.

Erin Holve AcademyHealth

Kenneth Hornbuckle Eli Lilly and Company

Hripcsak Columbia University

Lucy Hsu National Institute of Health

Wei Hua US Food & Drug Administration

Betsy Humphreys National Institute of Health

Li-Shiuan Hung Columbia University

Kwan Hur US Department of Veteran Affairs

Vojtech Huser National Institute of Health

Rima Izem US Food & Drug Administration

Hector S. Izurieta, MD, MPH US Food & Drug Administration

Guoqian Jiang Mayo Clinic

Yiding Jiang NYU Langone Medical Center

Jigar Desai Pfizer

Byun Jung Hyun Ajou University

Chris Kakkanatt Pfizer

Lisa Kammerman AstraZeneca

Tian Kang Columbia University

Eva Katz Johnson & Johnson

Hugh kawabata Bristol-Myers Squibb

Leslie Kenna US Food & Drug Administration

David M Kent MD Tufts University

Mark Khayter Ephir Inc

Sajan khosla Saama Technologies, Inc

John Kilbourne National Institute of Health

Jessica Kim US Food & Drug Administration

Christopher Knoll Johnson & Johnson

Julianna Kohler Deloitte

harish kondamadugu AstraZeneca

Ronald Krall

Fabricio Kury National Institute of Health

Steven Labkoff IMO Inc

Michelle LaCour US Army

Christophe Lambert University of New Mexico

Ling Lan US Food & Drug Administration

Cedric Lane US Food & Drug Administration

Lisa Lang National Institute of Health

Joan Leavey Weill Cornell Medical College

Joo-Yeon Lee US Food & Drug Administration

Amy Leval Johnson & Johnson

Mark Levenson US Food & Drug Administration

Rebecca Levin United BioSource Corporation

Matthew Levine Columbia University

Jennifer Lin Takeda Pharmaceuticals

Yu Lin US Food & Drug Administration

Robert LoCasale AstraZeneca

John Logie GlaxoSmithKline

Ajit Londhe Johnson & Johnson

Alyssa Long National Institute of Health

Yang Lu University of California, Los Angeles

Yun Lu US Food & Drug Administration

Robert Lubwama Merck & Co

Thomas Ly US Food & Drug Administration

Qianli Ma Humana

Larry Ma Johnson & Johnson

David Madigan Columbia University

Asha Mahesh Johnson & Johnson

Gleb Malikov Odysseys Inc

Joanna Mantzuranis AstraZeneca

Jim Manzi

Angela Mariotto National Institute of Health

David Martin US Food & Drug Administration

James Masterson US Army

Amy Matcho Johnson & Johnson

Michael Matheny Vanderbuilt University

Panagiotis Mavros Merck & Co

Doris McGinness IMO

Malcolm McRoberts NantHealth

Michael Klumpenaar Deloitte

Evan Minty University of Calgary

Parsa Mirhaji Albert Einstein College of Medicine

Nareesa Mohammed-Rajput Johns Hopkins

Montse Soriano Gabarro Bayer

Jeffrey Morgan Deloitte

Catherine Mueller

Jerald Mullersman, MD, PhD, MPH

Rich Murray United BioSource Corporation

Karthik Natarajan Columbia University

Eileen Navarro

Kenney Ng IBM

Michael Nguyen US Food & Drug Administration

Phung Anh (Alex) Nguyen Taipei Medical University

Toan Ong University of Colorado Denver

Rita Ouellet-Hellstrom US Food & Drug Administration

Casey Overby University of Maryland

Dr. Joseph Overhage Cerner Corporation

Antonio Paredes US Food & Drug Administration

Sara Dempster AstraZeneca

Rae Woong Park Ajou University

Lili Peng AstraZeneca

Mingkai Peng University of Calgary

Adler Perotte Columbia University

Paul Petraro Bayer

Simone Pinheiro US Food & Drug Administration

Abhishek Pratap Sage Bionetworks & University of Washington

Douglas Pratt US Food & Drug Administration

Nicole Pratt University of South Australia

Frank Pucino National Institute of Health

Nick Puntikov Odysseys Inc

Javan Quintela Signet Accel

Judy Racoosin US Food & Drug Administration

Christian Reich IMS Health

Stephanie Reisinger Evidera

Jenna Reps Johnson & Johnson

Peter Rijnbeek Erasmus MC

Mitra Rocca US Food & Drug Administration

Dr. Jane Ruby Indivior, Inc

Patrick Ryan Johnson & Johnson

Pat Ryan

Raj Sabharwal AcademyHealth

Priya Sapra Trinity Pharma

Suchi Saria Johns Hopkins University

Stephen Schachterle Pfizer

Lisa Schilling University of Colorado Denver

Peter Schotland US Food & Drug Administration

Martijn Schuemie Johnson & Johnson

Anando Sen Columbia University

Anthony Sena Johnson & Johnson

Trevor Shaddox University of California, Los Angeles

Bob Sharrar United BioSource Corporation

Rongjun Shen Pfizer

Dahye Shin Ajou University

Azadeh Shoaibi US Food & Drug Administration

Nze Shoetan AstraZeneca

Scottie Siewers Maine Medical Center

Helen Sile US Food & Drug Administration

Lorraine Silsbee National Institute of Health

Jamie Skipper Office of the National Coordinator for Health Information Technology

Aaron Smith-McLallen Independence Blue Cross

David Sontag New York University

Alfred Sorbello US Food & Drug Administration

Ergin Soysal University of Texas

Judy Staffa US Food & Drug Administration

Paul Stang Johnson & Johnson

Richard Starr Georgia Institute of Technology

Ron Stewart Amgen Inc

Marc Suchard University of California, Los Angeles

Jimeng Sun Georgia Institute of Technology

Helena Sviglin US Food & Drug Administration

Casey Sydnor Commonwealth Informatics, Inc.

Ana Szarfman US Food & Drug Administration

Nicholas Tatonetti Columbia University

Rosenie Thelus US Army

Rosenie Thelus US Army

Zoltan Thinsz Karolinska Institutet

Anto Thomas NantHealth

Tony Thomas Johnson & Johnson

Yuxi Tian University of California, Los Angeles

Jawahar Tiwari US Food & Drug Administration

Joseph Tonning US Food & Drug Administration

Don Torok Ephir Inc

Lisa Torosyan US Food & Drug Administration

Matthew Tyson AstraZeneca

Brandon Ulrich B2i Healthcare

Levon Utidjian Children's Hospital of Philadelphia

Rohm Vashisht Stanford University

Melissa Vaughan

Veronica Sansing US Food & Drug Administration

Erica Voss Johnson & Johnson

Kin Wah Fung National Institute of Health

Magnus Wallberg WHO Uppsala Monitoring Centre

Xia Wang AstraZeneca

jian wang US Food & Drug Administration

Cunlin Wang US Food & Drug Administration

Bruce Weaver US Food & Drug Administration

Rachel Weinstein Johnson & Johnson

Emily Welebob-Yost AstraZeneca

Brian Wilson Tufts University

Lesley Wise Takeda Pharmaceuticals

Jason Woo US Food & Drug Administration

Yonghui Wu Google

Suji Xie US Army

Hua Xu University of Texas

Alexandre Yahi Columbia University

Qin Ye AstraZeneca

Soo Yeon Cho Ajou University

Zhong Yuan Johnson & Johnson

Lilly Yue US Food & Drug Administration

Richard Zhang US Food & Drug Administration

Changgeng zhao Novartis

Yueqin Zhao US Food & Drug Administration

Vivienne Zhu Medical University of South Carolina

Maryan Zirkle PCORI

Appendix C - Evaluation Questions

1. How valuable did you find each component of the OHDSI Symposium? a. Welcome to the journey: Overview of OHDSI : past, present, future (presentation by

Patrick Ryan) b. OHDSI in action: Real-world evidence for clinical characterization (presentation by

George Hripcsak) c. OHDSI in action: Open-source analytics for patient-centered evidence (presentation by

Jon Duke) d. OHDSI collaborator showcase (poster presentations and software demonstrations) e. Panel Discussion – Experiences from the OHDSI international data network (moderated

by Christian Reich) f. Panel Discussion – The Value and Challenges of Evidence from Observational Data: A

Multi-Stakeholder Perspective (moderated by David Madigan)

2. How did you find the breakdown and time distribution between presentations, posters, demos, and panels? What would you like to see differently at future symposiums?

3. At the the OHDSI Symposium, we discussed the type of work we're collaborating on in terms of two dimensions: analytical use case (data management, clinical characterization, population-level estimation, patient-level prediction) and lifecycle (methods research, open-source development, clinical applications). We highlighted work across all use cases throughout the lifecycle, with particular emphasis on the foundational community-building activities to allow all stakeholders to get started in collaborating with the OHDSI community. Moving forward, please help us prioritize where OHDSI should focus across these dimensions for future community events (and offer suggestions for any other areas we may be missing):

a. Observational data management i. Methodological research

ii. Open-source analytics development iii. Clinical applications

b. Clinical characterization i. Methodological research

ii. Open-source analytics development iii. Clinical applications

c. Population-level estimation i. Methodological research

ii. Open-source analytics development iii. Clinical applications

d. Patient-level prediction i. Methodological research

ii. Open-source analytics development iii. Clinical applications

e. Other, please specify.

4. Do you have any feedback on the logistics of the: a. Website content and registration b. Call for participation: posters/demonstrations

c. Conference location and venue

5. Do you have any other comments, questions, or concerns?

6. What stakeholder group(s) do you represent? (Government, Academia, Industry, Health System/provider, Patient, Other)

7. What disciplinary perspective(s) do you represent? (Epidemiology, statistics, informatics, computer science, clinical science, health policy, other)