Presenter: Pamela Peele, Ph.D.Vice President, Health Economics

February 28, 2012

Learning About Your Clinical Programs

• One of the nation’s largest Integrated Delivery Systems• 5th in NIH funding, affiliated University of Pittsburgh• $9.0 billion in total operating revenue• More than 54,000 employees• More than 3,000 employed physicians and 5,000 affiliated

physicians• 21 hospitals: 4,500+ beds and 43 regional cancer centers• 400 clinical locations; home care; rehab, urgent care• 1.8 million members in Insurance Division programs • 20,000+ contracted network providers• Global and Commercial Enterprise (UK; Italy; Qatar; Ireland;

Cyprus)• $500 million commitment to information technology

UPMC Today

2

• 2nd Largest in Nation Provider Led• 3rd Largest Operating in PA• 1.8M Members• Annual Revenues ~$4B• Fastest Growing Medicaid Plan • Fastest Growing Children’s Health• Highest Commercial Satisfaction

J.D. Power• Top 10 Nationally in Medicaid Quality• 4 Star Medicare Plan• Highest Ranked Provider

Satisfaction (PA)• National Business Group on Health

Platinum Winner past three years

UPMC Insurance Services Division Highlights

3

• 1.8 Million covered lives across all insurance products

• Medicare Advantage• Medicare FFS• Commercial (fully insured and ASO)• Medicaid• Special Needs Plan (SNP)• Children’s Health Insurance Plan (SCHIP)• Community Care (behavioral health management plan)

UPMC Health Plan

4

Levels of Analytics Framework

5

Standard ReportsWhat happened?

AlertsWhat actions are needed?

Query DrilldownWhat exactly is the problem?

Ad hoc ReportsHow many, how often, where?

Statistical AnalysisWhy is this happening?

OptimizationWhat’s the best that can happen?

Predictive ModelingWhat will happen next?

ForecastingWhat if these trends continue?

Degree of Intelligence

Com

petit

ive A

dvan

tage

From Tom Farre, “The Analytical Competitor”, in Analytics: The Art and Science of Better, ComputerWorld Technology Briefing.

UPMC HP: 2009

UPMC HP: 2006

The TWO Pieces of the Puzzle

6

1. Program Description

2. Analytic Team

1. What are trying to do or change?

Concrete objectives of the program with outcomes linked to program actions

2. How will we do it?

What are the tasks to be done, who does them, to whom,and when?

3. How will we know if we are successful?

Outcomes, hypothesis testing, measurement

MAKE AN INTERNAL TEMPLATE DOCUMENT

Program Description

7

• Pre/Post evaluations – Confounded by unobservables and time-specific changes – Selection criteria often undermines validity

• Randomized Controlled Trial– Often impractical for regulatory, operational, or member/provider

satisfaction reasons

• Matched cohorts– Requires scientifically sound matching techniques– If continuous enrollment is required, death takes a holiday

Avoiding Regression to the Mean

8

• Intention to Treat– Requires careful thought on the inclusion/exclusion criteria

• Differences in Differences– Powerful method for the real world– Eliminates the concern over uniformly distributed unobservables

Avoiding Regression to the Mean

9

Pediatric Weight Management Program Objectives

10

• A clinic-based program that utilizes a multi-disciplinary team approach to reduce BMI in overweight or obese children

•Collaborative program with Children’s Community Pediatrics and Children’s Hospital of Pittsburgh

•Evaluation focused on children with 2 or more weight measures within 182 days (July 2010 to January 2012) and BMI percentile of 85% or greater

• Research question: Did children enrolled in the program have a greater change in body mass index (BMI) percentile than children not enrolled in the program?

Analysis Design

11

1.Primary analysis: Compared the difference in the change in BMI percentile between the intervention and comparison groups using a matched difference in differences comparison (kernel matching)

• Matching variables: age, gender, baseline BMI percentile, duration between first and last visit, and the number of visits

2.Robustness checks: Compare results of kernel matching to results using exact matching, nearest neighbor propensity score matching, and two OLS regression models

Primary Findings

12

Difference-in –differences: Δintervention-Δcomparison=-0.43-0.02=-0.45

Conclusions and limitations

13

•Primary conclusions• Children enrolled in the Pediatric Weight Management program

experienced significantly greater reductions in their BMI percentile compared to children not enrolled in the program

• These the program and designing a more rigorous study of the outcomepromising findings, if clinically meaningful, provide evidence to warrant continuing s going forward

•Limitations• Program attrition: 86 children (46%) did not have a return program

visit• Very little information is known about the children in the comparison

group

Levels of Analytics Framework

17

Standard ReportsWhat happened?

AlertsWhat actions are needed?

Query DrilldownWhat exactly is the problem?

Ad hoc ReportsHow many, how often, where?

Statistical AnalysisWhy is this happening?

OptimizationWhat’s the best that can happen?

Predictive ModelingWhat will happen next?

ForecastingWhat if these trends continue?

Degree of Intelligence

Com

petit

ive A

dvan

tage

From Tom Farre, “The Analytical Competitor”, in Analytics: The Art and Science of Better, ComputerWorld Technology Briefing.

• Excel

• Access

• Crystal Reports

Staff - 2006

18

Current Staff

19

• Industry Knowledge• Data visualization skills• Data ECTL (extraction, cleaning, transformation, loading) skills• Statistics• Health Services Research • Data Mining• Financial modeling & evaluation• Presentation, writing, and communication skills

• Formally trained but NOT blinded by their training– Challenge deeply held beliefs

Staff Skills and Backgrounds

20

• Database: SQL, Toad• Statistics: SAS, STATISTICA, STATA, R• Data Mining: STATISTICA, SAS Enterprise Miner, R• Modeling & Simulation: MATLAB, Mathematica, Vensim• GIS: ArcGIS

Tools

21

• Data Overload– No Knowledge

• No Learning

• Misleading Data Perspective

Issues

22