Why Data MattersBuilding and Sustaining a

Business Case

NEAUC Conference

June 18, 2014

Presentation

• Evaluation versus Performance Measurement• Types of Performance Measures

– Inputs– Outputs– Outcomes– Impacts

• Data Sources• Process

2

EVALUATION VERSUS PERFORMANCE MEASUREMENT

3

Comparison

4

Evaluation

• What are the goals?

• How is my program performing compared to goals or expectations?

• How does it compare to other programs?

• How can the program improve?

5

Performance Measurement• How can I measure?

– My organization’s efforts and inputs

– Outcomes of those efforts

– How we impacted clients

– How we impacted utility

• How has this changed over time?

• How does my organization compare?

• What are higher performers doing?

• Are those actions related to results?

• Can I implement those actions?

6

TYPES OF PERFORMANCE MEASURES

7

Types of Measures

8

Inputs

• Staff hours

• Equipment

• Travel costs

• Supplies

9

Input ExampleDelivery Costs

10

Agency Job Cost

Year# of Jobs

Mean Cost

Percentiles

P10 P25 P50 P75 P90

2010 530 $2,744 $764 $1,297 $2,423 $4,105 $5,155

2011 762 $3,436 $605 $2,004 $3,435 $4,958 $5,871

2012 369 $4,324 $904 $2,622 $4,310 $6,039 $7,068

Total 1,661 $3,413 $763 $1,679 $3,314 $4,912 $6,070

Input ExampleItemized Costs

11

UGI Electric2010 2011 2010 2011

Field Support $521,520 $764,932 $119,334 $252,962Administration $114,529 $167,622 $37,482 $35,671Inspections $19,840 $12,830 $7,385 $11,390No Measures $4,768 $4,075 $1,950 $1,600Rehab Pilot $94,774 $68,742 $0 $0Conservation Pilot $0 $50,000 $0 $0TOTAL $755,161 $1,068,201 $166,151 $301,623

Outputs

• Number of customers applied

• Number of customers enrolled

• Service delivered

• Participant characteristics

• Services coordinated with other programs

12

Output ExampleCustomers Served

13

Number

Referrals 51,868

Defaulted 10,166

Cancelled 17,006

Graduates 1,011

Moved 8,480

Re-certifications 8,512

New Enrollments 19,401

Output ExampleCustomer Characteristics

14

Federal Poverty Level

2011 2012 2013

Number Percent Number Percent Number Percent

0-50% 892 25% 860 28% 739 26%51-100% 1,252 36% 1,079 35% 906 32%101-150% 832 24% 725 23% 633 23%Unknown 533 15% 450 14% 517 19%Total 3,509 100% 3,114 100% 2,795 100%

Output ExampleCustomer Characteristics

15

2011 2012 2013Number Percent Number Percent Number Percent

Child 2,133 62% 1,923 62% 1,749 64%Elderly 369 11% 317 10% 325 12%

Output ExampleService Type

16

Enrollments RecertificationsPhone 731 257Mail 14,748 4,663Office 1,141 299Home 10 --Total 16,630 5,219

Output ExamplePayment Type

17

Payment Type EnrollmentsMinimum Payment 11%Percent of Bill Payment 54%Percent of Income Payment 16%Annualized Average Payment 7%Agency Selected Payment 12%

Output ExampleGrant Type

18

Number of Grants

Percent Payments Percent

Electric 3,350 87% $783,576 85%Oil 380 10% $106,469 11%Natural Gas 72 2% $19,418 2%Propane 43 1% $9,911 1%Kerosene 21 1% $5,982 1%Coal 2 <1% $738 0%Total 3,868 100% $926,094 100%

Output ExampleMeasures Installed

19

Measures2011 2012

# % # %

Audit 743 98% 352 95%

LIURP Intake 739 97% 352 95%

Repairs 675 89% 318 86%

Blower Door Test 587 77% 303 82%

Air Leakage Reduction 533 70% 276 75%

Door Work 526 69% 280 76%

Air Sealing 499 65% 282 76%

Efficiency Testing 446 59% 205 56%

HVAC Clean & Tune 395 52% 219 59%

Pipe Insulation 404 53% 213 58%

Energy Education 345 45% 262 71%

Window Repair/Replace 378 50% 185 50%

Measures2011 2012

# % # %

Water Measure 327 43% 197 53%

Attic Insulation 320 42% 204 55%

Lighting 248 33% 148 40%

Mileage 330 43% 217 59%

Dryer Work 231 30% 165 45%

Ventilation 160 21% 113 31%

Wall Insulation 159 21% 95 26%

Thermostat 100 13% 100 27%

Insulation 96 13% 43 12%

Header Insulation 140 18% 68 18%

Attic Preparation 83 11% 44 12%

Refrigerator 56 7% 20 5%

Outcomes

• Reduction in bill

• Reduction in energy burden

20

Outcome ExampleBurden Reduction

21

2009 National LIHEAP Survey

Pre-LIHEAP

Post-LIHEAP

≤ 5% 13% 37%

6% - 10% 32% 29%

11% - 15% 19% 17%

16% - 20% 15% 8%

21% - 25% 7% 4%

>25% 14% 7%

Outcome ExampleBurden Reduction

22

2009 National LIHEAP Survey

All Senior DisabledChild

Under 18Non-

Vulnerable

Pre-LIHEAP 16% 14% 17% 16% 16%

Post-LIHEAP 11% 9% 11% 12% 10%

Outcome ExampleHeat Restoration

23

2009 National LIHEAP Survey

Restored Heat Due to Electric/

Gas Shut Off

Restored Heat Due to Fuel

Running Out

Restored Heat Due to Broken

Equipment

61% 69% 58%

Impacts

• Bill payment coverage rates increased

• Service terminations declined

• Energy usage declined

24

Impact ExampleCustomer Survey

25

Has the winter temperature improved, worsened, or stayed the same since receiving weatherization services?

Change in Winter Temperature Number of Respondents

Improved 19Worsened 1Stayed the Same 0Did Not Notice a Difference 8Total 28

Impact ExampleArrearage Impacts

26

Pilot Payment Program

Treatment GroupNet

Change# Pre PostGross

Change

ALL GRAD 561 $276 $251 -$26** -$119**

1 – Graduated Discount 213 $257 $254 -$3 -$96**

2 – Discount &Audit 170 $332 $289 -$43* -$136**

3 – Discount & Counseling 178 $246 $210 -$36** -$129**

Impact ExampleService Terminations

27

Service Terminations

Utility 1 Utility 2 Utility 3 Utility 4

Pre 5% 14% 15% 4%

Post 7% 8% 4% 2%

Gross Change 1% -6%** -10%** -3%**

Net Change -1% -6%** -10%** 0%

Impact ExampleUsage Impacts

28

Obs.Usage Savings Net Savings

Pre PostkWh/

ccf%

kWh/ ccf

%

Electric Baseload

62 14,108 13,475 633 4.5% 915 6.5%

Electric Heating 43 27,808 26,343 1,465* 5.3% 1,295 4.7%

Gas Heating 471 1,588 1,411 177** 11.2% 156** 9.8%

Impact ExampleMajor Measures

29

# Major Measures

Obs.Usage (ccf) Savings

Pre Post ccf %

0 58 1,575 1,408 167** 10.6%

1 198 1,555 1,475 80** 5.1%

2 254 1,543 1,380 162** 10.5%

3 167 1,594 1,368 226** 14.2%

4 or more 53 1,610 1,340 271** 16.8%

All 730 1,565 1,402 163** 10.4%

Impact ExampleCost-Effectiveness

30

ccf Saved

Mean Cost

Payback Years

$/ccf Saved Lifetime

Air Sealing/ Leakage Reduction

7.68 $509 13 $1.25

Attic Insulation 4.68 $915 21 $2.06Wall Insulation 8.27 $1,520 12 $1.16Misc. Insulation 6.81 $1,190 15 $1.41Window Work 51 $834 16 $1.58Thermostat 57 $142 2 $0.24

DATA SOURCES

31

Agency Records

• Most accessible

• Should be put in a database

• May not be needed if good program database

• Data– Customers served– Characteristics – income, poverty level, elderly,

children– Services provided

32

Public Use Data• Available for free download• Characterize eligible population in service territory• Programming skills needed• Data

– Number eligible– Geography– Characteristics – income, poverty level, elderly, children,

language– Energy costs

33

Customer Survey

• Real time feedback

• Requires staff time

• Document methodology

• Data– Customer characteristics– Satisfaction– Self-reported impacts

34

Program Database

• Program manager – state or utility

• Canned reports

• Queries

• Data– Customers served– Characteristics – income, poverty level, elderly,

children– Services provided

35

Utility Data

• Difficult to obtain

• Easier for utility managed program

• Requires software and programming skills

• Data– Customer type – heating, water heating, baseload– Energy usage– Energy bills– Customer payments– Energy assistance

36

PROCESS

37

Process

• Start with available data

• Identify performance measures

• Determine additional data sources

• Collect additional data

• Develop additional performance measures

38

SUMMARY

39

Summary

• Performance measurement overlaps with evaluation

• Start with program goals

• Work with available data

• Identify ways to enhance data

• Measure performance over time

• Identify areas for improvement

• Impact measures require more data and analysis

40

Contact

Jackie Berger, Ph.D.

President and Co-Founder

APPRISE

32 Nassau Street, Suite 200

Princeton, NJ 08542

609-252-8009

jackie-berger@appriseinc.org

www.appriseinc.org

•  

41