Residential Connected Thermostats at the RTF

Josh RushtonPNW Smart Thermostat Regional Workshop

January 11, 2018

Presentation Overview

• RTF measure background• Current RTF measure (Planning UES)• Candidate Research Approach

2

RTF Measure Background

3

Defining a new RTF measure

• Clear measure definition is essential• Definition may be wide or narrow, as in…

– Install a low-flow showerhead, vs.– Install a low-flow showerhead to replace an existing showerhead that

has a measured flow-rate ≥ 2gpm

• Different definitions lead to different savings estimates and research questions

• Final definition depends on program interest, RTF resources– No need to pursue every technically sound path to savings

4 – Background

Defining this new RTF measure• New RTF UES (Residential Connected T-Stats) approved Nov., 2016

– Next RTF engagement by November, 2019 (sunset date)– Current measure reflects RTF thinking at time of approval

• Recognized three potential savings mechanisms– Reduced heating/cooling due to more scheduled setback – Reduced heating/cooling due to setback from occupancy detection– Improved heat pump efficiency from advanced controls

• Diverse vendor offerings– Many products/services claim savings from the listed mechanisms– Secret savings algorithms vary across products and over time– Presents a challenge for measure definition

• Known data sources (see Additional Slides for 2016 snapshot) – Evidence of real savings– Wide uncertainty in empirical estimates– Significant variation across products/services

5 – Background

RTF starting

point

Current RTF Measure

6

RTF Measure Approach• How to manage savings uncertainty in a diverse and dynamic

market space? – Don’t want to be an obstacle to innovation – Do need a clearly defined measure and way to manage uncertainty

going forward

• RTF approach is a Planning UES Measure• Measure highlights

– Eligibility defined in terms of product features• EPA qualified products list may help next measure update• Current measure draws "wide" eligibility boundaries

– Recognizes large uncertainty– Research Strategy (required for all RTF Planning measures)

• Sketches candidate approach to improving savings reliability• Wide range of products is a research challenge • Candidate method feasibility not clear for this measure

7 – Current Measure (Planning UES)

Measure EligibilityConnected Thermostats • 7 day programmable or learning-based scheduling• Wi-Fi enabled, with remote access• Occupancy detection (e.g., motion sensor, geo-locating) and automatic

HVAC reduction when home is unoccupied

Heat Pump Optimization Layer• Minimize auxiliary electric resistance heat using lockout informed by

outdoor temperature sensor or local weather via internet

Services• Applies to a thermostat that does not qualify for the Connected

Connected Thermostat measure• 7 day programmable or learning-based scheduling• Wi-Fi enabled, with remote access

8 – Current Measure (Planning UES)

RTF chose not to create a Services measure at 2016 meeting due to lack of data.

Given interest (and data), this could still be pursued with a new measure proposal.

Other measure details• Baseline is Pre-conditions. Assumes primary motivation for

purchasing eligible device is a thermostat “upgrade”

• Measure Life is 5 years.– Less than what one major vendor has found for their products – Chose shorter life because RTF measure not brand-specific and RTF

expects some eligible products to be removed or disabled early due to• Technical problems• Customer satisfaction• Availability of newer/shinier products

• Measure Identifiers (3). Separate UES for each combination– Heating system type (Electric-resistance furnace or Heat pump)– Geographic heating zone (HZ1 or HZ2 or HZ3)– Delivery mechanism (Retail or Direct Install)

• Only affects cost assumptions

9 – Current Measure (Planning UES)

Savings Assumptions

Heating Savings = [Typical Heating Consumption] x [%Savings_Heating]Cooling Savings = [Typical Cooling Consumption] x [%Savings_Cooling]

10 – Current Measure (Planning UES)

Modeled separately for each HVAC System Type and heating zone (climate)

Based on T-Stat studies (and assumptions)

UES values built up from RBSA/SEEM models and assumed savings rates for each savings mechanism• Run-time reduction from increased scheduled setback• Run-time reduction from occupancy-informed setback• Advanced heat pump controls

Estimated Savings Rates11 – Current Measure (Planning UES)

• Heating, electric-resistance furnace – Percent savings: 6% (2% to 10%)*

• Heating, Heat Pump – Percent savings: 14% (8% to 20%)*

• Cooling, any system type– Percent savings: 6% (2% to 10%)*

* Nominal 90% confidence intervals; see note on next slide** See additional slides

ETO study** of Nest devices in homes with natural gas heat

ETO study** of Nest devices in homes with ducted HPs

Assumed same as heating run time reduction

Estimated Savings (UES values)12 – Current Measure (Planning UES)

All applications cost-effective under RTF assumptions.

Nominal 90% confidence bounds…• Reflect statistical

uncertainty in ETO study estimates

• Not a full picture of UES uncertainty0

250

500

750

1000

1250

1500

HZ1 HZ2 HZ3 HZ1 HZ2 HZ3

Electric FAF Heat Pump

kWh

Savings and 90% Confidence Bounds

Heating Savings Cooling Savings

Candidate Research Approach

13

Research considerations

• Not feasible to study continuous stream of new products

• ENERGY STAR® certification protocol defines standardized performance metrics / reporting data – Run-time reduction metric (relative to assumed

baseline) – Resistance heat utilization data

• Candidate approach hopes to leverage data created by certification protocol to obtain flexible and cost-effective approach for estimating savings

14 – Candidate Research

Candidate Approach• Use billing analysis to estimate ∆kWh for each home• Calculate performance metrics from thermostat data

– Some version of run-time reduction metric – A metric based on resistance heat utilization data– Granularity may be limited by vendor data (e.g.,

composite values for separate groups of homes)• Fit a regression model to estimate relationship

between ∆kWh and performance metrics• Apply regression coefficients to new thermostat data

to estimate savings for new products and existing products at later times

15 – Candidate Research

Research limitations and concerns

• Research and future savings estimates rely on thermostat data collected and provided by vendors– Possible conflict of interest – Vendor willingness is unknown

• Approach relies on relationship between performance metrics and kWh consumption / savings– Assumes researchers can reliably estimate the

relationship (and that the relationship exists) – Will need to judge validity of applying estimated

relationship to products not included in research samples

16 – Candidate Research

Additional slides: ENERGY STAR Certification Data

17

E-STAR Certification

• References for Connected Thermostat (CT) demonstration method– Description of algorithms in Appendix B of

Version 1.0 (rev. Dec, 2016)– Latest code documentation and sample output

avalaible here– Ongoing developments and discussions are

tracked on the E-STAR CT Specification Page• Hope to use data generated by certification

tools to improve RTF CT savings estimates

18

Method Details (1)• Certification is based on calculations for a sample of in situ CTs.

Vendors feed site data to E-STAR tool:– List of sites with metadata (zip-code, serial number, etc.) – Hour-level data for each site (measured indoor air temperatures,

thermostat et-points, HVAC run-times, etc.)

• “Thermostat Module” performs the following analysis for each site: – Fit separate “thermal demand” models for “core cooling” and “core

heating” days (models fit run-time to a function of indoor-outdoor ΔT)– Estimate baseline run-time for each season using the fitted model and

some assumed baseline indoor air temperature– Estimate percent run-time reduction as the difference between season-

total baseline run-time and season-total actual run-time, divided by season-total baseline run-time

– Metric is very sensitive to assumptions about baseline indoor air temperature: Estimate of percent run-time reduction is basically the same thing as percent reduction in the portion of indoor-outdoor ΔT that exceeds a site-specific threshold, τ.

19

Method Details (2)• “Statistics Module” calculates statistics for testing compliance criteria

– Calculate 95% confidence interval (CI) for mean percent run-time reduction during core heating (weighted approach accounts for climate regions; assuming Gaussian distribution). Confirm that CI lower bound ≥8%.

– 20th percentile of heating run-time reductions is at least 4%. I.e., calculated run-time reduction is at least 4% for at least 80% of units (after possible weighting for climate regions)

– Similar requirements for cooling

• Trouble makers– Heat pumps: Thermostat module calculates “resistance utilization”

(total time resistance heat is on, divided by total time unit is in heating mode) for 5-degree temperature bins (no test metric at this time – just a reporting requirement)

– Variable Capacity: Method doesn’t include any special provisions for variable capacity systems at this time (run-time metric doesn’t make sense for these)

20

T-Stat Module Calculations (1)Calculation details for a site’s heating run-time reduction• Filter for “core heating days” with sufficiently data, 𝑑𝑑 = 1,2, … ,𝐷𝐷• Temperature variable ∆𝑇𝑇𝑑𝑑,ℎ is simple hour-level difference: indoor

temperature (measured by the thermostat), minus outdoor air temperature (from NOAA)

• Run-time variable 𝑅𝑅𝑇𝑇𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎, 𝑑𝑑 is total heating run-time for each day (record by thermostat)

• Thermal demand: 𝑇𝑇𝐷𝐷𝑑𝑑 = 124∑ℎ=124 ∆𝑇𝑇𝑑𝑑,ℎ − τ + (τ is explained below)

• Run-time responsiveness to thermal demand: 𝛼𝛼 =⁄∑𝑑𝑑=1𝐷𝐷 𝑅𝑅𝑇𝑇𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎, 𝑑𝑑 ∑𝑑𝑑=1𝐷𝐷 𝑇𝑇𝐷𝐷𝑑𝑑

• Modeled run-time: 𝑅𝑅𝑇𝑇𝑚𝑚𝑚𝑚𝑑𝑑𝑚𝑚𝑎𝑎, 𝑑𝑑 = 𝛼𝛼 × 𝑇𝑇𝐷𝐷𝑑𝑑• Fit procedure (finding 𝛼𝛼 and τ):

– Select initial value τ=0– Calculate 𝑇𝑇𝐷𝐷𝑑𝑑 and 𝛼𝛼 based on this τ– Calculate sum of squared errors SSE = ∑𝑑𝑑=1𝐷𝐷 𝑅𝑅𝑇𝑇𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎, 𝑑𝑑 − 𝑅𝑅𝑇𝑇𝑚𝑚𝑚𝑚𝑑𝑑𝑚𝑚𝑎𝑎, 𝑑𝑑

2

– Use iterative process (a version of gradient descent) to find 𝛼𝛼 and τ that minimize SSE

21

T-Stat Module Calculations (2)• Baseline

– Use baseline indoor air temperature assumption (instead of T-stat data) to calculate ∆𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑,ℎ

– Use fitted τ to calculate baseline thermal demand for each day: 𝑇𝑇𝐷𝐷𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑 = 1

24∑ℎ=1..24 ∆𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑,ℎ − τ +

– Use fitted 𝛼𝛼 to estimate baseline runtime 𝑅𝑅𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚, 𝑑𝑑 =𝛼𝛼 × 𝑇𝑇𝐷𝐷𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑

• Estimate % run-time reduction: ∑𝑅𝑅𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚, 𝑑𝑑 − ∑𝑅𝑅𝑇𝑇𝑚𝑚𝑚𝑚𝑑𝑑𝑚𝑚𝑎𝑎, 𝑑𝑑

∑𝑅𝑅𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚, 𝑑𝑑=∑ ∆𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑,ℎ − τ + − ∑ ∆𝑇𝑇𝑑𝑑,ℎ − τ +

∑ ∆𝑇𝑇𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑,ℎ − τ +

22

Equivalent expression is all about ∆T. This shows how run-time reduction estimate is closely tied baseline temperature assumptions.

Site-level outputs provided by certification tool

• Heating equipment type, weather station ID, CT identifier• Output variables related to thermal demand:

– Fitted threshold τ and responsiveness coefficient 𝛼𝛼– Number of core heating days 𝐷𝐷 included in analysis – Mean thermal demand, actual and baseline: 1

𝐷𝐷∑𝑑𝑑=1𝐷𝐷 𝑇𝑇𝐷𝐷𝑑𝑑 and 1

𝐷𝐷∑𝑑𝑑=1𝐷𝐷 𝑇𝑇𝐷𝐷𝑏𝑏𝑎𝑎𝑏𝑏𝑚𝑚,𝑑𝑑

– Multiplying thermal demand variables by 𝛼𝛼 yields actual and baseline run-time– Constant value assumed for indoor air temperature in the baseline (typically 90th

percentile of actual indoor temperature over core heating days)• No other summary values provided for actual indoor temperature

– Other variables that are algebraic combinations of the above (e.g., total run-time reduction)

• Fit statistics such as RMSE• Ranking information such as what percentile this site is in terms of

estimated run-time reduction• Filter-related data such as number of days with both heating and cooling or

with insufficient data• Resistance heat utilization data (for heat pumps)

– Total run-time and auxiliary (resistance) run-time over all core heating days– Resistance utilization rates by outdoor temperature bin (0-5⁰F, 5-10⁰F, …, 55-60⁰F)

23

Tool outputs and RTF savings estimates

Research Strategy: Use run-time and heat pump resistance variables as explanatory terms in a pre/post billing analysis• May only be able to get group averages for

different variables (vendor prerogative) • Cover a few products and heating zones to

establish relationship between run-time reduction metric and kWh savings

• Need to construct metric for resistance

24

Additional slides: Data Sources

25

Regional Data Sources Available in November, 2016

26 – Connected Thermostats

Sponsor ThermostatHeating System Post-install period

Sample sizes: Participant

/ComparisonHeating Savings

Savings % of heating

load

Savings % of cooling

load Uncertainty NotesRegional Connected Thermostat studies

ETO Nest ASHPpre: 6/12 - 7/13post: 1/14 - 2/15

106 / 199 645 kWh 14%90% CI: 376 to 914 kWh

8% to 20%

all participants: 176 / 40

600 kWh 12%

well behaved bills: 97/40

850 kWh 15%

ETO NestGas central heat

pre: 9/13 to 11/14post: 1/15 to 10/15

200 / 1000 34 therms 6%90% CI: 13 to 55 therms

2% to 10%

ETO LyricGas central heat

pre: 9/13 to 11/14post: 1/15 to 10/15

200 / 1000 -29 therms -5%90% CI: -55 to -3 therms

-9% to -1%

PSEHoneywell Vision Pro 8000

Gas heatpre: 8/12 to 7/13post: 12/13 to 11/14

1000 / 1000 17 therms 2.5%*90% CI: 5 to 29 therms

0.7%* to 4%*

29% of Treatment group couldn't get a thermostat installed. Savings expressed per installed thermostat .The most scientifically rigorous analysis of the group.*% savings based on territory-wide estimate of gas heat load from RBSA.

National Grid (Massachusetts)

Ecobee Gas heatpre: 1/09 - ?post: 3/11 - 4/12

86 / 0 82 therms ?90% CI**: 63 to 100

therms

Not from region**No Comparison group - results in smaller confidence interval

BPA Nest ASHPpre: 2/13 - 1/14post: 9/14 - 8/15

Large (due to small Comparison group)