baseline analysis cbp, amp, and dbp steve braithwait, dan hansen, and dave armstrong christensen...
TRANSCRIPT
Baseline AnalysisCBP, AMP, and DBP
Steve Braithwait, Dan Hansen, and Dave Armstrong
Christensen Associates Energy Consulting
DRMEC Spring Workshop
May 7, 2014
May 2014 1
May 2014 2
Presentation Outline
Objectives Methodology
Data Performance measures
Aggregator program (CBP and AMP) results Demand Bidding Program (DBP) results
May 2014 3
Objective: Assess Performance of Alternative Baseline Types
For each Utility and Notice type: All customers, with BL adjustment as chosen All customers, simulated with universal selection
of the BL adjustment Sum of individual BL vs. portfolio BL (constructed
from aggregated customer loads), for AMP and CBP only
Examine unadjusted and day-of adjustments with 20%, 30%, 40%, 50% caps, and uncapped
May 2014 4
Analysis Details
For actual program event days The “true” baseline is the estimated reference
load from the ex post evaluation For event-like non-event days
The “true” baseline is the observed load
May 2014 5
Performance Measures (1)Percentage Baseline Error
Percentage BL error for each customer/portfolio-event day is: Percentage error = (LP
d – LAd) / LA
d
LAd = actual, or “true” baseline load on day d
LPd = “predicted” baseline to be evaluated
Positive value = over-estimated baseline (implies over-stated program load impact)
Negative value = under-estimated baseline (implies under-stated program load impact)
May 2014 6
Performance Measures (2) Accuracy
Accuracy is measured as the median absolute percentage error (MAPE) Calculate the absolute value of the percentage error for
each customer/event-day Calculate the median of values across customer/event-
days (mean can be misleading due to extreme values) Higher values correspond to larger baseline errors
May 2014 7
Performance Measures (3)Bias
Bias is measured by the median percentage error, without taking the absolute value
Positive values indicate upward bias (i.e., the program baseline tends to over-state the “true” baseline)
Negative values indicate downward bias (i.e., the program baseline tends to under-state the “true” baseline)
Nominated Customers by Choice of BL Adjustment – CBP and AMP
May 2014 8
May 2014 9
Accuracy (Median Abs. % Error)PG&E CBP-DO
May 2014 10
Bias (Median % Error)PG&E CBP-DO
May 2014 11
Percentiles of % Errors – PG&E CBP-DOActual Events, by Adjustment Cap
May 2014 12
Percentiles of % Errors – PG&E CBP-DOSimulated Events, by Adjustment Cap
Summary: Accuracy & Bias (Aggregated Indiv.; Universal Adj.; 40% cap)
May 2014 13
Summary: Percentiles of % Errors(Aggregated Indiv.; Universal Adj.; 40% cap)
May 2014 14
May 2014 15
Summary of Findings Accuracy and bias measures vary by utility, program
and notice type Suggests that factors other than baseline type and
adjustment caps may be most important, such as types of customers (e.g., highly variable load) and event-day characteristics (e.g., event on isolated hot day)
Day-of adjustment often improves accuracy and reduces bias, but level of cap is less important Largest errors typically occur for Unadjusted BL and
Unlimited cap BL with small median error (e.g., 1%) can have >10%
errors in 20 percent of cases
16May 2014
DBP Results:PG&E Distribution of % Errors
-100.0%
-90.0%
-80.0%
-70.0%
-60.0%
-50.0%
-40.0%
-30.0%
-20.0%
-10.0%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Unadj. 20% 30% 40% 50% No Cap
% B
as
eli
ne
Err
or
Baseline Method
5th %ile
10th %ile
25th %ile
Median
75th %ile
90th %ile
95th %ile
17May 2014
DBP Results:SCE Distribution of % Errors
-40.0%
-30.0%
-20.0%
-10.0%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Unadj. 20% 30% 40% 50% No Cap
% B
ase
line
Err
or
Baseline Method
5th %ile
10th %ile
25th %ile
Median
75th %ile
90th %ile
95th %ile
Summary
Day-of adjustments tend to improve baseline accuracy and reduce bias
The analysis provides support for making the day-of adjustment the default option
The effectiveness of the day-of adjustment is not very sensitive to the level of the cap
May 2014 18
May 2014 19
Questions?
Contact – Steve Braithwait or Dan Hansen, Christensen Associates Energy ConsultingMadison, Wisconsin [email protected] [email protected] 608-231-2266
Appendix
SCE – CBP DO SDG&E – CBP DO PG&E – AMP DO SCE – AMP DO
May 2014 20
May 2014 21
Accuracy (Median Abs. % Error)SCE CBP-DO
May 2014 22
Bias (Median % Error)SCE CBP-DO
May 2014 23
Percentiles of % Errors – SCE CBP-DOActual Events, by Adjustment Cap
May 2014 24
Percentiles of % Errors – SCE CBP-DOSimulated Events, by Adjustment Cap
May 2014 25
Accuracy (Median Abs. % Error)SDG&E CBP-DO
Accuracy – Med. Abs. Err. (MW)SDG&E CBP DO
May 2014 26
May 2014 27
Bias (Median % Error)SDG&E CBP-DO
May 2014 28
Accuracy (Median Abs. % Error)PG&E AMP-DO
May 2014 29
Bias (Median % Error)PG&E AMP-DO
May 2014 30
Accuracy (Median Abs. % Error)SCE AMP-DO
May 2014 31
Bias (Median % Error)SCE AMP-DO