a critical assessment of tcb18 electricity · identification of omitted cost drivers stochastic...

A critical assessment of TCB18 electricity

Dr Srini Parthasarathy, Partner Alan Horncastle, Partner Professor Emmanuel Thanassoulis, Associate Professor Subal Kumbhakar, Associate Charles Blake, Project Manager Hannes Seidel, Analyst

5 June 2020

Strictly confidential © Oxera, 2020.

1

Key messages 2 Strictly confidential

Significant issues in TCB18 Themes and headline issues

◄Prevalence of data errors ◄Defining the input variable ◄Adjusting for differences in input prices ◄Indirect cost allocation

Data collection and construction

◄Cost driver analysis ◄Sensitivity to the sample selected ◄Selecting candidate cost drivers ◄Aggregation of NormGrid ◄Adjusting for environmental factors

Model development

◄Returns-to-scale assumption ◄Outlier analysis ◄Validation of DEA outputs ◄Identification of omitted cost drivers ◄Stochastic frontier analysis (SFA) ◄Dynamic efficiency (frontier shift)

Application and validation

3

Sumicsid’s level of transparency falls short of good practice.

Sumicsid’s outputs do not contain the necessary information to clearly follow its analysis, or to validate its analysis and sources without considerable effort.

Strictly confidential

2

Transparency 4 Strictly confidential

Transparency


Model development


5 Strictly confidential



Transparency Modelling codes and sourcing

6

• the report was not sufficiently detailed to follow each step of the analysis without the considerable effort we have undertaken • our replication of TCB18 results was close enough to conclude on the quality of the benchmarking

• Sumicsid could have published modelling codes that would have ensured that third parties are able to follow each step in its analysis • publishing modelling codes would allow third parties to follow each step, identify errors or

assumptions that have been made in the modelling process, and provide constructive comments • this would not entail the publishing of (confidential) data

• Sumicsid did not adequately source all of the external information it had used in the analysis • for example, it did not present the environmental complexity weights used to adjust NormGrid in the

main report or associated appendices • one TSO was removed from the dataset without clear reasoning


Sumicsid’s outputs do not contain the necessary information for third parties to follow its analysis, or to validate its analysis and sources without considerable effort

Transparency Modelling codes and sourcing: example of good practice

7


Simple commands with descriptions of the code

Provide anonymised outputs and clearly

reference where the results are used in the report

Provide interpretation of the results

3




◄Prevalence of data errors ◄Defining the input variable ◄Adjusting for differences in input prices ◄Indirect cost allocation


Model development


9




Data collection and construction Data errors (I) (section 3.1)

10

• significant data errors and inconsistencies are contained in the final dataset • some TSOs flagged data inaccuracies to Sumicsid, which were not corrected for the final analysis • empirical analysis relies on the accuracy of the data being used, and real data tends to be noisy

• Sumicsid’s application of DEA is deterministic, meaning that it is particularly sensitive to data errors • data errors will therefore have a significant impact on TSOs’ estimated efficiency scores. For

example: • one TSO’s TOTEX is overstated by 32% • in another case, the value for weighted lines is overstated by 27%

• data errors will also change the shape of the efficient frontier, affecting other TSOs • an appropriate solution would have been rigorous data-cleaning and comprehensive sensitivity

analysis


Data collection and construction Data errors (II) (section 3.1)

11

• Monte Carlo simulation considered in international and national exercises indicates that: • most TSOs’ efficiency scores are highly sensitive to the addition of small errors, for example the

second TSO in the chart may be between 55% and 100% efficient • we cannot be certain (at the 90% significance level) that four additional TSOs are not operating

efficiently based on data errors alone • the errors simulated are less severe than actual errors in the TCB18 dataset • a regulator needs to be confident that the proposed savings are feasible, so uncertainty inherent in the

data and modelling needs to be taken into account


Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 3.1.

Data collection and construction Defining the input variable (section 3.2)

12

• treatment of the cost categories (OPEX and CAPEX) is unnecessarily restrictive • Sumicsid assumes that OPEX and CAPEX are equivalent and controllable

• OPEX and CAPEX are unlikely to be equivalent as they are subject to different normalisations • the significant variance in CAPEX shares across TSOs may be indicative of fundamentally

different operating models or reporting differences • several alternatives that Sumicsid has overlooked exist and are used in regulation

• developing separate OPEX and CAPEX (or even more disaggregate) models, recognising the trade-offs between them

• constructing TOTEX as a weighted sum of OPEX/CAPEX • modelling OPEX and CAPEX as separate inputs

• all of these have advantages • two input models can account for heterogeneity in expenditure mix • separate OPEX/CAPEX models can account for more specific drivers capturing relevant

heterogeneity (e.g. position in investment cycle can be accounted for) • when estimating the two-input model one TSO assessed to be efficient in Sumicsid’s model

becomes inefficient and two become efficient


Data collection and construction Accounting for differences in input prices (section 3.3)

13

• differences in input prices across TSOs are insufficiently accounted for • Sumicsid only adjusts 5.9% of the cost base for differences in input prices • not normalising the costs risks conflating uncontrollable price-level differences with inefficiency

• in international benchmarking exercises (e.g. in benchmarking of rail infrastructure managers), all expenditure may require appropriate normalisation for price differences • one option could be to adjust all OPEX and CAPEX. Results are sensitive to the method of

indexation (e.g. choice of PLI, proportion of expenditure adjusted) and require robust justification



Data collection and construction Indirect cost allocation (section 3.4)

14

• the allocation rule used in TCB18 is not justified or sense-checked • in some cases, the allocation of indirect expenditure is driven by large cost items that are unrelated

to where indirect costs are incurred • most TSOs’ estimated efficiency scores are insensitive to the allocation of indirect expenditure

• the allocation of expenditure to benchmarked activities is a conceptually important issue and may have a more material impact on a different sample



4

Model development 15 Strictly confidential

Model Development


◄Cost driver analysis ◄Sensitivity to the sample selected ◄Selecting candidate cost drivers ◄Aggregation of NormGrid ◄Adjusting for environmental factors

Model development





Model development Cost driver analysis (section 4.1)

17

• the cost driver analysis is not transparent. It is not explained: • which cost drivers were considered in the project—how were they selected? • how the final set was derived—how were some drivers discarded? • how this final model was selected—what were the alternatives?

• cost driver analysis is based on unsubstantiated assumptions—for example: • inference from OLS, robust OLS results is not valid in the presence of inefficiency • functional form (linear) is not substantiated with evidence—alternative functional forms could result

in alternative models • the intercept is manually set to zero, with no justification

• the statistical assessment of the final model is limited and not informative • the value of adjusted R-squared in a regression of costs against NormGrid is not informative.

NormGrid is a representation of the average costs


Model development Sensitivity to the sample selected (I) (section 4.2)

18

• the estimated relationship between costs and cost drivers is highly sensitive to the sample on which the model is estimated • Sumicsid removed one TSO from the reference set, but does not explain the basis for this—

e.g. what tests were performed? • the estimated coefficients in the cost model are highly sensitive to the inclusion/exclusion of some

TSOs • NormGrid and weighted lines may have positive or negative coefficients depending on the exact

sample • on average, an increase in the cost driver is not associated with an increase in costs

• the relationship between the costs and cost drivers in Sumicsid’s model is highly dependent on the exact sample of TSOs used and thus unreliable


Minimum Median Maximum Intercept -13713113 -9717442 2641704 Adj. NormGrid -0.14 -0.09 0.54 Transformer power 2629.86 4103.23 4535.93 Weighted lines -3498.12 64817.61 71005.72 Observations 761 76 76

Note: In these estimations we omit one TSO (i.e. 5 observations) at a time hence the sample of 81 observations reduces to 76. Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Table 4.5.

Model development Sensitivity to the sample selected (II) (section 4.2)

19

• the year of assessment (2017) is arbitrary and the estimated efficiencies are highly sensitive to it • one TSO that is a peer in Sumicsid’s analysis is estimated to be 82% efficient prior to 2016

• large changes in efficiency should not occur if the model is robust. Such changes may be indicative of model development issues, for example: • relevant cost drivers may be missing, e.g. declining asset health may lead to higher expenditure • the input price inflation might have been understated, thus large investments may seem inefficient

• it is especially important to justify the large swings and how these coincided with a change in the efficiency of the TSOs



Model development Selecting candidate cost drivers (section 4.3)

20

• restriction of outputs to asset-based measures is unusual and unsubstantiated • asset-based measures create issues when the asset structure differs between TSOs • lack of comparison to models involving pure outputs means the cost drivers remain unvalidated • using alternative (asset-based) outputs to capture the same characteristics results in significantly

different results (e.g. transformer power and circuit end power are both proxies for capacity) • it is considered best practice to arrive at appropriate cost drivers through iterative consultations with

companies and stakeholders



Model development Aggregation of NormGrid (section 4.4)

21

• construction of NormGrid is not robustly justified • Sumicsid has not presented empirical evidence as to how the OPEX and CAPEX weights are

derived • the distribution of assets across TSOs is heterogenous. Thus, the weight attached to an asset

class has an impact on estimated efficiencies • changing the weights implied by regression analysis has an impact on some but does not validate

the approach for others (where the impact is less material) given other limitations • using components of NormGrid as the output variables in DEA significantly changes the distribution

of efficiencies


0%10%20%30%40%50%60%70%80%90%

100%

Estim

ated

effi

cien

cy

Sumicsid's model Components ModelSource: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 4.6.

Key messages Adjusting for environmental factors (section 4.5)

22

• environmental adjustments to NormGrid are not supported by evidence • we estimate a counterintuitive (negative) relationship between cost per NormGrid and environmental

adjustment (i.e. TSOs operating in ‘more complex’ environments have lower unit costs) • the biggest driver of environmental adjustment is the proportion of area covered in forest

• more operationally intuitive drivers of expenditure, such as urbanity, mountainous area and the proportion of surface area covered in infrastructure, have a small impact on the overall environmental adjustment


Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 4.8. Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 4.7.

00.050.1

0.150.2

0.250.3

0.350.4

0.45

Env

ironm

enta

l wei

ght -

1

Urban Infrastructure Forest Shrubland Wetland

5

Application and validation 23 Strictly confidential



Model development

◄Returns-to-scale assumption ◄Outlier analysis ◄Validation of DEA outputs ◄Identification of omitted cost drivers ◄Stochastic frontier analysis (SFA) ◄Dynamic efficiency (frontier shift)





Application and validation Returns-to-scale assumption (section 5.1)

25

• returns-to-scale assumption is not supported by statistical evidence • Sumicsid states that it has used two tests to support its returns to scale assumption:

• a Banker F-test for returns to scale on the DEA efficiencies • a test that the sum of coefficients in a log-linear regression is different from 1

• even performing the same tests we find no conclusive evidence supporting the non-decreasing returns to scale (NDRS) assumption made in TCB18

• the assumption has a significant impact on the shape of the efficient frontier and the efficiencies • any decision made must be robustly justified by both conceptual and empirical evidence


Test Null hypothesis

Alternative hypothesis

P-value Favoured RTS

Banker test (DEA) CRS NDRS 49% CRS

Banker test (DEA) NDRS VRS 5% VRS

Intercept (OLS in levels) CRS VRS 39% CRS

Sum of coefficients (OLS in logs) CRS VRS 9% VRS

Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, section 5.1.

Application and validation Outlier analysis (section 5.2)

26

• outlier procedures are not justified and are scientifically inadequate • Sumicsid relies on ARegV, but the outlier procedure specified in ARegV is neither legally binding nor

sufficient in an international benchmarking context • Sumicsid’s dominance test is not scientifically valid and is biased against the detection of

outliers. It is also not valid in small samples • we recommend the use of a bootstrap-based dominance test instead

• the super-efficiency test adopted by Sumicsid does not test whether any of the remaining TSOs are super-efficient in the new sample and is biased against detecting outliers in volatile samples • we recommend a sequential application. We have also considered how appropriate thresholds for

outlier detection can be defined • our alternative tests—a bootstrap-based dominance test, iterative super-efficiency—are able to

identify potential outliers and result in a more homogeneous comparator set


Application and validation of DEA outputs (section 5.3)

27

• DEA outputs were not discussed in Sumicsid’s final report and remain unvalidated • examining them reveals that most TSOs do not have NormGrid as the main driver of their efficiency

• this contrasts Sumicsid’s claim that NormGrid is the strongest cost driver • peer analysis appears to have been ignored • there is no discussion of whether the identified peers for inefficient TSOs are appropriate

• it is possible to perform a precise one-to-one assessment of the homogeneity of the units • alternatively, one could assess the weights (i.e. ‘scaling factors’) on peers—some ‘inefficient’ TSOs

are benchmarked against TSOs that are more than 12 times smaller



Application and validation Identification of omitted cost drivers (section 5.4)

28

• second-stage analysis forms the core of Sumicsid’s model validation • second-stage analysis has no theoretical basis for model validation purposes • Sumicsid has not considered necessary tests to justify the approach

• second-stage analysis would not support Sumicsid’s own model—neither NormGrid nor weighted lines will come out relevant in the second-stage regressions • this is not an argument to exclude NormGrid and weighted lines from the model. Rather, these

results indicate that second-stage analysis of this kind is not able to detect omitted cost drivers • it cannot be argued (on the basis of this procedure) that no relevant cost drivers were omitted


Omitted cost driver Coefficient on omitted cost driver P-value Relevant (omitted)

variable? Adj. NormGrid -1.1E-10 0.48 No Transformer power -2E-06 0.01 Yes Weighted lines -2.6E-05 0.19 No Environment 0.06014 0.88 No Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Table 5.2.

Application and validation Stochastic frontier analysis (section 5.5)

29

• Sumicsid has not used SFA or any other benchmarking method to validate its model • SFA is not deterministic (unlike DEA) and is thus particularly relevant when we suspect a lot of noise

in the data • Sumicsid argues that the sample is too small for a full-scale application of SFA and does not

present the results from SFA models • this argument is not specific to SFA; most empirical investigations (including DEA) perform better

on larger samples • sample size alone is not a sufficient justification for ignoring SFA

• SFA indicates that noise is being identified as inefficiency by Sumicsid’s application of DEA • only if the LR test detects statistically significant inefficiency and coefficients are intuitive could

meaningful efficiency estimates be derived • this suggests that caution is warranted against interpreting any estimated inefficiency in the

DEA as actual inefficiency rather than statistical noise


SFA estimator Does the LR test detect statistically significant inefficiency?

Intuitive interpretation of coefficients?

Pooled SFA No Yes Time-invariant SFA No Yes Time-varying SFA No Yes Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Table 5.4.

Application and validation Frontier shift (section 5.6)

30

• dynamic efficiency results (-4.1% p.a.) cast further doubt on the validity of results • such a large and negative frontier shift result differs from what is often applied in regulatory contexts

and could be indicative of model mis-specification • cost drivers poorly explain changes in efficient costs over time and differences in efficient costs

between TSOs • similar to the individual inefficiency estimates, frontier shift is highly volatile and indifferent from zero • the relative efficiency scores and frontier shift estimates in the TCB18 study are fundamentally

related—it cannot be that one piece of evidence is deemed more robust for regulatory applications than another, given both parameters are estimated using the same data, model and methodology


Pooled SFA Panel SFA (time-invariant)

Panel SFA (time-varying)

NormGrid 0.551*** 0.929*** 0.891*** Transformer power 0.520*** 0.147 0.12 Weighted lines 0.0146 0.065 0.115 Time trend 0.0464 0.0284*** 0.0229 Constant 2.688* -1.593 -0.933 Implied frontier shift (% p.a. ) -4.7% -2.9% -2.3% Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Table 5.5.

6

Conclusion and recommendations


Overall summary of our assessment

• we have issued a series of principles-driven recommendations • significant additional work is needed to ensure that the outcomes from the study are

robust and fit for purpose

32

TCB18 suffers from fundamental weaknesses that mean that the estimated efficiency scores cannot be used for regulatory, operational or valuation purposes

We have not presented the combined results in which all our recommendations are jointly implemented

• we have identified numerous fundamental issues with Sumicsid’s analysis • given the lack of relevant data, we could not develop alternative models with outputs as

cost drivers • the overarching conclusion from data processing and SFA is that the data is too noisy


Full assessment

• our full assessment of the issues and their impact can be found in the main report

Strictly confidential 33

Recommendations Transparency and process

34

• clear conceptual and empirical justifications for any assumptions • publish modelling codes (that are anonymised) and provide detailed description of

process and outputs • establish an iterative data-collection procedure to minimise data errors • make effective use of panel data and estimate static and dynamic efficiencies to assess

the validity of the models • use alternative model specifications and evidence from alternative estimators such as

SFA, economic and operational perspectives to motivate assumptions and cross-check results

• it can be helpful to consider debriefs involving all the parties on process and methodology to help future studies


Recommendations Analysis

35

• evaluate the impact of data errors using various methods (e.g. Monte Carlo) • as real data is noisy, derive confidence intervals around any results, including estimated

efficiency scores • capture input price differences and other heterogeneity robustly • perform scientifically valid model development

• based on realistic modelling assumptions • test the sensitivity to changes in the sample • develop alternative model specifications

• the causes of year-to-year efficiency fluctuation must be explored • asset-based outputs—if used—must be validated with pure output models • develop an appropriate outlier-detection procedure for international benchmarking • analyse the outputs of DEA to ensure that they are consistent with intuition • the impact of omitted cost drivers should be assessed through sensible robustness

checks—not flawed second-stage analysis


Contact: Dr Srini Parthasarathy +44 (0) 20 7776 6612 [email protected]

Oxera Consulting LLP is a limited liability partnership registered in England no. OC392464, registered office: Park Central, 40/41 Park End Street, Oxford OX1 1JD, UK; in Belgium, no. 0651 990 151, branch office: Avenue Louise 81, 1050 Brussels, Belgium; and in Italy, REA no. RM - 1530473, branch office: Via delle Quattro Fontane 15, 00184 Rome, Italy. Oxera Consulting (France) LLP, a French branch, registered office: 60 Avenue Charles de Gaulle, CS 60016, 92573 Neuilly-sur-Seine, France and registered in Nanterre, RCS no. 844 900 407 00025. Oxera Consulting (Netherlands) LLP, a Dutch branch, registered office: Strawinskylaan 3051, 1077 ZX Amsterdam, The Netherlands and registered in Amsterdam, KvK no. 72446218. Oxera Consulting GmbH is registered in Germany, no. HRB 148781 B (Local Court of Charlottenburg), registered office: Rahel-Hirsch-Straße 10, Berlin 10557, Germany.

Although every effort has been made to ensure the accuracy of the material and the integrity of the analysis presented herein, Oxera accepts no liability for any actions taken on the basis of its contents.

No Oxera entity is either authorised or regulated by the Financial Conduct Authority or the Prudential Regulation Authority within the UK or any other financial authority applicable in other countries. Anyone considering a specific investment should consult their own broker or other investment adviser. Oxera accepts no liability for any specific investment decision, which must be at the investor’s own risk.

© Oxera 2020. All rights reserved. Except for the quotation of short passages for the purposes of criticism or review, no part may be used or reproduced without permission.

www.oxera.com Follow us on Twitter @OxeraConsulting

a critical assessment of tcb18 electricity · identification of omitted cost drivers stochastic...

Documents