why government projects go over budget …and what to do about it

1 Hillwatch Inc., PO 4824, Station E, Third Ave., Ottawa, Ontario, K1S 5H9, Canada

Ph: (613) 238-8700, Fax: 866 310 4955, [email protected] Visit www.hillwatch.com

Why Government Projects Go Over Budget

…And What To Do About It

By Scott Proudfoot, Hillwatch Inc.

Major capital projects are an unwinnable game for any party in power. Projects that fall behind

schedule and go over budget are a predictable certainty. Each errant project hammers a dent in the

Government’s reputation. With opposition and media scrutiny, the dents add up. The inevitable

result is a diminished reputation for managerial competence - one precursor to eventual electoral

defeat. To absorb fewer dents, you have to change the rules of the game!

Problems That Are Deeper and Wider

The Harper Government entered office determined to re-equip the Canadian Military. To prove the

adage: ‘No good deed goes unpunished’, a series of bad news stories has resulted.

The Joint Support Ship project blew up when the winning bid was grossly over-budget. The Close

Combat Vehicle project wasted copious amounts of time and money before being mercifully cancelled.

Our second-hand subs seem to spend more time in the shop then under the seas. The ‘priority’ Fixed

Wing Search and Rescue procurement has been kicking around for a decade. That the fighter jet project

will cost more than originally budgeted is virtually certain.

The military, which needs the equipment, is the loser but not the sole one. Each failed or stalled project

hammers a dent in the Government’s reputation for managerial competence. These projects are chum

in the water for the media and opposition parties who smell the blood and gather to thrash and feed.

The columnist Andrew Coyne, amongst others, blames the problems on the Canadian industrial benefit

preferences built into bids. Others cite the lack of project management expertise, poor morale and high

turnover in government staff. Still others point to overly complex government requirements.

The Harper Government has not been ignoring the problem. The push for P3 projects is part of an

overall effort to mitigate project creep, ambiguous outcomes and public sector risk. Another thrust is to

fix budget levels and demand departments shave their requirement or number of units requested.



Industry consultations have been on-going.

Experts have convened and reported. More

Cabinet and senior official scrutiny has been

announced and 3rd Party Expert Panel Review

of larger projects.

One wants to be optimistic but a large caution

flag has to be waved!

These are process changes. Typically,

administrative failure in bureaucracy is dealt

with by focusing a scarce resource – senior

decision-makers’ time and attention - in the

hope of finding a solution. Sometimes it

works, often it doesn’t.

Third party expert reviews have their place

but building them into every major project has

a potential cost. It is another layer of

oversight for a system that cannot keep to a

schedule. There is a high risk of

‘backscratching’ as current officials hire

former officials for great consulting gigs. It is

a way of establishing ‘plausible deniability’ for

future mistakes. Most relevantly, experts are

largely unsuccessful in forecasting future costs

and riski.

The unmet challenge is overcoming history

and a high statistical probability of failure.

A long-term study of megaprojects

determined that nine out of ten such projects

had cost overruns. Overruns of 50% in real terms are common; cost overruns over 50% are not

uncommon. Despite computerization and changes in forecasting methods there have been no signs of

improvement over decades.

Another study of transportation infrastructure projects found inaccuracy in cost forecasts was 44.7% for

rail; 33.8% for bridges and tunnels and 20.4% for roads. Over a 70 year period, cost-forecast accuracy

had not improved.

Projects and % cost overruns

Suez Canal, Egypt 1,900%

Scottish Parliament Building, Scotland 1,600

Sydney Opera House, Australia 1,400

Montreal Summer Olympics, Canada 1,300

Concorde Supersonic Aeroplane, UK, France 1,100

Troy and Greenfield Railroad, USA 900

Excalibur Smart Projectile, USA, Sweden 650

Canadian Firearms Registry, Canada 590

Lake Placid Winter Olympics, USA 560

Medicare transaction system, USA 560

Bank of Norway headquarters, Norway 440

Furka Base Tunnel, Switzerland 300

Verrazano Narrow Bridge, USA 280

Boston’s Big Dig Artery/Tunnel project, USA 220

Denver International Airport, USA 200

Panama Canal, Panama 200

Minneapolis Hiawatha light rail line, USA 190

Humber Bridge, UK 180

Dublin Port Tunnel, Ireland 160

Montreal Metro Laval extension, Canada 160

Copenhagen Metro, Denmark 150

Boston–New York–Washington Railway, USA 130

Great Belt Rail Tunnel, Denmark 120

London Limehouse Road Tunnel, UK 110

Brooklyn Bridge, USA 100

Shinkansen Joetsu high-speed rail line, Japan 100

Ontario Smart Meter Program, Canada 100

Channel Tunnel, UK, France 80

Karlsruhe–Bretten light rail, Germany 80

London Jubilee Line extension, UK 80

Bangkok Metro, Thailand 70

Mexico City Metroline, Mexico 60

High-speed Rail Line South, The Netherlands 60

Great Belt East Bridge, Denmark 60%



A recent study examined data from 245 large dams in 65 different countries. The findings show the

construction costs of large dams are on average +90% higher than their budgets at the time of approval,

in real terms. This was before accounting for the impacts on human society and environment, and

without including the effects of inflation and debt servicing.ii.

Defence projects, large IT projects, bridges, tunnels, LRTs, Subways, Opera Houses, Parliament Buildings

– the pattern repeats. A high proportion of government projects are over-budget; a significant portion

spectacularly so. Despite new methodologies and better tools, the trend has not improved. Large

private sector projects can be similarly afflicted (e.g., the Barrick Gold and the Pascua-Lama project in

Chile).

This problem is not restricted to large projects. A survey of US homeowners who remodeled their

kitchens expected the job to cost $18,658 on average but, in fact, paid $38,769 on average. If we

individually reviewed our own home or work-related projects, most of us would prove no better than

government planners.

There is a persistent human pattern of repeated failure when it comes to estimating the schedule and

cost of projectsiii. This should give pause to the Harper Government as it grapples with defence (and

other) projects going forward.

But, there may be a light at the end of the tunnel. This illumination has been provided by the work of

researchers on megaprojects which incorporates their own findings with recent insights from the fields

of neuroscience, cognitive psychology, and behavioral economics.

Strategic Misrepresentation

The term Strategic Misrepresentation has been coined to describe why large government projects have

such a poor record. The thesis is as follows:

Projects go over budget, take longer and generate fewer benefits because politicians want to build

monuments and/or officials face fierce competition for intra-departmental and inter-departmental

funding. Project cost estimates, schedules and the reputed benefits are skewed to ensure that favoured

projects are approved

Politicians and officials are not rational and independent arbiters of the public interest but people with

biases and preferences who advocate for some projects over others. Instead of carefully weighing all

the facts and then deciding; our leaders decide what to do and pick the facts to support their chosen

course. This process, and the risks attached, are aided and abetted by technologists and engineers who

push the envelope for the newest and latest toys. They are also supported by contractors, unions,



consultants, investors, bankers, lawyers and developers who stand to financially gain and tout the

‘economic development’ benefits of the projects.

Advocates, within and outside government, come forward with the ‘best-case’ scenario for these

projects. Since it is widely understood government projects are seldom abandoned once commenced,

the risks are downplayed and the benefits oversold.

“What we often get is an inverted Darwinism; the survival of the un-fittest. It is not the best

projects that get chosen, but those that look best on papers. And the projects that look best on

paper are those with the largest cost underestimates and benefits overestimates. They are

disasters waiting to happen.”iv

It is hard to argue this does not happen since there are multiple examples of where it has. As this article

was being written, the Ontario Auditor-General provided a case study wrapped up in a bow.

In 2004, the Ontario Government announced an aggressive program for installing residential smart

hydro meters. They did so without any cost-benefit analysis or business case prior to the decision. The

projected net benefits were grossly overstated by half a billion because not all costs were counted. The

implementation and integration costs were much higher than budgeted. The projected reduction in

peak use of electricity never materialized. The program generated widespread complaints and

dissatisfaction from customers. Costs escalated to $2 billion instead of the $1 billion originally

announced. A political agenda drove a poorly-planned, poorly-implemented project that achieved few

of its objectives and cost twice as much as originally promised.

But, strategic misrepresentation does not explain why all government projects go over-budget. Every

public project is not a reflection of grandiose political ambitions or a perverse internal bureaucratic

competition for funding.

It has less explanatory value when it comes to defence projects in Canada. In my experience, most

defence officials are competent and well-intentioned. Established costing methodologies are used.

Officials try to make good choices. Wanting the latest technology is a rational objective for anyone in the

business of delivering and avoiding lethal effects.

The mistakes being made are unintentional. The roots of these mistakes are forms of self-deception

difficult to detect or recognize after the fact.

The Planning Fallacy

Research in the fields of neuroscience, cognitive psychology, and behavioral economics has exposed the

irrational aspects and hidden biases of our decision-making. Crudely summarized, the way in which our



brain is wired (particularly the interaction between the ‘automatic’ mind and the ‘rational’ mind) can

lead to multiple mistakes in judgment. We jump to the wrong conclusions and make poor decisions.

The psychologists Daniel Kahneman and Amos Tversky spend most of their careers studying decision-

making with a particular focus on tracing errors in the machinery of cognition. They cited two inter-

connected cognition biases as primarily causing what they called the Planning Fallacy. (This work helped

Kahneman win the Noble Prize in Economics in 2002.)

WYSIATI (What you see is all that there is): We see the future as an incremental adjustment to all that

we know and believe about the present. We underestimate the unknown and unpredictable nature of

the future and falsely assume we have more control than we do. We are programmed to: “see the

world as more tidy, simple, predictable and coherent than it really is”v. But, the future turns out

different than expected.

“What we anticipate seldom occurs; what we least expect generally happens.” Benjamin Disraeli

This is just as true for managers of large capital projects. Unanticipated ‘surprises’ delay and increase

the cost and risks of their projects. Large projects are inherently risky due to long planning periods and

the different moving parts. Managerial experience may be limited and/or turnover can be common.

There is the risk of too many cooks in the kitchen as multiple players are involved with key decisions.

There can be high levels of design and technology risk.

Miscalculating the risk profile of any of these variables can lead to cascade effects that knock-on to

other parts of the project; delaying schedules and escalating costs.

Optimistic Overconfidence: People overestimate both their abilities and their prospects.

“Hundreds of studies have shown that people overrate their health, leadership ability,

intelligence, professional competence, sporting prowess, and managerial skill. People also hold

the nonsensical belief that they are inherently lucky. Most people think they are more likely than

the average person to attain a good first job, to have gifted children, and to live to a ripe old age.

They think they are less likely than the average person to be the victim of an accident, crime,

disease, depressions, unwanted pregnancy or earthquake.”vi

Optimistic overconfidence is a useful species trait. It helped our species leave Africa over 80,000 years

ago and take over the world. It is a major driver of capitalism. A certain level of positive delusion makes

us more confident, happier and better able to cope with the world. It has less utility when planning and

predicting the costs of large capital projects.

The planning fallacy is the over-reliance on the ‘inside’ view held by the participants to a project who are

focused on the specific and unique circumstances of their own project.



“When forecasting the outcomes of risky projects, executives too easily fall victim to the planning

fallacy. In its grip, they make decisions based on delusional optimism rather than on a rational

weighting of gains, losses, and probabilities. They overestimate benefits and they underestimate

costs. They spin scenarios of success while overlooking the potential for mistakes and

miscalculations.”vii

For a different set of circumstances, both strategic misrepresentation and the planning fallacy lead to

project plans unrealistically close to best-case scenarios.

For Kahneman and Tversky, the antidote was to counter the ‘inside’ view with an ‘outside’ view and

compare and then revise your projections against the statistics of similar projects.

Reference Class Forecasting

Planning experts have taken their insights and developed Reference Class Forecasting (RCF) - away of

systematically taking an outside view of a planned projectviii. It involves three steps:

Identification of a relevant reference class of past, similar projects – broad enough to be

statistically meaningful and narrow enough to be comparable with the specific project.

Establish a probability distribution for the selected reference class. This requires access to

empirical data.

Compare your specific project with the reference class distribution in order to estimate the most

likely outcome for your project. Then use that data to revise your cost estimates and schedule

based on the statistics of the comparable projects.

Many government projects are delivered on time and on budget. Under a Reference Class Forecasting

system, any representative data set of projects would include those projects along with the ones that go

off the rails. RCF is not benchmarking to the worst projects just the average of a representative list of

comparable projects.

The underlying assumptions of RCF are:

Attempting to estimate and roll-up your costs and risks over multiple years is not reliable, and

Revising your estimates upwards as needed based on the actual aggregated and averaged costs

of similar projects will be more reliable.

Compared to 3rd Party Expert Reviews, RCF should have the virtue of being quicker, cheaper and more

accurate.



This methodology has now taken a life of its own. The UK Ministry of Transport uses it with all their

infrastructure projects. Other governments are using it. Private sector companies in the business of

large projects are gravitating towards it. It has been endorsed by the American Planning Association and

Association for the Advancement of Cost Engineering.

The Parliamentary Budget Office (PBO) has applied this methodology to the Joint Support Ships –

Feasibility of the Budget for the Acquisition of Two Joint Support Ships, PBO, February 28th, 2013. More

recently, the PBO also applied it to the Arctic Offshore Patrol Ships – Budget Analysis for the Acquisition

of a Class of Arctic/Offshore Patrol Ships, October 28, 2014.

The PBO used third party softwareix to compare the JSS estimates against actual budgets incurred for

eight similar naval vessels built in the last 20 years. To compare apples with apples, they put all their

data in present dollars and treated the other vessels as if they were the same weight as the proposed

JSS. The PBO then calculated the probability of the JSS being successfully completed under certain

budget levels.

The originals JSS budget was set at $2.1 billion. It was later revised to $2.6 billion. A third estimate will

soon be provided but the PBO worked with the earlier public numbers. The PBO analysis suggests there

was less than a 5% probability that the JSS could be completed for $2.6 billion and only a 50/50 chance it

could be done for $4.13 billion.

We would expect a government entity to work with prudent estimates. The UK Department of

Transportation works with an 80% probability threshold. In the case of the JSS, based on the PBO

analysis, that would mean setting aside a reserve capital budget of $5.13 for the project. This is almost

double the Government’s last public estimate.

In the case of the Arctic Offshore Patrol Vessels, the PBO used a data set of 14 similar ship projects.

After various calculations, it suggested, that for the stated budget, the Government would have

difficulty establishing an 80% confidence level for building even four ships instead of the eight, then six,

that have been discussed.

Not surprisingly, the Government and the PBO are disputing each others’ numbers. Effectively, they are

using two separate accounting methodologies and talking past each other. DND is using the ‘inside

view’ – a traditional bottom-up estimation of the costs based on a detailed requirement. PBO is using

the ‘outside view’ - a variant of Reference Class Forecasting. One is trying to develop a fixed cost

estimate for projects to be completed over the next ten years. The other is using statistical analysis to

determine the probability of those estimates being correct given experience on similar projectsx.

But the Government should consider whether the PBO, in its own jargon-choked, wonkish wayxii, is

actually providing them with good costing and political advice!



It’s The Accounting, Stupid!

Ministers and Officials spend on the basis of delegated authority from Parliament. Officials

(intentionally or unintentionally) have been incorrectly pricing the risks and costs of some projects.

Ministers have been approving and passing on those numbers to Parliament.

Because of the size of large capital projects, budget overruns have an outsized impact on departmental

budgets and overall government spending. Whether more deficit financing or forced cuts elsewhere,

poorer financial governance is the end-result.

The antidote to optimistic costing in any private or public enterprise is sound accounting practice!

Reference Class Forecasting provides a more objective and prudent method of estimating these risks

and costs. It is a counterweight to both deliberate and/or accidental misrepresentation.

If Ministers of the Crown are approving a project they are told will cost $3 billion (but there is a high

probability the project will cost $5 billion and take longer to complete), then Ministers should have that

information when taking the initial decision. They may go ahead but it would be with their eyes open

and a proper reserve budget established. They will have met their responsibility to present Parliament

with the most accurate estimates available.

Reduce Political Risk!

Beyond good financial governance, parties in power, or those hoping to be in power, have a vested

interest in getting the accounting right on major capital projects.

These days, few people enter Cabinet with prior experience running large organizations. Fewer still have

managed large capital projects. The chance that a Minister in charge of major capital projects has some

relevant actual experience is a low probability event. Beyond good intentions, the political class brings

no special competence to the task. They are neither helpless nor in control of events. They are publicly

responsible for projects but heavily reliant on the information and execution provided by officials.

The damage these errant projects do to a government’s reputation will vary based on circumstances. It

is not strictly about the dollars. A few senators and a measly half-million dollars in expense claims has

consumed months of media and public attention. Even with billions of dollars involved, ‘bad news’

projects usually are an intermittent news story.

The odd project going astray is politically manageable. The risk comes from a slow drip of similar stories.

The reputation of a government begins to suffer with the reporting of what seem to be a cluster of bad

projects. The cascade of bad news creates the impression of a government losing control of the



procurement process. Ministers provide project estimates only to revise them upwards or try to

explain away repeated schedule delays. People stop believing the government knows what it is talking

about or what it is doing. It appears evasive. It loses credibility. Procurement becomes a target-rich

subject for the Opposition and the Media who work hard to keep the stories cycling.

Beyond cancelling all projects and coming to a complete halt, Governments have little control over

whether bad projects cluster. Good or bad clusters are a by-product of a random distribution of results.

Human psychology will ignore a cluster of good results but fixate on the bad ones and automatically

assume they are more common.

Governments lose the confidence of the voters for any number of reasons but a principal cause is they

come to be perceived as less managerially competent. Over-budget projects play to the narrative of

incompetence.

By design, Reference Class Forecasting is more financially cautious and prudent. It simply assumes that

a government has a high probability of performing no better than other governments who managed

similar projects. Therefore, under an RCF budgeting system the starting assumptions for any project is a

longer schedule and a higher reserve budget to cover unanticipated costs.

If the initial schedules and budgets are set on this basis, fewer projects will go over-budget. It does not

eliminate a cluster of bad projects but it should reduce their size and frequency. There will then be less

public attention and criticism. The Government has mitigated the associated political risks.

Pros and Cons

Moving to Reference Class Forecasting (RCF) would seem to be a no-brainer for governments. Most

OECD countries provide information on the final costs of large projects. Database software and

statistical tools can be purchased off the shelf. It is possible to generate thousands of data-sets on

government projects.

However, resistance can be expected:

Algorithms over Experts: RCF involves admitting your internal experts and their methods are not

reliable. Senior officials and Ministers would be replacing their judgments and giving control to,

statistical algorithms. That can be hard for the Type ‘A’ individuals who habituate the senior ranks of

government. They are not inclined to believe their judgments are so fallible or they are destined to

repeat the mistake of their predecessors. This is called hubris but a status quo, even a dysfunctional

one, finds many supporters.



Fewer Projects: An inevitable result of moving to Reference Class Forecasting is higher reserve capital

budget for new projects. Unless the Government sets aside more money, it would have to postpone or

cut planned projects. For example, under the Canada First Defence Strategy, the Government is

committed to a series of projects under that funding envelope. It is clear the available funds will fall

short of the likely cost of those projects. RCF would make that glaringly apparent. There is a practical

limit to how many dollars can be moved from other spending priorities. Since all departments have a

long shopping list of upcoming projects, an RCF budgeting requirement would lead to many projects on

those lists being unaffordable.

This may be more perception than reality. Cost overruns in existing projects routinely lead to the

postponement or cancellation of projects. Departmental spending priorities change with the times, or

in the case of DND, with the nature of perceived threats. Delays in listed projects are a chronic feature

of the existing systems. Governments may slow down capital spending for fiscal reasons. Government

spending plans, presented as fixed, are a highly discretionary wish list.

Still, officials, and those with a vested interest in certain projects, are likely to believe something has

been taken away and will push back. RCF would present a shock to the system. It requires

determination to implement. After the initial caterwauling, departments and suppliers would adjust.

Still Expensive: It can be argued that RCF solves the bookkeeping problem but it still does not address

the issue that government projects are still too expensive.

That is correct. Improving the bookkeeping does not remove government’s responsibility for

management of these projects. The onus is still on officials to manage projects more cost-effectively.

The Government can provide some incentives by allowing departments, which deliver projects under

the reserve budget, to redirect unused capital to other projects.

Contractors would Game the System: It will be argued that Government contractors will find a way to

spend up to the new higher reserve budgets. That is certainly possible. Experienced government

contractors are adept at using the system to extract more revenue from projects.

But, contractors cannot game the system without the cooperation of officials who themselves realize

their numbers make no sense. Using over-optimistic spending plans as a tool to curb the ‘greed’ of

contractors is a derogation of management. The incentive to game the system is strongest on the part of

both contractors and officials when budgets are unrealistic.



Conclusion

There is a need for better management of these projects that Reference Class Forecasting does not

address. RCF is mostly a way to strip-out the fantasy elements of how risks and costs are initially

established.

Governments have a choice. They can continue to allow the battering of their reputation and electoral

prospects, or place procurement costing on a more conservative and prudent basis, and reduce their

personal political risk.

When fiscal responsibility and political self-interest align, smart politicians should say ‘Yes’.

i For a great tour through the predictive failures of experts, read Dan Gardner’s, Future Babble: Why Expert Predictions Fail and Why We Believe Them Anyway, McClelland & Stewart. Philip Tetlock’s, Expert Political Judgment, Princeton Paperbacks, is also a great examination of predictive failure. ii Energy Policy, Volume 69, June 2014, Pages 43–56, “Should we build more large dams? The actual costs of

hydropower megaproject development” iii We do not know how many defence projects or IT projects or Infrastructure projects come in over or under

budget and by how much. Do some projects perform better than others? Do small projects have the same problems as big projects? The data exists within Government to answer these and other questions but there is no incentive to aggregate the data. The results are likely to point to problems both expensive and not easily solved. iv Flyvbjerg, Brent, “ Mega-Delusional: The Cures of the Megaprojects,” New Scientist, December, 2013, pp.28-29

v P. 203, Thinking Fast and Slow, Daniel Kahneman, Random House, 2011

vi Pp. 511-512, The Better Angels of Our Nature, Steven Pinker. Penguin, 2012

vii P. 252, Thinking Fast and Slow, Daniel Kahneman

viii From Nobel Prize to Project Management: Getting Risks Right, Brent Flyvbjerg, Project Management Journal, Vol

37, no 3, August 2006. ix The PBO used TruePlanning which is a proprietary cost estimating tool which is said to be used by the US

Department of Defense, Sikorsky Aircraft, NASA, BAE Systems, Gulfstream, United Technologies and Boeing. x The Parliamentary Budget Office does some good work and some so-so work but it is a lousy communicator. It is

using Reference Class Forecasting, without explaining what it is or why it is doing so. Conditional probability algorithms are better markers when establishing long- term budget forecasts, but only mathematicians, actuaries and behavior economist will find this intuitively obvious. The rest of the world needs it patiently explained. xii

The Parliamentary Budget Office is a worthwhile public policy initiative poorly executed. Their reports are written by econometricians for econometricians. Reading their reports is tough-slogging - the eyes glaze over. The first word in their name is Parliamentary. That would suggest they should write their reports so the average Member of Parliament, who is not an econometrician, would want to read them. The press and the opposition extract what is critical of the government and ignore the rest. If the PBO wants to fulfill their mandate and have a broader impact on public policy discussions, they should consider making their research accessible.

why government projects go over budget …and what to do about it

Documents