george mason university decisionsmason.gmu.edu/~aloerch/intro651.pdf · operations research*...

1

George Mason UniversityThe School of Public Policy and the School of Information Technology & Engineering

Enterprise Engineering & Policy01/22/2001 1

Cost AnalysisS.J. Balut

Cost Analysis and Research DivisionInstitute for Defense Analyses

Th. R. GulledgeThe School of Public Policy and

The Department of Systems Engineering & Operations ResearchGeorge Mason University



DecisionsFor our purposes, a decision is judgement based on:

Experience

Intuition

Analysis

2



AnalysisAnalysis is:

A practical philosophy on how best to aid a decisionmaker with acomplex of choice under uncertainty.

Application of the scientific method problems of economic choice.

Characteristics of analysis:

Systematic

Analytical

Cannot supplant decisionmaking

Cannot prove optimality

Continuous dialogue between analyst and decisionmaker



Analysis(Continued)

Elements of analysis:Continuous iterative cycleExplores alternatives

CostsEffectiveness

Compare in light of the consequencesPresent to decisionmaker

Brings to bear value judgementsRejects some alternativesMakes decisionsAsks for more alternatives within framework

established by decisions already made

3



The Iterative Nature of Analysis*

Clarifying the

problem

Determining objectives

and criteria

Opening new

alternatives

Questioningassumptions

Interpretingresults

Evaluatingcosts and

effectiveness

Examining alternatives

for feasibility

Searching outand designingalternatives

Collectingdata and

Information

Buildingand testing

models

Iteration

* Taken from E.S. Quade, Analysis for Policy Decisions. North-Holland, 1989.



Policy AnalysisPolicy Analysis is a form of applied research

carried out to acquire a deeper understanding ofsociotechnical issues and to bring about better solutions.

The general framework for policy analysis is presented by E.S. Quade:

Quade, E.S. Analysis for Public Decisions, Third Edition. New York: North-Holland, 1989.

4



Policy Analysis Studies

Operations Research, Systems Analysis, Cost-Benefit Analysis, Economic Analysis, and Cost-Effectiveness Analysis are often employed in Policy Analysis Studies.

Policy Analysis is broader than the above, taking into account the political and organizational difficulties associated with public decisions and their implementation.



Wicked ProblemsPolicy Analysis problems fall in the most difficult category. As noted

by Quade ,

The problems of public policy are thus likely to be "wicked" problems; that is, they may have no definite formulation or stopping rule to tell the problem solver when he has a good solution. Moreover, a proposed solution is not true or false but good or bad. There may be neither an immediate nor even an ultimate test of a solution - the set of solutions may seem endless. Even for the comparatively simple problems involved in locating as freeway or changing a school curriculum, there is not a solution but a resolution - and this must rely on political

5



Examples of Public PolicyProblems

Analysis alone is clearly not sufficient to provide a solution to such questions as:

How much of a county budget should be allocated to welfare and what portion of this to birth control clinics in several cities?

Are metropolitan transportation needs better served by a rapid transit system or my more and higher-performance freeways?

Is there some legislative action that might end the increase in juvenile delinquency?



Cost-Benefit Analysis*

Cost-Benefit Analysis is an estimation and evaluation of net benefitsassociated with alternatives for achieving defined goals.

Cost-Benefit Analysis is a generic term embracing a wide range of evaluative procedures which lead to a statement assessing costs and benefits relevant to project alternatives.

The variety of problems addressed and the ingenuity which must be exercised in estimating costs and benefits make it particularly difficult, if not impossible, to design an all-purpose Cost-Benefit Analysis procedure.

* Peter G. Sassone and William A. Schaffer. Cost-Benefit Analysis: A Handbook . San Diego: Academic Press, 1978.

6



Operations Research*

Operational Research is the attack of modern science on complex problems arising in the direction and management of large systems of men, machines, materials, and money in industry, business, government, and defense. Its distinctive approach is to develop a scientific model of the system, incorporating measurements of factors such as chance and risk, with which to predict and compare the outcomes of alternative decisions, strategies, or controls. The purpose is to help management determine its policy and actions scientifically.

* British Operational Research Society, Operational Research Quarterly, Vol. 13 #3 (1962), 282.



Systems Analysis*Systems Analysis may be defined as inquiry to assist decisionmakers in choosing preferred future courses of action by

1. systematically examining and reexamining the relevant objectives and the alternative policies or strategies for achieving them; and

2. comparing quantitatively where possible the economic costs, effectiveness (benefits), and risks of the alternatives.

It is more of a research strategy than a method or technique; and in its present state of development it is more like an art than a science, although scientific methods are utilized wherever possible. In sum, systems analysis may be viewed as an approach to, or a way of looking at, complex problems of choice, usually under conditions of uncertainty.

* Gene H. Fisher. Cost Considerations in Systems Analysis. New York: American Elsevier, 1971.

7



Economic AnalysisEconomic Analysis is a tool that is used in the

search of more efficient courses of action.

The elements of an Economic Analysis, and a discussion of economic choice in general, are presented in Chapter 7 of

C.J. Hitch and R.N. McKean. The Economics of Defense in the Nuclear Age, R-346. Santa Monica: The RAND Corporation, 1960.



Economic Analysis(1972)

• The definition of Economic Analysis is given in DoD Instruction 7041.3, Economic Analysis and Program Evaluation for Resource Management, October 18, 1972.– Economic Analysis is a systematic approach to the

problem of choosing how to employ scarce resources and an investigation of the full implications of achieving a given objective in the most efficient and effective manner.

• As of 1972 Economic Analysis and Systems Analysis are synonymous within the Department of Defense.

8



CostNES Dictionary

The amount paid or payable for the acquisition of materials, property, or services........

Gene H. Fisher*

The cost of something is measured by the resources used to attain it.

The cost of attaining an objective at some point in time is measured by the resources not available for use in attaining alternative objectives.

Economic Costs are benefits lost. It is for this reason that economic costs are often referred to as Alternative Costs or Opportunity Costs. It is in the alternatives, the foregone opportunities, that the real meaning of cost must always be found.




Effectiveness*• In cases where there is a single goal to accomplish, the

effectiveness of any alternative is the extent to which the goal is achieved.

• It will be helpful to define the two concepts:

• Effectiveness Scale: A scale indicating the degree to which the objective has been achieved.

• Effectiveness: The position on the effectiveness scale that the alternative has reached.

* E.S. Quade , Analysis for Public Decisions, 3rd Edition. New York: North-Holland, 1987, p. 98.

9



Effectiveness*Effectiveness is the performance or output received

from an approach or program.Outputs are the products, functions, tasks, services,

or capabilities and organization exists to produce, accomplish, attain, or maintain.

Output measures are useful descriptors of functions, tasks, or missions performed by an organization, and of capabilities possessed.

* DoD Instruction 7041.3 (Enclosure 3), 1972.



Cost-EffectivenessAnalysis*

Cost-Effectiveness Analysis is a method for examining alternative means of accomplishing a desired military objective/mission for the purpose of selecting weapons and forces which will provide the greatest military effectiveness for the cost.

* NES Dictionary

10



The Economic Analysis Process*Define Purposes, Goals, and Objectives

Formulate Assumptions

Determine Alternatives

Determine Costs Determine Benefits

Compare and Rank alternatives Using Techniques ofEconomic Analysis

Evaluate Uncertainties

Data, Information, Experience, Judgment, and Analysis Are Required

at Each Step in the Process

Factors Other that Benefits and Costs

DECISION

Quantifiable Quantifiable

Non-Quantifiable Non-Quantifiable

* Source: Department of Energy, Economic Analysis: Methods, Procedures, Life-cycle Costing, Reviewing, and Validating , DOE/MA-0063, January, 1982,



Key PointIf the output is "valued" by the

market, then the analysis is a Cost-Benefit Analysis.

If the output is not "valued" by the market, then the analysis is a Cost-Effectiveness Analysis.

The inputs are always "valued" by the market.

11



Characteristics ofMilitary Systems*

In a consumer-oriented industry, the variables considered in decisionmaking can often be reduced to a common denominator, usually related to cost.

One attribute of a military system is the absence of a market mechanism to determine the relation between a proposed system's cost and its military worth (effectiveness).

Thus, cost and effectiveness are frequently incommensurable; i.e., there is no generally accepted standard for measuring the relation between the value of a military system and its cost.

* See Hitch and McKean, Chapter 9.



Summary of Key Point∆ Value Benefit in this use Benefit in next best use= −( ) ( )

Output Units Measure Input Units* Measure

Market Valued Dollars Revenue Dollars Cost

Non-Market Valued Physical Effectiveness Dollars Cost

* In some cases cost may not be measured in dollars. For example, certain strategic resources (e.g., key scientists), if lost, may involve large opportunity costs. The assignment of a dollar value to this cost may be difficult, but for the purposes of this presentation we assume that such an assignment is possible.

12



Cost-Benefit AnalysisConsider a project that involves a change from the status quo. Let:

vi = valuation ith individual attaches to change: i=1,2,....,n

vi > ⇒0 Change increases ith individual's welfare and "v" is the maximum amount that would be paid for the change

vi < ⇒0 Change decreases ith individual's welfare and "v" is the least amount that would be accepted to agree to the change

Project sanctioned if: v ii

∑ > 0



Cost-Effectiveness AnalysisConsider the same problem, except add a constraint E on

effectiveness or C on cost. Let:

m number of gainers

m number of losers

m m n

1

2

1 2

==

+ =,

The, Cost-Effectiveness Analysis calculations take the form of either:

Minimize vii

m

=∑

1

2

Maximize vii

m

=∑

1

1

St v Eii

m

:=∑ >

1

1

St v Cii

m

:=∑ <

1

2

13



Cost-Benefit Analysis Cost-Effectiveness Analysis

Systems Analysis

Cost Analysis

Policy Analysis



Cost EstimatingNES Dictionary

Cost and price estimating is defined as the art of predetermining the lowest realistic cost and price of an item or activity which assures a normal profit.

Gene H. Fisher*... "making an estimate" of the cost of something implies taking a rather detailed set of specifications and "pricing them out."


14



Cost Estimating and Analysis*"Differences in Disciplines"

"The first issue is the difference in disciplines between cost estimating and analysis...... Even though the two disciplines are functionally separate, they are interrelated so that one depends on the other. Cost estimating, the process of predicting or forecasting the cost of a work activity or a work output, depends on the output of cost analysis, the process of studying and organizing past cost and future estimates."

"...the difference between estimating and analysis must continually be put in perspective. Each has its own function and purpose in the life-cycle of a program. While they are interrelated their purposes should never be misconstrued to accept one as a form of the other."

* Quotes from N. Albert, "Merger Concerns and Objectives," Dollars and Sense, March/April, 1989.



Summary of Terminology• Policy Analysis is the broadest analytical framework.

• Policy Analysis includes a consideration of all “intangibles.”

• Systems Analysis is a subset of policy analysis in that distributional effects are excluded from consideration.

• Cost-Benefit Analysis and Cost-Effectiveness Analysis fall under the Systems Analysis Framework.

• Cost-Effectiveness Analysis is an analytical approach that uses cost and effectiveness as decision criteria.

• Since 1968 Cost-Effectiveness Analysis has been known as Economic Analysis within the DoD.

• Cost-Benefit Analysis is an analytical approach that uses costs and market valued benefits as decision criteria.

• Cost Analysis is a subset of Cost-Benefit Analysis and Cost-Effectiveness Analysis.

15



Hierarchy•Policy Analysis (PA)

Encompasses SA, adding a concern for distributional effects (who pays, who benefits, implementation, acceptance, etc.)

Seeks to do all that SA does, but to see that it is done with equity.

•Systems Analysis (SA)

Grew out of OR and may be considered to encompass it, adding economic considerations and inquiry into goals and their interactions with means.

Seeks to see that the right thing is done, and not only better, but more cheaply.

•Cost-Effectiveness (CE)

Belongs with SA (involves economic considerations but not distributional effects).

Is a tool or procedure of SA

Is an incomplete SA

Almost stands alone provided....…

•Operations Research (OR)

(Originally) sought no more than to do something better, to use scientific methods to get the most out of available resources



Cost-Effectiveness andPoliticians*

Consider the following quote by Quade *

I regard the acceptance of cost-effectiveness analysis in an environment populated with politicians as remarkable --- for politics is geared to winning the voters by outlining the benefits of programs before the means and costs are determined.

* E.S. Quade , A History of Cost-Effectiveness, P-4557. Santa Monica: The RAND Corporation, 1971.

16



BackgroundEconomic Theory and Defense

DecisionmakingA Stroll Through History

PeopleEvents

Trends



History of Analysis

Throughout Draw on economic theory

1940s Rise of Operations Research

1950s/early 1960s Rise of Systems Analysis

Late 1960s/early 1970s Fall of Systems Analysis

1970s/1980s Emphasis on Cost Analysis

1990s Rise of Cost and Operational Effectiveness Analysis

17



Rise of Operations Research

1941 Operations Research invented

1942 ASWORG established

During WWII Operations Research evolved

After WWII Separation of mission responsibility broke down

1946 RAND established

1947 Key West Agreement - DoD established

1948 "Weapon Systems Analysis"

1949 "Systems Analysis"



Rise of Systems Analysis

1950 RAND Cost Analysis Department established1951-56 Novick pioneered Cost Analysis1958 Defense Reorganization Act - PERT developedLate 1950s DoD Civilians losing control1960 RAND suggested applying economic analysis to

government decisions1961 McNamara appointed Hitch - Hitch established

DASD (Systems Analysis) - Beginning of centralized decisionmaking using cost-effectiveness as criteria

1961-65 "Whiz Kids" won battles1965 President Johnson extended PPBS to all Federal

Agencies - Systems Analysis office elevated to ASD

18



Fall of Systems Analysis

1965 Flag officers began speaking out against cost-effectiveness

1968 Schlessinger warned Congress of limitations

1969 Laird de-emphasized role of Systems Analysis - Term "Economic Analysis" substituted

1971-72 Laird/Packard resolve to fix procurement ills

1972 Shift towards "Program Evaluation" - ASD(SA) changed to ASD (Program Analysis & Evaluation)

Through 1970s Cost focus shifted from planning to programming and budgeting



Emphasis on Cost Analysis

1971 Design to Cost

1972 CAIG established

1974 VAMOSC

1981 Carlucci Initiatives

1984 ICE

1985 Service Cost Analysis Centers

19



Rise of Cost and OperationalEffectiveness Analysis

1991 DoD Instruction 5000.2 requiresCOEAs

1992 Weapon System Cost Analysis Division established

1997 Evaluation of Alternatives



Institutional Requirements forCost-Effectiveness Analysis

Department of Defense*"cost and operational effectiveness analyses shall be prepared and considered at milestone decision reviews of acquisition category I programs, beginning with Milestone I, Concept Demonstration Approval.""....(they) shall be tailored and implemented in support of acquisition category II, III, and IV programs as deemed appropriate by the DoD component Acquisition Executives."

Non-Defense Part of Government**"Benefit-cost analysis is recommended as the technique to use in a formal economic analysis of Government programs or projects. Cost-effectiveness analysis ... can be appropriate when the benefits from competing alternatives are the same or when a policy decision has been made that the benefits must be provided."

* DoD Instruction 5000.2, Part 4, Section E, "Cost and Operational Effectiveness Analysis, February 23, 1991.

**Office of Management and Budget Circular No. A-94, Revised, October, 1992.

20



DoD Instruction 7041.3

This document outlines policy guidance and establishes a framework for consistent application of:

1. Economic analysis on proposed programs, projects, and activities,

2. Program evaluations of on-going activities, and

3. Establishes the Defense Economic Analysis Council.



Efficiency and Effectiveness*The determination of efficiency and effectiveness is implicit in the

assessment of the cost effectiveness of alternative approaches, and is accomplished by:

Systematically identifying the benefits and other outputs and costs associated with alternative programs, missions, and functions and/or of alternative ways for accomplishing a given program.

Sensitivity Analysis on key decision variables and assumptions on which decisions are based.

Evaluating alternative methods of financing investments, such as lease or buy.

Using the benefits and costs to compare the relative merits of alternatives as an aid in making trade-offs between alternatives, recommending the cost-effective alternative, and in establishing or changing priorities.

* DoD Instruction 7041.3, p. 2

21



Two Clear-Cut RolesIn the Economic Analysis literature, two

individuals are referred to:

The analyst, and

The decisionmaker.

This division usually does not exist in practice; the roles of the analyst and the decisionmaker invariably overlap.



Role of the EconomicAnalyst

The function of the analyst is to present a schedule of alternatives and not to optimize in the sense of recommending the selection of a particular alternative.

A crucial role for the analyst is that of deciding which alternatives should be considered by the decisionmaker.

22



The Analytical StepsDefine the objective(s),Construct the alternatives,Measure the cost of the alternatives,Measure the effectiveness of the alternatives,Discuss the findings:

Quantitative results,Omitted factors,Risks and uncertainties,Judgement calls required of decisionmakers,Recommendations (two views).



Example of Equal Effectivenessand Equal Cost Comparisons*

EFFEFF

COST COST

z

x

v

z

x vz z

zz

x

x

x

x

v

v

v v

vx

System A

System B

System C

* Taken from DoD Instruction 5000.2, Part 4, Section E, 23 February 1991

z

23



The Presentationto the Decisionmaker

Under the assumption of a fixed budget constraint, the analyst would eliminate from consideration all systems but the one that yielded the greatest effectiveness.

If effectiveness were predetermined, the analyst would attempt to minimize cost.

Since neither scenario is likely, the analyst must assume that a range of levels of effectiveness and cost might be acceptable to the decisionmaker.



The “Best” Alternative

“The determination of a rule to tell you what is “best” is seldom solved even in the most ordinary and prosaic circumstances. Consumer Reportsdoes not tell you which washing machine to buy -- it merely lists dollar costs and some important characteristics and indicators for performance that you should consider. There may be quite a problem, however, in deciding which characteristics and indicators are significant to your choice. In the immensely more complicated problems of public policy, we can seldom do much better than take a similar approach, that is, display the various consequences, costs, and spillover associated with each alternative and their sensitivities to changes and uncertainties --the approach we have earlier called the ‘score-card’ approach -- and put the criterion problem squarely up to the decision-makers” (Quade*,pp. 221-22)

* E.S.Quade, Analysis for Public Decisions, 3rd Edition. New York: North-Holland, 1987.

24



Qualities of a GoodAnalysis

HonestAlternatives not arbitrarily suppressedQuantitative factors selected without bias

Calculations accurateOpen

ExplicitVerifiableSelf-correcting

Objective

UnbiasedNot self serving



Limitations of Analysis

Analysis is necessarily incomplete,Measures of effectiveness are inevitably approximate,No satisfactory way exists to predict the future,

Analysis falls short of scientific research,Analysis tends to ignore factors that cannot be reduced to

numbers, and tends to over-emphasize those that can,The approach may bias against the new in favor of the old,Cost-Effectiveness analysis appears biased against new

systems to replace those now in operation.

25



Some Common FallaciesUnderemphasis of problem formulation,

Inflexibility in the face of evidence,

Adherence to cherished beliefs,

Parochialism,

Communications failures,

Over-concentration on the model,

Excessive attention to detail.

Neglect of the question,

Incorrect use of the model,



Some Common Fallacies(Continued)

Disregard of the limitations,

Concentration on statistical uncertainty,

Inattention to uncertainties,

Use of side issues as criteria,

Substitution of the model for the decisionmaker,

Neglect of the subjective elements,

Failure to reappraise the work.

george mason university decisionsmason.gmu.edu/~aloerch/intro651.pdf · operations research*...

Documents