Measuring Efficiency

CRJS 4466EA

Introduction

• It is very important to understand the effectiveness of a program, as we have discovered in all earlier chapters

• But, it is equally important to be able to inform stakeholders about program costs, and more importantly, how program outcomes compare to program costs

Cost-benefit analysis

• Expressed in monetary terms• E.g. “a cost-benefit analysis of a program to

reduce cigarette smoking would focus on the difference between the dollars expended on the anti-smoking program and the dollar savings from reduced medical care for smoking-related diseases, days lost from work, and so on” (Rossi, Freeman and Lipsey, 1999)

Cost-effectiveness analysis

• Outcomes are expressed in substantive terms

• E.g. a cost-effectiveness analysis of the same smoking program “would estimate the dollars that had to be expended to convert each smoker into a non-smoker” (Rossi, Freeman and Lipsey, 1999)

Acceptance of efficiency assessment

• Derives from business applications• Very acceptable in human services to assess

efficiency, but concerns remain about acceptable procedures– Unfamiliarity with procedures

– Debate regarding the appropriateness of applying monetary values to social program outcomes

– An unwillingness to let go of what may not be working

Timing of efficiency analysis

• Most commonly undertaken at the planning stage or post program

• Before analysis is based on estimates of the future– Presumes a given magnitude of positive net impact– Not based on empirical information therefore over and

under-estimates can occur– Most important when the program is unlikely to be

abandoned once implemented– Important when resource commitments are anticipated

to be high

After analysis

• Most commonly efficiency analysis occurs after outcome evaluation completion

• “in comparative terms, the issue is to determine the differential “payoff” of one program versus another – for example, comparing the reduction in arrest rates for drunken driving brought about by an educational program with that of a program that pays for taxis to take people home after they have imbibed too much. In ex post analyses, estimates of costs and outcomes are based on studies of the types described in previous chapters on program monitoring and impact evaluations” (Rossi, Freeman and Lipsey, 1999)

The concepts of cost-benefit and cost-effectiveness analyses

• Cost-benefit analyses– Requires tangible and intangible estimates of program costs, both

direct and indirect– Once the benefits and costs are specified, they are translated into a

common measure (usually a monetary unit)– A particular economic perspective is adopted– Controversy around procedures used to convert inputs and outputs

into monetary values– “clearly, the assumptions underlying the definitions of the

measures of costs and benefits strongly influence the resulting conclusions. Consequently, the analyst is required, at the very least, to state the basis for the assumptions that underlie the analysis” (Rossi, Freeman and Lipsey, 1999)

• Cost-effectiveness analyses– Less controversial– Merely requires monetizing the program’s costs– Benefits are expressed in outcome units– “an ex ante cost-effectiveness analysis allows potential

programs to be compared and ranked according to the magnitudes of their effects relative to their estimated costs” (Rossi, Freeman and Lipsey, 1999)

– “…inputs and outputs – replace, to a considerable extent, estimates and assumptions” (Rossi, Freeman and Lipsey, 1999)

The uses of efficiency analyses

• Can’t be done when outcomes are unknown or unestimable

• Senseless for outcome evaluations that show no significant net effects

• Senseless for programs in progress that do not yet have program impact information

Methodology of cost-benefit analysis

• “benefits and costs must be defined from a single perspective because mixing points of view results in confused specifications and overlapping or double counting”(Rossi, Freeman and Lipsey, 1999)

• Accounting perspectives– Individual target

– Program sponsor

– Community aggregates (communal)

Individual – target accounting perspective

• Takes the point of view of the units that are the program target

• Can provide for the highest benefit-to-cost results because this group rarely bears the cost of the intervention

Program sponsor accounting perspective

• Takes the point of view of the funding source in valuing benefits and specifying cost factors

• “analysis from this perspective is designed to reveal what the sponsor will pay to provide a program and what benefits (or “profits”) should accrue to the sponsor” (Rossi, Freeman and Lipsey, 1999)

• Most appropriate perspective when choices need to be made by the sponsor between alternative programs in the face of fixed budget

The communal accounting perspective

• Takes the perspective of the community or society as a whole (usually in terms of total income)

• Most comprehensive approach but also most complex

• Special efforts are made to account for secondary or indirect project effects – effects on groups that are not directly involved in the intervention (Rossi, Freeman and Lipsey, 1999)

• “Can also be expanded to include equity considerations, that is the distributional effects of programs among different subgroups” (Rossi, Freeman and Lipsey, 1999)

• Communal perspectives include costs and benefits from the other types of perspectives but monetize and value items differently

• But communal perspectives include opportunity costs

• Generally the communal perspective is the most politically neutral

Case Studies

• Exhibit 11-F: Components of Cost-Benefit Analyses from Different Perspectives

• Exhibit 11-G: Hypothetical Example of Cost-Benefit Calculation from Different Accounting Perspectives (Rossi, Freeman and Lipsey, 1999)

Measuring costs and benefits

• Potential problems:– Identifying and measuring all program costs and

benefits

– Expressing all costs and benefits in terms of a common denominator (translating them into money values)

– Social programs do not often produce results that can be valued accurately by means of market prices

– Sometimes cost-effectiveness analysis should be used because with this form of analysis benefits needn’t be valued

Monetizing outcomes

• Money measurements

– Estimate of direct monetary benefits

• Market valuation

• Econometric estimation

• Hypothetical questions

• Observing political choices

• Opportunity costs

• Secondary effects (externalities)

• Shadow prices

– Derived prices meant to reflect their true benefit and cost

Comparing costs to benefits

• Simple method (net benefit)– Subtract costs from benefits

• Ratio of benefits to costs– More difficult to interpret