A Multi-Objective, Multi-Criteria Approach for Evaluating IT Investments: Results from Two Case Studies

G. S. KearnsInformation Resources Management

JournalVol. 17, No. 1, pp. 37-62

Jan-Mar 2004

Outline Introduction The IT Investment Decision The Analytic Hierarchy Process The IT Investment Model An Information Systems Example Evidence From Two Case Studies Results of Investment Decisions Discussion Study Contributions Conclusions

Introduction (1/3)

A majority of CEOs IT investments were economically

infeasible Confidence about the future ability of

IT to provide strategic advantages Economic analysis of IT returns

relies on quantitative measures

Introduction (2/3) Traditional approach

Have not proven useful in the economic evaluation of IT-based investments

Single criteria techniques Discounted cash flow, cost/benefit analysis Bias towards the tangible benefits

IRR or net present value may ignore the ‘soft’ , qualitative benefits of IT applications

Strategic applications Require a method

Reliably measure all benefits in a consistent manner that is understood and supported by management

Introduction (3/3) Maximizing returns from IT investments requires a total

portfolio planning approach Can not be accomplished by valuing each investment

individually Mutually exclusive, mutual dependencies Should not be combined due to the total risk

Combined with integer programming and the Analytic Hierarchy Process

Support a multi-objective, multi-criteria approach Address several issues hindering the success of IT

investments The purpose of the paper

Demonstrate the MOMC approach to IT investment analysis The applicability of the proposed model using an illustrative

example of five information systems projects The results of two case studies in which the model was

successfully applied

The IT Investment Decision (1/3) There is little persuasive evidence

Investment in IT positively impacts the financial position of the firm or increases productivity

Measurement problem Time period between investment and realized

benefits The direction of causality is difficult to prove

The study examines a more direct method of influencing business performance Improving the quality of the IT investment

portfolio

The IT Investment Decision (2/3) Traditional financial accounting measures

Past evaluation of IT investments suffer from Isolation Difficulty in valuation of benefits Low explanatory power

Ignore basic investment tenets All financial measures are sensitive to the valuation of

benefits The approach assumes that each investment stands

on its own merits without regard to other investments Some investments generally have failing marks under

ROI and passing marks under net presents value Such as ERP

The IT Investment Decision (3/3) IT-related investments represent in excess of half

the annual capital expenditures for many firms An agreed-upon approach to measuring IT

investments does not exit Returns on IT investments have been

unsatisfactory The selection of IT-based investments

Produce the highest value for the firm Value must reflect a combination of both quantitative

and qualitative criteria A decision support process is needed that will

incorporate all relevant decision criteria

The Analytic Hierarchy Process AHP applications are numerous

Strategic planning, microcomputer selection, etc AHP combines with other techniques

multi-dimensional scaling and integer and linear programming

No prior illustrations of this use The MOMC is an effective measurement process

Rank alternative investments according to criteria Corporate strategies

The strict time constraints of the planning process Support consensus among a diverse group of individuals Reflect investment precedence or exclusivity constraints Incorporate both quantitative and qualitative criteria Be understood by management

The IT Investment Model (1/7) Corporate strategies used as

project ranking criteria The importance of linking IT strategies

to corporate strategies has been well known

Traditional discounted cash flow techniques lack linkage to corporate strategy

AHP facilitate specification of criteria based upon corporate strategies

The IT Investment Model (2/7) Level of difficulty

Include The flexibility of the measurement process in

reflecting changes Perform sensitivity analysis Produce viable alternative solutions Provide an explanatory trail

AHP methodology Use a paired-comparisons approach The criteria indicators represent typical

investment alternatives The sum of each criterion’s value becomes the

investment’s global score for final ranking

The IT Investment Model (3/7) Explanatory power

The most valuable feature of AHP A convenient framework for concise

representation Offer a formal, systematic, consistent

approach When combined with an integer optimizing model

The weights can readily be compared Managers are able to see into the process

The IT Investment Model (4/7) Creating consensus

AHP is highly effective in distilling information from groups and fostering consensus

By paired comparisons An important foundation for acceptance

AHP creates quantitative rankings Use a systematic approach to capture priorities Measures the consistency of the overall process

Cost, precedence, and exclusivity constraints Resource constraints limit the number of

investments Precluded investment may be due to overlap in

functionality or competition for non-cash resources Convert the multi-criteria resource allocation

problems into integer programming maximization-type problems

The IT Investment Model (5/7) Structuring

the AHP hierarchy

The IT Investment Model (6/7) AHP theory

An overall view of the complex relationships Help the decision-maker assess the

importance of the issue Support meaningful comparisons between

attributes Steps of using

Establish the decision hierarchy Create input data and make paired-comparisons

of the decision elements Estimate the relative weights of the decision

elements Aggregate the relative weights of decision

elements to arrive at a final set of ratings For the decision alternatives

The IT Investment Model (7/7) Incorporating quantitative and

qualitative investments In practice and theory

No consensus on the appropriate mechanism for ranking IT investments

Objective evaluation method Net present value, cost-benefit analysis, project

risk, value analysis, benchmarking, multiple criteria approach, DSS evaluation, aggregate scoring technique, and anecdotal evidence

Subjective method Attitude surveys and the opinions of users and

analysts

An Information Systems Example (1/4)

A simple hierarchy illustration Includes both financial and non-financial criteria

Compare on the basis of corporate strategies Investment risk Revenue enhancing Operating efficiency Customer satisfaction Market growth

An Information Systems Example (2/4) Steps

Define the decision hierarchy The goal is to rank the decision alternatives

Input the data Expert ChoiceTM

The input data are manipulated using matrix algebra to produce the relative weights or priorities

Aggregation of all weights to produce a vector of composite relative weights between the criteria and the alternatives

An Information Systems Example (3/4)

An Information Systems Example (4/4) Optimizing using integer programming

Maximize the AHP priority weights with the resource constraint

The optimal solution is (1,1,0,1,0) The objective function value is equal to 0.709

Higher values signify higher overall returns for the IT investments

Evidence From Two Case Studies (1/7) Research methodology

Two case studies Use the IT investment model Contextual conditions could impact the outcomes

The goals Ascertain the efficacy of the proposed ranking mechanism Collect and report the attitudes, behaviors, and perceptions

Results were reviewed by the CIOs with minor corrections and revisions

Use multiple cases Allow the investigator to replicate the results and improves generali

zability The study will show

Management involvement is necessary Organizational structure affects the success of the ranking process

Hot and Lukewarm

Evidence From Two Case Studies (2/7)

Case study background of companies Two U.S. utility companies

North-central region and southern region Similarities

Generators of electricity, retail and wholesale markets, sold surplus power, and controlled their own transmission and distribution systems

Both had CIOs committed to IT planning

Evidence From Two Case Studies (3/7)

Hot Smaller company under greater competitive pressure Highly participative management structure with younger

management Previous experience in non-regulated industries

Highly committed to planning and the strategic use of IT Lukewarm

Relatively secure markets Issues of deregulation

Shortly put markets under competitive pressures With traces of political rivalry Top management was without experience outside their

field CEO and CIO had previous experience in non-regulated

industries Committed to planning and increasing returns on IT

investments

Evidence From Two Case Studies (4/7) IT planning and evaluation - Hot

Interest in IT planning and using IT strategically

Want a system Satisfy all areas of management

Ask IT management for assistance in identifying technologies

That might allow revision of business processes to improve efficiencies and customer service

A combination of project evaluation tools ROI, payback, and a corporate model Useful but probably unreliable

Evidence From Two Case Studies (5/7) IT planning and evaluation - Lukewarm

Delegate all IT planning to the CIO Complain about the time and cost of implementing

systems IT steering committee

Composed of several senior managers Rely heavily on the opinion of the CIO

The IT plan contained A wish list of applications that continually changes with

the political climate Use a cost/benefit and payback approach Selection of projects depends on

How well managers could creatively assign dollars to benefits

Evidence From Two Case Studies (6/7) The Hot results

The decision criteria and sub-criteria Originally developed by a team of IT managers Later modified by other members of management

The participators were familiarity with AHP prior to the session

Use a modified Delphi technique to decide the weights IT management played an impartial advisory role

The initial analysis was completed Working with managers from finance, engineering, and

marketing Use both the AHP and integer programming models

Disadvantage Total time involved in making the paired-comparisons and

estimating other parameters Advantage

Their understanding of the process would help to make future estimates easier and cut the time requirement

Select five IT investments with a capital requirement in excess of $18.5 million

Evidence From Two Case Studies (7/7) The Lukewarm results

Expected to benefit from the results of the Hot experience Partly implemented and with less success Less efficient session

A cross-functional management team Review and refine the comparisons after individual discussions with

managers The team would have final authority The CEO supported the process but didn’t participate directly

On the advice of the team The investments identified as strategic, high cost, and high risk were

evaluated 26 investments were analyzed

Many were overlapping and mutually exclusive 8 investments were selected with a capital cost in excess of $34 million

Problems Many managers continually requested revisions of the management

team Use a spreadsheet program Perform a modified ROI analysis on the selected projects

IT managers felt The direction was an improvement

Results of Investment Decisions (1/5) Acceptance

Managers form both companies Enthusiasm The documentation for the methodology improved their

understanding and made it easier for new managers to grasp

Hot The internal environment and organizational structure

are more conducive to acceptance of new processes Lukewarm

Acceptance of the methodology had removed a major burden from IT planning

No longer incurred the wrath of managers who had not been funded

This supports One of the benefits of the MOMC approach is the

balancing of conflicting objectives of different users and stakeholders

Results of Investment Decisions (2/5) Status of the IT investments selected

There was no immediate pressure to cut capital investments

Hot Lower earnings-per-share Delay one project to conserve cash and deploy

resources to the other projects in order to realize the benefits more quickly

All of the projects were on or under schedule and under budget

Lukewarm Benefit from reduced political tensions Most of the projects were on schedule and within

budget The delayed project had suffered from a political tug-of-

war about infrastructure issues IT projects were an outstanding success

Results of Investment Decisions (3/5) Status of selection process

Hot Managers were continuing to modify and enhance the model They wanted to be able to analyze individual investments on a

stand-alone basis The use of a program to quickly generate an initial set of

paired-comparisons Two strategic categories emphasized on valuation of intangible

benefits Tow over $1 million categories emphasized on risk analysis

Lukewarm The CEO had to contend with several presidents of the

operating companies Less time to focus on IT

Time period was not sufficient Little had been accomplished towards improving the process,

primarily documentation of the process and the training of new managers

The CIO was confident The next round of investment proposals would be handled more

expeditiously

Results of Investment Decisions (4/5) Generalizable findings

In one firm The CEO had greater knowledge of IS The CEO worked closely with the CIO and other managers followed t

he lead In the other firm

The CEO had superficial knowledge The CEO did not work closely with the CIO

Hot had capitalized on the new process to insure success and reduce the time requirement on management

By extending the model and adding administrative controls Lukewarm accomplished less

Managers in both firms had an improved attitude The new process improved the quality of information available t

o measure investment proposals, increased the involvement of managers, and added credibility to the final results

An investment’s potential return may be reduced Because of implementation problems The inability to control quality during software system development

Results of Investment Decisions (5/5) Summary

Discussion (1/2) Benefits

The ability of the model to handle a large number of criteria

The ability to represent both tangible and intangible items

The ability to model exclusivity and dependency of investments

The ability to quickly reflect revisions The explanatory power of the model The support for group decision-making

Limitations The lack of a financial measure of profitability The overall time requirements for management The problem of valuing intangibles, although

ameliorated, remained

Discussion (2/2)

Study Contributions (1/2) Provide a tested process for prioritizat

ion and selection of IT investments Identify benefits and limitations inher

ent within the process Identify facilitators and inhibitors and

generalizable findings to the approach

Assist the introduction of the process

Study Contributions (2/2) Suggestions for future research

Further case studies Suggested from different industries Provide more insights into the completeness of

the approach Examine the impact of contextual variables on

the success of the IT investment model The balancing of investment risk was not

tested in this study The relationship between process credibility

and subsequent development and implementation remains unresolved

Conclusions The MOMC approach merits attention as

a investment selection and ranking tool Utilize AHP and integer programming Improve the IT investment process

Strictly quantitative approaches have not yielded satisfactory results

Subjective approaches lack explanatory power and can not be easily adjusted to reflect new knowledge

Basing selection criteria on business strategies ensures the alignment of IT investments with these strategies