Project evaluation toolsDecisions on where to invest the company's resources to achieve a technological innovation have a major impact on the future competitiveness of the company. Therefore, trying to get involved in the right projects is worth an effort, both to avoid wasting the company's time and resources in meaningless activities, and to improve the chances of success.However, in a continuous improvement context, ideas for change and projects which need significant resources might be prioritised rather than selected, with a view to all projects eventually being addressed.In short, project evaluation aims at analysing research and development projects, or activities or ideas, for any or all of the following purposes: Getting an overall understanding of the project. Making priorities among a set of projects. Taking a decision about whether or not to proceed with a project. Monitoring projects, eg by following up the parameters analyzed when the project was selected. Terminating projects and evaluating the results obtained.Why and when are they used?To provide information to assess the value of a potential project with particular reference to the estimation of costs, resources and benefits, in order to take a decision about whether or not to proceed with the project. A secondary use is for monitoring and terminating projects.How do they work?A very long list of specific techniques have been developed over the years, and are still being developed and used today, both within companies and in the academic world. Most of those techniques can be described by means of some common elements which form the backbone structure of any project evaluation technique.Figure 1Main elements of Project Evaluation

Theinputs:the data on important aspects of the projects and the business environment which are needed to analyze them. The inputs will be assembled from various sources, and care should be taken to ensure its certainty, although some inputs will surely be very subjective. At the end of the day, an evaluation can only be as good as the data that go into it.Table 1Typical inputs to Project EvaluationTypical inputs

Technologicalthe technical activities which will have to be undertaken, maturity of technology, company's technological position

Internalpotential technical success, familiarity with the area of the project, role of individuals and of different functions within the organization

Financialexpected benefit, likely cost, both of project and consequent actions

Marketsize and attractiveness of the market, competitive position

Businessclarification of objectives, fit with company's strategy, level of top-management support, key success factors

Weighting: as certain data may be given more relevance than other (eg of market inputs compared with technical factors), in order to reflect the company's strategy or the company's particular views. The data is then processed to arrive at the outcomes. Many techniques includebalancingbetween projects, as the relative value of a project with respect to other projects is an important factor in situations of competition for limited resources.Portfolio manageportmenttechniques are specifically devoted to deal with this factor. The techniques might also incorporate how the results of the evaluation have to becommunicatedamong the interested parties, as well as the way to proceed in order to make the finaldecision. Eventually, no technique should be allowed to take decisions, as this is a management responsibility. Even when expert systems are used managers have the final word.Project evaluation usually assumes that there is a choice of projects in which to invest, but businesses may only be able to do R&D in collaborative projects or contract situations and if they are not asked to be involved in any projects there is no choice available in practice. Even in these situations, project evaluation is paramount to grasp what might be expected from pursuing each opportunity.With the apparent increase in networking, companies could move from one collaborative project to another as opportunities arise. Project evaluation in such situations actually means strategic direction (deciding which projects to accept). However accepting a project which is essentially a contract or order is quite different from selecting projects in which the company is making the financial investment. Companies are less likely to decline project opportunities and this could imply a deviation from the preferred strategy of the company. Even in these cases, project evaluation techniques can and should be used to assess the value and risk of those opportunities, in order to know whether the company's resources could be better used for other purposes.Nevertheless, all these different situations and contexts can have a specific answer within project evaluation techniques. The answer can be either to use different techniques for different situations, or, better, to introduce variations within the same technique (eg by using different inputs or different weighting depending on the context).Specific techniquesProject evaluation methods have evolved in response to changing needs, although 'old' techniques are still in use today. The earlier methods were based on financial assessment, and even now this forms the back-bone of most practical methods.One basic classification of all potential techniques might be: Techniques mainly or uniquely based on a financial assessment. Techniques mainly based on human judgment. Learning techniques, which explicitly take account of past experience in order to improve future decisions.Most of the techniques in practical use by industry incorporate a mixture of financial assessment and human judgment. A more detailed list of types of techniques is shown intable2.Table 2Type of Project Evaluation techniquesTechniquesShort description

Financial ratio methods Among the longest established, easily-applied methods Criticised as of limited accuracy The key is the ratio (financial benefit) /(cost) in which the estimations of benefits should be agreed by marketing

Cash flow analysis Requires the estimation of cash outflows and inflows It can be sofhisticated by considering discounting factors

Score index methods They generalize the simple Ratio Analysis by considering the probability for technical and market success The estimations of probability are usually made by experts Some of these methods include discount factors: some of the measures are Internal Rate of Return (IRR) and Return of Investment (ROI)

Mathematical methods They are based on the optimisation of allocation of R&D resources through mathematical programming The success of the methods depends strongly on how well the benefit is understood and how well the input data is converted into variables The algorithms tend to be customised and they can incorporate experience through expert systems

Matrix methods They employ subjective considerations for proper measurement of management information Matrix methods use correlation techniques in order to identify relationships that constitute the basis for decision-making

Check-lists Method that includes the reminders of the factors which are important in decision-making Simple and rapid way to assess a project with little effort Can be considered as starting point for more sophisticated methods as SWOT analysis

Relevance and decision trees It is an approach for structured thinking It requires a very clear objective or long term goal, establishing a clear differentiation between goals and means for achieving them Critical path analysis and decision trees are examples of these methods

Multicriteria & table methods Scoring procedures to incorporate judgments based on a number of criteria Various criteria may be used such as economic and financial factors and concepts or decision theory The scoring of criteria is usually complemented with the use of weight factors in order do distinguish theimportance of each criteria

QFD It is used in many fields of design and engineering It is based on the identification of customer requirements and the means for achieving those requirements Includes a scoring procedure with weighting of the factors

Experience based methods They are based on the analysis of conditions that are usually related to the success or failure of a project (quality of execution, synergies, etc.) Success or failure is predicted according to the answers to questions on those mentioned factors The inputs for those factors should come from members in different departments

Vision The vision is that of an individual (a Chief Executive or a product champion) defying conventional wisdom and bringing about a breakthrough It is common when information is scarce and it can be sustained by irrational methods When it works it has many virtues: speedy, incisive and changing the scene

Many techniques used today are totally or partially software based, which have some additional benefits in automating the process. In any case, the most important issue, for any method, is the managers' interpretation of the direct outcomes.There is no best technique. The extent to which different techniques for project evaluation can be used will depend upon the nature of the project, the information availability, the company's culture and several other factors. This is clear from the variety of techniques which are theoretically available and the extent to which they have been used in practice. In any case, no matter which technique is selected by a company, it should be implemented, and probably adapted, according to the particular needs of that company.ChecklistA checklist is a reminder of the factors (a list of factors) which are important in making a decision. Most useful criteria for evaluating any type of research or development project are essentially independent of the business field and the business strategy. These criteria include technical and commercial details, research and development realities, legal and financial factors, company targets and company strategy, etc.The requirements for the use of this technique are minimal, and the effort involved in using it is normally low. Another advantage of the technique is that it is very easily adaptable to the company's way of doing things. However, checklist can be a starting point for more sophisticated methods where the basic information can be used for better focus. One simple and useful example is a SWOT analysis, where projects are assessed for their Strengths, Weaknesses, Opportunities and Threats.Therefore, this technique can be developed further and the analysis interaction and feedback can be easily managed using simple information technology. Different ways to sophisticate the technique might be: To include some quantitative factors among the whole list of factors. To assign different weights to different factors. To develop a systematic way of arriving to an overall opinion on the project.A simple checklist could be one made up of a lists of factors which have been formed to affect the success of a project and which need to be considered at the outset. In the evaluation procedure a project is evaluated against each of these factors using a linear scale, usually 1 to 5 or 1 to 10 (seefigure2). The factors can be weighted to indicate their relative importance to the organisation.Figure 2Example of how to use a checklist

A value in this technique lies in its simplicity but by the appropriate choice of factors it is possible to ensure that the questions address, and are answered by, all functional areas. When used effectively this guarantees a useful discussion, an identification and clarification of areas of disagreement and a stronger commitment, by all involved, to the ultimate outcome.Checklist exampleTable3shows an example of a checklist, developed by the Industrial Research Institute, that, in princpiple, could be applied to any type of R&D activities: research, product development and process development.Table 3List of potential factors for project evaluationCorporate ObjectivesFits into the overall objectives and strategyCorporate image

Marketing and DistributionSize of potential marketCapability to market productMarket trend and growthCustomer acceptanceRelationship with existing marketsMarket shareMarket risk during development periodPricing trend, proprietary problem, etc.Complete product lineQuality improvementTiming of introduction of new productExpected product sales life

ManufacturingCost savingsCapability of manufacturing productFacility and equipment requirementsAvailability of raw materialManufacturing safety

Research and developmentLikelihood of technical successCostDevelopment timeCapability of available skillsAvailability of R&D resourcesAvailability of R&D facilitiesPatent statusCompatibility with other projects

Regulatory and legal FactorsPotential product liabilityRegulatory clearance

FinancialProfitabilityCapital investment requiredAnnual (or unit) costRate of return on investmentUnit pricePayout periodUtilization of assets, cost reduction and cash-flow

Using the list oftable3as a reference, any company should be able to develop its own checklist. This adaptation could take several forms: By sophisticating the checklist as suggested above. By choosing part of or adding different factors that better suit the company and its environment. For example, if the company strongly relies on external resources, either from other companies or from technological infrastructures, either for the development or for the production activities, this should probably be reflected in the list.Cash flow analysisIn its simplest form this approach requires the completion of a standard form which asks for estimates of the expected cash outflows and inflows for the project. This can be done on a yearly basis for longer term projects or on a shorter time scale for shorter term projects, even down to weekly.Table 4Cash outflows and inflowsCash OutflowCash InflowNet Cash Flow

1997C0B0B1-C1

1998C1B1B2-C2

1999C2B2B3-C3

2000C3B3B4-C4

When portrayed as a cash flow diagram a typical project might look as shown infigure3.Figure 3Diagram to plot the evolution of cash flow over time

A frequently used form of analysis applied to this data is to calculate the break even point. This is the point at which the cumulative net cash flow is equal to zero and hence the cash inflow has covered all the cash outflows (seefigure4).Figure 4Break even point

The reason for this measure being seen as important is that the shorter the time scale to break-even, or payback, the less risk there is that the environment will change dramatically and hence seriously affect the estimates used in the cash flow projections. However, the analysis completely ignores the likely returns after the break even point and therefore is not really a good indicator of the real potential of a project. In some cases, people choose to ignore the time value of money, although it is not difficult to handle this by including cost of capital payments on the net cash outflows as part of the costs.An easier way to do this, however, and one commonly encountered is to use the discounted cash flow approach. Here the cost of capital is incorporated directly into the calculation in a way which also takes into account the actual year of expenditure. The result is a figure which represents the present value of both the cost and benefit stream and ultimately a net present value which is the difference between the two. For example:

A net present value greater than zero therefore indicates that a project will produce a surplus after all costs are paid and is therefore likely to be worth undertaking. It is also possible to calculate a Benefit to Cost ratio which can provide another indication of value.In some cases it may not be easy to identify the true cost of capital and at the same time some people prefer to think in terms of rate of return on investment (ROI). In such circumstances it is not difficult to identify what has been termed the internal rate of return which is calculated by setting PVB = PVc and solving the resulting equation for i. This can be done very simply, as can the NPV calculation, using widely available computer programmes.Decision analysisThere is inevitably uncertainty surrounding the information which is used in any evaluation exercise. The extent of this will depend upon the type of project and the environment in which the organisation operates. One way of incorporating this into the evaluation procedure is through the use of probability estimates. This can be done in a number of different ways. For example, it is possible to adjust the calculations made by other techniques using estimates of the probability of commercial success and/or of technical success to create a rank index as followsRank Index = Pt PB (B-C)The potential variability in costs and benefits is very likely to be influenced by the time to completion and by the competitive environment. It is possible to take this into account using a simple 3 estimate approach. This requires that people are prepared to consider pessimistic and optimistic values as well as the most likely. This approach has been used in the planning of highly complex technical projects as is described in the project planning section.However these methods tend to ignore the multi stage process which most projects go through. A useful way of illustrating this is through the use of the decision tree approach.Figure5clearly illustrates that it is possible to stop a project at intermediate stages if progress is not up to expectations. In this case there are few stages and following each one there is an estimate showing the probability of success(s) or of failure(f).Figure 5Example of a decision tree

Watch out forTypical problems that might arise in the implementation and use of project evaluation are as follows: Thinking that subjective estimates are exact representations of future outcomes. People who are not committed to the process and who provide information without due thought to its likely accuracy and the effect of this on the selection decision. Unnecessary change of goals and resource conflicts with other projects which will lead to an inability to achieve the desired outcomes. The technique becoming a routine that the project has to suffer, being not applied with the intensity and effort required. Furthermore, any technique needs to be applied rigurously to avoid that those with a long experience in dealing with it can learn how to 'cheat' it. Changes occuring in the company context should have an impact in the application of project evaluation. Although there might be a learning process involved in order to master the technique, hence requiring it is applied consistently over a period of time, at the same time it needs to continuosly adapt itself to every new situation.

Further information The report: EIRMA Working Group 47, 'Evaluation of R&D projects'. It provides an overall explanation and analysis of project evaluation specific techniques, including both a method to assess which technique might be more suitable for a specific company and context, and specific cases of application in companies. The application of simple financial methods can be seen in many books either on financial management or on project evaluation.