Risk AnalysisRisk Analysis

Risk ManagementRisk Management

References: T. Lister, “Risk Management is Project Management

for Adults”, IEEE Software, May/June 1997, pp 20–22. M.J. Carr, “Risk management May Not Be for

Everyone”, IEEE Software, May/June 1997, pp 21–24. B. Kitchenham and S. Linkman, “Estimates,

Uncertainty and Risk”, IEEE Software, May/June 1997, pp 21–24.

R.C. Williams, J.A. Walker and A.J. Dorofee, “Putting Risk Management into Practice”, IEEE Software, May/June 1997, pp 75–82.

E.H. Conrow and P.S. Shishido, “Implementing Risk Management on Software Intensive Projects”, IEEE Software, May/June 1997, pp 83–89.

References: T. Lister, “Risk Management is Project Management

for Adults”, IEEE Software, May/June 1997, pp 20–22. M.J. Carr, “Risk management May Not Be for

Everyone”, IEEE Software, May/June 1997, pp 21–24. B. Kitchenham and S. Linkman, “Estimates,

Uncertainty and Risk”, IEEE Software, May/June 1997, pp 21–24.

R.C. Williams, J.A. Walker and A.J. Dorofee, “Putting Risk Management into Practice”, IEEE Software, May/June 1997, pp 75–82.

E.H. Conrow and P.S. Shishido, “Implementing Risk Management on Software Intensive Projects”, IEEE Software, May/June 1997, pp 83–89.

Risk ManagementRisk Management

Risk management is a critical aspect of successful software engineering where the aim to to deliver software on time and in budget (with all the functionality you promised).

But how can you know what will go wrong before you start the project?

How does it affect your cost estimates? How does it affect your delivery schedule? How do you manage your exposure to risk?

Risk management is a critical aspect of successful software engineering where the aim to to deliver software on time and in budget (with all the functionality you promised).

But how can you know what will go wrong before you start the project?

How does it affect your cost estimates? How does it affect your delivery schedule? How do you manage your exposure to risk?

ChallengesChallenges

Completed projects have three state: on-time, late and early (an empty set).

A survey of 8,380 commercial software-intensive projects found that: 53% of the projects were “challenged”.

Over budget Behind schedule Fewer features and functions than originally

specified. 31% of projects were cancelled.

Completed projects have three state: on-time, late and early (an empty set).

A survey of 8,380 commercial software-intensive projects found that: 53% of the projects were “challenged”.

Over budget Behind schedule Fewer features and functions than originally

specified. 31% of projects were cancelled.

ChallengesChallenges

On average, these projects had cost increases of 189% and schedule slippage of 222% versus the original estimates at the time they were completed or cancelled!

Of the completed projects, an average of only 61% of the originally specified features were delivered!

On average, these projects had cost increases of 189% and schedule slippage of 222% versus the original estimates at the time they were completed or cancelled!

Of the completed projects, an average of only 61% of the originally specified features were delivered!

What is Going Wrong?What is Going Wrong?

Are these problems the result of poor management? Inexperience? Negligence? Stupidity?

It is the result of poor risk management and costing strategies.

In order to fix the problem, we have to know what risk is and how to manage it effectively.

Are these problems the result of poor management? Inexperience? Negligence? Stupidity?

It is the result of poor risk management and costing strategies.

In order to fix the problem, we have to know what risk is and how to manage it effectively.

Defining RiskDefining Risk

Risk is the probability of failing to achieve particular objectives: Cost Performance Schedule

Risk is more – it is also the consequence of failing to achieve those objectives.

There are various techniques to estimate the probability of the risk occurring, but true risk values can almost never be mathematically computed.

Risk is the probability of failing to achieve particular objectives: Cost Performance Schedule

Risk is more – it is also the consequence of failing to achieve those objectives.

There are various techniques to estimate the probability of the risk occurring, but true risk values can almost never be mathematically computed.

Sources of RiskSources of Risk

There are numerous sources of risk within a project. Common risk group and factors are discussed below.

Project level Excessive, immature, unrealistic, or unstable

requirements. Lack of user involvement. Underestimation of project complexity or

dynamic nature. Project attributes

Performance shortfalls (includes errors and quality).

Unrealistic cost or schedule (estimates and/or allocated amounts).

There are numerous sources of risk within a project. Common risk group and factors are discussed below.

Project level Excessive, immature, unrealistic, or unstable

requirements. Lack of user involvement. Underestimation of project complexity or

dynamic nature. Project attributes

Performance shortfalls (includes errors and quality).

Unrealistic cost or schedule (estimates and/or allocated amounts).

Sources of RiskSources of Risk

Management Ineffective project management (multiple levels

possible). Engineering

Ineffective integration, assemble and test, quality, control, speciality engineering, or systems engineering (multiple levels possible).

Unanticipated difficulties associated with the user interface.

Management Ineffective project management (multiple levels

possible). Engineering

Ineffective integration, assemble and test, quality, control, speciality engineering, or systems engineering (multiple levels possible).

Unanticipated difficulties associated with the user interface.

Sources of RiskSources of Risk

Work environment Immature or untried design, process, or

technologies selected. Inadequate work plans or configuration control. Inappropriate methods or tool selection or

inaccurate metrics. Poor training.

Work environment Immature or untried design, process, or

technologies selected. Inadequate work plans or configuration control. Inappropriate methods or tool selection or

inaccurate metrics. Poor training.

Sources of RiskSources of Risk

Other Inadequate or excessive documentation or

review process. Legal or contractual issues (such as litigation,

malpractice, ownership, intellectual property) Obsolescence (includes excessive schedule

length). Unanticipated difficulties with subcontracted

items. Unanticipated maintenance and/or support costs.

Other Inadequate or excessive documentation or

review process. Legal or contractual issues (such as litigation,

malpractice, ownership, intellectual property) Obsolescence (includes excessive schedule

length). Unanticipated difficulties with subcontracted

items. Unanticipated maintenance and/or support costs.

Sources of RiskSources of Risk

But wait, there’s more… For moderate and high complexity software systems, there

are a number of issues that can significantly increase costs and result in schedule slippage.

Performance-dominated requirements generation (by either buyer or seller or both) results in an early focus on performance without due consideration for: Budget – typically inadequate for desired performance

level. Design – performance demands push the design to near

the feasible limit of achievable performance. Usually leads to cost blow-out.

Major decisions made too early – lack of understanding of relationship between cost, schedule and performance leads to increased costs and schedule slippage.

But wait, there’s more… For moderate and high complexity software systems, there

are a number of issues that can significantly increase costs and result in schedule slippage.

Performance-dominated requirements generation (by either buyer or seller or both) results in an early focus on performance without due consideration for: Budget – typically inadequate for desired performance

level. Design – performance demands push the design to near

the feasible limit of achievable performance. Usually leads to cost blow-out.

Major decisions made too early – lack of understanding of relationship between cost, schedule and performance leads to increased costs and schedule slippage.

Sources of RiskSources of Risk

Excessive optimism in assessing the limits of performance achievable for a given budget and time frame is likely to result in cost and schedule over-run.

These are all risks that may result in the project being challenged.

US Defence work mandates the risk assessment take place and that it follow a directive (DoD directive 5000.1) and a standard (EIA/IEEE J-STD-016). However, defence projects still run over time and budget, ie., their risk assessment and management is deficient!

Excessive optimism in assessing the limits of performance achievable for a given budget and time frame is likely to result in cost and schedule over-run.

These are all risks that may result in the project being challenged.

US Defence work mandates the risk assessment take place and that it follow a directive (DoD directive 5000.1) and a standard (EIA/IEEE J-STD-016). However, defence projects still run over time and budget, ie., their risk assessment and management is deficient!

Common DeficienciesCommon Deficiencies

There are four main risk management deficiencies.1. Buyer and seller often have risk management

processes that are weakly structured or ad hoc. May exist on paper, but not followed. No clear mechanism to mitigate risk. The problem is that there is not a single set of

guidelines as to how the risk management process is to be implemented.

There are four main risk management deficiencies.1. Buyer and seller often have risk management

processes that are weakly structured or ad hoc. May exist on paper, but not followed. No clear mechanism to mitigate risk. The problem is that there is not a single set of

guidelines as to how the risk management process is to be implemented.

Common DeficienciesCommon Deficiencies

2. Risk assessment is often too subjective and not adequately documented. A weak risk assessment methodology can

introduce considerable doubt as to the accuracy and value of the results.

Mathematical operations on uncalibrated risk assessment scores is meaningless.

Lack of detail makes the nature of the risk incomprehensible.

Buyer and seller may use different risk assessment methodologies which makes comparing results difficult or impossible.

2. Risk assessment is often too subjective and not adequately documented. A weak risk assessment methodology can

introduce considerable doubt as to the accuracy and value of the results.

Mathematical operations on uncalibrated risk assessment scores is meaningless.

Lack of detail makes the nature of the risk incomprehensible.

Buyer and seller may use different risk assessment methodologies which makes comparing results difficult or impossible.

Common DeficienciesCommon Deficiencies

3. Risk assessment processes generally focus on the probability associated with a specific event without consideration for the consequences. Recall that risk is a combination of an events

probability and its consequences. Therefore, there is a need to track both factors

over time.4. Risk assessment and risk handling plans often

unlinked and risks are not tracked fully.

3. Risk assessment processes generally focus on the probability associated with a specific event without consideration for the consequences. Recall that risk is a combination of an events

probability and its consequences. Therefore, there is a need to track both factors

over time.4. Risk assessment and risk handling plans often

unlinked and risks are not tracked fully.

ExampleExample

TRW project – ≥1,000,000 LOC (Ada) Command and Control System.

Key challenges: Tight performance demands. Other contractors developed external interfaces

concurrently. Significant user interaction. Evolving requirements. Fixed price contract.

A flexible architecture and design was needed. Had to uncover potential risk issues early and develop a process that allowed changes to be made easily.

TRW project – ≥1,000,000 LOC (Ada) Command and Control System.

Key challenges: Tight performance demands. Other contractors developed external interfaces

concurrently. Significant user interaction. Evolving requirements. Fixed price contract.

A flexible architecture and design was needed. Had to uncover potential risk issues early and develop a process that allowed changes to be made easily.

ExampleExample

The result was a modular design using measurable metrics built on a foundation of management visibility.

Building blocks: Flexible and easily modifiable software architecture in

order to deal with the evolving requirements. SE process that allowed changes during the

development cycle. Suitable environment and tools. Trained personnel.

Risks were identified and tracked. Whenever requirements changes occurred, the risks associated with the changes were considered, existing risks re-evaluated, and the impact on performance, cost and schedule analysed.

The result was a modular design using measurable metrics built on a foundation of management visibility.

Building blocks: Flexible and easily modifiable software architecture in

order to deal with the evolving requirements. SE process that allowed changes during the

development cycle. Suitable environment and tools. Trained personnel.

Risks were identified and tracked. Whenever requirements changes occurred, the risks associated with the changes were considered, existing risks re-evaluated, and the impact on performance, cost and schedule analysed.

Risk Management OverviewRisk Management Overview

Established a risk review board (RRB). Led by project manager. Met monthly. Included representatives from each of the

functional and support areas (software, hardware, test, quality assurance, configuration management, ...).

Process was iterative. Documented feedback to both project management

and the customer.

Established a risk review board (RRB). Led by project manager. Met monthly. Included representatives from each of the

functional and support areas (software, hardware, test, quality assurance, configuration management, ...).

Process was iterative. Documented feedback to both project management

and the customer.

RiskManagement

RiskAssessment

RiskHandling

RiskMonitoring

RiskPlanning

RiskAnalysis

RiskIdentification

Feedback

Risk ManagementRisk Management

All risks are documented. This includes a short description of the risk type (schedule, cost, technical); severity (low, moderate, high); risk mitigation plan; and status of the risk mitigation activity.

Metrics must be used to measure the effect of the risk and to identify if it is indeed occurring.

Everyone on the project is encouraged to identify risks during any management or technical meeting. Do not shoot the messenger!

Once a risk is identified and a risk mitigation plan created, it is assigned to a responsible person. Risks are monitored and plans updated until the risk is reduced to an acceptable level.

Read the article by Conrow and Shishido to find out how specific risk issues were dealt with on the TRW project.

All risks are documented. This includes a short description of the risk type (schedule, cost, technical); severity (low, moderate, high); risk mitigation plan; and status of the risk mitigation activity.

Metrics must be used to measure the effect of the risk and to identify if it is indeed occurring.

Everyone on the project is encouraged to identify risks during any management or technical meeting. Do not shoot the messenger!

Once a risk is identified and a risk mitigation plan created, it is assigned to a responsible person. Risks are monitored and plans updated until the risk is reduced to an acceptable level.

Read the article by Conrow and Shishido to find out how specific risk issues were dealt with on the TRW project.

Putting Risk Management Into Practice

Putting Risk Management Into Practice

It is all well reading and talking about risk management, but what happens when you are responsible for it? Where can you find a tried and tested risk management

plan? How do you put what you read into practice? How do you identify the risks you are exposed to? Which strategies work well? Which don’t?

Risk management must be a continuous activity throughout the project.

Everyone must be involved in risk management and mitigation if it is to work.

The customer and the supplier must continuously and independently manage their own lists of risks. Some risks may be jointly managed.

It is all well reading and talking about risk management, but what happens when you are responsible for it? Where can you find a tried and tested risk management

plan? How do you put what you read into practice? How do you identify the risks you are exposed to? Which strategies work well? Which don’t?

Risk management must be a continuous activity throughout the project.

Everyone must be involved in risk management and mitigation if it is to work.

The customer and the supplier must continuously and independently manage their own lists of risks. Some risks may be jointly managed.

SponsorshipSponsorship

For risk management to succeed, there must be a commitment from a suitably powerful individual – typically the customer project manager.

The sponsor must be visibly involved in the project’s life cycle.

Often all the customer needs to do is ask for the latest status on identified risks on a regular basis.

For risk management to succeed, there must be a commitment from a suitably powerful individual – typically the customer project manager.

The sponsor must be visibly involved in the project’s life cycle.

Often all the customer needs to do is ask for the latest status on identified risks on a regular basis.

Focus ActivitiesFocus Activities

Generate a list of risk statements, evaluate them for probability and impact, classify them and rank them in priority order. Develop mitigation strategies for them.

Remember that there is little point spending having 5 people spend 2 days developing a mitigation strategy for an identified risk that has a low probability, low priority and a low impact (ie., less that 10 days delay to the schedule in this case). Don’t spend more time and money managing the risk than what the cost of the risk actually is!

Document all risks – remember, it’s harder to ignore if it’s written down.

Generate a list of risk statements, evaluate them for probability and impact, classify them and rank them in priority order. Develop mitigation strategies for them.

Remember that there is little point spending having 5 people spend 2 days developing a mitigation strategy for an identified risk that has a low probability, low priority and a low impact (ie., less that 10 days delay to the schedule in this case). Don’t spend more time and money managing the risk than what the cost of the risk actually is!

Document all risks – remember, it’s harder to ignore if it’s written down.

HintsHints

Document all risks. Be precise – vague wording is ineffective.

Evaluate the likelihood of the risk (estimate satisfactory) Identify the impact of risk (its consequences).

Quick and dirty estimates (high, medium, low) is typically sufficient.

Assign the risk to an individual. Make all risks visible to everyone. Track and monitor all risks. Classify or group risks. Different people will see different

aspects of the same risk (project manager, configuration manager, software engineer, …).

Prioritise risks and re-evaluate periodically.

Document all risks. Be precise – vague wording is ineffective.

Evaluate the likelihood of the risk (estimate satisfactory) Identify the impact of risk (its consequences).

Quick and dirty estimates (high, medium, low) is typically sufficient.

Assign the risk to an individual. Make all risks visible to everyone. Track and monitor all risks. Classify or group risks. Different people will see different

aspects of the same risk (project manager, configuration manager, software engineer, …).

Prioritise risks and re-evaluate periodically.

HintsHints

You can’t fix everything! Not every risk can be mitigated; every condition has a number of possible outcomes (consequences) and each will create a risk that could be mitigated. Identify the critical risks and watch them for significant changes.

Identify the risks that can be accepted – you could live with them if they occurred.

Identify the risks that are better managed by someone else.

Track risks and monitor them – how else do you know when they are gone? Metrics (measures), indicators and triggers can help you watch for significant changes.

You can’t fix everything! Not every risk can be mitigated; every condition has a number of possible outcomes (consequences) and each will create a risk that could be mitigated. Identify the critical risks and watch them for significant changes.

Identify the risks that can be accepted – you could live with them if they occurred.

Identify the risks that are better managed by someone else.

Track risks and monitor them – how else do you know when they are gone? Metrics (measures), indicators and triggers can help you watch for significant changes.

HintsHints

Are you sure you can close the risk? Closing a risk because you took an action, any action, is not controlling the risk, it is abdicating to it.

Controlling a risk means: Altering mitigation strategies when they become

ineffective. Taking action on a risk that becomes important

enough to require mitigation. Taking a pre-planned contingency action. Dropping to a watch-only mode at some specific

threshold. Closing the risk when it no longer exists. Check to make sure that your actions reduce or

manage the risk.

Are you sure you can close the risk? Closing a risk because you took an action, any action, is not controlling the risk, it is abdicating to it.

Controlling a risk means: Altering mitigation strategies when they become

ineffective. Taking action on a risk that becomes important

enough to require mitigation. Taking a pre-planned contingency action. Dropping to a watch-only mode at some specific

threshold. Closing the risk when it no longer exists. Check to make sure that your actions reduce or

manage the risk.

EstimatesEstimates

When bidding for contracts we need to make several estimates in order to determine schedule, cost, etc.

Estimate uncertainty occurs because an estimate is a probabilistic assessment of a future condition.

Estimators produce an estimate based on an estimation model. This model itself can be a source of errors.

When bidding for contracts we need to make several estimates in order to determine schedule, cost, etc.

Estimate uncertainty occurs because an estimate is a probabilistic assessment of a future condition.

Estimators produce an estimate based on an estimation model. This model itself can be a source of errors.

Measurement ErrorMeasurement Error

This occurs if some of the input values into a model have inherent accuracy limitations.

For example, function points are assumed to have a 12% inaccuracy. Therefore a project with an estimated 1,000 function points, could end up with anywhere between 880 and 1,120 function points in reality.

Applying a model of 0.2 person days per function point means your estimate will have a range of 176 to 224 person days, with a most likely value of 200 person days.

This occurs if some of the input values into a model have inherent accuracy limitations.

For example, function points are assumed to have a 12% inaccuracy. Therefore a project with an estimated 1,000 function points, could end up with anywhere between 880 and 1,120 function points in reality.

Applying a model of 0.2 person days per function point means your estimate will have a range of 176 to 224 person days, with a most likely value of 200 person days.

Model ErrorModel Error

All empirical models are abstractions of reality. No model includes all factors that affects the final outcome. Factors not explicitly included in the model contribute to model error.

For example, a model of 0.2 person days per function point is usually obtained from results previously observed. It is unlikely that future projects will achieve exactly the same ratio, but it is expected that the model is “all right on average”.

If you have an estimation model with an inherent 20% inaccuracy and your project is 1,000 function points, then your estimate is likely to be between 160 and 240 person days.

All empirical models are abstractions of reality. No model includes all factors that affects the final outcome. Factors not explicitly included in the model contribute to model error.

For example, a model of 0.2 person days per function point is usually obtained from results previously observed. It is unlikely that future projects will achieve exactly the same ratio, but it is expected that the model is “all right on average”.

If you have an estimation model with an inherent 20% inaccuracy and your project is 1,000 function points, then your estimate is likely to be between 160 and 240 person days.

Errors are AdditiveErrors are Additive

Estimate error and model error are additive. The real range of values is between 140.8 and 268.8 person days with a most likely value of 200 person days.

Using the mean magnitude of the relative error to assess model accuracy can be misleading. There is no a priori reason for underestimation and overestimation to be of equal size.

Underestimation is naturally bounded: the minimum effort required for an activity is zero.

An overestimate is not naturally bounded. A more likely probability distribution for the effort

estimate is one based on a Gamma Distribution.

Estimate error and model error are additive. The real range of values is between 140.8 and 268.8 person days with a most likely value of 200 person days.

Using the mean magnitude of the relative error to assess model accuracy can be misleading. There is no a priori reason for underestimation and overestimation to be of equal size.

Underestimation is naturally bounded: the minimum effort required for an activity is zero.

An overestimate is not naturally bounded. A more likely probability distribution for the effort

estimate is one based on a Gamma Distribution.

Gamma DistributionGamma Distribution

f(x) = (x)–1exp(–x)/() where x ≥0, and are both real and

positive, and () is the Gamma function which is a generalisation of the factorial function. Often = 2 and = 19.

f(x) = (x)–1exp(–x)/() where x ≥0, and are both real and

positive, and () is the Gamma function which is a generalisation of the factorial function. Often = 2 and = 19.

Gamma DistributionGamma Distribution

Assumption ErrorAssumption Error

Occurs when we make incorrect assumptions. For example, the estimate of 1,000 function points id based on the assumption that we have correctly identified all of the customers requirements.

If you can identify the assumptions, you can determine the risk associated with it.

If there is a 10% probability that the requirements complexity has been underestimated, and if it has you estimate that another 100 function points will be required, you obtain the estimates risk exposure: Risk exposure = (E2 – E1) P2 E1 = effort of original assumption. E2 = effort if alternative assumption is true. P2 = probability that the alternative assumption is true.

Occurs when we make incorrect assumptions. For example, the estimate of 1,000 function points id based on the assumption that we have correctly identified all of the customers requirements.

If you can identify the assumptions, you can determine the risk associated with it.

If there is a 10% probability that the requirements complexity has been underestimated, and if it has you estimate that another 100 function points will be required, you obtain the estimates risk exposure: Risk exposure = (E2 – E1) P2 E1 = effort of original assumption. E2 = effort if alternative assumption is true. P2 = probability that the alternative assumption is true.

ContingencyContingency

This risk exposure corresponds to the contingency you need to protect yourself against the assumption error.

The probabilistic nature of risk means that the contingency will not protect a project if the original assumption is invalid.

Risk exposure and model error are independent. If you include uncertainty in the model, then you

should not include it in the risk assessment. Do not double count the uncertainty/risk.

This risk exposure corresponds to the contingency you need to protect yourself against the assumption error.

The probabilistic nature of risk means that the contingency will not protect a project if the original assumption is invalid.

Risk exposure and model error are independent. If you include uncertainty in the model, then you

should not include it in the risk assessment. Do not double count the uncertainty/risk.

ContingencyContingency

Risk management should be across the entire organisations project portfolio.

The organisation should hold all of the contingency days in a “central fund” which can be drawn on by projects as needed. This is similar to the philosophy used by insurance companies.

Risk management should be across the entire organisations project portfolio.

The organisation should hold all of the contingency days in a “central fund” which can be drawn on by projects as needed. This is similar to the philosophy used by insurance companies.