wp vendor selection

Vendor Selection Decisions

4300 Wilson Boulevard, suite 350

arlington, virginia 22203

703-399-2100

www.decisionlens.com

www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008

Selecting the best vendor is a difficult challenge.

The stakes can be high, jobs may be on the line,

and, adding to this complexity, multiple stake-

holders from various factions within the organi-

zation may demand that their input and needs be

addressed throughout the selection process.

Overview of Vendor SelectionMost organizations approach vendor selection (also called “source selection”) by

setting up an evaluation board representing the internal “customers.” The require-

ments are defined, and the vendors are asked to respond through an RFP. They are

then evaluated against a checklist of requirements by each evaluator, and sheets

with checkboxes showing “meets requirements” or “does not meet requirements” are

produced. The result is long lists of checkboxes at a detailed level, with little insight

into whether the vendors are capable of delivering to the key criteria the organization

must meet.

Oftentimes evaluators aren’t able to define a real difference between vendors using

a requirements checkbox evaluation. Another challenge is that a vendor may be an

expert at producing proposals (or hire professional proposal writers) and know how to

address a requirements checklist so that the company looks good, regardless of how

well each requirement is met.

After collating all of the requirements checkboxes and coming down to an even match

between vendors, the decision then becomes one of cost. This is a flawed process that

results in the following questions:

• How can we define our objectives and make the tough trade-offs to determine which

vendor we should choose?

• How can we elicit input from and drive consensus across the evaluation team in an

equitable manner?

• Can we focus the discussion so that we are more efficient in the evaluation process,

while also elevating us from the hidden agendas and inherent biases that others are

bringing to the table?

• How do we tease out the key differences among the vendors to really set them apart,

enabling us to make the “best value” decision?

• How do we know if the vendor’s price is justified by its benefits?

• How can we best defend the decision with senior management and with the ven-

dors?

The members of an organization must work together to assess its relative value—

strengths and weaknesses—across all of the objectives that it’s trying to achieve.

The members need to develop the evaluation criteria as a team, carefully defining the

meaning of each; decide which capabilities are relatively more important than others

in a quantified manner; and finally evaluate the vendors according to strengths and

weaknesses against each of the criteria, collecting all of the relevant comments and

ensuring that the inputs of the evaluation board are captured and aligned in consensus

rather than as checklists of “meets” or “does not meet.”

This white paper covers the

best-practice approach for

vendor selection decisions

and is meant for anyone

involved in a vendor

selection, product selection,

source selection, acquisi-

tion plan, procurement or

contract planning function.


The DeciSion LenS VenDor SeLecTion ProceSS

Decision Lens has developed a best-in-class vendor selection process that is used by Fortune 500 companies and agencies across

federal and state governments. This process was developed through an exhaustive 18-month effort to evaluate procurement best

practices across industries, and was then implemented as the core of the “Acquisition Center of Excellence” for the U.S. intelligence

community. It has since been vetted out through hundreds of vendor selections and has proved exceptionally effective at defending

against protests from vendors.

The rigorous process treats the vendors fairly, and each stage is entirely transparent in terms of the evaluators’ priorities and

analysis of vendors’ performances.

Step 1: Form vendor selection evaluation boardThe evaluation board for the selection is

set up with representation from across

the organization. Often there are two

different evaluation boards created—a

Technical Assessment evaluation board,

consisting of expert users and opera-

tional managers who will evaluate the

vendors’ relative technical merits, and a

Business Assessment evaluation board,

which will evaluate the vendors’ contrac-

tual terms and conditions, past business

conduct, etc.

Can the evaluation board be grouped

into specific functional areas?

Yes, you can divide the evaluation board

into Tier 1 and Tier 2 groups. Tier 1

groups are at the higher level, evaluating

the rollup to the top criteria. The Tier

2 groups are more functionally focused

groups that do the feed-in evaluations of

the proposals with specific strengths and

weaknesses in their areas of expertise.

Step 2: Develop state-ment of work (SOW)The evaluation board defines a list of

needs that are as independent from

one another as possible—mutually

exclusive and completely exhaustive.

Step 3: Develop proposal evaluationThe list of needs is used to develop spe-

cific selection criteria, which are placed

into a hierarchy illustrating all of the

criteria and their relationships to one

another. Criteria may include items such

as technical approach, project man-

agement plan, quality of data, system

application support and maintenance,

corporate quality/experience and ease of

use. Criteria also may be subdivided into

contributing subcriteria. (See Figure 1.)

Figure 1: Hierarchy of criteria

Form SourceSelectionEvaluation Board

DevelopStatementof Work(SOW)

DevelopProposalEvaluation

Review & AssessProposals

Conduct SensitivityAnalysis

MakeBest-ValueSelection

Justify &Document Decision


The group defines the relative impor-

tance of the criteria through a set of pair-

wise comparisons, asking not only if one

criterion is more important than another,

but by how much?

The Decision Lens tools can be used to

build the criteria, define them and then

use them to make the decision. The group

creates a value model using all of the

quantified criteria.

How is cost treated?

Cost is usually treated as an independent

variable. You first evaluate the vendors

based on the benefits. You

want the technical evaluation and busi-

ness terms evaluation to be untainted by

cost. Once you have rated each vendor

against the criteria, you are presented

with the specific value rating of each

vendor towards your needs.

You can then do a cost analysis by look-

ing at the relative benefit scores vs. the

costs of the vendors. An independent

decision is made as to the value/cost

tradeoff, and the final cost analysis

should not be formulaic; rather, you have

to assess the value of the final configura-

tion vs. the cost.

Figure 2: Cost vs. value tradeoff

If Vendor #1 provides 30% more value

and only 20% more cost, you make a

cost/benefit tradeoff decision. This is

a much more powerful approach than

simply selecting a vendor because it

offers the lowest cost.

How is risk evaluated?

Risk should not be included as a sepa-

rate criterion. You will be assessing risk

throughout each criterion in the model.

The risks will be captured in the weak-

nesses that you evaluate for each vendor,

as risk for the specific criterion.

How is past performance evaluated?

You could develop a criterion for “past

performance,” but past performance

should be evaluated across all of the

criteria. In other words, one criterion of

“past performance” may not get a weight

that accurately reflects its influence in

relation to the strengths and weaknesses.

Instead, we recommend that past per-

formance be evaluated throughout the

decision. When participants evaluate the

vendors on their strengths and weak-

nesses under each criterion, the past per-

formance would be calculated as

part of that.

How are the criteria weighted?

The hierarchy is broken down into a

series of judgments (pair-wise compari-

sons) at each level. (See Figure 3.)

For example, you are asked the question

“Would you give more value to an vendor

for its technical approach than for its

project management plan?” You would

then judge the two criteria being com-

pared on a scale ranging from 1 (equal

importance) to 9 (extreme difference

in importance).

Figure 3: Pair-wise comparisons

How is the overall value model that will be used to assess the vendors developed?

The result of the pair-wise comparison process is a set of ratio scale priorities showing the value of each of the criterion to the

decision makers. (See Figure 4.)

Decision Lens derives the priorities from the group members and tracks the consistency of their logic and how they applied their

comparisons. This is not forcing them to agree—disagreement is okay—but the process is done explicitly so that all of the judg-

ments across the evaluation board are represented.


Figure 4: Priority graph

“We needed a decision

process that would

receive buy-in from

all stakeholders …

transmission,

distribution and IS.”

Bill TsoliasManager, Energy Management Group,

National Grid

What drives the Request for

Proposal (RFP)? When does the RFP

process begin?

Don’t send out the RFP until you have

developed your selection criteria. The

criteria will guide the RFP response. You

should include the criteria and defini-

tions in the RFP itself, but not

the weights.

What if the RFP has already

been sent?

The model can be structured to reflect

what went into the RFP. Or if you have

specific potential issues not addressed

in the RFP, you can develop a model that

exactly matches the RFP and a second

model to ensure you have captured all of

your considerations.

What if there are optional

requirements? Are those included

in the upfront hierarchy model?

If all of the vendors are going to be

addressing the options in the model, then

optional requirements should be kept in.

You’ll want to note how important that

option is in your overall consideration

as well.

How are rating scales developed?

Once you have defined the relative impor-

tance of all of the criteria, you create a

ratings scale for each criterion. These

are rulers that will be used to evaluate

exactly how well each vendor’s solution

addressed the criteria (the strengths and

weaknesses of the vendor).

The ratings scales are detailed; an

“excellent” rating has a specific amount

of major strengths, just as “poor” has a

specific amount of major weaknesses.


(See Figures 5 and 6.)

Figure 5: Example of ratings definitions used in ratings scales

Figure 6: Ratings scales

raTing WeighT DeScriPTion

Proposal Ratings, Adjectives, and Definitions

Exceptional 1 Offeror’s proposal demonstrates an exceptional understanding of the goals and objectives of the

acquisition. One or more major strengths exist. No major weaknesses exist. Strengths significantly

outweigh the weaknesses. Expected to cause no disruption in schedule, increase in cost, or

degradation in performance. Will require no organizational emphasis and monitoring to

overcome difficulties.

Very Good .8 Offeror’s proposal demonstrates a very good level of understanding of the goals and objectives of

the acquisition. Strengths outbalance weaknesses that exist. Any weaknesses are easily correctable.

Expected to cause minimal disruption of schedule, increase in cost or degradation of performance.

Will require a low level of organizational emphasis and monitoring to overcome difficulties.

Good .5 Offeror’s proposal demonstrates a good level of understanding of the goals and objectives of the

acquisition. There may be strengths or weaknesses or both. Weaknesses are not offset by strengths,

but the weaknesses do not significantly detract from the offeror’s response. Expected to cause

minimal to moderate disruption in schedule, increase in cost, or degradation in performance. Will

require low to medium level of organizational emphasis and monitoring to overcome difficulties.

Marginal .2 Offeror’s proposal demonstrates a marginal level of understanding of the goals and objectives of

the acquisition. Weaknesses have been found that outbalance any strengths that exist. Weaknesses

will usually be difficult to correct. Expected to cause moderate to high disruption in schedule,

increase in cost, or degradation in performance. Will require medium to high organizational

emphasis and monitoring to overcome difficulties.

Unacceptable 0 Offeror’s proposal demonstrates a poor understanding of the goals and objectives of the acquisition.

No major strengths exist, and one or more major weaknesses exist. Weaknesses clearly surpass any

strengths. Weaknesses are expected to be very difficult to correct or are not correctable. This feature

is so poorly understood and demonstrated that it presents an extremely high risk to the success of

the program. Expected to cause significant, serious disruption in schedule, increase in cost, or

degradation in performance. Will require significant or constant, high level of organizational

emphasis and monitoring to overcome difficulties.


Step 4: Review and assess proposalsA quick read of the proposals is first

completed. This is designed to help

evaluators calibrate the range of vendor

proposal methods and approaches.

The discussion that results is very

important, and the loss of these com-

ments could be a fatal error. Without

a structured process for gathering

comments and evaluating them against

the criteria, the group will not come

to a logical decision.

With Decision Lens, the comments

are organized right in the software

(see Figure 7) or can be captured in

an Excel file.Figure 7: Documenting comments using Decision Lens

How are a vendor’s strengths and weaknesses evaluated?

Identify the specific strengths and weaknesses of the vendor under each criterion.

Avoid looking at it from a general “meets the requirements” point of view. There are

major strengths and minor strengths, major weaknesses and minor weaknesses. Risks

should be translated into weaknesses.

Each of the participants will write an individual evaluation report for each of the

vendor’s proposals under each of the criteria, listing the major and minor weaknesses

that he or she has found in the proposal. He or she will evaluate the strengths and

weaknesses (major and minor); include a “Clarification Request” for information in

the proposal that is inadequate for evaluation; and identify errors, minor omissions,

misunderstandings and contradictory statements.

What is a Consensus Evaluation Report (CER)?

The evaluation reports written by the participants are then brought together into a

Consensus Evaluation Report (CER). The strengths and weaknesses for each vendor

under each criterion are made explicit in the consensus document and are then used

to rate the vendors.

The CER does not necessarily indicate complete agreement but instead indicates the

preferred choice of the group.

How are the vendors rated?

Because each rating scale specifically states the strengths and weaknesses required

to meet each rating level on the ruler (i.e. “Excellent” or “Poor”), the group is generally

very aligned in their ratings. In other words, two members of the evaluation board

should not be opposite from one another on the scale if the strengths and weaknesses

are being applied against the scales correctly. (See Figure 8.)

If there is a member of the group who gives radically different ratings than the overall

group, this discrepancy should be addressed. If the group cannot come to agreement

on the ratings, then the difference of opinion must be documented as a comment only

and the group should move on.

“Applying Decision Len’s

innovative process for

decision making enabled

our Technical Evaluation

Committees (TEC) to focus

on key business and

technical drivers in eval-

uating vendor proposals.

The committees were able

to quickly and effectively

discern major strengths

and weaknesses in ven-

dor proposals and consis-

tently apply their logic in

selecting the best vendor.”

Lenetta Mccampbell Senior Director,

On Board Systems, Amtrak


Figure 8: Ratings

Step 5: Conduct sensitivity analysisSensitivity analysis is a powerful diag-

nostic tool. It enables the team to test

the “what-ifs.” For example, what if the

importance of the technical approach is

increased? Does that change the priority

of the vendors? At what point does one

vendor pass another?

In January of 2006, Harvard Business

Review published an entire issue on deci-

sion making. One of the key criticisms

was that in most decision-making pro-

cesses, there is no means to introduce

ad-hoc objections or changes to see how

the decision would be affected.

The Decision Lens process enables you to

introduce changes throughout the model,

providing key insights while saving time

and political capital. If senior manage-

ment does not agree with the evaluation,

the team should seek the reason for the

disagreement (the criteria used, the pri-

ority given to the criteria, the rating of

the vendors, etc.). The reason can then

be addressed and evaluated for further

refinement. (See Figure 9.)

Figure 9: “What-if” sensitivity analysis

Step 6: Make best-value selectionOnce all of the ratings are complete for each vendor across the criteria, you will have a specific, quantified measure showing the

value of each vendor against the objectives of your organization. (See Figure 10.)

You then may evaluate the relative “value” score against costs, and can determine which vendor best meets your needs. For example,

if you are more cost sensitive, then you may not be willing to make a tradeoff of 30% more benefits from one vendor to another for

just 10% more cost. If cost is less of an issue, you may make this tradeoff.

What if multiple vendors are to be chosen?

Decision Lens optimization capabilities allow you to enter a total cost figure for each vendor. Define your available budget as a

budget pool in the optimizer, and then optimize to get the highest value for cost. The calculation performed in Decision Lens is

to maximize “benefit priority/cost.”


Figure 10: Ratings scoresheet

Step 7: Justify and Document DecisionThe power of this process is that each

stage is explicitly and rigorously defined

and quantified. Specific judgments from

each evaluation board member are used

to drive the value of the criteria. Explicit

strengths and weaknesses and comments

are used to drive the vendor ratings. The

group works collaboratively, but consen-

sus is not forced at any stage; transpar-

ency and explicit judgment are required.

As such, any future review or audit

has all of the necessary information to

debrief senior management, vendors and/

or your customers.

conclusionThe vendor selection process is indeed

challenging, but when rigor is applied

through each step of the process with

a sound methodology and the ability to

analyze both qualitative and quantitative

evaluation criteria together, you arrive

at a robust and defensible decision. It is

critical that a major vendor selection pro-

cess be able to bring together all of the

stakeholders into a common collabora-

tion that generates buy-in, and that their

judgments, comments and evaluation

points be captured throughout the pro-

cess as well. Decision Lens is ideally

suited for this process and has been

proven through numerous “best-value”

vendor selection decisions to be a true

best-practice approach.