aoos project selection criteria061407

7/31/2019 AOOS Project Selection Criteria061407

1/26

Alaska Ocean Observing System

Conceptual Design Development

June 18, 2007

Ginny FayMeghan Wilson


2/26

Realism

Capability

Flexibility

Ease of use

Cost

Easy Computerization

Criteria for Project RankingSystems


3/26

Realism

Reflect the reality of the decision making circumstances,

including the multiple objectives of agencies andstakeholders.

Common measurement system to evaluate projects.

Take into account limitations on facilities, capital,

personnel. Include a risk factor.

Capability

Address multiple timeframes

Simulate various situations

Internal situations (e.g., changes in staff)

External situations (e.g., interest rates, appropriations)

Optimize the decision

Criteria for Project Ranking Systems


4/26


Flexibility

Provide valid results within a range ofconditions. Easily modified or self-adjusting to changes in

circumstances.

Ease of Use Convenient, low execution time, easy to use

and understand. Needs no special interpretation, no hard-to-

acquire data, no excessive personnel orunavailable equipment.

Parameters match one to one with real worldvariables that are significant to the project.

Expected outcomes should be easily simulated.


5/26


Cost

Data gathering and modeling costs should be lowrelative to the project cost.

All costs should be considered, and their totalshould definitely not be greater than the potentialbenefits of the project.

Easy Computerization

Convenient to gather and store information in a

computer database. Easy to manipulate the data in the model such as

through a spreadsheet.


6/26

Two basic model or ranking types Numeric - use numbers as inputs.

Nonnumeric - don't use numbers as

inputs.

Important model facts to remember:

Models don't make decisions; people do. Models only partially reflect reality.

The Nature of the Project Ranking Systems


7/26

First step -- develop model categories based on program goalsand objectives.

Categories generated at arms length (blind to potentialprojects) by agencies and stakeholders.

Categories should be weighted to represent their relativeimportance to achieving program goals.

Resulting project scores reflect how much their predictedoutcomes contribute to goals achievement.

Some AOOS identified categories: Ecosystem health

Marine operations Public health Living resources Homeland security Weather & climate Natural hazards

Project Ranking Systems


8/26

Project Ranking Systems

Some factors are recurring, others haveone-time impact.

Ranges of uncertainty are helpful for

hard-to-estimate factors.

Thresholds or critical values of acceptanceor rejection can be assigned.

Items will contain differing levels ofspecificity.

Avoid category overlap to avoid double

counting.


9/26

Type of Project Ranking Systems

Nonnumeric Models

The Sacred Cow

Suggested by a senior and powerful official

Stopped at successful conclusion or when suggesting

official realizes it was a mistakeThe Operating Necessity

Project is required to keep the system going

Ask if system is still worth operating

The Competitive Necessity

Needed to compete

Operating necessity projects take precedence overcompetitive necessity projects


10/26

Program Leverage Judged on potential match or partnerships, filling a gap,

strengthening a weak link or extending into a newdesirable direction.

Comparative Benefit Model Sort

Divide a list of projects into groups of

good - fair - poor.

Rank projects within groups.

One person or a selection committee picks projects.

Models widely used.

Peer review - outside referee picks projects.

Type of Project Ranking Systems


11/26

Unweighted 0-1 Factor Model

List relevant factors on a preprinted form andhave top management evaluators determinewhether or not the project qualifies for each ofthe factors. Sum the responses and see if theproject qualifies for enough factors to accept.

Weighs all factors as equally important.

No gradation of degree to which factor is met.

Unweighted Factor Scoring Model

Grade how well a factor is met by a project on ascale, usually 5 pt. (e.g., 5 - very good, 1 - verybad).

All factors still weighted equally

Numeric Systems - Scoring


12/26

Weighted Factor Scoring ModelAdd weights reflecting the relative importance of

the factors and multiply the weight of each factorby its score. Sum these values and compare to athreshold value.

This method can provide a sensitivity analysis toidentify an component for project improvement.

Don't include marginally relevant factors

Constrained Weighted Factor Model Similar to above except additional criteria enter

the model as constraints (things that must bepresent or absent in order for the project to beacceptable) rather than as weighted factors



13/26

Advantages of Numeric Scoring Models

Allow multiple criteria to be used forevaluation and decision.

Structurally simple and easy to use.

A direct reflection of management policy.

Easily altered to meet changes in theenvironment and in management policy.

Weighted scoring models acknowledge thatsome factors are more important thanothers.

Easy sensitivity analysis to see trade-offsbetween several criteria.



14/26

Disadvantages of Numeric Scoring Models

Output is a relative measure, noutility is reflected, thus no directindication of project support.

Generally linear, elements areassumed to be independent.

Tendency to include too many

criteria.Unweighted models assume equalimportance of all criteria.



15/26

Analysis Under High Uncertainty

Uncertainty in Organizations

Most uncertainty is about the timing and thecosts.

Three areas of uncertaintyTiming of project and cash flows.

What the project will accomplish.

Side effects or unforeseen consequences. Pro-forma documents help reduce

uncertainty.


16/26

Focuses decision maker's attention on the

nature and the extent of the uncertainty ofsome variables in the decision-making process.

Uses probability distributions for each of theuncertain variables, instead of the point

estimates that financial analysis utilizes.

Decision Analysis.

General Simulation Analysis Only count costs and times that result from new

proposed project.

Risk Analysis


17/26

Subjective vs. Objective

Objective - Measurement taken by reference to anexternal system.

Subjective - Reference to a standard that isinternal to the system.

Quantitative vs. Qualitative Difference is that one may apply the law of

addition to quantitative data and not toqualitative.

Reliable vs. Unreliable Data source is reliable if repetitions of a

measurement vary by less than a pre-specifiedamount.

Comments on Measurements


18/26

Suggestions on Ranking Systems-Examples

2006 2008 Project Evaluation Criteria

(Alaska Department of Transportation)


19/26

Puget Sound Near Shore PartnershipEstuary and Salmon Restoration Program

http://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdf
http://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdfhttp://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdf


20/26

Center for Neighborhood Technology Project ScoringCriteria (multiple projects)

http://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdf
http://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdf


21/26

Belize National Protected Areas System (bio-physical and land use characteristics)

http://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdf
http://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdf


22/26

NOAA Risk and VulnerabilityAssessment Tool (RVAT)

Hazard Frequency+

Area Impactx

Magnitude = Total

StormSurge

2 4 5 30

Wind 3 5 4 32

Flood 4 4 4 32

CoastalErosion

3 2 3 15

(Frequency + Area Impact) x Magnitude = Total Score


23/26

Ecosystem Health

Ocean conditions

3-D models of oil spill trajectories

Maritime Operations

Collision avoidance system

Uncharted rocks, reefs and shoals

Weather and wave conditions Surface current velocities and direction

Improve vessel safety and maximize transportationefficiency

Vessel icing conditions Ocean forecast models for currents and shears

Tidal predictions in time and space

Hazardous icing conditions


24/26

Public Health Harmful plankton blooms Contamination of cultured shellfish

Living Resources Effects of eddies on mixing Fish survival as a function of ocean conditions Effects of large eddies on fish migration Fish habitat State of the ocean data Fisheries variability Climate variability Ocean circulation and mixing

Ocean productivity estimates Fish habitat locations and extents Forecast of important food web components Forage fish abundances and distributions Inventory of marine habitats


25/26

Homeland Security Vessel tracking Search and rescue times are lengthy Vessel locations

Weather and Climate Satellite: SAR and QuikSCAT Weather conditions Limited weather observations in Alaska

Surface observations of wind and waves in real time Climate variability Wind and wave observations

Natural Hazards Storm surge models Vessel icing Vessel safety in winter Coastal flowing Coastal flooding Coastal erosion

Prediction of erosion events


26/26

Contact us Steve Colt, ISER

[email protected]

786-1753

Ginny Fay, EcoSystems [email protected]

333-3568

Meghan Wilson, ISER - [email protected]

- 786-5408

website: www.iser.uaa.alaska.edu
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]

aoos project selection criteria061407

Documents