Download - AOOS Project Selection Criteria061407
-
7/31/2019 AOOS Project Selection Criteria061407
1/26
Alaska Ocean Observing System
Conceptual Design Development
June 18, 2007
Ginny FayMeghan Wilson
-
7/31/2019 AOOS Project Selection Criteria061407
2/26
Realism
Capability
Flexibility
Ease of use
Cost
Easy Computerization
Criteria for Project RankingSystems
-
7/31/2019 AOOS Project Selection Criteria061407
3/26
Realism
Reflect the reality of the decision making circumstances,
including the multiple objectives of agencies andstakeholders.
Common measurement system to evaluate projects.
Take into account limitations on facilities, capital,
personnel. Include a risk factor.
Capability
Address multiple timeframes
Simulate various situations
Internal situations (e.g., changes in staff)
External situations (e.g., interest rates, appropriations)
Optimize the decision
Criteria for Project Ranking Systems
-
7/31/2019 AOOS Project Selection Criteria061407
4/26
Criteria for Project Ranking Systems
Flexibility
Provide valid results within a range ofconditions. Easily modified or self-adjusting to changes in
circumstances.
Ease of Use Convenient, low execution time, easy to use
and understand. Needs no special interpretation, no hard-to-
acquire data, no excessive personnel orunavailable equipment.
Parameters match one to one with real worldvariables that are significant to the project.
Expected outcomes should be easily simulated.
-
7/31/2019 AOOS Project Selection Criteria061407
5/26
Criteria for Project Ranking Systems
Cost
Data gathering and modeling costs should be lowrelative to the project cost.
All costs should be considered, and their totalshould definitely not be greater than the potentialbenefits of the project.
Easy Computerization
Convenient to gather and store information in a
computer database. Easy to manipulate the data in the model such as
through a spreadsheet.
-
7/31/2019 AOOS Project Selection Criteria061407
6/26
Two basic model or ranking types Numeric - use numbers as inputs.
Nonnumeric - don't use numbers as
inputs.
Important model facts to remember:
Models don't make decisions; people do. Models only partially reflect reality.
The Nature of the Project Ranking Systems
-
7/31/2019 AOOS Project Selection Criteria061407
7/26
First step -- develop model categories based on program goalsand objectives.
Categories generated at arms length (blind to potentialprojects) by agencies and stakeholders.
Categories should be weighted to represent their relativeimportance to achieving program goals.
Resulting project scores reflect how much their predictedoutcomes contribute to goals achievement.
Some AOOS identified categories: Ecosystem health
Marine operations Public health Living resources Homeland security Weather & climate Natural hazards
Project Ranking Systems
-
7/31/2019 AOOS Project Selection Criteria061407
8/26
Project Ranking Systems
Some factors are recurring, others haveone-time impact.
Ranges of uncertainty are helpful for
hard-to-estimate factors.
Thresholds or critical values of acceptanceor rejection can be assigned.
Items will contain differing levels ofspecificity.
Avoid category overlap to avoid double
counting.
-
7/31/2019 AOOS Project Selection Criteria061407
9/26
Type of Project Ranking Systems
Nonnumeric Models
The Sacred Cow
Suggested by a senior and powerful official
Stopped at successful conclusion or when suggesting
official realizes it was a mistakeThe Operating Necessity
Project is required to keep the system going
Ask if system is still worth operating
The Competitive Necessity
Needed to compete
Operating necessity projects take precedence overcompetitive necessity projects
-
7/31/2019 AOOS Project Selection Criteria061407
10/26
Program Leverage Judged on potential match or partnerships, filling a gap,
strengthening a weak link or extending into a newdesirable direction.
Comparative Benefit Model Sort
Divide a list of projects into groups of
good - fair - poor.
Rank projects within groups.
One person or a selection committee picks projects.
Models widely used.
Peer review - outside referee picks projects.
Type of Project Ranking Systems
-
7/31/2019 AOOS Project Selection Criteria061407
11/26
Unweighted 0-1 Factor Model
List relevant factors on a preprinted form andhave top management evaluators determinewhether or not the project qualifies for each ofthe factors. Sum the responses and see if theproject qualifies for enough factors to accept.
Weighs all factors as equally important.
No gradation of degree to which factor is met.
Unweighted Factor Scoring Model
Grade how well a factor is met by a project on ascale, usually 5 pt. (e.g., 5 - very good, 1 - verybad).
All factors still weighted equally
Numeric Systems - Scoring
-
7/31/2019 AOOS Project Selection Criteria061407
12/26
Weighted Factor Scoring ModelAdd weights reflecting the relative importance of
the factors and multiply the weight of each factorby its score. Sum these values and compare to athreshold value.
This method can provide a sensitivity analysis toidentify an component for project improvement.
Don't include marginally relevant factors
Constrained Weighted Factor Model Similar to above except additional criteria enter
the model as constraints (things that must bepresent or absent in order for the project to beacceptable) rather than as weighted factors
Numeric Systems - Scoring
-
7/31/2019 AOOS Project Selection Criteria061407
13/26
Advantages of Numeric Scoring Models
Allow multiple criteria to be used forevaluation and decision.
Structurally simple and easy to use.
A direct reflection of management policy.
Easily altered to meet changes in theenvironment and in management policy.
Weighted scoring models acknowledge thatsome factors are more important thanothers.
Easy sensitivity analysis to see trade-offsbetween several criteria.
Numeric Systems - Scoring
-
7/31/2019 AOOS Project Selection Criteria061407
14/26
Disadvantages of Numeric Scoring Models
Output is a relative measure, noutility is reflected, thus no directindication of project support.
Generally linear, elements areassumed to be independent.
Tendency to include too many
criteria.Unweighted models assume equalimportance of all criteria.
Numeric Systems - Scoring
-
7/31/2019 AOOS Project Selection Criteria061407
15/26
Analysis Under High Uncertainty
Uncertainty in Organizations
Most uncertainty is about the timing and thecosts.
Three areas of uncertaintyTiming of project and cash flows.
What the project will accomplish.
Side effects or unforeseen consequences. Pro-forma documents help reduce
uncertainty.
-
7/31/2019 AOOS Project Selection Criteria061407
16/26
Focuses decision maker's attention on the
nature and the extent of the uncertainty ofsome variables in the decision-making process.
Uses probability distributions for each of theuncertain variables, instead of the point
estimates that financial analysis utilizes.
Decision Analysis.
General Simulation Analysis Only count costs and times that result from new
proposed project.
Risk Analysis
-
7/31/2019 AOOS Project Selection Criteria061407
17/26
Subjective vs. Objective
Objective - Measurement taken by reference to anexternal system.
Subjective - Reference to a standard that isinternal to the system.
Quantitative vs. Qualitative Difference is that one may apply the law of
addition to quantitative data and not toqualitative.
Reliable vs. Unreliable Data source is reliable if repetitions of a
measurement vary by less than a pre-specifiedamount.
Comments on Measurements
-
7/31/2019 AOOS Project Selection Criteria061407
18/26
Suggestions on Ranking Systems-Examples
2006 2008 Project Evaluation Criteria
(Alaska Department of Transportation)
-
7/31/2019 AOOS Project Selection Criteria061407
19/26
Puget Sound Near Shore PartnershipEstuary and Salmon Restoration Program
http://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdf
http://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdfhttp://pugetsoundnearshore.org/esrp/esrp_proposalrequest.pdf -
7/31/2019 AOOS Project Selection Criteria061407
20/26
Center for Neighborhood Technology Project ScoringCriteria (multiple projects)
http://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdf
http://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdfhttp://www.cnt.org/tsp/pdf/Criteria%20Project%20Scoring%20-%202003.pdf -
7/31/2019 AOOS Project Selection Criteria061407
21/26
Belize National Protected Areas System (bio-physical and land use characteristics)
http://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdf
http://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdfhttp://biological-diversity.info/Downloads/NPAPSP/Site_scoring.pdf -
7/31/2019 AOOS Project Selection Criteria061407
22/26
NOAA Risk and VulnerabilityAssessment Tool (RVAT)
Hazard Frequency+
Area Impactx
Magnitude = Total
StormSurge
2 4 5 30
Wind 3 5 4 32
Flood 4 4 4 32
CoastalErosion
3 2 3 15
(Frequency + Area Impact) x Magnitude = Total Score
-
7/31/2019 AOOS Project Selection Criteria061407
23/26
Ecosystem Health
Ocean conditions
3-D models of oil spill trajectories
Maritime Operations
Collision avoidance system
Uncharted rocks, reefs and shoals
Weather and wave conditions Surface current velocities and direction
Improve vessel safety and maximize transportationefficiency
Vessel icing conditions Ocean forecast models for currents and shears
Tidal predictions in time and space
Hazardous icing conditions
-
7/31/2019 AOOS Project Selection Criteria061407
24/26
Public Health Harmful plankton blooms Contamination of cultured shellfish
Living Resources Effects of eddies on mixing Fish survival as a function of ocean conditions Effects of large eddies on fish migration Fish habitat State of the ocean data Fisheries variability Climate variability Ocean circulation and mixing
Ocean productivity estimates Fish habitat locations and extents Forecast of important food web components Forage fish abundances and distributions Inventory of marine habitats
-
7/31/2019 AOOS Project Selection Criteria061407
25/26
Homeland Security Vessel tracking Search and rescue times are lengthy Vessel locations
Weather and Climate Satellite: SAR and QuikSCAT Weather conditions Limited weather observations in Alaska
Surface observations of wind and waves in real time Climate variability Wind and wave observations
Natural Hazards Storm surge models Vessel icing Vessel safety in winter Coastal flowing Coastal flooding Coastal erosion
Prediction of erosion events
-
7/31/2019 AOOS Project Selection Criteria061407
26/26
Contact us Steve Colt, ISER
786-1753
Ginny Fay, EcoSystems [email protected]
333-3568
Meghan Wilson, ISER - [email protected]
- 786-5408
website: www.iser.uaa.alaska.edu
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]