written by william tibben iact 424/924 the design process: evaluating plans william tibben sitacs...
Post on 18-Dec-2015
213 views
TRANSCRIPT
Written by William Tibben
IACT 424/924The Design Process:
Evaluating Plans
William Tibben
SITACS
University of Wollongong
17 September 2002
Written by William Tibben
Overview
• Definitions• Brief example to illustrate why we evaluate• A real life example of an evaluation process
– Case Study of Commonwealth Gov Tender
• Move to the theory – the need to determine priorities
• Final focus on technical matters.
Written by William Tibben
Definition
• Evaluation – to determine the value or worth of, to appraise (Cassell Concise English Dictionary)
• Some common anxieties that evaluation should relieve– Will it work?
– Will it meet the design criteria?
– How much is it going to cost?
Written by William Tibben
What is evaluation in this context?
• Evaluation in this context refers to a recognised point in time where the detail of the design is evaluated in relation to the primary goals that have been established
• Is distinguished from the “normal” and ongoing evaluation that one undergoes while doing work (eg am I doing it right?)
Written by William Tibben
What is evaluation in this context?
• Represents a point where goals can be re-assessed to determine if the detailed design will meet the goals.
• A point in which the detailed design can be altered (fine tuned) to better meet goals
• A point in which goals can altered to take account of contradictions between goals
Written by William Tibben
Scenario 1: The Design Criteria states that a backup line needs to be provided for the ISDN serial link between office 1 and office 2
Tender 1.• $120,000• Main Serial Line –
Telstra ISDN line• Back up Line – Dial-
up modem using PSTN
Tender 2• $130,000• Main Serial Line –
Telstra ISDN Line• Back up line – ‘Line-
of-sight’ microwave link owned by business.
Written by William Tibben
Which tender?
• Which tender would you choose?– Tender 1 - the cheapest
Or
– Tender 2 – a separate and higher bandwidth back-up link
Or
– Go for a change in design criteria – make the main link the microwave link (avoid costs for the ISDN line from Telstra) and use a dial up modem backup
Written by William Tibben
• In summary, we often need to balance the desire for technical excellence and cost
• It can be a point where management and engineering collide.
Written by William Tibben
Evaluating and resolving conflicting objectives
One example of an evaluation task is a Commonwealth Government Tender Assessment Procedure
1. Technical Proposal – if OK then move onto
2. Budget
Written by William Tibben
Comm Government Tender
• Technical Proposal – – knowledge and experience in relevant fields –
50%– Managerial and financial capabilities – 20%– General capabilities – 30%
Written by William Tibben
Knowledge and Experience in Relevant fields
• The design document should detail how the proposed design will meet the the design requirements
• The document should detail a timeline as to how the project will be executed
Written by William Tibben
Managerial and Financial Capabilities
• Does the tenderer have appropriate personnel (CV)?
• Is the tenderer able to substitute a similarly qualified person should the primary person become unavailable?
• Is the tenderer able to complete the work within the budget allocated?
Written by William Tibben
General Capabilities
• Is the tenderer or good reputation?
• Does the tenderer have a good understanding of your business and the environment in which it works?
• Does the tenderer have good contacts within the industry?
Written by William Tibben
Lastly,…
• Once the tenders have satisfactorily satisfied the technical requirements then tenders should be compared on the basis of cost
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
1. Strategic Assessment2. Technical Assessment3. Cost-benefit assessment4. Risk Analysis
• Where do each of these forms of evaluation relate to when considering the Comm. Gov. Tender?
Written by William Tibben
1. Strategic Assessment
2. Technical Assessment
3. Cost-benefit assessment
4. Risk Analysis
• Technical Proposal – – knowledge and experience
in relevant fields – 50%
– Managerial and financial capabilities – 20%
– General capabilities – 30%
• Budget
Written by William Tibben
1. Strategic Assessment
2. Technical Assessment
3. Cost-benefit assessment
4. Risk Analysis
• Technical Proposal – – knowledge and experience
in relevant fields – 50%
– Managerial and financial capabilities – 20%
– General capabilities – 30%
• Budget
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
• Strategic Assessment– Objectives: support for corporate vision– Information System (IS) Plan: legacy systems– Organisation Structure: enhance of destroy?– Management Information System (MIS)– Personnel: manning levels and skill base– Corporate Image: will it affect customer
perceptions
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
• Technical Assessment– Functionality: will the end product work?
End-end connectivity issuesApplicationsSecurity
Metrics – refer lecture 8: design requirements
– Scalability: is the network able to grow without major problems
– Adaptability: will the project be able to incorporate new technologies in the future?
– Manageability: can we monitor network operations and make necessary changes easily?
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
• Cost-benefit Analysis• Costs
– Development cost– Setup cost– Operational costs
• Benefits– Direct benefits (reduction in salary bills)– Indirect benefits (increased accuracy, increased timeliness,
more user friendly)– Intangible benefits (Better customer and supplier
relationships, better information flows/problem solving)
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
• Cost-benefit Analysis– If benefits > cost, that is good– If benefits < cost – that is bad
• How do you measure intangible benefits?
Written by William Tibben
Four foci of evaluation (Cotterall and Hughes1995)
• Risk Analysis– What is the likelihood that an event will result
in the project not meeting its objectives?– A cost-benefit style of analysis can also be used
to quantify possible losses. “What if” analysis– The difficulty is determining the likelihood of
an event occurring or accounting for an unpredictable set of circumstances
Written by William Tibben
Resolving Conflicting Goals
• It is in the evaluation phase when goals can collide. How does one resolve these?
• McCabe (1998) suggests that you prioritise design gaols.
• What kind of prioritisation occurred with the Comm Gov Tender?
Written by William Tibben
Resolving Conflicting Goals
• McCabe (1998) also suggests that prioritisation of goals allows one to better clarify future directions. This was demonstrated in Scenario 1described at the beginning of the lecture.
Written by William Tibben
Technical Assessment
• How does one determine the technical characteristics of a proposal?– Functionality: will the end product work?
End-end connectivity issuesIsolate specific services and determine whether flows
indicated on the diagram are appropriate
Written by William Tibben
Teare, 1999, p. 384
Written by William Tibben
Teare, 1999, p. 385
Written by William Tibben
Milestone 3 Scenario
• A distributed database for record keeping
• Interactive training modules demonstrating fine art of coffee and tea making
• Streaming Audio and Video for meetings
• Ask the question whether specific technologies are suitable for the kind of service you wish to deliver.
1. Best effort
2. Deterministic service
3. Guaranteed Service
Written by William Tibben
Criteria used in your last mile stone exercise1. Response Time
2. Accuracy
3. Availability
4. Maximum Network Utilisation
5. Throughput
6. Efficiency
7. Latency
Written by William TibbenFill in the blanks
Best Effort Deterministic Guaranteed
Response Time
Accuracy
Availability
Network Utilisation
Throughput
Efficiency
Latency
Written by William Tibben
Technical Assessment
• Scalability: is the network able to grow without major problem– Are group sizes appropriate?
• McCabe (1998, p. 185)– In a broadcast environment (eg Ethernet) background
broadcast traffic should be < 2%– Need to remember that routing protocols also account
for background traffic and need to be considered» e.g Connection oriented protocols need to
exchange state information between end points in order to maintain track of the packet sequence, errors and buffer overflow
Written by William Tibben
Technical Assessment
• Scalability: is the network able to grow without major problem– Is there sufficient room for growth in the
protocols you have chosen?– All protocols have a common upgrade path to
higher bandwidths. You should be choosing protocols to ensure that they can be speeded-up as increased traffic demands
Written by William Tibben
McCabe, 1998, p.193
Written by William Tibben
Technical Assessment
• Adaptability: will the project be able to incorporate new technologies in the future?– Best effort may be OK today but your applications may
demand Deterministic Service or Guaranteed Service in the future
– If you can foresee that there will be a future requirements for more demanding specifications you will need to consider whether you should move to those protocols now.
Written by William TibbenNBMA=Non-Broadcast Multiple Access McCabe, 1998, p.195
Written by William TibbenMcCabe, 1998, p.196
Written by William Tibben
Technical Assessment
• Manageability: can we monitor network operations and make necessary changes? – In-band monitoring – relies on existing network
to transmit management data
– Out-of-band monitoring uses a separate network to transfer information (e.g. a dialup line using the PSTN)
Written by William Tibben
Prototyping
• Prototyping represents another important evaluation mechanism
• Begins to stray into the area of next lecture validating plans
Written by William Tibben
References
• MacCabe, D. D. 1998, Practical Computer Network Analysis and Design, Morgan Kaufman Publishers, San Francisco, California.
• Teare, D. 1999, Designing Cisco Networks, Cisco Press Indianapolis.