iact 424/924 the design process: evaluating plans

Written by William Tibben

IACT 424/924The Design Process:

Evaluating Plans

William Tibben

SITACS

University of Wollongong

17 September 2002


Overview

• Definitions• Brief example to illustrate why we evaluate• A real life example of an evaluation process

– Case Study of Commonwealth Gov Tender

• Move to the theory – the need to determine priorities

• Final focus on technical matters.


Definition

• Evaluation – to determine the value or worth of, to appraise (Cassell Concise English Dictionary)

• Some common anxieties that evaluation should relieve– Will it work?

– Will it meet the design criteria?

– How much is it going to cost?


What is evaluation in this context?

• Evaluation in this context refers to a recognised point in time where the detail of the design is evaluated in relation to the primary goals that have been established

• Is distinguished from the “normal” and ongoing evaluation that one undergoes while doing work (eg am I doing it right?)


What is evaluation in this context?

• Represents a point where goals can be re-assessed to determine if the detailed design will meet the goals.

• A point in which the detailed design can be altered (fine tuned) to better meet goals

• A point in which goals can altered to take account of contradictions between goals


Scenario 1: The Design Criteria states that a backup line needs to be provided for the ISDN serial link between office 1 and office 2

Tender 1.• $120,000• Main Serial Line –

Telstra ISDN line• Back up Line – Dial-

up modem using PSTN

Tender 2• $130,000• Main Serial Line –

Telstra ISDN Line• Back up line – ‘Line-

of-sight’ microwave link owned by business.


Which tender?

• Which tender would you choose?– Tender 1 - the cheapest

Or

– Tender 2 – a separate and higher bandwidth back-up link

Or

– Go for a change in design criteria – make the main link the microwave link (avoid costs for the ISDN line from Telstra) and use a dial up modem backup


• In summary, we often need to balance the desire for technical excellence and cost

• It can be a point where management and engineering collide.


Evaluating and resolving conflicting objectives

One example of an evaluation task is a Commonwealth Government Tender Assessment Procedure

1. Technical Proposal – if OK then move onto

2. Budget


Comm Government Tender

• Technical Proposal – – knowledge and experience in relevant fields –

50%– Managerial and financial capabilities – 20%– General capabilities – 30%


Knowledge and Experience in Relevant fields

• The design document should detail how the proposed design will meet the the design requirements

• The document should detail a timeline as to how the project will be executed


Managerial and Financial Capabilities

• Does the tenderer have appropriate personnel (CV)?

• Is the tenderer able to substitute a similarly qualified person should the primary person become unavailable?

• Is the tenderer able to complete the work within the budget allocated?


General Capabilities

• Is the tenderer or good reputation?

• Does the tenderer have a good understanding of your business and the environment in which it works?

• Does the tenderer have good contacts within the industry?


Lastly,…

• Once the tenders have satisfactorily satisfied the technical requirements then tenders should be compared on the basis of cost


Four foci of evaluation (Cotterall and Hughes1995)

1. Strategic Assessment2. Technical Assessment3. Cost-benefit assessment4. Risk Analysis

• Where do each of these forms of evaluation relate to when considering the Comm. Gov. Tender?


1. Strategic Assessment

2. Technical Assessment

3. Cost-benefit assessment

4. Risk Analysis

• Technical Proposal – – knowledge and experience

in relevant fields – 50%

– Managerial and financial capabilities – 20%

– General capabilities – 30%

• Budget



• Strategic Assessment– Objectives: support for corporate vision– Information System (IS) Plan: legacy systems– Organisation Structure: enhance of destroy?– Management Information System (MIS)– Personnel: manning levels and skill base– Corporate Image: will it affect customer

perceptions



• Technical Assessment– Functionality: will the end product work?

End-end connectivity issuesApplicationsSecurity

Metrics – refer lecture 8: design requirements

– Scalability: is the network able to grow without major problems

– Adaptability: will the project be able to incorporate new technologies in the future?

– Manageability: can we monitor network operations and make necessary changes easily?



• Cost-benefit Analysis• Costs

– Development cost– Setup cost– Operational costs

• Benefits– Direct benefits (reduction in salary bills)– Indirect benefits (increased accuracy, increased timeliness,

more user friendly)– Intangible benefits (Better customer and supplier

relationships, better information flows/problem solving)



• Cost-benefit Analysis– If benefits > cost, that is good– If benefits < cost – that is bad

• How do you measure intangible benefits?



• Risk Analysis– What is the likelihood that an event will result

in the project not meeting its objectives?– A cost-benefit style of analysis can also be used

to quantify possible losses. “What if” analysis– The difficulty is determining the likelihood of

an event occurring or accounting for an unpredictable set of circumstances


Resolving Conflicting Goals

• It is in the evaluation phase when goals can collide. How does one resolve these?

• McCabe (1998) suggests that you prioritise design gaols.

• What kind of prioritisation occurred with the Comm Gov Tender?


Resolving Conflicting Goals

• McCabe (1998) also suggests that prioritisation of goals allows one to better clarify future directions. This was demonstrated in Scenario 1described at the beginning of the lecture.


Technical Assessment

• How does one determine the technical characteristics of a proposal?– Functionality: will the end product work?

End-end connectivity issuesIsolate specific services and determine whether flows

indicated on the diagram are appropriate


Teare, 1999, p. 384


Teare, 1999, p. 385


Milestone 3 Scenario

• A distributed database for record keeping

• Interactive training modules demonstrating fine art of coffee and tea making

• Streaming Audio and Video for meetings

• Ask the question whether specific technologies are suitable for the kind of service you wish to deliver.

1. Best effort

2. Deterministic service

3. Guaranteed Service


Criteria used in your last mile stone exercise1. Response Time

2. Accuracy

3. Availability

4. Maximum Network Utilisation

5. Throughput

6. Efficiency

7. Latency

Written by William TibbenFill in the blanks

Best Effort Deterministic Guaranteed

Response Time

Accuracy

Availability

Network Utilisation

Throughput

Efficiency

Latency



• Scalability: is the network able to grow without major problem– Are group sizes appropriate?

• McCabe (1998, p. 185)– In a broadcast environment (eg Ethernet) background

broadcast traffic should be < 2%– Need to remember that routing protocols also account

for background traffic and need to be considered» e.g Connection oriented protocols need to

exchange state information between end points in order to maintain track of the packet sequence, errors and buffer overflow



• Scalability: is the network able to grow without major problem– Is there sufficient room for growth in the

protocols you have chosen?– All protocols have a common upgrade path to

higher bandwidths. You should be choosing protocols to ensure that they can be speeded-up as increased traffic demands


McCabe, 1998, p.193



• Adaptability: will the project be able to incorporate new technologies in the future?– Best effort may be OK today but your applications may

demand Deterministic Service or Guaranteed Service in the future

– If you can foresee that there will be a future requirements for more demanding specifications you will need to consider whether you should move to those protocols now.

Written by William TibbenNBMA=Non-Broadcast Multiple Access McCabe, 1998, p.195

Written by William TibbenMcCabe, 1998, p.196



• Manageability: can we monitor network operations and make necessary changes? – In-band monitoring – relies on existing network

to transmit management data

– Out-of-band monitoring uses a separate network to transfer information (e.g. a dialup line using the PSTN)


Prototyping

• Prototyping represents another important evaluation mechanism

• Begins to stray into the area of next lecture validating plans


References

• MacCabe, D. D. 1998, Practical Computer Network Analysis and Design, Morgan Kaufman Publishers, San Francisco, California.

• Teare, D. 1999, Designing Cisco Networks, Cisco Press Indianapolis.

iact 424/924 the design process: evaluating plans

Documents