lec#3 - selection of techniques and metrics.pdf

7/21/2019 Lec#3 - Selection of Techniques and Metrics.pdf

1/33

3-1

2010 Raj Jain www.rajjain.com

Simulation

, Modeling and Analysis of

Computer Networks

(ECE 6620)

Dr. M. Hasan Islam

Selection

of Techniques

and Metrics(Chapter#3)

Art of Computer Systems Performance Analysis

By R. Jain


2/33

3-2


Overview

Criteria for Selecting an Evaluation Technique

Three Rules of Validation

Selecting Performance Metrics

Commonly Used Performance Metrics

Utility Classification of Metrics

Setting Performance Requirements


3/33

3-3


Criteria for Selecting an Evaluation Technique


4/33

3-4


Three Rules of Validation

Do not trust the results of an analytical model untilthey have been validated by a simulation model or

measurements.

Do not trust the results of a simulation model until

they have been validated by analytical modeling or

measurements.

Do not trust the results of a measurement until theyhave been validated by simulation or analytical

modeling.


5/33

3-5


Selecting Performance Metrics


6/33

3-6


Selecting Metrics

Include: Performance Time, Rate, Resource

Error rate, probability

Time to failure and duration

Consider including: Mean and variance

Individual and Global

Selection Criteria:

Low-variability Non-redundancy

Completeness


7/333-7


Compare two different congestion control algorithmsfor computer networks

A computer network consists of a number of end systems

interconnected via a number of intermediate systems

End systems send packets to other end systems on thenetwork

Intermediate systems forward packets along the right path

The problem of congestion occurs when the number of

packets waiting at an intermediate system exceeds the

systems buffering capacity and some of the packets have

to be dropped

Case Study: Two Congestion Control Algorithms


8/333-8


Case Study: Two Congestion Control Algorithms

System: Consists of networks Service: Send packets from specified source to

specified destination in order

Possible outcomes:

Some packets are delivered in order to the correct

destination

Some packets are delivered out-of-order to the destination

Some packets are delivered more than once (duplicates) Some packets are dropped on the way (lost packets)


9/333-9


Performance metrics: For packets delivered in order

Time-rate-resource metrics Response time to deliver the packets

Delay inside the network for individual packets

Determines the time that a packet has to be kept at the source end

station using up its memory resources

Lower response time is considered better

Throughput

Number of packets per unit of time

Larger throughput is considered better

Processor time per packet on the source end system

Processor time per packet on the destination end systems

Processor time per packet on the intermediate systems


10/333-10



Variability of the response time Highly variant response results in unnecessary retransmissions

Probability of out-of-order arrivals

Undesirable since they cannot generally be delivered to the user

immediately

In many systems, the out-of-order packets are discarded at the

destination end systems

In others, they are stored in system buffers awaiting arrival of

intervening packets

In either case, out-of-order arrivals cause additional overhead(buffers)

Probability of duplicate packets

Consume the network resources without any use


11/333-11



Probability of Lost packets Undesirableresults in retransmissions, delays, loss of data,

disconnection

Probably of disconnect

Excessive losses result in excessive retransmissions and could cause

some user connections to be broken prematurely


12/333-12


Case Study (Cont)

Fairness The network is a multiuser system. It is necessary that all users be

treated fairly. It is defined as a function of variability of

throughput across users

For any given set of user throughputs (x1, x2,..., xn), the following

function can be used to assign a fairness index to the set:

Fairness Index Properties: For all non- negative values it is always lies between 0 and 1

If all users receive equal throughput, the fairness index is 1

If kof n receivex and n-kusers receive zero throughput: the fairness

index is k/n.


13/333-13


Redundant Parameters

After a few experiments, throughput and delay were foundredundant

All schemes that resulted in higher throughput also resulted in

higher delay, therefore, the two metrics were removed from the

list and instead a combined metric called power

A higher power meant either a higher throughput or a lower

delay


14/333-14


Variance in response time Also dropped since it was redundant with the probability of

duplication and the probability of disconnection

A higher variance resulted in a higher probability of

duplication and a higher probability of prematuredisconnection

Thus, in this study a set of nine metrics were used tocompare different congestion control algorithms

Redundant Parameters


15/333-15


Response time is defined as the interval between ausers request and the system response

Simplistic definition



16/333-16



Two definitions of response time


17/333-17


Turnaround time Basically the responsiveness for a batch stream

Time between the submission of a batch job and the

completion of its output

Time to read the input is included in the turnaround time

Reaction Time

Time between submission of a request and the beginning of

its execution by the system

To measure the reaction time, one has to able to monitor the

actions inside a system since the beginning of the execution

may not correspond to any externally visible event



18/333-18


Stretch factor Ratio of response time at a particular load to that at

the minimum load

Response time of a system generally increases as

the load on the system increases

For a timesharing system, for example, the stretch

factor is defined as the ratio of the response time

with multiprogramming to that withoutmultiprogramming



19/333-19


Common Performance Metrics (cont)

Throughput Rate (requests per unit of time) at which the requests can be

serviced by the system

For batch streams, the throughput is measured in jobs per second

For interactive systems, throughput is measured in requests per

second

Jobs per second

Requests per second

Millions of Instructions Per Second (MIPS)

Millions of Floating Point Operations Per Second (MFLOPS)

Packets Per Second (PPS)

Bits per second (bps)

Transactions Per Second (TPS)


20/333-20


Common Performance Metrics (Cont)

Throughput of a system generally increases as the load on the system

initially increases and after a certain load, the throughput stops increasing;

in most cases, it may even start decreasing

Nominal Capacity: Maximum achievable throughput under ideal workload

conditions. e.g., for networks nominal capacity is called bandwidth in bits

per second. The response time at maximum throughput is too high Usable capacity: Maximum throughput achievable without exceeding a

pre-specified response-time limit

Knee Capacity: throughput at Knee = Low response time and High

throughput

It is also common to measure capacity in terms of load, for example, the

number of users rather than the throughput


21/333-21


Capacity


22/333-22



Efficiency: Ratio of maximum usable capacity (achievable throughput) to

nominal capacity

Maximum throughput from a 100-Mbps (megabits per second) Local

Area Network (LAN) is only 85 Mbps, its efficiency is 85%.

For multi-processors system ratio of the performance of an n-processor

system to that of a one-processor system is its efficiency Utilization: The fraction of time the resource is busy servicing requests.

Average fraction used for memory

processors, are always either busy or

idle, so their utilization is in terms of

ratio of busy time to total time. For otherresources, such as memory, only a

fraction of the resource may be

used at a given time; their utilization is

measured as the average fraction used

over an interval


23/333-23



Reliability: Usually measured by probability of errors or the mean time between errors

(error-free seconds)

Availability:

Fraction of the time the system is available to service users requests

Time during which the system is not available is called downtime; the

time during which the system is available is called uptime

Better indicator of uptime Mean Time to Failure (MTTF)

Mean Time to Repair (MTTR)

MTTF/(MTTF+MTTR)


24/333-24


In system procurement studies, the cost/performanceratio is commonly used as a metric for comparing two

or more systems

The cost includes the cost of hardware/software

licensing, installation, and maintenance over a givennumber of years

The performance is measured in terms of throughput

under a given response time constraint For example, two transaction processing systems may

be compared in terms of dollars per TPS



25/33

3-252010 Raj Jain www.rajjain.com

Utility function of a performance metric, can be categorized into three classes

H igher is Better or HB

System users and system managers prefer higher values of such metrics.

System throughput is an example of an HB metric.

Lower is Better or LB

System users and system managers prefer smaller values of such metrics.

Response time is an example of an LB metric

Nominal i s Best or NB

Both high and low values are undesirable. A particular value in the middle is

considered the best

Very high utilization is considered bad by the users since their response times

are high

Very low utilization is considered bad by system managers since the system

resources are not being used

Some value in the range of 50 to 75% may be considered best by both users

and system managers



26/33




27/33



One problem performance analysts are faced with isthat of specifying performance requirements for asystem to be acquired or designed

Typical system requirements - Examples:

The system should be both processing and memoryefficient. It should not create excessive overhead

There should be an extremely low probability thatthe network will duplicate a packet, deliver a

packet to the wrong destination, or change the datain a packet.


28/33



General Problems (may not applied to all):

Non-Specific - No clear numbers are specified. Qualitative words such as

low, high, rare, and extremely small are used instead

Non-Measurable - no way to measure a system and verify that it meets the

requirement

Non-Acceptable - Numerical values of requirements, if specified, are setbased upon what can be achieved or what looks good. If an attempt is

made to set the requirements realistically, they turn out to be so low that

they become unacceptable (achievable)

Non-Realizable - Often, requirements are set high so that they look good.

However, such requirements may not be realizable (achievable)Non-Thorough - No attempt is made to specify a possible outcomes

(study of all possible outcomes and failures)

SMART

Specific, Measurable, Acceptable, Realizable, and Thorough


29/33


A LAN basically provides the service of transportingframes (or packets) to the specified destination station

Given a user request to send a frame to destination

station D, there are three categories of outcomes:

Frame is correctly delivered to D

Incorrectly delivered (delivered to a wrong

destination or with an error indication to D)

Not delivered at all

Service: Send frame to Destnation

Case Study 3.2: Local Area Networks


30/33


Performance requirements for three categoriesSpeed: If the packet is correctly delivered, the time

taken to deliver it and the rate at which it is

delivered are important. This leads to the following

two requirements:

Access delay at any station should be less than 1 second

Sustained throughput must be at least 80 Mbits/sec



31/33


Reliability: Six different error modes were considered important.Probability requirements for each of these error modes and their

combined effect are specified as follows:

Probability of any bit being in error must be less than 10-7

Probability of any frame being in error (with error indication set)

must be less than 1%

Probability of a frame in error being delivered without error

indication must be less than 10-15

Probability of a frame being mis-delivered due to an undetected

error in the destination address must be less than 10-18

Probability of a frame being delivered more than once (duplicate)

must be less than 10-5

Probability of losing a frame (due to all sorts of errors) must be less

than1%



32/33


Availability: Two fault modes were considered significant First - time lost due to the network re-initializations

Second - Time lost due to permanent failures requiring field

service calls

Requirements for frequency and duration of these faults are: Mean time to initialize LAN must be less than 15 milliseconds

Mean time between LAN initializations must be at least 1 minute

Mean time to repair a LAN must be less than 1 hour

Mean time between LAN partitioning must be at least half a week


All of the numerical values specified above were checked for realizability by

analytical modeling, which showed that LAN systems satisfying these

requirements were feasible


33/33


Make a complete list of metrics to compare Two personal computers

Two database systems

Two disk drives Two window systems

Submission Date: 17 Feb 12pm

Assignment-2

lec#3 - selection of techniques and metrics.pdf

Documents