1

Advancing Metrics on the Standards

TrackLab Testing Results

draft-morton-ippm-advance-metrics-01

Al Morton July 2010ACK: Len Ciavattone for NetProbe

2

Outline Definition-centric metric advancement Possible Report Template

Supports the Protocol Action Request Examples of the Definition-centric

approach: Revised Lab test methods Results for one implementation

Open call for participation in Lab tests

3

Definition-Centric Process (revised) ,---. / \ ( Start ) \ / Implementations `-+-' +-------+ | /| 1 `. +---+----+ / +-------+ `.-----------+ ,-------. | RFC | / |Check for | ,' was RFC `. YES | | / |Equivalence..... clause x -------+ | |/ +-------+ |under | `. clear? ,' | | Metric \.....| 2 ....relevant | `---+---' +----+---+ | Metric |\ +-------+ |identical | No | |Report | | Metric | \ |network | +--+----+ |results+| | ... | \ |conditions | |Modify | |Advance | | | \ +-------+ | | |Spec +----+RFC | +--------+ \| n |.'+-----------+ +-------+ |request | +-------+ +--------+

4

Key Points (the sub-points) Start with an RFC

Focus on a specific clause Run test(s) with Implementations

Test plan is customized to a specific clause Evaluate Measurements & Compare

Expected measurement results are Clear Obvious place to take action if text is found to be

ambiguous Final state is Report Dev. for Protocol Action Req.

5

Test Set-up (more details in I-D)

Network Emulator

NetProbe MS-1

NetProbe MS-2

NetProbe MS-3

NetProbe MS-4

eth2 10.201.0.254

eth3 10.202.0.254

eth4 10.203.0.254

eth5 10.204.0.254

negotiated 100baseTx-FD

10.201.0.1

10.202.0.1

10.203.0.1

10.204.0.1

ManagementLAN

Test LANs

6

NetProbe 5.8.5 Runs on Solaris (and Linux, occasionally) Pre-dates *WAMP, functionally similar Software-based packet generator Provides performance measurements

including Loss, Delay, PDV, Reordering, Duplication, burst loss, etc. in post-processing on stored packet records

7

Report Template – Loss Threshold 3.1. One-way Delay, Loss threshold, RFC

2679 (procedure) 3.1.1. NetProbe Lab results for Loss Threshold 3.1.2. XXX Lab Results for Loss Threshold

<your compliant implementation here> 3.1.3. Conclusions on Lab Results for Loss

Threshold

8

Section 3.1 – Loss Threshold See Section 3.5 of [RFC2679], 3rd bullet point and also

Section 3.8.2 of [RFC2679]. 1. configure a path with 1 sec one-way constant delay 2. measure (average) one-way delay with 2 or more

implementations, using identical waiting time thresholds for loss set at 2 seconds

3. configure the path with 3 sec one-way delay (or change the delay while test is in progress, measurements in step 2)

4. repeat measurements 5. observe that the increase measured in step 4 caused all

packets to be declared lost, and that all packets that arrive successfully in step 2 are assigned a valid one-way delay.

Results: 22 of 38 packets were declared lost (post-process).

9

Section 3.2: First-bit to Last-bitSee Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330].

1. configure a path with 1000 ms one-way constant delay, and ideally including a low-speed link (10-baseT, FD)

2. measure (average) one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 32 octet IP payload)

3. maintain the same path with 1000 ms one-way delay 4. measure (average) one-way delay with 2 or more implementations,

using identical options and equal size large packets (e.g., 1400 octet IP payload)

5. observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

10

3.2: First-bit to Last-bit, NetProbe

1400 byte payload 32 byte payload Delay for each mode (one mode) Delay Diff Expected Diff microseconds microseconds microseconds microseconds

1001621 1000356 1265 1094.4 1002735 1000356 2379 1094.4

TxDelay(us)

1000500

1001000

1001500

1002000

1002500

1003000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58

11

3.2: First-bit to Last-bit, NetProbe

Bi-modal delay behavior in the Network Emulator for UDP payload 96 octets & up Same for Poisson, Periodic, 1 sec, 10ms spacing Not evident in tests on the Management LAN

Delay increased with larger payload, but more than expected (170 us for low mode)

Suggestions to investigate further are welcomeTests on Management LAN, 100baseTx-FD 1400 byte payload 32 byte payload Ave Delay Ave Delay Diff Expected Diff microseconds microseconds microseconds microseconds

443.37 143.57 299.8 109.44 (190.36)

12

Other Examples One-way Delay, RFC 2679

This test is intended to evaluate measurements in sections 3 and 4 of [RFC2679].

Average delays before/after 2 second increase

Average pre-increase delay, microseconds 1000276.6 Average post 2s additional, microseconds 3000282.6 Difference (should be ~= Y = 2s) 2000006

Error Calibration, RFC 2679 This is a simple check to determine if an implementation reports the error

calibration as required in Section 4.8 of [RFC2679].

13

Summary Key clauses of RFC 2679 evaluated in Lab One Implementation: NetProbe Other volunteers? Comments on approach to reporting?

draft-morton-ippm-advance-metrics-01

What other RFCs should we target first?