On-Site Training PreviewProof of Concept Test

Overview

Deck Overview

• This is a sample deck from a two day instructor led course.

• The course consists of presentations, labs and in class discussion.

• Each course is tailored for the end user and generally only applies to them.

• For more information visit the Router Analysis website or email Steve Noble

• All materials are © Router Analysis / SONN

Section 1 Part A

What is a Proof of Concept Test?

What is a Proof of Concept Test?

• A PoC is a test plan designed to prove out a design. You normally call a PoC when you have reached a point in the bidding process where you have > 1 vendor who appears to meet your requirements.– A PoC is used to compare N vendors to see that they fit a set of

requirements the customer has listed.

– A PoC is used to test each vendor separately and compare the results with the RFP information they were given.

– A PoC can be used to help lower the price.

What is a Proof of Concept Test?

• It is important to do a Proof of Concept test (PoC). Here are some concepts of a good PoC:– Note: PoCs can go by other names, bake off, bench mark test, etc.

– A well designed PoC will allow you to choose not only the better vendor, but be more secure in your choices.

– The PoC should be set up in a way that every vendor is able to be directly compared.

The Rules of a PoC

Section 1 Part B

Rule #1Always Discuss and Set the Rules of the PoC

PoCs require strict guidelines for each vendor to follow. Why?

• To be able to directly compare results.

• To keep everything on the level.

• To make sure the PoC is run fairly and you get the right information.

Rule #2Keep Good Communication with the Vendors

During the Tests

• You will need to grade and keep notes about the tests as they progress to be able to directly compare results.

• Vendors will need to keep your own notes and compare them with you after each test and at the end of the day.

• This helps vendors to develop a document trail of where they passed/failed and if you are owed any updates after the PoC.

Rule #3Empower Yourself in the Ways of Proper Testing

• You need to drive at the PoC including system configuration.

• Tests should be based on a 3-5 year plan if possible.

• Tests should be combined with each other rather then doing stand alone testing.

• Vendors should be required to submit one version of code to be used for all testing.

Rule #4Stay Aware During Testing

• Take notice of any configuration changes

• If there are hardware or software issues, ask for tracking information

Test Plans and Practical Issues

Section 1 Part C

Building Test Plans

• The test plan should include clear rules on how the testing will be executed, what equipment is to be provided and the code expected to be on the SUTs.

• The testing should be done using a standard IMIX (such as the Light Reading IMIX or Agilent’s IMIX*) not one provided by a specific vendor which favors that vendor.

• The test cases should include both bandwidth intensive and PPS intensive tests, not just one or the other.

• The test plan should consist of combined tests; i.e. tests that build on-top of each other. Based on your current topology.

• Utilize general test cases available from test vendors such as the Agilent Journal of Test Methodologies.

Communicating With Vendors

• Discuss the test plan

– Work with the vendor to understand common testing issues

• Combining tests in a reasonable way

• Avoiding the pitfalls of loosely defined test cases

– Attempt to define what the grading criteria will be

• Tell the vendor what the correct amount of equipment is

– Base the hardware on the testing equipment that will be available

– Make sure to specifically define the types of interfaces I.e. 10GE LAN

• Explain the value of the tests you are running

– Explain how your tests are based on your real world goals and expectations

Practical Issues

• Many vendor representatives do not have experience doing

proper system testing

– Confirm the type and number of tester interfaces available

• If the testing equipment is limited scale tests, involving traffic will not be very

useful.

– Check that the vendors will be able to provide the number of interfaces necessary, if not

try to come up with ideas to work through the issue.

– If the testing is being done on-site, ensure that the testing site is prepared to provide

power and HVAC for the equipment provided.

Practical Issues Continued

• It is important to understand if a test has passed either fully

or conditionally

– When doing scale tests define solid numbers. E.g. 1k IS-IS L1 Routes

– When doing traffic tests work to avoid running interfaces at 100% as issues such as byte-

stuffing and overhead can cause issues.

– When combining tests define them as including the last test and be sure to run

background traffic from the previous tests to confirm a full pass.

– When defining an access list, be sure to provide information about the type such as

simple or extended (source/dest ip, source/dest ip/port).

– Set time limits, number of re-tries allowed for a test and the time allowed between re-

tries before a fail is called.

– Define importance to the tests so that vendors know what is required and what the

minimum pass is.

When Things Go Wrong

… and they will

Section 1 Part D

Why Things Go Wrong

• There will always be issues when tests are being done, most issues revolve around improper configuration of either the SUT or the tester.– To troubleshoot follow the following steps:

• Confirm that the System Under Test is connected to the correct ports on the tester and other devices.

• Confirm that the ports are all up and functional.

• Confirm that all necessary features are configured on both the SUT and the tester.

• … More information in the full training.

Issues Completing Tests

• There are many reasons why a test may not be completed

– Time runs out

• Sometimes there is not enough time to do everything the you want. By making

sure you understand the value of the different tests you can determine whether to

continue or just move on.

• Sometimes it may take much longer to configure/debug a test than you expect

– Hardware/Software issue

• Based on the rules of the BMT you will have different options such as allowing a

repeat at a later date, within the next x hours, or giving a fail.

Thank You!