Trading Systems:Testing at the Confluence of FT & NFT

Alexey Zverev, Managing DirectorAlyona Bulda, QA Project LeadIvan Bobrov, HFT Analyst

Introduction

1. Introduction2. Our topic - how to find / reproduce "toxic issues" which can happen under

the load or in real live environment conditions3. We will talk about:

a. A classical approach where FT and NFT are separate streamsb. What is missed herec. Some approaches we use or may apply to fill this gap between FT and

NFT (applying to trading systems)4. Our presenters

FTfocus on:

• Test scenarios are based on functional requirements

• Test tools which allow to run simplistic tests such as to send an order -> to verify a response

• Prepare a functional library starting from simple scenarios and adding more complex ones step by step

• Test scenarios are automated

• A test Library is enhanced as we proceed with its testing and production clients scenarios

• An automated library will be reused as a regression test library which is executed against each release after go live and can be taken for consequent projects

NFTfocus on:

• Testing is based on non-functional requirements

• Load scenarios are driven by the technical characteristics in PROD.

• The main target of the tests is to assess non-functional requirements: latency, capacity and fault tolerance

• A lot of efforts are invested in correct latency measurements and determining conditions when the system breaks

• Further regressions are done to compare system performance indicators with the previous releases

What do we miss when we keep FT and NFT separated ?

- Any complex system must sustain a certain level of

concurrency. Some issues appear when several events happen

simultaneously (Race conditions).

Example 1: Clients send a CxlReplace request and approx. at the same

time

the ME sends a fill back.

These 2 events meet at

the Order Management

server, and the state of

the order can get corrupted.

What do we miss when we keep FT and NFT separated ?

- Production order books and various system object states are

more complex than in our FT scripts, and no amount of test cases

can cover all possible scenarios

Example 2: A User tries to

hit two different implied

orders coming from the

same parent order

which is being restated

in the middle of the process.

What do we miss when we keep FT and NFT separated ?

- Components may behave differently under the load

Example 3: The System must support various

throttling limitations and the requirements of

the system behaviour. The correctness

of functional system behavior can be

tested only under the load.

- Integer memory overflow

Example 4: Turnover (VWAP) fields in the statistics messages and

incremental OrderID overflow;

* the picture was taken from https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcS9nlDdLAmuvmyRqqZsanbyfwL601nJlOvtHGEfJBS4DKnmSTANUA

What do we miss when we keep FT and NFT separated ?

- Dynamic Mass events which happen during

DLC

Example 5: Market Open,

Market Close, Auctions,..

The state of the system

at exact instance

when the system goes

from one session to another

should be verified with the load

applied against the system.

Detailed Verification Of The System Behavior Under Random Load

We need to make sure that we provide extensive coverage We need to make sure that the load test mimics what happens in the production system

We need to verify that the system works as expected

- basic consistency check - market data must be internally consistent- order book should be available and not be different

+ Recreation of exact steps of real production issues

- It is difficult to record multiple network streams

Performing Functional Testing Against The System Under The Load

to leave a part of the system intact by the load

to adjust functional tests to cope with the activity of the load

to make scripts in case when we perform tests actively interacting with the load * scenarios must be flexible* sustain interaction with the script

Testing With The Baseline

The example of the cycle of a Clearing System

A detailed, step by step analysis

The “Mini Robots” Solution

The “Mini Robots” Solution

Questions & Answers-tO

Thank You-tO