sean mcclure - approaching real-time business intelligence - trading at the speed of light

www.excellerate4success.com

Approaching Real-Time Business Intelligence

Trading at the Speed of Light

Sean McClure, Ph.D.

Business Analytics, Excellerate Inc.

[email protected]

http://www.excellerate4success.com


Overview

Introducing Excellerate

Real-Time BI and HFT

Information at High Frequency

Strategies at High Frequency

Developing and Deploying Models

Executing and Monitoring Real-Time

Systems

Summary

Big Data

Data Mining

Metrics

Themes

Meta Data

Prediction



About Us!

Business Intelligence Service Providers

• Dedicated to bringing top quality business

intelligence expertise to successful

growing organizations (SGOs);

• Aggressively researching industry best

practices and best-in-breed software tools

to deliver high-end analytics and data

mining expertise;

• Business Intelligence model supported by

Subject Matter Experts (SMEs) in key

business areas.



• Data latency: time taken to collect and store the data;

• Analysis latency: time taken to analyze the data and turn it

into actionable information; and

• Action latency: the time taken to react to the information and

take action.

• Real-time business intelligence technologies are designed to reduce all three

latencies as close to zero as possible;

• Traditional BI only seeks to reduce data latency.

Real -Time Business Intelligence

Defining “Real-Time”

Three types of latency1:

Approaching “zero” latency



Debit and Credit Fraud Detection

Marketing

Inventory Control

Supply-chain Optimization

Customer relationship management (CRM)

Dynamic pricing and yield management

Data validation

Call center optimization

Transportation industry

Operational intelligence and risk management

Real Time BI in various industries

Finance (biggest candidates)

Real -Time Business Intelligence

1

2

3



Traditional long-

term Investing

High-

frequency

Trading

Algorithmic/

electronic trading

Low

High

Ex

ec

uti

on

La

ten

cy

Position Holding Period Short Long

High Frequency Trading (HFT)

• Trading platform that uses

powerful computers to transact

a large number of orders at

very fast speeds;

• Uses complex algorithms to

analyze multiple markets and

execute orders based on

market conditions;

• Traders with fastest execution

speeds will be more profitable

than traders with slower

execution speeds (arbitrage

opportunities).

In the U.S., high-frequency trading accounts for

~73% of all equity trading volume5

Best Case Study for “Real-Time” Intelligence

Figure 1



• Timestamp

• Security ID

• Bid Price

• Ask Price

• Available bid volume

• Available ask volume

• Last trade price

• Last trade size

• Option-specific data

Date/time quote

originated (>20ms)

Price at which the last trade in the

security cleared

Highest price available for

sale of the security

Lowest price entered for

buying the security

Provided by other

market participants

through limit orders

Total demand

Total supply

Properties of Tick Data – Quote, Trade, Price and Volume Information

High Frequency Information

Actual size of the last

executed trade




Recent microstructure research and advances in econometric

modeling tell us there are unique characteristics to tick data.

irregularly spaced

quotes arriving randomly

very short time intervals

(low-frequency data is opposite)

Irregularities provide a wealth of information not available in low-

frequency data. Inter-trade durations may signal changes in market

volatility, liquidity, and other variables.

Volume of data allows for statistically precise inferences.

Number of observations in a single day of tick data = 30 years of daily

observations

time

Tick Data



Modeling the Arrivals of Tick Data creates a host of opportunities not

available at low-frequency

• Time distance between quote arrivals carries information

quote processes

trade processes

price processes

volume processes


time

Duration models

Estimate the factors affecting the

duration between ticks

Low Trade

Duration

Higher likelihood

of unobserved

good news

High Trade

Duration

Higher likelihood of

unobserved bad

news

Low Price

Duration

Increased

Volatility

Low Volume

Duration

Increased Liquidity

Absence of Trade

Lack of news, low levels

of liquidity, trading halts,

trader motivations




Minute 1 Minute 2 Minute 3 Minute 1 Minute 2 Minute 3

Most modern computational techniques have been developed

to work with regularly spaced data (easy to process)

High frequency data-sampling methods developed to

overcome irregularities in tick data by sampling at

predetermined periods of time

Figure B Figure A

Traditional Approach Linear Time-Weighted Interpolation

Data sampling methods overcome irregularities in high-frequency data

for ease of processing

lasttt qq ,ˆ

lastnext

lastlasttnexttlasttt

tt

ttqqqq

)(ˆ

,,,quote



The price of the security in the inefficient market begins adjusting before/after

the news becomes public ( “information leakage” and “overshooting”)

Many solid trading strategies exploit both the information leakage and

overshooting to generate consistent profits

Information Arrival Time Information Arrival Time

Good News Bad News

Inefficient market response

Efficient

market

response

Efficient market response

Inefficient market response

Incorporation of information in efficient and inefficient markets

Security Price Adjustments to Information




High-Frequency Strategies

Traders leverage state-of-the-art IT technology to implement trading

strategies that have high-frequency opportunities;

High-frequency trading strategies typically fall into four main categories.

HFT-based Strategies

Electronic

Liquidity

Provision

Statistical

Arbitrage

Liquidity

Detection

Others

Spread

Capturing

Market Neutral

Arbitrage

Sniffing/Pinging/

Sniping

Latency

Arbitrage

Rebates

Cross Asset,

Cross Market &

ETF

Quote Matching Short Term

Momentum

Trading on High-Frequency Information




- Spread Capturing

Liquidity providers profit from the spread between bid and ask prices by

continuously buying and selling securities;

Executed predominantly using limit orders

High-speed transmission of orders and

low-latency execution required for

successful implementation of liquidity

provision strategies.

Liquidity Provision Strategies

Bid Offer Price

Asking Price

Market Price Bid-Ask Spread

Ask

Limit Buy Orders Limit Sell Orders

Market Buy Orders Market Sell Orders

Market Transactions




Predictions based on the Limit Order Book

Direction of market price

movement

buy sell

Direction of market price

movement

buy sell

• Shape of limit order book is

predictive of impending

changes in market price;

• Exploited by market-maker

traders;

• Depends on probability

distribution for arriving market

orders;

• Shape can be estimated when

book not observable.




Statistical Arbitrage

Leverages states of the art technology to profit

from small and short-lived discrepancies

between securities;

Arbitrageurs generate profits by selling the

asset on the market where it is valued higher

and simultaneously buying it on another

market where it is valued lower.

“Stat-Arb” rests squarely on data mining. It finds

statistical relationships in large amounts of data

and builds a model of those relationships;



Identify securities

that trade in

frequency unit

Measure difference

between prices of

identified securities

Select most stable

relationships

TtSSS tjtitij ,1,,,,

2

1

,,

min

T

t

tijji

S

Estimate

distributional

properties of the

difference

T

t

tt ST

SE1

1

T

t

ttt SEST

S1

2

1

1

Monitor and act

upon differences

in security prices

SSESSS jit 2,,

SSESSS jit 2,,

Once gap in prices

reverse, close out

position/stop loss

Detecting Statistical Anomalies in Price Levels




Asset Class Fundamental Arbitrage Strategy

Foreign Exchange Triangular Arbitrage

Foreign Exchange

Uncovered Interest Parity (UIP)

Arbitrage

Equities Different Equity Classes of the

Same Issuer

Equities Market Neutral Arbitrage

Equities Liquidity Arbitrage

Equities Large-to-Small Information

Spillovers

Futures and the Underlying Asset Basis Trading

Indexes and ETFs Index Composition Arbitrage

Options Volatility Curve Arbitrage

Fundamental Arbitrage Strategies by Asset Class




• Modeled relationships tested on lengthy

spans of tick data

• Forecasting validity

• Various market situations

Model Development

Back Testing

Model Development/Deployment

• Linear Econometric Models

• Autoregressive (AR) Estimation

• Moving Average (MA) Estimation

• Autoregressive Moving Average (ARMA)

• Cointegration

Volatility Modeling

• To model observed volatility

clustering = ARMA or GARCH

NonLinear Econometric Models

Allows for modeling of complex nontrivial

relationships in data

• Taylor series expansion

• Threshold autoregressive model

• Markov switching model

• Nonparametric estimation

• Neural Networks

Models used in HFT

• Academic research and

proprietary extensions

• Modeling predominantly

in Matlab /R,

• c++ for back-tests and

transition into production

Ideas

Tools



Model Accuracy Analysis

Model Development/Deployment

Point Forecasts • predict price will reach certain

level /point

• regression of realized values

from historical data against out of

sample forecasts

Directional Forecasts • makes decisions to enter into

positions based on expectations

of system going up or down

(without target)

Accuracy Curves • compares the accuracy of

probabilistic forecasts

• HFT models done with TSA

curves

Miss Rate (%)

Hit

Rate

(%

)

Model A

Model C

Model B

Random Forecast

0.0

100

100 %

Back-Testing Econometric Models

Accuracy Curve



best way to route the order to the exchange

best point in time to execute a submitted order (non-market order)

best sequence of sizes in which the order should be optimally processed

• Algorithms spanning order-execution processes

• Designed to optimize trading execution once the buy-

and-sell decisions have been made elsewhere

Executing Real-Time Systems

1) Market Aggressiveness Selection algorithms designed to choose between market

and limit orders for optimal execution;

1) Price-Scaling algorithms designed to select the best execution price according to

the pre-specified trading benchmarks; and

1) Size-optimization algorithms that determine the optimal ways to break down large

trading lots into smaller parcels to minimize adverse costs (cost of market impact)

Execution Optimization Algorithms

Common Types

1)

2)

3)




Market Aggressiveness

Selection

Price-Scaling Size Optimization

• Balances passive and

aggressive trading using

optimization

)()(min

RiskCost

)()(),()(

))(()(

)()(

XgXfPP

Risk

PPECost bo

)(

),(

)(

)(

Xg

aXf

P

a

aP

Pb

Benchmark execution price

Realized execution price

Deviation of trading outcome

Market price at order entry

Temp impact due to liquidity

Price impact due to info leak

• Tries to obtain the best

price for the strategy

2,1 )(min tbttt PPEt

tb

t

tt

P

P

,

1 )(

Realized price

Trading aggressiveness

Benchmark price

Strike Algorithm

Plus Algorithm

Wealth Algorithm

• Minimizes the cost of

execution relative to a

benchmark

• Designed to capture gains in

periods of favorable prices

• Tries to trade with

position undetected

• Large order packets are

broken up for least

amount of market impact

(“Stealth Trading”)

Execution Optimization Algorithms



Receive/archive real-

time tick data on

securities of interest

Apply back-tested

econometric models to the

tick data obtained in 1

1

2

3

4

5

6

Send orders and keep track of

open positions/P&L values

Monitor run-time trading behavior,

compare with predefined parameters,

manage the run-time trading risk

Evaluate trading

performance relative to

predetermined benchmarks

Ensure trading costs incurred

during execution are within

acceptable ranges

1 – 4: run-time

5 – 6: post-trade

Each functions built with

independent alert systems

that notify monitoring

personnel of problems,

unusual patterns etc.


HFT Business Cycle



Summary

Introducing Excellerate

Real-Time BI and HFT

Information at High Frequency

Strategies at High Frequency

Developing and Deploying Models

Executing and Monitoring Real-Time

Systems

Big Data

Data Mining

Metrics

Themes

Meta Data

Prediction



Sean McClure, Ph.D.

Business Analytics, Excellerate Inc.

[email protected]

Thank You



References

1) Richard Hackathorn, "Active Data Warehousing: From Nice to Necessary," Teradata Magazine (June 2006), AR-4835

2) cdn.avangate.com/web/images/articles/fraud-lock.jpg

3) genesissolutions.com/wp-content/uploads/2009/10/3.3.3-MROSupply-307x195.jpg

4) partnerc.com/images/iStock_000007068822Small.jpg

5) http://www.economist.com/node/5475381?story_id=E1_VQSVPRT


sean mcclure - approaching real-time business intelligence - trading at the speed of light

Documents

data latency

business intelligence

trade price

data analysis latency

business analytics

high frequency strategies

realtime intelligence

ms price