Automated Selection and

Robustness

For Systematic Trading Strategies

Dr Tom Starke

• Traditional approachParameter optimisation on train set

Validation on test set

This is what we can do now:System Parameter Permutation

Supervised compound metric optimisation

Domain Regression

Step-Down Correlation Minimisation

Strategy Selection Automation

Walk-forward optimisation

Traditional Approach

Train Data

Test Data

Bad result - bad luck

Is there another way?

Too many parameters is bad news

for systematic trading strategies

But what is too many?

• Data mining bias very real

Overfitting is hard to detect

Solution:

Use all available backtest data

System Parameter

PermutationEvaluation all possible outcomes*

Turning Data Mining from Bias to Benefit Through System Parameter Permutation

Mean should

be positive

PnL Distribution of all backtest resultsPitfall: parameter space has unreasonable bounds

Domain Regression:The train performance informs the test

performance.

Train performance informs test performance

Correlation of results from backtest of train and test sets

BAD!

• Picking a single parameter set may not

produce very good results

Selecting several of the best strategies increases

the probability of arriving closer to the

regression line

Step-Down Low-Correlation Approach

Parameter sets should be as uncorrelated as possible

One More:

Use random subsamples instead of fixed test

period

Train

Test

Randomised test sets

Estimating PnL Shortfall

BRAC*- Build,Rebuild and Compare

Out-of-sample performance scales with the

mean of the PnL distribution of N

subsamples.

*Building Reliable Trading Systems - Keith

Fitschen

Choosing subsamples for

BRAC test

Mean PnL shortfall scales with OOS performance

Performance of full data set

Adjusted mean PnL of

subsamples

Bootstrapping

Expected

OOS PnL

Compound Metrics

Manually supervised ML

Optimisation: Monte-Carlo vs Grid Search

GA’s may not give the full story.

Walk-forward optimisation

System Parameter Permutation Profitability

Domain Regression OOS Expectation

Step-Down Selection Optimising Outcome

BRAC PnL Shortfall

Walk-Forward Optimisation Algorithm

Human-Supervised Metric

ConstructionOutput Metric

Bootstrapping PnL Shortfall

Questions?