N-tuple S&P Patterns Across Decades, 1950s to 2011
A. G. (Tassos) Malliaris
Mary MalliarisLOYOLA UNIVERSITY CHICAGO
The Korea Institute of Finance and
The Athenian Policy Forum Conference
Seoul, Korea
June 16-18, 2013
Purpose
• To investigate the Up and Down movements of the S&P 500 from 1950 through 2011
• And to use that information to forecast the direction tomorrow of the S&P 500
Data
• Daily closing prices from 1/3/1950 to 7/19/2011
• Over 15,000 observations
• were transformed into Up [U] or Down [D] by comparing today’s value to yesterday’s
• Up and Down movements were recorded from 1 up to 7 days
Up [U] and Down [D] movements per decade, 1950 through 2011.
Two-Day Patterns Across Decades
Three-Day Patterns
Four to Seven-Day patterns
• Strings were formed, for example, UDDU through DUDUDDU
• From 16 to 128 patterns [too many to display on a slide]
• Columns also created counting the number of Up movements within a string
Number of Ups in Two Days
Number of Ups in Three Days
Number of Ups in Four Days
Number of Ups in Five Days
Forecasting
• Training set: data from January 1950 through December 2009 [15,087 rows].
• Validation set: January 2010 through mid-September 2011 [387 rows]
• Training set patterns were used as inputs for models that were then used to predict the Validation set
Forecasting Methodologies
• Pattern Forecasts with 1 to 7 Days
• Decision Tree
• Neural Network
• Random Forecasts
The Forecast Decision by Pattern, an Example
Training set 4-day patterns
Count
DDUD 760
DDUU 1050
Validation Set Pattern Up through Today
Most Likely Move Tomorrow
DDU U
Decision Tree Forecasts
• Decision Tree Methodology: C5.0
• Software Package: SPSS Modeler
• Target: Tomorrow’s Direction
• Inputs: : the up-down patterns from one to seven days, the number of up days in 1 to five days, and the closing value today
Decision Tree Variable Importance• This Modeler technique also ranks the input
variables in terms of importance, with more important variables occurring higher up on the tree.
• The variables ranked highest in importance to the forecast were
the direction today
the closing value today
the 7-day pattern [for example UDUUDDU]
number of Up movements in the last three days.
Neural Network Forecasts
• Neural Network Methodology: Back Propagation; two hidden layers
• Software Package: SPSS Modeler
• Target: Tomorrow’s Direction
• Inputs included: the up-down patterns from one to seven days, the number of up days in 1 to five days, and the closing value today
Neural Network Variable Importance
• SPSS Modeler also ranks the input variables in terms of their importance to the neural network
• The variables ranked highest in importance with this methodology were
the patterns in the 4 day through 7 day strings
the number of Ups in the past three days
the closing value
Comparing Methods by Calculating Gain or Loss
Daily Gain/Loss Summed Over 387 Days
Change in Index
Actual Direction
Predicted Direction
Amount of Gain/Loss
-5 Down Down 5-5 Down Up -55 Up Down -55 Up Up 5
Random Forecast
• A random number was generated for each day in the validation set
• If the random value was less than .5 then Down was predicted, otherwise the forecast was Up.
• The amount of gain/loss was calculated over the Validation set
• This was repeated 100,000 times
RANDOM WALK SIMULATIONSTotal gains/losses for 100,000 simulated random forecasts for 387 days
Minimum -1028.62Maximum 1048.12Mean -0.0995Std Dev 235.4787
Number of correct directional forecasts for each methodology
Method Down UpNumber Correct
2 Day String 72 123 195
3 Day String 41 164 205
4 Day String 41 164 205
Decision Tree 43 168 211
5 Day String 45 167 212
7 Day String 48 164 212
6 Day String 40 178 218
1 Day String 0 219 219
Neural Net 61 161 222
Total amount gained or lost per strategy, and comparison to the random simulations
Type of ForecastTotal Amount Gained/Lost
Amount as Std Deviations in the Random
Distribution2 Day String -21.08 -0.087Decision Tree 168.16 0.7153 Day String 174.34 0.7424 Day String 174.34 0.7421 Day String 193.96 0.8257 Day String 306.92 1.3035 Day String 307.22 1.3056 Day String 335.84 1.426Neural Net 393.4 1.67
Conclusions• The number of up movements is greater than
down movements across the decades.
• Patterns of Up and Down movement from the past can be useful for forecasting future movement
• The highest gain from simple patterns came from using the 6-day strings
• The neural network, a more complex methodology, yielded the greatest gain