prequential analysis of stock market returns · 2003. 2. 18. · david a. bessler and robert...
TRANSCRIPT
May 2001
Prequential Analysis of Stock Market Returns
David A. Besslerand
Robert Ruffley
Abstract
The paper considers the Brier score and a covariance partition due to Yates to study theprobabilistic forecasts of a vector autoregression on stock market returns. Probabilistic forecastsfrom a model and data developed by Campbell (1991) are studied with ordinary least squares.Both calibration measures and the Brier score and its partition are used for model assessment.The partitions indicate that the ordinary least squares version of Campbell’s model does notforecast stock market returns particularly well. While the model offers honest probabilisticforecasts (they are well calibrated), the model shows little ability to sort events which occur intodifferent groups from events that do not occur. The Yates-partition demonstrates this shortcoming, while calibration metrics do not.
____________ Bessler is a Professor at Texas A&M University. Ruffley is a former graduate student inEconomics at Texas A&M University. Thanks are extended to John Y. Campbell for sharing hisdata with us. John L. King’s earlier association with Bessler is acknowledged as helpful for thispaper.
1
Prequential Analysis of Stock Market Returns
Given data which arrives in sequence, prequential analysis uses currently available data to
produce probability distributions on future observations (Dawid 1984) . Such a system is judged
on its forecasting ability and not on a priori grounds such as agreement with prior theory or
within sample goodness of fit. Dawid (1985) suggests that probability calibration be used to
judge the adequacy of probability forecasts.1 Kling (1987) applies prequential analysis to study
the distribution of turning points. Kling and Bessler (1989) use prequential analysis to model a
small macroeconomic vector autoregression. Covey and Bessler (1992) apply the technique to
test for Granger's full causality.2 These applications use calibration as the sole metric of
performance. Calibration is a test of whether an issued probability agrees with its relative
frequency, ex post. So, for example, if a prequential model issues a probability of .25 for one
hundred events, we should observe (ex post) that twenty five of these events occurred if this
model is to be labeled “well-calibrated.”
An alternative metric for evaluating probabilistic forecasts is the mean probability score,
otherwise known as the Brier Score (Brier 1950). The Brier score has received considerable use
in evaluating weather forecasts (see Murphy and Winkler 1977), but relatively little use in
economics. As one exception to this last statement we mention Zellner, Hong and Min (1991),
who use the Brier score to rank probability forecasts of turning points from various fixed and
time-varying-parameter models of aggregate output. The Brier score is a quadratic scoring rule
which has a rich history of use in motivation and evaluation of subjective probabilities, see
deFinetti (1937, 1965 and 1974) and Savage (1971) for theoretical developments on the
quadratic scoring rule and Nelson and Bessler (1989) for an empirical test of the optimal
2
property of this rule. We follow Kling and Bessler (1989) and Zellner, et al (1991) in suggesting
the use of optimal scoring rules for the evaluation of forecasts from econometric models.
One advantage of using a Brier score over calibration is that the Brier score can be
decomposed into components which index both calibration and resolution (sorting). In studying
financial data we may be interested in differences in probability forecasts assigned to events
(rates of return) that ultimately occur versus probabilities assigned to events that do not occur.
This “sorting” characteristic is not captured by calibration metrics. The Brier score thus gives
analysts more information (more than calibration measures) on the performance of a set
probability forecasts.3 Sanders (1963) provides one such partition. Murphy (1973) decomposes
Sanders’ resolution into an outcome variance index and an alternative measure of resolution.
Both the Sanders and Murphy decompositions work off of fixed probability vectors and thus
offer little, beyond usual calibration metrics, where forecasts are continuous (Kling and Bessler
(1989, pp.482-83)). Yates (1982) provides a covariance decomposition which applies to both
discrete and continuous probability forecasts. The Yates-partition is applied in this paper. We
are aware of no applications of the Yates-partition for evaluation of forecasts from financial or
econometric models.
This paper applies prequential analysis, using a standard calibration test and the Yates-
partition of the Brier score to two forecasting models of the U.S. stock market. As decisions
involving stock prices are inherently embedded in uncertainty, and as many if not most decision
theories require the entire probability distribution (e.g. expected utility theory), such methods
(not necessarily the one advocated here) are prima facie of interest. The models we consider
build on an earlier paper by Campbell (1991). First we study probability forecasts from an
ordinary least squares estimated version of Campbell's (1991) three variable vector
3
autoregression (we refer to this as the OLS-VAR) of stock market returns, the dividend price
ratio and short term interest rates. The second model entertained is a vector random walk in each
of the three variables from Campbell's model. The latter is of interest since it provides a
baseline set of probability forecasts which can be compared to the more substantive, knowledge-
based, forecasts from Campbell’s model. Our interest is not in assessing Campbell’s model in
particular, as the data are dated (some would say old) and not particularly relevant to up-to-date
or real time decision-making. Rather, our interest is to study probabilistic forecasts using the
Yates’ partition. By applying the results to Campbell’s model and his data, we provide readers
with a clear example of how such forecasts may be made and a clear indication of the type of
results they may expect to find in a well-designed econometric model (Campbell’s model).
An advantage of Campbell's model is that it incorporates the findings of Campbell (1987)
and Fama and Schwert (1977) that the level of short term interest rates helps forecast stock
market returns and the findings of Fama and French (1988) and Campbell and Shiller (1988) that
the dividend price ratio helps forecast stock market returns. We are particularly interested in
whether the additional information from the OLS-VAR results in improved forecasts of stock
market returns.
The outline of the paper is as follows. Section two provides greater detail on Campbell's
model and replicates his results. Section three discusses prequential data analysis, testing for
calibration and probability partitions (Brier Score and Yates-partition). Section four presents the
results. Section five concludes the paper.
Campbell's Model and Replication
Campbell's (1991) U.S. stock market model is a three variable, one lag, vector
autoregression consisting of real stock returns, the dividend price ratio and interest rates. The
4
data are measured monthly from 1926 thru 1988, with the first year being used for startup lags
for estimation. Following Campbell, the sample is broken into two subgroups, to take into
consideration the Federal Reserve Board-Treasury Accord. Prior to 1951, the Federal Reserve
Board held interest rates fairly constant. After 1951 the Board allowed rates to move more
freely. Thus, the model is run over three periods: (1) entire sample: January 1927 through
December 1988; (2) Pre-Treasury Accord period: January 1927 through December 1951; and
(3) Post-Treasury Accord period: January 1952 though December 1988.
The stock return series (h t) is the log of the real stock return over a month where the real
stock index is measured as the value weighted New York Stock Exchange Index for the CRSP
tapes deflated by the consumer price index. The interest rate series ( r t ) is the one-month
Treasury bill rate minus a one-year backward moving average. The dividend price ratio (d/p t ) is
the ratio of total dividend paid over the previous year divided by the current stock price.
Justification for including these variables in a stock market return model is given in
Campbell(1991). He also estimates a monthly VAR with six lags and a quarterly VAR. This
paper focuses on the one lag VAR on monthly data.
To ensure that the analysis in this paper is consistent with Campbell's analysis, we first
replicate his results, which are shown in table one of his paper (Campbell 1991, page 166).
Instead of using Generalized Method of Moments (GMM) this paper takes the route of using
ordinary least squares (OLS). The drawback of using OLS versus GMM is that the latter
produces a heteroskedastic consistent variance-covariance matrix, while OLS does not. The
reason we used OLS is that ordinary least squares provides an easy form for recursive
forecasting and coefficient updating using the Kalman filter. All of the estimation is carried-out
using RATS (Doan) software.
5
Table 1 contains the results produced by OLS. For the entire sample period and pre-
Treasury accord period, the coefficients are almost identical with Campbell's table 1. For the
post-Treasury accord period, the estimated coefficients differ slightly, but differ by no more than
.01 in any one case. However, we see considerable disagreement in the estimated standard
errors, with no consistent pattern of over or under estimation. Based on these results, the
findings of our paper do not necessarily reflect on Campbell's model; rather they reflect on our
OLS version of Campbell’s model.
Probability Forecasting
Let xt N = (x1t ,...., xmt), t=1,..., n be observed values of the mx1 vector time series Xt. At time n,
given known values xt , t = 1, . . . , n, a set of probability distributions Pn,k = (Pn+j ; j=1,..., k) for
unknown quantities xn+j , j=1,..., k are issued. A rule P which associates a choice Pn,k with each
value of n and any possible set of outcomes xt , t = n+1,..., n+k is a "prequential forecasting
system" (PFS) (Dawid, 1984). A prequential forecasting system is judged as "good" or "bad"
through the sequence of probabilities it actually issues and subsequent outcomes and not
through a priori considerations, such as agreement with theory or goodness of fit.
To judge the adequacy of prequential probabilities, Dawid (1985) proposed using
probability calibration. A PFS is said to be well-calibrated if the ex post relative frequency of all
events whose probability is P* is in fact P*. For example, a well-calibrated PFS should plot along
a 45 degree line with the relative frequency on the y-axis and issued probability on the x-axis.
If the xi,t+k are continuous random variables with continuous distribution functions Fi,t+k,
the random fractiles Ui,t+k = Fi,t+k (X i,t+k ), t=1..n, are independent uniform (U[0,1]) random
variables (Dawid, 1984). If the Xi,t+k are discrete with cumulative distribution functions Fi,t+k,
6
then the random fractiles Ui,t+k also have distribution functions of the form G(ui,t+k) = ui,t+k, though
the functions are not continuous. In either case the assessment of the PFS reduces to a test of
hypothesis that the observed sequence ui,t+k = F i,t+k (xi,t+k) is from a probability distribution with
the cumulative distribution G(ui,t+k)=ui,t+k. If this hypothesis cannot be rejected, then the PFS is
considered to be well-calibrated.
The estimated cumulative distribution function G(i,t+k) for Ui,t+k, is obtained by taking the
observed sequence ui,t+k = Fi,t+k (xi,t+k), t=1..n, ordering the sequence from low to high
ui,k(1),..,ui,k(n) and calculating
G ui,j(j) ' (j/n); j ' 1,..,n. (1)
The empirical cumulative distribution function is referred to as the "calibration function" (Bunn,
1984). For a well-calibrated PFS the calibration function should "look like" a 45-degree line.
A test of calibration can be made by testing the observed fractiles (ui's) from the sequence
of probability forecasts Pt,k. If there is a sequence of n such forecasts, then under the null
hypothesis (well calibration), any subinterval of length L (where 0# L# 1) will have n*L
observed fractiles. If there are J non-overlapping sub-intervals that exhaust the unit interval,
then a chi-squared goodness of fit statistic can be applied:
χ2 ' jJ
j'1
(aj & Ljn)2
Ljn- χ(J&1) . (2)
where aj is the actual number of observed fractiles in the interval j and Lj is the length interval
j. Under weak conditions, not requiring independence for the distributions underlying the
7
forecasts and under the null hypothesis of calibration, the test statistic will be distributed as chi-
squared with J-1 degrees of freedom (Dawid (1984)).
The Brier Score
Evaluation of a prequential system by way of its calibration property is not the only
possibility open to researchers (see the recent paper by Diebold, Hahn and Tay and the
prequential papers listed in our introduction for papers which focuses on calibration). The
probability score introduced by Brier (1950) is an alternative metric which has received
considerable attention in the literature. The Brier score considers calibration and resolution. As
resolution is a measure of a model's ability to "sort" or partition uncertain events into subgroups
which have probability measures that differ from long-run relative frequencies, it should prove
helpful in econometric applications.
Below we summarize known results on the Brier Score. We do so first for the case of
single event trials -- the event (say A) obtains or it does not obtain. We then present the
generalization to multiple event trials -- at each time t one of k possible events obtains. Our
source on this section is Yates (1988).
Let f represent the probabilistic forecast for an event that the forecaster is trying to
predict. Let d represent the outcome index where:
d = 1, if the event occurs
d = 0, if the event does not occur.
The Brier Score (1950) or probability score is then represented, for the single forecast case, as:
PS(f,d) ' (f & d)2. (3)
8
PS reaches a minimum value of 0 when the forecast is perfect (f=d=1). PS is maximum at 1 when either
the forecaster is absolutely certain that the event will occur when in fact it does not occur (f=1,d=0) or
the forecaster is certain that the event will not occur and, in fact, it does occur (f=0,d=1).
Over N occasions, indexed by i=1, . . . , n, the mean of PS is given by
PS(f,d) ' 1N j
N
i'1(fi & di)
2 . (4)
Sanders (1963) and Murphy (1972 and 1973) have decomposed PS into various components including
measures of calibration and resolution.
Yates (1982) further decomposed PS allowing for additional analysis. His formulation, called
"covariance decomposition" is given as:
PS(f,d) '
Var(d) % MinVar(f) % Scat(f) % Bias 2 & 2(Cov (5)
Var(d) represents the variance of the outcome index and defined as:
Var(d) ' d(1 & d) (6)
where
d '1N j
N
i'1di . (7)
Var(d) represents the factors of forecasting which are out of the forecasters control. That is, it represents
9
the base rate in which the target A occurs. The remaining terms in (5) reflect factors that are under the
forecaster's control. Thus, the forecaster wants to minimize MinVar(f), Scat(f) and Bias2 and maximize
Cov(f,d) to obtain the lowest PS.
The Bias is defined as
Bias ' f & d (8)
where
f ' 1N j
N
i'1fi . (9)
Bias is labeled "calibration in the large" or the mean probability judgement. It reflects the overall mis-
calibration of the forecast, i.e. how much the probability assessments are to high or too low. Thus, Bias2
is the amount of calibration error regardless of the direction of error.
The Cov(f,d) term is defined as
Cov(f,d) ' slope var(d) (10)
where Slope is defined as:
Slope ' f1 & f0 (11)
and
10
f1 '1N1
jN1
j'1f1j (12)
f0 '1N0
jN0
j'1f0j . (13)
f21 represents the conditional mean probability forecast for event A over the N1 occurrences for which
the event actually occurs. f2 0 represents the conditional mean probability forecast for event A over the
N0 occurrences that the event does not occur. The maximum value of Slope is 1 which occurs when the
forecaster always reports f=1 and the event does occur and f=0 and the event does not occur. Cov (f,d)
reflects the model's ability to make distinctions between individual occasions in which the event occurs
or does not occur. So, covariance is at the heart of the forecasting problem (Yates(1988)).
Scat(f) is given by:
Scat(f) ' 1N
N1Var(f1) % N0Var(f0) (14)
where
Var(f1) '1N1
jN
j'1(f1j & f1)
2 (15)
and
11
var(f0) '1N0
jN
j'1(f0j & f0)
2 (16)
Var(f1) is the conditional variance of the probability judgements for event A on those N1 times when A
actually occurs. Var(f0) is the conditional variance of the probability forecasts for event A on those N0
times when the event does not occur. Scat(f) is a weighted average of Var(f0) and Var(f1). It can be
interpreted as an index of overall scatter or noise contained in the forecaster’s probability statements.
MinVar(f) is the minimum forecast variance defined as:
MinVar(f) ' Var(f) & Scat(f) (17)
It represents the overall variance in the forecaster's probabilities if there were no scatter about the
conditional means f1 and f0. Thus, MinVar(f) measures how responsive the forecaster is to information
not related the event's occurrence.
The Brier score can also be formulated for a multiple event case. Let A1 , . . , A k represent a k-
event outcome space partition with K$2. Let dk represent the outcome index for each event k = 1, . . . ,
K. Let fk represent the probability forecast for each event k = 1, . . . ,K. In vector form d = (d1 , . . , dk)
and f = (f1 , . . , fk). The multiple event probability score (Murphy, 1972) is given
as:
PSM(f,d) ' (f & d)) (f & d) where 0 # PSM #2 (18)
Let i = 1, . . . , N index multiple-event forecasts fi and outcome indexes di over N different events, then
the mean of PSM is
12
PSM ' jK
k'1PSk (19)
where PSk represents the probability mean score for the kth event in the partition.
The covariance decomposition for the multiple event forecast is
PSM(f,d) ' jK
k'1Var(dk) % j
K
k'1MinVar(fk)
% jK
k'1Scat(fk) % j
K
k'1Bias 2
k
& 2jK
k'1Cov(fk,dk)
(20)
Each term in the multiple event case has an interpretation similar to that given in the single event case
discussed above.
Bootstrap Methodology
An OLS version of the 3-variable VAR model of Campbell (1990) and the 3-variable random
walk model were used to generate forecasts using the chain-rule of forecasting. Probability forecasts
from these models were generated using a bootstrap like procedure as outlined below.
The general VAR model is given as:
φ(B)tXt ' εt (21)
13
Here φ(B)t refers to the 3x3 autoregressive parameter matrix, whose elements are individually
polynomial functions of the lag operator B. The elements of φ(B)t are allowed to change over time
throughout the forecast interval, thus, they are indexed by t. Xt represents a 3x1 vector of stock prices,
dividend price ratios and interest rates observed in period t. The variable ε(t) represents a 3x1 vector of
residuals which are uncorrelated through time, but may be correlated in contemporaneous time.
Following a suggestion by Fair (1986), two sources of uncertainty are used to model the
probability distributions - uncertainty due to the lack of knowledge of φt and uncertainty in the one-
step-ahead forecasts (call this ut+1, a 3x1 vector).
At each date the elements of φ(B)t are assumed to normally distributed with mean φ(B)t and
covariance Vt=PtP't. Here φ(B)t and Vt are the estimated parameter and covariance matrices, found from
updating equation (21) with the Kalman filter at each date t. Uncertainty in φ(B)t is modeled by
making draws from the probability distribution used to describe φ(B)t. A particular draw φ(B)t is
obtained as
φ(B)(t ' φ(B)t % Pte (22)
where e is a vector of standard normal draws.
Uncertainty due the one-step-ahead forecast errors is modeled by drawing (call this draw u*t+1)
from the normal distribution with mean vector of zero and covariance matrix equal to the empirical
covariance matrix Σt on one-step-ahead forecast errors, ut+1. These latter errors are obtained from the
historical forecast performance on earlier data points. Thus, an initial period is required to obtain
estimates of Σt. To accomplish this, each sample is divided into three intervals. All three sample sets are
dealt with in a similar fashion. The first 24 observations in each sample are used to obtain OLS
estimates of φ(B) t, the next 48 observations are then used to simulate one-step-ahead forecasts. By
14
recursively forecasting Xt+1 and updating φ(B)t over this interval, we obtain a sample of 48 one-step-
ahead forecast errors from which we form our initial Σt. The remaining observations in each period are
used to model and evaluate the one-step-ahead probability forecasts.
The one-step-ahead forecast for Xt+1 is given as follows
X (
t%1 ' φ(B)(t Xt % u (
t%1 (23)
Repeating draws on e and ut+1 100 times at each date yields 100 point forecasts of Xt+1 at each time t in
the latter half of each period. The model is then moved forward one data point. The Kalman filter is
applied to obtain new estimates, φ(B)t+1 and Vt+1. In addition, the actual observed Xt+1 and the mean
forecasted Xt+1 are used to update Σt+1. Equation (23) is then reapplied 100 times in order to obtain the
forecast distribution for Xt+2. This procedure is repeated for each of the remaining data points in each
sample set.
Following each month's forecast, the actual outcome is compared to its forecasted distribution in
order to determine the observed fractile for that month.
Results
Table 2 summarizes the chi-squared statistic for the pre-Treasury Accord, post-Treasury Accord
and the entire sample period. At each date, the observed data point will determine the realized fractile by
where it falls under the cumulative distribution function for that date. By the probability integral
transform these should be uniformity distributed on (0,1). We break the interval (0,1) into 20 non-
overlapping and exhaustive classes and test uniformity of the realized fractiles using a chi-squared
"goodness of fit" test. Hence, there are 19 degrees of freedom and a critical chi-squared statistic value
of 30.114 at the 5 percent significance level.
In almost all cases, the random walk has a lower chi-squared statistic than does the OLS-VAR
15
for a one-step ahead forecast. For the pre-Treasury Accord period (1927-1951) and the entire sample
period (1927-1988), forecasts from both the OLS-VAR and the random walk model are rejected as
being well-calibrated. For the post-Treasury Accord period (1952-1988) the OLS-VAR is well-
calibrated for one, two, and three step ahead forecasts for all series. The random walk model is well-
calibrated for all series in the one-step and two-step forecast case; it is only well-calibrated for the
dividend price ratio in the three-step ahead case in the post-Treasury Accord period. Both models tend
to be less well-calibrated as the forecast horizon increases; although there are certainly exceptions to
this. The chi-squared goodness of fit tests indicate that the OLS-VAR model offers prequentially well-
calibrated forecasts only over the post-Treasury Accord period. The discrepancy between the pre and
post accord results is striking and indicative that the simple VAR is probably not capturing fundamental
pre-accord regularities.
Calibration plots for one-step ahead forecasts from the OLS-VAR and the random walk model
over the three time periods are given in Figures 1, 2 and 3. In each plot the horizontal axis is the issued
fractile and the vertical axis the, after-the-fact, relative frequency. So for a model to be well-calibrated
the fractile-relative frequency plot should be the 45 degree line. Figure 1 is for the post-accord period.
Note here that both the random walk and OLS-VAR plot close to the 45 degree line, with the OLS-VAR
closer to the 45 degree line for stock returns and interest rates over much of the issued fractiles. Figures
2 and 3 show much poorer calibration, considerable deviation form the 45 degree line for both models.
The calibration plots mirror quite well the chi-squared “goodness of fit” tests on one-step-ahead
forecasts presented in table 2; that is, both models are clearly not appropriate on pre-accord data.
Below we consider Brier Scores and their partitions on the same pre and post accord data for
both the OLS-VAR and a random walk forecast. Brier scores and covariance decomposition for the
entire sample, pre-Treasury accord and post-Treasury accord periods are contained in Tables 3 through
16
5, respectively. We have not considered the sampling distributions of these scores or their
decompositions, thus our results should be viewed as indicative and not definitive.
The row labeled "score" in Tables 3, 4 and 5 is the Brier score. Components which make up the
covariance decomposition are given beneath it. The columns labeled 1-step, 2-step and 3-step
represent, the Brier score or a component associated with one month ahead, two month ahead and 3
month ahead forecasts. The heading labeled "OLS-VAR” is the OLS version of Campbell's three
variable VAR model. The heading labeled random walk represents the results from a random walk
model.
Generally, the Brier scores increase as the forecast horizon increases (steps increases). This
holds for both the random walk and the OLS-VAR. The only case when this does not occur is for the
OLS-VAR forecasts of stock market returns using the entire sample period. Admittedly, the increase
may be very small as in the case of the stock market returns forecast in the post-Treasury accord sample
period where the one-step, two-step and three-step Brier scores for Campbell' model are .6314, .6339
and .6350, respectively. The increase may be large as in the case of the dividend price ratio forecast in
the post-Treasury accord sample period where the one-step, two-step and three-step Brier scores for the
OLS-VAR model are .1520, .2139 and .2698, respectively. The increase in the Brier scores indicates a
deterioration in both models’ abilities to forecast as the horizon increases. Not a surprising result. Of
course, since the Brier score is composed of various attributes, one of which is not under the model's
control, a more meaningful assessment may be through the individual score components.
Note that variability not under the forecaster's control (DVAR) decreases for the stock market
return and increases for the dividend price ratio and interest rate series. Note further that DVAR on
interest rates increases in post-accord data (contrast DVAR in tables 4 and 5 on interest rates).
This appears to agree with the prior notion that pegging interest rates in the pre-accord period made
them a much less uncertain series in the pre-accord period as compared with the post-accord period.
17
Stock market returns and the dividend price ratio appear to be easier to forecast over the post-accord
period (DVAR on both series is lower in table 5 than in table 4).
Across all horizons and for all variables forecasted, the conditional minimum forecast
(MinVarf), declines. As the number of forecast steps increases the amount of forecast variance that
must be tolerated for the forecaster to apply his expertise declines. Except for the dividend price ratio in
the post-accord period, scatter declines as the number of forecast steps increases for all periods and all
variables. This indicates that the noise or excess variability in the forecast declines as the number of
forecast periods is increased. Also, it shows that, at least for the one-step to three-step forecast, the
models are better at ignoring extraneous information than including important information in making a
forecast.
Except for the OLS-VAR forecast of stock returns in the post-Treasury accord period, bias
increases as the number of forecast steps increases. It is interesting to note that bias tends to increase
more for the random walk model than for the OLS-VAR model.
In general, covariance declines across all sample periods and forecast variables. The exceptions
are the stock market forecasts for the OLS-VAR model over the entire sample period, random walk over
the entire sample period and the OLS-VAR model over the pre-treasury accord period. A declining
covariance indicates that as the number of forecast steps increases, both models weaken in their ability
to respond to information related to forecasting the variable.
With respect to the entire sample period, the OLS-VAR model gives a higher (poorer) Brier
score for the one-step, two-step and three-step forecasts of the dividend price ratio and the one-step
ahead forecast of the interest rate. For the post-Treasury Accord period, the OLS-VAR model gives a
higher Brier score only for the one-step ahead interest rate forecast. For the pre-Treasury Accord
period, the OLS-VAR model gives a higher Brier score as in the entire sample period. It appears as
though the OLS-VAR model is not as good at forecasting the dividend price ratio as a random walk
18
model. This may be due to the fact that the forecasts are made on a monthly basis, while the time-series
is constructed by using total dividends paid over the previous year divided by the current stock price.
Although the Brier score gives an overall indication of the forecaster's ability, the components
of the covariance decomposition provides a clearer indication of the forecaster's ability to forecast. For
all sample periods and all variables, MinVarf is either almost zero or the estimates given by the OLS-
VAR model are less than the ones given by the random walk models. Recall that MinVarf is the
conditional minimum forecast variance given the covariance of forecasts and outcome index. It
represents the minimum amount of forecast variance that must be tolerated given that the forecaster
applies his fundamental forecasting abilities. The OLS-VAR model has a superior characteristic over
the random walk model in that it requires a lower minimum forecast variance.
In only six cases, does the OLS-VAR give a higher bias. None of these instances occur in the
stock market forecast, whereas five occur in the dividend price ratio forecast. Thus, the OLS-VAR
appears to be able to match mean forecasts to relative frequencies better than the random walk model.
Over the entire period, for dividend price ratio forecasts and interest rate forecasts, the OLS-VAR has
lower covariance than the random walk model. But for the stock market forecasts, the OLS-VAR
results in higher covariance in seven out of nine cases. Thus, for the interest rate series and the dividend
price ratios, the random walk model exhibits more forecasting skill, while the OLS-VAR shows more
skill in forecasting stock prices.
In all sample periods and all forecasts, the OLS-VAR gives a better (lower) measure of scatter.
It is superior to the random walk model by being less responsive to information not related to
forecasting for either of the three time series. Stated alternatively, the forecasts from the OLS-VAR
contain less noise. Note that since the forecast variance is the sum of MinVarf and scatter and that
Campbell's model in general had both a lower scatter and Minvarf, then it is also true that the OLS-
VAR has a lower forecast variance than the random walk model.
19
Overall, the OLS-VAR model outperforms the random walk in, scatter and MinVarf. But it is
outperformed in terms of covariance, except in forecasting the stock market. One issue which arises is
why does the OLS-VAR have better scatter, but offers worse covariance estimates. This is possibly due
to the types of models being compared. The VAR model acts as a filter, trying to remove unimportant
information while letting important information pass through. A random walk model does not
discriminate and allows all information to pass through, whether it is relevant or not. Since scatter
represents how responsive the forecaster is to information unrelated to the event, while covariance
represents how responsive the forecaster is to information related to the event, it is possible that the
OLS-VAR does well in screening out irrelevant information, but at some expense of screening out
relevant information for interest rate and dividend price ratio forecasts.
In the case of forecasting the stock market, the OLS-VAR does comparatively well in screening out
irrelevant information, while still incorporating relevant information. Actually, Campbell’s motivation
for his model appears to be forecasting stock market returns, first and foremost, and not so much
forecasting dividend price ratios and interest rates.
The covariance measure reflects differences between average probabilities assigned to events
(fractiles) that occurred and average probabilities associated with fractiles that did not occur. We want
such differences to be large. In table 6 we list these differences. Notice that these differences are tiny
for stock market returns at every horizon, quite large at every horizon for the dividend price ratio and
moderate for interest rates. To the extent that these differences capture the essence of forecasting, we
have to conclude that our ability to forecast stock market returns with this model is not very good.
One point which we wanted to address in this paper is whether the Brier score and its partition is
able to provide additional information with respect to forecast performance relative to the chi-squared
statistics on calibration. In table 7, we summarize performance measures using zero-one indicators for
each series, time period, and forecast horizon. A zero (0) indicates that the random walk forecast
20
outperformed the OLS-VAR for that measure on that particular series, time interval and horizon. A one
(1) indicates that the OLS-VAR outperformed the random walk model.
From Table 7 notice that there is no clear dominance of the OLS-VAR over the random walk
using the chi-squared statistics. Of the 27 cases studied (three variables, three horizons and three time
periods), OLS dominates in 11 cases and is dominated by the random walk in 16. Notice, however, that
in terms of Brier score components, the OLS-VAR-forecasts do have some dominating characteristics.
In particular, scatter and minimum variance components of the OLS-VAR forecasts of the dividend
price ratio (d/p) and interest rate (r) dominate the random walk forecasts across all three time periods
and almost all forecast horizons. Further, in seven of the nine cases, the OLS-VAR forecasts dominate
the random walk forecasts in terms of covariance. Just the opposite is true for dividend price ratio and
interest rate forecasts in terms of the covariance metric -- the random walk forecasts are superior in
terms of the covariance component of the Brier score. Recall, that covariance is at the heart of the
forecasting effort (the difference in average probability assigned to events which ultimately occur versus
average probabilities of events which ultimately do not occur). The covariance metric appears to tell us
that the OLS-VAR does not offer much in sorting relative to the random walk forecast for two of the
three series. Campbell’s’s model was setup (primarily) to forecast stock market returns (h) and not
interest rates (r) or necessarily dividend price ratios(d/p). So perhaps these results are not unreasonable.
However, as noted in table 6, the OLS-VAR does not offer large differences in probabilities between
events which occur and events which do not occur. Nevertheless, it does better than the random walk
model. This last result and interpretations holds for the overall Brier score results as well; the OLS-VAR
performs well over all three time periods and all three horizons in forecasting stock market returns
(relative to the random walk); while its does not perform particularly well in any of the forecasts of the
dividend price ratio and offers mixed results in forecasting interest rates. The OLS-VAR generally gets
good marks (relative to the random walk) in terms of scatter, minimum variance and bias.
21
Discussion
This paper applies prequential analysis to two models of the US stock market. Probability
calibration, mean probability scores (Brier scores) and their partitions are considered. A three variable
VAR, introduced earlier by Campbell, consisting of real stock returns, short term interest rates and
dividend price ratio, is fit to 1927-1988 monthly data using ordinary least squares. Probability forecasts
from an ordinary least squares version of this model (OLS-VAR) are compared to probability forecasts
from a random walk. This paper finds that the OLS-VAR and the random walk model are not well
calibrated for the pre-Treasury Accord period, (before 1952). For the post-Treasury-Accord sample
period (1952-1988), the random walk model is slightly better calibrated for the one-step ahead forecast,
but the OLS-VAR is better calibrated for the two and three-step ahead forecasts. The OLS-VAR tends
to have lower Brier scores than the random walk model for all three series and forecast horizons.
The Yates-partition of the Brier score indicates that the OLS-VAR produces a lower minimum
forecast variance, bias and scatter. But it gives a smaller covariance, except for forecasting stock market
returns. Thus, OLS-VAR is better at screening out information not relevant to issuing forecasts.
However, this improvement comes at the a cost of failing to incorporate some relevant information. The
exception to this is forecasting stock market returns. Here the OLS-VAR outperforms the random walk.
Generally, we would like a model to offer well-calibrated forecasts and sort events into groups
where the assessed probability of events which obtain approach one and probabilities on events which
do not obtain approach zero. Tests of calibration and the bias component of the Yates-partition measure
well-calibration. The covariance component of the Yates-partition measures a model’s ability at sorting.
Our OLS version of Campbell’s VAR does result in well-calibrated probability forecasts over the post-
Treasury accord period. It does not result in large differences in probabilities of stock market return
events which occur versus stock market return events which do not occur. Thus, we do not offer our
model as particularly helpful in forecasting stock market returns. Our model is “honest”, as it reports
22
well-calibrated forecasts over the post-accord period; yet it is not particularly good, as it is not able to
offer a large difference in probabilities between events which ultimately obtain versus events which do
not obtain (for stock market returns). This last point is a strong reason why one might want to use the
additional insights offered by the Brier score and its Yates-partition, rather than confining his/her study
to just the calibration properties of a particular set of probability forecasts. We suggest that recent
papers (Kling and Bessler (1989) for example) which focus on calibration ought to be re-evaluated
under the more general Brier score and its probability partition.
Further research on probability forecasting is certainly warranted. In particular, this model could
be studied using non-normal draws in calculating probability forecasts. As stock market returns are
generally characterized by “fat tailed” distributions, this suggestion is worth consideration. However,
even under alternative error assumptions, the resulting model should be judged according to its
prequential performance--by the sequence of probabilities it issues and subsequent realizations. Further,
work could be done on the sampling distribution of both the Brier score and its covariance partitions.
Here we made statements about the “goodness” of a set of forecasts based on the Brier score and its
Yates-partition. This is consistent with deFinetti’s use of scoring rules as a metric of performance in
probability forecasting (see deFinetti (1965)). One could go further with this idea and make
probabilistic statements on the hypothesis that the difference between two Brier scores (say one from
model A the other from model B) is equal to zero. 4 By extension one might consider, as well,
distributions on the differences between each of the five components in the Yates partition. We leave
these topics for future research.
23
Footnotes
1. For a model to be well-calibrated, events that are assigned a probability of n percent should occur in
ex post assessment with n percent relative frequency. Calibration acts as a long-run assessment of a
model's ability to issue realistic probability forecasts.
2. Actually there are several applications of prequential analysis in the more general literature. Dawid
(1986) reviews this literature under the heading of probability forecasting. A recent application which is
not labeled “prequential analysis”, but is none the less the same, Diebold, et al. 1999 study calibration
properties of probability forecasts of foreign exchange.
3. Resolution refers to the ability of a model to sort individual outcomes into groups which differ from
the long run relative frequency.
4. A point deFinetti probably would not agree with as he was a strict subjectivist, having no clear
attraction to hypothesis testing in the usual sense of the word; see deFinetti (1974).
24
Table 1. Ordinary Least Squares (OLS) VAR Estimates versus Campbell’s Generalized Method of Moments(GMM)Estimates.
_________________________________________________________________________
right-hand side variablesdep. var. ht-1 d/pt-1 rt-1 R2
OLS GMM OLS GMM OLS GMM OLS GMM
(1927 - 1988)
ht .107 .107 .331 .331 -.423 -.424 .024 .024
(.037) (.063) (.155) (.183) (.201) (.195)
d/pt -.007 -.007 .963 .963 .018 .018 .936 .937
(.002) (.005) (.009) (.028) (.012) (.010)
rt .007 .007 -.039 -.040 .669 .669 .450 .450
(.005) (.005) (.021) (.010) (.028) (.061)
(1927 - 1951)
ht .142 .142 .482 .483 .926 .926 .028 .028
(.059) (.091) (.281) (.466) (.643) (.712)
d/pt -.012 -.012 .934 .935 -.033 -.033 .901 .901
(.004) (.007) (.019) (.045) (.043) (.041)
rt .005 .005 -.019 -.019 .308 .309 .092 .101
(.005) (.006) (.024) (.026) (.056) (.161)
(1952 - 1988)
ht .048 .048 .500 .490 -.723 -.724 .065 .065
(.047) (.060) (.227) (.227) (.163) (.192)
d/pt -.001 -.001 .978 .980 .034 .034 .959 .959
(.020) (.003) (.009) (.011) (.007) (.009)
rt .013 .013 -.105 -.017 .739 .739 .548 .547
(.009) (.012) (.046) (.058) (.033) (.009)______________________Here ht refers to real stock market returns in period t; d/pt refers to the dividend price ratio in period t; rt the real interestrate in period t. The numbers in parentheses are standard errors. These variables are defined and discussed in Campbell(1991).
25
Table 2. Chi-Squared “Goodness of Fit” Tests on OLS-VAR and Random Walk Probability Forecastson Horizons of 1,2, and 3 Steps Ahead.________________________________________________________________________________________________________________________________________________________________________
Chi-Squared Statistics1
ForecastedVariable
OLS-VAR Random Walk
1step 2step 3step 1step 2step 3step
(1927-88)
ht 197.85 232.23 229.61 169.58 425.05 563.87
d/pt 486.79 544.34 490.66 482.78 451.29 437.58
rt 206.40 215.29 213.78 144.24 205.12 371.44
(1927-51)
ht 106.17 106.68 110.74 75.81 184.26 229.20
d/pt 156.46 151.72 141.46 123.75 117.42 94.04
rt 479.55 383.28 495.65 443.39 485.23 551.67
(1952-88)
ht 11.00 15.71 13.38 6.42 42.49 92.16
d/pt 15.59 15.80 21.95 3.72 14.70 26.44
rt 20.16 22.84 20.66 14.72 19.51 43.30
_______________1. Five per cent critical value is 30.11
Score is the Brier score, lower values of which suggest “better” performance. The Yates-decomposition is given by the five numbers below score in each column: Score = DVAR + Minvar + Scatter + Bias - 2 Cov
26
Table 3. Brier Scores and Decompositions of OLS-VAR and Random Walk Forecasts, 1928 - 1988.1________________________________________________________________________________________________________________________________________________________________________
OLS-VAR Random Walk
1 step 2 step 3 step 1 step 2 step 3 step
(stock market returns)
score .7162 .7050 .7145 .7787 .8257 .8447
Dvar .6502 .6491 .6488 .6502 .6491 .6488
Minvar .0000 .0001 .0000 .0002 .0000 .0000
Scatter .0155 .0148 .0154 .0277 .0170 .0134
Bias .0556 .0540 .0546 .1098 .1588 .1604
2Cov .0051 .0130 .0045 .0092 .0009 .0021
(Dividend price ratio)
Score .2663 .3787 .4539 .2509 .3508 .4296
Dvar .7035 .7037 .7039 .7035 .7037 .7039
Minvar .1908 .0962 .0569 .2093 .1198 .0743
Scatter .1022 .0929 .0827 .1025 .1013 .0965
Bias .0003 .0020 .0046 .0004 .0024 .0054
2Cov .7304 .5161 .3942 .7648 .5764 .4505
(Interest rates)
Score .5739 .6145 .6421 .5608 .6307 .6664
Dvar .6569 .6569 .6578 .6569 .6569 .6578
Minvar .0110 .0003 .0009 .0279 .0095 .0047
Scatter .0691 .0365 .0251 .1359 .0876 .0623
Bias .0020 .0029 .0027 .0067 .0255 .0457
2Cov .1651 .0847 .0443 .2665 .1488 .1042
1.Score is the Brier Score, lower values of which suggest “better” performance. The Yates-decomposition is given by the five numbers below score in each column: Score = DVAR + Minvar +Scatter + Bias - 2 Cov.
27
Table 4. Brier Scores and Decompositions of OLS-VAR and Random Walk Forecasts, 1928 - 1951.1________________________________________________________________________________________________________________________________________________________________________
OLS-VAR Random Walk
1 step 2 step 3 step 1 step 2 step 3 step
(stock market returns)
score .7588 .7602 .7659 .8254 .8555 .8617
Dvar .6881 .6845 .6845 .6881 .6845 .6845
Minvar .0002 .0001 .0001 .0001 .0000 .0001
Scatter .0122 .0114 .0121 .0205 .0132 .0097
Bias .0749 .0757 .0773 .1193 .1566 .1754
2Cov .0166 .0115 .0081 .0027 - .0012 .0081
(Dividend price ratio)
Score .3459 .4947 .5836 .3370 .4718 .5475
Dvar .7028 .7046 .7063 .7028 .7046 .7063
Minvar .1324 .0463 .0192 .1455 .0633 .0034
Scatter .0984 .0735 .0580 .1078 .0963 .0824
Bias .0151 .0235 .0269 .0139 .0230 .0281
2Cov .6028 .3531 .2267 .6630 .4154 .3027
(Interest rates)
Score .5348 .5491 .5553 .4987 .5712 .6319
Dvar .4988 .4940 .4940 .4988 .4940 .4940
Minvar .0005 .0005 .0004 .0028 .0009 .0004
Scatter .0183 .0165 .0175 .0299 .0214 .0187
Bias .0351 .0473 .0525 .0308 .0833 .1323
2Cov .0178 .0092 .0091 .0636 .0284 .0134
1.Score is the Brier Score, lower values of which suggest “better” performance. The Yatesdecomposition is given by the five numbers below score in each column: Score = DVAR + Minvar +Scatter + Bias - 2 Cov
28
Table 5. Brier Scores and Decompositions of OLS-VAR and Random Walk Forecasts, 1952 - 1988.1________________________________________________________________________________________________________________________________________________________________________
OLS-VAR Random Walk
1 step 2 step 3 step 1 step 2 step 3 step
(stock market returns)
score .6314 .6339 .6350 .7301 .7411 .7640
Dvar .6307 .6310 .6307 .6307 .6310 .6307
Minvar .0002 .0000 .0000 .0006 .0002 .0000
Scatter .0190 .0125 .0108 .0746 .0387 .0262
Bias .0028 .0029 .0032 .0428 .0811 .1121
2Cov .0212 .0127 .0097 .0186 .0099 .0050
(Dividend price ratio)
Score .1520 .2193 .2698 .1474 .2281 .2834
Dvar .6471 .6471 .6471 .6471 .6471 .6471
Minvar .3628 .2720 .2143 .3810 .2819 .2253
Scatter .1064 .1285 .1382 .1101 .1483 .1674
Bias .0016 .0025 .0031 .0003 .0003 .0002
2Cov .9659 .8308 .7331 .9911 .8496 .7566
(Interest rates)
Score .6060 .6708 .7207 .6257 .6998 .7184
Dvar .7319 .7334 .7349 .7319 .7334 .7349
Minvar .0293 .0083 .0022 .0341 .0108 .0061
Scatter .1293 .0760 .0547 .1617 .1078 .0788
Bias .0032 .0048 .0060 .0076 .0160 .0249
2Cov .2876 .1517 .0771 .3095 .1682 .1263
29
Table 6. Average Probabilities Assigned to Events which Occur minus Average Probabilities Assignedto Events which do not Occur, by series (h, d/p, r), forecast horizon (t+1, t+2, t+3) and time period, forOLS-VAR and Random Walk.________________________________________________________________________________________________________________________________________________________________________
1927 - 1988 1921 - 1951 1952 - 1988
series OLS RW OLS RW OLS RW
h t+1 .0039 .0071 .0121 .0020 .0168 .0147
h t+2 .0100 .0007 .0084 -.0009 .0101 .0078
h t+3 .0035 .0016 .0059 .0059 .0077 .0040
d/p t+1 .5217 .5436 .4289 .4717 .7463 .7658
d/p t+2 .3667 .4095 .2500 .2948 .6419 .6565
d/p t+3 .2880 .3200 .1605 .2143 .5665 .5846
r t+1 .1257 .2028 .0178 .0638 .1965 .2114
r t+2 .0645 .1133 .0093 .0287 .1034 .1147
r t+3 .0337 .0792 .0092 .0136 .0525 .0859____________Numbers in this table are derivable from those contained in Tables 3,4, or 5 and the formula: slope =cov/dvar, where cov and dvar are defined as in tables 3,4, and 5. The slope is the difference in averageprobabilities associated with events which obtain versus the average probabilities associated with eventswhich do not obtain. It can be found statistically as the “slope” found by regressing probabilityjudgements on outcome indexes (see Yates (1988, page 284) for further discussion).
30
Table 7. Indicators of dominance: VAR(1) versus the Random Walk (0) for chi-squared statistics, Brierscores, and its components.________________________________________________________________________________________________________________________________________________________________________
series chi sq Br Sc Covar Sct Mvar Bias
steps ahead
1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3
- - - - - - - - - - - - - - - - - -
h (27-88) 0 1 1 1 1 1 0 1 1 1 1 0 1 0 - 1 1 1
h (27-51) 0 1 1 1 1 1 1 1 - 1 1 0 1 0 - 1 1 1
h (52-88) 0 1 1 1 1 1 1 1 1 1 1 1 1 1 - 1 1 1
d/p (27-88) 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
d/p (27-51) 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1
d/p (52-88) 0 0 1 0 1 1 0 0 0 1 1 1 1 1 1 0 0 0
r (27-88) 0 0 1 0 1 1 0 0 0 1 1 1 1 1 1 1 1 1
r (27-51) 0 1 1 0 1 1 0 0 0 1 1 1 1 1 - 0 1 1
r (52-88) 0 0 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1
_________________A one (1) indicates the VAR outperforms the random walk on the particular measure for forecasts at thehorizon listed at the head of the column. A zero indicates the random walk outperformed the VAR onthe measure. A "-" indicates the two models have the same score (the models tied with respect to thatmeasure).The variables h, d/p, and r refer to stock market returns, dividend - price ratio and interestrates, respectively.
31
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1 R
elat
ive
Fre
quen
cy
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
Dividend Price Ratio 1952-1988.
Stock Returns 1952-1988.
Interest Rates 1952-1988
Figure 1. Calibration Plots on Probabilistic Forecasts of Dividend Price ratio, Stock Market Returns andInterest Rates, 1952 - 1988 Data, by model -- OLS-VAR (Campbell’s model) and Random Walk.
32
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
Dividend Price Ratio 1926-1951
Stock Returns 1926-1951
Interest Rates 1926-1951
Figure 2. Calibration Plots on Probabilistic Forecasts of Dividend Price ratio, Stock Market Returns andInterest Rates, 1926 - 1951 Data, by model -- OLS-VAR (Campbell’s model) and Random Walk.
33
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
0
0.2
0.4
0.6
0.8
1
Rel
ativ
e F
requ
ency
0 0.2 0.4 0.6 0.8 1 Fractiles
Campbells Model Random Walk Model
Dividend Price Ratio 1926-1988
Stock Returns 1926-1988
Interest Rates 1926-1988
Figure 3. Calibration Plots on Probabilistic Forecasts of Dividend Price ratio, Stock Market Returns andInterest Rates, 1926 - 1988 Data, by model -- OLS-VAR (Campbell’s model) and Random Walk.
34
References
Brier, Glen W., "Verification of forecasts expressed in terms of probability," Monthly Weather Review,78(1950): 1-3.
Bunn, Derek W. Applied Decision Analysis. New York: McGraw-Hill.
Campbell, John Y., "A variance decomposition for stock returns," Economic Journal, 101(1991): 157-179.
Campbell, John Y., "Stock returns and the term structure," Journal of Financial Economics, 18(1987):373-399.
Campbell, John Y., and Robert Shiller, "The dividend-price ratio and expectations of future dividendsand discount factors," Review of Financial Studies, 1(1988): 195-228.
Covey, Ted, and David A. Bessler, "Testing for Granger's full causality," Review of Economics andStatistics, 74(1992): 146-153.
Dawid, A.P., "Statistical theory: A prequential approach," Journal of Royal Statisticical Society,147(1984): 278-297.
Dawid, A.P., "Probability forecasting," in Encyclopedia of Statistical Sciences, Vol. 7. New York:Wiley.
deFinetti, B. “La Prevision: ses lois logiques, ses sources subjectives” Annales de l’Institute HeniPoincare, 7(1937):1 -67.
deFinetti, B. “Methods for Discriminating Levels of Partial Knowledge Concerning a Test Item,” BritishJournal of Mathematical and Statistical Psychology 18(1965, part I):87-123.
deFinetti, Bruno. Theory of Probability, New York: Wiley 1974.
Diebold, F., J. Han and A. Tay. “Multivariate Density Forecast Evaluation and Calibration in FinancialRisk Management: High-Frequency Returns on Foreign Exchange,” Review of Economic and Statistics81(1999):661-673.
Dow, Charles, H., "Scientific stock speculation", The Magazine of Wall Street, (New York), 1920.
Fair, Ray, C., "Evaluating the predicative accuracy of models," in Z. Griliches and M.D. Intriligator(eds.) Handbook of Econometrics Vol. III(New York: Elsevier Science Publishing Co., 1986), 1980-1995.
Fama, Eugene, "Efficient capital markets: A review of theory and empirical work," Journal of Finance,25(1970): 383-417.
Fama, Eugene, F. and Kenneth R. French, "Dividend yields and expected stock returns," Journal ofFinancial Economics, 22(1988): 3-25.
35
Fama, Eugene, F. and Kenneth R. French, "Business conditions and expected returns on stocks andbonds," Journal of Financial Economics, 22(1989): 23-49.
Fama, Eugene, F. and Robert G. Schwert, "Assert returns and inflation," Journal of FinancialEconomics, 5(1977): 115-46.
Hanson, Lars, Peter, "Large sample properties of generalized method of moments estimators,"Econometrica, 50(1982): 1029-1054.
Kim, Myung Jig, Charles Nelson, and Robert Startz, "Mean reversion in stock prices? A reappraisal ofthe empirical evidence," Review of Economic Studies, 58(1991): 515-528.
Kling, John, L., "Predicting the turning points of business and economic time series," Journal ofBusiness, 60(1987): 201-238.
Kling, John, L., and David A. Bessler, "Calibration based prediction distributions: An application ofprequential analysis to interest rates, money and prices," Journal of Business, 62(1989): 477-499.
Murphy, A.H., "A new vector partition of the probability score," Journal of Applied Meteorology,12(1973): 595-600.
Nelson, R. G. and D. A. Bessler. 1989. "Subjective Probabilities Elicited Under Proper and ImproperScoring Rules: A Laboratory Test of Predicted Responses." American Journal of AgriculturalEconomics 71:363-369.
Sanders, F., "On subjective probability forecasting," Journal of Meteorology, 2(1963): 191-201.
Savage, L. “Elicitation of Personal Probabilities and Expectations,” Journal of the American StatisticalAssociation 66(1971):783-801.
Yates, Frank, "Analyzing the accuracy of probability judgements for multiple events: An extension ofthe covariance decomposition," Organizational Behavior and Human Decision Processes, 41 (1988):281-299.
Zellner, A., Chansik Hong and Chung-Ki Min, "Forecasting turning points in international outputgrowth rates using Bayesian exponentially weighted autoregression, time-varying parameter, and pollingtechniques," Journal of Econometrics, 49 (1991): 275-304.
36