early warning systems with real-time data...el estudio analiza crisis cambiarias en ocho países de...
TRANSCRIPT
Banco de México
Documentos de Investigación
Banco de México
Working Papers
N° 2017-16
Early Warning Systems with Real-Time Data
September 2017
La serie de Documentos de Investigación del Banco de México divulga resultados preliminares de
trabajos de investigación económica realizados en el Banco de México con la finalidad de propiciar elintercambio y debate de ideas. El contenido de los Documentos de Investigación, así como lasconclusiones que de ellos se derivan, son responsabilidad exclusiva de los autores y no reflejannecesariamente las del Banco de México.
The Working Papers series of Banco de México disseminates preliminary results of economicresearch conducted at Banco de México in order to promote the exchange and debate of ideas. Theviews and conclusions presented in the Working Papers are exclusively the responsibility of the authorsand do not necessarily reflect those of Banco de México.
Tjeerd M. BoonmanBanco de México
Jan P .A.M. JacobsUniversity of Groningen, University of Tasmania,
CAMA and CIRANO
Gerard H. KuperUniversity of Groningen
Alber to RomeroBanco de México
Early Warning Systems with Real-Time Data*
Abstract: This paper investigates the performance of early warning systems in real-time, usingforecasts of indicators that were available at the moment predictions are to be made. The study analyzescurrency crises in eight Latin American and Central and Eastern European countries, distinguishing anestimation period 1990-2009 and a prediction period 2010-2014. We apply two varieties of earlywarning systems: the signal approach and the logit models. For both methods we find that usingforecasts of the indicators worsens the predictive ability of early warning systems compared to using themost recently available information (ex post).Keywords: Real-time data, Early warning system, Signal approach, Logit model, Emerging economiesJEL Classification: E47, G01, F31, C23, E58.
Resumen: En este trabajo se investiga el desempeño de los sistemas de alerta temprana en tiemporeal, utilizando los pronósticos de indicadores que se encontraban disponibles al momento de hacer laspredicciones. El estudio analiza crisis cambiarias en ocho países de América Latina y de Europa Centraly Oriental, separando el período de estimación, 1990-2009, del período de predicción, 2010-2014. Seaplicaron dos métodos de sistemas de alerta temprana: el método de señales y el de regresión logística.Con ambos métodos encontramos que el uso de pronósticos de los indicadores reduce la capacidadpredictiva de los sistemas de alerta temprana en comparación con el uso de la información disponiblemás reciente (ex post).Palabras Clave: Datos en tiempo real, sistemas de alerta temprana, método de señales, regresiónlogística, economías emergentes
Documento de Investigación2017-16
Working Paper2017-16
Tjeerd M. Boonman y
Banco de MéxicoJan P .A.M. Jacobs z
University of Groningen, University of Tasmania,CAMA and CIRANO
Gerard H. Kuper x
University of GroningenAlber to Romero **
Banco de México
*The opinions expressed here are those of the authors and do not necessarily represent Banco de México's or itsboard of governors' opinions. We thank Georgia Bush, Gabriel Cuadra, Mariela Dal Borgo, Juan Pablo Graf,Arturo Lamadrid, Fabrizio López-Gallo, Lorenzo Menna, participants at the EEFS 2017 annual meeting inLjubljana, Slovenia, and seminar participants at the University of San Diego, San Diego State University,EGADE Guadalajara and Banco de México for their valuable comments and suggestions. We also thank DiegoCid and Stefano Lord for their excellent research assistance. y Dirección General de Estabilidad Financiera. Email: [email protected]. z University of Groningen, University of Tasmania, CAMA and CIRANO. Email: [email protected]. x University of Groningen. Email: [email protected]. ** Dirección General de Estabilidad Financiera. Email: [email protected].
1 Introduction
The high costs of financial crises for the public sector as well as private investors has led to
a large number of empirical studies that intend to detect crises in a timely manner, through
the use of Early Warning Systems (EWSs). EWSs are quantitative models that predict
extreme events based on statistical information and mechanisms from the past.1 EWSs
for financial crises have been criticized for several reasons, one of which is that they are
only useful for detecting past crises (Frankel and Saravelos, 2010).2 An important reason
is that the data used by forecasters to construct the EWS are not available in a timely
manner. Another reason is that indicators are selected with the benefit of hindsight. EWSs
are typically tested out-of-sample using the most recent data, known as current-vintage
data. When the EWS detects crises using current-vintage data but fails to detect crises for
the out-of-sample prediction period, then the EWS provides a false sense of security.
We focus on a particular type of financial crisis, the currency crisis, which is defined
as a large, sudden depreciation or devaluation of the exchange rate, or an episode with
high pressure on the exchange rate that may result in large losses of international reserves
and/or a hike in domestic interest rates to defend the exchange rate (Berg, Borensztein
and Patillo, 2005). We set up two types of early warning systems for currency crises,
the signal approach and the logit model. We apply each EWS to a panel of four Latin
American and four Central and Eastern European countries, in the period 1990–2014. We
first estimate our EWSs on the in-sample period (1990–2009), and then predict currency
crises for the out-of-sample period (2010–2014) with two types of real-time data, with
and without employing information that is not available at the moment predictions have
to be made.1EWSs are applied to predict financial crises, but also for natural disasters such as tsunamis, earthquakes,
droughts and epidemics (Choo, 2009).2Other reasons are that the out-of-sample performance of EWSs has been rather poor—even for promi-
nent papers (Berg and Patillo, 1999). A more fundamental problem with early warning signals is that theseare meant to predict crises with the aim to avoid their occurrence; but if these signals are used for policypurposes, the predicted crises will be avoided, which means that model predictions will not be accurate anylonger. Bussiere (2013) refers to this contradiction as an “impossibility theorem”. A second problem issimilar in essence, but leads to the opposite conclusion: when Early Warning Systems predict a crisis thismay lead to self-fulfilling prophecies by market participants.
1
Figure 1: Real-time data trapezoid
Vintage years
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1 2
2000 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 2000 8.0
2001 6.9 6.9 6.9 6.9 6.9 6.9 6.9 6.9 6.9 6.9 2001 6.9
2002 7.8 7.8 7.8 7.8 7.8 7.8 7.8 7.8 7.8 7.8 2002 7.8
2003 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 2003 8.1
2004 1.5 1.6 1.6 1.6 1.6 1.6 1.6 1.6 1.6 1.6 2004 1.6
2005 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 2005 0.3
2006 7.1 6.7 10.1 10.6 10.9 10.9 10.9 10.9 10.9 10.9 2006 10.9 7.1
2007 6.8 8.3 10.1 10.7 10.4 10.5 10.5 10.5 10.5 10.5 2007 10.5 8.3
2008 9.3 8.6 1.6 0.5 0.4 0.1 0.1 0.1 0.1 2008 0.1 8.6
2009 9.2 4.3 8.1 8.5 9.1 10.4 10.6 10.6 2009 10.6 4.3
2010 2.2 7.8 7.4 6.7 6.0 5.1 5.1 2010 5.1 7.8
2011 6.6 6.1 4.9 3.5 2.9 1.7 2011 1.7 6.1
2012 4.5 0.2 -2.3 -7.2 -6.9 2012 -6.9 0.2
2013 0.0 -1.8 -3.3 -4.2 2013 -4.2 -1.8
2014 -1.3 -2.8 -3.0 2014 -3.0 -2.8
Data set
Notes: Example of a real-time data trapezoid (left panel). Columns show vintages, and rows subsequentobservations for years. Italicized numbers are subject to revisions. The cells with the darkest backgroundsrepresent first estimates, the cells with lighter backgrounds are second estimates, third, and fourthestimates of the data. Data that are not italicized represent data which are no longer revised. The rightpanel shows data concepts that we use in our analysis, current-vintage data (data set 1) and secondestimates (data set 2).
In this paper, we investigate the performance of EWSs for currency crises with real-
time data, which reflects the notion that data are subject to revision over the course of
time. Real-time data are typically displayed in the form of a real-time data trapezoid.
Figure 1 provides an example. Vintages are in columns and observations are in rows. As
we move across columns, from left to right, later vintages are displayed and we observe
how indicators are revised over time. To illustrate the real-time data trapezoid, let’s take
the values for the year 2012, which corresponds to the row 2012. The first estimate of the
variable in 2012 is a forecast made one year earlier in 2011 (4.5). This forecast is adjusted
in 2012 (0.2), while in 2013 the first official data observation is published −2.3, which
is revised to −7.2 in 2014, followed by another data revision in 2015, which brings the
value at −6.9.
We assess the sensitivity of EWS currency crisis predictions to the choice of two
alternative data sets shown in Figure 1. As our first data set, we employ current-vintage
2
data which implies using information of the most recent vintage, as shown in the first
column of the right panel of the figure. The second data set that we employ contains
second estimates (Right panel, data set 2), for this we take forecasts for year t published
in the beginning of year t. We find that using second estimates worsens the ability of
EWSs in signalling crises compared to using currently available data. This is in line with
the critique that EWSs perform well to predict past crises, but do poorly in predicting
future crises.
To review the performance of alternative EWSs on forecasting currency crises in
emerging economies in the late 1990s Berg, Borensztein and Patillo (2005) use only in-
formation that was available at the time, no information about actual outcomes was used
in the forecasts. They use the internal IMF July 1999 forecasts and subsequent internal
forecasts to compare the performance of alternative EWSs. Gunther and Moore (2003)
compare the performance of an EWS for banking crises in the period 1996–1998 using
first releases (first official publication of the data after the period has ended) versus current
vintage data that was available in May 2000 and find that the EWS with current vintage
data performs better than the EWS with first releases. Reagle and Salvatore (2005) test
the robustness of predicting the Asian 1997–1998 financial crisis without and with data
revisions. They estimate a probit model for a cross-section of a group of 54 emerging
economies with three sets of data: the original, unrevised 1996 World Bank data, and
the 1999 and 2004 updates of the 1996 World Bank data. They conclude that data revi-
sions lead to significant changes in the model’s estimates, which presents a problem for
researchers that should be recognized and addressed.
The importance of using early estimates in EWSs has been acknowledged in the liter-
ature, but real-time data have not been implemented on a widespread scale. Frankel and
Saravelos (2012) comment that predictions issued in real time would be impressive, but
also especially difficult. Lo Duca and Peltonen (2013) remark that real-time data sets that
contain information on the revisions of data after the first publication do not exist yet for
several countries in their sample of developed and emerging economies. Data availability
3
issues seem to be the major reason why implementation has not been widespread; early
estimates for emerging economies are not publicly available.
Alessi and Detken (2011) use quasi real-time data. Although it sounds similar to
real-time data, the authors actually mean current-vintage data. In their own words:
The caveat is that we use the most recent vintage of data and not a true real
time data set with unrevised data. Nevertheless, we use conservative lags to
proxy for standard publication lags and thus real time data availability. [...]
The performance of real variables could possibly be worse in a true real time
setting compared to a quasi real time setting, as real variables can be heavily
revised (i.e., the quality of current vintage data can be much better than the
quality of real time vintages). [page 524]
Lo Duca and Peltonen (2013) also use quasi real-time data, in their case to predict fi-
nancial stress events. Holopainen and Sarlin (2016) use quasi real-time data to predict
banking crises. Their contribution to the crises EWS literature is to assume that the fore-
caster uses all available accounting and market-based information, with a lag. However,
an important limitation still holds as data revisions may occur after the first estimate.
Our approach differs from Berg, Borensztein and Patillo (2005) in several ways. First,
we explicitly focus on the difference between second estimates and current-vintage data
when making out-of-sample predictions. Second, we use the consensus forecasts avail-
able in Haver Analytics, whereas they used internal predictions from the IMF, which are
not publicly available. Third, we consider a longer period. As a consequence, our sample
includes more and different crises, both from the 1990s (e.g. Mexico 1994-1995, Russia
1998, Brazil 1999) and the next decade (e.g. Argentina 2001–2002, and the Global Fi-
nancial Crisis that hit most emerging economies in 2008–2009). Including more crises
makes our analysis more complete, because of the variety in the crises (Kaminksy, 2006).
Compared to Gunther and More (2003) and Reagle and Salvatore (2005) we use sec-
ond estimates to predict currency crises whereas they use first releases to predict banking
crises.
4
Our results can be useful for policymakers as it draws attention to the issue of data
availability. Since current-vintage data is not available on time, a realistic out-of-sample
evaluation of EWSs requires early estimates of the indicators. Our results show that the
crisis prediction results are worse when using early estimates, which demonstrates the
necessity to dedicate more efforts in producing forecasts of the indicators themselves.
The remainder of this paper is structured as follows. Section 2 reviews definitions of
currency crises and EWSs for currency crises. Section 3 describes the two types of EWSs
we employ. After the description of the data in Section 4, we present empirical results in
Section 5. Section 6 concludes.
2 Currency Crises and Early Warning Systems
Currency crises not only occur in countries fixed exchange rate regimes, but also in coun-
tries with flexible exchange rates which, in principle, should be more resistant to currency
crises. One would expect continuous market adjustment to limit the buildup of pressures
leading to extreme currency overvaluation and subsequent large discrete currency declines
as may occur under fixed exchange rate regimes.
Pegged and intermediate exchange rate regimes are indeed associated with greater
susceptibility to currency crises, particularly in developing and emerging market coun-
tries with more open capital accounts (Ghosh, Ostry and Tsangarides, 2010). However,
many countries with floating exchange rates have experienced currency crises. A possible
explanation is the fact that countries reporting their currencies as on a floating rate regime
are often quite reluctant to allow their currencies to float due to so-called fear of floating
behavior (Calvo and Reinhart, 2000), and de facto follow a pegged exchange rate regime
(Glick and Hutchison, 2011).
5
2.1 Currency Crisis Definition
How to identify currency crises has been debated since the mid 1990s. Two approaches
can be distinguished: the successful attack approach and the speculative pressure ap-
proach. In the successful attack approach a currency crisis is identified when a currency
depreciates significantly. Frankel and Rose (1996) identify a currency crisis when two
conditions are met: (i) the depreciation of the nominal exchange rate of the currency is
larger than 25% in a year, and (ii) the rate of the nominal depreciation must be 10 per-
centage points larger than it was in the previous year. Variations of this approach have
emerged, with differences in the threshold and sample frequency.
The second approach, known as the speculative pressure approach, is inspired by
Girton and Roper (1977) and later used by Eichengreen, Rose and Wyplosz (1995) and
many others for currency crises. This approach does not only take into account actual
devaluation or depreciation of the currency, but also includes periods of great stress of
the exchange rate. The latter occurs when the monetary authorities avoid a devaluation
or depreciation through the use of its international reserves or by increasing interest rates.
Although the ‘currency attack’ was unsuccessful, one may argue that periods of exchange
rate pressure should be considered a crisis. The measure for speculative pressure is an
index of the weighted average of changes in the exchange rate, reserves and interest rate.
The index is known as the Exchange Market Pressure Index (EMPI). Many variations have
been proposed, with the adjusted definition from Kaminsky, Lizondo and Reinhart (1998)
being the most common for emerging economies. The index consists of the weighted
average of monthly changes in the nominal exchange rate versus the US dollar and the
monthly percentage change of the international reserves, measured in US dollars. In
contrast to Eichengreen, Rose and Wyplosz (1995), they do not include the interest rate
in the definition. Their argument is that in emerging economies interest rate spreads
are not always available or useful. Hawkins and Klau (2000) comment that there are
periods where interest rates were controlled rather than market-determined. However, the
omission of interest rates in the EMPI is also recognized as a shortcoming in identifying
6
crises (Berg, Borensztein and Patillo, 2005). Two often cited examples are the attacks
on the Argentinian peso and the Hong Kong dollar in 1995 and 1997 respectively, which
were deterred by a rapid increase in domestic interest rates (Hawkins and Klau, 2000, and
Klaassen and Jager, 2011). In Section 4.1 we explain how we identify currency crises.
2.2 Early Warning Systems for Currency Crises
Currency crises can be costly, particularly when they lead to sovereign debt or banking
crises. It is therefore important to signal currency crises in a timely manner such that a
crisis can be avoided, or the impact can be reduced. Early warning systems are models
that send a signal well in advance of a crisis. Over the years dozens of EWSs have
been developed which differ widely in the definition of a currency crisis, the estimation
period and the countries included in the database, the inclusion of indicators, the forecast
horizon and the statistical or econometric method. For overviews see Kaminsky, Lizondo
and Reinhart (1998) or Abiad (2003). The differences between the EWSs make it hard to
compare the studies.
3 Methodology
This section introduces two of the most used types of EWS to predict currency crises,
the signal approach and the discrete choice model. The approaches will be described in
the next subsections. The final subsection compares the out-of-sample currency crisis
prediction performance using current-vintage data and using second estimates.
3.1 Signal Approach
Eichengreen, Rose and Wyplosz (1995) and Frankel and Rose (1996) introduce the event
study graph to analyze and predict currency crises. The method involves a graphical
comparison of the performance of indicators in times of crisis versus their performance
in tranquil periods. Kaminsky, Lizondo and Reinhart (1998) extend this methodology
7
to what is known as the signal approach. This approach consists of two stages. In the
first stage the indicators that are expected to play a role in the crisis, such as inflation,
debt as a percentage of GDP and the current account, are selected. Typically, a visual
inspection of the event study graph determines whether the indicator shows a special,
extraordinary behavior before a crisis. This helps to restrict the number of potential crisis
indicators. In the second stage a threshold is determined for each indicator by minimizing
the probability of not signalling a crisis that occurred (type I error) and the probability
of signalling a crisis that did not occur (type II error). If the variable exceeds a pre-
established threshold, then a crisis is signaled and the value of 1 is assigned to the binary
variable, and zero otherwise. For each threshold we construct a contingency table as in
Table 1. A represents the number of observations in which the model signals a crisis that
actually took place (correct crisis signals); B corresponds to the number of observations
in which the model signals a crisis that did not take place (false alarms, also known as
type II errors); C is the number of observations in which the model does not signal a
crisis that actually took place (missed crises, a.k.a type I errors); and D is the number of
observations in which the model does not signal a crisis that did not take place (correct
non-crisis signals).
Table 1: Contingency table of crisis realizations and signals.
Realization
Signal Crisis No crisis
Crisis A BNo crisis C D
The main advantages of the signal approach are that the method does not impose any
parametric structure on the data, and that the method is more accessible and informa-
tive than tables of coefficient estimates. The main disadvantage is that the approach is
intrinsically univariate as we analyze the individual contribution instead of the marginal
contribution conditional on other variables (Frankel and Rose, 1996).
8
Determining the Level of the Threshold
The higher the threshold, the less likely it is for the indicator to send a crisis signal. This
will result in less false alarms (type II errors), but also in more missed crises (type I
errors). A lower threshold leads to less missed crises, but more false alarms. The optimal
threshold depends on the relative costs of the two error types. There are several ways to
determine the optimal threshold. We use the noise-to-signal ratio and the loss function.3
The letters (A, B, C, D) used in the formulas below refer to the categories in Table 1.
Noise-to-Signal Ratio
The first method is used in the original signal approach model of Kaminsky, Lizondo and
Reinhart (1998). The noise-to-signal (NtS) ratio is defined as:
NtS =B/(B +D)
A/(A+ C)(1)
The lower the NtS, the better the variable identifies the actual crisis. Indicators with
an NtS equal to or greater than 1 should be discarded, since these do not have intrinsic
predicting power. According to this criterium false alarms and missed crises are treated
equally.
Loss Function
Alessi and Detken (2011) define an alternative criterium, which allows taking into account
the policy maker’s risk aversion. The loss function of the policy maker is defined as:
L = θC
A+ C+ (1− θ) B
B +D, (2)
where θ reflects the policy maker’s relative risk aversion between missed crises (type I
error) and false alarms (type II error). θ can take any value between 0 and 1. A θ smaller
3We also applied the ROC curve methodology, which provides similar results as the loss-functionmethod. Results are available upon request.
9
than 0.5 implies that the policy maker is less risk averse towards missing a signal for a
crisis than for receiving a false alarm. The costs of a false alarm are the costs of taking
preventive actions, the risk of a self-fulfilling prophecy and the loss of trust in the policy
makers when false alarms become frequent. Generally, policy makers and practitioners
will prefer to be ‘better safe than sorry’. For them missed crises are far more important
than false alarms. Bussiere and Fratzscher (2006) mention two arguments for the relative
importance of missed crises versus false alarms. First, false alarms are less costly from
a welfare perspective than missed crises. Second, false alarms may have been caused by
appropriate policy initiatives that were taken when the fundamentals were so weak that a
crisis was predicted.
A central banker can always realize a loss of min[θ; 1 − θ] by disregarding the indi-
cator. If θ is smaller than 0.5, the benchmark is obtained by ignoring the indicator, which
amounts to never having any signals issued so that A = B = 0. The resulting loss accord-
ing to Equation (2) is θ. If θ exceeds 0.5, the benchmark indicates that there is always a
crisis, i.e. assuming a signal is always issued so that C = D = 0. The resulting loss is 1 −
θ. An indicator is useful to the extent that it produces a loss lower than min[θ; 1 − θ] for
a given θ. The usefulness of an indicator can then be defined as:
U = min[θ; 1− θ]− L, (3)
where the maximum value is to be determined.
3.2 Logit Models
Logit and probit models are widely used in EWSs for financial crises, including cur-
rency crises. Compared to the signal approach, the logit and probit models have advan-
tages. First, the methods consider all the variables simultaneously (Kaminsky, Lizondo
and Reinhart, 1998), and second, the independent variables can have a nonlinear effect on
10
the probability of a crisis, which is appropriate because of the presence of strong nonlinear
effects in currency crises mechanisms (Bussiere, 2007).
3.2.1 Binary Logit Model
In the binary logit model the dependent variable is dichotomic and takes the value of 1
if the event occurs and 0 otherwise. In this setup, Yit represents a binary variable for
country i ∈ {1, . . . , N} at time t ∈ {1, . . . , Ti} where Ti denotes the number of time
periods considered for the ith country. The probability of an event is characterized by the
logistic distribution. That is, for each country, the probability of the event is given by:
P (Yit = 1) =exp(Xitβ)
1 + exp(Xitβ), i = 1, . . . , N ; (4)
where Xit denotes a vector of exogenous variables and β the vector of slope parameters.
The odds ratio, which is useful for interpretations, is determined as
P (Yit = 1)
1− P (Yit = 1)= exp(Xitβ). (5)
A common alternative for discrete choice models is the probit model. However, in
our setup the logit model is preferred over the probit model because a crisis event has
a relatively low frequency (as is the case in currency crises and sovereign debt crises).
The reason for which the logit model is preferred is because the logistic distribution (logit
model) has heavier tails than the normal distribution (probit model) (Manasse, Roubini
and Schimmelpfennig, 2003; Bussiere, 2007). However, differences are small (Cornelli,
2014).
3.3 Out-of-Sample Performance
We estimate an Early Warning System for the period 1990–2009, with current-vintage
data. Then we compare its out-of-sample prediction for the period 2010–2014, with (i)
current-vintage data, and (ii) original, second estimates. We include the currency crises
11
that occur in the last half of 2008 (the fall of Lehmann Brothers) in the in-sample period,
because it has elements not seen in earlier crises such as the role of advanced economies.
According to Frankel and Saravelos (2012) leading indicators that most frequently ap-
peared in earlier reviews are not statistically significant indicators in the Global Financial
Crisis.
Measures for the Out-of-Sample Performance
The methods to determine the optimal threshold, the Noise-to-Signal or loss function
approach (see Section 3.1), can also be used to measure the out-of-sample performance
of the signal approach and the binary logit model. For the logit model an additional
measure is available which is the quadratic probability score (QPS) proposed by Diebold
and Rudebusch (1989) to evaluate out-of-sample forecasts. This measure indicates how
close, on average, the predicted probabilities Pt and the observed realizations Zt are. The
QPS is given by
QPS =1
T
T∑t=1
2(Pt − Zt)2.
The QPS ranges from 0 to 2, with a score of 0 corresponding to perfect accuracy (well-
predicted crisis, or a well-predicted tranquil period), and a score of 2 corresponding to a
perfect false signal (missed crisis or false alarm).
4 Data
We focus on emerging economies, in particular on two regions: Latin America and Cen-
tral and Eastern Europe (CEE) for which we have second estimates available. Countries
in these regions implemented market reforms in the 1990s after a period of domestically-
oriented economic policies. All countries have experienced political and institutional
changes, changes in exchange rate regimes and at least one currency crisis since the early
1990s. In terms of total GDP and GDP per capita the regions are comparable, as shown
12
in Table 2. For Latin America we select Argentina, Brazil, Mexico and Venezuela. These
countries are the largest economies in terms of GDP and share economic and institutional
features, such as the importance of commodities, and a history of changes in exchange
rate regimes, and political and institutional changes. For CEE we select the four largest
economies in the region: Russia, Poland, Czech Republic and Hungary.
Table 2: Comparison GDP and GDP per capita, 2014
Country GDP Ranking GDP per capita RankingArgentina 332.6 26 7, 956 64Brazil 1, 206.1 11 5, 970 76Mexico 1, 067.1 13 8, 626 60Venezuela 186.9 42 6, 057 75Czech Rep. 157.1 45 14, 945 44Hungary 117.2 54 11, 888 50Poland 429.5 22 11, 305 54Russia 999.8 14 6, 844 69
Notes: total real GDP in billions of constant 2005 USD. Real GDP per capita in con-stant 2005 USD.Ranking refers to world ranking (authors’ calculations).Source: World Bank (2014).
4.1 Currency Crisis Dating
To identify currency crises we combine the speculative pressure approach (EMPI) and
the successful attack approach (large depreciation). An advantage of the speculative pres-
sure approach is that the method works well under both fixed and floating exchange rate
regimes. Frankel and Saravelos (2010) state that the inclusion of reserves is particularly
relevant for countries with fixed exchange rate regimes, because capital flight and crisis in-
cidence are present through larger drops in reserves rather than exchange rate weakness.
This is very useful for our panel that starts in the early 1990s, when various emerging
economies had their currencies pegged. A disadvantage of the speculative pressure ap-
proach is that the threshold is based on the standard deviation of the variables. When
the movements in a particular currency crisis are relatively strong then these may prevent
13
other depreciations that should be identified as currency crises to become visible, in other
words: crises with very large EMPI values ‘push out’ other crises. The successful attack
approach is not based on the standard deviation and is therefore used as a complementary
crisis identification method.
For the speculative attack approach we choose for the EMPI with two components,
changes in nominal exchange rate and changes in reserves. We do not include the interest
rate for three reasons. First, the number of observations reduces, as interest rates are not
available for all countries and years. Second, some interest rates are highly unrealistic.
For example, Venezuela has a nominal rate in the period 2011-2014 lower than 1% annual
in most months (source: IFS). Third, as Angkinand, Li and Willett (2006) observed when
comparing crisis dating alternatives:
. . . while the three-component indices with interest rates may be able to pick
up the exchange market pressure from interest rate hikes, they might miss the
crisis periods when authorities intervenes the market with reserves only. In
terms of picking up the mild EMP from selling reserves, the two-component
indices without interest rates seem to be superior to the three-component in-
dices.” [page 16].
The EMPI is standardized with the mean and standard deviation for the in-sample period
(in our case: 1990–2009) and not the entire period (in-sample and out-of-sample), to keep
the out-of-sample exercise as pure as possible. For each country an index is constructed
and a crisis is identified when the index exceeds a threshold, for which Kaminsky, Li-
zondo and Reinhart (1998) use three standard deviations of the index. For the successful
attack approach we use the Frankel and Rose (1996) definition of a crisis. We refine the
definition by also including higher frequency time frames: monthly, three-months and six-
months periods. The reason is that several crises (in particular in the 2008–2009 period)
are not identified because the depreciation takes place during two calendar years (and in
none of the years is large enough to classify as a currency crisis), or the depreciation is
(partially) reversed in the same year.
14
We exclude periods with hyperinflation as they should be categorized as inflation
crises instead of currency crises. We use Cagan (1956)’s hyperinflation definition: a
monthly increase in consumer prices of 50% or more. We define a calender year with
hyperinflation if there is at least one month of hyperinflation.
We label the year prior to the crisis as a crisis run-up year. This run-up year is used
since we are interested in an early warning system, so we want to detect a currency crisis
before it actually occurs. Additionally, we use a window exclusion period of 12 months,
which implies that a crisis that takes place within 12 months after a previous crisis is not
considered a separate crisis, but a continuation of the previous crisis.
The resulting currency crises are shown in Table 3. The fifth column of the table
shows there have been 17 currency crisis episodes in the period 1990–2009, and the last
column shows there have been seven currency crisis episodes in the out-of-sample period
2010–2014.
4.2 Explanatory Variables
As explanatory variables we use 10–15 economic and financial indicators that correspond
to different types of currency crises (Kaminsky, 2006), and that are typically used in stud-
ies on EWSs for currency crises. The following variables are available as second estimates
for the out-of-sample period 2010–2014: current account balance as a percentage of GDP,
import cover, import growth, real GDP growth in the US, a selection of commodity price
indices (agricultural goods, metals, energy), inflation, real GDP growth, and gross fixed
capital investments. The second estimates are taken from the Focus Economic Consensus
of Haver Analytics (HA). The database offers monthly reported forecasts for economic
indicators. Current account balance and inflation are not available in HA, so we take these
from World Economic Outlook (WEO)—in a semi-annual vintage frequency (April and
October). We take the second estimates from HA as published in January for the year in
course, and from WEO as published in October of the previous year. This is an ad hoc
choice, that is a trade-off between accuracy and timeliness. Forecasting over a long time
15
Table 3: Identification of currency crises, 1990–2014
Country Starting Definition 1: Definition 2: Combined Combinedyear EMPI Depreciation 1990–2009 2010–2014
Argentina 1992 2002 2002, 2013,2014
2002 2013, 2014
Brazil 1995 1999 1999, 2001-2002, 2008
1999, 2001-2002, 2008
Mexico 1990 1994-1995, 2008 1994-1995,2008-2009
1994-1995,2008-2009
Venezuela * 1990 1994, 1995-1996, 2002-2003, 2009,2011, 2013
1993-1996,2002, 2011,2013
1993-1996,2002-2003,2009
2011, 2013
Czech Republic 1993 1997, 2008-2009 1997, 2008-2009
Hungary 1990 1993, 2008-2009 1996, 2008-2009, 2011
1993, 1996,2008-2009
2011
Poland 1991 2008-2009, 2011 2008-2009 2011Russia ** 1994 1995, 1998 1994-1995,
1998-1999,2008-2009, 2014
1998-1999,2008-2009
2014
Notes:Starting year: We excluded years with hyperinflation, and years without available data.Definition 1: a crisis is identified when the EMPI is greater than three times the standard deviation.Definition 2: a crisis is identified when: (i) annual depreciation is greater than 25%, (ii) annual depreciation10 percentage points higher than previous year’s annual depreciation. Additionally, a crisis is identified inthe year in which depreciation of monthly, three-months or six-months periods exceeds 25%.Combined: combines definitions 1 and 2, with the implementation of a 12 months window exclusion pe-riod. We distinguish the in-sample period (1990–2009) in the penultimate column from the out-of-sampleperiod (2010–2014) in the ultimate column.* There is no data available on the reserves in 2014 in Venezuela, so no EMPI was calculated.** Currency crises that occur at the start of the in-sample period are excluded, because the run-up periodis not part of the in-sample period. The 1994-1995 crisis in Russia is therefore excluded.Data source: IFS.
period typically leads to greater variation relative to the actual outcome, while forecasting
over a shorter period may lead to more precision, but also leaves less time for corrective
action. Berkmen et al. (2009) use consensus growth forecast changes, which has the ad-
vantage of pooling across various forecasters and potentially suffering from less bias than
the WEO. For the same argument we prefer to use the consensus data from HA, and only
include WEO predictions when these data series are not available in HA.
The following variables are not available as second estimates, but are essential to in-
clude in any EWS for currency crises: general government gross debt as a percentage
16
of GDP, domestic credit from financial institutions, broad money growth, foreign direct
investments, portfolio investments, and changes in the US interest rate. For these vari-
ables and for current-vintage data we use International Financial Statistics (IMF), Haver
Analytics and World Development Indicators. Details on the variables, their sources, fre-
quency and availability are shown in the Appendix. All explanatory variables have been
standardized, using the mean and standard deviation of the in-sample period (1990-2009)
to keep the out-of-sample prediction exercise as pure as possible.
5 Empirical Results
Since early warning systems are only useful when they send an early warning, it is com-
mon to use indicators from year t− 1 to detect a possible crisis in year t. In other words,
the model links the dependent variable (the crisis dummy) with a selection of indicators
from the year prior to the crisis entry on the dependent variable (the crisis dummy).
5.1 Signal Approach
In the signal approach we determine the optimal threshold separately for each indicator.
For each of the thirteen selected variables we analyze the performance according to two
criteria: the Noise-to-Signal ratio and the loss function (the latter with a relatively high
penalty for missed crises, i.e. θ ≥ 0.5). We select the five indicators that are in top po-
sitions according to these two criteria: Current Account to GDP ratio, import cover, M2
growth, domestic credit by financial institutions and changes in food prices. Of these, M2
growth and domestic credit by financial institutions are not available as real-time data,
that is second estimates are not available, which makes them not suited for our compar-
ison. We discard changes in food prices, because we consider the countries produce and
consume (export and import) different foods, where the prices are not uniformly increas-
ing and decreasing (the commodity price lottery as documented by Blattman, Hwang and
17
Williamson, 2007). We continue our analysis with the other two indicators, change in
import cover and the Current Account to GDP ratio.
Import cover, defined as the ratio of reserves to imports, is a signal of currency sta-
bility. The ratio is typically expressed in months and can be interpreted as the number
of months that a country can continue to import financed by international reserves only.
A deterioration in the import cover can be caused by a decrease in reserves and/or an
increase in imports. A decrease in the import cover ratio is associated with a higher prob-
ability of a currency crisis. We use two values for the thresholds of this indicator, −0.7σ
and −0.1σ. Since all indicators have been standardized, the threshold can be expressed
simply as−0.7 and−0.1. The first (−0.7) is the optimal threshold according to the noise-
to-signal ratio and the loss function with a relatively mild penalty for missed crises. For
stronger penalties for missed crises, the optimal threshold is−0.1. For both thresholds we
show the contingency table, in Table 4. With a threshold of −0.7 the approach generates
only 18 false alarms, but at the cost of having a high number of missed crises. With a
lower threshold in absolute terms (−0.1), the approach identifies more crises (11), but
also generates more false alarms (73). The latter threshold would be preferred by a policy
maker that is more averse to missed crises than to false alarms.
Table 4: Contingency tables for change in import cover
Threshold −0.7 −0.1
Correct crisis signal 6 11Missed crisis (type I error) 11 6False alarm (type II error) 18 73Correct non-crisis signal 101 46
The pattern of the indicator, the currency crises and the threshold is shown in Figure 2.
Using a threshold of −0.7 six crises are correctly identified: Argentina in 2002, Brazil
in 1999 and 2001, Venezuela in 1993 and 2002, and Czech Republic in 1997. With a
threshold of −0.1 eleven crises are correctly identified. Apart from these six crises, also
18
the crises of Mexico in 2008, Czech Republic in 2008, Hungary in 2008, and Russia in
1998 and in 2008 are identified correctly.
Another indicator that is commonly used in EWSs for currency crises is the current
account balance, which is a signal of an economy’s external trade position, and indirectly
the capital account.4 A worsening current account deficit puts the exchange rate under
pressure and may result in a depreciation of the currency, and/or depletion of international
reserves to finance the deficit. As with the previous indicator, we use several threshold
values, −1.4, −0.3 and 0.8. The first threshold is optimal when missed crises are not
punished stronger than false alarms (optimal threshold according to Noise-to-Signal ra-
tio), the second threshold is optimal when missed crises are punished relatively mildly
(optimal threshold according to loss function with equal risk aversion for missed crises
and false alarms), and the third threshold is optimal when missed crises are punished rela-
tively strongly (optimal threshold according to loss function with higher risk aversion for
missed crises than for false alarms). For both thresholds we show the contingency tables
in Table 5. The pattern of the indicator, the currency crises and the thresholds are shown
in a time line in Figure 3.
Table 5: Contingency tables for current account balance to GDP ratioThreshold −1.4 −0.3 0.8
Correct crisis signal 4 12 16Missed crisis (type I error) 13 5 1False alarm (type II error) 5 50 89Correct non-crisis signal 114 69 31
The performance of the signal approach using the change in import and the current ac-
count balance is very similar.
4A current account deficit (surplus) is typically accompanied by a capital account surplus (deficit).
19
Figure 2: Signal approach for the change in import cover in eight countries during 1990–2009.
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1993 1995 1997 1999 2001 2003 2005 2007 2009
Argentina
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1996 1998 2000 2002 2004 2006 2008
Brazil
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Mexico
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Venezuela
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1994 1996 1998 2000 2002 2004 2006 2008
Czech Republic
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Hungary
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1992 1994 1996 1998 2000 2002 2004 2006 2008
Poland
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
1995 1997 1999 2001 2003 2005 2007 2009
Russia
Notes: The dots represent the realization of the binary variable that takes the value of 1 if there was a crisisand zero otherwise. The two horizontal lines represent the thresholds, the bottom line the threshold of−0.7 and the top line the threshold of −0.1. The solid line is the standardized movement of the change inthe import cover, with a 1 year lag.
20
Figure 3: Signal approach for the current account balance to GDP ratio.
-2.5
-1.5
-0.5
0.5
1.5
2.5
1993 1995 1997 1999 2001 2003 2005 2007 2009
Argentina
-2.5
-1.5
-0.5
0.5
1.5
2.5
1996 1998 2000 2002 2004 2006 2008
Brazil
-2.5
-1.5
-0.5
0.5
1.5
2.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Mexico
-2.5
-1.5
-0.5
0.5
1.5
2.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Venezuela
-2.5
-1.5
-0.5
0.5
1.5
2.5
1994 1996 1998 2000 2002 2004 2006 2008
Czech Republic
-2.5
-1.5
-0.5
0.5
1.5
2.5
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Hungary
-2.5
-1.5
-0.5
0.5
1.5
2.5
1992 1994 1996 1998 2000 2002 2004 2006 2008
Poland
-2.5
-1.5
-0.5
0.5
1.5
2.5
1995 1997 1999 2001 2003 2005 2007 2009
Russia
Notes: This figure shows the signal approach for the current account balance as a percentage of GDP foreight countries from 1990 to 2009. The dots represent the realization of the binary variable that takes thevalue of 1 if there was a crisis and zero otherwise. The two horizontal lines represent the thresholds: thebottom line is the threshold of −1.4 and the top line is the threshold of −0.3. The solid line is thestandardized movement of the current account to GDP ratio, with a one-year lag.
21
Out-of-Sample Performance
We compare the performance of using second estimates versus current-vintage data in the
predictions. We use the thresholds that were estimated in the in-sample period (1990–
2009), to predict crises in the out-of-sample period 2010–2014.
Table 6: Signal approach: out-of-sample results for the change in import cover.
Threshold -0.7 Threshold -0.1Country Actual Current 2nd Estimates Current 2nd Estimates
crisis Vintage Vintage
Argentina 2013–2014 2011–2013 2011, 2012 2011–2014 2011, 2012, 2014Brazil – 2011, 2014 2011, 2012 2011, 2013, 2014 2011, 2012Mexico – – – 2011, 2013, 2014 2011–2013Venezuela 2011, 2013 2011, 2012, 2014 – 2011–2014 2011, 2012Czech Rep. – 2012 – 2011, 2012 2011, 2012, 2014Hungary 2011 – – 2011, 2013, 2014 2011, 2012, 2014Poland 2011 – – 2011, 2012, 2014 2011, 2012, 2014Russia 2014 2011, 2012 – 2011–2014 2011, 2012, 2014
# correct crises 2 0 6 4# missed crises 4 6 0 2# false alarms 9 4 19 17# correct non-crises 17 22 7 9
Noise to Signal ratio 1.04 N/A 0.73 0.98Usefulness of the loss function a la Alessi and Detken (2011) with:θ = 0.5 0.08 -0.04 0.32 0.17θ = 0.6 -0.08 -0.24 0.22 0.04θ = 0.7 -0.24 -0.43 0.15 -0.07θ = 0.8 -0.39 -0.62 0.08 -0.17
Notes: Current-vintage refers to the data vintage as available in June 2015. Second estimates refer to theuse of forecasts compiled by Haver Analytics in the month January for the current year. The top section ofthe table contains the crisis years, as took place according to our definition (column 2), and according to thepredictions with the Signal Approach, using current-vintage data (columns 3 and 5) and second estimates(columns 4 and 6). The middle section of the table summarizes the model’s performance. The observationsin the out-of-sample period are divided over the four possible categories: correctly predicted crises, missedcrises, false alarms and the correctly predicted tranquil years. The bottom section of the table shows two cri-teria that we used to measure and compare the out-of-sample performance.
For the change in the import cover as shown in Table 6 we compare column 3 versus 4,
and column 5 versus 6. We can see that the current-vintage data provides better predic-
22
tion results than the second estimates, because more crises are correctly picked up, the
noise-to-signal ratio is lower and the usefulness is higher for all θ. Monetary authorities
will prefer the threshold with the highest value (−0.1), because this will lead to a higher
number of correctly predicted crises, reflected in a higher usefulness (comparing columns
3 and 5 of Table 6, e.g. for θ = 0.5: 0.10 ≥ -0.10).
For the other indicator, current account balance as a percentage of GDP, we discard
one threshold (−1.4), because no crisis signal was sent, neither when using current-
vintage data nor when using second estimates. Therefore in Table 7 we show the per-
formance with thresholds −0.3 and 0.8.
Table 7: Signal approach: out-of-sample results for current account balance as a percent-age of GDP.
Threshold -0.3 Threshold 0.8Country Actual Current 2nd Estimates Current 2nd Estimates
crisis Vintage Vintage
Country actual Current 2nd estimate Current 2nd estimatecrisis vintage of forecast vintage of forecast
Argentina 2013-2014 — — 2011-2014 2012-2014Brazil — 2013-2014 2012-2014 2011-2014 2011-2014Mexico — — — 2012-2014 2011-2014Venezuela 2011, 2013 2011, 2013-2014 — 2011-2014 2011-2014Czech Rep. — — — 2011 2011, 2013Hungary 2011 — — — —Poland 2011 2011-2012 2013 2011-2013 2011-2014Russia 2014 2011-2014 2011-2014 2011-2014 2011-2014
# correct crises 4 1 5 5# missed crises 2 5 1 1# false alarms 7 7 18 20# correct non-crises 19 19 8 6
Noise to Signal 0.49 1.96 0.79 0.89Usefulness, based on loss function with:θ = 0.5 0.22 0.00 0.26 0.24θ = 0.6 0.08 -0.18 0.15 0.13θ = 0.7 -0.06 -0.36 0.06 0.04
Notes: see Table 6.
23
We find very similar results for this second indicator. Current-vintage data generates
better results in terms of lower noise-to-signal ratios, and higher usefulness values. The
difference is marginal when using the threshold of 0.8. For the threshold of −0.3 the cur-
rency crises in Venezuela in 2011 and 2013 as well as the crisis in Poland 2011 are picked
up when we use current-vintage data, but not with second estimates. Monetary authorities
that are averse to missed crises would prefer the highest threshold (0.8), because this will
lead to less missed crises. However, our results do not capture the possible actions taken
when a crisis signal arrives and the costs, including the loss in credibility.
The predictions based on current-vintage data are better than the ones based on second
estimates. More crises are predicted correctly and higher usefulness figures are obtained
for all levels of θ.
5.2 Binary Logit Model
We estimate the model for the in-sample period (1990–2009) of the pooled dataset for
eight countries. We use the Hausman test to decide between fixed or random effects. In
addition, we use a regional dummy to distinguish the four Latin American countries from
the four Central and Eastern European countries. We include all potentially relevant vari-
ables, even when there could be multicollinearity among variables. Under multicollinear-
ity the estimates are unbiased and consistent, but the standard deviations may be inflated.
This does not affect the predictions.
The outcomes of the logit model are shown in Table 8, for the pooled observations of
the eight countries for the period 1990–2009. The dependent variable is a dummy variable
with value 1 if the country experiences a currency crisis. A value of 1 is also assigned to
the pre-crisis year, the run-up to the currency crisis.
We now turn to the performance in the out-of-sample period, by comparing the gener-
ated probabilities of a currency crisis with the actual outcome. The out-of-sample perfor-
mance is shown graphically in Figure 4. We perform the out-of-sample predicting with
(i) current-vintage data, and (ii) second estimates. In Figure 4 we observe that predictions
24
Table 8: Binary logit regressions with currency crisis dummy as the dependent variable
Variables Coefficient Std. ErrorCAGDP −0.520 (0.470)∆IMPCOVER −0.643 (0.611)INFLAT 1.213∗ (0.666)GDPGROW 1.513∗ (0.892)IMPGROW −2.759 ∗ ∗∗ (0.937)∆USGROW −0.262 (0.275)AGRI 0.908 ∗ ∗∗ (0.350)METAL −1.032 ∗ ∗ (0.429)ENERGY 0.137 (0.328)FDI 0.288 (0.350)∆GDEBTGDP 0.698 (0.615)∆CREDFIN 0.678 (0.590)∆PFINV 0.141 (0.468)∆M2GROW 0.046 (0.526)∆USINT 0.373 (0.325)REGION −0.846 ∗ ∗ (0.406)REGION × CAGDP 0.507 (0.634)REGION ×∆IMPCOVER 0.427 (0.880)REGION × INFLAT −0.711 (0.741)REGION × GDPGROW −1.714 (1.148)REGION × IMPGROW 1.723 (1.156)REGION × FDI 0.208 (0.531)REGION ×∆GDEBTGDP −1.009 (0.808)REGION ×∆CREDFIN −0.401 (0.737)REGION ×∆PFINV −0.004 (0.546)REGION ×∆M2GROW 0.092 (0.714)Log likelihood -57.428McFadden pseudo R2 0.366Total observations 132Observations of crisis 46
Notes: Countries in the sample: Argentina, Brazil, Mexico, Venezuela,Czech Republic, Hungary, Poland and Russia; annual data from 1990to 2009. CAGDP = Current Account to GDP ratio; ∆IMPCOVER= percentage change in import cover (reserves / imports); INFLAT =consumer price inflation; GDPGR = Real annual GDP growth; IMP-GROW = imports growth; ∆USGROW = change in US real GDPgrowth; AGRI, METAL, ENERGY are the annual changes in the com-posite price indices for agricultural products, metals and minerals,and energy, respectively; FDI = Foreign Direct Investment to GDPratio; GDEBTGDP = Gross central government debt to GDP ratio;CREDFIN = domestic credit given by financial institutions; PFINV =portfolio investments to GDP ratio; M2GROW = Annual M2 growth;USINT = interest rate on 10 year US treasury bonds; REGION =dummy variable with value 1 for the (four) CEE countries. All vari-ables in this in-sample estimation are current vintage data.
25
based on current-vintage data produce a higher probability of crises than predictions based
on second estimates. The predictions based on current-vintage data yield a low number
of missed crises, but at the cost of a high number of false alarms. In contrast predictions
based on second estimates do not pick up many crises and issue few false alarms.
Figure 4: Probability of currency crises: binary logit regressions
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Argentina
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Brazil
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Mexico
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Venezuela
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Czech Republic
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Hungary
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Poland
0%
20%
40%
60%
80%
100%
2011 2012 2013 2014
Russia
Notes: The dots represent the realization of the binary variable that takes the value of 1 if there was a crisisand zero otherwise. The solid line represents the probability of a crisis when using the current-vintagedata. The dotted line represents the probability of a crisis when using the second estimates.
To compare the out-of-sample performance we determine a cut-off probability to dis-
tinguish crisis signals from non-crisis signals. Through a grid search we find that the
26
Table 9: Binary logit model: out-of-sample performance.
Country Actual Crisis Current Vintage 2nd Estimates
Argentina 2013-2014 2011, 2012, 2013 2013Brazil – 2012, 2014 2013, 2014Mexico – 2011, 2012, 2013 –Venezuela 2011, 2013 2011, 2012, 2013 2011, 2013Czech Republic – 2011, 2012 –Hungary 2011 2011, 2012 –Poland 2011 2011 –Russia 2014 – –
# correct crises 5 3# missed crises 1 3# false alarms 11 2# correct non-crises 12 21
Noise to Signal 0.57 0.17
Usefulness for the loss function a la Alessi and Detken (2011) with:θ = 0.5 0.80 0.73θ = 0.6 0.79 0.68θ = 0.7 0.78 0.63θ = 0.8 0.79 0.59
Notes: Current-vintage refers to the data as available in June 2015. 2nd es-timates refers to the use of forecasts compiled by Haver Analytics in themonth January for the current year for HA data, and October of the previ-ous year for WEO-data.The top section of the table contains the crisis years according to our def-inition (column 2), and according to the predictions with the logit model,using current-vintage data (column 3) and second estimates (column 4).The middle section of the table summarizes the model’s performance. The29 observations in the out-of-sample period are divided over the four pos-sible categories: correctly predicted crises, missed crises, false alarms andthe correctly predicted tranquil years.The bottom section of the table shows two criteria that we used to measureand compare the out-of-sample performance.
27
Table 10: Quadratic probability score for binary logit regressions: out-of-sample results2011–2014.
Current-Vintage 2nd Estimates
All countries 0.413 0.433
Latin America 0.542 0.453Central Europe 0.285 0.414
Argentina 0.515 0.570Brazil 0.910 0.375Mexico 0.537 0.212Venezuela 0.084 0.763
Czech Republic 0.289 0.053Hungary 0.310 0.662Poland 0.148 0.624Russia 0.364 0.431
Countries with crises 0.289 0.599Countries without crises 0.578 0.213
Notes: Current-vintage refers to the data as available in June2015. 2nd estimates refers to the use of estimates compiled byHaver Analytics in the month January for the year in course. TheQPS score ranges between 0 and 2, where a lower score reflectsa better out-of-sample performance.
optimal cut-off (in terms of the lowest Noise-to-Signal ratio, and the highest values for
usefulness for the loss function a la Alessi and Detken, 2011) is 50%. Thus, when the
probability of a crisis exceeds 50% then the logit model calls a crisis. The results are
shown in Table 9. We observe that using current-vintage data identifies 5 crises (versus
3 when using second estimates), although at the cost of more false alarms (11 versus 2).
Note that according to the Noise-to-Signal ratio the second estimates are considered better
for crisis predictions. We give more attention to the usefulness because we consider the
costs of missed crises higher than the cost of false alarms; the Noise-to-Signal ratio does
not make any distinction.
A more formal statistic for the performance of the predictions, the Quadratic Prob-
ability Score, is presented in Table 10. Here we can observe that current-vintage time
data do not always perform better than second estimates. In particular for countries where
28
no crisis took place (Brazil, Mexico and Czech Republic), the QPS for predictions based
on second estimates is lower than the QPS for predictions based on current-vintage data,
indicating that predictions based on second estimates are better. For countries where a
currency crisis took place the reverse holds: the QPS is lower for predictions based on
current-vintage data than for predictions based on second estimates.
6 Conclusion
In this paper we focus on the use of real-time data for early warning systems for currency
crises. EWSs have received many critiques, one of which is related to data availability.
The use of realized data for EWSs is unrealistic and not feasible in practice, since these
are not available when predictions are made. We select eight emerging economies from
two regions: Argentina, Brazil, Mexico and Venezuela from Latin America and Czech
Republic, Hungary, Poland and Russia from Central and Eastern Europe (CEE). These
countries form a more or less homogeneous sample in terms of size of the economy,
comparable economic history since the 1990s including financial crises, and switches in
exchange rate regimes. We analyze these countries in a pooled data set, covering the
period from 1990 to 2014.
The signal approach is often used as EWS for currency crises. Based on the in-sample
data and two criteria we determine two critical thresholds for each indicator separately.
In the out-of-sample period we use these thresholds to identify crises. Comparison of
current-vintage time data and second estimates shows that the latter perform worse in
signalling crises. This conclusion also holds for logit models, the second EWS that we
analyze. Predictions with current-vintage yields higher probabilities of crises than predic-
tions with second estimates; fewer crises are missed, but more false alarms are generated.
29
We conclude that in our models current-vintage data perform better than second esti-
mates in terms of predicting currency crises. In other words, based on second estimates
a high number of crises will not be signaled in time. Some possible explanations are
that the second estimates (which are based on consensus expectations) tend to smooth out
extremes, and that the realizations were more dramatic than predicted in crisis episodes.
A limitation of our work is the small number of crises in the out-of-sample prediction
period, which implies that the results should be taken with care. Future research will
consist of expanding the number of countries. In addition we aim at investigating the im-
pact of using other real-time data, like the descriptive statistics of the information in early
estimates.
Predicting financial crises, including currency crises, is notoriously hard, even in ret-
rospect. Taking properly account of the information that is available at the moment a
researcher has to prepare predictions, makes it even more difficult. For supervisors, who
assume that the costs of missed crises are much higher than the costs associated with
false alarms, this is not a comforting finding, because current-vintage data is not available
on time for making genuine out-of-sample predictions. The EWS literature has focused
mainly on applying different methodologies on current vintage data sets. Given our find-
ings it would be better to dedicate more resources in producing forecasts of the indicators
themselves.
References
Abiad, A. (2003), “Early warning systems: A survey and a regime-switching approach”,
IMF Working Paper 32, International Monetary Fund, Washington, DC.
Alessi, L. and C. Detken (2011), “Quasi real time early warning indicators for costly asset
price boom/bust cycles: A role for global liquidity”, European Journal of Political
Economy, 27, 520–533.
30
Angkinand, A. P., J. Li, and T. Willett (2006), “Measures of currency crises: A survey”,
mimeo Claremont Graduate University, Claremont, CA.
Berg, A., E. Borensztein, and C. Pattillo (2005), “Assessing early warning systems: How
have they worked in practice?”, IMF Working Paper 52, International Monetary Fund,
Washington, D.C.
Berkmen, Pelin, Gaston Gelos, Robert Rennhack, and James P. Walsh (2009), “The
global financial crisis: Explaining cross-country differences in the output impact”,
IMF Working Paper WP/09/280, International Monetary Fund.
Blattman, C., J. Hwang, and J. G. Williamson (2007), “The impact of the terms of trade on
economic development in the periphery, 1870–1939: Volatility and secular change”,
Journal of Development Economics, 82, 156–179.
Bussiere, M. (2007), “Balance of payment crises in emerging markets. How early were
the ‘early’ warning signals?”, ECB Working Paper 713, European Central Bank, Frank-
furt am Main, Germany.
Bussiere, M. (2013), “In defense of early warning systems”, Document de Travail 420,
Banque de France, Paris.
Bussiere, M. and M. Fratzscher (2006), “Towards a new early warning system of financial
crises”, Journal of International Money and Finance, 25, 953–973.
Cagan, P. (1956), “The monetary dynamics of hyperinflation”, in M. Friedman, editor,
Studies in the quantitative theory of money, University of Chicago Press, Chicago.
Calvo, G. A. and C. M. Reinhart (2000), “Fear of floating”, NBER Working Papers 7993,
National Bureau of Economic Research, Cambridge, MA.
Choo, C. W. (2009), “Information use and early warning effectiveness: Perspectives and
prospects”, Journal of the American Society for Information Science and Technology,
60, 1071–1082.
31
Cornelli, F. (2014), “Comparing the performance of logit and probit Early Warning Sys-
tems for currency crises in emerging market economies”, IMF Working Paper 65, In-
ternational Monetary Fund, Washington, DC.
Diebold, F. X. and G. D. Rudebusch (1989), “Scoring the leading indicators”, The Journal
of Business, 62, 369–391.
Eichengreen, B., A. K. Rose, and C. Wyplosz (1995), “Exchange rate mayhem: The
antecedents and aftermath of speculative attacks”, Economic Policy, 21, 251–312.
Frankel, J.A. and A. K. Rose (1996), “Currency crashes in emerging markets: An empir-
ical treatment”, Journal of International Economics, 41, 351–366.
Frankel, J.A. and G. Saravelos (2010), “Are leading indicators of financial crises useful for
assessing country vulnerability? Evidence from the 2008-2009 global crisis”, NBER
Working Paper 16047, National Bureau of Economic Research.
Frankel, J.A. and G. Saravelos (2012), “Can leading indicators assess country vulnera-
bility? Evidence from the 2008–09 global financial crisis”, Journal of International
Economics, 87, 216–231.
Ghosh, R., J. D. Ostry, and C. Tsangarides (2010), “Exchange rate regimes and the sta-
bility of the international monetary system”, IMF Occasional Paper 270, International
Monetary Fund, Washington, D.C.
Girton, L. and D. Roper (1977), “A monetary model of exchange market pressure applied
to the postwar Canadian experience”, American Economic Review, 4, 537–548.
Glick, R. and M. Hutchison (2011), “Currency crises”, FRBSF Working Paper series 22,
Federal Reserve Bank of San Francisco, San Francisco, CA.
Gunther, J. W. and R. R. Moore (2003), “Early Warning Models in real time”, Journal of
Banking and Finance, 27, 1979–2001.
Hawkins, J. and Klau M. (2000), “Measuring potential vulnerabilities in emerging market
economies”, BIS Working Papers 91, BIS, Basel, Switzerland.
32
Holopainen, M. and P. Sarlin (2016), “Toward robust early-warning models: A horse
race, ensembles and model uncertainty”, ECB Working Paper 1900, European Central
Bank, Frankfurt, Germany.
Kaminsky, G. L. (2006), “Currency crises: Are they all the same?”, Journal of Interna-
tional Money and Finance, 25, 503–527.
Kaminsky, G. L., S. Lizondo, and C. M. Reinhart (1998), “Leading indicators of currency
crisis”, IMF Staff Papers 45/1, International Monetary Fund, Washington, DC.
Klaassen, F. and H. Jager (2011), “Definition-consistent measurement of exchange market
pressure”, Journal of International Money and Finance, 30, 74–95.
Lo Duca, M. and T. A. Peltonen (2013), “Assessing systemic risks and predicting systemic
events”, Journal of Banking and Finance, 37, 2183–2195.
Manasse, P., N. Roubini, and A. Schimmelpfennig (2003), “Predicting sovereign debt
crises”, IMF Working Paper 221, International Monetary Fund, Washington, DC.
Reagle, D. and D. Salvatore (2005), “Robustness of forecasting financial crises in emerg-
ing market economies with data revisions—a note”, Open Economies Review, 16, 209–
216.
33
A Variables: Definitions and Sources
Indicator Definition Source and vintage Source and realizeddata
1 Inflation Annual change in the consumerprice inflation index
WEO: 1999–2020 WEO: 1990–2014
HA: 2006–2020 WDI: 1990–2014,except Arg, Ven
2 Broad moneygrowth
Annual change in M2 N/A WDI: 1990–2014
3 Real GDP growth GDP in local currency, in constantprices; annual growth
WEO: 1999–2020 WEO: 1990–2014
HA: 2006–2020 HA: 1990–2014WDI: 1990–2014
4 Gross fixed capi-tal formation
Annual growth of real gross fixedcapital formation
HA: 2006–2020 HA: 1993–2014,WDI: 1990–2014
Idem, % of GDP WEO: 2011–2015 WEO: 1990–2014,WDI: 1990–2014
5 Governmentgross debt toGDP
Government gross debt as a per-centage of GDP
N/A WEO: 1990–2020
6 Short term debt Short term debt as a % of total debt N/A WDI: 1990–20137 Domestic credit Domestic credit provided by finan-
cial sector, as % of GDPN/A WDI: 1990-2014
8 Portfolio invest-ments
Portfolio investments (bonds andequity) inflows
N/A WDI: 1990–2013/2014
9 FDI inflows Foreign Direct Investments inflows,as % of GDP
N/A WDI: 1990–2014
10 Current Accountbalance
Current Account balance, as % ofGDP
WEO: 2004–2015 WEO: 1990–2014
11 Imports growth Annual change in value of importsof goods (USD)
HA: 2006–2015 HA: 1995–2014,WDI: 1990–2014
12 Import cover Calculation: International reserves /(Imports / 12)
HA: 2006–2015 HA: 1995–2014
12B International re-serves
International reserves (excludinggold), in billions of USD
HA: 2006–2015 HA: 1995–2014
WDI: 1990–201412C Value of imports
of goodsValue of imports of goods; billionsof USD
HA: 2006–2020 HA: 1995–2014
Continues on the next page.
34
Indicator Definition Source and vintage Source and realizeddata
13 US interest rate 10-year US government bond yield N/A WDI: 1990–201414 US real GDP
growthQuarterly real GDP in USA; con-verted into annual growth rate
HA: 2009–2015 WDI: 1990–2014
HA: 2009–2015 HA: 2009–201515 Agriculture price
indexAgriculture products price index(2000 = 100), incl. vegetable oils,soy beans, rice, wheat, maize, meat,bananas, seafood, sugar, coffee, teaand cocoa
WB: 2002–2015 WB: 1991–2014
16 Metals price in-dex
Metals and minerals price index(2000 = 100), incl. aluminum, cop-per, iron ore, lead, nickel, steel, tinand zinc
WB: 2002–2015 WB: 1990–2014
17 Energy price in-dex
Energy price index (2000 = 100),consists of coal, crude oil and nat-ural gas
WB: 2002–2015 WB: 1990–2014
18 Regional dummy Distinguish Latin America andCEE
Data sources:WEO: World Economic Outlook. Semi-annual reports.WDI: World Development Indicators. Annual reports.HA: Haver Analytics. Monthly reports.
35