RELIABILITY AT RISKThe supervision of financial models as a casestudy for reflexive economic sociology

A Javier IzquierdoUniversidad Nacional de Educacioacuten a Distancia Madrid Spain

ABSTRACT This article focuses on nancial lsquomodel riskrsquo supervision as atest case for a reexive approach to the sociology of contemporary nancialmarkets Model risk is customarily dened as a statistically signicantrelation between the expectation of a trading loss by a rm and its strategicuse of somehow awed econometric models for asset pricing and markettrading purposes It made its rst public appearance in the mid-1990s as acomponent of a new generation of nancial risk management systems forthe nancial derivatives industry Since then it has been assimilated by themost sophisticated national and international nancial regulatory bodies Aperfect illustration of the thesis of the progressive lsquoembedding of theeconomy into economicsrsquo the forensic practice of nancial reliability trials(backtesting) faces a deep pragmatic dilemma how to distinguish trulyunpredictable error from negligent risk management behaviour in a wildlyrandomized social environmentKey words nancial engineering nancial supervision market risk modelrisk reexive economic sociology social randomness

I believe that Value at Risk econometrics is the alibi bankers will giveshareholders (and the bailing-out taxpayer) to show documented due dili-gence and will express that their blow-up came from truly unforeseeablecircumstances and events with low probability ndash not from taking large risks theydid not understand

(Taleb 1997b 2)

The kind of lsquoquantsrsquo who had thought up Value at Risk econometricmodels were also the ones doing relative value and arbitrage tradingWas it possible that smart people could engineer their way round the failsafemechanisms Was it possible to fool VaR and take hidden risks

(Dunbar 2000 147 on the 1998 deacutebacirccle of Long-Term CapitalManagement)

DOI 10108014616690120046950 69

European Societies3(1) 2001 69-90

copy 2001Taylor amp Francis Ltd

ISSN1461-6696 print

1469-8307 online

1 Introduction

In an inuential theoretical characterization of the radicalized reexiveform of modern culture emergent in advanced industrial societies duringthe second half of the twentieth century sociologist Anthony Giddens hasstressed technological reliability as one of the key elements of this new idealtype of sociocultural conguration The main distinctive feature of thisnew modality of lsquoimpersonal trustrsquo (Shapiro 1987) involved in moderninstitutions is precisely that it is lsquovested not on individuals but on abstractcapacitiesrsquo Giddens thus portrays reliability as a lsquoform of ldquofaithrdquo in whichtrust is vested on probable outcomesrsquo Being the product of a radical sus-pension of our common-sense judgement the increasingly uncriticalgranting of the highest moral value to numerical measures of the degreeof uncertainty of future events expresses lsquoa compromise with somethingmore than mere cognitive understanding it represents in fact areliance upon vague and partial understandings of the ldquoknowledge baserdquoof expert systemsrsquo (Giddens 1993 26ndash7) Among the most importantsanctuaries where this new cult is preached Giddens and the other cham-pions of the theory of lsquoSecond Modernityrsquo invariably single out one par-ticular social arena the new globally integrated variety of capital market

More precisely what these theorists frequently identify as the veryepitome of an explicitly reexive new form of social life are the complexsociotechnical assemblies of the sophisticated risk management systemsused in the trading of nancial derivative products and services such asfutures options and swaps (Giddens 1995 153) Recent theoretical trendsin the sociology of science and technology have developed a variation onthe theme of technical reliability as impersonal trust-building around anetwork theory of lsquoheterogeneous alliancesrsquo which link the living andhighly unstable memories of human biological bodies to the inert anddurable memories incorporated in the natural and technical design ofphysical objects in the form of lsquotechno-economic networksrsquo (Callon 1991)A natural sequel of the techno-economic networks approach to the soci-ology of nancial markets has been the introduction of a generalizedpolitical model namely that of advanced-liberal lsquogovernment at a distancersquo(Miller and Rose 1990) The global game of nancial competition andregulation is then reinterpreted as a concrete realization of a more generaltrend in modern political culture to lsquogovernmentalizersquo (Foucault) indi-vidual collective action by inscribing abstract expert knowledge in theform of technical reliability into the very institutional substrate of socialarrangements (Porter 1995)

Having mainly dealt with the latter research programmes in myprevious work on the sociology of nancial markets (Izquierdo 1999a) Iwill here present a slightly different approach to the subject matter freely

EUROPEAN SOCIETIES

70

inspired by the reexive sociological models of economic cognition andaction developed by Mirowski (1990 1991) and Boltanski and Theacutevenot(1991) My particular account of how the naughty phantom of socialreexivity terries contemporary nancial competition and regulation willfocus on one unexpected outcome of the huge industrial success attainedby a social science endeavour applied nancial economics After tryingunsuccessfully to reduce to mathematics and computer logic a world ofwildness nobody had constructed by design nancial economics hasturned into an articial social systems engineering project This projectlsquocomputational nancial engineeringrsquo intends to build from scratch thekind of transparent cultural machines that could be fully understandableand controllable by the king-scientist However put to work in a marketenvironment where stability is increasingly determined by their beingaccurate computational (econometric) models of nancial risk now haveto face the perverse effects of their own success in the form of new uncon-trolled types of nancial risk Among these technologically inducedsecond-order types of nancial risks is the so-called lsquomodel riskrsquo whichhas recently been the subject of intense regulatory controversy on aninternational scale

The economics of nancial modelling is discussed in the second sectionThe third section presents the concept of lsquomodel riskrsquo while the fourthoffers the basic material on the regime shift in the domain of internationalnancial risk supervisory procedures The nal section constructs a linkbetween the uncertainty of conventional supervisory judgements oneconometric modelsrsquo performance and the ambiguous scientic status ofthe econometric theory of nancial randomness Some troubling socio-logical hypotheses are discussed in the conclusion

2 The political economy of nancial risk modelling

We can distinguish between two separate economic uses of nancial riskmodels First there is what may be called an industrial use associated withthe cost and long-term monopolistic returns of the competitive strategiesdevised by individual rms in the incipient marketplace for computationalnancial risk management systems In the short run however the mainpreoccupation of the users of this nancial experts system is the directnancial use of internal risk control models the gains in allocativeefciency of capital reserves obtained as a consequence of disposing ofmore accurate mathematical models for nancial risk management

The supervisory controversy over sound bank internal risk managementsystems and safe capital reserves (Swary and Topf 1993) falls into thissecond economic dimension of nancial econometrics expert knowledge

Reliability at risk IZQUIERDO

71

The level of capital safety requirements of nancial intermediaries (theratio of reserves to nancial assets) is a key factor in the market competi-tiveness of these rms Traditional base assetndashliability gures in thebalance sheets of nancial dealers need to be complemented with newtypes of standardized risk-accounting data that change the level of capitalsafety requirements Due to the new supervisory regulations (see below)the quantitative calculus of expected trading losses internally performedby a nancial rm has a direct and strong effect on the level of reservesrequired to fully insure a rmrsquos creditors and shareholders against bank-ruptcy Thus it has a direct effect on the rmrsquos nancial bottom line andprotability

However the human social activity of mathematical economic andeconometric modelling is still amenable to a third more direct and explicitkind of economic analysis in terms of costndashbenet and riskndashreturn calcu-lations a second-order type of nancial risk known by nancial analystsengineers and traders as model risk (Derman 1996a 1996b) The publi-cized deacutebacirccle in September 1998 of the large and sophisticated hedgefund Long-Term Capital Management is the most telling example of thedevastating effect that can be produced by these strange forms of nan-cial risk a truly reexive form of economic risk that is produced by theactions of risk-adverse nancial agents themselves using mathematicalasset-pricing models in an intensive and extensive manner to buildnancial insurance policies or risk-hedging instruments the famouslsquonancial derivativesrsquo products such as futures options and swaps con-tracts (Steinherr 1998)

3 Dening model risk

Model risk has been dened as a kind of nancial risk that lsquoresults fromthe inappropriate specication of a theoretical model or the use of anappropriate model but in an inadequate framework or for the wrongpurposersquo (Gibson et al 1998 5) The particular risks and uncertaintiesimplied by the practice of formal scientic inquiry (modelling estimatingand testing) into the economics of nancial markets activity are the verylsquofundamentalsrsquo in the economic sense of the market value of formalnancial knowledge understood as a key competitive resource in themodern world of nance Hence the multiple economic sources of model riskare associated with the almost innite manners of constructing a wrongtheoretical model or using a correct model in the wrong way

In a rst economic approximation the concept of model risk accountsfor the fact that the existence and utilization of different types of formalasset-pricing econometric models can give rise to a wide diversity of

EUROPEAN SOCIETIES

72

theoretical prices for a similar type of nancial product As the discrep-ancy between these theoretical bidndashask prices resolves itself in the marketprocess the use of theoretical prices as inputs for the decision-makingprocess of trading and dealing in real nancial markets is revealed as amajor factor of economic success and failure in contemporary nancialglobal competition1 The second methodological approach to theconcept of model risk focuses on the existence of different types and levelsof error in the practice of economic modelling at the base-theoreticalhypothesis translation into mathematical expression statistical datainputs arithmetical calculations computer softwiring or trading misuses

Model risk in nancial markets appears each time an asset-pricingmodel does not take into account some relevant factor of price variationor else wrongly assumes that the motion of certain stochastic variables canbe imitated by a deterministic process or thinks that price changes can bedescribed by a normal frequency distribution with limit variance range Inother cases ndash even if the model could be thought of as lsquocorrect in prin-ciplersquo or at least not patently erroneous from the point of view of theformal logical arguments mathematical proofs probabilistic test andlsquoencompassingrsquo checks commonly used in academic econometric diagnos-tics ndash markets can disagree with its results in the short term The data usedcan also have been badly estimated or collected or there may have beena mistake during the heuristics searching for its analytical solution Themodel may also have been badly calibrated to mimic real market statisticsThere may even have been coding errors in programming it into the com-puter or the model may have been used in an incorrect way by the naluser (eg a trader may have applied it to price for instruments or marketsfor which it lacked predictive validity) and so on

As has been observed by most nance scholars and professional deriva-tives traders the core mathematical and statistical assumptions built intostandard neoclassical pricing models suffer tremendously when they con-front the structures and processes of real-world nancial trading and rm-wide risk management While equilibrium asset-pricing models forexample characteristically assume that markets are composed of atomizedagents who cannot substantially inuence each other or individuallymanipulate aggregate market prices imitative contagion and herd behav-iour are ubiquitous in real markets and giant reputed investors occasion-ally also lsquomove the marketsrsquo Common nancial models furthermore take

Reliability at risk IZQUIERDO

73

1 A New York-based nancial consultancy Capital Market Risk Advisors (CMRA)recently estimated that 40 per cent of total derivatives-related trading losses were dueto modelling errors up to 27 billion US dollars during 1997 This same consultingrm estimates that in the period 1987 to 1997 some 47 billion of total cumulativederivative losses of 238 billion dollars were due to pricing errors caused by wrong dataor wrong assumptions in asset-pricing models (Stix 1998 27)

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

1 Introduction

In an inuential theoretical characterization of the radicalized reexiveform of modern culture emergent in advanced industrial societies duringthe second half of the twentieth century sociologist Anthony Giddens hasstressed technological reliability as one of the key elements of this new idealtype of sociocultural conguration The main distinctive feature of thisnew modality of lsquoimpersonal trustrsquo (Shapiro 1987) involved in moderninstitutions is precisely that it is lsquovested not on individuals but on abstractcapacitiesrsquo Giddens thus portrays reliability as a lsquoform of ldquofaithrdquo in whichtrust is vested on probable outcomesrsquo Being the product of a radical sus-pension of our common-sense judgement the increasingly uncriticalgranting of the highest moral value to numerical measures of the degreeof uncertainty of future events expresses lsquoa compromise with somethingmore than mere cognitive understanding it represents in fact areliance upon vague and partial understandings of the ldquoknowledge baserdquoof expert systemsrsquo (Giddens 1993 26ndash7) Among the most importantsanctuaries where this new cult is preached Giddens and the other cham-pions of the theory of lsquoSecond Modernityrsquo invariably single out one par-ticular social arena the new globally integrated variety of capital market

More precisely what these theorists frequently identify as the veryepitome of an explicitly reexive new form of social life are the complexsociotechnical assemblies of the sophisticated risk management systemsused in the trading of nancial derivative products and services such asfutures options and swaps (Giddens 1995 153) Recent theoretical trendsin the sociology of science and technology have developed a variation onthe theme of technical reliability as impersonal trust-building around anetwork theory of lsquoheterogeneous alliancesrsquo which link the living andhighly unstable memories of human biological bodies to the inert anddurable memories incorporated in the natural and technical design ofphysical objects in the form of lsquotechno-economic networksrsquo (Callon 1991)A natural sequel of the techno-economic networks approach to the soci-ology of nancial markets has been the introduction of a generalizedpolitical model namely that of advanced-liberal lsquogovernment at a distancersquo(Miller and Rose 1990) The global game of nancial competition andregulation is then reinterpreted as a concrete realization of a more generaltrend in modern political culture to lsquogovernmentalizersquo (Foucault) indi-vidual collective action by inscribing abstract expert knowledge in theform of technical reliability into the very institutional substrate of socialarrangements (Porter 1995)

Having mainly dealt with the latter research programmes in myprevious work on the sociology of nancial markets (Izquierdo 1999a) Iwill here present a slightly different approach to the subject matter freely

EUROPEAN SOCIETIES

70

inspired by the reexive sociological models of economic cognition andaction developed by Mirowski (1990 1991) and Boltanski and Theacutevenot(1991) My particular account of how the naughty phantom of socialreexivity terries contemporary nancial competition and regulation willfocus on one unexpected outcome of the huge industrial success attainedby a social science endeavour applied nancial economics After tryingunsuccessfully to reduce to mathematics and computer logic a world ofwildness nobody had constructed by design nancial economics hasturned into an articial social systems engineering project This projectlsquocomputational nancial engineeringrsquo intends to build from scratch thekind of transparent cultural machines that could be fully understandableand controllable by the king-scientist However put to work in a marketenvironment where stability is increasingly determined by their beingaccurate computational (econometric) models of nancial risk now haveto face the perverse effects of their own success in the form of new uncon-trolled types of nancial risk Among these technologically inducedsecond-order types of nancial risks is the so-called lsquomodel riskrsquo whichhas recently been the subject of intense regulatory controversy on aninternational scale

The economics of nancial modelling is discussed in the second sectionThe third section presents the concept of lsquomodel riskrsquo while the fourthoffers the basic material on the regime shift in the domain of internationalnancial risk supervisory procedures The nal section constructs a linkbetween the uncertainty of conventional supervisory judgements oneconometric modelsrsquo performance and the ambiguous scientic status ofthe econometric theory of nancial randomness Some troubling socio-logical hypotheses are discussed in the conclusion

2 The political economy of nancial risk modelling

We can distinguish between two separate economic uses of nancial riskmodels First there is what may be called an industrial use associated withthe cost and long-term monopolistic returns of the competitive strategiesdevised by individual rms in the incipient marketplace for computationalnancial risk management systems In the short run however the mainpreoccupation of the users of this nancial experts system is the directnancial use of internal risk control models the gains in allocativeefciency of capital reserves obtained as a consequence of disposing ofmore accurate mathematical models for nancial risk management

The supervisory controversy over sound bank internal risk managementsystems and safe capital reserves (Swary and Topf 1993) falls into thissecond economic dimension of nancial econometrics expert knowledge

Reliability at risk IZQUIERDO

71

The level of capital safety requirements of nancial intermediaries (theratio of reserves to nancial assets) is a key factor in the market competi-tiveness of these rms Traditional base assetndashliability gures in thebalance sheets of nancial dealers need to be complemented with newtypes of standardized risk-accounting data that change the level of capitalsafety requirements Due to the new supervisory regulations (see below)the quantitative calculus of expected trading losses internally performedby a nancial rm has a direct and strong effect on the level of reservesrequired to fully insure a rmrsquos creditors and shareholders against bank-ruptcy Thus it has a direct effect on the rmrsquos nancial bottom line andprotability

However the human social activity of mathematical economic andeconometric modelling is still amenable to a third more direct and explicitkind of economic analysis in terms of costndashbenet and riskndashreturn calcu-lations a second-order type of nancial risk known by nancial analystsengineers and traders as model risk (Derman 1996a 1996b) The publi-cized deacutebacirccle in September 1998 of the large and sophisticated hedgefund Long-Term Capital Management is the most telling example of thedevastating effect that can be produced by these strange forms of nan-cial risk a truly reexive form of economic risk that is produced by theactions of risk-adverse nancial agents themselves using mathematicalasset-pricing models in an intensive and extensive manner to buildnancial insurance policies or risk-hedging instruments the famouslsquonancial derivativesrsquo products such as futures options and swaps con-tracts (Steinherr 1998)

3 Dening model risk

Model risk has been dened as a kind of nancial risk that lsquoresults fromthe inappropriate specication of a theoretical model or the use of anappropriate model but in an inadequate framework or for the wrongpurposersquo (Gibson et al 1998 5) The particular risks and uncertaintiesimplied by the practice of formal scientic inquiry (modelling estimatingand testing) into the economics of nancial markets activity are the verylsquofundamentalsrsquo in the economic sense of the market value of formalnancial knowledge understood as a key competitive resource in themodern world of nance Hence the multiple economic sources of model riskare associated with the almost innite manners of constructing a wrongtheoretical model or using a correct model in the wrong way

In a rst economic approximation the concept of model risk accountsfor the fact that the existence and utilization of different types of formalasset-pricing econometric models can give rise to a wide diversity of

EUROPEAN SOCIETIES

72

theoretical prices for a similar type of nancial product As the discrep-ancy between these theoretical bidndashask prices resolves itself in the marketprocess the use of theoretical prices as inputs for the decision-makingprocess of trading and dealing in real nancial markets is revealed as amajor factor of economic success and failure in contemporary nancialglobal competition1 The second methodological approach to theconcept of model risk focuses on the existence of different types and levelsof error in the practice of economic modelling at the base-theoreticalhypothesis translation into mathematical expression statistical datainputs arithmetical calculations computer softwiring or trading misuses

Model risk in nancial markets appears each time an asset-pricingmodel does not take into account some relevant factor of price variationor else wrongly assumes that the motion of certain stochastic variables canbe imitated by a deterministic process or thinks that price changes can bedescribed by a normal frequency distribution with limit variance range Inother cases ndash even if the model could be thought of as lsquocorrect in prin-ciplersquo or at least not patently erroneous from the point of view of theformal logical arguments mathematical proofs probabilistic test andlsquoencompassingrsquo checks commonly used in academic econometric diagnos-tics ndash markets can disagree with its results in the short term The data usedcan also have been badly estimated or collected or there may have beena mistake during the heuristics searching for its analytical solution Themodel may also have been badly calibrated to mimic real market statisticsThere may even have been coding errors in programming it into the com-puter or the model may have been used in an incorrect way by the naluser (eg a trader may have applied it to price for instruments or marketsfor which it lacked predictive validity) and so on

As has been observed by most nance scholars and professional deriva-tives traders the core mathematical and statistical assumptions built intostandard neoclassical pricing models suffer tremendously when they con-front the structures and processes of real-world nancial trading and rm-wide risk management While equilibrium asset-pricing models forexample characteristically assume that markets are composed of atomizedagents who cannot substantially inuence each other or individuallymanipulate aggregate market prices imitative contagion and herd behav-iour are ubiquitous in real markets and giant reputed investors occasion-ally also lsquomove the marketsrsquo Common nancial models furthermore take

Reliability at risk IZQUIERDO

73

1 A New York-based nancial consultancy Capital Market Risk Advisors (CMRA)recently estimated that 40 per cent of total derivatives-related trading losses were dueto modelling errors up to 27 billion US dollars during 1997 This same consultingrm estimates that in the period 1987 to 1997 some 47 billion of total cumulativederivative losses of 238 billion dollars were due to pricing errors caused by wrong dataor wrong assumptions in asset-pricing models (Stix 1998 27)

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

inspired by the reexive sociological models of economic cognition andaction developed by Mirowski (1990 1991) and Boltanski and Theacutevenot(1991) My particular account of how the naughty phantom of socialreexivity terries contemporary nancial competition and regulation willfocus on one unexpected outcome of the huge industrial success attainedby a social science endeavour applied nancial economics After tryingunsuccessfully to reduce to mathematics and computer logic a world ofwildness nobody had constructed by design nancial economics hasturned into an articial social systems engineering project This projectlsquocomputational nancial engineeringrsquo intends to build from scratch thekind of transparent cultural machines that could be fully understandableand controllable by the king-scientist However put to work in a marketenvironment where stability is increasingly determined by their beingaccurate computational (econometric) models of nancial risk now haveto face the perverse effects of their own success in the form of new uncon-trolled types of nancial risk Among these technologically inducedsecond-order types of nancial risks is the so-called lsquomodel riskrsquo whichhas recently been the subject of intense regulatory controversy on aninternational scale

The economics of nancial modelling is discussed in the second sectionThe third section presents the concept of lsquomodel riskrsquo while the fourthoffers the basic material on the regime shift in the domain of internationalnancial risk supervisory procedures The nal section constructs a linkbetween the uncertainty of conventional supervisory judgements oneconometric modelsrsquo performance and the ambiguous scientic status ofthe econometric theory of nancial randomness Some troubling socio-logical hypotheses are discussed in the conclusion

2 The political economy of nancial risk modelling

We can distinguish between two separate economic uses of nancial riskmodels First there is what may be called an industrial use associated withthe cost and long-term monopolistic returns of the competitive strategiesdevised by individual rms in the incipient marketplace for computationalnancial risk management systems In the short run however the mainpreoccupation of the users of this nancial experts system is the directnancial use of internal risk control models the gains in allocativeefciency of capital reserves obtained as a consequence of disposing ofmore accurate mathematical models for nancial risk management

The supervisory controversy over sound bank internal risk managementsystems and safe capital reserves (Swary and Topf 1993) falls into thissecond economic dimension of nancial econometrics expert knowledge

Reliability at risk IZQUIERDO

71

The level of capital safety requirements of nancial intermediaries (theratio of reserves to nancial assets) is a key factor in the market competi-tiveness of these rms Traditional base assetndashliability gures in thebalance sheets of nancial dealers need to be complemented with newtypes of standardized risk-accounting data that change the level of capitalsafety requirements Due to the new supervisory regulations (see below)the quantitative calculus of expected trading losses internally performedby a nancial rm has a direct and strong effect on the level of reservesrequired to fully insure a rmrsquos creditors and shareholders against bank-ruptcy Thus it has a direct effect on the rmrsquos nancial bottom line andprotability

However the human social activity of mathematical economic andeconometric modelling is still amenable to a third more direct and explicitkind of economic analysis in terms of costndashbenet and riskndashreturn calcu-lations a second-order type of nancial risk known by nancial analystsengineers and traders as model risk (Derman 1996a 1996b) The publi-cized deacutebacirccle in September 1998 of the large and sophisticated hedgefund Long-Term Capital Management is the most telling example of thedevastating effect that can be produced by these strange forms of nan-cial risk a truly reexive form of economic risk that is produced by theactions of risk-adverse nancial agents themselves using mathematicalasset-pricing models in an intensive and extensive manner to buildnancial insurance policies or risk-hedging instruments the famouslsquonancial derivativesrsquo products such as futures options and swaps con-tracts (Steinherr 1998)

3 Dening model risk

Model risk has been dened as a kind of nancial risk that lsquoresults fromthe inappropriate specication of a theoretical model or the use of anappropriate model but in an inadequate framework or for the wrongpurposersquo (Gibson et al 1998 5) The particular risks and uncertaintiesimplied by the practice of formal scientic inquiry (modelling estimatingand testing) into the economics of nancial markets activity are the verylsquofundamentalsrsquo in the economic sense of the market value of formalnancial knowledge understood as a key competitive resource in themodern world of nance Hence the multiple economic sources of model riskare associated with the almost innite manners of constructing a wrongtheoretical model or using a correct model in the wrong way

In a rst economic approximation the concept of model risk accountsfor the fact that the existence and utilization of different types of formalasset-pricing econometric models can give rise to a wide diversity of

EUROPEAN SOCIETIES

72

theoretical prices for a similar type of nancial product As the discrep-ancy between these theoretical bidndashask prices resolves itself in the marketprocess the use of theoretical prices as inputs for the decision-makingprocess of trading and dealing in real nancial markets is revealed as amajor factor of economic success and failure in contemporary nancialglobal competition1 The second methodological approach to theconcept of model risk focuses on the existence of different types and levelsof error in the practice of economic modelling at the base-theoreticalhypothesis translation into mathematical expression statistical datainputs arithmetical calculations computer softwiring or trading misuses

Model risk in nancial markets appears each time an asset-pricingmodel does not take into account some relevant factor of price variationor else wrongly assumes that the motion of certain stochastic variables canbe imitated by a deterministic process or thinks that price changes can bedescribed by a normal frequency distribution with limit variance range Inother cases ndash even if the model could be thought of as lsquocorrect in prin-ciplersquo or at least not patently erroneous from the point of view of theformal logical arguments mathematical proofs probabilistic test andlsquoencompassingrsquo checks commonly used in academic econometric diagnos-tics ndash markets can disagree with its results in the short term The data usedcan also have been badly estimated or collected or there may have beena mistake during the heuristics searching for its analytical solution Themodel may also have been badly calibrated to mimic real market statisticsThere may even have been coding errors in programming it into the com-puter or the model may have been used in an incorrect way by the naluser (eg a trader may have applied it to price for instruments or marketsfor which it lacked predictive validity) and so on

As has been observed by most nance scholars and professional deriva-tives traders the core mathematical and statistical assumptions built intostandard neoclassical pricing models suffer tremendously when they con-front the structures and processes of real-world nancial trading and rm-wide risk management While equilibrium asset-pricing models forexample characteristically assume that markets are composed of atomizedagents who cannot substantially inuence each other or individuallymanipulate aggregate market prices imitative contagion and herd behav-iour are ubiquitous in real markets and giant reputed investors occasion-ally also lsquomove the marketsrsquo Common nancial models furthermore take

Reliability at risk IZQUIERDO

73

1 A New York-based nancial consultancy Capital Market Risk Advisors (CMRA)recently estimated that 40 per cent of total derivatives-related trading losses were dueto modelling errors up to 27 billion US dollars during 1997 This same consultingrm estimates that in the period 1987 to 1997 some 47 billion of total cumulativederivative losses of 238 billion dollars were due to pricing errors caused by wrong dataor wrong assumptions in asset-pricing models (Stix 1998 27)

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

The level of capital safety requirements of nancial intermediaries (theratio of reserves to nancial assets) is a key factor in the market competi-tiveness of these rms Traditional base assetndashliability gures in thebalance sheets of nancial dealers need to be complemented with newtypes of standardized risk-accounting data that change the level of capitalsafety requirements Due to the new supervisory regulations (see below)the quantitative calculus of expected trading losses internally performedby a nancial rm has a direct and strong effect on the level of reservesrequired to fully insure a rmrsquos creditors and shareholders against bank-ruptcy Thus it has a direct effect on the rmrsquos nancial bottom line andprotability

However the human social activity of mathematical economic andeconometric modelling is still amenable to a third more direct and explicitkind of economic analysis in terms of costndashbenet and riskndashreturn calcu-lations a second-order type of nancial risk known by nancial analystsengineers and traders as model risk (Derman 1996a 1996b) The publi-cized deacutebacirccle in September 1998 of the large and sophisticated hedgefund Long-Term Capital Management is the most telling example of thedevastating effect that can be produced by these strange forms of nan-cial risk a truly reexive form of economic risk that is produced by theactions of risk-adverse nancial agents themselves using mathematicalasset-pricing models in an intensive and extensive manner to buildnancial insurance policies or risk-hedging instruments the famouslsquonancial derivativesrsquo products such as futures options and swaps con-tracts (Steinherr 1998)

3 Dening model risk

Model risk has been dened as a kind of nancial risk that lsquoresults fromthe inappropriate specication of a theoretical model or the use of anappropriate model but in an inadequate framework or for the wrongpurposersquo (Gibson et al 1998 5) The particular risks and uncertaintiesimplied by the practice of formal scientic inquiry (modelling estimatingand testing) into the economics of nancial markets activity are the verylsquofundamentalsrsquo in the economic sense of the market value of formalnancial knowledge understood as a key competitive resource in themodern world of nance Hence the multiple economic sources of model riskare associated with the almost innite manners of constructing a wrongtheoretical model or using a correct model in the wrong way

In a rst economic approximation the concept of model risk accountsfor the fact that the existence and utilization of different types of formalasset-pricing econometric models can give rise to a wide diversity of

EUROPEAN SOCIETIES

72

theoretical prices for a similar type of nancial product As the discrep-ancy between these theoretical bidndashask prices resolves itself in the marketprocess the use of theoretical prices as inputs for the decision-makingprocess of trading and dealing in real nancial markets is revealed as amajor factor of economic success and failure in contemporary nancialglobal competition1 The second methodological approach to theconcept of model risk focuses on the existence of different types and levelsof error in the practice of economic modelling at the base-theoreticalhypothesis translation into mathematical expression statistical datainputs arithmetical calculations computer softwiring or trading misuses

Model risk in nancial markets appears each time an asset-pricingmodel does not take into account some relevant factor of price variationor else wrongly assumes that the motion of certain stochastic variables canbe imitated by a deterministic process or thinks that price changes can bedescribed by a normal frequency distribution with limit variance range Inother cases ndash even if the model could be thought of as lsquocorrect in prin-ciplersquo or at least not patently erroneous from the point of view of theformal logical arguments mathematical proofs probabilistic test andlsquoencompassingrsquo checks commonly used in academic econometric diagnos-tics ndash markets can disagree with its results in the short term The data usedcan also have been badly estimated or collected or there may have beena mistake during the heuristics searching for its analytical solution Themodel may also have been badly calibrated to mimic real market statisticsThere may even have been coding errors in programming it into the com-puter or the model may have been used in an incorrect way by the naluser (eg a trader may have applied it to price for instruments or marketsfor which it lacked predictive validity) and so on

As has been observed by most nance scholars and professional deriva-tives traders the core mathematical and statistical assumptions built intostandard neoclassical pricing models suffer tremendously when they con-front the structures and processes of real-world nancial trading and rm-wide risk management While equilibrium asset-pricing models forexample characteristically assume that markets are composed of atomizedagents who cannot substantially inuence each other or individuallymanipulate aggregate market prices imitative contagion and herd behav-iour are ubiquitous in real markets and giant reputed investors occasion-ally also lsquomove the marketsrsquo Common nancial models furthermore take

Reliability at risk IZQUIERDO

73

1 A New York-based nancial consultancy Capital Market Risk Advisors (CMRA)recently estimated that 40 per cent of total derivatives-related trading losses were dueto modelling errors up to 27 billion US dollars during 1997 This same consultingrm estimates that in the period 1987 to 1997 some 47 billion of total cumulativederivative losses of 238 billion dollars were due to pricing errors caused by wrong dataor wrong assumptions in asset-pricing models (Stix 1998 27)

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

theoretical prices for a similar type of nancial product As the discrep-ancy between these theoretical bidndashask prices resolves itself in the marketprocess the use of theoretical prices as inputs for the decision-makingprocess of trading and dealing in real nancial markets is revealed as amajor factor of economic success and failure in contemporary nancialglobal competition1 The second methodological approach to theconcept of model risk focuses on the existence of different types and levelsof error in the practice of economic modelling at the base-theoreticalhypothesis translation into mathematical expression statistical datainputs arithmetical calculations computer softwiring or trading misuses

Model risk in nancial markets appears each time an asset-pricingmodel does not take into account some relevant factor of price variationor else wrongly assumes that the motion of certain stochastic variables canbe imitated by a deterministic process or thinks that price changes can bedescribed by a normal frequency distribution with limit variance range Inother cases ndash even if the model could be thought of as lsquocorrect in prin-ciplersquo or at least not patently erroneous from the point of view of theformal logical arguments mathematical proofs probabilistic test andlsquoencompassingrsquo checks commonly used in academic econometric diagnos-tics ndash markets can disagree with its results in the short term The data usedcan also have been badly estimated or collected or there may have beena mistake during the heuristics searching for its analytical solution Themodel may also have been badly calibrated to mimic real market statisticsThere may even have been coding errors in programming it into the com-puter or the model may have been used in an incorrect way by the naluser (eg a trader may have applied it to price for instruments or marketsfor which it lacked predictive validity) and so on

As has been observed by most nance scholars and professional deriva-tives traders the core mathematical and statistical assumptions built intostandard neoclassical pricing models suffer tremendously when they con-front the structures and processes of real-world nancial trading and rm-wide risk management While equilibrium asset-pricing models forexample characteristically assume that markets are composed of atomizedagents who cannot substantially inuence each other or individuallymanipulate aggregate market prices imitative contagion and herd behav-iour are ubiquitous in real markets and giant reputed investors occasion-ally also lsquomove the marketsrsquo Common nancial models furthermore take

Reliability at risk IZQUIERDO

73

1 A New York-based nancial consultancy Capital Market Risk Advisors (CMRA)recently estimated that 40 per cent of total derivatives-related trading losses were dueto modelling errors up to 27 billion US dollars during 1997 This same consultingrm estimates that in the period 1987 to 1997 some 47 billion of total cumulativederivative losses of 238 billion dollars were due to pricing errors caused by wrong dataor wrong assumptions in asset-pricing models (Stix 1998 27)

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

for granted that economic information is a public good while in practicethere are different rhythms of accessing and analysing it They also assumethat transaction costs are minimal and that markets are highly liquid butliquidity lsquosqueezesrsquo and large jumps in prices and volatilities are all typicalof real legally and organizationally constructed markets The sameapplies to the standard assumption of levelled debt capacity and regulatoryneutrality There is a wide variation in the nancial and legal costs ofrunning a banking business depending on the different institutional andsocial statuses of the agents

4 Supervising model risk technical controversies and publicchoices

Byzantine academic debates over how to dene measure and reducemodel risk are central to the supervisory controversy over the calculationof so-called lsquomarket riskrsquo banking capital requirements Having provedpowerless to accommodate its standard bureaucratic norms for externalbanking examination to the ever faster rhythm of technical innovation innancial derivatives markets the main international banking supervisoryagency the Basle Committee for Banking Supervision (BCBS) of the Bankof International Settlement (BIS) has recently given a Copernican-turnto the tradition of central banking supervision a tradition whose mostconspicuous example is the 1988 Basle Capital Accord (BCA) (Swary andTopf 1993 133ndash4) Confronted with the constant failures of mandatoryand universal supervisory standards the BCBS now tries to enlist into itsteam the adaptive powers of the decentralized mechanism of innovation-based market competition that allows most nancial rms to continuallyimprove internal risk management systems by heavily investing in humancapital and RampD (Dunbar 1998)

Setting global market risk supervisory standards

The BCBS intended to integrate the fast-evolving organizational know-how of the derivatives industry into its extended supervisory repertoire ndashthe 1996 Amendment to the BCA (ABCA) ndash by targeting the bankrsquos owninternal control systems and not as was previously done its real invest-ment portfolio At the end of the 1980s the trading book and off-balanceoperations (mostly derivatives contracts) had gained so much space in thebalance sheets of the savings and loans and commercial banks that thenational and international regulatory authorities began to fear thattogether with traditional credit risk retail banks would now be strongly

EUROPEAN SOCIETIES

74

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

affected by that class of devastating risk specic to the investment bankingand securities dealer business namely market risk Authorities perceivedan increasing probability that an adverse sudden and coordinate pricemovement across diverse markets terms and instruments worldwide couldproduce such a huge quantity of trading losses that the precautionarycapital reserves which serve as guarantees for depositors would beseverely affected and trigger a spiral of nancial panics and bankruptciesWith the US savings and loans disaster reaching its peak at the beginningof the 1990s the initial rhetorical concern of public authorities overmarket risk translated into a concrete programme for adapting regulatorycapital requirements to the new reality

In 1988 the BCBS succeeded in having its members sign the rst inter-national protocol for harmonizing national banking capital standards theBasle Capital Accord (BCA) The BCA prescribed the acceptance of a setof common procedural rules a system of direct external supervisionknown as the lsquostandard approachrsquo (Basle Committee 19881998) Bymechanically applying the same broad criteria for credit analysis thedifferent national authorities could determine in a crude but normalizedway what should be the correct and safe level of capital reserves for a bankin possession of a diversied credit portfolio to insure its depositors andshareholders against a huge wave of credit defaults regardless of thenational legislation This common measure of banking safety was knownas the lsquoCooke Ratiorsquo2

However only two years later the supervisory norms of the BCA hadbecame outdated by the new investment practices of its regulatory sub-jects that is by massive exchange-traded and OTC3 derivatives tradingThe BCA strictly focused on the regulation of credit risk capital require-ments the amount of capital that must be set aside to insure banksrsquo bottomlines against risks of credit default and said almost nothing about theincipient problem of market risk precautionary capital

Thus shortly after the BCA began to be applied by national authori-ties the BCBS was already seriously entertaining the possibility of amend-ing it and including new precautionary standards against market risk A

Reliability at risk IZQUIERDO

75

2 The BCA required banks to raise their reserve to reach at least 8 per cent of total assetsweighted by risk class It distinguished two components or lsquotiersrsquo of banking capitalTier 1 or lsquocorersquo capital (stock issues and disclosed reserves) and Tier 2 or lsquosupple-mentaryrsquo capital (perpetual securities undisclosed reserves subordinated debt withmaturity greater than ve years and shares redeemable at the option of the issuer)Finally the Accord established a set of risk capital weights to ponder capital require-ments against different types of nancial instruments (Swary and Topf 1993 450ndash6)

3 OTC is for lsquoover-the-counterrsquo or tailor-made derivatives contracts such as foreignexchange options or so-called lsquoswaptionsrsquo (options on interest rate swaps) Contrary topublicly exchanged nancial securities OTC derivatives are privately negotiated mainlybetween an investment bank and its client corporation

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

new regulatory proposal was devised to encourage the international adop-tion of a new simple transparent and amply agreed procedure to deter-mine with sufcient precision the extra quantity of capital reserves neededby the banks with huge portfolios of derivatives and other high-risksecurities

At the end of 1996 the BCBS issued an advisory report that recom-mended banks to use their own internal risk measurement models and theirown computerized systems of rm-wide risk management to determinefor themselves the proper quantity of market risk capital reserves (BasleCommittee 1996a 38ndash50) There was a double argument in support ofthis proposal (1) to prot socially from the private information and entre-preneurial know-how accumulated during years of daily risk managementand (2) to publicly prot from the rmsrsquo own selsh interests in improv-ing the quality of its risk management system to gain competitive advan-tage With the coming into force in January 1997 of the Amendment tothe BCA (ABCA) that allowed banks to use their own internal riskmanagement models to autonomously determine the proper amount ofmarket risk capital reserves public supervisory authorities have come toperform rather indirect and abstract new inspectorate tasks centredaround a set of very technical procedures for risk management systemsquality auditing

In this new regulatory regime effective banking safety levels can onlybe guessed indirectly by supervisory authorities by means of checking thetechnical reliability and organizational exibility of banksrsquo internal riskmanagement systems

The design of banksrsquo internal control systems value-at-riskeconometric modelling

Opposed to the former lsquostandard approachrsquo to banking supervision thenew supervisory regime for market risk capital reserves is known as thelsquointernal models approachrsquo (Jorion 1997a 50) Many of the internal riskcontrol systems developed by the banks who are active in the globalderivatives markets are based on the application of a class of generalizedequilibrium asset-pricing econometric models known as Value-at-Risk(VaR) models The basic principle of VaR management the daily calcu-lation of a broad aggregate gure of maximum potential losses had beendeveloped within the community of the biggest Wall Street investmentbanks almost since the aftermath of the October 1987 stock-market crash

VaR models tackle the following computational problem how todetermine the maximum nancial loss expected with a signicant proba-bility for a given condence level that could be suffered by a properly

EUROPEAN SOCIETIES

76

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

diversied asset portfolio during a given period of time as a consequenceof an adverse and pronounced movement in nancial prices coordinatedacross different markets instruments maturities or countries (see Jorion1997a 86ndash93) Technically a VaR gure is a probabilistic measure offuture economic value or to be more precise a mathematical expectationof nancial losses dened as the mean probability associated with a givenevent times the economic value assigned to this event The informationprovided by VaR numbers is an estimation of the maximum pecuniarylosses (eg ve million euros) attached to a numerical probability of occur-rence (1 per cent) a statistical condence level (99 per cent) ndash and there-fore to some theoretical frequency distribution (eg gaussian) ndash and aperiod of time (one day) That is of each 100 trading days one shouldexpect that only during one of these onersquos investment portfolio could reacha maximum cumulated daily loss of ve million euros and that with amargin of error of plusmn1 The amplitude of this error interval thus accountsfor the possibility of a maximum-loss event occurring twice during thechosen time period

The most common procedure used to calculate VaR gures is called thelsquohistorical methodrsquo This is a two-step econometric procedure originallycodied by JP Morgan into its proprietary risk management software Risk-metricsTM (Guldimann 2000) It works in the following manner It is rstof all necessary to arrange a complete and extended numerical databasethat is a multidimensional matrix of previous fundamental parameterchanges in the most frequently traded nancial instruments This shouldconstitute a reliable sample of the long-term behaviour of markets and willallow the user to estimate a set of robust statistical trends in the relationsbetween (1) the market prices of a broad range of investment contracts(end-of-the-day quotes of shares index bonds futures etc) (2) its volatil-ities that is the mean deviations of every single market price from its meanhistorical level and (3) its correlations or the statistically signicant co-efcients of mutual inuence between the long-term motion of eachsecurity and the historical motion of each and every other security relatedto it These three types of sample statistics (mean values volatilities andcorrelations) are the variables which are subject to econometric treatmentwithin VaR models typically constructed in the form of equilibrium asset-pricing models obeying the well-known meanndashvariance principle of neo-classical nance theory (optimal risk spread dened as the minimumaggregate variance of mean expected returns for any given level of sub-jective risk-aversion)

A much used alternative approach to VaR calculations ndash and favouredby Bankers Trust with its computer application RaRoc2020TM (Falloon1995) ndash is taken not from classical portfolio theory but from the theory ofarbitrage-free option pricing (Jorion 1997a 77) In this case the key

Reliability at risk IZQUIERDO

77

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

variables of the model are not correlations or historical volatilities butfundamental risk parameters that can be derived from the MertonndashBlackndashScholes option-pricing model delta gamma vega theta rho etc In thisapproach each nancial contract is decomposed or lsquogranulizedrsquo into aseries of basic risk factors lsquodelta-riskrsquo lsquogamma-riskrsquo etc (Merton 1995a)Huge masses of these little risk lsquograinsrsquo or lsquoparticlesrsquo are then aggregatedusing statistical correlation techniques until a single gure results thatmeasures the risk-adjusted return on all the capital invested in the marketTwo other statistical simulation techniques are widely used to complementthe analysis in terms of historical volatilities and risk factors Monte Carlosimulations (based on articially calibrated computational samples andstochastic processes) and lsquostress testingrsquo a qualitative assessment of therobustness of different portfolio structures under extreme-value con-ditions (see Dunbar 1999)

Reliability trials backtesting

The 1996 ABCA established a series of minimum general lsquotechnicalrsquorequirements that banksrsquo internal risk management systems need to fullThe initial validation and periodic revision of bank internal models underits jurisdiction was a task assigned to national banking supervisoryauthorities The amendment of 1996 was also accompanied by a comple-mentary advisory report that established a set of criteria for nationalsupervisory authorities to conduct quality audits of banksrsquo VaR internalmodels (Basle Committee 1996b) The aim of this complementary reporton lsquobacktestingrsquo procedures was to add an incentive mechanism for com-pliance with regulatory norms to assure the public that if banks wanted togain supervisory approval for using their internal risk managementsystems as lsquoregulatory alliesrsquo they would have to adopt the necessary (andcostly) measures to improve their accuracy

The report in question detailed how to conduct a series of standardstatistical counter-trials or lsquobacktestsrsquo to formally asses the performance ofbank internal modelsrsquo risk measures in relation to the actual risk levels inthe market To guarantee that banks would indeed devote the requiredefforts and resources to maintain update and improve their internal modelsthe report stipulated that the different national supervisory authoritieswould conduct quarterly examinations of their forecasting performanceThese exams would monitor the quality of the internal statistical infor-mation used by bank CEOs in the decision-making process to set a safe levelof market risk capital reserves Hence the ultimate aim of the modelexamination is to guarantee that the VaR gures of aggregate nancial riskwould comply with some minimum econometric reliability requirements

EUROPEAN SOCIETIES

78

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

As dened in this 1996 BCBS supplementary document backtestingtrials consist in the comparison of VaR theoretical measures calculated bya particular nancial econometric model for a time horizon of one daywith actual nancial prot and loss daily gures that is the effectivelsquotrading outcomesrsquo realized at the end of each business session (BasleCommittee 1996b 2) As we have seen theoretical VaR measures areintended to encompass within them (almost) all trading outcomesexpected at the end of the day leaving outside of its coverage only a tinyfraction of these (ie the most improbable ones) whose size is given bythe condence level chosen to calibrate the model In this respect theBCBS report established that the percentage of trading outcomes that thetheoretical VaR measures produced by the banks must cover should belsquoconsistentrsquo with a condence level of 99 per cent

Therefore to assess the degree of statistical effectiveness of a bankrsquosVaR econometric models the public examiner must (1) count the numberof lsquoexceptionsrsquo produced by the model that is how many times the actualtrading outcomes at the end of the day fall outside the theoretical expecta-tion produced by the model and (2) determine if the number of excep-tions is consistent with the obligatory coverage level of 99 per cent Forexample for a recommended sample of 250 trading days a daily VaRmeasure calibrated for a 99 per cent condence level should cover onaverage 248 of the 250 observed trading outcomes leaving only twoexceptions unforecasted by the safety calculus4 If the model produces say125 exceptions it must be lsquoclearrsquo to the external public auditors that some-thing is wrong The bank must then compensate for the forecasting weak-ness of its model with a proportional rise in the multiplying factor appliedto its capital reserves that happens to attain the desired condence levelof 99 per cent

However the main problem with which VaR econometric models exter-nal examiners have to deal is how to interpret an ambiguous backtestingresult That is still using the former example one that produces a numberof exceptions only slightly higher than two ndash say four or seven ndash a gurethat from a strictly probabilistic point of view is not a conclusive signalabout the actual predictive strength or weakness of the model To solvethis fundamental supervisory uncertainty the BCBS document establisheda second set of quantitative criteria to clearly demarcate three differentinterpretative zones a lsquosafetyrsquo zone (green) a lsquocautionrsquo zone (yellow) and alsquodangerrsquo zone (red) The green zone extends to all backtesting results ndashbetween zero and four exceptions in a normalized sample of 250 ndash that

Reliability at risk IZQUIERDO

79

4 To make a trade-off between the regularity of the supervisory exams and the repre-sentativeness (in the statistical sense) of the data used by the models the BCBS rec-ommended carrying this backtesting exam on a quarterly basis the evaluation focusingon trading data from the last twelve months ie a sample of 250 observations

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

lsquofrom a mathematical probabilistic point of viewrsquo suggest no doubts aboutthe predictive soundness of the model In this case no supervisory actionis undertaken in the sense of rising capital requirements Within the yellowzone fall those results that produce non-conclusive doubts about the fore-casting ability of the model ndash between ve and nine exceptions ndash and whosereading by the supervisor could be accompanied by a rise of between 040and 085 points in the multiplying factor applied to the existing base capitalreserves Finally those outcomes which are equal to or exceed 10 excep-tions are located in the red zones and all must be countered by a one-pointrise in the multiplying factor

Again this system of zones has its own problems as the supervisoryreport recognized If the examiner is too stern about the numerical thresh-olds that demarcate the different zones she can commit two types of sta-tistical errors in her lecture of backtesting results either she can classifyas defective a model that is actually valid or she can admit as correct amodel that is actually faulty These types of problems are largely posed bythose backtesting results which are included within the yellow zonebecause standard statistical calculations show that the probabilities for amodel to produce outcomes between ve and nine exceptions are similarfor acceptable (99 per cent coverage) and rejectable (98 or 97 per cent)models

To aid the examiner to overcome this problem the BCBS reportincluded two tables with numerical calculations of existing theoreticalprobabilities to obtain a given number of exceptions for a sample of 250observations for different coverage levels of the model (99 per cent 98 percent 97 per cent 96 per cent and 95 per cent) These calculations showthat there exists a high probability of erroneously rejecting a valid modelwhen for a condence level of 99 per cent the examiner chooses a par-ticularly low number of exceptions as the threshold for rejection (if thethreshold is set to one exception valid models would be rejected by exam-iners in 919 per cent of cases) Of course if the threshold of the maximumnumber of exceptions that can be produced by a model to be validated israised the probability of incurring this type of error is lowered Howeverthe probability of making the inverse error is raised for a rejection thresh-old of seven or more exceptions the calculations of the Committee indi-cate that a model with a coverage of only 97 per cent (a non-valid model)will be erroneously accepted in 375 per cent of cases

5 Types of randomness error and responsibility

A further answer to the problems posed by of the ambiguity of backtest-ing results is provided by another Basle Committee recommendation

EUROPEAN SOCIETIES

80

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

The Committee eventually advises the supervisor to require the bank tosupply a set of complementary information of a qualitative nature bothabout the precise econometric and computational architecture of themodel under supervision and about the lsquospecialrsquo character of non-coveredtrading outcomes5 This means that when there is not enough quantita-tive evidence about the technical reliability of the risk model banks arestill allowed to try to document explain away and possibly justify on acase-by-case basis the causes of every exception detected through thebacktesting

The bankrsquos model risk counter-experts do in fact routinely elaboratecomplex interpretative documents to try to explain away even the mostagrant backtesting exceptions If for example a bank were to fail to raiseits bottom-line capital level to insure creditors against adverse asset pricemovements produced by an abrupt social rupture in a foreign country thebank VaR modellers would present supervisory authorities with news-paper clips and dossiers that qualify such an exceptional lsquoexceptionrsquo as oneof those completely unpredictable and hence uninsurable random econ-omic events that supervisors conventionally allocate to the correct prob-abilistic margin of 1 per cent normal measurement error6 However if thesame failure were to apply to the occurrence of an adverse price changeof the kind that is considered by neoclassical nancial economists to bestrictly governed by so-called lsquoendogenous market forcesrsquo such as recur-rent stationary cycles in aggregate consumer demand or stable stochastictrends in macroeconomic growth rates the fact of an eventual bankruptcycould hardly be publicly justied as the consequence of unnoticed and

Reliability at risk IZQUIERDO

81

5 lsquoThe burden of proof in these situations should not be on the supervisor to prove thata problem exists but rather should be on the bank to prove that their model is funda-mentally sound In such a situation there are many different types of additional infor-mation that might be relevant to an assessment of the bankrsquos modelrsquo (Basle Committee1996b 8)

6 The tale of the lsquoperfect nancial stormrsquo is grosso modo the scheme of the justicatoryarguments put forward by defendants in the governmental inquiry that was set up afterthe private bail-out of the large hedge fund Long-Term Capital Management goinglsquotechnically bankruptrsquo in September 1998 In this particular account the star role ofthe lsquoextreme eventrsquo is played by the default of Russian sovereigns (Dunbar 2000 xiii)Curiously enough the fact of not being directly subject to Basle Committee internalmodelsrsquo regulations was one of the reasons for the fundrsquos extraordinary success aslsquoglobal central banker for volatilityrsquo during the aftermath of the autumn 1997 Asiancrisis (ibid 178) but also played an important role in its eventual debacle exactly oneyear later In his careful reconstruction of the LTCM catastrophe nancial journalistNicholas Dunbar claims that despite the shock of the Russian bonds default the realproblems of the fund were in a larger part caused by the growing management promi-nence conceded to lsquoRisk Aggregatorrsquo the awed in-house VaR management softwareof LTCM lsquoThe Risk Aggregator has been the subject of much debate As is now clearit either didnrsquot work properly or was misused by the LTCM partners ndash none of whomwill now accept responsibilityrsquo (ibid 186)

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

unintended lsquomodelling errorsrsquo in the face of lsquoradical market uncertaintyrsquoThe surest bet here for the supervisory examiners should be the presenceof strategic lsquofakersquo movements intended to make cheap low-quality nan-cial risk management policy appear to comply with high-quality high-costrisk management supervisory standards What I would like to suggest hereis that serious doubts and criticisms from academics and practitioners alikehave recently crept into this regime of conventional peaceful techno-economic coordination between private bank modellers and supervisoryexaminers To get rid of the frightening ghost of sudden nancial deacutebacirccleno longer sufces to magically conjure as do conventional nancialmodellers the perfect isolation of stable economic functions from non-stationary sociohistorical processes

Adopting the language of lsquostandard econometricsrsquo as common currencyin the political debate over global nancial stability is no longer as uncon-scious an administrative behaviour as it used to be To be sure the mid-1990s academic controversy over the management and regulatory uses ofVaR econometric models has produced a large repertoire of methodo-logical theoretical and epistemological justications for adversarial typesof econometric practice7 Among the most remarkable arguments putforward in this detective-forger social reexive game is the banksrsquo riskmodellers accusation of arbitrariness formulated against public supervisorsfor setting the standard condence levels according to which backtestingresults are to be judged in complete disagreement with the empirical sta-tistical structure of real market uctuations When you choose a con-dence level of 99 per cent it means that only one out of each 100 tradingdays your losses can exceed the VaR value computed by the model Butthe true meaning of the condence level is really an artefact of the adop-tion of a more fundamental (and disputed) theoretical assumption namelythat of a characteristic probability distribution In neoclassical nancialeconometrics statistical condence is but the offspring of gaussian math-ematical laws (the well-known lsquoergodicrsquo and lsquocentral-limitrsquo theorems) andwhen these mathematical theorems are rejected as a proper algorithmic

EUROPEAN SOCIETIES

82

7 A fast foray into this controversy is provided by the published exchange between twonancial experts Philippe Jorion nance professor at the University of CaliforniaIrvine and one of the principal academic advocates of VaR models and Nassim Taleba respected senior option trader and derivatives engineer who is critical of VaR (seeJorion 1997b Taleb 1997a 1997b Stix 1998) For Jorion on the one hand the purposeof VaR models is not as is usually stated lsquoto describe the worst possible outcomesrsquo butmore modestly lsquoto provide an estimate of the range of possible gains and losses Manyderivatives disasters have occurred because senior management did not inquire aboutthe rst-order magnitude of the bets being takenrsquo (Jorion 1997b 1) Taleb on the otherhand discredits VaR econometrics as mere lsquocharlatanismrsquo arguing that lsquoit tries to esti-mate something that is not scientically possible to estimate namely the risks of rareevents It gives people misleading precision that could lead to the buildup of positionsby hedgers It lulls people to sleeprsquo (Taleb 1997a 1)

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

representation of the empirical frequency distribution of price changes sois statistical condence as a means for technological reliability

Following the path initially tracked by the same nancial rms theyaudit supervisors have a decidedly lsquomildrsquo conception of nancial ran-domness But as has been pointed out many times by the most incisivecritics of nancial neoclassical econometrics there exists a agrant gapbetween the tractable mathematical models of mild randomness generallyassumed by applied portfolio theory and the type of lsquowildrsquo randomness inwhich as is characteristic of true historical processes extraordinary eventsare always in some sense lsquotoo probablersquo (Mandelbrot 1997b 57ndash74) Stillpublic regulators and private nancial competitors alike have traditionallypreferred to assume that lsquorandomnessrsquo is the source of mostly insignicantand easily reversible economic events and that truly irreversible economicevents such as large-scale or long-term price variations have nothing todo with randomness but are the product of deterministic necessary andthus predictable causes

This classical reassuring principle for the administrative vision and div-ision of the world ndash the well-known gaussian axiom that randomness canonly be understood as a microscopic phenomena ndash is today in trouble inthe world of derivatives trading As much by the sheer brutality of recentmarket events as by the strategic necessity to adapt to changes in publicsupervisory norms nancial practitioners have been called upon to reectupon the obscure and disputable modelling conventions that sustain themyth of technological reliability in the world of applied nancial econo-metrics In fact even the very senior executives who run the risk manage-ment divisions of the biggest world investment banks are beginning todoubt the key feature of neoclassical nancial theory and engineeringpractice that you can separate deterministic from random forces8

The irony here is that the strong point put forward by rational (scien-tic) criticism of nancial management and regulatory practice is in this

Reliability at risk IZQUIERDO

83

8 Witness the crystal-clear account by prominent market professional Robert Gum-merlock former managing director of Swiss Bank Corporation one of the worldrsquosbiggest investment banks lsquoThe magnitude of a 5ndash10 standard-deviation move is notdebatable ndash that is given What is debatable is how often it happens and thatrsquos wherepeople get confused In the textbook world of normal distributions a 10 standard-deviation move is more than a one in a million event In nancial markets we know itis not so we have to decide how often it can happen The troublesome thing about fattail distributions is that they sever the link between ordinary and extraordinary eventsUnder a purely normal distribution the extraordinary events are strictly governed byprobabilities policed by the standard deviation With fat tailed distributions outlierscan occur with maddening frequency and no amount of analysis of the standard devi-ations can yield useful information about themrsquo (cited in Chew 1994 64) It is indeedremarkable that practitionersrsquo indictments against orthodox statistical nancial riskmeasurement do read almost exactly the same as some of the most recent publicstatements by the very nemesis of academic neoclassical nancial econometrics lsquoThe

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

case and as it should be totally unacceptable for supervisors The reason forthis is that to accept the statistical spectre of lsquowildrsquo randomness as a moreaccurate scientic description of the typical spectral shape of real-wordnancial risk would mean to reject any role whatsoever for public super-vision in the nancial services industry9

Minimum supervisory requirements for banking capital reserves onlymake sense in a world were nancial risk is statistically deterministic it canbe modelled as a predictable phenomenon in the probabilistic sense andtherefore as something that falls under the domain of human control evenif this control is exercised under the subtle mathematical routines of sto-chastic dynamic programming (Sent 1998) For banking capital risk super-vision to have a positive social welfare effect nancial catastrophe mustbe understood as something that can be prevented For only under thishypothesis can some level of regulatory capital reserves be called safe ora sudden bankruptcy attributed to a failure to comply with supervisoryrequirements Using this lsquoclassicalrsquo framework of analysis nancialmanagement can be judged to have lsquofailedrsquo and legal responsibility forlsquomismanagementrsquo can be sought on an individual basis

However if the speculative motion of nancial prices is a non-deterministic process of a second-order class as critics of neoclassicalnancial econometrics argue then nancial catastrophe cannot be pri-vately or socially prevented In this later scenario no regulatory level ofrisk capital reserves (including full investments coverage) can be reallydeemed lsquoprotectiversquo and no nancial damage to the bankrsquos creditors orshareholders (even instantaneous bankruptcy) can be understood as theproduct of lsquomismanagementrsquo Human responsibility is rather translatedinto the language of unforeseen unintended random lsquoerrorrsquo In this

EUROPEAN SOCIETIES

84

mathematics underlying portfolio theory handles extreme situations with benignneglect it regards large market shifts as too unlikely to matter or as impossible to takeinto account According to portfolio theory the probability of these large uctua-tions would be a few millionths of a millionth of a millionth of a millionth (The uc-tuations are greater than 10 standard deviations) But in fact one observes spikes on aregular basis ndash as often as every month ndash and their probability amounts to a few hun-dredthsrsquo (Mandelbrot 1999 70)

9 But also paradoxically to deny any productive role for the nancial engineerrsquos com-putational stylization of the economic process As has been acknowledged by Peter LBernstein in his bestseller history of the triumphal march of mathematical nancialeconomics in the academy and the marketplace lsquoMandelbrot remains on the periph-ery of nancial theory both because of the inconvenience to analysts of accepting his argu-ments and because of the natural human desire to hope that uctuations will remain withinfamiliar boundsrsquo (Bernstein 1992 132 my italics added) The said Benoicirct Mandelbrothas recently restated his old arguments as to the weak scientic status of nancialeconometrics taking nancial engineering as a new target for his clever invectiveslsquoAvant de srsquoengager dans lrsquoingeacutenieacuterie nanciere et ses ldquoproduits deriveacutesrdquo il srsquoimposedrsquoabord de ldquosrsquoassurer bien du faitrdquo on ne laisse pas agrave lrsquoingeacutenieur le loisir de prendreagrave sa charge les regrets du savantrsquo (Mandelbrot 1997b 9)

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

alternative theoretical framework it should come as no surprise that bankmanagersrsquo overall judgement on public supervisory procedural norms forconducting model risk audits is that they are doing more harm than goodto our collective economic welfare

Tearing down the conventional administrative boundaries that separateordinary from extraordinary economic events as suggested by theMandelbrotian hypothesis of lsquowildrsquo randomness would imply that thosemanagement decisions backed up by VaR results concerning precau-tionary capital allocation that were considered the most agrantly lsquounjus-tiablersquo under the gaussian statistically deterministic supervisoryframework could be excused as the product of lsquosheer bad luckrsquo Thus con-fronted as it is by the hyperbolic stochastic dynamics of contemporarynancial prices the everyday administrative banking maintenance of thetwin social constructs that dene the institutional core of a capitalistmarket economy ndash accounting value and commodity money ndash desperatelydemands lsquothat something be treated as effectively invariant even as weknow all along it is notrsquo (Mirowski 1991 579)

6 Conclusion forgers and critics

An intellectual adventure barely forty years old the mathematical eco-nomic theory of equilibrium asset pricing in perfectly competitive capitalmarkets has by now consolidated into one of the most dynamic andrespected subelds of economics Together with the exploding job marketand lsquoindecentrsquo salaries paid to hundreds of young MBAs in mathematicalnance during the past two decades the 1991 and 1997 Nobel prizes ineconomics awarded to the pioneering models of lsquoefcientrsquo portfolio selec-tion risk pricing and capital arbitrage economic routines by HarryMarkowitz Williams Sharpe and Merton Miller and the lsquooptimalrsquodynamic risk management and synthetic (derivative) asset replicationschemes by Robert C Merton and Myron Scholes stand as irrefutableproof of the scientic economic and political success of this most esotericbody of social knowledge On the other hand highly publicized recentderivatives-driven nancial catastrophes such as the October 1987 NYSEmarket crash (Jacobs 1999) Metallgessechschaft and Orange Country in1994 (Jorion 1995) Barings in 1995 (Millman 1995) and Long-TermCapital Management in 1998 (Dunbar 2000) have raised serious concernsabout the scientic shortcomings and technological dangers of appliedmathematical nancial economics so-called nancial engineering

In our contemporary high-tech world the ancient dialectics of publicdomain expertise versus private and secret information has reached a peakin the arena of global nancial marketsrsquo competition The complex

Reliability at risk IZQUIERDO

85

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

reexive social patterns characteristic of the world of derivative nancialproducts services and markets engineering ndash with its characteristicsequence of innovation competition and regulation cycles (Abolaa 1996)ndash offer one of the most intricate present variants of this classical dialecticsof authorized knowledge corrupted into strategic forgery and then re-cycled as learned criticism10 This new technological art of the articiallsquoreplicationrsquo or lsquocomputational synthesisrsquo of historically stable economicfunctions (Crane et al 1995) is being increasingly self-consciously under-stood by its practitioners as a sort of counterfeiting game11 In such a gameof economic competition social reexivity is pervasive and sooner or laterwinning strategies are defeated as a consequence of their own success Nomatter how impressive its trading record in the short term the lsquorisk fakersquomanufactured by the nancial engineer would eventually have to beauthenticated by the most harsh critic of economic technology namelyeconomic history (Mandelbrot 1997a 17ndash22)

In a series of working papers and ofcial advisory reports publishedduring the second half of the 1990s the Basle Committee established defacto supervisory procedural rules for the correct way to conduct standardforensic trials to test the technological reliability of banksrsquo risk controlmodels Avant-garde academics and professionals have lately attackedstandard backtesting model risk supervisory methods for being unable toacknowledge the probabilistic subtleties of real-world nancial risk Thisparticular denouncement reproduces and updates the well-known market-libertarian criticism of the bureaucratic lsquorigidityrsquo characteristic of indus-trial organizations (there comprised the ever-outdated character ofnormalized quality control testing procedures as a core chapter) so dearto Austrian and Chicago School radical liberal versions of neoclassicaleconomic analysis

EUROPEAN SOCIETIES

86

10 The lsquospiralrsquo of regulation and innovation-driven market competition characteristic ofthe international industry for advanced nancial services (Merton 1995a) is perhapsonly paralleled by the hyper-complex reexive dynamics described by the emergenceof digital network security standards under re from hacker attacks In the world ofcomputer security research and development the well-known ambivalent characterof the computer lsquohackerrsquo seriously compromises the moral separation border betweenconstructive forensic criticism and destructive informed forgery (Hollinger 1991) Ifthe technical complexity of the standard repertory of forensic authentication trialsdeveloped by government security agencies for the surveillance of telephone andcomputer communications has improved dramatically during the past ten years it hasbeen largely as a learning by-product of ad hoc public prosecution actions conductedin the face of ever more sophisticated new forms of network-computer fraud andforgery (Shimomura and Markoff 1997)

11 For a sociological analysis of the strategic games of counterfeiting being traditionallyplayed between the detective and the criminal (Kaye 1995) the scholar critic and thescholar rogue (Grafton 1990) and more generally between the public expert and thereexive forger see the excellent book by Bessy and Chateauraynaud (1995)

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

In ratifying their belief that if uncertain as to its ultimate aggregateeconomic outcomes a minimum of publicly induced procedural stan-dardization in the market for proprietary nancial risk control systems isalways better than lsquopurersquo market competition freed from any type (director indirect) of macroeconomic controls the reply of regulators is on theother hand patterned under the no less ancient grammar of engineer-typedenouncements of market lsquowhimsrsquo understood as coordination lsquofailuresrsquo(Boltanski and Theacutevenot 1991 334) In fact far from accomplishing theirintended objective that is reducing aggregate levels of risk in contem-porary nancial markets the most probable effect of this new repertoireof lsquonormalizedrsquo lsquoclearrsquo lsquosimplersquo and lsquofastrsquo meta-statistical tests of modelrisk as global nancial supervisory tools would be the improvement of theindustrial quality of applied nancial econometric models

In the lsquoexponentially innovativersquo environment of contemporary capitalmarkets (Merton 1995b) it is indeed increasingly difcult to identify anddistinguish from an a priori theoretical point of view the kind of behav-iour we would otherwise label as lsquosmartrsquo lsquorecklessrsquo or overtly lsquocriminalrsquoThat is successful research on the dynamics of innovative behaviour likethat on the management of high-risk technologies depends on the explo-ration of a more fundamental theoretical topic how to attribute meritsand blames ndash or even legal responsibility and lsquoauthorshiprsquo on an indi-vidual basis ndash in a social environment where lsquochancersquo is always a little tooprobable

It has been argued (Meier and Short 1983) that since modern indus-trial life is inherently risky and for this very reason the connectionbetween purposive action and observable social consequences is radicallyambiguous it would always be controversial to point the nger at someparticular type of social risk (nancial risk) as the outcome of criminalconduct (nancial fraud and forgery) And all the more so when as is thecase with the nancial services global industry innovativeness is centralto responsible behaviour My ultimate claim in this article is thereforethat the inverse should also hold that because modern empirical socialscience is also a high-risk enterprise it should be deemed no less con-troversial to dissolve any suspicion of criminal conduct (nancial fraudand forgery) into the ambiguous dustbin of unintended scientic error(model risk)12

Reliability at risk IZQUIERDO

87

12 In the words of a prominent expert in the eld efcient policy rules against scienticmisconduct must lsquobe able to distinguish error from fraud unintentional and evencareless mistakes from intentional misconduct and misstatements from deceptivemisrepresentationrsquo (Bernardine Healy Director National Institute of Health1991ndash93 Statement 1 August 1991 at the Hearings on Scientic Fraud conducted byCongressman John D Dingellrsquos House Subcommittee on Oversight and Investi-gations of the Committee on Energy and Commerce cited in Kevles 1998 306)

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

Acknowledgements

This article draws on research materials previously presented at differentfora I gratefully thank Jens Beckert (Free University Berlin) RichardSwedberg (Stockholm University) Michel Callon (CSI Paris) ZoltanSzanto (University of Budapest) and Mirjam Sent and Arjo Klamer(EIPE Rotterdam) for giving me the opportunity to present my researchto different audiences Initial drafts beneted from lively conversationswith Antonio Escohotado and sharp criticism by Fabian Muniesa Thisversion has beneted from valuable comments by Phil Mirowski andLaurent Theacutevenot and from the general discussion at the lsquoNew Econ-omic Sociology in Europersquo 2000 Conference at Stockholm UniversityFor all legible English expressions found in the text the reader must thankRichard Swedberg who kindly and patiently edited the nal version I amresponsible for any remaining obscurities

ReferencesAbolaa Mitchell (1996) Making Markets Cambridge MA Harvard University

PressBasle Committee (19881998) International Convergence of Capital Measurement

and Capital Standards Basle Julymdashmdash (1996a) Amendment to the Capital Accord to Incorporate Market Risks Basle

Januarymdashmdash (1996b) Supervisory Framework for the use of lsquoBacktestingrsquo in Conjunction with

the Internal Models Approach to Market Risk Capital Requirements Basle JanuaryBernstein Peter L (1992) Capital Ideas New York Free PressBessy Christian and Chateauraynaud Francis (1995) Experts et faussaires Paris

MeacutetailieacuteBoltanski Luc and Theacutevenot Laurent (1991) De la justication Paris GallimardCallon Michel (1991) lsquoReacuteseaux technoeacuteconomiques et irreversitibiliteacutesrsquo in

Robert Boyer Bernard Chavanee and Olivier Godard (dirs) Les gures delrsquoirreversibiliteacute en eacuteconomie Paris Editions de lrsquoEHESS pp195ndash230

Chew Lillian (1994) lsquoShock treatmentrsquo Risk September 63ndash70Crane Dwight Froot Mason Kenneth Scott Perold Andreacute Merton Robert

C Bodie Zvi Sirri Eric and Tufano Peter (1995) The Global Financial SystemBoston MA Harvard Business School Press

Derman Emanuel (1996a) lsquoModel riskrsquo Risk May 34ndash7mdashmdash (1996b) lsquoThe value of models and modelling valuersquo Journal of Portfolio

Management 22 106ndash44Dunbar Nicholas (1998) lsquoThe accord is dead ndash long live the Accordrsquo Risk

October 9mdashmdash (1999) lsquoThis is the way the world endsrsquo Risk December 26ndash32mdashmdash (2000) Inventing Money New York WileyFalloon William (1995) lsquo2020 Visionsrsquo Risk October 20ndash2

EUROPEAN SOCIETIES

88

mdashmdash (1998) lsquoRogue models and model copsrsquo Risk September 24ndash31Gibson Rajna Lhabitant Franccediloise-Serge Pistre Natalie and Talay Denis

(1998) lsquoInterest rate model risk what are we talking aboutrsquo HEC LausanneWP no 9803

Giddens Anthony (1993) Consecuencias de la modernidad trans A Lizoacuten MadridAlianza

mdashmdash (1995) Modernidad e identidad del yo trans JL Gil Barcelona PeniacutensulaGrafton Anthony (1990) Forgers and Critics Princeton NJ Princeton University

PressGuldimann Till (2000) lsquoThe story of Riskmetricsrsquo Risk January 56ndash8Hollinger Richard (1991) lsquoHackers computer heroes or electronic highwaymenrsquo

Computers amp Society 21 6ndash17Irving Richard (1996) lsquoBanks grasp VaR nettlersquo Risk June S16ndashS21Izquierdo A Javier (1999a) lsquoTechno-scientic culture and the Americanisation of

international nancial marketsrsquo paper presented at the 4th EuropeanConference of Sociology Amsterdam

mdashmdash (1999b) lsquoDe la abilidadrsquo PhD dissertation Departmento de CambioSocialUniversidad Complutense de Madrid

Jacobs Bruce (1999) Capital Ideas and Market Realities London BlackwellJorion Philippe (1995) Big Bets Gone Bad New York Academic Pressmdashmdash (1997a) Value at Risk Chicago Irwinmdashmdash (1997b) lsquoIn defense of VaRrsquo Derivatives Strategy JanuaryKaye Brian (1995) Science and the Detective New York VCHKevles Daniel (1998) The Baltimore Case New York NortonMandelbrot Benoicirct (1997a) Fractals and Scaling in Finance New York Springermdashmdash (1997b) Fractales hasard et nance Paris Flammarionmdashmdash (1999) lsquoA multifractal walk down Wall Streetrsquo Scientic American February

70ndash3Meier Robert and Short James (1983) lsquoThe consequences of white-collar crimersquo

in H Edelhertz (ed) White-Collar Crime An Agenda for Research LexingtonMA Lexington Books pp 23ndash49

Merton Robert C (1995a) lsquoFinancial innovation and the management and regu-lation of nancial institutionsrsquo Journal of Banking and Finance 19 461ndash81

mdashmdash (1995b) lsquoA functional perspective of nancial intermediationrsquo FinancialManagement 24 23ndash41

Miller Peter and Rose Nicholas (1990) lsquoGoverning economic lifersquo Economy andSociety 19 1ndash31

Millman Gregory J (1995) The Vandals Crown New York Free PressMirowski Philip (1990) lsquoLearning the meaning of a dollar conservation prin-

ciples and the social theory of value in economic theoryrsquo Social Research 57689ndash717

mdashmdash (1991) lsquoPostmodernism and the social theory of valuersquo Journal of Postkey-nesian Economy 13 565ndash82

Porter Theodore (1995) Trust in Numbers Princeton NJ Princeton University PressSent Mirjam (1998) lsquoEngineering economic dynamicsrsquo in J B Davis (ed) New

Economics and its History London Duke University Press pp 41ndash62

Reliability at risk IZQUIERDO

89

Shapiro Susan P (1987) lsquoThe social control of impersonal trustrsquo American Journalof Sociology 93 623ndash58

Shimomura Tsutomu and Markoff John (1997) Takedown trans H SilvaMadrid Aguilar

Steinherr Alfred (1998) Derivatives The Wild Beast of Finance New York WileyStix Gary (1998) lsquoA calculus of riskrsquo Scientic American May 92ndash7Swary Itzhak and Topf Barry (1993) La desregulacioacuten nanciera global trans

Eduardo L Suaacuterez Meacutexico DF FCETaleb Nassim (1997a) lsquoInterview the world according to Nassim Talebrsquo Deriva-

tives Strategy Januarymdashmdash (1997b) lsquoAgainst value at riskrsquo unpublished paper electronic copy available

at httppw1netcomcom~ntalebjorionhtm

Javier Izquierdo is Associate Professor of Sociology at UNED the SpanishNational University for distance education in Madrid He has published workson economic sociology the sociology of economics and the history of stochasticconcepts in the social sciences He has a PhD in sociology from UniversidadComplutense de Madrid and wrote a dissertation on the rise of nancialengineering and derivatives-driven nancial catastrophes A book based on thisdissertation Delitos faltas y Premios Nobel (Crimes Misdemeanours and NobelPrizes) will appear in Spanish in 2001

Address for correspondence Departamento de Sociologiacutea I Facultad de CienciasPoliacuteticas y Sociologiacutea Universidad Nacional de Educacioacuten a Distancia Sendadel Rey sn 28040 Madrid Spain E-mail jizquierpoliunedes

EUROPEAN SOCIETIES

90