EvaluationofDonorSupporttoPublicFinancialManagement(PFM)ReforminDevelopingCountries

Analyticalstudyofquantitativecross‐countryevidence

FINALREPORT

November2010

PaolodeRenzio(UniversityofOxfordandODI)

MattAndrews

(HarvardUniversity)

ZacMills(Independentconsultant)

OverseasDevelopmentInstitute111WestminsterBridgeRoad

LondonSE17JDUnitedKingdom

Tel:+44(0)2079220300Fax:+44(0)2079220399

www.odi.org.uk

2

TableofContents Page

ExecutiveSummary 3 Introduction 5Background,objectivesofthestudyandanalyticalapproach 5Literaturereviewandpreviousfindings 7Keyvariablesanddatacollection 11Resultsoftheanalysis:PEFAlarge‐Nsample 21Resultsoftheanalysis:HIPCmedium‐Nsample 29Conclusionsandimplicationsforoverallevaluation 31 References 35Appendices 38AcknowledgementsTheauthorswouldliketothankthemembersoftheManagementGroup,EdHedger,GregSmith and HafsaMahtab at ODI, JoachimWehner at LSE, participants at theWorld Bankseminaranddonorofficialswhoprovideddataandinformation.

3

ExecutiveSummaryThis study ispartof abroaderevaluationofdonor support toPFMreforms indevelopingcountries.ItbringstogetheravailablequantitativeevidenceonthequalityofPFMsystems,to assess the factors that are associated with and may have determined cross‐countrydifferencesandvariationsovertime,withaparticularfocusontheimpactofdonorsupportfor PFM reforms. The bulk of the analysis draws on data from PEFA assessments in 100countries,dataondonorsupporttoPFMreformsdirectlycollectedfromsomeofthedonoragencies most active in this area, and a large dataset on other economic/social,political/institutionalandaid‐relatedvariablesthatwereidentifiedasrelevantfrompreviousresearch.There are a number of findings from the cross‐country econometric analysis that arerelevantfortheevaluation,andforbroaderdonorapproachesandpoliciesonPFMreforms.Theycanbesummarisedasfollows: Economic factors are most important in explaining differences in the quality of PFM

systems.Aid‐relatedfactors,ontheotherhand,havemore limitedexplanatorypower.As a consequence, PFM systems are more likely to improve responding to changingeconomiccircumstances,ratherthantodonorefforts.

More specifically, countries with higher levels of per capita income, with largerpopulations and with a better recent economic growth record are characterised bybetter quality PFM systems. On the other hand, state fragility, defined as being in aconflictorpost‐conflictsituation,hasanegativeeffectonthequalityofPFMsystems.

Interestinglyforthepurposesoftheevaluation,donorPFMsupportisalsopositivelyandsignificantly associated with the quality of PFM systems. On average, countries thatreceivedmorePFM‐relatedtechnicalassistancehavebetterPFMsystems.However,theassociationisveryweak:anadditional40‐50millionUS$peryearwouldcorrespondtoahalf‐pointincreaseintheaveragePEFAscore(equivalentto,say,achangefromCtoC+).

These results remained consistent through a number of robustness checks andmodelchanges.Interestingadditionalresultscomefromusingmorerecentdataorfocusingonlow‐income countries only. In these cases, the share of total aid provided as generalbudgetsupportisalsopositivelyandsignificantlyassociatedwithbetterPFMquality.Inotherwords aidmodalities, andnot just direct support toPFM reforms, contribute toexplaining differences in the quality of PFM systems in some of the poorer countrieswheremostdonoreffortsareconcentrated.

Finally,differentaspectsofdonorsupportdifferintheirrelationshipwithmorespecificPFMprocesses.A longerperiodofdonorengagement, forexample, is associatedwithbetterperformanceinupstream,dejureandconcentratedprocesses.Thismaybeduetodonors’ historical tendency to paymore attention to these simpler reform areas, butcouldalsoreflectthefactthatdownstream,defactoanddeconcentratedprocessestakelongertoimprove.

ThelevelofdonorPFMsupportisalsomorestronglyassociatedwithscoresfordejureandconcentratedPFMprocesses,againhighlightinghowdonorPFMsupportseemstofocus more on rules, procedures and specific actors within government. Results arereversedwhenitcomestoupstreamvs.downstreamprocesses.Here,theassociationis

4

strongerwithdownstreamprocesses,possiblyhighlightingthelargeamountsoffundingdevotedtoIFMISprojects,atypicaldownstreamPFMreform.

Atthesametime,theseresultssufferfromanumberofseriouslimitationsandchallenges,includingthefollowing: Dataqualityremainsanissue,especiallywhenitcomestoinformationaboutdonorPFM

support. Given the limitations of the information provided by donors, we focused onyearlydisbursementsforPFM‐relatedactivities.ThisgivesundueweighttolargeprojectssuchasIFMISintroduction,attheexpenseof‘softer’interventions.Wealsofocusedondata post‐2002, for which availability is much greater. This means that we cannotcaptureearlierdonorPFMsupport,whenthefoundationsforPFMreformswerelaidinsomeofthecountriesincludedinoursample.

WhilethepositiveandsignificantrelationshipbetweendonorPFMsupport(andGBSasashareoftotalaidincertaincases)ononehandandqualityofPFMsystemsontheotheris particularly encouraging for the purposes of the evaluation, it clearly cannot beinterpretedascausalgiventhenatureof thedata. Itcouldmerelyreflect thefact thatdonors tendtoprovidemorePFM‐relatedassistance (andmoreGBS) tocountries thathavealreadyachievedacertainsuccess in improvingthequalityof theirPFMsystems.Despite various attempts at tackling this issue, we could not prove the direction ofcausality.

Assessingthe impactofdonorsupportonPFMreformsrequirestrackingthequalityofPFM systems over time. Given the lack of sufficient time‐series data, the analysisassumes that a higher PEFA score today is a valid proxy for past reform success. Itsfindings,however,areonlypartlyconfirmedbyevidencefromasmallerdatasetlookingatchangesinPFMsystemsoverthepastdecadein19Africancountries.

These limitations and challengespoint to theneed to interpret the results of the analysispresented in this paper with a lot of caution. Moreover, they highlight the need tocomplementthesequantitativefindingswithin‐depthqualitativeresearchatcountrylevel,explainingnotonlyifandwhendonorPFMsupporthashadanimpactonPFMsystems,butalsowhyandhow itdid.Casestudycountries,however,canandshouldbeselectedtakingintoaccountsomeoftheinsightsprovidedinthispaper.

5

IntroductionThis Report is submitted by ODI to the Management Group for the “Analytical Study ofQuantitativeCross‐CountryEvidence” inaccordancewithContractNo.C97027signedwiththeSwedishInternationalDevelopmentCooperationAgency(Sida).TheworkwascarriedoutovertheperiodfromMarchtoSeptember2010byateamformedbyPaolodeRenzio(UniversityofOxfordandOverseasDevelopmentInstitute,teamleader),Zac Mills (independent consultant) and Matt Andrews (Harvard University). Ed HedgermanagedandcoordinatedthecontractatODI.GregSmithprovidedresearchsupportduringtheearlystagesoftheproject.Thereportcomprisesthefollowingsections:a) Background,objectivesofthestudyandanalyticalapproachb) Literaturereviewandpreviousfindingsc) Keyvariablesanddatacollectiond) Resultsoftheanalysis:PEFAlarge‐Ne) Resultsoftheanalysis:HIPCmedium‐Nf) ConclusionsandimplicationsforoverallevaluationBackground,objectivesofthestudyandanalyticalapproachThisanalyticalstudyofquantitativecross‐countryevidenceonPublicFinancialManagement(PFM) in developing countries is part of a broader joint evaluation, initiated by theevaluation departments of Danida, Sida, DFID and the African Development Bank, inconsultationwith theOECD‐DAC EvaluationNetwork. The evaluation aims to answer twosetsofquestions:a) WhereandwhydoPFMreformeffortssucceed?b) WhereandhowdoesexternalsupporttoPFMreformeffortscontributemosteffectively

totheirsuccess?More specifically, the present study brings together all available quantitative evidence onthequality of PFM systems across countries andover time, to assess the factors that areassociated with and may have determined cross‐country differences and variations overtime.Oneofthekeyissuesforthestudyrelatestohowbesttouseavailablemethodologicaltools toanalyseexistingquantitativedata, inorder toprovideat least tentativeorpartialanswers to the two evaluation questions. On one hand, the analysis aims to assesswhatcountry characteristics are associatedwith, or can be considered as causes of, successfulPFMreformefforts.Atthesametime,itneedstofocusmorespecificallyonthecontributionof donor support to PFM reforms to such success. Therefore, for question (a) the focusneedstobeoncontextualfactorsandconditions(economic,political,etc.)thatcanbeusedas independentvariables, inordertoassesstheirassociation/correlationwithvariations inthe dependent variable (the quality of PFM systems) across countries and over time. Forquestion (b), on the other hand, a somewhat narrower approach is needed, aimed at

6

isolating themore specific impact of donor support to PFM reforms,while holding otherfactorsconstant.Inordertoaddresstheseissues,whiletakingintoaccountthescarceavailabilityofdata,thestudyfocusesontwotypesofanalysis.Thefirsttypeofanalysis(henceforthPEFAlarge‐N)applies econometric techniques using the full cross‐country dataset from PEFA (PublicExpenditureandFinancialAccountability)assessmentsasthedependentvariable,covering100countries(seeAppendix1).Thislarge‐Nanalysistakesadvantageofthewidercoverageand availability of cross‐sectional PEFA data. It focuses on cross‐country variation, and isbasedon twoassumptions that arequite strong, and thatneed tobe kept inmindwheninterpreting the results. First, that the PEFA methodology is a good way to assess andmeasure thequalityofPFMsystems.WhilePEFA indicatorshavebeendesigned toassesscountries’compliancewith“good internationalpractices” (PEFA2005:5), there isevidencethat “good public financial management means different things in different countries”(Andrews2009:7).Second, thatahigherPEFAscore today isavalidproxy forpast reformsuccess. Given the lack of sufficient time‐series data to assess changes over time, cross‐country variation is assumed to depend mostly on the outcomes of recent PFM reformefforts, rather than on better initial conditions or other contextual factors. Multivariateregression techniques, designed tobest fit thenatureof the variables and theunderlyingmodel, areutilised to test the impact and significanceof various factorson thequalityofPFM systems across countries. For most of the explanatory factors, data refers to 2002‐2006,totakeintoaccountpossiblelagsintheireffectsandaddresssomeofthelimitationsofcross‐countrydata.Thesecondtypeofanalysis (henceforthHIPCmedium‐N) reliesonthemuchmore limiteddata that tracks changes in the quality of PFM systems over time. A smaller datasetwascollated using HIPC1 and PEFA assessments for 19 countries in Sub‐Saharan Africa (seeAppendix2)2,includingcountrieswhererepeatPEFAassessmentshavebeencarriedout,inorder tobuild apanel dataset that covers theperiod from2001 to themost recentPEFAassessment. As such a medium‐N sample is too small for any econometric treatment,simpler analytical methods are utilised, highlighting country clusters and relevant andinteresting patterns or configurations of factors that are associated with PFM reformsuccess,verifyingandvalidatingthefindingsfromthePEFAlarge‐Nanalysis.Thecombinationof these twodifferentkindsofanalysis is thebestpossiblewayofusingavailabledata to shedsome lighton theevaluationquestions inways thatgobeyond thecountry‐specific(butmuchmoredetailed)evidencethatcanbegeneratedthroughthecasestudies.Atthesametime,itshouldbestressedthatthefindingsfromthisstudyarelimitedtobroadcorrelationsandidentificationofrelevantpatterns,ratherthanstrongandspecificcausal linkages.Nevertheless,we believe that the studymakes an important contributionthat does not only advance quantitative work on this subject, but can also nicelycomplementmorein‐depthqualitativecasestudies.1TheHighlyIndebtedPoorCountries(HIPC)initiativepromotedaseriesofAssessmentsandActionPlans(AAPs)meanttotrackchangesinthequalityofPFMsystemsandidentifynecessaryreformsfortheirimprovement.Forfurtherdetails,seeIDA/IMF(2003).2Fordetailsofthemethodusedtocompilethedataset,seedeRenzioandDorotinsky(2007).

7

Literaturereviewandpreviousfindings3In recent years, donor‐supported PFM reform programmes have covered a range ofinitiatives aimed at strengthening the rules and procedures which underpin the budgetprocess in aid recipient countries. These have typically focused on a number of standardinterventions, which include improving the comprehensiveness of budget operations,building better links between annual allocations, medium‐term policy objectives andperformanceindicators,andcomputerisingbudgetmanagementandexpenditurecontrols4.Whathascertainlychangedovertime,however, isthescaleofresourcesinvestedandthenumberof actors involved.A recentWorldBankevaluationofPublic SectorReform (PSR)programmes(WorldBank2008),which includesupporttobudgetreforms,showsthatthenumber of World Bank‐financed projects with a substantive PSR component quadrupledbetweentheearly1990sand2005,increasingfromlessthan10%tomorethan20%oftotalprojects5.Data from theOECDDAC’sdatabase includingalldonors showsaneven starkerincrease in committed funds for activities related to public sector financial management,whichgrewmorethanten‐fold,fromUS$85.1min1995toUS$930.6min2007.Duringthesame period of time, the number of donor agencies involved in providing technicalassistanceinthePFMareahasrisentoover25(IMF2007:22).Given such interest and investment, it is somewhat puzzling that so little evidence andanalysisexistsonthecomparativeperformanceofPFMsystemsacrosscountriesandovertime, on the factors that underpin successful PFM reforms, and on the role that donoragenciescanplayinPFMreformprocessesindevelopingcountries.Oneofthekeyreasonsforthis, inevitably,islackofavailablecomparativedata.Aswillbeshownbelow,effortsatassessingthequalityofPFMsystemsusingstandardizedmethodologiesonlystartedaboutadecade ago, when HIPC assessments were launched. In recent years, however, theintroduction and gradual expansion in the use of the PEFA methodology has providedresearcherswithmoreandbetterdatatostartaddressingtheevaluationquestions.Therehavebeenonlytwocross‐countrycomparativeanalysesofPEFAassessmentdatasofar, looking at the performance of PFM systems both across different areas of budgetmanagement and across countries. General analysis by de Renzio (2009a) of 57 PEFAassessments highlights how average indicator scores tend to deteriorate the further onemoves through the budget cycle (from formulation to execution, reporting and scrutiny).DrawingonadatasetofdisaggregatedPEFAscoresfor31Africancountries,Andrews(2010)also investigates patterns or ‘themes’ in performance across PFM process areas. Hereorganisesthe73PEFAindicatordimensionsintoclustersagainstthebudgetcycle.Hisfirstfinding isconsistentwiththatofdeRenzio (2009a) forawiderspanofcountries.AveragePEFAscoresdecline in theprogression fromupstreambudget formulation todownstreamfinancialmanagementandaccountabilityprocesses.Onaverage,formalbudgetpreparationand legislativebudgetreviewscoremoststrongly,withexternalauditand legislativeauditanalysis shown to be among the weakest processes. The implication is that budgets are‘bettermadethantheyareexecuted’.

3PartsofthissectiondrawonarecentreviewinHedgeranddeRenzio(2010).4WorldBank(1998)andIMF(2007).5ForSubSaharanAfrica,suchproportionreaches37%.

8

OtherinterestingfindingscomefromfurtherdistinctionsthatAndrews(2010)makesamongPEFA indicators. First, he distinguishes PFM reforms linked to legislation, processes andprocedures(i.e.dejurereforms),fromthoselinkedtotheimplementationorestablishmentofnewpractices (i.e.defacto reforms), findingthataveragescoresforde juredimensionsare consistently higher than for de facto ones. In other words, improvements in budgetpractices lagbehindreformsinbudget lawsandprocesses.Second,Andrewscontraststheperformance of PFM process areas involving small groups of ‘concentrated’ actors, withprocesses which engage broader sets of ‘de‐concentrated’ actors. Out of the total 64disaggregated ‘budget cycle’ dimensions (excluding therefore those linked to budgetoutcomes),26are limited toconcentratedactors suchas theBudgetDepartmentorDebtManagementUnit,whiletheremaining38dimensionsrelatetoactorssuchaslineministriesor Parliament. The evidence shows that countries score higher against the first set ofmeasures,suggestingthatactorconcentrationisassociatedwithbetterfunctioningofPFMsystems.As far as cross‐country comparisons are concerned, statistical analysis carried out by deRenzio (2009a) highlights some interesting cross‐country patterns. For example, countriesthat are richer andmore democratic have better quality PFM systems,while countries inSub‐Saharan Africa and South Asia perform worse on average. However, these binaryassociations are not necessarily significant from a statistical point of view, as for eachcountrytheoverallaveragePEFAscoremightbedeterminedbyanumberofthesefactorsand categories interacting contemporaneously. Analysed throughmultivariate regressions,infact,theonlyvariablesassociatedwithsignificantchangesinPEFAscoresareincomeleveland aid dependency. Even these findings are ambiguous. It is not surprising that higherincome levelsare significantlyassociatedwithhigherqualityofPFMsystems.But it isnotclear that income levelper se is thedriverof better PFMperformance, rather thanothervariablesthatareoftenhighlycorrelatedwithincome,suchaseducationlevelsortheshareof government revenues that accrues to taxes rather than rents. The positive associationwith aid dependency, apart from the very small coefficient,may in fact reflect a reversecausalitywherecountrieswithbetterbudgetinstitutionsreceivemoreaid.InhisanalysisofPFMperformanceacrossAfricancountries,Andrews(2010)usesaslightlydifferent set of explanatory variables, such as: a) level of income and income growth; b)degreeofcountrystabilityorfragility;c)dependenceon‘rents’asmajorrevenuesources;d)lengthofuninterruptedreformperiods;e)typeofadministrativeheritage.Byorganisingthe31African countries included inhis analysis into five separatePFM ‘performance leagues’according to their average PEFA scores, Andrews investigates the influence of eachcontextualvariableuponPFMsystemstrength.Hisfindingsrevealthefollowingtrends:a) TheeconomicgrowthratehasastrongerassociationwithhigherqualityPFMthanthe

absolute level of income. In fact, some low‐income but relatively fast‐growing AfricancountriesfeatureinthehighestPFMperformanceleague.

b) CountrystabilityappearsconducivetoPFMprogress.Fragilestates–identifiedusinganIMF classification – dominate the lowest league of PFM performance. These displayparticular weaknesses in strategic budgeting, budget transparency, budget executionandinternalcontrol.

9

c) ‘Rentier’ states (i.e. thosewhichaccruemost revenue fromexternal sources, includingnatural resources, trade taxes and donor funding) tend to have weaker PFM systemscomparedwith ‘fiscalstates’ (i.e.thosewhichcollectamajorityoftheirrevenuesfromdomesticcitizens).

d) CountrieswithaPRSP6formorethanthreeyearsachievehigherPEFAscoresinalmostallPFMprocessareas.TheexistenceofaPRSPisusedasaproxymeasureforbroadreformcommitment, as it may ‘lock in’ pro‐developmental policy choices and reformprogrammes.

e) The evidence on administrative heritage is ambiguous, except for the downstreamexternal accountability dimension, where Francophone countries tend to score loweragainstthePEFAindicatorswhencomparedtoAnglophoneones.

ThetwostudiesbyAndrewsanddeRenzioareuseful insheddingsomeinitial lightonthefactorsthatmightinfluencethequalityofPFMsystemsacrosscountries,butsaylittleabouttheroleandinfluenceofdonorsanddonorassistance.Therefore,welookedatthebroaderliteratureonaideffectiveness,inparticularatcross‐countrystudieslookingattheimpactofforeignaidonrecipientcountrygovernance.Again,thereare justahandfulofpapersthatlookspecificallyattheeffectofaiddependenceandaidfragmentationonmeasuresofthestrengthofinstitutions.Usingdifferentpossiblemeasuresofstatecapacity,fromindicatorsof bureaucratic quality to taxation efforts, Knack (2001) and Brautigam and Knack (2004)findthathigherlevelsofaiddependenceareindeedassociatedwithdeclinesinthequalityofgovernanceandintaxrevenuesasashareofGDP.Thisishowtheyjustifythisfinding:

“theway large amounts of aid are delivered canweaken institutions rather than build them.Thiscanhappenthroughthehightransactioncoststhataccompanyaid,thefragmentationthatmultiple donor projects and agendas promote, problems of "poaching," obstruction ofopportunities to learn, and the impactof aidon thebudgetprocess. Lessdirectly,but just asimportant,high levelsofaidcancreate incentivesthatmake itmoredifficulttoovercomethecollective action problems involved in building a more capable and responsive state.”(BrautigamandKnack2004:260‐1)

ThesefindingsaresupportedbyevidenceinMossetal.(2006),whoemphasisehowheavyrelianceonexternalfundingsourcescangeneratean‘aid‐institutionsparadox’,wherebyasaid increases, the recipientgovernment’saccountability relationship towards its citizens isweakened. Further evidence and tests provided by Ear (2007) also confirm the negativeimpact of aid on governance, even if just the technical cooperation component of aid isconsidered. The impact of aid dependency on revenue generation is slightly morecontroversial. Cross‐country analysis presented inMoss et al. (2006) and Remmer (2004)presents furtherevidence in supportof theargument that “high levelsofaiddependencehave failed to create strong incentives for governments to marshal new resources ordevelopmental aims: instead, aid has simultaneously fostered the growth of governmentspendingandthereductionofrevenueeffort”(Remmer2004:88).Ontheotherhand,somecase study evidence points to the fact that under specific circumstances aid flows areassociatedwithagrowthindomesticrevenues(FagernasandRoberts2004,PackandPack1990).Finally,KnackandRahman(2007)arguethatit’snotjustthelevelofaiddependence

6APovertyReductionStrategyPaper(PRSP)isacomprehensivedevelopmentplanthatmanyAfricancountriesformulatedinordertogainaccesstodebtreliefafter1997.

10

that causes institutional deterioration, but also the degree of fragmentation of donorinterventions, determinedbyhowmanydonors arepresent in a recipient country andbytheirsharesofthetotalaidthatthecountryreceives.Mostof the literaturementionedaboveusesanaggregatemeasureofaiddependenceasthemainindependentvariable,withoutrecognisingthatforeignaidcomesinmanydifferentformsandguise,whichmighthavedifferential impactson institutions.Theonly twomorespecificfindingsrelatetotechnicalcooperationandtoaidfragmentation.Inotherwords,itisimportanttokeepinmindthatthe‘quality’ofaidmightbeasimportantasits‘quantity’indeterminingitsimpactongovernanceandinstitutionsinrecipientcountries.Whataboutthemore specific component of aid that this study aims to look at? At present, there are nocomprehensive studies looking at the impact of donor support for PFM reforms on thequalityofPFMsystems.TheonlyexistingcomprehensiveevaluationofthiskindofsupportisincludedintheWorldBankevaluationmentionedabove(WorldBank2008,Wescott2009).The evaluation finds that “about two thirds of all countries that borrowed for financialmanagement showed improvement in this area”,with public financialmanagement being“themost consistent area of improvement in the case studies”when compared to otheraspectsofpublicsectorreforms(WorldBank2008:xv).Morespecifically,usingCPIAdataasayardstickforimprovementsinthequalityofbudgetinstitutions,theevaluationfindsthat64%ofcountriesthatreceivedanysupportforPFMreformprogrammessawtheirCPIAPFMindicator score increase, compared to 32% in countries that did not receive such support(Wescott 2009:147).On the other hand, the evaluation also notes a number of problemswiththewayinwhichdonorsprovidedsupporttoPFMreforms.Forexample,itnoteshowheavydonorinvolvementoftenleadstoasituationinwhich“expectationsandobjectives[ofbudgetreforms]tendtobemoreambitiousandglobal,reflectingthedonors’ listofthingsthatneed fixing rather than thegovernment’s listof things it is ready todo” (WorldBank2008:40).Theinsistence“onafullarrayofpublicreforms”,theevaluationobserves,meansthat “[World Bank] staff often lack the time and resources to develop a fully tailoredproduct.Sotheresultislikelytobeonesizefitsall,offtheshelf”(2008:41).In summary, the existing literature that looks at determinants of PFM quality across thedevelopingworld,attheimpactofforeignaidongovernanceandatdonorsupporttoPFMreforms isquitescarce,butneverthelessprovidesusefulbackgroundandsome interestingelementsforouranalysis.Itidentifiesanumberofvariablesthatneedtobeincludedinourmodels. Intermsofgeneralcountrycharacteristics, itprovidespreliminaryevidenceoftheimportance of, for example, levels of income and income growth, strength of democraticinstitutions, government revenue sources, political stability and administrative heritage inshaping the quality of PFM systems. When looking at the influence of foreign aid, ithighlightstherolenotonlyofoverallaiddependencylevels,butalsoofaidfragmentationinaffectinggovernancestandardsinaidrecipientcountries.Importantstatisticalissueslinkedtoomitted variables and thepossibility of reverse causation are alsomentioned as issuesthat need to be considered in further research. Our study aims to build on these initialfindingstakingadvantageoflargerdatasetsthatarebecomingavailable.Inthenextsectionwethereforeturntoissuesofdefinitionandmeasurement.

11

KeyvariablesanddatacollectionThedatasetthatwascompiledtocarryouttheanalysisincludesthreesetsofvariables:(a)dataonPFMsystems(dependentvariable);(b)dataondonorsupporttoPFMreforms(mainindependentvariable);and(c)dataonother independentandcontrolvariables.Thesearedescribedindetailbelow.DataonPFMsystemsThere are very limited sources of information and cross‐country data which can be usedreliablytoassessandcomparethequalityofPFMsystemsacrosscountriesandovertime.Forthelarge‐N,cross‐countryanalysis,thetwodatasetsthatweconsideredweretheWorldBank’s Country Policy and Institutional Assessment (CPIA) and the set of indicatorsdevelopedbythePEFASecretariat.CPIA’sindicator13isproducedbytheWorldBankaspartofanannualinternalperformanceratingexercise,andmeasures the ‘QualityofBudgetaryandFinancialManagement’alongthree dimensions: a) comprehensive and credible budgeting linked to policy priorities; b)effective financialmanagement systems; and c) timely and accurate accounting and fiscalreporting. The indicator ranks about 75 countries on a six‐point scale (1‐6). Despite itsrelevanceandcoverage, theCPIA indicator suffers fromtwomaindrawbacks: (i) its singlenumericalvalueprovidesvery limiteddetailand informationonPFMsystemperformance;and(ii)thereisanecdotalevidenceofCPIAratingsbeingbasedonsubjectivejudgementandaffectedbylendingdecisions,thereforeintroducingimportantmeasurementerrors7.We opted instead for the use of PEFA data. The PEFA Performance MeasurementFrameworkforPFM(PEFA2005)isthemostcomprehensiveattemptthusfaratconstructinga framework to assess the quality of budget systems and institutions. It comprises 28indicatorswhichassessinstitutionalarrangementsatallstagesofthebudgetcycle,togetherwith cross‐cutting dimensions and indicators of budget credibility. It also includes threeadditionalindicatorsondonorpractices.Thedatasetweworkedwithincludedtheresultsofnational‐level assessments for 107 countries and territories. Of these, 7 countries weresubsequently excluded from our sample: Kosovo and the Palestinian Territories becausethey are territories and not states, limiting the availability of other relevant data; Tuvalubecauseanimportantamountofothernecessarydatawerenotavailable;Norwaybecauseit is a clear outlier being a high‐income country; and Bangladesh, Gabon and Nicaraguabecause the PEFA assessments had too large a number ofmissing indicators. The overallsample therefore includes data for 100 countries8. Two thirds of the assessments werecarriedoutbetween2008and2009.Only42ofthe100assessmentreportshavebeenmadepubliclyavailable,while24arestillconsideredtobeatdraftstage.In order to transform PEFA scores into the dependent variable to be used in our large‐Nanalysis,wefollowedaseriesofsteps.First,weonlyconsideredindicatorsPI‐5toPI‐28,as

7SeeArndt(2008).8AfulllistisincludedinAppendix1,alongsidetheyearinwhichtheassessmentwascarriedoutandthestatusofthereport.

12

indicatorsPI‐1toPI‐4coverPFMsystemoutcomesandperformance,andnotthequalityofPFM systems per se. Second, for multi‐dimensional indicators we used sub‐indicator/dimensionscoresratherthansummaryindicatorscoresinordertofullyexploittheinformationcontainedinthePEFAscores.Thisalsoallowedustoavoidthedownwardbiasintroduced by theM1 scoringmethodology,where summary indicators are based on thelowestscoringdimension,or‘weakestlink’.Third,weconvertedtheletterscoresincludedinPEFAreports intonumericalscores,withhigherscoresdenotingbetterperformance(fromA=4toD=1).Fourth,and finally,weconstructedourdependentvariable in threedifferentways:a) As an overall simple average of the 64 numerical scores that include all sub‐

indicators/dimensionsforindicatorsPI‐5toPI‐28;b) Asaveragesofnumericalscoresforsub‐indicators/dimensions ineachofsixclustersof

indicatorsgroupedbyphaseofthebudgetcycle9.Thisgeneratessixsub‐indicesthatwillbeusedseparatelyasdependentvariables;

c) Asindividualscoresforeachofthe64sub‐indicators/dimensionsinindicatorsPI‐5toPI‐28.Thisgeneratesapanel‐typedatasetof64dimensions*100countries.

Itcouldbearguedthattheseun‐weightedaveragesaretoosimplisticanindicatortoallowfor cross‐country comparisons. The PEFA Secretariat has also warned about a number ofissuesregardingaggregatingPEFAscoresandcomparingthemacrosscountries(PEFA2009).Inordertocheckthatvaluesforthevariableweconstructeddidnotsufferfromsubstantialbiases,weusedtwoprocedures,onestatisticalandonesubstantive.Forthestatisticalone,weimputedvaluesformissingobservationsandappliedPrincipalComponentAnalysis(PCA)toboththeoverallandtheclusteraverages.PCAisastatisticaltechniquedesignedtodetecttheunderlyingstructureofanumberofrelatedvariables,andreducetheirnumberthroughthe creation of a new variable (or variables) that reflect that structure10. In our case, itgeneratedanalternative‘summary’PEFAscorebasedontheinformationincludedinthe64underlyingdimensions.Thesubstantiveprocedurewasbasedonthecreationofanumberofmore parsimonious indices taking only sub‐indicators/dimensions that donor assistancetends to focus on more directly into account, such as those linked to MTEFs, budgetclassification, internal controls, etc. Inboth cases, the resulting variableswere veryhighlycorrelatedwiththeoverallaveragescores(>0.95).Webelievethatthisprovidessufficientevidencethattheoverallaveragesusedinouranalysisinfactcapturerelevantaspectsofthequality of PFM systems, and do not suffer from major biases. Finally, we think that thesamplesizeislargeenoughtoreducetheriskofinvalidcomparisons.

9TheseclustersareslightlydifferentfromtheonesincludedinthePEFAmethodology,andhavebeenrearrangedtoincreasetheirlevelofinternalconsistency.Forfurtherdetails,seeAppendix3,Andrews(2010:8)andAndrews(2007).10AsdescribedbyVyasandKumaranayake(2006:460),“PCAisamultivariatestatisticaltechniqueusedtoreducethenumberofvariablesinadatasetintoasmallernumberof‘dimensions’.”Rabe‐HeskethandEveritt(2006)addthat“thebasicideaofthemethodistotrytodescribethevariationofthevariablesinasetofmultivariatedataasparsimoniouslyaspossibleusingasetofderiveduncorrelatedvariables,eachofwhichisaparticularlinearcombinationofthoseintheoriginaldata.”ImputationbyChainedEquations(ICE)wasutilisedtogeneratethe103missingobservations.

13

Figure1.DistributionofoverallaveragePEFAscores

Overall average PEFA scores across the 100‐country sample vary between aminimum of1.38(Guinea‐Bissau)toamaximumof3.58(SouthAfrica),withameanof2.44andamedianof2.47.Figure1showsthescoredistribution.Ascanbeseen,mostcountrieshaveaveragescoresbetween2and3,which in theoriginalPEFAmethodologywould fallbetweena ‘C’anda‘B’score.However,therearealso20countriesthatwouldbroadlyscorebetweena‘D’anda‘C’,whilethereareonly11countrieswithanaveragescoreof‘B’andabove.These aggregate averages inevitably hide a lot of the underlying information related toindividual indicatorsanddimensions.Toaddress this shortcoming,wehavealso lookedataveragescoresacrossmorespecificPFMareas,createdbygroupingthe64PEFA indicatordimensionsinsixinternallyconsistentclustersthatfollowthebudgetcycle(seeAppendix3).These are: (a) strategic budgeting; (b) budget preparation; (c) resourcemanagement; (d)internal control, audit and monitoring; (e) accounting and reporting; and (f) externalaccountability (Andrews 2010:8). Figure 2 compares average scores and their distributionacrossthesesixclusters.Ascanbeseen,‘budgetpreparation’istheareainwhichcountriesperformbest,withanaveragescorefortherelevantPEFAdimensionsof2.77.The‘resourcemanagement’and‘accountingandreporting’clustersarealsocharacterisedbyresultsthatare slightlyabove‐average.Performance in theother threeclusters isworse. Inparticular,‘externalaccountability’ ischaracterisedbythelowestaveragescorefortherelevantPEFAdimensions, 1.99. It has to be noted, however, that given the great variationwithin eachclusterthedifferencesbetweentheaveragescoresarenotstatisticallysignificant.

14

Figure2.AveragePEFAscoresbyPFMcluster

Note: box‐plots include information on the minimum, first quartile, median, third quartile and maximumaveragescoresforthegroupofPEFAdimensionsineachPFMcluster.

Whiletheseresultsconfirmthepreviousfindingthat,forexample,budgetsarebettermadethanexecuted,theyalsohighlighttwointerestingexceptions.Ononehand,thedownstream‘accounting and reporting’ cluster performs better than average, while the upstream‘strategicbudgeting’clusterdoesnot.Thisisparticularlysurprising,as‘strategicbudgeting’refersmostly to the adoptionof amedium‐termperspective inbudgeting, an areawheredonors have invested substantial resources. It may however reflect the difficulties thatcountries face in introducing medium‐term budgeting frameworks that goes beyond themere projections of aggregate revenues and expenditures, but reflects well‐developedsectoralplansandlinkscapitalspendingwithitsrecurrentimplications.Wealsolookedatwhethercountriestendtoscorebetterorworseacrossthesixclusters,bycalculating the rank correlation coefficients for each pair of clusters. In other words, wewantedtocheckthelikelihoodthatanycountrywithahighaveragescoreinaclusterdoesbetterinotherclusters,too.Surprisingly,coefficientsarequitelow,andrangefrom0.30to0.75. This indicates that countriesdonotnecessarily score consistently across the variousclusters.Whilewetakethisintoaccountintheanalysisthatfollows,thisiscertainlyanareathatdeservesfurtherattentionandresearch.Finally, we checked to see whether missing observations could be the source of anyimportant biases. As indicated above, we excluded from the analysis three countries forwhichthereweretoomanymissingobservations.Whileabouthalfof thecountries inoursamplehavesomePEFAdimensionthatwasnotscored,inthegreatmajorityofcasesthisislimitedtolessthanfiveofthe64PEFAdimensionsthatweconsidered.Onlytwocountrieshavemorethan10missingobservations(MacedoniaandGambia).Abouthalfofthemissingobservationsaredueto the fact thatPEFAassessmentshadamore limitedscopeanddidnotcoverthefullsetof indicators,while lessthanonethirdareduetoa lackofsufficient

15

information to justifya specific rating.As faras specific indicatorsareconcerned, the twoindicatorswiththemostmissingobservations(morethan15)areindicatorPI‐8ontransfersto sub‐national governments and indicator PI‐15 on tax collection. Otherwise, missingobservations are quite evenly spread across the full range of indicators. Furthermore,countrieswithahighernumberofmissingobservationsarenotsignificantlydifferentfromthe rest of the sample in terms of their levels of income, region or any other relevantcharacteristic. In summary, we do not think that, after excluding countries with veryincomplete assessments,missing observations constitute a seriousmeasurement problemthatmightunderminetheanalysisthatwillfollow.For the second part of our analysis, focused on a medium‐N sample of African HIPCcountries,ouraimwastoaddressoneofthemainweaknessesofthePEFAdataset.Whilemorethan150assessmentshavebeencarriedoutsince2005,theyprovideonlyasnapshotofPFMsystemperformanceacrosscountries11. Interestingly,however,PEFAindicatorscanatleastpartiallybemappedontoexistingpreviousassessments,helpingtoextendtheirtimeseriestomorerecentyears.Forthemedium‐Nanalysis,therefore,wecombinedPEFAdatawith data resulting from anothermethodology for assessing the quality of PFM systems,developedjointlybytheIMFandtheWorldBanktotestthesystemsincountriesqualifyingfor debt relief under the Highly Indebted Poor Countries (HIPC) initiative. These HIPCAssessments were carried out in 2001 and 2004 in 23 countries12, and scored countrysystemsagainstbenchmarksforfifteenindicatorscoveringallstagesofthebudgetcycle13.Despite some of the limitations of this methodology14, it constitutes the only codified‘historical’evidencethatallowsforaconsistenttrackingofthequalityofPFMsystemsovertime.Onthisbasis,wecompiledasmallpaneldatasetthattrackschangesin11indicatorsofPFMqualityfor19countriesinSub‐SaharanAfrica,manyofwhichhavealsohadtwoPEFAassessments (see Table 1 below). The dataset covers the period from 2001 to the mostrecent PEFA for each country, and groups indicators in three clusters, namely: (a)transparencyand comprehensiveness (INFO), (b) linkingbudgets, policies andplans (POL),and(c)control,oversightandaccountability(CTRL)15.

11Thetotalnumberincludesrepeatandsub‐nationalassessments.AsofOctober2009,21countrieshadcarriedoutrepeatPEFAs,butoftenwithinatimespanthatisoftennotsufficienttoseeandjustifysubstantivechanges.12SeeIDA/IMF(2005).In2006threeadditionalcountrieswereassessed.13In2004,anadditionalindicatoronprocurementwasadded.14First,calibrationofthebenchmarksdidnotcapturethesignificantvariationsobservedforsomeindicatorsacrosscountriesandovertime.Second,theactualassessmentsinsomecasesrevealedinsufficientevidencetojustifythescoring.Third,theindicatorsomittedimportantdimensionssuchastaxadministration,fiscaldecentralisationandparliamentaryaccountability.15Forthedetailsofwhichindicatorsfallundereachcluster,seeAppendix4.Formoredetailonthemethodology,seedeRenzioandDorotinsky(2007)anddeRenzio(2009b).

16

Table1.QualityofPFMsystemsacross19AfricanHIPCs,2001‐2010

Source:IDA/IMF(2005)andPEFAassessments.Basedonauthors’calculations.Note:NumericalscoresarebasedonmethodologydescribedindeRenzioandDorotinsky(2007).‘2007’denotesPEFAassessmentscarriedoutin2005‐07.‘2010’denotesPEFAassessmentscarriedoutin2008‐10.

What the results show is that only five of the 19 countries (Burkina Faso, Cameroon,Ethiopia, Mali and Zambia) for which historical data is available saw an uncontroversialimprovement in the quality of their PFM systems as measured by the subset of HIPCindicators. Four countries saw their PFMquality deteriorate (Benin, Gambia, Rwanda andUganda).Fortheothertencountries,it ismoredifficulttodetectacleartrend.Ghana,forexample,recordedimpressiveimprovementsbetween2001and2007,tothensufferaslightworseningof itsoverall score in2010.Mozambique’soverall scorehas improvedover thewhole period, but has seen some considerable fluctuations. Tanzania has consistentlyperformedamongthebestinthe19‐countrysample,buthasseenarecentslide.Intermsofperformance across the three different clusters, most improvements happened in linkingbudgets, policies and plans, where nine countries increased their score, while the areawheretheleastprogresswasmadewascontrol,oversightandaccountability.The correlationwith the overall PEFA averages used for the large‐N sample is quite high(0.76), which ensures broad consistency between the two scoringmethods. At the sametime, there are some considerable differences. For example, the overall PEFA scores forRwanda,MalawiandMozambiqueareconsiderablyhigherusingthe64dimensionsaveragethanthesub‐setofindicatorsthatcanbemappedbackontothepreviousHIPCassessments.Theseresults thereforeneedtobetakenwithsomecaution,probablyasdescribingbroad(albeit somewhat incomplete) trends rather than specific changes in the quality of PFMsystems.DataondonorsupporttoPFMreformsDetailed and reliable data on donor support for PFM reforms is also difficult to find. TheOECD‐DAC Creditor Reporting System (CRS) database, the main global source of data ondetailed aid flows, includes a sub‐sector purpose code for ‘public sector financialmanagement’ (15120). This includes information about donor commitments anddisbursements for aid activities in support of PFM reforms, going back to 1995 forcommitments, but only to 2002 for disbursements. Despite its apparent relevance as a

17

source for data on donor support to PFM reforms, the quality, reliability andcomprehensivenessoftheCRSdataishighlyquestionable.Analysisoftheunderlying‘micro‐data’(i.e.thespecificentriessubmittedbydonoragencies)revealsthatnotonlyanumberof activities included should not be classified as support to PFM reforms, but also theomissionofmanyactivitiesthatshouldbeincludedunderthisclassification.Inothercases,thedetailsavailabledonotallowfortheverificationoftherelevanceoftheaidactivityforPFM reforms. This is especially true for multilateral agencies, such as the EuropeanCommissionortheWorldBank,whicharenotfullDACmembersandthereforehavemorelimitedreportingobligations.These limitations constituted a serious challenge for the study. A substantive effort wasthereforeput into first‐handcollectionofdonordata.To facilitate the task,wetargetedasub‐setof13donoragenciesthatareparticularlyactiveinthefieldofPFMreforms.Theseincluded the members of the Management Group (Sida, Danida, DFID, and the AfricanDevelopment Bank), plus the Dutch, Norwegian and French aid agencies, the EuropeanCommission, the World Bank, the IMF, USAID, and the Asian and Inter‐AmericanDevelopment Banks. While this sample does not ensure full coverage of data on donorsupportforPFMreforms,webelievethatitprovidesagoodpictureandasuitableproxyfortheamountofsupportprovidedtoPFMreforms16.Eachagencywassentadata request form(Appendix5)andasked toprovide informationabout actual disbursements for technical assistance and other activities related to PFMreformsupportovertheperiod1995‐2008.Whilelimitedinformationwasrequestedforthe100countriesinthePEFAlarge‐Nsample,weaskedformoredetaileddataforthemedium‐NsampleofcountriesincludedintheHIPC/PEFApaneldataset(suchasthePFMfocusarea,the main inputs provided, etc.). Of the 13 agencies contacted, ten replied, with varyingdegrees of completeness17. Unfortunately, no agency provided the more detailedinformationrequestedfortheHIPCmedium‐Nsample,whichinevitablylimitedthedepthofouranalysis.Forthethreemissingagencies18,welookedatinformationavailableonpublicwebsitesorwentthroughthemicro‐dataintheirentriesintheCRSdatabase,selectingonlytheactivitiesthatcouldbeidentifiedasdirectlyrelatedtoPFMreforms.Inanumberof caseswehad tomake some judgement calls. Someagencies, forexample(suchastheWorldBank,theAfricanDevelopmentBankandDanida),classifygeneralbudgetsupportoperationsasPFMinterventions.Thisgreatlydistortedthedata,aswewerelookingfor a measure of direct support to PFM reforms. Budget support, while it can indirectlystrengthenPFMsystems, ismainlyaimedat financinggeneralgovernmentoperations.Wetherefore decided to either exclude these operations, or only include the share thatwasindicated as being directly associatedwith PFM reform initiatives,where this informationwasavailable. Inotherdubiouscases,wesimplytrustedthedataprovidedtousbydonoragencies.

16Infact,thesedonoragenciescollectivelyprovidemorethan90percentoftotaldonorcommitmentsfor‘PublicSectorFinancialManagement’asrecordedintheDAC/CRSdatabasefortheperiod1995‐2008.17DataforDanidaandFrenchaidwereintheendexcludedasinformationwasnotsufficient.18EuropeanCommission,USAIDandAsianDevelopmentBank.

18

Theyearlyaverageofdonordisbursementsineachcountrywasthenincludedinourdatasetasthemainindependentvariable,basedonthehypothesisthathigheramountsoftechnicalassistanceshouldbeassociatedwithbetterqualityPFMsystems19.Thereareanumberofproblematicissueswiththisapproach.Forexample,datamightbedrivenbylargeprojectssuchasthe introductionof IntegratedFinancialManagement InformationSystems(IFMIS),which require substantial hardware investments, at the expense of ‘softer’ interventionsthatmightaffectPFMqualityata lowercost,as inthecaseofMediumTermExpenditureFrameworks (MTEFs). Given the greatly improved availability of disbursement data from2002onwards,wemostly utilised the sumof PFM‐relateddisbursements over the period2002‐2006. While this makes sense in a number of ways, it also introduces a furtherpotential sourceofbias inourdata.Ononehand, focusingon recentdonorPFMsupportmeansthatwearemorelikelytocapturemoredirecteffectsonaspectsofthePFMsystemthat are part of the same consensus that is behind the construction of the PEFAmethodology20.Ontheotherhand,ourdatadonot includeenough informationonearlierdonor PFM support, when the foundations for PFM reforms were laid in some of thecountriesincludedinoursample.Figure3.TotaldonorPFMsupport,1995‐2008

According to the data we collected, overall donor funding for PFM‐related activitiesincreased from around US$60m in 1995 to more than US$400m in 2008 (see Figure 3above). The largest recipients of technical assistance in support of PFM reforms between2002and2006wereAfghanistan(US$33.2mperyear)andMorocco(US$25.1mperyear).InAfghanistan’s case, the bulk of the funding came from the Asian Development Bank, theWorld Bank and DFID, funding a diverse set of PFM‐related activities. In Morocco, PFM

19Thisgoesagainstsomeofthemoregeneralfindingsintheliteratureaboutthenegativeeffectsofoverallaiddependency,basedontheassumptionthattechnicalassistancetargetedatspecificreformsdoesnotsufferfromthesamedrawbacks.20Unfortunately,thismightalsogenerateastatisticalproblemofendogeneity.However,2002‐06istheperiodprecedingthetimewhenmostPEFAassessmentswerecarriedout,allowingustoatleastpartlyaddresstheissueofreversecausation(moreonthisbelow).

19

support is linkedmostly to a seriesofWorldBank loans forpublic administration reform,aimedatimprovingtheeffectivenessofpublicresourcemanagement.Othercountriesthatreceivedmore than US$15m per year over the same period are Indonesia, Tanzania andMozambique.Attheotherendofthescale,thegroupofcountriesthatreceivedthesmallestamounts(i.e.lessthanUS$10,000peryear)indonorsupportforPFMreformsoverthesameperiod include mostly small island states (St. Lucia, St. Kitts and Nevis, Kiribati and theSeychelles) and Belize21. The World Bank and DFID were the two donors, among thosesurveyed,thatprovidedthelargesttotalamountsinsupportofPFMreforms.Interestingly,thecorrelationbetweenourdataondonorPFMsupportanddatatakenfromtheDAC/CRSdatabaseisalow0.23.Giventheeffortthatwentintoensuringthereliabilityofthedatawecollected,wethinkthatthisisprobablyareflectionofthepoorerqualityofCRS data, something which should be taken into account in further analyses, and whichhighlightstheneedfordonoragenciestoincreasethecomprehensivenessandreliabilityoftheirCRSreporting.Apart fromdirectsupport toPFMreforms,someadditionalvariableswere included in thedatasettocaptureotheraspectsofdonors’ impactonPFMsystems inrecipientcountries.Theseare:a) A variable capturing the share of total aid being channelled through programme aid

modalities (general budget support and sector programmes) calculated fromDAC/CRSdata,withthehypothesisthathighersharesofprogrammeaidarepositivelycorrelatedwiththequalityofPFMsystems;

b) A variable looking at the lengthof donor engagementonPFM issues in each country,measuredasthenumberofyearspassedsincethefirstWorldBankprojectinsupportofPFMreforms.TheunderlyinghypothesisinthiscaseisthatlongerengagementshouldbeassociatedwithbetterqualityofPFMsystems;

c) Dummy variables for the presence of an IMF programme and aWorld Bank PRSC, inorder tocapture thepresenceofPFM‐relatedconditionalities,hypothesisingapositiveimpactofPFM‐relatedconditionalitiesonthequalityofPFMsystems.

d) AnindexofoverallaidfragmentationcalculatedfromDACdata,withthehypothesisthathigherfragmentationimpactsnegativelyonthequalityofPFMsystems22;

OthervariablesApart from the two main variables of interest – ‘quality of PFM systems’ (dependentvariable)and‘donorsupportforPFMreforms’(independentvariable)–anumberofotherindependentvariableswereincludedintheanalysis.Theyrepresentotherfactorswhichcanbehypothesisedashavinganinfluenceonthelevelandchangeofthedependentvariable,includingthosewhichhavealsobeenidentifiedasrelevantfactorsbythepreviousempiricalanalysessummarisedabove.Foreaseofreference,thesevariablesaresummarisedinTable

21Itcouldbearguedthatpercapitafigures,ormoregenerallyfiguresadjustedforcountrysize,provideabetterindicationofdonorPFMinvestments.Weneverthelessthinkthatitmakesmoresensetofocusonoverallfiguresatthispoint,andcontrolforcountrysizeintheanalysis.22SeeKnackandRahman(2007)bothtoseehowtheindexiscalculatedandforthemainargumentsbehindthehypothesis.

20

2 below, distinguishing between economic/social and political/institutional variables, andspecifyingthehypothesisedimpactonthequalityofPFMsystems.Table2.OtherindependentvariablesincludedinthePEFAlarge‐Nanalysis

Variable Effecta Detailedhypothesis/ReferencesEconomic/socialvariablesGDPpercapita + Richercountrieshavebettergovernmentcapacityandabetter‐

educatedpopulationwhichwillholdgovernmentaccountableforhowitmanagespublicfinances.SeedeRenzio(2009a)

RecentGDPgrowth + EconomicperformanceandincreasingresourcesallowforincreasingfiscalandreformspacenecessaryforPFMreforms.SeeAndrews(2010)

Aiddependency ‐ Aiddependencyworsensgovernancestandardsanddistortsincentivesforreform.SeeKnack(2002)andBrautigamandKnack(2004)

Resourcedependency ‐ The‘resourcecurse’hypothesisclaimsthatdependenceonnaturalresourcerevenuesunderminesPFMandothergovernancereformprospects.SeeAndrews(2010)anddeRenzioetal.(2009).

Population ? TherecouldbeeconomiesofscaleininvestinginPFMsystemsinlargercountries;otherwisepublicfinancesmightbeeasiertomanageinsmallercountries.SeedeRenzio(2009a)

Adultliteracy + Incountrieswithabettereducatedpopulation,governmentwillhavebettercapacity,andcitizenswillholdgovernmentaccountable.SeeKaufmannetal.(1999)23

Tradeopenness + Openeconomiesneedtobebettermanaged,andaremoreexposedtoexternalpressureforbettereconomicmanagement.SeeTreisman(2000)andBusseetal.(2007)

Technologicaldiffusion + Electricpowerconsumption,telephonesubscriptionsanduseofpersonalcomputersandtheinternetallmakeiteasierforgovernmentstoadoptPFMreformsandforcitizenstoholdgovernmentaccountable.SeeKakabadseetal.(2003)

Political/institutionalvariablesAdministrativeheritage ? Inheritedbudgetsystemsaffectpresent‐dayPFMquality.See

Andrews(2010)andLaPortaetal.(1999)Democracy + Indemocraticregimes,citizenswillholdgovernmentaccountable

anddemandbettermanagementofpublicresources.SeeWehneranddeRenzio(2010)andRivera‐Batiz(2002)

Statefragility/conflict ‐ Countriesinfragileorinconflict/post‐conflictsituationshavemoredifficultiesincarryingoutPFMreforms.Capacityisveryweak,informalityispredominant,andpoliticalwillislacking.SeeAndrews(2010)

Regionaldummies ? Regionaldummiescaptureothercountrycharacteristics,suchasgeographyandculture,thatmayaffectPFMreforms.

a)Expectedsignoftheregressioncoefficient.Questionmarksdenotecaseswherethisisnotclear,orwherethevariableiscategorical.

Whilemeasuringmostof thevariables included in the tableabove isnotproblematic, for‘democracy’ and ‘state fragility’ measurement is less straightforward. A number ofalternativemeasuresofdemocracyexist,anddebatearoundtheirusefulness isverymuch

23Kaufmannetal.(1999)notethecorrelationbetweenadultliteracyandgovernance,butarguethatthecausallinkgoesfromthelattertotheformer.

21

alive24.WesettledforoneofthemostcommonlyutilisedmeasuresbyFreedomHouse,butalso checked whether our results changed when other indicators were used instead. Forstate fragility, we decided to use a dummy variable recording the presence of UNpeacekeepersinthecountrysince1995,thereforeequatingstatefragilitywithaconflictorpost‐conflictsituation25.FurtherdetailsonmeasuresutilisedandsummarystatisticsforallvariablesarereportedinAppendix 6. We were unable to include in our analysis a number of other potentiallyrelevant variables, related for example to levels of corruption and government tax effort.Thiswasduetothefactthatdatasources(e.g.theInternationalCountryRiskGroup)didnotcoverasufficientnumberofcountries.Resultsoftheanalysis:PEFAlarge‐NsampleTheaveragePEFAscoresthatwereusedasthevariabletobeexplainedcanbeconsideredtorepresentameasureoftheoverallqualityofPFMsystems.Assumingthatthepresent‐dayqualityof such systemsaccurately reflects the resultsof past reformefforts,what canbesaidabout the factors thatdeterminedsuccessful reformeffortsacrossoursampleof100countries?Andmorespecifically,whatwastheimpactofdonorassistanceinsupportofsuchefforts? The main objective of the PEFA large‐N analysis was to provide some tentativeanswers to these questions. Most PEFA assessments from which PFM quality scores arederivedtookplacein2008‐9(seeAppendix1).Forourexplanatoryvariables,weuseddatathat cover the five years between 2002 and 2006, in order to assess the impact of pastconditionsandactivitiesonpresentPFMsystemsanddue tosomeof thedata limitationsdescribedintheprevioussection.Given the mostly cross‐sectional nature of our data (meaning comparing data acrosscountries, and not over time), the standard econometric method to be used is OrdinaryLeastSquares(OLS)regression.Inthissection,wereportontheanalysisthatwascarriedoutand its results, including a number of robustness checks to verify their strength against anumber of alternative models and measures, and highlighting their limitations and thecautionneededininterpretingthem.Asafirststep,welookedatsomebivariatecorrelations,toseetheextenttowhichourdataconfirmed both previous findings from the literature and some of our preliminaryhypotheses. Figure 4 below presents scatter‐plots of average PEFA scores against incomelevels,incomegrowth,aiddependencyanddonorPFMsupportovertheperiod2002‐2006.Overour 100‐country sample, data showpositive correlationswith income levels, incomegrowthandPFMaid,andanegativecorrelationwithoverallaiddependency,asexpected,despitelargevariance.24See,forexample,Munck(2009)andCheibubetal.(2010).25Otherdefinitionsoffragility,suchasthoseusedbytheWorldBankorinthe‘StateFragilityIndex’producedbythePolityIVProject,inourviewaretoobroadtocapturethekeyelementsoffragilityweareinterestedin.

22

Figure4.AveragePEFAScoresandkeyexplanatoryfactors(bivariatescatter‐plots)

Wethenturnedourattentiontomultivariateanalysis,tounderstandhowtheseandothervariables identified jointly affected PFM quality. We first ran a series of exploratoryregressions using sub‐groups of variables to establish which economic, political and aid‐relatedvariableswere significantly associatedwith changes in thequalityofPFMsystemsandshouldbekeptinsubsequentstagesoftheanalysis.TheresultsareshowninTable3.Table3.PreliminaryregressionsbygroupsofvariablesDependentVariablePEFAAvg.Score

Economic/socialvariables

Political/institutionalvariables

Aid‐relatedvariables

GDPpercapita(log) 0.32*** RecentGDPgrowth 0.04*** Population(log) 0.08** Resourcedependency ‐0.37*** Adultliteracy 0.001 Tradeopenness ‐0.002** Technologicaldiffusion Electricity

Computers0.000.00

Observations 94 R‐squared 0.57 Admin.heritage Britishcolony

Frenchcolony ‐0.15

‐0.26*

Democracy ‐0.04 Statefragility ‐0.38*** Regionaldummies SSA

LACECA

‐0.070.090.31**

Observations 100 R‐squared 0.33

23

Aiddependency ‐0.02***PFMsupport 0.24***Aidfragmentation 0.17GBSas%oftotalODA 0.24IMFprogramme 0.02WBPRSC 0.14Lengthofdonoreng. ‐0.003Observations 97R‐squared 0.25Notes:OrdinaryLeastSquaresregressions.Onlycoefficientsarereported.*Significantat10%;**significantat5%;***significantat1%.Amongtechnologicaldiffusionvariables,wedroppedtelephonesubscriptionsandinternetusersastheywerehighlycorrelatedwiththeothers.

Economicandsocialvariablesaretheonesthatexplainalargerpartofthevariationinthequality of PFM systems (shown by a higher R‐squared value of 0.57), and most of theircoefficientshavetheexpectedsign.HigherlevelsofGDPpercapitaandofrecenteconomicgrowtharepositivelyandsignificantlyassociatedwithbetterPFMsystems,asisthesizeofthe country’s population, whereas dependency on natural resource revenues and tradeopennesshavenegativecoefficients.Whilethisresultconfirmsthestandard‘resourcecurse’hypothesis, that linksresource‐dependentcountrieswithworsegovernancestandards,thesign for the trade openness variable is not in the expected direction. At this preliminarystage,however,itisnotworthadvancingfurtherexplanations,asresultsmightbedrivenbyomitted variables. Among political and institutional variables, the ones with a significantcoefficient(apartfromtheEasternEuropeanregionaldummy)arethedummiesforformerFrenchcoloniesandforconflict‐affectedstates,bothwithnegativecoefficientsthatconfirmthe findings of previous research. The level of democracy also comes close to standardsignificance levels, alsowith the expected sign26. Aid‐related variables, in this preliminaryphaseoftheanalysis,onlyexplainaquarterofthedifferencesinthequalityofPFMsystems(R‐squared=0.25). Results highlight the negative and significant association between theoveralllevelofaiddependencyandPFMqualityand,interestingly,apositiveandsignificantone of donor PFM support. Another variable that comes very close to standard levels ofstatistical significance, namely the presence of a PRSC, also has the expected positivecoefficient.Inthefollowingstepoftheanalysis,webuiltthemainmodeltobeusedforouranalysisbybringing together the variables that were shown to be significant in the exploratoryregressions,plusafewothersofspecificimportance,suchassomeofthekeyaidvariables,regional dummies (to control for various regional characteristics), and the level ofdemocracy,coveringtheperiodfrom2002to2006.Theresultsareshown inTable4, firstusing standard OLS regression, but also using Weighted Least Squares (WLS) to addressissuesofheteroskedasticity,whichpersistedevenwhenweusedtherobuststandarderrorsoption.Some of the key results from the preliminary regressions also hold in the comprehensivemodel.Onceagain,higherlevelsofincome,incomegrowthandpopulationaresignificantlyassociatedwith better quality PFM systems. Countries that are richer, larger, and have a26ThenegativesignisduetothefactthatFreedomHouseclassifiesstrongerdemocracieswithlowervaluesofthevariable.

24

good economic growth record in recent years are alsomore successful at reforming andimprovingtheirPFMsystems.Morespecifically,countrieswithdoubletheincomepercapitacanbeexpectedtohaveanaveragePEFAscorewhichisalmosthalfapointhigher,holdingother factors constant. Similarly, countries which are poorer and with a worse recentperformance in terms of economic growth, will also be characterised by PFM systems oflower quality. Donor support for PFM reforms retains a positive (although slightly lesssignificant) coefficient. Its low value, however, indicates that very large injections oftechnical assistancewould be coupledwith only a small increase in PEFA average scores.Accordingtotheanalysis,anadditionalUS$40‐50mperyearinPFMassistance(thatis,morethan doubling the amount received by the country that has received themost in recentyears, Afghanistan)would be associatedwith just a half‐point change in the overall PEFAaverage score, holding other factors constant. Finally, the presence of peacekeepers as aproxy for state fragility maintains its significant negative association with PFM quality,highlightingtheimportanceofpoliticalstabilityforPFMimprovements.Table4.ThedeterminantsofthequalityofPFMsystemsDependentVariablePEFAAvg.Score

OLS

WLS

GDPpercapita(log) 0.44*** 0.45***RecentGDPgrowth 0.02** 0.02*Population(log) 0.08** 0.08***Resourcedependency ‐0.19 ‐0.19Tradeopenness ‐0.001 ‐0.001Frenchcolony ‐0.09 ‐0.09Democracy ‐0.02 ‐0.01Statefragility ‐0.22** ‐0.21*Aiddependency ‐0.01 0.01PFMsupport 0.01** 0.01*GBSas%oftotalODA 0.76 0.71Lengthofdonorengagement 0.01 0.01SSAdummy 0.11 0.10LACdummy 0.02 0.009ECAdummy 0.17 0.15Observations 93 93R‐squared 0.68 0.68Notes:Onlycoefficientsarereported.*Significantat10%;**significantat5%;***significantat1%.These initial results shed some light on the factors that drive differences in cross‐countryPFMperformance. For the purposes of the overall evaluation, the positive and significantrelationship with PFM‐related technical assistance is particularly encouraging, though itclearlycannotbeinterpretedasacausal.Itcouldmerelyreflectthefactthatdonorstendtoprovide more PFM‐related assistance to countries that have already achieved a certainsuccessinimprovingthequalityoftheirPFMsystems.Theissueofendogeneityalsoneedsto be taken into account in interpreting the relationship between income levels and PFMquality.Wecomebacktothisproblemfurtherbelow.Inorder tobetterunderstand thenatureof thepossible impactofdonorPFMsupportonaveragePEFAscores,weintroducedsomeinteractiontermsinthebasicmodel.Thesewereaimedat testingwhether donor PFM supportwas in someway linked to thepresenceof

25

higher levels of programme aid or of conditionalities linked to IMF or World Bankprogrammes.Noneofthese interactionterms,however,reachedconventionalsignificancelevels.WethenusedthesamemodeltoexplainvariationnotintheoverallPEFAaveragescore,butineachoneofthesixPFMclusterscoreslinkedtothedifferentphasesofthebudgetcycle.While there inevitably are differences in the results (some of which aremore difficult tointerpret), the overall trends did not change dramatically. Some results are worthhighlighting: (a) donor PFM support retains a positive and significant association withaveragePEFAscores inmostof thesixPFMclusters; (b)coefficients for income levelsandgrowthmaintain their positive sign and significance inmost clusters, except for ‘strategicbudgeting’; (c) in ‘strategic budgeting’, GBS as a share of total aid becomes significant,possibly indicating thatsuccessfulmedium‐termbudgetprojectionsaremoresuccessful inthosecountrieswheredonorschannelmoreoftheiraidthroughgovernmentchannels27;(d)‘external accountability’ scores in former French colonies and in Latin American countries(whichareallformerSpanishorPortuguesecolonies)aresignificantlylower28.Aseriesofrobustnesscheckswerethencarriedoutinordertoaddressanumberofissuesrelated to thedataand thenatureofourvariables,and tocheck forconsistencywith theresultsweobtainedinourmainmodel.First,welookedatthepossibilityofmeasurementerrorsbiasingourresultsbyrepeatingtheanalysisusingalternativemeasuresforanumberofourvariables,suchasdemocracy,statefragility,and theuseofaidmodalities. Fordemocracy,weusedbothFreedomHouseandPolityIVassourcesofdatatoassesswhetherthestrengthofdemocraticinstitutionshadanyeffectonPFMquality.Similarly,welookedatdifferentexistingindicesofstatefragility,andexpanded our definition of programme aid to include both general budget support andsectorprogrammes.Inallcases,ourresultswerenotaffected.Interestingly,however,whenwesubstitutedthePFMsupportvariablethatweconstructedfromdonordatawithsimilardatafromtheDAC/CRSdatabase,thestatisticalsignificanceofthecoefficientvanished.Thisisnotsurprising,giventhe lowcorrelationbetweenthetwovariablesreportedabove.WeinterpretthisasafurtherconfirmationoftheworsequalityoftheinformationcontainedintheCRSdatabase.Second, we ran an identical set of regressions using data for our explanatory variablesaveraged over the period 2004‐08 rather than 2002‐06, to testwhether our resultsweredependentonthetimeperiodconsidered.Again,theoverallstoryholds,withtheadditionalfinding that over this more recent period the share of GBS in total aid flows is alsosignificantlyandpositivelyassociatedwiththequalityofPFMsystems.Ratherthanprovide

27Itshouldalsobenoted,however,thattheR‐squaredstatisticforthe‘strategicbudgeting’regressionisthelowestofthesixclusters(0.29),indicatingthatothervariablesmightbetterexplainvariationsinthisPFMcluster.28Thisresultcouldbeinterpretedintwodifferentways:eitherexternalauditisreallylesseffectiveincountriescharacterisedbyacivillaw(asopposedtoacommonlaw)legalbackground,orthePEFAmethodologyisnotdesignedtoadequatelyassesstheexternalauditsystemsthatareprevalentincivillawcountries,havingbeenconceivedwithacommonlaw,Westminster‐stylegovernmentinmind.

26

evidence in support of the positive impact of budget support, however, this mightmoresimplyreflecttheincreasingimportanceofGBSasanaidmodalityinrecentyears,andthefactthatdonoragenciesrewardcountrieswithbetterPFMsystemsbyshiftingmoreoftheiraidtodirectlysupportthegovernmentbudget.Third, theconversionofPEFAscores from letter tonumericalscoresandtheirsubsequentaveraging means that we treated a variable that is ordinal in nature as if it were acontinuous variable. Using OLS with an ordinal dependent variable is known to lead tobiasedresults.Oneofthestatisticaltechniquesthatcorrectforthisproblemisorderedlogitregression. For this purpose, we used individual PEFA dimension scores, rather thanaverages, as our dependent variable (as in Andrews 2010). This allowed to also assesswhether countries scored higher on de jure rather than de facto, upstream rather thandownstream, and concentrated rather than deconcentrated PFM processes, testing thefindings reported in Andrews (2010) for a larger sample of countries. These differencesprovetobesignificant,confirmingtheresultsofpastresearchthatcountriesmakebudgetsbetter thantheyexecutethem,pass lawsbetter thanthey implement them,andprogressfurtherwhen responsibility for reforms lieswithin core groups in theMinistry of Finance.Moreover, even after controlling for these differences, the results linked to our othervariables still hold. The positive and significant coefficients for income levels and incomegrowthremain,pushingcountriesupanotchintheirordinalPEFAdimensionscores.HigherdonorPFMsupportisassociatedwithhigheraveragePEFAscoresinthiscase,too,whiletheGBS share of total aid seems to be associated with positive changes in PFM quality forcountrieswithlowerPEFAdimensionscores.Fourth,wewantedtocheckthatourresultswererobusttosamplechanges,andnotdrivenbyspecificoutliersorspecificgroupsofcountriesacrossthevariousvariablesincluded.Wetherefore repeated our analysis with the following changes: (a) excluding very smallcountries (withpopulationsof less than1m),as theseoftenskewresults; (b)excluding, inturn, middle‐income countries and low‐income countries; (c) excluding countries withparticularlyhighvaluesofdonorPFMsupport(inexcessofUS$15mperyearovertheperiod2002‐06);(d)excludingcountrieswherePEFAassessmentswerestillindraftform,toensurethatthelowerqualityofsuchassessmentswasnotundulyaffectingresults.ResultsforthemainvariablesofinterestarereportedinTable5below.Table5.Partialregressionresults(robustnesscheckswithsamplechanges)DependentVariablePEFAAvg.Score

Excl.smallcountries

Excl.MICs

Excl.LICs

Excl.PFMoutliers

Excl.draftPEFAs

GDPpercapita(log) 0.42*** 0.35*** 0.50*** 0.43*** 0.33***RecentGDPgrowth 0.03** 0.02* 0.03** 0.02** 0.02Population(log) 0.03 0.07 0.03 0.08** 0.06*Statefragility ‐0.24** ‐0.27** ‐0.22 ‐0.23** ‐0.21*PFMsupport 0.01** 0.01** 0.01 0.01 0.01*GBSas%oftotalODA 0.71 1.21* 0.71 1.03* 0.94Observations 75 70 58 89 71R‐squared 0.73 0.59 0.66 0.68 0.73Notes:OrdinaryLeastSquaresregressionswithrobuststandarderrors.Onlycoefficientsarereported.AllregressionsincludeallothervariablesfrommainmodelinTable4.*Significantat10%;**significantat5%;***significantat1%.

27

Oncemore,resultsshowedtobereasonablyrobusttomanyofthesechanges.MostofthevariablesthatwerefoundtobesignificantlyassociatedwithchangesinthequalityofPFMsystemsmaintainthedirectionandthesignificanceoftheircoefficient.Someresultsdeservefurthercomments,however.First,thepositiveassociationbetweenPFMqualityanddonorPFM support seems to be driven by the small number of countries that receive largeamountsofPFMassistance.WhenthefivecountriesreceivingmorethanUS£15mperyear(Afghanistan, Morocco, Indonesia, Tanzania and Mozambique) are excluded from theanalysis, the PFM support variable loses significance. Second,whenonly poorer countriesare considered, income levels become less important in explaining differences in PFMquality, while increasing shares of GBS in total aid become positively and significantlyassociatedwithbetterPFMquality.Giventhattheanalysisappliestoamorehomogeneousgroupofcountries,theresultforincomelevelscanbeexpected.TheresultforGBSismoreinteresting, as it indicates that in low‐income countries it is not just PFM support, amongdonor interventions, that is significantly associated with changes in the quality of PFMsystems. Aid modalities also count. Once again, however, endogeneity issues limit theconclusions that canbedrawn from this interesting finding. Third, state fragility andPFMsupport are significant factors among poorer countries. When low‐income countries areexcludedfromtheanalysis,bothvariableslosesignificance.Finally,weneededtoaddresstheveryimportantissueofreversecausationforsomeofthekey explanatory variables that were found to be significant. As indicated above, therelationship between both income levels and donor PFM assistance on one side, and thequalityofPFMsystemsontheother(whichissignificant,positiveandrobusttoanumberofspecificationchanges)cannotbeinterpretedascausal.Whileitcouldbethathigherlevelsofincome and donor PFM support bring about improvements in PFM systems, the oppositecouldbejustastrue.Inotherwords,betterqualityPFMsystemsmightexplain,ratherthanbeexplainedby,higherGDPpercapitaandhigherlevelsofdonorsupportforPFMreforms.MorespecificallyforPFM‐relatedaid,whatourresultsmightbecapturingisthetendencyofdonoragenciestoinvestmoremoneyincountriesthathavealreadyshownthecapacityandwillingnesstoreformtheirPFMsystems,andthathavegonepastsimplerinitialreformsandarestartingtoengagewithmorecomplex(andmoreexpensive)reforms.Suchendogeneityissuesareverydifficulttoaddresswiththekindofdatathatwehadatourdisposal,whichhavenotime‐seriesdimensionthatcanshedlightonhowPFMsystemshaveevolvedwithineachcountry.Nevertheless,welookedintothepossibleuseofthestatisticaltechniquemostusedtoaddressreversecausation,whichreliesontheidentificationofso‐called ‘instrumental variables’. These are variables which are highly correlated with theendogenousexplanatoryvariablebutdonotsufferfromthesameproblemswithrelationtoreversecausation.Inourcase,thechallengewastofindadequateinstrumentalvariablesforbothincomelevelsanddonorPFMsupport.In the case of income,we found that 1995 levels of GDP per capita and infantmortalityexplain most of the differences in the more recent income levels used in our analysis,withoutbeingcorrelatedwiththequalityofPFMsystemsin2008.Usingthesetwovariablesasinstruments,wefindthatatwo‐stageleastsquaresproceduregeneratesresultsthatare

28

very similar to theOLS ones,whichmeans thatwe canmore comfortably claim that it ishigherincomelevelsthatbringaboutbetterPFMquality,andnottheotherwayaround.When it came to disentangling the endogeneity of donor PFM support, unfortunatelywecould not find any suitable instrumental variable that could help us determine in whichdirectionthecausationreallyworks.Forexample,wetriedusing:(a)administrativeheritagedummies,astheymayreflectnotonlygeneraldonorinterest,butmorespecificinterestinsupportinginstitutionalreforms(asmanyinstitutionshavecolonialorigins);(b)indicatorsofthe past quality of institutions, as this could have shaped donors' intentions to supportinstitutional developmentwithout being highly correlatedwith PFMquality today; (c) thedegreeofeconomicandpoliticalstabilityoverthepast10‐15years;(d)dummyvariablesfortheexistenceofaPRSPorforHIPCstatus,asmuchoftheoriginaldecisiontoallowcountriesto enter HIPC were linkedmore tomacro‐stability and indebtedness rather than to PFMquality; (e) other variables linked to the length and depth of donor engagement withbroader public sector governance. None of these variables, however, could explain,individuallyorjointly,recentlevelsofdonorPFMsupport.Whilethismaybeareflectionofthecomplexinter‐relationshipsthatexist inrealitybetweendonorandgovernmenteffortsat reforming country institutions, itmayalsobedue to thebiaseswenotedabove in thedata that were collected for the donor PFM support variable. The nature of donor PFMinterventions, moreover, does not lend itself easily to statistical analysis that treats allcountriesasequalcases.Country idiosyncrasiesmightplayan importantrole inexplainingthe difficulties we faced in interpreting the results of the analysis. In some cases, forexample, large donor PFM supportmight be due to an expensive computerized financialmanagementsystemthat isatalladequatetothecontext,andthereforefails togenerateimprovements in theoverall qualityof PFMsystems. Inother cases, smaller interventionsmaybemuchmorecontextuallyrelevant,leadingtomoreimportantimprovements.A final set of regressions focused more specifically on how different aspects of donorengagement inPFMreformprocessesaffecteddifferentPFMprocess types,asdefinedbyAndrews(2010). Inotherwords,wewantedtocheck iftherewereanysignificant linkagesbetween variables such as donor PFM support and share of programme aid, and certainaspectsofPFMquality,i.e.thosemoredirectlyrelatedtoupstreamratherthandownstreamprocesses,tode jureratherthandefactoreforms,andtoPFMprocessescharacterisedbyactor concentration rather than deconcentrated responsibility. We then calculated PEFAaverage scores for the indicator dimensions belonging to each of the types of PFMprocesses,andusedthemasdependentvariables.Table6reportstheresults.WhileGBS as a shareof total aid doesnot bear anydirect relationshipwith anyof thesespecific types of PFM processes, some interesting patterns can be seen for donor PFMsupportandthelengthofdonorengagement.First,alongerperiodofdonorengagementissignificantly andpositively associatedwithupstream,de jureand concentratedprocesses.Theseresultsmayberelatedtothefactthatdonorshavehistoricallypaidmoreattentiontothesesimplerreformareas,ratherthangettinginvolvedinthemessier,andmoredifficult,challengesofimplementingPFMreformsthataddressproblemsingovernment‐widebudgetexecution.This,inturn,couldexplaintheverysmallimpactofPFMsupportonoverallPFMquality.Theycouldalsostemfromthefactthatdownstream,defactoanddeconcentratedprocesses take longer to develop and improve, with donor engagement so far not being

29

sufficientlylongandsustainedtobeassociatedwithsubstantiveimprovements.Second,thelevelofdonorPFMsupporthasamoresignificantandstrongerassociation(thecoefficientdoublesinsize)withscoresfordejureandconcentratedPFMprocesses,againhighlightinghowdonorPFMsupportseemstofocusmoreonrules,proceduresandspecificactorswithingovernment. The results are somewhat reversed when it comes to upstream vs.downstreamprocesses.Here, the association is strongerwithdownstreamprocesses. Thebestexplanationwecanoffer isthatthiscouldberelatedtothelargeamountsoffundingdevotedtoIFMISimplementation,atypicaldownstreamPFMreform.Table6.Partialregressionresults(PEFAscoresbytypeofPFMprocess)DependentVariablePEFAAvg.ScoresbyPFMprocess

Up‐stream

Down‐stream

Dejure

Defacto

Conc.actors

Deconc.actors

PFMsupport 0.01 0.02*** 0.02** 0.01** 0.02*** 0.01*GBSas%oftotalODA 1.42 0.38 0.77 0.50 0.46 0.73Lengthofdonorengagement 0.02** 0.007 0.02*** 0.002 0.02*** 0.002Observations 93 93 93 93 93 93R‐squared 0.52 0.70 0.64 0.69 0.65 0.67Notes:OrdinaryLeastSquaresregressionswithrobuststandarderrors.Onlycoefficientsarereported.AllregressionsincludeallothervariablesfrommainmodelinTable4.*Significantat10%;**significantat5%;***significantat1%.

In summary, the PEFA large‐N analysis considered a numberof economic/social, political/institutional, andaid‐relatedvariables that couldexplaindifferences in thequalityofPFMsystems across countries. It found that income levels, economic performance (measuredthroughaveragegrowthrates),populationsizeanddonorPFMsupportaresignificantlyandpositively correlatedwith better PFM systems,while state fragility negatively affects PFMquality. These results are robust to a series of tests and changes in econometricmodels.Moreover,whentheanalysisfocusesonmorerecentyearsoronlow‐incomecountriesonly,theshareofGBSintotalaidalsobecomessignificantinexplainingdifferencesinthequalityofPFMsystems,withapositivecoefficient.Finally,donorsupport,bothintermsoffinancialresourcesandof lengthofengagement, seems toaffectdifferent typesofPFMprocessesdifferently.Oneofthemoreinterestingfindingsforthepurposesoftheevaluationthatthisstudyispartof,thatofthepositivecorrelationbetweendonorPFMaidandqualityofPFMsystems in recipient countries, suffers from serious endogeneity issues that wewere notable to resolve, and that will need to be addressed by utilising other approaches andtechniques,includingqualitativecasestudiesthatcanclarifyandunpacktheexistingcausallinkagesthrougha‘processtracing’methodology(GeorgeandBennett2005).Resultsoftheanalysis:HIPCmedium‐NsampleA proper assessment of the effectiveness and impact of donor support to PFM reformswouldrequiredatathatcapturechangesinthequalityofPFMsystemsovertime.OneofthemainlimitationsoftheanalysiscarriedoutsofarbasedonPEFAdataonlyisthatitcomparesacross countries at a certain point in time, and therefore on the assumption that betterqualityPFMsystemstodayreflectsuccessfulpastPFMreforms.Moreover,itsuffersfromaseries of statistical problems that limit the scope of the analysis and the validity of theresults. We therefore complement the PEFA large‐N analysis with a non‐econometric

30

analysis of 19 African HIPC countries for which time‐series data is available, thoughwithmore limited coverage of PFM systems (see Appendix 4), and somewhat less reliableindicators.Nevertheless,table1abovepresentedsummaryscoresfortheperiod2001‐2010,basedoninformationdrawnfrombothHIPCandPEFAassessments.Table7belowlinkschanges inthequalityofPFMsystemstoasetofpossibleexplanatoryvariables. It ranks countries according to whether they saw a clear improvement in thequalityoftheirPFMsystemsovertheperiodconsidered(+),manifestedanuncleartrend(=)or experienced a worsening level of PFM quality. It also shows the overall difference insummary scores between 2001 and 2010. Data for various explanatory variables arepresented alongside HIPC/PEFA scores to partly replicate themodel utilised for the PEFAlarge‐Nanalysis,thistimecoveringalongerperiodoftime,from1995to200829.Table7.ThedeterminantsofthequalityofPFMsystemsin19AfricanHIPCs,2001‐2010

Country 2001‐2010Variation

AvgGDPpc

AvgGrowth

Admin.Herit.

AvgDemocracy

AvgAidDep.

TotalPFMaid

AvgGBS/Totalaid

US$PPP % FHindex % US$p/c %Zambia + +8 1005 3.8 GBR 4.0 21.4 0.71 9.4BurkinaFaso + +4 878 6.0 FRA 4.2 14.4 0.06 15.0Ethiopia + +4 548 6.8 N/A 4.8 12.4 0.04 4.3Cameroon + +1 1741 4.0 FRA 6.1 5.7 0.04 3.1Mali + +1 856 5.5 FRA 2.4 15.4 0.06 10.0

Average 1006 5.2 4.3 13.9 0.18 8.4Ghana = +8 1039 5.0 GBR 2.4 10.6 0.18 11.5Mozambique = +2 548 7.6 POR 3.4 28.7 0.47 15.0Niger = +1 550 4.1 FRA 4.2 14.3 0.05 9.9Guinea = 0 940 3.7 FRA 5.5 7.7 0.12 0.3Tanzania = 0 879 5.6 GBR 4.0 13.8 0.32 17.5Madagascar = ‐1 826 3.7 FRA 3.1 13.5 0.09 6.8Malawi = ‐1 636 4.9 GBR 3.3 22.1 0.63 7.8Senegal = ‐1 1399 4.3 FRA 3.3 8.2 0.07 2.6Chad = ‐2 1049 7.2 FRA 5.6 11.0 0.18 5.0SãoTomé = ‐2 1383 6.8 POR 1.7 23.4 1.35 0.9

Average 925 5.3 3.6 15.3 0.35 7.7Rwanda ‐ ‐2 686 10.4 BEL 6.0 23.6 0.42 16.4Uganda ‐ ‐2 784 7.4 GBR 4.6 13.4 0.25 9.3Gambia ‐ ‐4 1033 4.6 GBR 5.2 13.0 0.43 1.7Benin ‐ ‐5 1188 4.6 FRA 2.1 10.0 0.17 8.7

Average 923 6.7 4.5 15.0 0.32 9.0Note:Allvariablesreferto1995‐2008,exceptforGBS/Totalaid(2002‐06).

What the results show is that variables that are significantly correlated with higher PFMquality (measuredthroughPEFAscores)donotseemtoexplainmuchofthechangesovertimeinPFMqualitymeasuredthroughtheHIPC/PEFAindicators.Countriesthatsawaclearimprovement do have a higher average income level and lower aid dependence thancountries that either worsened or showed an unclear trend, but they are predominantlyformer French colonies, have a lower average growth rate and did not receive higheramounts of either PFM support or of general budget support. If we consider only the

29Resourcedependencywasnotincludedasonlyonecountry(Cameroon)isconsideredresource‐dependent.RegionaldummieswerenotrelevantasallcountriesincludedareinSub‐SaharanAfrica.DonorPFMsupportispresentedastheaverageofyearlypercapitaPFMsupportovertheperiod1995‐2008,totakecountrysizeintoaccount.

31

difference in score between 2001 and 2010, rather than the scores in the variousassessments, thepictureagainchanges. In thiscase,countrieswhose2010score ishigherthan the 2001 did receive higher amounts of general budget support, but have loweraverageincomes,economicgrowthratesandpercapitaPFMsupport.Moreover,manyofthesedifferencesacrosscountrygroupsarenot likelytobesignificant,given the high levels of within‐group variance. The better‐performing group includescountrieswith income levels that rangefromUS$1741(Cameroon) toUS$548(Ethiopia),andwithlevelsofpercapitadonorPFMsupportthatvariesbetweenUS$0.71(Zambia)andUS$ 0.04 (Ethiopia and Cameroon) a year. Similarly, the group of countries who saw adeclineinthequalityoftheirPFMsystemsincludesbothRwandaandGambia,whoreceivedoneofthehighestandoneofthelowestlevelsofgeneralbudgetsupportasashareoftotalaid,andUgandaandBenin,whichscoreverydifferentlywithregardtothestrengthoftheirdemocraticinstitutions.Inotherwords,whattheHIPCmedium‐Nanalysisshows is that theresults fromthePEFAlarge‐N analysis need to be taken with some caution. Some of the differences that existbetween present PFM quality and reform history need to be taken into account wheninterpretingresults,eventhoughthemuchsmallersampleclearlydoesnotpermittocarryout a similarly detailed analysis. Similarly, the lack of more detailed information on PFMsupportindonorresponsestoourdatarequestdidnotpermitamorein‐depthanalysisofhow specific PFM‐related activities funded by donors may have influences PFM reformoutcomes, either in general or in specific PFM areas. The best that the HIPC medium‐Nanalysiscandogiventheexistingconstraints,asweargueintheconclusions,istoguidethechoiceofcasestudycountriesthatcanprovidemoredetailedcomparativeinsightsintothedynamicsofPFMreforms,inordertobetterexplaintheirsuccessesandfailures.ConclusionsandimplicationsforoverallevaluationThisstudybroughttogetheravailablequantitativeevidenceonthequalityofPFMsystemsacrossalargenumberofcountries,withtheaimofassessingthefactorsthatareassociatedwithandmayhavedeterminedcross‐countrydifferencesandvariationsovertime.InordertooperationaliseandmeasurethequalityofPFMsystems,wedrewfromtwomainsources:a cross‐country dataset of PEFA assessments carried out worldwide in more than 100countries, and a more limited panel dataset based on a combination of HIPC and PEFAindicators,covering19Africancountriesovertheperiod2001to2010.AspecificaimofthestudywastoinvestigatetheeffectivenessandimpactofdonorPFMsupportonthequalityofPFMsystems in recipientcountries.For thispurpose,andgiventhescarcityand lackofreliability of existing data, we devoted substantial resources to the compilation of a newdatasetgatheringinformationondonoractivitiesinsupportofPFMreformsovertheperiod1995 to 2008. Finally, we included in the analysis a series of economic/social,political/institutional and aid‐related variables that were identified as having a potentialinfluenceonPFMqualityandreforms.ThefirstphaseoftheanalysisfocusedontheuseofeconometrictechniquestoidentifythekeyvariableswhichweresignificantlyassociatedwithchangesinthequalityofPFMsystems.

32

Results largely confirmed some of the findings of previous research. Among the differentsetsoffactorsconsidered,economicandsocialonesexplainmorethanhalfoftheexistingdifferencesinthequalityofPFMsystems,whileaid‐relatedfactorsexplainonlyaquarterofthevariation.Countrieswithhigherincomelevels,goodrecenteconomicperformance,andlarger populations have better average PEFA scores. State fragility, defined as countriesbeing in a conflict or post‐conflict situation and measured through the presence ofpeacekeepingforces,isnegativelyassociatedwiththequalityofPFMsystems.More interestingly, the analysis found a significant and positive association, albeit with averysmallcoefficient,betweendonorPFMsupportandaveragePEFAscores.Ifmorerecentdataareused,orwhentheanalysisfocusesonlow‐incomecountriesonly,theshareoftotalaidprovidedasgeneralbudgetsupportisalsopositivelyassociatedwithbetterPFMquality.Theseresultssurvivedanumberofrobustnesschecks.Moreover,iflookingatbothfinancialresources and length of engagement with PFM reforms, higher donor support is morestrongly associated with improvements in de jure and concentrated PFM processes.Substantive investments in IFMIS systems, however, seem to have had a positive (albeitlimited)impactonthequalityofPEFAdimensionsrelatedtobudgetexecution.TheimpactofdonorPFMsupportonPEFAscorescannotbeproven,however,giventhatthecross‐sectionalnatureofavailabledatadoesnotexcludereversecausation,orthepossibilitythatitcouldbecountrieswithhigherPEFAscoresthatreceivemorePFM‐relatedassistance,rather than theotherwayaround.Resultsmayalso reflectbiases in thedata themselves,givenlimitationsinthedatacollectionprocess.Inthissense,improvementsinPEFAscoresmayinfactbearlittlerelationwithdonorPFMsupport,andmayreflectinsteadisomorphictendencies, a hypothesis supported by the fact that de jure PEFA dimensions scoreconsistentlybetterthandefactoones.Variousattemptsweremadeatfindingadequateinstrumentalvariablesthatcoulduntanglethe direction of causation, but without success. The problem of endogeneity has beenwidelynotedanddiscussedintheliteraturelinkingforeignaidandinstitutions30.Acemoglu(2005) is generally pessimistic about the possibility of identifying causal relationships incomparativepoliticaleconomy,andarguesthatrobustnon‐causalrelationshipsneverthelessareofvaluetoadvancementsintheoreticalanalysisandpolicymaking.Inourcase,asarguedbelow,thepositiveandsignificantassociationbetweenPEFAscoresanddonorPFMsupportprovidesaninterestingbasisforthecountrycasestudiesthatwillcomplementtheanalysiscarriedoutinthispaper.Results fromthesecondphaseof theanalysis, lookingat changes inHIPC/PEFA indicatorsovertime,onlypartlysupportthefindingsmentionedabove,andhighlightthefactthatthecharacteristicsofAfricanHIPCcountriesthatweresuccessfulinimplementingPFMreformsvarysubstantially,evenacrossthefewvariablesthatwerefoundtobesignificantinthefirstphaseoftheanalysis.AmoredetailedlookattheinfluenceofspecifictypesofdonorPFMsupportonPFMsystemswasunfortunatelypreventedbyalackofsufficientdataprovidedbydonoragencies.

30SeeKnack(2001),Svensson(2000),Tavares(2003),AlesinaandWeder(2002),BrautigamandKnack(2004),RajanandSubramanian(2007),andEar(2007).

33

Infact,probablythebestwaytogainamoredetailedunderstandingofthefactorsaffectingPFMreforms,andtoaddressthedifficultyofconfirmingthedirectionofcausalitybetweendonor PFM support and the quality of PFM systems, is to complement the present studywith in‐depth qualitative case studies that can unearth the underlying dynamics of PFMreformsandtherolethatdonorsplayedinsupportingthem.Oneofthekeyissuesthatthisstudycanhelpaddress,therefore,isthechoiceofcountriesthat shouldbe the focusof thecase studyapproach.Both thePEFA large‐Nand theHIPCmedium‐Nanalysescanassistinthisregard.AssuggestedbyLieberman(2005),theresultsofalarge‐Nanalysiscaninformthechoiceofin‐depthcasestudiesthroughtheanalysisofregressionresiduals,or thedifferencesbetweentherealvaluesof thedependentvariable(averagePEFAscores)andthosegeneratedbytheregression.Casesshouldbechoseneitherwithin those with very small residuals, in the attempt to confirm the findings from theregression analysis, or within the group of countries with larger residuals, to open theinvestigation to possible alternative models and explanations (see Figure 5 for a plot ofresiduals).Countries thatbelong to the firstgroup includeAfghanistan,Cambodia,Ghana,Honduras, PapuaNewGuinea and Sierra Leone. Belize, India,Malawi, Tunisia and Yemenbelonginsteadtothelattergroup.Figure5.PEFAlarge‐Nregressionresiduals

FurthersuggestionscanbedrawnfromtheHIPCmedium‐Nanalysis.GiventhatoneofthemainpurposesoftheevaluationistoassesstheimpactofdonorPFMsupportonthequalityofPFMsystems, ideallycasestudiesshould includebothcountries thatweresuccessful inimplementingPFMreforms (and therefore improve theirPFMsystems)andcountries thatwerenot.AssuggestedindeRenzio(2009b),thisshouldbecoupledwithdifferentlevelsof

34

donor PFM support. For example, looking at the case of Zambia (and possibly alsoMozambique) could shed light on how substantive donor PFM support could have led toimproved PFM systems. Uganda andMalawi could be examples of less successful donorsupport,whileBurkinaFasoandCamerooncould shed lightonother factors shapingPFMreformoutcomes,giventheirsuccessdespitelowerdonorPFMsupport.In summary, evaluating the effectiveness of donor support to PFM reforms in developingcountries isstill largelyunfinishedbusiness.Whilethisstudyhasusedexistingquantitativedata to identify some preliminary trends and interesting associations, further work isneeded. More quantitative research will need to be carried out as new and better databecomeavailable (forexample,on thequalityofPFMsystems,ondonorsupport forPFMreforms,andongovernments’owneffortstostrengthenPFMsystems),tofurthertestsomeof the findings presented above. Qualitative case studies will be fundamental, too, tocomplementandaddressthemanyshortcomingsofquantitativeanalysis,mostnotablythedifficultiesinexplainingnotonlyifandwhendonorPFMsupporthashadanimpactonPFMsystems,butalsowhyandhowitdid,takingintoaccountthedifferencesincountrycontextinwhichPFMreformstakeplace.

35

ReferencesAcemoglu,D.(2005)‘Constitutions,Politics,andEconomics:AReviewEssayonPerssonand

Tabellini'sTheEconomicEffectsofConstitutions’.JournalofEconomicLiterature,43(4),1025‐1048.

Alesina,A.andB.Weder(2002)‘DoCorruptGovernmentsReceiveLessForeignAid?’American

EconomicReview,92(4),1126‐1137.Andrews,M. (2010)HowFarHavePublic FinancialManagementReformsCome inAfrica?RWP10‐

018FacultyResearchWorkingPaperSeries.Cambridge,MA:HarvardKennedySchool.Andrews,M.(2009)‘GoodGovernmentMeansDifferentThingsinDifferentCountries’,Governance,

23(1):7‐35.Andrews,M. (2007) “Whatwould an ideal PFM system look like?” inA. Shah (ed.)Budgeting and

BudgetaryInstitutions.Washington,DC:WorldBank.Arndt, C. (2008) ‘The Politics of Governance Ratings’, International Public Management Journal,

11(3),275‐297.Brautigam,D.andS.Knack(2004) ‘Foreignaid, institutions,andgovernance insub‐SaharanAfrica’.

Economicdevelopmentandculturalchange,52(2),255‐286.Busse, M. et al. (2007) Institutions, Governance and Trade. Hamburg: Hamburg Institute of

InternationalEconomics(HWWI).Cheibub, J.A., J. Gandhi and J.R. Vreeland (2010) ‘Democracy and Dictatorship Revisited’, Public

Choice,143:67‐101.de Renzio, P. (2009a) Taking Stock:What do PEFA Assessments tell us about PFM systems across

countries?ODIWorkingPaper302.London:OverseasDevelopmentInstitute.de Renzio, P. (2009b) Assessing Budget Institutions and Budget Reforms in Developing Countries:

Overviewoftheoreticalapproachesandempiricalevidence.Approach&MethodologyfortheEvaluation of Donor Support to Public Finance Management (PFM) Reform in DevelopingCountries.PartA.Oxford:Fiscus

deRenzio,P.,P.GomezandJ.Sheppard(2009)‘BudgetTransparencyandDevelopmentinResource‐

DependentCountries’,InternationalSocialScienceJournal,57:57‐69.deRenzio,P.andW.Dorotinsky(2007)TrackingProgressintheQualityofPFMSystemsinHIPCs:An

updateonpastassessmentsusingPEFAdata.Washington,DC,PEFASecretariat.Ear,S.(2007)‘DoesAidDependenceWorsenGovernance?’InternationalPublicManagement

Journal,10(3),259‐286.Fagernas,S.andJ.Roberts(2004)FiscalImpactofAid:ASurveyofIssuesandSynthesisofCountry

StudiesofMalawi,UgandaandZambia.ESAUWorkingPaper11.London:OverseasDevelopmentInstitute.

36

George,A.L.andA.Bennett(2005).Casestudiesandtheorydevelopmentinthesocialsciences.Cambridge,MA:MITPress.

Hedger,E.andP.deRenzio(2010)WhatdoPublicFinancialManagementassessmentstellusabout

PFMreform?ODIBackgroundNote.London:OverseasDevelopmentInstitute.IDA/IMF (2005) Update on the Assessments and Implementation of Action Plans to Strengthen

Capacity of HIPCs to Track Poverty‐Reducing Public Spending.Washington DC: InternationalDevelopmentAssociationandInternationalMonetaryFund.

IDA/IMF(2003)CountryAssessmentandActionPlan(AAP)forHIPCs:Questionnaire,Benchmarks,

Explanations,StandardTables.WashingtonDC:InternationalDevelopmentAssociationandInternationalMonetaryFund.

IMF(2010)BudgetInstitutionsandFiscalPerformanceinLow‐IncomeCountries.IMFWorkingPaper

WP/10/80.WashingtonDC:InternationalMonetaryFund.IMF (2007b).Fiscal PolicyResponse to Scaled‐UpAid: StrengtheningPublic FinancialManagement.

Washington,DC:InternationalMonetaryFund.Kakabadse,A.,N.K.KakabadseandA.Kouzmin(2003)‘ReinventingtheDemocraticGovernance

ProjectthroughInformationTechnology?AGrowingAgendaforDebate’,PublicAdministrationReview,63(1):44‐60.

Kaufmann,D.,A.KraayandP.Zoido‐Lobatón(1999)GovernanceMatters.PolicyResearchWorking

Paper2196.WashingtonDC:WorldBank.Knack,S.(2001)‘AidDependenceandtheQualityofGovernance:Cross‐CountryEmpiricalTests’.

SouthernEconomicJournal,68(2),310‐329.Knack,S.andA.Rahman(2007)‘DonorFragmentationandBureaucraticQuality’.Journalof

DevelopmentEconomics,83,176‐197.LaPorta,R.,F.Lopez‐de‐Silanes,A.Shleifer,andR.Vishny(1999)‘TheQualityofGovernment’

JournalofLaw,Economics,andOrganization,15(1):222‐279.Lieberman, E. S. (2005) "Nested Analysis as aMixed‐Method Strategy for Comparative Research."

AmericanPoliticalScienceReview,99(3):435‐452.Moss,T.,G.Pettersson,etal.(2006).AnAid‐InstitutionsParadox?AReviewEssayonAidDependency

andStateBuilding inSub‐SaharanAfrica.WorkingPaperNo.74.Washington,DC:Centre forGlobalDevelopment.

Munck,G.(2009)MeasuringDemocracy:ABridgeBetweenScholarshipandPolitics.Baltimore,MD:

JohnsHopkinsUniversityPress.Pack,H.andJ.R.Pack(1990)‘IsForeignAidFungible?TheCaseofIndonesia’TheEconomicJournal,

100(399),188‐194.PEFA(2009)IssuesinComparisonandAggregationofPEFAAssessmentResultsOverTimeandAcross

Countries.Washington,DC:PublicExpenditureandFinancialAccountability(PEFA)Secretariat.

37

PEFA(2005).PublicFinancialManagementPerformanceMeasurementFramework.Washington,DC:PublicExpenditureandFinancialAccountability(PEFA)Secretariat.

Rabe‐Hesketh,S.andB.S.Everitt(2007)AHandbookofStatisticalAnalysesUsingStata.4thedition.

BocaRaton,FL:ChapmanandHall.Rajan, R. and A. Subramanian (2007) ‘Does Aid Affect Governance?’ American Economic Review,

97(2),322‐327;Remmer,K.L.(2004)‘DoesForeignAidPromotetheExpansionofGovernment?’AmericanJournalof

PoliticalScience,48(1),77‐92.Rivera‐Batiz,F.L.(2002)‘Democracy,GovernanceandEconomicGrowth:TheoryandEvidence’,

ReviewofDevelopmentEconomics,6(2):225‐247.Svensson,J.(2000)‘ForeignAidandRent‐Seeking’,JournalofInternationalEconomics,51,437‐461.Tavares,J.(2003)‘DoesForeignAidCorrupt?’EconomicsLetters,79,99‐106.Treisman,D.(2000)‘TheCausesofCorruption:ACross‐NationalStudy’,JournalofPublicEconomics,

76:399‐457.Vyas, S. and L. Kumaranayake (2006) “Constructing socio‐economic status indices: how to use

principalcomponentsanalysis.”HealthPolicyandPlanning,21(6):459–468.Wehner,J.andP.deRenzio(2010)Citizens,legislators,andexecutivedisclosure:Thepoliticaldeterminantsoffiscaltransparency.Unpublishedmanuscript.London:LondonSchoolofEconomics.Wescott, C. G. (2009). "World Bank Support for Public Financial Management and Procurement:

FromTheorytoPractice."Governance,22(1):139‐153.WorldBank (2008).PublicSectorReform:WhatWorksandWhy?An IEGEvaluationofWorldBank

Support.Washington,DC:WorldBank,IndependentEvaluationGroup.

38

APPENDICESAppendix1.ListofPEFAcountriesAppendix2.ListofHIPCcountriesAppendix3.PFMClustersAppendix4.HIPC/PEFAindicatorsAppendix5.DonordatarequestformAppendix6.Listofvariablesandsummarystatistics

39

Appendix1:ListofPEFACountries Country Date Status Country Date Status

1 Afghanistan Jun.08 Final‐P 51 Lesotho Jul.09 Draft2 Albania Jul.06 Final‐P 52 Liberia Jun.09 Final3 Armenia Oct.08 Final‐P 53 Madagascar May08 Final4 Azerbaijan Jan.08 Final 54 Malawi Jun.08 Final5 Barbados Oct.06 Draft 55 Maldives Nov.09 Final6 Belarus Apr.09 Final 56 Mali Dec.08 Final‐P7 Belize Oct.08 Draft 57 Mauritania Mar.08 Final8 Benin Sep.07 Final‐P 58 Mauritius Jun.07 Final‐P9 Bolivia Aug.09 Draft 59 Moldova Jul.08 Final‐P

10 Botswana Feb.09 Final‐P 60 Montenegro Jul.09 Final11 Brazil Oct.09 Draft 61 Morocco May09 Final‐P12 BurkinaFaso Apr.07 Final‐P 62 Mozambique Feb.08 Final‐P13 Burundi Feb.09 Final 63 Namibia Nov.08 Final14 Cambodia Mar.10 Draft 64 Nepal Feb.08 Final‐P15 Cameroon Jan.08 Final 65 Niger Dec.08 Draft16 CapeVerde Dec.08 Final‐P 66 Pakistan Jun.09 Final17 CentralAfricanRepublic Jun.08 Final 67 PapuaNewGuinea Mar.09 Draft18 Chad Jul.09 Final 68 Paraguay Apr.08 Final‐P19 Colombia Jun.09 Draft 69 Peru Apr.09 Final‐P20 Comoros Jan.08 Final 70 Philippines Oct.07 Draft21 Congo,Dem.Republicof Mar.08 Final 71 RussianFederation Jan.07 Draft22 Congo,Republicof Mar.06 Final‐P 72 Rwanda Jun.08 Final‐P23 Coted'Ivoire Nov.08 Final‐P 73 Samoa Oct.06 Final‐P24 Dominica Apr07 Draft 74 S.TomeandPrincipe Jan.10 Final25 DominicanRepublic Nov.09 Final 75 Senegal Dec.07 Final26 Egypt May,09 Draft 76 Serbia Feb.07 Final‐P27 ElSalvador May09 Final‐P 77 Seychelles Dec.08 Final28 Ethiopia Oct.07 Final‐P 78 SierraLeone Dec.07 Final‐P29 FijiIslands Jun.05 Final 79 SolomonIslands Nov.08 Final‐P30 FYRMacedonia Aug.07 Final‐P 80 SouthAfrica Sep.08 Final‐P31 Gambia Mar.09 Draft 81 St.KittsandNevis Sep.09 Draft32 Georgia Nov.08 Final‐P 82 St.Lucia Nov.09 Draft33 Ghana Jan.10 Final 83 St.VincentandGren. Sep.06 Final34 Grenada Mar.10 Final‐P 84 Sudan May09 Draft35 Guatemala Mar.10 Draft 85 Swaziland Mar.10 Draft36 Guinea Jul.07 Final 86 Syria Mar.06 Final37 GuineaBissau May09 Final 87 Tajikistan Jun.07 Final‐P38 Guyana Dec.07 Draft 88 Tanzania Jun.09 Final39 Haiti Jan.08 Final‐P 89 Thailand Oct.09 Final40 Honduras Apr.09 Final 90 TimorLeste Feb.07 Final‐P41 India Mar.10 Final‐P 91 Togo Nov.08 Draft42 Indonesia Oct.07 Final‐P 92 Tonga Sep.07 Final43 Iraq Jun.08 Final 93 TrinidadandTobago Dec.08 Final‐P44 Jamaica May07 Final‐P 94 Tunisia Mar.10 Draft45 Jordan Apr.07 Final‐P 95 Turkey Dec.09 Final46 Kazakhstan Jun.09 Final 96 Uganda Jun.09 Final‐P47 Kenya Mar.09 Final‐P 97 Ukraine Mar.07 Final‐P48 Kiribati Dec.09 Draft 98 Vanuatu Nov.09 Final49 KyrgyzRepublic Oct.09 Final 99 Yemen Jun.08 Final‐P50 LaoPDR Dec.09 Draft 100 Zambia Jun.08 Final

Note:‘Final‐P’denotesreportsthathavebeenmadepubliconthePEFAwebsite.

40

Appendix2:ListofHIPC/PEFACountries Country HIPCAssessments PEFAAssessments

1 Benin 2001,2004 20072 BurkinaFaso 2001,2004 20073 Cameroon 2001,2004 20084 Chad 2001,2004 20095 Ethiopia 2001,2004 20076 Gambia 2001,2004 20097 Ghana 2001,2004 2006,20108 Guinea 2001,2004 20079 Madagascar 2001,2004 2006,2008

10 Malawi 2001,2004 2006,200811 Mali 2001,2004 200812 Mozambique 2001,2004 2006,200813 Niger 2001,2004 200814 Rwanda 2001,2004 200815 SãoTome&Principe 2001,2004 2007,201016 Senegal 2001,2004 200717 Tanzania 2001,2004 2006,200918 Uganda 2001,2004 2006,200919 Zambia 2001,2004 2005,2008

41

Appendix3:PFMClusters

Detailed description of the six PFM clusters used in the analysis, with PEFA indicator dimensionsincludedineach.Source:Andrews(2010)

42

Appendix4:HIPC/PEFAindicatorsThe table below explains how PEFA indicators and information was used to update HIPCindicatorscoresformorerecentyears.

HIPCIndicator PEFAa Notes

1.Coverageofthebudgetorfiscalreportingentity

IntroPI‐8PI‐9

TheHIPCindicatorisnotdirectlyrelatedtoanyonePEFAindicator,butinformationisoftenavailableindifferentpartsofthePEFAreport.

2.Degreeofspendingbeingfundedbyinadequatelyreportedextra‐budgetarysources

PI‐7(i) Directlyrelated.Conversionsometimesdifficultduetodifferencesinthresholds.

3.Reliabilityofthebudgetasaguidetofutureoutturn

PI‐1PI‐2

Directlyrelatedandeasilyconvertible.

4.Inclusionofdonorfunds PI‐7(ii) Directlyrelatedandeasilyconvertible.

5.Classification PI‐5 Directlyrelatedandeasilyconvertible.

6.Identificationofpoverty‐reducingspending

InformationnotavailableinPEFAreports.Indicatordropped.

Form

ulation

7.Integrationofmedium‐termforecasts

PI‐12 Directlyrelatedandeasilyconvertible.

8.Evidenceofbudgetexecutionproblems–arrears

PI‐4 Directlyrelated.LimitedinformationavailableinsomePEFAreports.

9.Effectivenessofinternalcontrolsystem

PI‐18PI‐20PI‐21

Easilyconvertiblebasedontheinformationavailablefordifferentindicators.

10.Trackingsurveysareinuse PI‐23 Directlyrelatedandeasilyconvertible.

Execution

11.Qualityoffiscalinformation PI‐22(i) Directlyrelatedandeasilyconvertible.

12.Regularityoftimelyinternalfiscalreporting

InformationnotavailableinPEFAreports.Indicatordropped.

13.Regularfiscalreportstrackpovertyspending

InformationnotavailableinPEFAreports.Indicatordropped.

14.Transactionsarerecordedintheaccountsinatimelyfashion

InformationnotavailableinPEFAreports.Indicatordropped.Repo

rting

15.Timelinessofauditedfinancialinformation

PI‐26 Directlyrelatedandeasilyconvertible.

Procurem

ent

16.Efficiencyandeffectivenessofthepublicprocurementsystem

PI‐19 Indicatordroppedbecause(a)notincludedin2001HIPCassessment,and(b)PEFAindicatorlooksatdifferentissues.

a)IndicatesthesectionofthePEFAassessmentreportortheindicator(andindicatordimension)inthePEFAPerformanceMeasurementFramework.

Indicators1,2,4and5wereutilisedtogeneratethetransparencyandcomprehensiveness(INFO)score;indicators3,7and10wereutilisedtogeneratethelinkingbudgeting,planningand policy (POL) score; indicators 8, 9, 11 and 15 were utilised to generate the control,oversight and accountability (CTRL) score. Letter scores were transformed into numericalones(C=1,B=2,A=3)andaddedtogetherineachcluster.

43

Appendix5:Donordatarequestform1.Coverpage

2.PEFAlarge‐Nsample

3.HIPCmedium‐Nsample

44

Appendix6:Listofvariablesandsummarystatistics

Variable Description SourceEconomic/socialvariablesGDPpercapita PercapitaGrossDomesticProduct(US$PPP),average

2002‐06WorldDevelopmentIndicators

RecentGDPgrowth Year‐on‐yearGrowthinGrossDomesticProduct(%),average2002‐06

WorldDevelopmentIndicators

Aiddependency OfficialDevelopmentAssistanceasashareofGrossNationalIncome(%),average2002‐06

WorldDevelopmentIndicators

Resourcedependency Dummyvariableforcountrieswhichrelyheavilyonoilandmineralrevenues

InternationalMonetaryFund

Population TotalPopulation,average2002‐06 WorldDevelopmentIndicators

Adultliteracy Adultliteracyrates(%),average2002‐06 HumanDevelopmentReport

Tradeopenness SumoftotalimportsandtotalexportsasashareofGrossDomesticProduct(%),average2002‐06

WorldEconomicOutlook

Technologicaldiffusion Electricpowerconsumption(kWhpercapita)Telephonesubscribers(per100people)Personalcomputers(per100people)Internetusers(per100people)

WorldDevelopmentIndicators

Political/institutionalvariablesAdministrativeheritage DummyvariablesforformerBritishandFrench

coloniesQualityofGovernment

Democracy FreedomHousesumofCivilLibertiesandPoliticalRightsIndex(1=mostdemocratic,7=leastdemocratic)

FreedomHouse

Statefragility Dummyvariableforpresenceofpeacekeepersinthecountrysince1995

[DataobtainedfromMattAndrews]

Regionaldummies Dummyvariablesforregionoftheworldtowhichcountrybelongs

WorldDevelopmentIndicators

Aid‐relatedvariablesPFMsupport DonorsupportforPFMreforms OwndatacollectionAidfragmentation Fragmentationindexcalculatedonthebasisofdonor

sharesoftotalODAreceivedbycountry,average2002‐06

OECD/DACdatabase

GBSas%oftotalODA GeneralBudgetSupportasashareoftotalODAreceivedbycountry(%),average2002‐06

OECDDAC/CRSdatabase

IMFprogramme DummyvariableforpresenceofIMFprogrammeforatleastoneyearduring2002‐06

InternationalMonetaryFund

WBPRSC DummyvariableforpresenceofWorldBankPovertyReductionSupportCreditforatleastoneyearduring2002‐06

WorldBank

Lengthofdonorengagement

No.ofyearssincefirstWorldBankPFMproject [DataobtainedfromMattAndrews]

45

Variable Obs. Mean Std.Dev. Min MaxPEFAaveragescore 100 2.44 0.48 1.38 3.58GDPpercapita,PPPUS$ 99 3,902 2,159 251 19,189RecentGDPgrowth,% 100 4.96 3.23 4.60 9.33Population(millions) 100 30.8 112.0 0.047 1,080.0Resourcedependency 100 0.17 0.38 0 1Adultliteracy,% 99 75.22 21.37 26.14 99.00Tradeopenness 94 84.87 34.40 27.04 190.36Technologicaldiffusion

Electricity(kWhpc)Phone(%)Computer(%)Internet(%)

100100100100

730.9329.733.446.57

1199.99

29.434.147.85

0000

5666.81122.8418.0037.83

Democracy(FreedomHouse) 100 3.72 1.57 1 7Statefragility 100 0.16 0.37 0 1Aiddependency,% 98 9.19 10.72 0 43.10PFMsupport,US$m/year 100 3.07 5.59 0 33.21Aidfragmentation 99 0.80 0.10 0.36 0.91GBSas%oftotalODA 99 0.04 0.05 0 0.30IMFprogramme 100 0.58 0.50 0 1WBPRSC 100 0.20 0.40 0 1Lengthofdonorengagement(years) 100 12.04 6.69 ‐1 22