economy-wide modeling: uncertainty, verification, and ... · economy-wide modeling: uncertainty,...
TRANSCRIPT
Economy-WideModeling:Uncertainty,Verification,andValidation
June17,2016
PreparedfortheU.S.EPAScienceAdvisoryBoardPanel(SAB)onEconomy-WideModelingoftheBenefitsandCostsofEnvironmentalRegulation
ThispaperhasbeendevelopedtoinformthedeliberationsoftheSABPanelonthetechnicalmeritsandchallengesofeconomy-widemodeling foranair regulation. It isnotanofficialEPAreportnordoes itnecessarilyrepresenttheofficialpoliciesorviewsoftheU.S.EPA.
2
[Thispageintentionallyleftblank.]
3
TableofContents
1 Introduction............................................................................................................................4
2 UncertaintyinCostEstimation...............................................................................................52.1 UncertaintyinRegulatoryCostEstimates....................................................................................52.2 UncertaintyinimplementationofRegulatoryCostsinEconomicModels...................................8
3 UncertaintyinBenefitsEstimation.........................................................................................93.1 UncertaintyinRegulatoryBenefitsEstimates..............................................................................93.2 UncertaintyinIntroducingAirQualityImprovementBenefitsinEconomicModels.................11
4 UncertaintyinCGEModeling................................................................................................114.1 ParametricUncertainty..............................................................................................................114.2 StructuralUncertainty................................................................................................................13
5 AnalyticalApproaches...........................................................................................................135.1 SensitivityAnalysis.....................................................................................................................145.2 FormalUncertaintyAnalysis.......................................................................................................155.3 Inter-ModelComparisonExercises............................................................................................20
6 PresentationofUncertaintyAnalysis....................................................................................20
7 VerificationandValidationExercises....................................................................................247.1 ModelVerification......................................................................................................................257.2 Modelvalidation........................................................................................................................26
7.2.1 BackoftheEnvelopeModels.............................................................................................267.2.2 StatisticalValidationofModelParameters........................................................................267.2.3 CGEModelValidationExercises.........................................................................................277.2.4 ChallengesofModelValidation..........................................................................................28
8 ConcludingRemarks..............................................................................................................28
References....................................................................................................................................30
4
1 Introduction
Benefitcostandeconomicimpactanalysesofnationalregulationsdesignedtoimproveairqualityrequireinformationfromnumerousscientificdisciplinesandsourcestobelinkedandassessed.Eachcomponentoftheseanalyses(e.g.,engineeringassessment,airqualitymodeling,riskassessment,economicanalysis)maybesubjecttouncertaintyandthe linkagesbetweenthesecomponentsmayacttocompoundtheuncertainty. A recentNational Academy of Sciences study onmaking Environmental DecisionsUnderUncertaintynoted that“informed identificationanduseofuncertainties in theprocess isanessentialfeatureofenvironmentaldecisionmaking”(NAS,2013).In2002,theNASalsonotedthat:
“Evengreatuncertaintydoesnotimplythatactiontopromoteorprotectpublichealthshouldbedelayed.Decisionsaboutwhethertoact,whentoact,andhowaggressivelytoactcanonlybemadewith some understanding of the likelihood and consequences of alternative courses ofaction. The potential for improving decisions through researchmust be balanced against thepublic health costs incurred because of a delay in the implementation of controls. Completecertaintyisanunattainableideal.”
TheNASCommitteein2013classifieduncertaintiesaccordingtothreetypes:(1)statisticalvariabilityandheterogeneity,(2)deepuncertainty,and(3)modelandparameteruncertainty.Statisticalvariabilityandheterogeneityreferstoexogenoussourcesofuncertaintyinherentinthesystemorprocessunderstudy.Thesearesourcesofuncertaintythat,exante,cannotbereducedthroughadditionalresearchordatacollection,forexamplestochasticityindynamicprocesseswhosefundamentalsarewellunderstood.Deepuncertaintyreferstocaseswherethereisalackofunderstanding,ornotabledisagreement,aboutthefundamentalsofunderlyingprocessesimportanttounderstandingtheimpactofaninterventioninthesystemofinterest.Thesesourcesofuncertaintyarecharacterizedbycaseswhereadditionalresearchordatacollectionareunlikelytoreducetheuncertaintyinthetimebeforeapolicymakerneedstoselectacourse of action.Model and parameter uncertainty refers to uncertainty inmodels that are used torepresentunderlyingprocessesrelevanttounderstandingtheimpactofaninterventioninthesystemofinterest,whereamodelisdefinedasa“simplificationofrealitythatisconstructedtogaininsightsintoselectattributesofaparticularphysical,biologic,economic,orsocialsystem”(NRC,2009).Thesesourcesofuncertaintyariseduetocurrentlimitationsofscientificandeconomicprocesses,nowandinthefuture,andintheorycouldbereducedbyadditionalresearchpriortomakingadecision.
While all these sources of uncertaintymay be relevant for decisionmaking regarding environmentalregulations designed to improve air quality, this paper focuses primarily on model and parameteruncertainty related to benefit cost analysis and the potential use of CGE models for economy-wideanalysis.1Thiswhitepaperintroducessomeofthekeysourcesofuncertaintyassociatedwithbenefitcost
1Justasuncertaintymayaffectdecisionsaboutenvironmentalregulations,uncertaintymayaffectotherdecisions
intheeconomy.Whileoutsidethescopeofthispaper,dynamicstochasticgeneralequilibrium(DSGE)modelscan capture some aspects of decisionmaking under uncertainty by economic agents. For examples ofDSGEmodels applied to environmental issues see the white paper on Economy-Wide Modeling: Evaluating theEconomicImpactsofAirRegulations.
5
analysesofair regulationsandadditionalsourcesofuncertainty thatmayarisewhenextending thoseanalysestoincludeaneconomy-wideassessmentusingCGEmodels.Giventhesesourcesofuncertainty,thiswhitepaperdiscussesanumberofanalyticalapproachesthatmaybeusedtoprovidedecisionmakersandstakeholderswithadditionalinformationabouttherobustnessofmodelresults.Theseapproachesandothermethodsofmodelvalidationandverificationcanhelpprovideconfidence inthequalitativeconclusions of policy analysis based onmodeling results.While the focus of the discussion is on theestimationofbenefitsandcosts,thesamesourcesofuncertaintycanalsoaffectestimatesofeconomicimpacts.
Specifically, the white paper begins with a brief description on uncertainties inherent in traditionalengineering-basedcostassessmentsfollowedbyuncertaintiesthatmaybeassociatedwithincorporatingthis information in a CGEmodel. Subsequently, the paper provides a brief discussion of some of theuncertainties associatedwith traditional benefits assessment for regulations designed to improve airquality.ThewhitepaperthenconsiderssourcesofuncertaintyassociatedwithCGEmodeling.Followingtheintroductorydiscussionsonsourcesofuncertainty,thepaperconsidersapproachesforquantitativelyassessingtheimpactofuncertaintyontheresultsandmethodsforpresentingthoseresultsinamannerthatappropriatelyconveys the informationrelevant fordecisionmakersandstakeholders.Finally, thepaper considers approaches of model validation and verification for CGE models to further increaseconfidenceinmodelingresultsandreducepotentialsourcesofuncertainty.
2 UncertaintyinCostEstimation
Economy-wideanalysisofairregulationswillinpartbeinformedbytraditionalengineeringcostestimatesofpollutionabatement.Uncertaintyintheresultsoftheseengineeringanalyseswillinfluencetheoveralldegreeofuncertaintyintheresultsofeconomy-wideanalysis.ThissectionexaminesuncertaintiesarisingfromtheestimationofengineeringcoststhatmaybeusedasinputstoCGEmodels,whenCGEmodelsareusedtoperformregulatoryanalysis.Thissectionbeginsbyexamininguncertaintyinthecostinputsfromtheperspectiveofthecostanalystorengineerwhotypicallydevelopstheregulatorycostestimatesusedinregulatoryactions.Subsequently,thesectiondiscussesuncertaintiesthatmayariseusingtheseestimatestointroducearegulatoryshockintoaCGEmodel.
2.1 UncertaintyinRegulatoryCostEstimates
Entities affected by a regulationmay incur compliance costs in the process ofmitigating pollution tocomplywith the regulation. The largest components of compliance costs are typically the capital andoperating costs associated with pollution control equipment. Capital costs are often one-timeexpenditures related to the installation or retrofit of structures or equipment to reduce emissions.Operatingcostsarerecurringannualexpendituresassociatedwiththeoperationandmaintenanceofthepollutioncontrolequipmentandwilloftenincludemonitoring,reportingandrecordkeepingexpenditures.Administrativeandenforcementcostsmayalsoaccruetolocal,state,andfederalregulatoryagencies.
Analyststypicallyestimatecompliancecostsexpectedtobeincurredbyregulatedentitiesbeforetheruleis implemented. In many cases, these ex ante estimates of compliance costs are estimated by cost
6
engineersandpolicyanalyststofulfillstatutoryobligationsoutsideofbenefitcostanalysisorpurposesofevaluating the national economic impacts of the regulation. As has been discussed by the NationalAcademies of Science (NAS, 2013), these ex ante estimates of compliance costs are often based onengineeringmodels inwhich there is uncertainty over facility characteristics, the number of affectedfacilities,andthedegreeofregulatorycompliance.Thisuncertaintyoftenleadstocasesinwhichtheexanteestimatesofcompliancecostsdifferfromtheexpost(realized)compliancecosts.
The“RetrospectiveStudyoftheCostsofEPARegulation”highlightedanumberofreasonsthatexanteandexpostcompliancecostsmaydiffer(Kopitsetal.,2014).Forexample,technologicalinnovationandunforeseencomplianceoptionsmightalsocauseexantecoststodivergefromexpostcompliancecosts.Unanticipatedchanges inotherexogenous factors, suchas changes inenergypricesordemandshiftsindependentofregulation,mayalsocauseexpostcompliancecoststodivergefromtheexanteestimates.Furthermore,strategicbehaviorbyfirmscaninfluenceexantecompliancecostestimates,asmuchoftheinformationusedinestimatingcompliancecostscomesdirectlyfromindustry.
SinceEPAcompliancecostestimationisprimarilyperformedbycostanalystsandengineers,itisusefultoreview the framework developed by those professions to articulate the accuracy of engineering costestimates.Forexample,theEPAAirPollutionControlCostManual(2002)isdescribedasacomprehensivesetof“proceduresanddataforsizingandcostingcontrolequipment”forVOCs,PM,SO2,NOx,andsomeacidgases.ThismanualaswellastheAACEInternationalCostEstimatingClassificationSystem2havebeenused to provide context for the uncertainty associated with pollution control cost estimates forcompliancewithNationalAmbientAirQualityStandards(NAAQS).
Inthe2015FinalOzoneNAAQSRegulatoryImpactAnalysis(RIA),forexample,EPAstatedthatthereisarangeof ± 30percent for non-electrical generatingunit point source control costs, citing the EPAAirPollutionControlCostManual(2002).ThislevelofaccuracyisdescribedintheEPAAirPollutionControlCostManualasa“studyestimate.”AccordingtotheManual,thestudyestimateiswellsuitedforuseinregulatory development because it does not require detailed site-specific information necessary forindustrylevelanalyses.Theyalsocanbepreparedatarelativelylowcostwithrelativelyminimaldata.WhileinformationthatismoredetailedmaybeavailabletotheEPA,itisoftenproprietaryandconsideredconfidentialbusinessinformation.
In addition to the studyestimate, theEPAAirPollutionControlCostManualdiscussesother typesofestimates:
• Order-of-magnitude:Thisestimateprovidesarule-of-thumbprocedureappliedonlytorepetitivetypesofplantinstallationsforwhichthereexistsgoodcosthistory.Itserrorboundsaregreaterthan±30percent.Thesoleinputrequiredformakingthislevelofestimateisthecontrolsystem’scapacity.
• Scope,BudgetAuthorization,orPreliminary:Thisestimate,nominallyof±20%accuracy,requiresmoredetailedknowledgethanthestudyestimateregardingthesite,flowsheet,
2AACEInternational.RecommendedPracticeNo.17R-97.CostEstimateClassificationSystem–CostEstimatingandBudgeting.RevisedonNovember29,2011.Availableathttp://www.aacei.org/.
7
equipment,buildings,etc.Inaddition,roughspecificationsfortheinsulationandinstrumentationareneeded.
• ProjectControlorDefinitive:Theseestimates,accuratetowithin±10%,yetrequiremoreinformationthanthescopeestimates,especiallyinformationconcerningthesite,equipment,andelectricalrequirements.
• Firm,Contractor,orDetailed:Thisisthemostaccuratetypeofestimate(±5%)andrequirescompletedrawings,specifications,andsitesurveys.Consequently,detailedcostestimatesaretypicallynotavailableuntilrightbeforeconstructioniscommenced.
These levels of accuracy also correspond to those reported in Perry’s Chemical Engineers’Handbook,anotherimportantreferenceforcostsengineersandanalysts.
ThetypeofexantecompliancecostestimatecalculatedbyEPApriortoarulemakingwillberegulationspecific,giventheheterogeneity inavailable informationandscopeofregulations.Forexample,somerulemakingsmayaffectrelativelyfewfacilities,anddetailedinformationonfacilitycharacteristicsmaybeavailablepublicallyorthroughformalInformationCollectionRequests.Inothercases,regulationmayonlyaffectnewfacilitieswhereheterogeneityinfacilitycharacteristicsthatwouldaffectretrofitcostsarenota confounding factor. Compliance cost estimates for these actions may be more precise than forregulationslikeaNAAQS,wheretheillustrativecontrolsstrategiesexaminedinNAAQSRIAsmayrequirereducingemissionsatalargenumberoffacilities(potentiallyinthethousands)acrossmultiplesectorsoverrelativelylongtimehorizon.
In its air-related RIAs, the EPA has limited experiencewith evaluating uncertainty in compliance costestimates. However, some recent examples of where the Agency has examined the implication ofuncertainfactorsoncompliancecostsinclude:
• The levelof voluntaryemissions reductionsversus regulatoryemissions reductions foroil andnaturalgasemissionssources(EPA2012);
• whether stateswould adopt a rate-based ormass-based compliance approaches formeetingrequirementsoftheCleanPowerPlan(EPA2015a);
• alternativeassumptionsonthecostsofemissionscontrolsforNAAQS-relatedcostanalysis(EPA2015b);
• and,theinfluenceofimportantcostcomponentssuchasthevalueofnaturalgasthatiscapturedbyemissionscontrolsandroutedtosaleslinesorusedinprocessestooffsetotherfuelpurchases(EPA2012andEPA2016).
Asnotedinthe“WhitePaperonUsingCGEModelstoEvaluateSocialCostofAirRegulations”,regulatoryanalysis may also differ due to the available information on potential pollution control options. Forexample,insomecasespollutioncontrolequipmentmayalreadybeinusewithinpartsoftheindustryorcloselyrelatedindustries.Inothercases,theremaybeuncertaintyinhowstatesorlocalgovernmentsmayimplementregulationsandthereforeinthefacilitiesaffectedandthepollutioncontrolequipmentthatmightbeadopted.Forexample,inillustrativeattainmentanalysesconductedforsomeNAAQS,onceallidentifiedcontroltechnologieshavebeenappliedsomeareasofthecountrymaystillbemodeledas
8
out of compliance with the air quality standard. In some cases, EPA has had to assume the per toncompliance costs for “unidentified controls” that reduce emissions sufficiently to bring areas intoattainmentofalternativeairqualitystandards.Whenconsideringthecostofunidentifiedcontrols,theEPAhastraditionallyofferedsensitivityanalyseswithalessexpensiveandamoreexpensivecostpertonestimate.However,thereisnotafirmanalyticalbasisforthevalueschosenforthesensitivityanalyses;rather,theyroughlycorrespondtothesame+/-30percentrangeassumedforidentifiedcontrols.
2.2 UncertaintyinimplementationofRegulatoryCostsinEconomicModels
Asdiscussedin“Economy-WideModeling:SocialCostandWelfareWhitePaper”,therearefewexamplesinwhichaCGEmodelhasbeenusedtoestimatethecostsofregulationsdesignedtoimproveairquality.InthecaseswhereaCGEmodelhasbeenusedtoanalyzenon-priceregulations,suchastheemissionslimitandairqualityregulationspromulgatedbytheEPA,theyareoftenmodeledasaproductivityshock.Specifically,compliancewiththeregulationcreatesaneedforadditionalinputstoproducegoodsintheregulatedsectoralongwithpollutionabatement.Whilethetotalcostoftheseadditionalinputscanbederivedfromdetailedcompliancecostestimatesfromanengineeringorpartialequilibriummodel,itisnotalwaysclearhowtoallocatethetotalcostamongtheinputsspecifiedintheCGEmodel,becauseCGEmodelsarebytheirnatureanaggregated,parsimoniousrepresentationoftheeconomy.Onefrequentlyusedapproachistoallocatetheabatementcostsindirectproportiontotheinputs(i.e.,capital,labor,andintermediate goods) used in the regulated sector of the CGE model. In other words, regulatoryrequirementsdonot change theproportionof labor, capital, or other inputs in the firm’sproductionfunction.This“Hicks-neutral”allocationistheapproachtakenbyHazillaandKopp(1990)andJorgensonandWilcoxen(1990).BallardandMedema(1993)allocatealloftheabatementcoststocapitalandlaborinputs only. There is inherent uncertainty associated with the decision to model the production ofpollutionabatementashavingthesameproductionfunctionastheaffectedindustry.EvenincaseswheredetailedcompliancecostestimatesprovideadditionalinformationastothesharesofinputsexpectedtobeusedinpollutionabatementtheremaybeuncertaintyastohowtheinputsfromtheseengineeringcostmodelsmaptothegoodswithinaCGEmodel.Furthermore,theremaybeuncertaintyastohowthesubstitutionpossibilitiesbetweeninputstoproductionintheregulatedsectorintheCGEmodelmaptosubstitution possibilities in pollution control options that may be adopted under non-price basedregulations, or how future technology developmentmight be expected to change those substitutionpossibilities.
Itisalsoimportanttoconsiderboththespatialandtemporalallocationofcost.Theremaybeamismatchbetweenaffectedfacilitylocationswithininputandoutputmarketsandthescaleoftheeconomy-widemodel,whichistypicallyregionalornationalinscale.Thatis,itmaybedifficulttopreciselymodelchangesthat affect specific areas of the country in an aggregatedmodel. As a result, thismay lead to someuncertaintyintheresultsoftheaggregatemodel.
Likewise,costsmaybeincurredatdifferentpointsduringthetimehorizonoftheeconomy-widemodel.Withalongtimehorizon,itisnecessarytoconsidertheroleoftechnologicalchangeanditsimpactontheprices of inputs over time. Recent NAAQS RIAs include discussions of how technological change(specifically,innovationincontroltechnologies)mayreducecostsovertime,andtheempiricalliterature
9
alsohasnotedthatvariablecostsofproductionorenvironmentalabatementtendtodeclineovertimewithcumulativeexperience.WhilethestudyonthecostsandbenefitsoftheCleanAirActfrom1990to2020(EPA2011)accountedfor“learningcurve”effects(ortheextenttowhichthecostsofatechnologydeclineasexperiencewiththattechnologyincreases),regulatoryanalysesoftendonotattempttoadjustcosts to account for different assumptions about technological changeor theeffect of experienceoncontrol costs.3 As a result, when incorporating compliance costs in a CGE model, any impact oftechnologicalchangewillbedeterminedbytherepresentationoftechnologicalchangethatexistsinthemodel, rather thanassumptions that are incorporated in theengineering costestimates.AsnotedbyPindyck(2007),characterizingtheuncertaintyovertechnologicalchangeisinitselfdifficult.
3 UncertaintyinBenefitsEstimation
Aswould be the casewith entering costs into a CGEmodel, uncertainty in any inputs related to thebenefitsofaregulationwillpropagatethroughtheanalysisandleadtosomeuncertaintywithrespecttoresults.Whileacompletediscussionof thesourcesofuncertainty inbenefitsestimates isbeyondthescopeofthispaper,thissectionprovidesabriefsummaryofEPA’seffortstocharacterizeuncertaintyinthebenefitsestimatesinrecentRIAs.
3.1 UncertaintyinRegulatoryBenefitsEstimates
Inanycomplexanalysisusingestimatedparametersandinputsfromnumerousmodels,therearelikelytobemanysourcesofuncertainty.EPAbenefitsanalysesincludeinputsfrommanydatasources,includingemissions inventories, air quality data from models (with their associated parameters and inputs),populationdata,populationestimates,healtheffectestimatesfromepidemiologystudies,economicdataand parameters for valuing benefits, and assumptions regarding the future state of the world (e.g.,regulations,technology,emissions,andhumanbehavior).Theremaybeuncertaintyassociatedwitheachof these inputs. Understanding key uncertainties in each stage of the analysis, and how they mightinteract,canbe important forunderstandingthe informationcontained inthetotalquantifiedbenefitestimates.
WhiletheNationalResearchCouncil(NRC,2002,2009)reviewedEPA’smethodologyforcalculatingthebenefitsofreducingairpollutionandfoundittobereasonableandinformative,theyalsohighlightedtheneed to conduct rigorousquantitativeanalysesofuncertaintyand topresentestimates inaway thatrecognizes their inherent uncertainty. In response, EPA has continued to work to improve thecharacterizationofuncertainty inhealth incidenceandbenefitsestimates.QuantitativeapproachestouncertaintyanalysisforbenefitsassessmentinRIAshaveincluded:
• MonteCarloassessmentsthataccountforrandomsamplingerrorandbetweenstudyvariabilityinepidemiologicalandeconomicvaluationstudies(forpollutantsPM2.5andO3);
3Anexceptiontothisisrecentgreenhousegasemissionstandardsforlight-dutyandformedium-andheavy-duty
vehicles.Seehttps://www3.epa.gov/otaq/climate/documents/420r12901.pdfforoneexampleofhowlearningcurveshavebeenappliedinthesecontexts.
10
• alternativeconcentration-responsefunctions(PM2.5andO3);
• alternative income elasticities in the specification of willingness-to-pay functions used formortalityandmorbidityendpoints(O3);
• inclusionofthresholdsinsomeconcentration-responsefunctions(O3);
• alternativecessationlagsforlong-termmortality(O3);
• aconcentrationbenchmarkassessmentcharacterizingthedistributionofavoideddeathsrelativetospecificconcentrationsinthelong-termepidemiologicalstudiesusedtoestimatePM2.5-relatedmortality(PM2.5);and
• whenmonetizingclimatebenefits,theuseoffourpossiblemeasuresforthesocialcostofcarbon(SC-CO2)andsocialcostofmethane(SC-CH4),reflectingalackofconsensusontheappropriatediscount rateand toaccount for thepossibilityofhigher-than-expected impacts fromclimatechange.
Whilesuchtechniquescanprovidevaluableinformationaboutkeyuncertaintiesandhowtheyinfluencebenefitsestimates,theseapproachesmaystillfacechallengesinaccountingfortheroleofuncertaintyinother input variables, including emissions and air quality modeling, baseline incidence rates, andpopulation exposure estimates. Challenges also remain in addressing correlations between inputparametersandfullycharacterizinginputdistributions.Asaresult,reportedconfidenceintervalsandtherangeofestimatesmaypresentonlyapartialpictureoftheoveralluncertaintyinthefinalestimates.
In2009,EPAundertookaprojecttoidentifytheinputparametersandassumptionsthathavethepotentialto be significant contributors to the uncertainty in the benefits estimates produced by BenMAP, theprimarytoolEPAusestoestimatethehumanhealthimpactsandeconomicvalueofairqualitychanges.To assess the impact of uncertainty in these parameters and assumptions, sensitivity analysis wasconductedwhere a range of values for key parameterswas defined, and their effects on resultswascalculatedwhileholdingallotherparametersattheirmidvalues.Thestudyfoundthatthecomponentsof a PM2.5 benefits analysis contributingmost to uncertainty of themonetizedbenefits andmortalityincidencearetheestimateofthevalueofastatisticallife,thechoiceofconcentration-responsefunctionformortality,andthechangeinPM2.5concentration(Mansfieldetal.,2009).
As noted in the “Economy-Wide Modeling: Benefits of Air Quality Improvements White Paper”, thebenefitsofregulationscanvarysignificantlyoverspaceduetoanumberoffactors,includingthespatialheterogeneityinairqualityimpactsandthespatialvariationinpopulationandbaselineincidencerates.Anumberofstudieshaveanalyzedthesensitivityofbenefitsestimatestotheresolutionoftheairqualityinputsand/orunderlyingpopulationandincidenceratedata.Theyhavefoundthatdifferencesinspatialresolution,operationalized in thestudiesbyvaryingthesizeof thegridcellwithinwhichairquality isassumedtobehomogenous,canleadtosubstantiallydifferentbenefitsestimates(DeRidderetal.2014,Fannet al. 2011, Kheirbek et al. 2013, Li et al. 2015, Punger andWest 2013, Thompsonet al. 2014).However,thestudiesdonotfindaconsistentbiasintheresults.PungerandWest(2013)foundthatverycoarsegridresolutions(>250km)producemortalityestimatesthatarebiasedsubstantiallylowforPM2.5,
11
buttheyfindthatthemortalityestimatesforbothPM2.5andO3at36kmresolutionareslightlyhigherthantheresultsmodeledat12kmresolution.Lietal.(2015)alsofoundanegativebiasinmortalityestimatesatverycoarseresolutions.DeRidderetal.(2014)foundasimilarnegativebiasinexposuretoNO2whenmovingfromafinelyresolvedgrid(1km)toacoarserresolution(64km).Thompsonetal.(2014)foundO3healthimpactstobesensitivetoresolution,butdidnotfindasimilarpatternwithPM2.5,atleastwhencomparing36kmresultsto12kmresults.Fannetal. (2011)andKheirbeketal. (2013) foundthatveryspatiallyresolveddataallowedforbetteridentificationofareasofhighvulnerabilitywithincities.
3.2 UncertaintyinIntroducingAirQualityImprovementBenefitsinEconomicModels
Settingasideuncertaintiesintheupstreamelementsaffectingbenefitsestimates,anybenefitsanalysiswill produce a set of estimates that may include confidence intervals for some endpoints, may notcompletelyaccount forallof thebenefits (ordisbenefits)ofanaction,andwhichmaynotbedirectlytranslatable to the inputs needed for an economy-wide model. As discussed in the “Economy-WideModeling:BenefitsofAirQualityImprovementsWhitePaper,”challengesremaininfullyimplementingtheeffectofairqualityonthebehaviorandwell-beingofeconomicagentsinaCGEmodel.Todate,therehas been some limited inclusion of air quality impacts in CGE models including changes in medicalexpendituresandeffectsonthelaborforcethroughachangeinthetimeendowment.Expandingthosecategoriesofbenefitsor introducingamorecomplete treatmentofairqualitymayrepresentnotablechallenges and be associated with uncertainty surrounding the implementing methodology. In thesecases, theremaybeuncertaintyover the structureof themodel,orat least the componentsused toincorporatetheadditionalbenefitcategories.
Aswasthecasewhenconsideringtheinclusionofcostsinaneconomicmodel,itisimportanttoconsiderboth the spatial and temporal allocation of benefits.Whilemore spatially resolved datamay lead todifferentbenefitsestimates,thespatialresolutionofthebenefitsestimateistypicallymuchfinerthanthespatialresolutionofanyCGEmodelintowhichitmayserveasaninput.Asaresult,someaggregationofbenefitsestimatesand/orairqualitymodelingwillbenecessary.ThedegreetowhichthisaggregationinfluencesCGEmodeling results is an importantquestion. Likewise, the timingof benefits affects thebudgetconstraintandtimeendowmentavailabletohouseholds,whichmayinfluencetheresultsfromaCGEmodel.
4 UncertaintyinCGEModeling
AppliedGEanalysis,includingwithCGEmodels,haslongbeenknowntobesensitivetobothstructuralandparametricassumptions.Insomecases,theseuncertaintiesmaybeimportantfortheinterpretationofpolicyanalysisconductedusingCGEmodels.Beforediscussingapproachestoquantitativelymodelingandpresentinguncertaintyoverpolicyresults,thissectiondiscussestypesofuncertaintyrelatedtoCGEmodels.
4.1 ParametricUncertainty
TheCGEmodelscurrentlyusedinpolicyanalysisoftencontainhundreds,ifnotthousands,ofparameters,including those that define production technologies, household preferences, benchmark economic
12
activity,currentpolicies,andchangesintechnologiesandfactorendowmentsovertime.Thisparametricuncertaintymayhaveimplicationsforthetypeofanalysisconductedandtheinterpretationoftheresults.
OnewellrecognizedareaofuncertaintyinCGEmodelsaretheestimatesofelasticitiesthathelpdefineproductiontechnologiesandagentpreferences.Uncertaintyaroundelasticityparametershasfrequentlybeen a focus of analysts as themodeling results are often highly sensitive to these parameters (e.g.,Shoven and Whalley, 1984). The sensitivity of results to elasticities has been the subject of muchdiscussiongiventhecommonapproachofselectingthevaluesthroughacalibrationprocessasopposedeconometric estimation (Hansen and Heckman, 1996). However, selecting econometrically estimatedparametervaluesfromtheliteratureisnotwithoutitsownconcernsduetoinconsistenciesbetweentheempirical analyses and the structure of the CGEmodel and a large rangeof potentially contradictorystudiesthatprovideelasticityestimates(Canova,1995).Beckmanetal.(2011)demonstratedthatuseofcurrentandwell-researched substitutionelasticities for calibratedmodels canbecrucial forachievingdefensible behavior of a CGE model, and demonstrated an approach for validating modelbehavior/parameterizationusinghistoricalobservations.Giventhatnotallparametersofamodelmaybenecessarilyverifiedinsuchaprocess,sensitivityanalysis isonemeansoftryingtoassessareasonablerangeforthemodelingresultsofinterestgivenlikelyrangesoftheinputparameters.
ThesensitivityofCGEresultstoassumptionsand/orestimatesofsubstitutionelasticitieshasbeenwelldemonstrated.Forexample,FoxandFullerton(1991)findthatestimatesofwelfarechangesassociatedwithtaxreformcanbemoresensitivetoassumptionsaboutelasticityparametersthantheactuallevelofdetailaboutthetaxsystemincludedinthemodel.Elliotetal.(2012a)findthattheresultsofappliedCGEanalysesarelikelyfarmoresensitivetouncertaintyaroundassumptionsaboutelasticityparametersthanotherdatainputssuchasthebenchmarksocialaccountingmatrix.
Indynamicmodels,resultshavebeenfoundtobeparticularlysensitivetoadditionalparametersbeyondthesubstitutionelasticities.Forexample,Websteretal.(2001)findthatexogenoustemporalprojections,suchasthosedefininglaborproductivitygrowthandautonomousenergyefficiencyimprovementoverthemodels timehorizon,notably influence results.Additional studieshave foundmodel results tobeparticularly sensitive toassumptionsabout future technologyavailability,when such technologies aredirectlyrelatedtocompliancewiththepolicybeinganalyzed(e.g.,Clarkeetal.(2014)).
Previouseconomy-wideanalysesconductedbyEPAhaveprimarilyexaminedthesensitivityofresultstokey policy parameters. For example, in 2009members of the United States Congress requested EPAconductananalysisofHouseResolution2454,whichproposedamulti-sectorallowance-tradingprogramdesigned to reduce emissions of greenhouse gases.4 In that analysis, EPA examined the sensitivity ofresults from two CGE models (ADAGE and IGEM) under a number of scenarios in which key policyparameters(e.g.,availabilityofoffsets,energyefficiencyprovisions)werevaried(EPA,2010a).However,inthatanalysisandasupplementaryanalysis(EPA,2010c),EPAalsoexaminedthesensitivityofresultstokeyassumptionsregardingthefutureavailabilityofcertaintechnologies(e.g.,newnuclearpowerplants,
4http://thomas.loc.gov/cgi-bin/bdquery/z?d111:H.R.2454:
13
carboncaptureandsequestration).Theresultsoftheanalysiswerefoundtobesensitivetoexogenousassumptionsregardingtheavailabilityoftechnologiesinthefuture.
4.2 StructuralUncertainty
ThedevelopmentofanyCGEmodelinvolvesmanystructuralchoicestoaddressaparticularquestionorsetofquestions.Commonhigh-levelstructuralchoices includestaticvs.dynamic,myopicvs. forward-looking,thedegreeofsectoralandregionaldisaggregation,andclosedvs.openeconomy.Otherexamplesofstructuralchoicesincludethetradespecification(Armington,Heckscher-Ohlin,orMelitz),savingsandinvestmentclosures,departures fromthefullemploymentassumption,CESnestingstructure,andtheformoftheutilityfunction(e.g.,Cobb-DouglasorStone-Geary/LinearExpenditureSystem).Banse(2013)inareviewofnineglobalCGEmodelsusedforagriculturalanalysislist16structuralspecificationoptionsforlabormarketsand25optionsforcapitalmarkets.
Ingeneral,structuraltestingismoreresourceintensivethanparametricsensitivitytesting,thoughthisdependsuponthetypeofstructuralchangeandtherangeofparametricuncertainty.Therearemanyexamplesof singlemodel structural comparison testing in the literature.An illustrative sampleof thisliterature follows. Balistreri and Rutherford (2012) examine the differences between Armington andMelitztreatmentsofinternationaltradeinastudyofbordercarbonadjustments.Theyfindhigherleakagerates and more effective border adjustments under the heterogeneous firm structure of a Melitzapproach. Jacoby and Sue-Wing (1999) and McFarland et al. (2004) show that the incorporation ofvintaged capital in aCGEmodel raises the costsof reducing greenhousegasemissions.Babiker et al.(2009)comparesforward-lookingandrecursive-dynamicspecificationsforclimatepolicyanalysis.Theynotethatthemacro-economiccostsare lower intheforward-lookingversionbecauseoftheabilitytoshift consumption optimally over time. However, the forward-looking model, due to computationallimitations, also drops the full capital vintaging specification and contains fewer explicit emissionreductiontechnologies.Thesefactorslikelyhavecountervailinginfluencesontherelativemacroeconomiccosts. EPA addressed structural uncertainty in the aforementioned economy-wide analysis of HouseResolution 2454 by using twomodels: ADAGE and IGEM (EPA, 2010a). Themodels differ on severalstructuralfronts.ADAGEisacalibratedmodelandIGEMiseconometricallyestimated.ADAGEhasamoredetailedrepresentationoftheenergysectorandenergysectortechnologies;IGEMhasmorenon-energysectordetail.
5 AnalyticalApproaches
Uncertainty associatedwith engineering cost and health and environmental benefit inputs, analyticalassumptions, andmodel parameters and structure,may in some caseswarrant additional analysis tobetterexploretheexpectedimpactsofapolicy.Insomecases,thespecificsoftheregulationinquestionmay informwhether theadditional resourcesrequiredtoperformadditionaluncertaintyanalysis isofsufficient value. For example, Circular A-4, which “provides the Office ofManagement and Budget’s(OMB's) guidance to Federal agencies on the development of regulatory analysis,” suggests a formalquantitativeanalysisof relevantuncertaintiesbeconductedformajorrulesthatareexpectedtohave“annualeconomiceffectsof$1billionormore”(OMB,2003).Thissectionconsidersdifferentquantitative
14
approaches for considering uncertainty in policy analyses conducted using CGE models, includingsensitivityanalysis,formaluncertaintyanalysis,andinter-modelcomparisons.Foreaseofexposition,wedelineate sensitivity analysis and formal uncertainty analysis, by defining the former as analyses thatconsiderscenariosinwhichoneormoreinputsarevariedoutsideofawell-definedprobabilityspaceandthelatterasanalysesthatattempttocharacterizethedistributionofmodelresults.
5.1 SensitivityAnalysis
Themostcommonapproachtotestingtherobustnessofmodelingresultsistovaryanexogenousvariableor parameter and solve the model to obtain the new set of endogenous state variables under thealternative parameterization. In this paper, such an approach is defined as sensitivity analysis todistinguishitfrommoreformaluncertaintyanalysis(Section5.2)andinter-modelcomparisons(Section5.3).Weusethetermsensitivityanalysisas inclusiveofcomparativestaticsanddynamics,alongwithsituationsinwhichthemodelsallowfortheexistenceoftemporarydisequilibrium.
Itiscommonforsensitivityanalysistovaryasingleparameterinthealternativescenariostoisolatetheimpactithasontheresultsofinterest.Thisapproachallowsanalyststogatherinformationabouthowrobustqualitativeconclusions,basedonmodelingresults,arewithrespecttospecificassumptionsorkeyparameters.EPA’sGuidelinesforPreparingEconomicAnalysis(2010b)suggeststhat“[i]ncaseswherethedataareuncertain,ornoteasilyquantified,butmayhaveasignificantinfluenceontheresults,theanalystshould describe the weaknesses in the data and assumptions, and include some type of sensitivityanalysis.”
Sensitivity analysis that varies only one, or a few, parameters at a time could potentially provide anincomplete characterization of the uncertainty surrounding the results due to important interactionsbetweenparameterswithincomplexandnon-linearCGEmodels(Ableretal.,1999).Byitsnature,basicsensitivityanalysisislimitedbythefactthatisnormallyconductedoutsideofawell-definedprobabilityspace that guideswhich parameters to vary, and themagnitude bywhich they should be perturbed.However,incaseswithincompleteinformationaboutthecompletedistributionalparameterspaceand/orlimitedresourcesandtime,basicsensitivityanalysismayprovideusefulinformationastotherobustnessofmodelingresultsandprovidevaluableinformationfordecisionmakers(Pannell,1997).Furthermore,asnotedbyOMB(2003)inCircularA-4:
“Insomecases,thelevelofscientificuncertaintymaybesolargethatyoucanonlypresentdiscretealternativescenarioswithoutassessingtherelativelikelihoodofeachscenarioquantitatively.Forinstance,inassessingthepotentialoutcomesofanenvironmentaleffect,theremaybealimitednumberofscientificstudieswithstronglydivergentresults.Insuchcases,youmightpresentresultsfroma rangeof plausible scenarios, togetherwith any available information thatmighthelp inqualitativelydeterminingwhichscenarioismostlikelytooccur.”
While this description reinforces the benefits of sensitivity analysis, it also highlights the challengespresent in conducting sensitivity analysis, as concepts such as a range of plausible scenarios are notformallydefined.However,itispossibletobringsomestructuretosensitivityanalysis.Forexample,theCongressionalBudgetOffice(CBO)approachessensitivityanalysisinitsmacroeconomicestimatesusing
15
a defined process to address the lack of structure. Specifically, in its dynamic scoring approach CBOdetermines the twoparameters in themodel towhich the results aremost sensitive. Then potentialestimatesaregeneratedbyexaminingeachcaseinwhichthosetwoparametersareattheendsoftheirrangesandotherparametersareequaltocentralestimates,whereCBOultimatelyreportsthecasesthatshowthemostandleastfavorablebudgetaryoutcomes.5ItisalsoworthnotingthatCBOdoesnotconductaquantitativeuncertaintyanalysisforeachbillconsiderediftherangederivedfromtheanalysiscouldpotentiallybemisinterpreted.Forexample,incaseswheretheunderlyingconventionalcostestimateisassociated with significant uncertainty relative to the uncertainty in the macroeconomic model,conductingsensitivityanalysisoveronlytheparametersinthemacroeconomicmodelcouldproducearange of estimates that should not be interpreted as characterizing the full range, given significantadditionaluncertaintyintheconventionalcostestimate(CBO,2015).
SensitivityanalysismayalsobeusedtoexamineuncertaintieswithrespecttothestructureofCGEmodelsandhowthatmayaffecttheresultsofpolicyanalysis.Unlikeparametricsensitivityanalysis,structuralsensitivityanalysisexaminestheeffectofchangestotheunderlyingequationsinthemodel.Thistypeofanalysistypicallytakesplacewithinasinglemodel.Ingeneral,structuraltestingismoreresourceintensivethanparametricsensitivityanalysis,thoughthisdependsuponthetypeofstructuralchangeandtherangeofparametricuncertainty.
5.2 FormalUncertaintyAnalysis
Sensitivityanalysisofafewselectparameters,asdescribedinSectionError!Referencesourcenotfound.,mayallowananalysttogetageneralsenseofhowthemodel’sresultsmaydependonkeyparametervalues. However, due to the lack of amethodological structure, which leads the approach to ignoreinformationcontainedinthecovarianceofuncertainparameters,thisapproachisnecessarilyimprecise(Bernheimetal.,1989).Asamorestructuredalternative,modelershaveconsideredformaluncertaintyanalysisthattakesintoconsiderationadditionalinformationabouttheuncertainparameters.6
Thecurrentclassofappliedgeneralequilibriummodelsusedforpolicyanalysishavehundredsofinputparameters.Evenforarelativelysmallmodelwitheightparametersthateachhavefivepotentialvalues,ifthemodeltookoneminutetosolveandsavetheresultsacompletefactorialexperimentdesignthatconsidered every possible combination of parameterswould require ninemonths of processing time(Pannell,1997).ItislikelythatwithmoderncomputingresourcesmodelsthatsolveinoneminutelikelyhavefarmorethaneightuncertainparametersmakingPannell’sexampleevenmoreapt.Itisalsolikelythatparametersaremoreappropriatelycharacterizedbydistributionsthanbyadiscreteset,as intheexampleset forthbyPannell (1997).Therefore,Gaussianquadrature (e.g.,Herteletal.,2007),MonteCarlo(e.g.,Selinetal.2009),andlinearization(e.g.,Jorgesonetal.,2013)methodsaremorecommonlyusedtoconductformsofprobabilisticanalysis.Thesetechniquesprovidewaysofapproximatingintegralsoveruncertainmodelparameterstoobtainexpectedvaluesandconfidenceintervalsformodelresults.
5http://www.slideshare.net/cbo/dynamic-scoring-at-cbo-51635156
6Approachestomoreformalandstructureduncertaintyanalysishavealsobeendescribedas“systematicsensitivityanalysis.”
16
Thesebenefitscomeatthecostofadditionalcomputationandneedtodefineprobabilitydistributionsover theuncertainmodels inputsand/orparameters.This sectiondescribes thesevariousmethodsofuncertaintyanalysisandexamplesinwhichtheyhavebeenapplied.
Theendogenousvariables,Y,inaCGEmodelmayberepresentedastheresultoftheimplicitfunction
( ), ,Y H X β θ= , (1)
where X areexogenousvariablesandknownparameters, includingpolicyvariables, β areuncertain
parameters of the model, excluding calibration parameters which are defined by θ . The calibration
parameters,θ ,arechosensuchthatthemodelreplicatesbenchmarkconditions, 0Y ,suchthat
( )0 0, ,h X Yθ β= , (2)
where
( )( )0 0 0 0, , , ,H X h XY Yβ β= . (3)
Inmostcases,thegoaloftheanalyst istodeterminetheexpectedchangeinendogenousvariablesof
interestgivenachangeinpolicyparameters,representedasamovementfrom 0X to 1X ,anduncertainty
inthemodel’sparameters β ,
[ ] ( ) ( )( )
1 0 1 0
1 0
, , , ,
, ,
E Y Y E H X H X
E H X Y
β θ β θ
β θ
− = −⎡ ⎤⎣ ⎦
= −⎡ ⎤⎣ ⎦. (4)
GiventhenonlinearnatureofmostCGEmodelsitwillbethecasethat
( ) [ ]( )1 1, , , , ,E H X H X Eβ θ β θ≠⎡ ⎤⎣ ⎦ , (5)
such that evaluating the change in model output under the policy at the expected values of theparameterswill not be equal to expected changewhen fully considering the parametric uncertainty.Probabilistic analysis allows analysts to providemore robust assessments of the expected change in
economicvariablesbyintegratingoverthejointdistributionoftheuncertainparameters, ( )g β ,
[ ] ( ) ( )1 1 , ,E Y H X g dβ θ β βΩ
= ∫ (6)
andprovideavarianceassociatedwithmodel’sresultsbyevaluating
[ ]( ) ( ) [ ]{ } ( )2 2
1 1 1 1, ,E Y E Y H X E gY dβ θ β βΩ
⎡ ⎤− = −⎣ ⎦ ∫ . (7)
GiventhenatureofappliedCGEmodels,analyticalsolutionsfor(6)and(7)donotexistandnumericalapproximationsarerequired.Acommonapproachfornumericallyapproximatingasamplingdistribution
of 1Y fromwhich samplemoments can be calculated is aMonte Carlo simulation. The basic process
17
involvesconductingN simulationsof 1Y ,whereeachtimeanewsetofparameters, ˆnβ ,isdrawnforthe
jointprobabilitydistribution ( )g β .Theexpectedvaluesoftheendogenousvariablesin(6)maythenbe
estimatedbythesamplemean
[ ] ( )1 11
1 ˆ, ,N
nn
E Y H XN
β θ=
≈ ∑ , (8)
withthevariancecalculatedanalogously.Theaccuracyoftheapproximationwill,inpart,dependonthe
number of simulations conducted, with the error being of order 1 N and therefore requiring a
potentiallylargenumberofsimulations.
Incasesofhighlycomplexmodels,orrelativelyscarcecomputingresources,thenumberofsimulationsrequiredtogetreliableestimatesofthemomentsaroundthevariablesofinterestcanbeofconcern.Toaddress the computational burdens,Harrison andVinod (1992) suggest approximating theunderlyingprobabilitydistributionwithadiscretesetofpointsandprobabilities,allowingfor fasterconvergence.However, their approach of choosing the points for each parameter by dividing the support of themarginalprobabilitydistributionintoafinitenumberofequi-probableregions, isknowntounderstatethevariance(andallotherhigherorderevennumberedmoments)oftheparameterdistribution(MillerandRice,1983).ThisbehaviorisinheritedbytheapproximatejointdistributionoftheparametersandwhenusedintheMonteCarloapproachwillbiastheresults(DeVuystandPreckel,1997).
AsanalternativetoMonteCarlosimulationswithCGEmodels,Ardnt (1996)andDeVuystandPreckel(1997)introduceanapproachbasedontheGaussianQuadraturetoapproximatetheintegralsin(6)and(7).Thegeneralprincipleistocarefullychooseaseriesof J weights, jw ,eachassociatedwithasetof
parameters, jβ ,fromthejointprobabilitydistribution,suchthattheexpectationmaybeapproximated
as
[ ] ( )1 11
, ,J
j jj
E Y w H X β θ=
≈∑ , (9)
where1
1Jjj
w=
=∑ (withsimilarapproximationsholdingforhigherordermoments).Bycarefullyselecting
thesetsofparametersandcorrespondingweights,theanalystmay,insomecases,beabletoapproximatethemomentsaroundthemodeloutputswithfarfewermodelrunsthanundertheMonteCarloapproach,J N< .Ingeneral,thebasisforselectingtheweightsandcorrespondingparametersistoensurethatthemomentsaboutzero,uptoagivenorder,wouldbeadequatelyintegratedusingtheselectedvalues.Byusing higher order quadrature techniques greater accuracy may be achieved when integratingcomplicatedsystemsofnon-linearequations,suchasaCGEmodel,thoughtheabilitytoapproximatetheintegraldependsontheabilityofapolynomialtoapproximatethecurvatureofthemodel.
In their original application,DeVuyst andPreckel (1997) use a global, energy-explicit CGEmodelwithuncertainfossilfuelsupplyelasticitiestoevaluatetheimpactsofacarbontax.Theauthorsdemonstrate
18
theapproachwithsixuncertainparametersfromindependentdistributions,andfindthattheycanobtainareasonableapproximationof themeanandstandarddeviationof themodel’sendogenousvariableswithrelativelyfewmodelruns.Sincetheiroriginaldemonstration,GaussianQuadraturehasbeenusedtoestimate expected values and confidence intervals for key results fromCGEmodels inmore complexcontexts.Forexample,Herteletal.(2007)usethetechniquetoderivemeansandconfidenceintervalsfromaglobalCGEmodelwithdozensofuncertainparameters,whenstudyingthe impactof reducinginternationaltradebarriers.However,toapplythemethodtheyassumethatalluncertainparametersarenormallydistributedandindependent.
Forcommondistributionsandinthecaseofindependentuncertainparameters,techniquesforselectingweightsandparametricvaluesarereadilyavailable(MirandaandFackler,2002).TheoriginalapplicationusingthistechniquewithCGEmodelsfocusedoncaseswithsymmetricandindependentdistributionsforthe uncertain parameters. For the case of the non-symmetric distributions, non-diagonal covariancematrices,and/orlargeparameterspacestechniquesforGaussianQuadraturemaynotbereadilyavailableormaynotprovidethesamecomputationalimprovementoverMonteCarlosimulations.Suchlimitationsmaybe important, as Jorgensonetal. (2013)demonstrates thatassuming independenceofuncertainparameters in a CGE model can significantly bias confidence intervals around endogenous variablesdownward. However, Horridge and Pearson (2011) demonstrate an approach to applying GaussianQuadrature in the context of CGEmodels in cases with non-diagonal covariancematrices across theuncertainparameters.TheauthorsdemonstratethefeasibilityofthemethodusinganexamplefromtheGlobalTradeandAnalysisProject.
AnotheralternativetoMonteCarlosimulationsisbasedonthelinearizationoftheCGEmodel(e.g.,Paganand Shannon, 1985;Wigle, 1991). This approach is sometimes referred to as the Deltamethod. Forexample,Bernheimetal.(1989)andTuladharandWilcoxen(1998)usethisapproachtoderiveconfidenceintervalsaroundendogenousstatevariables inaCGEmodel.Themethodseekstoprovidetheanalystwith a confidence interval around the endogenous variable evaluated at the central tendency of theuncertain parameters. Themethod is based on linearizing themodel using a first order Taylor seriesexpansion around the endogenous variables with respect to the model’s parameters.7 Under thisapproximation,acovarianceformodel’spointestimateisderivedastheproductofthemodel’sJacobianwithrespecttoitsparametersandthecovariancematrixfortheparameters.Usingthenotationabove,themodelinlinearizedas
[ ]( ) ( )1 1, , , ,H X E H X J Jβ β ββ θ β θ ʹ≈ + Σ , (10)
where Jβ is the Jacobianof theCGEmodelwith respect to theuncertain parameters and βΣ is the
covariancematrixassociatedwiththeuncertainparameters.Thesecondtermin(10)providestheanalyst
7 Higher order approximationsmay also be applied to caseswhere there is significant curvature in the implicit
functionrelatingthelevelorchangeintheendogenousvariabletotheuncertainparameters(Bernheimetal.,1989).
19
withacovariancefortheendogenousvariablesbasedonthemodel’ssensitivitytothoseparametersandtheirdegreeofuncertainty.
Given the complexity ofmost applied CGEmodels, the Jacobian is computed bymeans of numericaldifferentiationandinmostcasesmodelershaveusedabasicforwardfinitedifferencetominimizethenumberofmodelsolvesrequiredtoderivetheJacobian.Jorgensonetal.(2013)haveshownthat,eveninthecaseofcomplexdynamicCGEmodels,themethodcanprovidestandarderrorsasapercentageofthemodel’sendogenousvariablesthatarereasonablyclosetothestandarderrorsasapercentageoftheexpectedendogenousvariablesinthecaseofMonteCarlosimulations.Theirresultssuggestthatinsomecases,dependingonthemodel, theDeltamethodmaybeabletoderiverelativestandarderrorswithfewersimulationsthanaMonteCarloapproach.
With the increased availability of computing resources and simulation techniques, Monte Carlosimulationshavebeenusedinnumerousrecentstudiestoexplorethesensitivityofmodelingresults(e.g.,Websteretal.,2001;Sokolovetal.,2009;Elliotetal.2012a;Elliotetal.2012b).Forexample,Elliotetal.(2012a)takeadvantageoflarge-scaleparallelprocessingtoconductMonteCarlosimulationsthatexplorethesensitivityofCGEmodelresultstoparametricuncertaintyinthecaseofaglobalCGEmodel.8Notably,they find that thebias associatedwithmodel results basedon running theCGEmodel at the centraltendencyofallparametersversusthecasewithfulluncertaintyinparametersissmall.However,theyfindthevariancearoundtheCGEresultswasstillsignificant.Inabootstrapanalysisoftheirresults,Elliotetal.(2012a) find that the number of simulations required to obtain information about the shape of thedistribution around endogenous variables might vary considerably depending on the uncertainparametersconsideredandtheendogenousvariablesofinterest.Forexample,Elliotetal.(2012a)findthatendogenousvariableswithlowercoefficientsofvariationrequirefewersimulationstoadequatelycapturethemeanandvarianceoftheresult,comparedtovariablewithrelativelyhighercoefficientsofvariation.
BasicMonteCarloanalysisreliesonthestronglawoflargenumbersandthecentrallimittheoremforitsconvergenceproperties.Intermsofestimatingthemeanoftheendogenousvariablestheapproximationerrortendstozeroasthenumberofsimulationstendstoinfinity;however,inpracticetheapproximationerrorisrandomduetotheprobabilisticnatureoftheanalysisandcanbelargeevenwhenthenumberofsimulationsislarge.InrecentMonteCarlostudies,researchershaveusedquasi-MonteCarlotechniquestomoreefficientlysampletheparameterspaceandthereforereducethenumbersimulationsrequired.The goal of these approaches is to impose additional structureon themethodused todraw randomparametersfortheanalysis,suchthattheapproximationerrorisreducedbeyondwhatitwouldbeinabasicMonteCarloanalysis.9Forexample,Websteretal.(2001)useLatinHypercubesamplingtoreducethenumberofrunsrequiredforadesireddegreeofaccuracy.
8 Similar to other studies, Elliot et al. (2012a) assume the uncertain parameters are normally distributed and
independent.
9SeeJudd(1998)fordetailsonquasi-MonteCarlotechniques.
20
Building uponMonte Carlo techniques, variance decompositionmethods, such as the Sobol method(Sobol,1993),producesensitivityindicesthatestimatethefractionalcontributionofaninputparameter(andparameterinteractions)tothevarianceofanoutput.Thesemethodsallowthemodelertotracethedependenciesbetweenoutputsandinputs.However,theycanbecomputationallyintensive(IoossandLemaitre, 2015). Although variance decomposition methods have been employed in environmentalmodelingandchemicalengineering, theyhavenotbeenwidelyused inconjunctionwithCGEmodels.However,inoneexampleMohora(2006)employstheSobolmethodinastudyoftheRomanianeconomyand finds that the variance decomposition identified that indirect effects and database structurecontributemost tooutputvariance. Interestingly,Mohora foundtheseresults tobe largelyconsistentwithasimplerparameterscreeningmethodusedinthestudy.
5.3 Inter-ModelComparisonExercises
Inter-modelcomparisonexercisesareameanstoexaminetheeffectofbothstructuralandparametricdifferences betweenmodels. These exercises convene severalmodeling teams to examinepolicy andmodelingissuesbycomparingmodelresultsforastandardizedsetofscenarios.Inter-modelcomparisonexercises can servemany purposes. The exercises bring togethermodelers from academia, industry,government,andotherorganizationstoexamineanissuefromdifferentperspectivesandusingdifferenttools. The section will discuss the basics of inter-model comparisons projects, past examples, andimportantconsiderationstotakeintoaccountwhendesigningsuchexercises.
Inter-modelcomparisonexercisesarestructuredaroundasetofpolicy-relevantresearchquestions.Eachofthemodelingteamsproduceresultsforacommonsetofscenarios(e.g.,withandwithoutapolicyoratechnology).Thescenariosandresultsaredevelopedandanalyzedtypicallyoveraseriesoftwotothreeworkshopsheld severalmonthsapart. Theseexercises serve several valuable functions. First, amulti-modelapproachtoatopicmayhighlightareasofrobustagreementaswellasidentifyandthekeyfactorsdrivingdisagreementacrossmodels.Second,theexercisehelpstoexplainthestrengthsandlimitationsofalternativemodelingapproaches.Third,theexercisesprovideaforumformodelerstoreceivepeerfeedbackonmodelingissuesthatrangefromdataandparameterselectiontostructuralspecificationsandmodelingtechniques.Finally,theexercisesidentifyfutureresearchandmodeldevelopmentneeds.
Thelongestrunningandawell-knownseriesofinter-modelcomparisonsisStanfordUniversity’sEnergyModelingForum(EMF)co-directedbyJohnWeyantandHillardHuntington10.Beguninthemid-1970’sEMFhascompletednearly30projectsonissuesrelatedtoenergyanditsrelationshiptotheeconomyandenvironment.Pastprojectshaveexploredtheoil,gas,coal,andelectricpowermarkets.EMFexerciseswithinthelastfifteenyearshaveemphasizedtheinteractionbetweenenergymarkets,climatechange,andclimatechangepolicy.Severalrecentinter-modelcomparisonexerciseshavealsofocusedonenergy-economy-climateinteractions(seee.g.,AMPERE(Kriegler,etal.2015)andLIMITS(Kriegleretal.,2013)andagriculture-economy-climateinteractions(seee.g.,AgMIP(Rosenzweig,2013)).
10Seehttps://emf.stanford.edu/.
21
Although inter-model comparison exercises are valuable to both the policymaking and modelingcommunities,thereareimportantchallenges.Forexample,itcanbedifficulttodiscernthekeyfactorsthatleadtotherangeofresultsacrossmodelsbecausethemodelsmaydifferbytype(e.g.,generalvs.partialequilibrium)inadditiontostructuralandparametricdifferences.OnenotableexceptiontothisisEMF29:TheRoleofBorderCarbonAdjustmentinUnilateralClimatePolicy(Bohringeretal.,2012).EMF29wasentirelycomprisedofCGEmodelsthatusedthesameunderlyingdataset(GTAP7.1).Inaddition,cautionshouldbetakenwheninterpretingtherangeofresultsfromcomparisonexercisesbecausetherangetypicallydoesnotrepresentthefullrangeofuncertainties.Instead,therangeofresultsrepresentthe uncertainties across the participating models and possibly over a handful of parameters orassumptions in the study. Finally, a study may not necessarily reflect all models as participation isvoluntaryandmaynotbesupportedbyexternalfunds.
6 PresentationofUncertaintyAnalysis
Whileadditionalsensitivityorformaluncertaintyanalysiscanprovideuseful informationaspreviouslydescribed,italsopresentsachallengetoanalystswhomustsummarizethisadditionalinformationinaformat that is readily understood by policy makers and stakeholders. When uncertainty analysis isconducted,CircularA-4guidesanalyststotryto“providesomeestimateoftheprobabilitydistributionofregulatory benefits and costs” (OMB, 2003). However, there are different formats in which suchinformationcanbeconveyed.Informationabouttheprobabilitydistributionofbenefitsandcostscanbepresentednumerically,verballyorgraphically(NAS,2013).
Anumericalrepresentationmayincludesummarystatistics,suchthecentraltendency(e.g.,meanandmedian), variances, confidence intervals, or other potentially relevant characteristics. A numericalpresentationoftheresultscanprovideasignificantamountofinformation,however,theapproachmayrequirethattheaudiencehavetheappropriateexpertisetounderstandandinterprettheinformation.Verbalrepresentationsconveyinformationusingcuessuchaslikelyorunlikelytodifferentiatepotentialoutcomesandcharacterizetheprobabilitydistributionofbenefitsandcosts.Whileverbalrepresentationsof uncertainty may be more approachable to a broader audience, there is the potential for theinterpretationtodifferacrossindividuals.TheUnitedNationsIntergovernmentalPanelonClimateChange(IPCC)hasattemptedtoaddressthisconcernbyestablishingspecificdefinitionsanduseguidelinesfortheverbalcharacterizationintheirassessmentreports(IPCC,2010).Graphicalrepresentationsofprobabilisticinformation have the potential to convey more information than basic verbal cues while remainingaccessibletobroaderaudiences.Graphicalrepresentationscantakedifferentformssuchashistogramsorcolorwheels.Forexample,Websteretal.(2008)presentsomeresultsoftheirprobabilisticCGEanalysisasprobabilitydistributions.Figure1,whichisfromtheiranalysis,presentsthelossinglobalconsumptioninagivensimulationyearacrossdifferentpolicystringencies(i.e.,Level1through4).
22
Figure1:DistributionofConsumptionLossacrossStringenciesinEPPACGEModel(Websteretal.,2008)
Inthecaseofdynamicanalyses,itcanbeusefultodepicttheresultsinawaythatprovidesinformationas tohowthedistributionof resultschangesover time.Forexample, in theirCGEanalysisElliotetal.(2012a)presentthemeanestimateofakeyoutputvariable,CO2emissionsintensity,overtimealongwiththedifferentconfidenceintervalsthataredifferentiatedbytheirshading(Figure2).
Figure2:CarbonIntensity[kgCO2/2004$ofGDP]fromCIM-EARTHCGEModel(Elliottetal.,2012a)
Awell-known example of colorwheels to convey probabilistic information is thework byMIT’s JointProgramontheScienceandPolicyofGlobalChangeoncommunicatinguncertaintysurroundingclimatechange (Figure3).11Thisplotpresents theprobabilityofdifferentclimateoutcomeswithandwithoutpolicy as two different pie charts where the area of the slices corresponds to the probability of theoutcome conditional on the policy assumption associated with the pie chart. The graphic eases
11Seehttp://globalchange.mit.edu/focus-areas/uncertainty/gamble.
23
interpretationoftheoutcomesandcomparisonacrossthetwoscenariosbycoordinatingthecolor-codingofthepieslices.
Figure3:MITGreenhouseGambleRepresentationofClimateChangeUncertainty11
Inadditiontopresentingtheresultsofuncertaintyanalysis,analystsmaychoosetoprovideadditionalinformationastothesourcesofuncertaintymostrelevanttotheresults.Forexample,Tornadoplotsmayalsobeuseful in communicating the resultsof sensitivityanalysis to showtheparametersofgreatestinfluenceovertheresultsofinterest.Jacobyetal.(2006)conductastudywiththeEPPACGEmodelinwhichtheyvarykeyparameters,onebyone,byplusandminusonestandarddeviation.Theythenplotthe effect of these experiments on thewelfare change of a given policy in order of greatest to leastinfluence(Figure4).
24
Figure4:ChangeinWelfareforaPolicywithPlusandMinusOneStandardDeviationChangeintheParameterwiththeEPPACGEModel(Jacobyetal.,2006)
However, tornadoplotscanalsobeusedtopresenttheresultsofprobabilisticanalyses.Forexample,standardizedregressionsmaybeusedtoanalyzetheresultsofMonteCarlosimulations,asdescribedinSection 5.2, to determine the influence of different parameters on the results of the analysis. Afterstandardizing the draws for the input parameters, they may be regressed on the output variable ofinterest to obtain standardized regression coefficients that represent the impact of a one standarddeviation change in the parameters on the output variable of interest. The results of such additionalanalysisareoftenpresentedintheformoftornadoplots.
7 VerificationandValidationExercises
The methods of conducting and presenting uncertainty analysis discussed in Sections 5 and 0 areapproaches to providing decision makers and stakeholders with additional information about therobustnessofmodelresultstohelpinstillconfidenceinthequalitativeconclusionsofpolicyanalysisbasedon modeling results. Additional confidence in the model results may be provided through formalapproaches to model verification and validation. Model verification and validation represent twoimportant, interrelated activities in the CGE model building process. Carson (2002) defines modelverificationasa“processandtechniquethatthemodeldeveloperusestoassurethathisorhermodeliscorrect and matches any agreed-upon specification and assumption.” Sargent (2013) defines modelvalidationasthe“substantiationthatamodelwithinitsdomainofapplicabilitypossessesasatisfactoryrange of accuracy consistent with the intended application of the model.” This section discussestechniquesformodelverificationandvalidationwithafocusonCGEmodels.
25
7.1 ModelVerification
Verification of a CGEmodel constitutes the initial phase of themodeling project (Carson, 2002). Thisprocessinvolvesanumberofiterativeproceduresthatinclude,amongotherthings,modelspecification,dataorganization,anddebuggingofthemodelcode.Themodelerwillusuallyperformanumberoftestsandsimulationstoexaminebasicfeaturesoftheunderlyingdataandeconomicstructure.Forexample,acommoninitialexperimentattemptstotestifthemodelcodeandtheimbeddedassumptionsreplicatethebenchmarkdataset.Ifthemodelisspecifiedcorrectly,theequilibriumsolutiontothemodelwithnoshocksshouldbeidenticaltothebaselinedata.Thischeckcanbeperformedateachstageofthebaselinesettingprocess.Forexample,RauschandRutherford(2008)describeasetofprogramsthattransformIMPLANdataontheU.S.economyintoaformatthatcanbeusedinaCGEmodelimplementedintheGAMS/MPSGEmodelinglanguage.Aftereachstepintheprocess,thebenchmarkistestedinasmall-scale,highlyaggregated,open-economymodel.Thesestepsincludeaggregatingthedata,translatingparameternames,diagonalizingthedatasothateachindustryproducesonlyonecommodity,mergingstateleveldata, and balancing interregional trade flows. If the benchmark is not replicated at any point, theprogramsstopsandanerrormessagepromptstheusertocorrecttheimbalancebeforemovingon.
DixonandRimmer(2013)describeabasichomogeneitytestthatmaybeperformedinthedevelopmentofsomeCGEmodels.Ifallmodelsectorsarerepresentedwithconstantreturnstoscale(CRTS)productionfunctions, then increasing the exogenous nominal variables by a percentage should return the samepercentagechangeintheendogenousnominalvalues.For instance,doublingthecostsofall inputstoproductionshouldresultinadoublingoftheunitcostofoutput.Ifthemodelcontainsrepresentationsofnominalrigidities,suchasstickywages,adherencetohomogeneityshouldnotbeexpected.Theeffectsof these deviations can be checked by turning off the features that cause the rigidity, such as arepresentationofstickywages,totestifthefeatureiscausingthedeviationfromthehomogeneouscase.Otherdeviationsinthehomogeneitytestmayrepresenterrorsinthemodel.
CGEmodelverificationcanalsobeconductedusingGDPaccounting identities.GDPcanbecomputedfromCGEmodeling resultsbyaddingup total incomesor totalexpenditures,wherebothcalculationsshouldproducethesamevalue.Eachof thesecalculationscanalsobedone inrealornominal terms.Thesechecksareusefulbecausetheyinvolvedifferentsetsofmodelvariables,andarethereforetestingfor consistency across themodel. The expenditure calculation involves themacroeconomic variablesconsumption, investment,governmentexpenditures,andnetexports.The incomecalculation includesfactorincomesforallagents,transferpayments,andtaxrevenues.
Thesocialaccountingmatrix(SAM)usedtocalibratethemodelprovidesanadditionalcheckonthemodel.TheSAMincludesallpaymentsbetweenindustriesandagents,aswellasinternationaltradebalances,andshouldbalancebeforeandafterapolicyshock.Thismeansthatthezero-profit,market-clearance,and income balance conditions hold for each industry, good, and agent in the model. (For moreinformationonSAMssee,interalia,PyattandRound(1985).)
26
7.2 Modelvalidation
While it has been argued that a simulation model can never be completely validated (Gass, 1983),subjectingamodeltovalidationtests–andmakingappropriateadjustmentstothemodelifnecessary–canincreaseconfidenceintheusefulnessofthemodelforpolicyanalysis.Anumberofmethodsformodelvalidation have been used by CGE modelers. This section describes several of these methods and anumberofexercisesthathavebeenconductedtovalidate(orinvalidate)CGEmodels,alongwiththecostsofmodelvalidation.Validationexercisescouldbeperformedwhenamodelisfirstdeveloped,atthetimeofmajorrevisions,orsimplyovertimetocheckthemodelhascontinuedvalidity.
7.2.1 BackoftheEnvelopeModels
DixonandRimmer(2013)suggestthatbackoftheenvelope(BOTE)modelscanbeusefultoolsforCGEmodelvalidation.Theseestimatescangiveasenseofthemagnitudeoftheexpectedresultsfromthefullmodel.ABOTEmodelsimplifiesthelarge-scalemodeltogaininsightonmodelresults.Thissimplificationcanbedonebyaggregatingthenumberofindustries,factorsofproduction,and/oragentsinthemodel.Afterasimplemodeliscalibratedtotheaggregateddata,themodelercanaddpolicyvariablestotesttoseeiftheexpectedmagnitudeofchangesinthemodelresultsareproduced.Next,themodelercanaddone featureata timeto theBOTEmodel to test theperformanceof these featuresof the large-scalemodelandtogaininsightabouthoweachfeatureinfluencestheresultsofsamplescenarios.
SimpleBOTEestimatesoftheGDPimpactsofapolicyshockcanbecalculatedusingdataandparameters,suchasindustrysize,taxrates,andelasticities.EachofthecomponentsoftheGDPcalculationcanbeusedtohelpexplainmodelresultsandtheoverallimpactofapolicyonGDP.Policiesthataffecteconomicactivity can influence the componentsofGDP indifferentdirections.Wemayexpect that a rule thatrequiresinstallationofnewcapitalwouldincreasetheinvestmentcomponentofGDPanddecreasetheconsumptioncomponent.OncethemacroeconomicimpactsofthesimplifiedmodelarewellunderstoodwiththeBOTEmodel,thefullCGEmodelcanbeusedtounderstandtheeffectsonspecificindustriesandhouseholds.
7.2.2 StatisticalValidationofModelParameters
DixonandRimmer(2013)suggesttheuseofstatisticaltechniques,suchasregressionanalysis,toimprovetheunderstandingofmicroeconomicresults.Theyusetheexampleofananalysisofemploymentchangesunder a scenario in which import tariffs and quotas are removed. The USAGE CGE model forecastsemploymentchangesbystate,whicharethencomparedtoaregressionanalysisthatseekstoforecastthesamechanges.Theregressionanalysiswaspreparedforthisvalidationexerciseusinghistoricaldatatoestimatearelationshipbetweenthetradepolicyandemployment.Byrankingtherelativeimpactsbystateandcomparingtheserankingsacrossthetwomodels,theregressionexerciseallowedtheauthorstoseewherestateswererepresenteddifferentlyineachmodelandultimatelyimprovetheCGEmodel.
Ideally,CGEmodelsshouldusegeneralequilibriumelasticitiesthataccountforthedegreeofflexibilityavailableacrossmanypartsoftheeconomy.Valuesthatareestimatedinapartialequilibriumframeworkcancausethemodeltobetooresponsiveornotresponsiveenoughtopolicyscenarios,whencomparedtohistoricaloutcomesanalyzedduringthevalidationprocess.
27
7.2.3 CGEModelValidationExercises
Althoughthepublished literature isrelativelysparse,anumberofauthorshaveundertakenvalidationexerciseswith theirmodels. Recent efforts have used increasingly sophisticated techniques thatmayallowforaprocessbywhichsignificantmodelingimprovementscanbemade.
Kehoeetal.(1995)comparedresultsfromtheirCGEanalysisoffiscalreforminSpain–replacementofacomplexsystemofindirecttaxeswithavalue-addedtax–during1985-1987withthechangesthatactuallyoccurred.Theycomparedchangesinconsumerandindustrialprices,industrialoutput,andanumberofmacroeconomic variables using two performance metrics: the weighted correlation coefficient andweightedR2.Kehoeetal.(1995)reportthattheirmodelperformedwell,particularlywhentheyincludedtwoexogenousshocksthataffectedtheSpanisheconomyduringtheperiodofanalysis(afallinthepriceofpetroleumandadeclineinagriculturalproductivityduetoadverseweatherconditions).Inaddition,they found that the results were robust to alternative specifications of the labor market andmacroeconomicclosurerules.
Kehoe (2005) evaluated the performance of threemultisector static CGEmodels used to predict theimpactoftheNorthAmericanFreeTradeAgreement(NAFTA).Kehoe(2005)reportsthatallofthemodelsdramaticallyunderestimatedtheimpactofNAFTAontradeamongsttheU.S.,Canada,andMexico.Themodelswerealsoreportedtohaveonlybeenabletocapturerelativeportionoftheimpactsindifferentsectors.Kehoe(2005)arguedthat,basedonhisanalysis,anewtheoreticalmechanismwasneededforgeneratinglargeincreasesintradeinproductcategoriesthatpreviouslyhadlittletrade.Inaddition,inorder to capture changes inmacro aggregates,models need to be better able to capture changes inproductivitybroughtaboutbytradeagreements.Whilethiseffortdidnotexplicitlyresult inmodelingimprovements, this retrospectiveanalysiswasable to identifypotentialareasof improvements in themodels.
Valenzuelaetal.(2007)developedamethodologyforvalidatingCGEmodelsonasector-by-sectorbasis,focusingonthewheatmarketintheGTAPmodel.Theyemployedastochasticsimulation,usingshocksderived from a time-seriesmodel tomeasure the randomness in annual output over the 1990-2001period. The residuals were used to create a distribution reflecting random productivity variation byproducingregion.Theseproductivityshocksgeneratedendogenousfluctuationsinproductionthatmatchthose in the data. Solving the CGEmodel repeatedly while sampling from this distribution yielded adistributionofcorrespondingmarketpricechangesforwheat,byregion.Standarddeviationsbasedonthesemodeledoutcomeswerethencomparedtoobservedoutcomesforyear-to-yearpricechanges.TheauthorsfindthatwhentheGTAPmodelfails,ittendstodosoinasystematicway,under-predictingpricevolatilityfornetexportersofwheat,andover-predictingvolatilityforimportingregions.Usingtheinsightsgainedfromthisexercise,theauthorswereabletomakemodificationstothemodelspecificationthatimprovedthefittothefluctuationsinthetime-seriesdata.
28
UsingamethodologysimilartothatofValenzuelaetal.(2007),Beckmanetal.(2011)examinedtheabilityof the GTAP-E model to reproduce historical price volatility in the petroleum market.12 WhereasValenzuelaetal.(2007)focusedonthesupply-side,Beckmanetal.(2011)includebothsupply-sideanddemand-side shocks. For the supply-side,Beckmanet al. (2011) fit a time-seriesmodel to countryoilproductionoverthe1980-2005timeperiod.Forthedemand-side,ageneralindicatorofeconomicactivity,GDP,wasused.Time-seriesresidualswerethenusedtocreateprobabilitydistributionsforrandomshockstotheunderlyingsupplyanddemandschedulesforpetroleum.Usingtheseshocks,standarddeviationsofoilpricechangesbasedonGTAP-Emodelrunsarethencomparedtothosefromthehistoricaldata.TheoriginalGTAP-Emodelsignificantlyunderestimatedpetroleumpricevolatilityincomparisonwiththedata.Inresponse,Beckmanetal.(2011)conductedanextensiveliteraturesearchandupdatedrelevantmodelparameters.There-parameterizedmodelperformedsignificantlybetter.To further test there-parameterizedmodel,Beckmanetal.(2011)alsoperformedastylizedmedium-runsimulationinwhichthey shockedpopulation, labor, capital, investment,oilprices, andTFPbyobservedchangesover the2001-2006period.Withthenowmoreinelasticdemandspecification,themodelwasabletocapturethebroadchangesinthepetroleummarketincludinganincreaseindemandthatoccurreddespitethesharpincreaseinpriceduringthisperiod.
7.2.4 ChallengesofModelValidation
Sargent(2013)pointsoutthatsimulationmodelvalidationcanbetime-consumingandcostly,andthatitisnotfeasibletoensurethatalarge-scalemodelisvalidoveritsentiredomain.SeveraloftheexamplescitedinDixonandRimmer(2013)werestatedtohavetakenyearstocomplete,duetothetimerequiredto collect the historical data, perform numerous runs, and conduct detailed analysis. Sargent (2013)further suggests that validating amodel for one specific purpose does notmean it is valid for otherapplications,andthatadditionalvalidationmaybenecessary.Furthermore,inthecontextofregulatoryanalysisbackcastingexercisesmayfaceadditionalchallengesduetothenatureofCGEmodelsastoolsto estimate changes instead of levels, and the unobserved counterfactual baselines necessary tounderstandthehistoricalchangebecauseofthepolicy.
8 ConcludingRemarks
EPAhasahistoryofconsideringuncertaintyindifferentcontextsandusingavarietyofqualitativeandquantitativeapproachestodeterminetherobustnessofpolicyoutcomes.However,asoutlined inthiswhite paper, there may be additional sources of uncertainty associated with extending benefit costanalysestoaneconomy-wideperspective.Thisleadstoquestions,suchas:
• Are certain types of uncertaintymore of a concernwhen evaluating social costs, benefits oreconomicimpactsinaneconomy-wideframework?
12TheGTAP-EmodelisavariantofthestandardGTAPmodelthatincludesmoredetailinthespecificationofthe
energysector.
29
• Arechallengesor limitations related to theseuncertaintiesmoreofaconcern than forpartialequilibriumapproachestoestimation?
AsnotedbytheNASintheir2013report:
“Althoughsomeanalysisanddescriptionofuncertaintyisalwaysimportant,howmanyandwhattypesofuncertaintyanalysesarecarriedoutshoulddependonthespecificdecisionproblemathand the effort to analyze specific uncertainties through probabilistic risk assessment orquantitativeuncertaintyanalysisshouldbeguidedbytheabilityofthoseanalysestoaffecttheenvironmentaldecisionathand.”
When quantitative analysis of uncertainty is warranted in an economy-wide framework, there aremultipleapproachestoconductingsuchanalysis,aslaidoutinSection5.Theseapproachesrangefromlimitedsensitivityanalysistoformalprobabilisticapproachesandhavedifferentinformation,resource,andtimerequirements,whichwillvaryonacase-by-casebasis,aswillthepotentialvalueoftheadditionalanalysis. Differences in analytical approaches in terms of the information provided and inputrequirementsleadstoquestions,suchas:
• Aresensitivityanalysesofimportantmodelparametersand/ormodelassumptionsatechnicallyappropriatewaytoassessuncertaintiesinvolvedineconomy-widemodelingofregulations?
• Aretherecircumstancesinwhichtheuseofmultiplemodelsshouldbeconsidered?
Beyond addressing model and parameter uncertainty that stem from limitations in scientificunderstanding, in some cases there may be questions about the fitness of a particular model for apurpose.Inthecontextofeconomy-wideanalysisofairregulations,thismayinclude:
• AretherebestpracticestoprovideconfidencethataCGEmodelisproducingcrediblewelfareoreconomicimpactestimates?
30
References
Abler, D., Rodriguez, A., and Shortle, J. 1999. “Parameter Uncertainty in CGE Modeling of theEnvironmentalImpactsofEconomicPolicies.”EnvironmentalandResourceEconomics14(1):75-94.
Arndt, C., and Robinson, S., and Tarp, F. 2002. “Parameter estimation for a computable generalequilibriummodel:amaximumentropyapproach.”EconomicModelling19(3):375—398.
Arndt, C. 1996. “An Introduction to Systematic Sensitivity Analysis via Gaussian Quadrature.” GTAPTechnicalPapers.7-1-1996.
Babiker,M.,Gurgel,A.,Paltsev,S.,andReilly,J.2009.“Forward-lookingversusrecursive-dynamicmodelinginclimatepolicyanalysis:Acomparison.”EconomicModeling26(6):1341-1354.
Balistreri,E.,andRutherford,T.“SubglobalCarbonPolicyandtheCompetitiveSelectionofHeterogeneousFirms.”EnergyEconomics:34(S2):S190-S197.
Ballard,C.,andMedema,S.1993.“TheMarginalEfficiencyEffectsofTaxesandSubsidiesinthePresenceofExternalities”JournalofPublicEconomics52(2):199-216.
Banse,M.,Shutes,L.,Dixon,P.,VanMeijl,H.,Rimmer,M.,Tabeau,A.,andRothe,A.2013.“FactorMarketsin General Computable Equilibrium Models.” Factor Markets Working Paper 47. Seehttp://www.factormarkets.eu/system/files/FM%20WP47%20Final%20Modelling%20Report.pdf.
Beckman,J.,Hertel,T.,andTyner,W.2011.“ValidatingEnergy-OrientedCGEModels.”EnergyEconomics33(5):799-806.
Bernheim,B.,Scholz,J.,andShoven,J.1989.“ConsumptionTaxationinaGeneralEquilibriumModel:HowReliable are Simulation Results?” In B. Bernheim and J. Shoven (Eds.),National Saving and EconomicPerformance.UniversityofChicagoPress.
Bohringer,C.,Rutherford,T.,Balistreri,E.,andWeyant,J.2012.“IntroductiontotheEMF29SpecialIssueontheRoleofBorderCarbonAdjustmentinUnilateralClimatePolicy.”EnergyEconomics34:S95-S96.
Canova,F.1995.“SensitivityAnalysisandModelEvaluation inSimulatedDynamicGeneralEquilibriumEconomies.”InternationalEconomicReview36(2):477-501.
Carson, J. 2002. “Model Verification and Validation” In Proceedings of the Winter 2002 SimulationConference1:52-58.
CBO. 2015. “Budgetary and Economic Effects of Repealing the Affordable Care Act.”https://www.cbo.gov/sites/default/files/114th-congress-2015-2016/reports/50252-Effects_of_ACA_Repeal.pdf
Clarke,L.,Fawcett,A.,Weyant,J.,McFarland,J.,Chaturvedi,V.,andZhou,Y.2014.“TechnologyandU.S.EmissionsReductionsGoals:ResultsoftheEMF24ModelingExercise.”TheEnergyJournal35(SI1):9-31.
31
DeRidder,K.,Viaene,P.,VanDeVel,K.,Brasseur,O.,Cheymol,A.,Fierens,F.2014.“TheImpactofModelResolutiononSimulatedAmbientAirQualityandAssociatedHumanExposure.”Atmósfera27:403-410.
DeVuyst,E.,andPreckel,P.1997.“SensitivityAnalysisRevisited:AQuadrature-BasedApproach.”JournalofPolicyModeling19(2):175-185.
Dixon,P.,andRimmer,M.2013.“ValidationinComputableGeneralEquilibriumModeling.”InP.DixonandD.Jorgenson(Eds.),HandbookofComputableGeneralEquilibriumModeling.NorthHolland.Elsevier.
Elliott,J.,Franklin,M.,Foster,I.,Munson,T.,andLoudermilk,M.2012.“PropagationofDataErrorandParametricSensitivityinComputableGeneralEquilibriumModels.”ComputationalEconomics39(3):219-241.
Elliott,J.,Foster,I.,Loudermilk,M.,andMunson,T.2012.“ImpactofEliminatingBiofuelsProductiononU.S.GasolinePrices:AnEquilibriumAnalysis.”CenterforRobustDecisionMakingonClimateandEnergyPolicyattheUniversityofChicago.RDCEPWorkingPaperNo.12-05.
EPA.2002.EPAAirPollutionControlCostManual.EPA/452/B-02-001.https://www3.epa.gov/ttncatc1/dir1/c_allchs.pdf.
EPA.2010a.EPAAnalysisoftheAmericanCleanEnergyandSecurityActof2009H.R.2454inthe111thCongress.http://www3.epa.gov/climatechange/Downloads/EPAactivities/HR2454_Analysis.pdf.
EPA.2010b.GuidelinesforPreparingEconomicAnalyses.EPA-240-R-10-01.http://yosemite.epa.gov/ee/epa/eerm.nsf/vwAN/EE-0568-50.pdf/$file/EE-0568-50.pdf.
EPA.2010c.SupplementalEPAAnalysisoftheAmericanCleanEnergyandSecurityActof2009H.R.2454inthe111thCongress.http://www3.epa.gov/climatechange/Downloads/EPAactivities/HR2454_SupplementalAnalysis.pdf.
EPA.2012.RegulatoryImpactAnalysis:FinalNewSourcePerformanceStandardsandAmendmentstotheNationalEmissionsStandardsforHazardousAirPollutantsfortheOilandNaturalGasIndustry.https://www3.epa.gov/ttnecas1/regdata/RIAs/oil_natural_gas_final_neshap_nsps_ria.pdf.
EPA.2015a.RegulatoryImpactAnalysisforCleanPowerPlanFinalRule.EPA-452/R-15-003.https://www.epa.gov/sites/production/files/2015-08/documents/cpp-final-rule-ria.pdf.
EPA.2015b.RegulatoryImpactAnalysisoftheFinalRevisionstotheNationalAmbientAirQualityStandardsforGround-LevelOzone.EPA-452/R-15-007.https://www.regulations.gov/#!documentDetail;D=EPA-HQ-OAR-2013-0169-0057.
EPA.2016.RegulatoryImpactAnalysisoftheFinalOilandNaturalGasSector:EmissionStandardsforNew,Reconstructed,andModifiedSources.EPA-452/R-16-002.https://www3.epa.gov/ttnecas1/docs/ria/oilgas_ria_nsps_final_2016-05.pdf.
Fann,N.,Roman,H.A.,Fulcher,C.M.,Gentile,M.A.,Hubbell,B.J.,Wesson,K.,Levy,J.I.2011.“MaximizingHealthBenefitsandMinimizingInequality:IncorporatingLocal-ScaleDataintheDesignandEvaluationofAirQualityPolicies.”RiskAnalysis31:908-922.
32
Fox,T.,andFullerton,D.1991.“TheIrrelevanceofDetailinaComputableGeneralEquilibriumModel.”EconomicsLetters36:67-70.
Gass,S.1983."Decision-AidingModels:Validation,Assessment,andRelatedIssuesforPolicyAnalysis."OperationsResearch,31(4):603-631.
Hansen,L.,andHeckman,J.1996.“TheEmpiricalFoundationsofCalibration.”TheJournalofEconomicPerspectives10(1):87-104.
Harisson, G., and Vinod, H. 1992. “The Sensitivity Analysis of Applied General Equilibrium Models:CompletelyRandomizedFactorialSamplingDesigns.”ReviewofEconomicsandStatistics74(2):357-362.
Hazilla,M.andKopp,R.1990.“SocialCostsofEnvironmentalQualityRegulations:AGeneralEquilibriumAnalysis.”JournalofPoliticalEconomy98(4):853-873.
Hertel, T. Hummels, D., Ivanic, M., and Keeney, R. 2007. “How Confident can e be of CGE-BaseAssessmentsofFreeTradeAgreements?”EconomicModelling24:611-635.
Horridge, J., and Pearson, K. “Systematic Sensitivity Analysiswith Respect to Correlated Variations inParametersandShocks.”GTRAPTechnicalPaperNo.30.
InstituteofMedicine. 2013.EnvironmentalDecisions in the FaceofUncertainty.Washington,DC: TheNationalAcademiesPress.
Iooss, B. and Lemaitre, P. 2015. “A Review of Global Sensitivity Analysis Methods.” In UncertaintyManagementinSimulation-OptimizationofComplexSystems.Dellino,G.andMeloni,C.(eds).SpringerUS.
IPCC. 2010. “Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on ConsistentTreatment of Uncertainties.” https://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf
Jacoby,H.,Reilly, J.,McFarland,J.,andPaltsev,S.2006.“TechnologyandTechnicalChange intheMITEPPAModel.”EnergyEconomics28:610-631.
Jacoby,H.andSue-Wing,I.1999.“AdjustmentTime,CapitalMalleability.”EnergyJournal20:73-92.
Jorgenson,D.,Goettle,R.,Ho,M.,andWilcoxen,P.2013.DoubleDividend:EnvironmentalTaxesandFiscalReformintheUnitedStates.MITPress.
Jorgenson,D.,andWilcoxen,P.1990.“EnvironmentalRegulationandU.S.EconomicGrowth.”RANDJournalofEconomics21(2):314-340.
Kehoe,T.2005.“AnEvaluationofthePerformanceofAppliedGeneralEquilibriumModelsontheImpactofNAFTA.”InT.Kehoe,T.Srinivasan,andJ.Whalley(eds.),FrontiersinAppliedGeneralEquilibriumModeling:EssaysinHonorofHerbertScarf,CambridgeUniversityPress.
Kehoe,T.,Polo,C.,andSancho,F.1995.“AnEvaluationofthePerformanceofanAppliedGeneralEquilibriumModeloftheSpanishEconomy.”EconomicTheory6:115-141.
33
Kheirbek, I.,Wheeler,K.,Walters, S.,Kass,D.,Matte,T.2013. “PM2.5andOzoneHealth ImpactsandDisparitiesinNewYorkCity:SensitivitytoSpatialandTemporalResolution.”AirQuality,Atmosphere&Health6:473-486.
Kriegler,E.,Riahi,K.,Bosetti,V.,Capros,P.,vanVuuren,D.,Weyant,J.,Edenhofer,O.2015.“SpecialSection:TheAMPEREIntermodelComparisonontheEconomicsofClimateStabilization.”TechnologicalForecastingandSocialChange90(Part.A):1-354.
Kriegler,E.,Tavoni,M.,Riahi,K.,vanVuuren,D.,2013.“IntroducingtheLIMITSSpecial Issue.”ClimateChangeEconomics04:1302002.
Kopits,E.,McGartland,A.,Morgan,C.,Pasurka,C.,Shadbegian,R.,Simon,N.,Simpson,D.,andWolverton,A.2014.“RetrospectiveCostAnalysesofEPARegulations:ACaseStudyApproach.”JournalofBenefitCostAnalysis5(2):173-193.
Li,Y.,Henze,D.K.,Jack,D.,Kinney,P.L.2016.“TheInfluenceofAirQualityModelResolutiononHealthImpactAssessmentforFineParticulateMatteranditsComponents.”AirQuality,Atmosphere&Health9:51-68.
Mansfield,C.,Sinha,P.&Henrion,M.,2009.InfluenceAnalysisinSupportofCharacterizingUncertaintyinHumanHealthBenefitsAnalysisInfluenceAnalysisinSupportofCharacterizingUncertaintyinHumanHealthBenefitsAnalysis:FinalReport,PreparedforU.S.EPA,OfficeofAirQualityPlanningandStandards.ResearchTrianglePark,NC.
Miller,A.,andRice,T.1983.“DiscreteApproximationsofProbabilityDistributions.”ManagementScience29:352-362.
Miranda,M.,andFackler,P.2004.AppliedComputationalEconomicsandFinance.TheMITPress.
McFarland,J.,Reilly,J.,andHerzog,H.2004.“RepresentingEnergyTechnologiesinTop-downEconomicModelsusingBottom-upInformation.”EnergyEconomics26(4):685-707.
Mohora, M. 2006. “RoMod: A Dynamic CGE Model for Romania.” Doctoral Dissertation. ErasmusUniversityRotterdam.Availableat:http://repub.eur.nl/pub/7455/few_mohora_20060217_thesis.pdf
NAS.2013.EnvironmentalDecisionsintheFaceofUncertainty.TheNationalAcademiesPress.
NRC.2002.EstimatingthePublicHealthBenefitsofProposedAirPollutionRegulations.TheNationalAcademiesPress.
NRC.2009.ScienceandDecisions:AdvancingRiskAssessment.TheNationalAcademiesPress.
OMB.2003.“RegulatoryAnalysis:APrimer.”CircularA-4.Washington,DC.Availableathttp://www.whitehouse.gov/omb/circulars/a004/a-4.html.
Pagan, A., and Shannon, J. 1985. “Sensitivity Analysis for Linearized Computable General EquilibriumModels.”InJ.PiggottandJ.Whaley(Eds.),NewDevelopmentsinAppliedGeneralEquilibriumAnalysis(pp.104—118).CambridgeUniversityPress.
34
Pannell, D. 1997. “Sensitivity Analysis of Normative Economic Models: Theoretical Framework andPracticalStrategies.”AgriculturalEconomics16(2):139-152.
Pindyck,R.S.2007."UncertaintyinEnvironmentalEconomics."ReviewofEnvironmentalEconomicsandPolicy1(1):45-65.
Punger,E.M.,West,J.J.2013.“TheEffectofGridResolutiononEstimatesofTheBurdenofOzoneandFineParticulateMatteronPrematureMortalityintheUSA.”AirQuality,Atmosphere&Health6:563-573.
Pyatt,G.,andRound,J.I.,eds.1985.Socialaccountmatrices:Abasisforplanning.Washington:TheWorldBank.
Rausch,S.,andRutherford,T.2008.“ToolsforBuildingNationalEconomicModelsusingState-LevelIMPLANSocialAccounts.”Availableat:http://www.mpsge.org/IMPLAN2006inGAMS/IMPLAN2006inGAMS.pdf
Rosenzweig,C.,Jones,J.W.,Hatfield,J.L.,Ruane,A.C.,Boote,K.J.,Thorburn,P.,Antle,J.M.,Nelson,G.C.,Porter,C.,Janssen,S.,Asseng,S.,Basso,B.,Ewert,F.,Wallach,D.,Baigorria,G.,Winter,J.M.2013.“TheAgriculturalModelIntercomparisonandImprovementProject(AgMIP):Protocolsandpilotstudies.”AgriculturalandForestMeteorology170:166-812.
Sargent,R.2013.“VerificationandValidationofSimulationModels.”JournalofSimulation7:12-24.
Selin,N.,Wu,S.,NamK.-M.,Reilly,J.,Palsev,S.,Prinn,R.2009.“GlobalHealthandEconomicImpactsofFutureOzonePollution.”EnvironmentalResearchLetters4:1-9.
Shoven,J.,andWhalley,J.1984.“AppliedGeneral-EquilibriumModelsofTaxationandInternationalTrade:AnIntroductionandSurvey.”JournalofEconomicLiterature22(3):1007-1051.
Sobol,I.1993.“SensitivityEstimatesforNon-linearMathematicalModels.MathematicalModellingandComputationalExperiments1:407-414.
Sokolov, A., Stone, P., Forest, C., Prinn, R., Sarofim, M., Webster, M., Paltsev, S., and Schlosser, C.“Probabilistic Forecast for Twenty-First-CenturyClimateBasedonUncertainties in Emissions (WithoutPolicy)andClimateParameters.JournalofClimate22:5175-5204.
Thompson, T.M., Saari, R.K., Selin, N.E.. 2014. “Air Quality Resolution for Health Impact Assessment:InfluenceofRegionalCharacteristics.”AtmosphericChemistryandPhysics14:969-978.
Tuladhar,S.,andWilcoxen,P.1998.“AnEconometricLookattheDoubleDividendHypothesis.”NationalTaxAssociationProceedings91:57-62.
Valenzuela, E., Hertel, T., Keeney, R., and Reimer, J. 2007. “Assessing Global Computable GeneralEquilibrium Model Validity using Agricultural Price Volatility.” American Journal of AgriculturalEconomics89(2):383-397.
Wang,C.andChen,J.2006.“ParameterUncertaintyinCGEModelingoftheMacroeconomicImpactofCarbonReductioninChina.”TsinghuaScienceandTechnology11(5):617-624.
35
Webster,M.,Babiker,M.,Mayer,M.,Reilly,J.,Harnisch,J.,Hyman,R.,Sarofim,M.,andWang,C.2002."UncertaintyinEmissionsProjectionsforClimateModels."AtmosphericEnvironment36(22):3659-3670.
Webster,M., Paltsev, S., Parsons, J., Reilly, J., and Jacoby, H. 2008. “Uncertainty in Greenhouse GasEmissionsandCostsofAtmosphericStabilization.”MITJointProgramontheScienceandPolicyofGlobalChangeReportNo.165.
Wigle,R.1991.“ThePagan-ShannonApproximation:UnconditionalSystematicSensitivity inMinutes.”EmpiricalEconomics16:25-49.