evidence review business advice - whatworksgrowth.org

Evidence Review

Business Advice

May 2014

Contents

1. Preface 2

2. Executive summary 4

3. Introduction 8

4. Impact evaluation 10

5. Methodology 13

6. Definition 16

7. Findings 17

8. Detailed findings 23

9. Summary of findings 32

10. How to use this review 34

11. References 36

Appendix A: Evidence Reviews 38

Appendix B: Search Terms and Sources 40

Appendix C: Findings for RCT studies 45

Appendix D: Case Studies 47

00

Evidence Review: Business Advice - May 2014 3

Preface

Thisreportpresentsfindingsfromasystematicreviewofevaluationsofbusinessinformation,adviceandmentoringprogrammes(‘businessadvice’)aimedatimprovingbusinessgrowthandotheroutcomes.

ItisthesecondofaseriesofreviewsthatwillbeproducedbytheWhat Works Centre for Local Economic Growth.TheWhatWorksCentreisacollaborationbetweentheLondon School of Economics and Political Science,Centre for CitiesandArupandisfundedbytheEconomic & Social Research Council,The Department for Communities and Local GovernmentandThe Department for Business Innovation & Skills.

Thesereviewsconsideraspecifictypeofevidence–impact evaluation–thatseekstounderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Toputitanotherwaytheyask‘didthepolicywork’and‘diditrepresentgoodvalueformoney’?Bylookingatthedetailsofthepoliciesevaluatedwecanalsostarttoanswerquestionsaboutdeliveryissues–forexample,whetherbusinessadvicepolicieswhichusemanagedbrokerage(i.e.a‘hands-on’approach)performbetterthanprogrammesthatarelighttouch(i.e.involvelittleornoengagementwithclients).

Evidenceonimpactandeffectivenessisclearlyacrucialinputtogoodpolicymaking.Processevaluation–lookingindetailathowprogrammesoperatedaytoday–providesavaluablecomplementtoimpactevaluation,butwedeliberatelydonotfocusonthis.Werecognisethatmaysometimescausefrustrationforpractitionersanddecision-makerswhoareresponsibleforthedeliveryofpolicy.However, we see these impact-focused reviews as an essential part of more effective policy making.Weoftensimplydonotknowtheanswerstomanyofthequestionsthatmightreasonablybeaskedwhenimplementinganewpolicy–notleast,doesitwork?Figuringoutwhatwedoknowallowsustobetterdesignpoliciesandundertakefurtherevaluationstostartfillingthegapsinourknowledge.This also helps us to have more informed discussions about process and delivery issues and to improve policy making.

Thesereviewsthereforerepresentafirststepinimprovingourunderstandingofwhatworksforlocaleconomicgrowth.Inthemonthsahead,wewillbeworkingwithlocaldecision-makersandpractitioners,usingthesefindingstohelpthemgeneratebetterpolicy.

Henry OvermanDirector,WhatWorksCentreforLocalEconomicGrowth

01

http://www.whatworksgrowth.org

http://www.whatworksgrowth.org

http://www.lse.ac.uk

http://www.lse.ac.uk

http://www.centreforcities.org/

http://www.arup.com/

http://www.esrc.ac.uk/

http://www.esrc.ac.uk/

https://www.gov.uk/government/organisations/department-for-communities-and-local-government

https://www.gov.uk/government/organisations/department-for-business-innovation-skills

https://www.gov.uk/government/organisations/department-for-business-innovation-skills


Executive Summary

Thisreportpresentsfindingsfromasystematicreviewofevaluationsofbusinessinformation,adviceandmentoringprogrammes(‘businessadvice’)aimedatimprovingbusinessgrowthandotheroutcomes.ItisthesecondofaseriesofreviewsthatwillbeproducedbytheWhatWorksCentreforLocalEconomicGrowth.

Thereviewconsideredalmost700policyevaluationsandevidencereviewsfromtheUKandotherOECDcountries.

Itfound23impactevaluationsthatmettheCentre’sminimumstandards.Thisisasmallerevidencebasethanforourfirstreview(onemploymenttraining)althoughthismaystillbelargerthantheevidencebaseformanyotherlocaleconomicgrowthpolicies.Itisaverysmallbaserelativetothatavailableforsomeotherpolicyareas(e.g.medicine,aspectsofinternationaldevelopment,educationandsocialpolicy).

02

Figure 1: Methodology


Overall, of the 23 evaluations reviewed, 17 found positive programme impacts on at least one business outcome. Four evaluations found that business advice didn’t work (had no statistically significant effects) and two evaluations found that business advice might be harmful.

ApproachThisreviewconsiderstheeffectivenessofbusinessadviceinimprovingfirmperformance(intermsofproductivity,employmentandotherperformancemeasures).Figure2providesasummaryofthenumberofevaluationsthatlookatdifferentprogrammefeatures.

Findings

What the evidence shows:

• Businessadvicehadapositiveimpactonatleastonebusinessoutcomein17outof23evaluations.

• Programmesthatusedahands-on,‘managedbrokerage’approachmayperformbetterthanthoseusingalighttouchdeliverymodelsuchasprovidingadvicethroughawebsite. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyandignoresthequestionofcosteffectiveness.

• Businessadviceprogrammesshowconsistentlybetterresultsforproductivityandoutputthantheydoforemployment.Resultsforsales,profitsandexportsaremixed.

Figure 2: Number of shortlisted evaluations by programme feature and context

0 5 10 15 20 25

Employment

Exports

Hybrid

National

Local

Both

Public

Private

Hybrid/Both

Not stated

Light touch

Managed brokerage

Pipeline forcing

Export promotion

Other

Comparison betweentypes of support

Sector specific

General

Prog

ram

me

obje

ctive

Nati

onal

vs

loca

lPu

blic

vs p

rivat

e De

liver

y m

odel

Sect

orsp

ecifi

city

Breakdown of studies by variable


Where the evidence is inconclusive:

• Inmostcases,programmeshadvagueormultipleobjectives,whichmakesmeasuringsuccessdifficult.

• Wefindnostrongdifferencesinresultsbetweenprogrammeswithmultipleobjectivesandprogrammeswithmorefocusedobjectives.

• Wefoundnoevidencethatwouldsuggestonelevelofdelivery–nationalorlocal–ismoreeffectivethananother.

• Itisdifficulttoreachanyconclusionsabouttheeffectivenessofpublic-ledvs.private-leddelivery.

• Overall,itisdifficulttoreachanystrongconclusionsonthelinkbetweenspecificprogrammefeaturesandbetterfirmoutcomes.

Where there is a lack of evidence:

• Thereisinsufficientevidencetoestablishtheeffectivenessofsectorspecificprogrammescomparedtomoregeneralprogrammes.

• Wefoundnohighqualityimpactevaluationsthatexplicitlylookattheoutcomesforfemale-headedorBMEbusinesses.

• Wefoundtwohigh-qualityevaluationsofprogrammesaimedatincubatingstart-ups.Bothprogrammesweretargetedatunemployedpeopleandshowmixedresultsoverall.However,thereisalackofimpactevaluationforDragons’Den-typeacceleratorprogrammesthataimtolaunchhigh-growthbusinessesandinvolvecompetitiveentry.

How to use these reviewsTheCentre’sreviewsconsideraspecifictypeofevidence,impactevaluation,whichseekstounderstandthecausaleffectofpolicyinterventions,andtoestablishtheircost-effectiveness.Inthelongerterm,theCentrewillproducearangeofevidencereviewsthatwillhelplocaldecision-makersdecidethebroadpolicyareasonwhichtospendlimitedresources.Figure3illustrateshowthereviewsrelatetotheotherworkstreamsoftheCentre.

Evidence reviews

Demonstrationprojects

You are here

Capacitybuilding

Understanding what works

More effective policy

Capacitybuilding

Capacitybuilding

Figure 3: What Works Centre work programme


Supporting and complementing local knowledgeTheevidencereviewsetsoutanumberof‘BestBets’–approachestobusinessadvicethathaveperformedmoststronglybasedonthebestavailableimpactevaluations.

However,the‘BestBets’donotaddressthespecificsof‘whatworkswhere’or‘whatwillworkforaparticularfirm’.Detailedlocalknowledgeandcontextremaincrucial.

‘BestBets’alsoraiseanoteofcautionforpolicymakersiftheydecidetointroduceaprogrammewhichhasnotworkedwellelsewhere.

Specific recommendationsThe23evaluationsofferarichsourceofmaterialforpolicymakerstouseindesigningspecificbusinessadvicepolicies.Inparticular,theevaluationswillbeofusetopolicymakersattwokeystagesinthepolicydesignprocess:determiningthepolicyoptions,andthenselectingthepreferredoption.

• Ifwewanttoknowwhatworksintheareaofbusinessadviceweneedtoimproveprogrammedesignandevaluation.Whendesigningaprogramme,localpolicymakersshouldidentify one or two clear programme objectives,andthenidentifyoutcomemeasuresthatarebothclearlyrelatedtotheprogrammeobjectives,andfeasibletomeasure.

• Businessadviceprogrammestendtobemoresuccessfulinincreasingfirms’productivitythaninincreasingtheiremployment.

• Onecomparativestudysuggeststhatsmaller, better-resourced programmes are more likely to achieve successandthanlarger‘hands-off’policies.Butitisunclearwhichoftheseapproachesismorecost-effective.

Filling the Evidence GapsThisreviewhasnotfoundanswerstosomeofthequestionsthatwillbeforemostinpolicymakers’minds.

Thesegapshighlighttheneedforimprovedevaluationandgreaterexperimentation,specificallyexperimentsthatfocuson:

• identifyinghowdifferentelementsofbusinessadviceprogrammedesigncontributetobetterorworseoutcomes;and,

• thevalueformoneyofdifferentapproaches.Only5ofthe23shortlistedstudiesincludedcost-benefitanalysis,andnotalloftheseusedmeasuresthatarecomparableacrossstudies.Thereisaclearneedformore,consistentanalysisofcost-effectivenessinbusinessadviceimpactevaluations.

Thisrequiresevaluationtobeembeddedinpolicydesign,andthinkingdifferentlyaboutthepolicycycleasawhole.


Introduction

Theprovisionofpubliclyfundedadvice,mentoringandsupporttobusinesses,andparticularlytoentrepreneursandsmallbusinesses,isubiquitousinOECDcountries.1Take-upofpublicsectorsupport,however,isgenerallyassumedtobelow.IntheUK,forexample,itisestimatedthatabout40%ofbusinesseshavereceivedformalexternaladvice,butonly20%receivedadvicefromapublicsectorprovider.2

Thisreviewfocusesonprogrammesthatarefundedbygovernmentandthatprovideinformation,structuredadviceorlongertermmentoringtofirms(hence‘businessadvice’).3Suchinterventionstypicallyaimtoincreaseratesoffirmcreation,toimprovebusinesssurvival,andtopromotebusinessproductivityandemploymentgrowth.

Whyarepolicymakerssointerestedinthiskindofbusinesssupport?Somegovernmentsmaywanttopromote‘enterpriseculture’onitsownmerits.Othersstartfromthefactthatsmallandmedium-sizeenterprises(SMEs)formthevastmajorityofbusinessesintheUKandotherdevelopedeconomies;4andthatsmall,newfirmsaccountforthemajorityofjobcreation.5

Intheory,publiclysupportedadvisoryservicescanbejustifiedontwogrounds–informationfailuresandwidereconomicimpacts.Inthefirstcase,wheninformationishardtoaccessorofvariablequality,firmsmayunder-investinservicesthatcouldsupporttheirbusinesses.Economistsrefertothesechallengesas‘informationasymmetries’.6Suchmarketfailuresmayresultwhenbusinessownersare:

• Unawareofinformationandadvicethatwouldbevaluabletothem;

• Unclearabouthowtoaccesssuchresources;

1 OECD(2002)OECDSmallandMediumEnterpriseOutlook,OECD,Paris2 www.gov.uk/government/uploads/system/uploads/attachment_data/file/32250/11-1288-research-barriers-to-use- of-business-support.pdf3 Asdistinctfromprogrammestoa)improveaccesstofinance;b)improvethegeneralbusinessenvironment andc)reduceburdens,regulationsandcosts.4 Curran(2000),Bennett(2008).5 SeeHaltiwangeretal(2010)fortheUS,andBravo-Bioscaetal(2011)fortheUK.6 TwooftheclassicstudiesoninformationasymmetriesareAkerlof(1970),Stiglitz(1979)andGrossmanandStiglitz (1980).Forageneraloverviewofinformationeconomics,seeStiglitz(2002).

03


• Concernedaboutthequalityofadviceoffered;

• Facingfinancialortimeconstraintsonaccessingadvicewhichexceedtheperceivedbenefits;and/or

• Worriedthatconfidentialinformationcouldendupinthehandsofcompetitors.

Inprinciple,publicpolicycansolvetheseproblemsandhelpbusinessestogrowbyprovidingimpartial,freeorsubsidisedadviceandmentoring.

BusinesssupportinterventionsmayalsobejustifiedbecauseSMEsareimportantforeconomicdevelopment.Ifinformation,mentoringandadvicecanhelpindividualfirmstogrow,thiscouldhavespillovereffects–or‘externalities’–fortheeconomyasawhole.Theseincludethecreationofmorejobs,moreinnovation,orlowerpricestoconsumers.

Whilethereisatheoreticalcaseforgovernmentintervention,inpractice,itisnotstraightforwardforgovernmenttoprovideeffectivebusinessadvice,andtherearedangersofpolicyfailureindoingso.Buttheliteraturealsohighlightsrealmarketfailuresformanystart-upsandearlystagefirms;thatsomeprogrammeshaveafarhigherimpactthanothers;andthattherearesignificantdifferencesinusertake-up(forexample,betweenmaleandfemale-headedbusinesses).Thisimpliesthatwell-designedinterventionscouldhavepositiveimpacts.

Consistentwithallofthis,inourreview,wefindanumberofeffectiveprogrammesbutwealsofindexamplesofpolicyfailure.Thissuggeststhatabetterunderstandingofwhatworkswouldaddsignificantvaluetothepolicymakingprocessinthisarea.


Impact evaluation

Governmentsaroundtheworldincreasinglyhavestrongsystemstomonitorpolicyinputs(suchasspendingonabusinesssupportprogramme)andoutputs(suchasthenumberoffirmswhohavegonethroughtheprogramme).However,theyarelesssuccessfulatidentifyingpolicyoutcomes(suchastheeffectofabusinessadviceprogrammeonfirmemployment).Inparticular,manygovernment-sponsoredevaluationsthatlookatoutcomesdonotusecrediblestrategiestoassessthecausal impactofpolicyinterventions.

Evaluationofcausalimpactsfocusonmeasuringthedifferencethatcanbeexpectedbetweentheoutcomeforfirms‘treated’inaprogramme,andtheaverageoutcometheywouldhaveexperiencedwithoutit.Pinningdowncausalityisacruciallyimportantpartofimpactevaluation.Estimates of the benefits of a programme are of limited use to policymakers unless those benefits can be attributed, with a reasonable degree of certainty, to that programme.

Thecredibilitywithwhichevaluationsestablishcausalityisthecriteriononwhichthisreviewassessestheliterature.

Using CounterfactualsEstablishing causality requires the construction of a valid counterfactual–i.e.whatwouldhavehappenedtoprogrammeparticipantshadtheynotbeentreatedundertheprogramme.Thatoutcomeisfundamentallyunobservable,soresearchersspendagreatdealoftimetryingtorebuildit.Thewayinwhichthiscounterfactualis(re)constructedisthekeyelementofimpactevaluationdesign.

A standard approach is to create a counterfactual group of similar individuals not participating in the programme being evaluated. Changesinoutcomescanthenbecomparedbetweenthe‘treatmentgroup’(thoseaffectedbythepolicy)andthe‘controlgroup’(similarindividualsnotexposedtothepolicy).

A key issue in creating the counterfactual group is dealing with the ‘selection into treatment’ problem.Selectionintotreatmentoccurswhenparticipantsintheprogrammedifferfromthosewhodonotparticipateintheprogramme.

04


Anexampleofthisprobleminbusinessadviceprogrammeswouldbewhenmoreambitiousfirmsapplyforadvice.Ifthishappens,estimatesofpolicyimpactmaybebiasedupwardsbecauseweincorrectlyattributebetterfirmoutcomestothepolicy,ratherthantothefactthatthemoreambitiousparticipantswouldhavedonebetterevenwithouttheprogramme.

Selectionproblemsmayalsoleadtodownwardbias.Forexample,firmsthatapplyforadvicemightbeexperiencingproblemsandsuchfirmsmaybelesslikelytogroworsucceedindependentofanyadvicetheyreceive.Thesefactorsareoftenunobservabletoresearchers.

So the challenge for good programme evaluation is to deal with these issues, and to demonstrate that the control group is plausible.Iftheconstructionofplausiblecounterfactualsiscentraltogoodpolicyevaluation,thenthecrucialquestionbecomes:how do we design counterfactuals? Box1providessomeexamples.

Box 1: Impact evaluation techniques

Onewaytoidentifycausalimpactsofaprogrammeistorandomlyassignparticipantstotreatmentandcontrolgroups.Forresearchers,suchRandomisedControlTrials(RCTs)areoftenconsideredthe‘goldstandard’ofevaluation.Properlyimplemented,randomisationensuresthattreatmentandcontrolgroupsarecomparablebothintermsofobservedandunobservedattributes,thusidentifyingthecausalimpactofpolicy.However,implementationofthese‘realworld’experimentsischallengingandcanbeproblematic.RCTsmaynotalwaysbefeasibleforlocaleconomicgrowthpolicies–forexample,policymakersmaybeunwillingtorandomise.7Andsmall-scaletrialsmayhavelimitedwiderapplicability.

Whererandomisedcontroltrialsarenotanoption,‘quasi-experimental’approachesofrandomisationcanhelp.Thesestrategiescandealwithselectiononunobservables,by(say)exploitinginstitutionalrulesandprocessesthatresultinsomefirmsquasi-randomlyreceivingtreatment.

Evenusingthesestrategies,though,thetreatmentandcontrolgroupsmaynotbefullycomparableintermsofobservables.StatisticaltechniquessuchasOrdinaryLeastSquares(OLS)andmatchingcanbeusedtoaddressthisproblem.

Notethathigherqualityimpactevaluationfirstusesidentificationstrategiestoconstructacontrolgroupanddealwithselectiononunobservables.Thenittriestocontrolforremainingdifferencesinobservablecharacteristics.Itisthecombinationthatisparticularlypowerful:OLSormatchingaloneraiseconcernsabouttheextenttowhichunobservablecharacteristicsdeterminebothtreatmentandoutcomesandthusbiastheevaluation.

7 Gibbons,NathanandOverman(2014).


Evidence included in the review We include any evaluation that compares outcomes for firms receiving treatment (the treated group) after an intervention with outcomes in the treated group before the intervention, relative to a comparison group used to provide a counterfactual of what would have happened to these outcomes in the absence of treatment.

Thismeanswelookatevaluationsthatdoareasonablejobofestimatingtheimpactoftreatmentusingeitherrandomisedcontroltrials,quasi-randomvariationorstatisticaltechniques(suchasOLSandmatching)thathelpmaketreatmentandcontrolgroupscomparable.Weviewtheseevaluationsasprovidingcredibleimpactevaluationinthesensethattheyidentifyeffectswhichcanbeattributed,withareasonabledegreeofcertainty,totheimplementationoftheprogrammeinquestion.AfulllistofshortlistedstudiesisgiveninAppendixA.

Evidence excluded from the reviewWeexcludeevaluationsthatprovideasimplebeforeandaftercomparisononlyforthosereceivingthetreatmentbecausewecannotreasonablyassumethatchangesforthetreatedgroupcanbeattributedtotheeffectoftheprogramme.

Wealsoexcludecasestudiesorevaluationsthatfocusonprocess(howthepolicyisimplemented)ratherthanimpact(whatwastheeffectofthepolicy).Suchstudieshavearoletoplayinhelpingformulatebetterpolicybuttheyarenotthefocusofourevidencereviews.


Methodology

Toidentifyrobustevaluationevidenceonthecausalimpactofbusinessadviceprogrammes,weconductedasystematicreviewoftheevidencefromtheUKandacrosstheworld.Ourreviewsfollowedafive-stageprocess:scope,search,sift,scoreandsynthesise.

Stage 1: Scope of Review WorkingwithourUserPanelandamemberofourAcademicPanel,weagreedthereviewquestion,keytermsandinclusioncriteria.Wealsousedexistingliteraturereviewsandmeta-analysestoinformourthinking.

05

Figure 1: Methodology


Stage 2: Searching for EvaluationsWesearchedforevaluationevidenceacrossawiderangeofsources,frompeer-reviewedacademicresearch,togovernmentevaluationsandthinktankreports.Specifically,welookedatacademicdatabases(suchasEconLit,WebofScienceandGoogleScholar),specialistresearchinstitutes(suchasCEPRandIZA),UKcentralandlocalgovernmentdepartments,andworkdonebythinktanks(suchastheOECD,ILO,IPPRandPolicyExchange).Wealsoissuedacallforevidenceviaourmailinglistandsocialmedia.Thissearchfoundcloseto700books,articlesandreports.AppendixBprovidesafulllistofsourcesandsearchterms.

Stage 3: Sifting EvaluationsWescreenedourlong-listonrelevance,geography,languageandmethods,keepingimpactevaluationsfromtheUKandotherOECDcountries,withnotimerestrictionsonwhentheevaluationwasdone.WefocussedonEnglish-languagestudies,butwouldconsiderkeyevidenceifitwasinotherlanguages.Wethenscreenedtheremainingevaluationsontherobustnessoftheirresearchmethods,keepingonlythemorerobustimpactevaluations.WeusedtheMarylandScientificMethodsScale(SMS)todothis.8TheSMSisafive-pointscalerangingfrom1,forevaluationsbasedonsimplecrosssectionalcorrelations,to5forrandomisedcontroltrials(seeBox2).Weshortlistedallthoseimpactevaluationsthatcouldpotentiallyscore3oraboveontheSMS.9Inthiscasewefoundnoevaluationsscoring4.Forexamplesofbusinessadviceevaluationsthatscore3and5ontheSMSscale,seeAppendixD.

Stage 4: Scoring EvaluationsWeconductedafullappraisalofeachevaluationontheshortlist,collectingkeyresultsandusingtheSMStogiveafinalscoreforevaluationsthatreflectedboththequalityofmethodschosenandqualityofimplementation(whichcanbelowerthanclaimedbysomeauthors).Scoringandshortlistingdecisionswerecross-checkedwiththeacademicpanelmemberandthecoreteamatLSE.Thefinallistofincludedstudiesandtheirreferencenumbers(usedintherestofthisreport)canbefoundinAppendixA.

Stage 5: Synthesising EvaluationsWedrewtogetherourfindings,combiningmaterialfromourevaluationsandtheexistingliterature.

8 Sherman,Gottfredson,MacKenzie,Eck,Reuter,andBushway(1998).9 Shermanetal.(1998)alsosuggestthatlevel3istheminimumlevelrequiredforareasonableaccuracyofresults.


Box 2: The Scientific Maryland Scale

Level 1:Correlation of outcomes with presence or intensity of treatment, cross-sectional comparisons of treated groups with untreated groups, or other cross-sectional methods in which there is no attempt to establish a counterfactual.Nouseofcontrolvariablesinstatisticalanalysistoadjustfordifferencesbetweentreatedanduntreatedgroups.

Level 2:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention (‘before and after’ study). Nocomparisongroupusedtoprovideacounterfactual,oracomparatorgroupisusedbutthisisnotchosentobesimilartothetreatmentgroup,nordemonstratedtobesimilar(e.g.nationalaveragesusedascomparisonforpolicyinterventioninaspecificarea).No,orinappropriate,controlvariablesusedinstatisticalanalysistoadjustfordifferencesbetweentreatedanduntreatedgroups.

Level 3:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (e.g. difference in difference). Somejustificationgiventochoiceofcomparatorgroupthatispotentiallysimilartothetreatmentgroup.Evidencepresentedoncomparabilityoftreatmentandcontrolgroupsbutthesegroupsarepoorlybalancedonpre-treatmentcharacteristics.Controlvariablesmaybeusedtoadjustfordifferencebetweentreatedanduntreatedgroups,buttherearelikelytobeimportantuncontrolleddifferencesremaining.

Level 4:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (i.e. difference in difference).Careful and credible justification provided for choice of a comparator group that is closely matched to the treatment group.Treatmentandcontrolgroupsarebalancedonpre-treatmentcharacteristicsandextensiveevidencepresentedonthiscomparability,withonlyminororirrelevantdifferencesremaining.Controlvariables(e.g.OLSormatching)orotherstatisticaltechniques(e.g.instrumentalvariables,IV)maybeusedtoadjustforpotentialdifferencesbetweentreatedanduntreatedgroups.Problemsofattritionfromsampleandimplicationsdiscussedbutnotnecessarilycorrected.

Level 5:Reserved for research designs that involve randomisation into treatment and control groups.Randomisedcontroltrialsprovidethedefinitiveexample,althoughother‘naturalexperiment’researchdesignsthatexploitplausiblyrandomvariationintreatmentmayfallinthiscategory.Extensiveevidenceprovidedoncomparabilityoftreatmentandcontrolgroups,showingnosignificantdifferencesintermsoflevelsortrends.Controlvariablesmaybeusedtoadjustfortreatmentandcontrolgroupdifferences,butthisadjustmentshouldnothavealargeimpactonthemainresults.Attentionpaidtoproblemsofselectiveattritionfromrandomlyassignedgroups,whichisshowntobeofnegligibleimportance.


Definition

Weincludedinourdefinitionofbusinessadviceandmentoringgovernmentfundedprogrammesthatfocusedon:

• Supportingindividualstosetuptheirownbusinesses

• Supportingexistingbusinessestogrow,wheregrowthmaybebroadlydefinedtoinclude:

• Improvedproductivity(intermsofsales/turnoverperemployee10orvalueaddedperemployee11)

• Growthinemployment

• Growthinturnover

• Growthinprofits

• Expansionintonewmarkets(particularlyoverseas).

Weexcluded:

• Financialsupportandaccesstofinanceschemes–wewilladdressthistopicindependentlyinafuturereview

• Incubatorprogrammes–duetoalackofevidenceontheirimpact.

10 Studies165,166,167and284.11 Studies169and170.

06


Findings

Thissectionsetsoutthereview’sfindings.Webeginwithadiscussionoftheevidencebase,andthenexploretheoverallpatternofpositiveandnegativeresults.Afterthisweconsiderspecificprogrammefeaturesinmoredetail.

Quantity and quality of the evidence baseFromaninitiallonglistof690studies,23evaluationsmetourminimumstandards.12Thisisasmallerevidencebasethanforourfirstreview(onemploymenttraining),althoughthismaystillbelargerthantheevidencebaseformanyotherlocaleconomicgrowthpolicies.Table1showsthedistributionofthestudiesrankedaccordingtotheSMS.

Table 1: Ranking studies by quality of implementation

SMS Score Number by implementation5 4

4 0

3 19

Total 23

Therearefourrandomisedcontroltrials,allofwhichscored5ontheSMS.13Wefoundnostudiesthatusedcrediblequasi-randomsourcesofvariation(i.e.scored4ontheSMS)toidentifypolicyimpacts.

19studiesscored3ontheSMS,andusevariationsonOLSormatchingtechniques.Thetechniquesappliedinthesestudiesmeanthatwecanbereasonablyconfidentthattheevaluationhasdoneagoodjobofcontrollingforallobservablecharacteristicsoffirmsorindividuals(forexample:firmage;size;sector),whichmightexplaindifferencesinfirmoutcomes.However,forthesestudies,itislikelythatunobservablecharacteristicssuchasentrepreneurialtalentorfirms’desiretogrowmaystillbeaffectingtheresults.Thisraisesconcernsthattheevaluationincorrectlyattributesbeneficialoutcomes

12 Manyofthestudiesnotincludedprovidedcasestudiesorprocessevaluationswhichareoftenvaluable,butare notthefocusofourreview.Seemethodologysectionforfurtherdiscussion.13 Studies162,163,282,287.

07


totheprogrammeratherthantothesefirmcharacteristics.WecanonlybefullyconfidentthatselectiononunobservableshasbeeneradicatedwithanRCTmethodology,whereparticipantsarerandomlyassignedtotreatmentorcontrolgroups.

Foreightofthese19studieswehaveconcernsoverthebaselineyearused(eitherpost-treatmentorunknown)soweneedtobecarefulininterpretingtheresults.Ifpositiveprogrammeeffectsarefeltimmediately,thenthesestudiesmayunderestimatetheimpact.Conversely,ifparticipationintheprogrammeinitiallyworsensperformance(e.g.becauseanentrepreneurisspendingtimeimplementingadviceorre-organisingthefirmratherthanfocusingonproduction)thenthesestudiesmayoverestimateimpact.

Type and focus of business adviceBroadlyspeakingtherearetwoseparatetypesofinterventionthatfallunderthebannerof‘businessadviceandmentoring’,thoseinwhichthefocusisonsupportingindividualstosetuptheirownbusinessandthoseinwhichthefocusisonsupportingexistingbusinessestogrow(where‘growth’maybedefinedinvariouswaysincludinggrowthinturnover,employment,profits,expansionintonewmarkets,etc).

Fiveofthe23evaluationslookedatsupporttoindividualstoestablishnewbusinesses,typicallymicro-enterpriseswith1-10employees.14Ofthose,fourwerefromtheUSA(ofwhichtwolookedatthesameprogramme–ProjectGATE-butfoundcontrastingresults).15OnewasfromGermany.16Inallcases,thefocusisonsupportingunemployedindividualstosetupeitherasown-accountworkersorasmicroenterprises,andonimprovingthesuccessrateofthosestart-ups.Theseprogrammesgenerallyinvolvetraininginbasicbusinessskillsandconceptsandsupportinsuchendeavoursaswritingabusinessplan.

Oftheremaining18evaluations:

• Eightcoverfourprogrammes,orgroupsofprogrammes,whichprovidegeneralbusinessadviceandaimtosupportfirmgrowth.ForBusinessLinkwehaveincludedboththeoriginalevaluationcommissionedbyBIS(study284)andthreeacademicfollowupstudiesthatexploreaspectsoftheprogrammeinmoredetail;17

• ThreecovervariousprogrammesrunbyUKTrade&Industryfocusedonpromotingexportsorhelpingfirmstoaccessforeignmarkets;18

• Threelookedatprogrammesprovidingsubsidiestoallowfirmstoaccessmarket-providedbusinesssupportservices;19

• TwolookedatthePLATOprogramme(Belgium),whichfocusesonpeerlearningandnetworksasaroutetobusinessgrowthandimprovement;20

• OnelookedatNewZealand’sTrade&Enterpriseprogrammetosupportfirmswithhighgrowthpotential;21

14 Studies163,217,276,282,287.15 Studies163,276,282,287.16 Study217.17 Studies166,167,177,284and285coverBusinessLink(UK);162coversCompetitiveProductivityorIPCC (Mexico);168coversthePueblaInstituteforCompetitiveProductivity;and172coversvariousservices(Germany).18 Studies165,169,183,184whichcoverOverseasMarketIntroductionService(OMIS),UK;PassporttoExport,UK; UKTIBusinessSupportservices(various,includingPassport),UK.19 Studies162,170,182.20 Studies174,286.21 Study269.


• Onedidnotlookataspecificprogramme,butanalysedtheimpactofvariousdifferentformsofexternaladvice(includingpubliclyprovidedbusinesssupport)onfirmperformance.22

Overall effects on business growth

Business support and advice had a positive effect on at least one business outcome in 17 out of 23 evaluations.

Ofthe23evaluationsreviewed,17foundpositiveprogrammeimpactsonatleastonefirmoutcome.Fourevaluationsfoundthatbusinessadvicedidn’twork(i.e.theyfoundnostatisticallysignificantevidenceofpositiveimpactsonfirmoutcomes)andtwoevaluationsfoundthatbusinessadvicemightbeharmful(i.e.statisticallysignificantnegativeimpactsonfirmsoutcomeswithnooffsettingpositiveeffects).23Inoneofthesetwocases,theauthorsattributethenegativeresultstotwofactors:thepossibilitythatbusinessadviceenablesweakfounderstoformmarginalbusinesses;andtheover-provisionofnon-intensivesupportinthisparticularprogramme.24

Table 2: Summary of effects of business advice & mentoring programmes

Finding No of studies Evaluation reference numbers

Supportmayhelp(positivecoefficientsonatleastonefirmoutcome)

17 162,163,165,166,167,168,169,174,177,183,269,182,184,195,284,285,

286,287

Supportdoesn’twork(nostatisticallysignificantfindings)

4 170,195,276,282

Supportmaybeharmful(nopositive,somenegativecoefficients)

2 172,217

Total 23

Table3summariseskeycharacteristicsoftheprogrammesintheshortlistedevaluations.Itoutlineswhichcharacteristicswereaddressedbywhichevaluations,andalsowheredirectcomparisonsweremade.

22 Study195.23 Study165foundanegativeeffectonexports,butapositiveeffectonthreeotherfirmsoutcomes.Wediscussthis furtherbelow.24 Study172.


Table 3: Overview of programme features

No. of studies

Findings (on balance of evidence)

Evaluation Reference Numbers

Overall

Programme objectiveEmployment 2 Mixed

Mostprogrammestargetmultipleorvagueobjectives.Onlythreeevaluationsfoundaclearlinkfromprogrammeobjectivestoimprovedfirmoutcomesforthatobjective.

-Helps(1) 287

-Harmful(1) 217

Exports 3 Mixed

-Helps(2) 169,184

-Harmful(1) 165

Hybrid 18 Mixed

-Helps(13) 162,163,166,167,168,174,177,183,183,269,284,285,

286

-Zero(4) 170,195,276,282

-Harmful(1) 172

National vs local deliveryNational 8 Mixed

Thereisnoevidencethatonelevelofdeliveryismoreeffectivethanothers

-Helps(5) 169,182,183,184,269

-Zero(2) 170,195

-Harmful(1) 165

Local 6 Mixed

-Helps(4) 162,168,174,286

-Zero(1) 276

-Harms(1) 172

Both 9 Mixed

-Helps(7) 163,166,167,177,284,285,

287

-Zero(1) 282

-Harmful(1) 217


No. of studies



Overall

Public vs private led deliveryPublic 3 Mixed

Thereisnoevidencethatonedeliverymodelismoreeffectivethanothers.

-Helps(2) 183,184

-Harmful(1) 165

Private 4 Mixed

-Helps(3) 168,182,286

-Zero(1) 170

Hybrid/Both 14 Mixed

-Helps(10) 162,163,166,167,174,177,269,284,285,

287

-Zero(2) 282,276

-Harmful(2) 172,217

Notstated 2 169,195

Delivery model/ techniqueLighttouch 1 Helps 168

Programmesthatusedahands-on,‘managedbrokerage’approachmayperformbetterthanthoseusinga‘lighttouch’approach(althoughthisconclusionisbasedononecomparisonstudy).

Managedbrokerage 4 Helps 162,174,269,286

Pipelineforcing 4 Mixed

-Helps(2) 163,287

-Zero(1) 282

-Harmful(1) 217

Exportpromotion 4 Mixed

-Helps(3) 169,183,184

-Harmful(1) 165

Other 9 Mixed

-Helps(5) 167,177,182,284,285

-Zero(3) 170,195,276

-Harmful(1) 172

Comparisonbetweentypesofsupport

1 Intensivesupportmoreeffectivethanlighttouch

166


No. of studies



Overall

Sector specificitySectorspecific 2 Mixed

Thereisinsufficientevidencetoestablishtherelativeeffectivenessofsectorspecificprogrammescomparedwithmoregeneralprogrammes.

-Helps(1) 182

-Harms(1) 172

General 21 162,163,165,166,167,168,169,170,174,177,183,184,195,217,269,276,282,284,285,286,287


Detailed findings

Thissectionofthereportlooksatwhetherthereisanyevidenceofalinkbetweenspecificprogrammefeaturesandoutcomes.Forexample,weconsiderwhetherpublic-ledorprivate-leddeliverymodelsareassociatedwithbetteroutcomes.

Thisisnotstraightforwardbecausepossiblerelationshipscouldalsobeexplainedbyanumberofother‘confoundingfactors’thatmaybeinplay.Inaddition,thereissignificantvarietyinthetypesofsupportthatareexploredinthedifferentevaluationswehaveconsidered.Forexample,programmesmaybebroadlycategorisedbythosethatofferassistancetoindividualswhowishtostartbusinessesandthosethatassistalreadyestablishedfirms.Aswellasofferingaverydifferenttypeofsupport,themeasurablegoalsofsuchprogrammesarealsocontrasting,makingbroadcomparisonparticularlychallenging.

Inouremploymenttrainingreview,wewereabletoaddressthisproblembylookingatstudiesthatmadeexplicitcomparisonsbetweenprogrammes.Forbusinesssupport,suchcomparisonsare,unfortunately,notreadilyavailableandsowecanonlylookforanygeneralpatternorcorrelationbetweenthefeatureandtheoutcomewe’reinterestedin(forexample,deliverytypeandemployment).

Programme objectives

Most programmes target multiple or vague objectives. Only three evaluations found a clear link from programme objectives to improved firm outcomes for that objective.

Thefocusoftheprogrammeswelookedatvariessignificantly,asreflectedinanumberofmeasurableobjectives.Whileasmallnumberofprogrammeshaveexplicit,singularobjectivesthatareestablishedfromtheoutset,suchasincreasingtheexportsofsmallfirms,othersformpartofwiderpackagesofmarketinterventionswithmorecomplexsetsofgoals.

Thesecontrastingobjectivesreflectdifferentialpreferencesthatpolicymakershaveintermsofprogrammedesign,andalsotheultimatebeneficiariesofthesupport.Ashighlightedpreviously,anumberoftheprogrammescoveredinourreviewarefocusedongettingunemployedindividualsintoworkbysupportingthemtoestablishtheirownbusinessesandbecomeself-employed.Theseareoftenpartofwideractivelabourmarketpoliciesthatultimatelyaimtoincreaseemploymentand

08


durationoftimeinwork.Programmesthatareaimedtowardsexistingsmallbusinessestakethisastepfurther,sharingacommonunderlyinggoal,encouragingsmallbusinessesthatarealreadyestablishedtogrowfurther.Thismightbeachievedby,forexample,increasingexportsorimprovingproductivityinordertoboostsalesandprofits.Thepictureisfurthercomplicatedintheevaluationstage,withsomeprogrammesbeingdirectlyevaluatedagainststatedobjectives,whileinothercasesresearcherslookatabasketofwideroutcomes.

Weareinterestedinwhetherprogrammesmeettheirstatedobjectivesandwhethermorecomplexprogrammesaremoreorlesslikelytobesuccessful.Forthepurposesofthereview,weclassifyprogrammesintotwogroups:afirstgroupwhereobjectivesareclearlystatedandasecondgroupwhereobjectivesaremultipleornotclearlystated.Programmesinthefirstgroupallhavesingleobjectivesandaimtoeitherincreaseemploymentorexports.Programmesinthesecondgrouptendtohavemultipleobjectives–forexample,BusinessLink.

BusinessLinkprovedespeciallyhardtoclassify.25Eachevaluationemphasisesthemainobjectivesslightlydifferently,resultinginalackofclarityabouttheoverarchingcoreaims.SeveralreferenceHartandRoper(2003)who,insummary,statethatBusinessLinkwasdesignedtoimprovetheperformanceofthesmallbusinesssector,specificallyexistingbusinessesofbetween20and100employees.26However,giventhattheprogrammewasestablishedasanetworkoflocally-runoffices,theexactscopevariedasaresultoflocalpreferences.Indeed,HartandRoper(ibid)suggestthat,insomecases,start-upswereassistedaswellasexistingsmallfirms,somethingthatispickedupbytwootherevaluationsthatciteraisingself-employmentratesandraisingentrepreneurshipindeprivedareasasadditionalobjectives.27Inordertoreflectthesevariations,wehavecategorisedallBusinessLinkevaluationsunderthe‘hybrid’bracket.

Table 4: Programme Objectives

Objective No of studies Study reference numbers

Increasedemployment 2 217,287

Increasedexports 3 165,169,184

Hybrid(multipleobjectives) 18 162,163,166,167,170,172,174,177,182,183,195,269,276,282,

285,286

Onlytwoevaluationsconsiderprogrammesthatarefocusedsolelyonincreasingemployment(butnotethatoneofthesescored5ontheSMS).Bothfocusedspecificallyonself-employment,anddemonstratedmixedresults.Self-employmenttrainingandcoachingprovidedtorecipientsofBridgingSupport(Germany)wasgenerallylesssuccessful.28Incontrast,theWashingtonandMassachusettsSelf-EmploymentDemonstrators(US)wereshowntoincreasethelikelihoodofbeinginemploymentby14%,withtransitionoccurring5.9monthsmorequickly.Forself-employment,thelikelihoodincreasedby5%withtransition2.4monthsquicker.29

Thethreeevaluationsthatassessedtheeffectivenessofexport-onlyfocusedprogrammesallfoundatleastonesignificantpositiveeffectforfirmoutcomes.Oneofthesedidnotlookdirectlyatfirmexports

25 Studies166,167,177,284and285.26 Study167.27 Studies177and284.28 Study217.29 Study287.


butfoundpositiveeffectsforindirectmeasuresofsuccess.30Ofthetwothatdidlookatexports,whilstAftercare(UK)wasfoundtonoticeablyincreasethelevelandgrowthofexportintensity,31theOverseasMarketIntroductionService(UK)ledtoamarginallynegativeimpactonexportscomparedtofirmsinthecontrolgroup(usinganSMS3methodology). 32Thisisdespitepositiveeffectsonotherindicatorssuchasturnovergrowth(£611,000higher),employment(sevenadditionaljobsperfirmonaverage)andfirmsurvivalprobability.

Theremainingprogrammeshaveobjectivesthatareeithernotclearlydefinedorinvolveseveraldifferentoutcomemeasures.TheevaluationofBusinessLink,forexample,showsmixedresults:inoneexample,BusinessLinkwasfoundtogenerate4.4%employmentgrowth,butnosignificanteffectsonsales;33anotherfoundsimilarresults,witha2%upliftinemploymentgrowthbutnoimpactonsales;34whilstathirdfoundonlytentativeevidencethatBusinessLinkincreasedproductivitygrowth.35TheGATEprogramme(US),whichtargetedasuiteofentrepreneurshiptrainingtothosereceivingunemploymentbenefits,wasprincipallyaimedatincreasingself-employment,butwasalsodesignedtoraiseindividualearningsanddevelophigh-growthfirms.36However,increasesinthenumberofhoursspentinself-employmentwereoffsetbydecreasesinformalemploymenthours,thusresultinginaneutralneteffectonnumberofhoursinemployment.Householdandbusinessearningswerenotsignificantlydifferentfortreatmentandcontrolgroups.

Wehavenotfoundanyevaluationsthatexplicitlycompareprogrammeswithdifferentoutcomes.Overall,wefindnostrongdifferencesbetweenprogrammeswithmultipleobjectivesandprogrammeswithmorefocusedobjectives.Thelackofaclearlinkbetweenprogrammeobjectivesandspecificmeasuresofsuccessinthemajorityofcasesmakesitveryhardtoassesstheoveralleffectivenessofbusinessadviceprogrammes.Ourfindingshereechoexistingevidencereviewsonbusinessadviceandmentoring.37

National vs. local delivery

We found no evidence that would suggest one level of delivery is more effective than others.

The23evaluationsinvolveddeliverymodelsforbusinesssupportatseveraldifferentscales.Someprogrammesweredeliveredthroughnational-levelorganisations,bothgovernmentdepartments38andarms-lengthstateagencies,39whilstotherswerewhollydevolvedtotheregionalorlocallevel.40However,themajorityofprogrammescoveredinourreviewweredeliveredbyaformofpartnership,ofteninvolvingoverarchingnationalcontrolwithelementsofdevolvedpoweranddelivery.

30 Study184.31 Study169.32 Study165.33 Study28534 Study17735 Study16736 Studies163,282.37 ForexampleCurran(2000),Bennett(2008).38 UKTradeandIndustryprogrammesinstudies165,169and183.39 Studies166,167,177,284and285assessBusinessLink,whichwasanarms-lengthagencyadministeredand managedbyBISbeforelaterbeingdevolvedtotheregionallevel.40 Initslatteryears,controloverBusinessLinkwasdevolvedtoRegionalDevelopmentAgencies,with individual‘Links’incorporatedindividuallyasentrepreneurialorganisations(seestudy166formoredetails).The PLATOnetworkwasestablishedandrunbyanindependentlocalassociationofbusinesses,withfinancialsupport fromtheregionalgovernment.


Table 5: National or Local Delivery

Delivery model No of studies Study reference number

National 8 165,169,170,182,183,184,195,269

Local 6 162,168,172,174,276,286

Hybrid 9 163,166,167,177,217,282,284,285,287

Noneoftheevaluationslooksdirectlyatthequestionofwhetherprogrammesaremoresuccessfulwhentheyarelocally,regionallyorcentrallymanaged.Whenweclassifiedevaluationsaccordingtothelevelatwhichtheprogrammeisdelivered,wefoundnoevidencethatwouldsuggestonelevelofdeliveryisinherentlymoreeffectivethanothers.Thisreflectsthefindingsofotherevidencereviews.41Bennett(2008)notesthatdecentralisationofbusinessadviceislikelytohaveprosandcons:localknowledge,take-upandserviceperformancemayrise,butcostsandservicecomplexitymayalsoincrease.

Public vs. private-led delivery

We found no evidence that would suggest one delivery model (public, private, hybrid) is more effective than others.

Whilethereviewfocusesonevaluationsofpublicly-fundedprogrammes,themanagementanddeliveryofsuchprogrammesisoftendividedandsharedbetweenpublicandprivatesectororganisations.

Inseveralcases,publicsectorfundingisprovidedtoestablishedbusinessnetworkstorunsupportandnetworkingprogrammes,withtheoverallsteerisverymuchprovidedbyprivatefirms.42However,inthemajorityofcases,ahybridstructureisutilisedwhichinvolvesoverarchingprojectmanagementbypublicsectoragenciesandthesub-contractingofspecificadviceandothersub-programmestoamixtureofprivatesectorgroups.43Often,businessesorindividualsarescreenedbycustomer-facingbusinessadvisorsfrompublicsectororganisations,beforethenbeingreferredtoamorespecialistadviserfromtheprivatesector.44Somedirectlysubsidisedprivatesectorbusinessconsultancyorprofessionalservices,forexamplethroughtheprovisionof‘consultancyvouchers’.45

Table 6: Public / private delivery models

Delivery model No of studies Study reference number

Public-led 3 165,183,184

Private-led 4 168,170,182,286

Hybrid 14 162,163,166,167,172,177,217,269,276,282,284,285,287

Notstated 2 169,195

Ofthethreeevaluationsthatlookedatpublic-ledprogrammedelivery,theresultscangenerallybe

41 Curran(2000),Bennett(2008).42 Studies168and286providegoodexampleofthisapproach.43 Seestudies162or163forexamples.44 BusinessLinkutilisedthismodel.45 Forexample,Study170.


describedasmixed.Twostudiesfindpositiveeffectsonatleastonefirmoutcome.Forexample,firmsutilisingtheUKTI’sregionalsupportnetwork(UK)experiencedbetween8%and15%increasedturnovergrowth.46WhiletheUKTIPassportprogramme(UK)didgenerateturnovergrowthwhichwas20%fasterthanthecontrolgroup(albeitwithoutleadingtogreaterprofits).47Thefinalprogrammeismoreunusualshowingpositiveimpactsintermsofemployment,turnoverandproductivity,butnegativeimpactsonexports(theprogrammeobjective).

Inthecaseofthesimilarlysmallsampleoffourprivate-ledprogrammescoveredintheevaluations,theresultsarealsosomewhatmixed.Oneevaluation,whichlookedattheRegionalBusinessDevelopmentprogramme(Sweden),foundthatthedistributionofprivate-sectorconsultancyvouchershadnosignificantimpactonemploymentorproductivity.48Programmesthatutilisedamore‘handson’managementstyletendedtoshowmorepositiveresults(apointwereturntobelow).Twoweremanagedandrunbyprivatesectorbusinessnetworksandprovidedbothmentoringandnetworkingopportunitieswithbusinessexecutivesstillinactiveemploymentwithotherfirms.49Positiveimpactswereseenonsalesandassetvalue.Inthecaseofthelatterstudy,firms’netassetvalueincreasedbynearly5%.50

Delivery Technique

Programmes which used a hands-on, ‘managed brokerage’ approach may perform better than those using a light touch approach (although this conclusion is based on only one comparison study).Taken at face value, this suggests that a strong relationship and a high level of trust between advisor and client may be important to the delivery of positive programme outcomes. It is not clear, however, which of these two approaches is more cost-effective.

Theprogrammescoveredinourreviewuseawidevarietyofdifferenttechniquesintheirdelivery.IntheirreviewoftheBusinessLinkservice,Moleetaldevelopedaseriesofmodelswhichbroadlycategorisedhowdifferentaspectsoftheadviceserviceoperated.51Wehaveadaptedthiscategorisationtoallowustocomparedifferentprogrammetypesinourreview:

• Light Touch programmesarethosewithlittleornoengagementorfollowupbetweenadvisorsandtheirclients,forexample,thosewhichprovideholisticadvicewhichdirectsthemtowardsothersources;52

• Managed Brokerage programmeswereessentiallysimilar,butwiththeadvisorinmoreofarelationship-buildingposition;theyprovidemoretargetedindividualadvice;53

• Pipeline Forcing programmesinvolvemoreintensiveadvicetotheclientinordertogetthemtotheendofthepipeline.54Wehavecategorisedmostoftheprogrammesfocusedonself-employmentaspipelineforcing.

Wealsoseparatedoutthreeprogrammeswhichfocusedsolelyonexportpromotion(aswefeltthatthesedidnotfitappropriatelyintoanyofthesethreecategories).

46 Seestudy183.47 Study184.48 Study170.49 Seestudies168and286.50 Study286.51 Study177.52 Seestudy167.53 Study162isagoodexampleofthis,withconsultantsdiagnosingproblemsthatpreventgrowth,suggesting solutionsandassistingwithimplementation.54 Study287fitsneatlyintothiscategory,theassistancebeingshorttermandintense.


Table 7: Delivery Techniques

Delivery technique No of studies Study reference number

Lighttouch 1 168

Managedbrokerage 4 162,174,269,286

ComparisonofLighttouchvsmanaged

1 166

Pipelineforcing 4 163,217,282,287

Other 9 167,170,172,177,182,195,276,284,285

Exportpromotion 4 165,169,183,184

Oneevaluationlookedata‘lighttouch’programme,whichfoundthatbusinessadvisoryservicesresultedingreaterlevelsofsales,withsomeevidencetosuggestthattheprobabilityofpatentsisalsoincreased.Itshouldbenoted,however,thattheservicesreceived,though‘lighttouch’innature,weretailoredtotheneedsofbusinesses.55

Programmesutilisingmanagedbrokeragetechniquesaspartoftheirdelivery,whichwouldinvolvesomelevelof“fundingandmoreinteractionbetweentheclientsandadvisorinarelationship-buildingapproach”,tendedtobesuccessfulinimprovingatleastonefirmoutcome.Allfourevaluationsfoundsomepositiveresults,oneofwhichwastheresultofanSMS5randomisedexperiment.56Inthiscase,monthlyprofitswerereportedtobe120%higherversusthecontrolgroup.

Onthebasisofthesestudies,itishardtoreachanyconclusionaboutwhichapproachismoreeffective.Thereis,however,oneexampleofanevaluationthatdirectlycomparesintensiveandlighttouchsupportadministeredtobusinesses.57Itfoundthatmoreintensiveadvisoryservicesweremoreeffectivethan‘lighttouch’methods,headingto2.2%higherannualemploymentgrowth.Itisimportanttonotethatthisstudydidnotuserandomisedallocationtechniques;insteadsupportmethodswerebroadlycategorisedandusedtomakelatercomparisons.Takenatfacevalue,thisevaluationsuggeststhatastrongrelationshipandahighleveloftrustbetweenadvisorandclientmaybeimportanttothedeliveryofpositiveprogrammeoutcomes.However,itisimportanttonotethatsuchadvicemaybemoreexpensive,andsoitisunclearwhichapproachismorecost-effective.

Severalprogrammes(notablyBusinessLink)usedamixtureofthesetechniques,oranothertechnique.Onceagain,thecategorisationofevaluationsoftheBusinessLinkpolicywaschallengingbecauseeachevaluationprovidesslightlydifferentmaterialonthecontentoftheprogramme,andthescopeoftheservicesprovidedvariedovertimeandbetweenareas.Themajorityofevaluationssuggestamixedapproachoflighttouchandmanagedbrokerageandso,forconsistency,wehaveclassifiedalltheBusinessLinkevaluationsas‘mixed’.Interestingly,resultsarefarmoremixedforthesestudies.Ofthoseclassifiedas‘mixed’or‘other’approaches,fivefoundpositiveresultsforatleastonefirmoutcome.58Oftheremainder,threefoundnoevidenceofpositiveeffects,whileonefoundnegativeimpactsofbusinessadvice.59

Programmesthatfocusedsolelyonexportpromotionalsofoundpositiveeffectsforsomefirmoutcomes,thoughitisimportanttoacknowledgethatthethreeevaluationsincludedinourshortlistalllookedat

55 Study168.56 Studies162,174,296and286.Study162isanRCT.57 Study166.58 Studies166,177,182,284and285.59 Studies170,195and276foundinsignificanteffects.Study172foundnegativeeffects.


initiativesdeliveredbyUKTI,thuswecannotconcludewithcertaintythatallexportfocusedprogrammesaregenerallysuccessfulintheiroutcomes.Wediscussfindingsfromthesethreestudiesfurtherbelow.

Sector targeting

There is insufficient evidence to establish the relative effectiveness of sector specific programmes compared with more general programmes.

Themajorityoftheprogrammesinourreviewwerenotrestrictedbyfirmsector,butwedidfindtwoevaluationsofprogrammesthatweremoretargeted.Onewastailoredspecificallytoadvancedtechnologyfirms,whilstanotherwasonlyopentoBritish-basedmanufacturers.60

Table 8: Sector Targeting

Targeting No of studies Study reference number

Sector-specific 2 172,182

General 21 162,163,165,166,167,168,169,170,174,177,183,184,195,217,269,276,282,284,285,286,287

Oneofthetwosector-specificstudiesshowednostatisticallysignificanteffectsonemploymentandfirmsurvival61.Theothershowedsignificantpositiveeffectsonsalesandemploymentgrowthforsmallandmediumenterprises.62Overall,therearenotenoughstudiesthatfocusonsector-specificprogrammestosaywhethertheseprogrammestendtoleadtomore/lesspositiveoutcomesthanthegeneralprogrammes.Therearealsonodirectcomparisonsavailable.

Programme outcomes for firms

Business advice programmes show consistently better results for productivity and output than for employment. Results for sales, profits and exports are also mixed, although in the latter cases the count of studies is small.

Assetoutabove,mostprogrammesinourshortlisttargetmultipleorvagueobjectives,whichmakesitdifficulttoassessprogrammesuccessdirectlyagainstobjectives.Wewereonlyabletodothisdirectcomparisonforfiveofthe23shortlistedevaluations.

Forthatreason,wealsolookdirectlyatprogrammeoutcomes,regardlessofstatedobjectives.Inthisinstance,weareinterestedinwhetherbusinessadviceinterventionsworkbetterforsomefirmoutcomesthanothers.Inparticular,wewanttoknowwhetherprogrammestendtohavepositiveimpactsonfirmoutcomesthatrelatecloselytokeylocalgrowthoutcomes(particularlyproductivityoremployment)aswellasonrelevantfirm-leveloutcomes(suchassales)thatmightnotbetiedtolocaleconomicgrowthobjectives.

60 Study172istechnology-focused,182manufacturing-focused.Afurtherevaluation(study183)lookedspecifically atprogrammeimpactfromtheperspectiveofthemanufacturingandrealestateindustries,thoughforconsistency wehavedecidednottoincludeitinourdiscussionhereastheprogrammeitselfwasnotitselfsector-targeted.61 Study172.62 Study182.


Table 9: Programme outcomes

Outcome type Studies Positive Zero Negative Share positive

Productivity 165,166,167,169,284 4 1 4/5

GVA 170,174,183,269,286 4 1 4/5

Sales/turnover 162,165,167,168,177,182,184,195,269,282,284,285

7 5 7/12

Employment 163,165,166,167,169,170,172,174,177,182,195,217,

282,284,285,286,287

8 8 1 8/17

Employmentduration/survival

172,217 1 1 0/2

Assets/capital 172,184,286 2 1 2/3

Exports 165,169 1 1 1/2

Profits 162,184,195 1 2 1/3

Earnings/income 163,276,287 1 2 1/3

Ownership 282 1 0/1

Unemploymentbenefits 163 1 0/1

Patents 168 1 0/1

ResultsaregiveninTable9.Wefindthatthe10evaluationslookingatproductivityoroutput(GVA)showthemostconsistentlypositiveresults(8/10positive).Incontrast,ofthe17studiesthatlookatemploymentoutcomes,onlyeightreportpositiveprogrammeeffects.Asmanyevaluationsreportzeroeffectsonemployment,andoneevenfindsnegativeeffects.Forthetwoprogrammesthatlookatemploymentdurationorsmallbusinesssurvival,resultsaresubstantiallyworsethanotheremploymentcases.Resultsforsalesoutcomesareslightlybetterthanforemploymentbutworsethanforproductivity(withsevenoutof12programmesreportingpositiveresults).

Wealsofindevaluationsofarangeofotheroutcomes,someofwhichareonlyrelevantforspecificprogrammes(forexampleunemploymentbenefits,whichisonlyreportedforProjectGATE,aprogrammetargetedspecificallyatjoblesspeople).63Overall,foranygivenoutcomeresultsarequitemixedandlimitedtoasmallnumberofevaluations.

Inprinciplewemightworrythatsomeevaluationsareonlyreportingsuccessfulfindings,orhidingnegativeresults.Inpracticeweseeanumberofzeroornegativeeffectsinourshortlistedstudies.Forinstance,study165evaluatesanexportsprogrammeandreportsnegativeeffectsonexports,butpositivecoefficientsonanumberofotheroutcomes(productivity,employmentandturnover).Oursiftstrategy(seemethodologysectionabove)shouldalsoremovestudieswhichmakeinflatedorinaccurateclaims.Itmightalsobethatproductivitybenefitsoccurfirst,whileemploymentbenefitstakelongertomaterialise.Evaluationsthattakealongertermperspectivewouldhelpusunderstandwhetherthisisthecase.

63 Study163.


Specific UK ProgrammesAnumberofourevaluationslookatthesameprogramme,allowingustoattemptcollectivejudgementsabouttheeffectivenessofspecificpolicies.

FivestudieslookspecificallyattheBusinessLink(UK)policy,examiningprogrammeeffectsonarangeoffirmoutcomes.64Theweightofevidencesuggeststhatthepolicyhashadmixedresultsacrosstheseoutcomes.ThreeofthefivestudiesconcludethatBusinessLinkhadapositiveandsignificantimpactuponemployment,between2.2%and4.4%.65Intensiveassistanceappearstobeparticularlyeffective.66However,impactonsalesappearstobeparticularlyweakinallstudies,withonlytentative(andstatisticallyinsignificant)indicationsthatBusinessLinkleadstoupliftsinsalesandproductivity.67Thisisparticularlynoteworthygiventhatoneofthemainobjectivesofthepolicyistoimprovetheproductivityofsmallbusinesses.Anotherthingtonoteisthatallofthefiveevaluationsareranked3ontheSMSscale,thelowestqualitytypeofevaluationthatwehaveconsidered.

WealsofoundfourstudiesthatconsidertheimpactsofUKTI(UK)programmesthatassistsmallfirmstoenternewforeignmarketsandincreasetheirexports.68Threeofthefourevaluationsshowpositiveprogrammeeffectswhenlookedatinthecontextofseveraloutcomevariables.69Surprisingly,onlytwoofthefourstudiesofexport-focusedprogrammeslookspecificallyattheimpactonexports;ofthoseonefindsasignificantpositiveimpactonexportingitself,70whilstoneactuallysuggeststheopposite.71GiventhatacoreaimofUKTIistoallowfirmstoenterforeignmarkets,thisraisesquestionsabouttheoveralleffectivenessofsomepoliciesindeliveringtheirstatedobjectives,evenifsupportdoesappeartobroadlybeneficialtofirms.

64 Studies166,167,177,284and285.65 Studies177,284and285.66 Studies284and166comparetheeffectivenessoflighttouchandintensiveservices.67 Studies167and284.68 Studies165,169,183,and184.69 Studies165,169and18470 Study16971 Study165


Summary of findings

What the evidence shows1. Business support and advice had a positive impact on at least one business outcome

in 17 out of 23 evaluations. Fourevaluationsfoundthatbusinessadvicedidn’tworkandtwostudiesfoundthatbusinessadvicemightbeharmful.(Onestudyfoundnegativeeffectsagainstthestatedobjective,althoughotherpositiveeffectswerealsorecorded.)

2. Business advice programmes show consistently better results for productivity and output (GVA) than for employment. Evaluationslookingatproductivityoroutputeachshowpositiveeffectsineightoutof10cases.Bycontrast,evaluationsthatlookatemploymentreportpositiveeffectsinonlyeightoutof17cases(witheightcasesofzeroeffects,andzeroornegativeresultsfortwofurtherstudieslookingatself-employmentduration/survival).Resultsforsales,profitsandexportsarealsomixed,althoughinthesecasesthecountofstudiesissmall.

3. Programmes that used a hands-on, ‘managed brokerage’ approach may perform better than those using a light touch delivery model. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyanddoesnotconsidercost-effectiveness.

Where the evidence is inconclusive4. In most cases programmes had multiple or vague objectives. Ofthefiveevaluations

thatlookedatprogrammeswithaclearlyidentifiedobjective(i.e.increaseemployment/exports)three found positive effects.

5. We find no strong differences between programmes with multiple objectives and programmes with more focused objectives. Thelackofaclearlinkbetweenprogrammeobjectivesandspecificoutcomemeasuresinthemajorityofcasesmakesitveryhardtoassesstheoveralleffectivenessofprogrammesupport,andtounpickwhatfeaturesoftheprogrammesarelinkedtospecificoutcomesforfirms.Itisalsolikelytoraiseoperationalcomplexityandprogrammerisks(suchasthechanceofunintendedconsequences).Ourabilitytounderstandwhatworksforbusinessadvicewouldbeimprovedifpoliciesweredesignedwithclearobjectivesthatrelatedtomeasurable,relevantfirmoutcomes.

09


6. We found no evidence that would suggest one level of delivery is more effective than others. Resultsforbothnationally-ledandlocally-ledprogrammesismixed,asitisfor‘hybrid’programmesthecombinenationalandlocaldeliverystructures.

7. It is difficult to reach any strong conclusions on the effectiveness of private-led versus public-led delivery. Resultsappeartobemixedforbothpublic-ledandprivate-ledprogrammes.

8. Overall, it is difficult to reach any conclusions on the link from specific programme features to better firm outcomes.Toimproveourunderstandingofwhatworks,policymakersshoulddesignprogrammesthatallowforexplicitcomparisonsofprogrammefeatures–forexample‘lighttouch’versus‘handson’deliverymodels,ordifferentkindsof‘accelerator’approachesforstartups(seebelow).

Where there is a lack of evidence 9. Thereisinsufficientevidencetoestablishtherelativeeffectivenessofsector-specific

programmescomparedwithmoregeneralprogrammes.

10. Wefoundnohigh-qualityimpactevaluationsthatexplicitlylookathowbusinessadviceaffectsoutcomesforfemale-headedorBMEbusinesses.Moreresearchisneededhere.

11. Wefoundtwohigh-qualityevaluationsofprogrammesaimedatincubatingstart-ups(referredtoaboveas‘pipelineforcing’).Bothprogrammesweretargetedatunemployedpeopleandshowmixedresultsoverall.However,thereisalackofimpactevaluationforDragons’Den-typeacceleratorprogrammesthataimtolaunchhigh-growthbusinessesandinvolvecompetitiveentry.


How to use this review

Thisreviewconsidersaspecifictypeofevidence–impact evaluation.Thistypeofevidenceseekstoidentifyandunderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Toputitanotherway,theyask,‘didthepolicywork’?

Thefocusonimpactreflectsthefactthatweoftendonotknowtheanswerstobasicquestionsthatmightreasonablybeaskedwhendesigninganewpolicy,suchas‘whatwillworkbestinthisscenario’and‘whatapproachoffersbestvalueformoney’.Beingcleareraboutwhatisknownwillenablepolicymakerstobetterdesignpoliciesandundertakefurtherevaluationstostartfillingthegapsinknowledge.

Supporting and complementing local knowledgeTheevidencereviewsetsoutanumberof‘BestBets’,whichoutlinetheapproachestobusinessadviceandmentoringhaveperformedmoststrongly,basedonthebestavailableimpactevaluations.

The‘BestBets’donotaddressthespecificsof‘whatworkswhere’or‘whatwillworkforaparticularindividual’.Insomecasesevaluationsdo breakoutresultsbyareatypeordifferentgroups.Butevenwhentheydo,detailedlocalknowledgeandcontextremaincrucial.

Reflectingthis,theoverallfindingsfromtheevaluationsshouldberegardedasacomplement,notasubstitute,forlocal,on-the-groundknowledge.

Businessadviceandmentoringinterventionswillneedtobetailoredandcarefullytargeted.Anaccuratediagnosisofthespecificlocalbusinesschallengesneedstobethefirststeptounderstandinghowtheevidenceappliesinanygivensituation.

‘BestBets’alsoraiseanoteofcautionforpolicymakersiftheydecidetointroduceaprogrammethathasnotworkedsowellelsewhere.

Specific recommendations

• Whendesigningaprogramme,localpolicymakersshouldidentify one or two clear programme objectives,andthenidentifyoutcomemeasuresthatarebothclearlyrelated

10


totheprogrammeobjectives,andfeasibletomeasure.Manyoftheprogrammeswelookedathadoverlycomplexorvagueobjectives,whichmakesassessingeffectivenessdifficult.

• Businessadviceprogrammesarebestplacedtohelpfirms’productivity, profits and sales–ratherthanraisingemployment.

• Programmes that used a hands-on, ‘managed brokerage’ approach may perform better than those using a ‘light touch’ delivery model. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyanddoesnotconsidercost-effectiveness.

Helping to fill the evidence gapsAsshouldbeclearfromthisreview,therearemanythingsthatwedonotknowabouttheeffectivenessofbusinesssupport.Muchofthepolicydebatefocusesonverybroadquestionsabouttheinstitutionalstructuresthatareputinplacetosupportbusinesses.Yetoveralltheevidenceprovidesnoclearsteeronwhetheroneparticulartypeofdeliverymodel(public/private;national/local)ismoreeffective.

Tohelpimprovebusinessadviceprogrammes,wewouldliketoseefarmorefocusonrobustlyevaluatingtheimpactofparticularaspectsofadviceprogrammesandcomparingtheircost-effectiveness.Forexample,thecostsoflighttouchversusmoreintensivesupportvarydramatically,yetwefoundonlyoneevaluationthatdirectlycomparedtheeffectivenessofthesetwotypesofsupport.Similarly,only5ofthe23shortlistedstudiesincludedcost-benefitanalysisthatassesscost-effectiveness,andnotalloftheseusedmeasuresthatarecomparableacrossstudies.Thereisaclearneedformore,consistentcost-benefitanalysisinbusinessadviceimpactevaluations.Webelievethatfurtherevaluationsofthiskind,involving,forexample,theprovisionofdifferenttypesofadvicetosimilarfirms,shouldbeapriorityforimprovingourunderstandingofwhatworksinbusinessadvice.Localflexibilitythatallowsforgreaterexperimentationprovidesanidealopportunitytoundertakesuchevaluations.

TheCentre’slonger-termobjectivesaretoensurethatrobustevidenceisembeddedinthedevelopmentofpolicy,thatthesepolicesareeffectivelyevaluated,andthatfeedbackisusedtoimprovethem.Toachievetheseobjectiveswewantto:

• Workwithlocaldecision-makerstoimproveevaluationstandardssothatwecanlearnmoreaboutwhatpolicieswork,where;and

• Establishupaseriesof‘demonstrationprojects’toshowhoweffectiveevaluationcanworkinpractice.

Interestedpolicymakerspleasegetintouch.


References

Acs,Z.J.,Audretsch,D.,Braunerhjelm,P.,&Carlsson,B.(2004).TheMissingLink:TheKnowledgeFilterandEntrepreneurshipinEndogenousGrowth.London:CEPR.

Aghion,P.,Blundell,R.,Griffith,R.,Howitt,P.,&Prantl,S.(2009).TheEffectsofEntryonIncumbentInnovationandProductivity.ReviewofEconomicsandStatistics,91(1),20-32.

Akerlof,G.(1970).Themarketfor‘lemons’:qualityuncertaintyandthemarketmechanism..QuarterlyJournalofEconomics,84(3),488-500.

Bartle,D.,andMorris,M.(2010)Evaluatingtheimpactsofgovernmentbusinessassistanceprogrammes:approachestotestingadditionality.ResearchEvaluation,19,275–280.

Batra,G.,&Mahmood,S.(2003).DirectSupporttoPrivateFirms:EvidenceonEffectiveness.Washington,DC:WorldBank.

Bennett,R.(2008).“SMEpolicysupportinBritainsincethe1990s:whathavewelearnt?”EnvironmentandPlanningC,26,375-397.

Benus,J.,Shen,T.,Zhang,S.,Chan,M.andHansen,B.(2010)GrowingAmericaThroughEntrepreneurship:FinalEvaluationofProjectGATE.WashingtonDC,USDepartmentofLaborEmploymentandTrainingAdministration.

Benus,J.M.,Wood,M.L.,Grover,N.,andAbtAssociates(1994)AComparativeAnalysisoftheWashingtonandMassachusettsUISelf-EmploymentDemonstrations.Cambridge,Mass,AbtAssociates.

Bravo-Biosca,A.,Marston,L.,&Westlake,S.(2011).VitalGrowth:Theimportanceofhigh-growthbusinessestotherecovery.London:NESTA.

Breinlich,H.,Mion,G.,Nolen,P.,andNovy,D.(2012)IntellectualProperty,OverseasSales,andtheImpactofUKTIAssistanceinEnteringNewOverseasMarkets(UKTI).London,UKTI.

Bruhn,M.,Karlan,D.S.,andSchoar,A.(2012)TheImpactofConsultingServicesonSmallandMediumEnterprises:EvidencefromaRandomizedTrialinMexico.YaleUniversity,DepartmentofEconomics.

Cumming,D.J.,andFischer,E.(2012)PubliclyFundedBusinessAdvisoryServicesandEntrepreneurialOutcomes.ResearchPolicy41,467–481.

Curran,J.(2000).WhatisSmallBusinessPolicyintheUKfor?EvaluationandAssessingSmall.InternationalSmallBusinessJournal,18(3),36-50.

Fairlie,R.W.,Karlan,D.,andZinman,J.(2012)BehindtheGATEExperiment:EvidenceonEffectsofandRationalesforSubsidizedEntrepreneurshipTraining.NBERWorkingPaper17804,Cambridge,Mass.NBER.

Gibbons,S.,Nathan,M.andOverman,H.(2014).EvaluatingSpatialPolicies.TownPlanningReview,forthcoming.

Girma,S.,Görg,H.,andPisu,M.(2005).Quantitativeanalysisandlinkedmicro-datastudyofUKTIservices-Finalreport.London,UKTI.

11


Godel,M.,andMantovani,I.(2012)EvaluationoftheimpactandcosteffectivenessofUKTI’sregionalnetworksupport.London,UKTI.

Grossman,S.J.,&Stiglitz,J.E.(1980).OntheImpossibilityofInformationallyEfficientMarkets.TheAmericanEconomicReview,70(3),393-408.

Haltiwanger,J.C.,Jarmin,R.S.,&Miranda,J.(2010).WhoCreatesJobs?Smallvs.Largevs.Young.Cambridge,MA:NBER.

Hart,M.,andRoper,S.(2003)ModellingtheEffectsofPublicSupporttoSmallFirmsintheUK-ParadiseGained?In:EuropeanRegionalScienceAssociation,EuropeanRegionalScienceAssociationConference.Jyväskylä,Finland,August2003.

Huggins,R.,&Williams,N.(2009).Enterpriseandpublicpolicy:areviewofLabourgovernmentinterventionintheUnitedKingdom.EnvironmentandPlanningC:GovernmentandPolicy,27(1),19-41.

Kosters,S.,andObschonka,M.(2011)PublicBusinessAdviceintheFoundingProcess:AnEmpiricalEvaluationofSubjectiveandEconomicEffects.EnvironmentandPlanningC,29,577–604.

Mansson,J.,andWiderstedt,B.(2012).TheSwedishBusinessDevelopmentProgram:Evaluationandsomemethodologicalandpracticalnotes.In:EuropeanRegionalScienceAssociation,EuropeanSocietyofRegionalAnalysis.Bratislava,Slovakia,2012.

Mole,K.,Roper,S.,Hart,M.,Storey,D.,&Saal,D.(2006).EconomicimpactstudyofBusinessLinklocalservice:finalreport.London:DepartmentforBusinessEnterpriseandRegulatoryReform.

Mole,K.,Hart,M.,Roper,S.,&Saal,D.(2008).DifferentialgainsfromBusinessLinksupportandadvice:atreatmenteffectsapproach.EnvironmentandPlanningC:GovernmentandPolicy,26(2),315-334.

Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2009)AssessingtheEffectivenessofBusinessSupportServicesinEnglandEvidencefromaTheory-BasedEvaluation.InternationalSmallBusinessJournal,27,557–582.

Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2011)Broaderordeeper?Exploringthemosteffectiveinterventionprofileforpublicsmallbusinesssupport.EnvironmentandPlanningA,43,87–105.

Oberschachtsiek,D.,Scioch,P.(2011)Theoutcomeofcoachingandtrainingforself-employment:astatisticalevaluationofnon-financialsupportschemesforunemployedbusinessfoundersinGermany.Nuremberg,InstituteforEmploymentResearchoftheFederalEmploymentAgency(IAB).

Ram,M.,&Jones,T.(2008).Ethnic-minoritybusinessesintheUK:areviewofresearchandpolicydevelopments.EnvironmentandPlanningC:GovernmentandPolicy,26(2),352-374.

Robson,P.J.A.,andBennett,R.J.(2000)SMEGrowth:TheRelationshipwithBusinessAdviceandExternalCollaboration.SmallBusinessEconomics,15,193–208.

Rogers,M.,andHelmers,C.(2008)IntellectualpropertyandUKTIPassportfirms-Stage2Report.London,UKTI.

Sanders,C.(2002)TheImpactofMicroenterpriseAssistancePrograms:AComparativeStudyofProgramParticipants,Nonparticipants,andOtherLow‐WageWorkers.SocialScienceReview,76,321-340.

Sherman,L.W.,Gottfredson,D.C.,MacKenzie,D.L.,Eck,J.,Reuter,P.,&Bushway,S.D.(1998).PreventingCrime:WhatWorks,WhatDoesn’t,What’sPromising.WashingtonDC:USDepartmentofJustice.

Schoonjans,B.,Cauwenberge,P.V.,andBauwhede,H.V.(2013)Knowledgenetworkingandgrowthinservicefirms.TheServiceIndustriesJournal,33(11),1051-1067.

Schumpeter,J.(1962).TheTheoryofEconomicDevelopment.Berlin:Springer.

Stiglitz,J.(1979).EquilibriuminProductMarketswithImperfectInformation.AmericanEconomicReview,69(2),339-345.

Stiglitz,J.(2002).InformationandtheChangeintheParadigminEconomics.TheAmericanEconomicReview,92(3),460-501.

Storey,D.(1994).UnderstandingtheSmallBusinessSector.London:Routledge.

VanCauwenberge,P.,VanderBauwhede,H.,andSchoonjans,B.(2013)Anevaluationofpublicspending:theeffectivenessofagovernment-supportednetworkingprograminFlanders.EnvironmentandPlanningC,31,24–38.

Wren,C.,andStorey,D.(2002)Evaluatingtheeffectofsoftbusinesssupportuponsmallfirmperformance.OxfordEconomicPapers,54,334–365.


Appendix A: Evidence Reviewed

Number Reference

162 Bruhn,M.,Karlan,D.S.,andSchoar,A.(2012)TheImpactofConsultingServicesonSmallandMediumEnterprises:EvidencefromaRandomizedTrialinMexico.YaleUniversity,DepartmentofEconomics.

163 Benus,J.,Shen,T.,Zhang,S.,Chan,M.andHansen,B.(2010)GrowingAmericaThroughEntrepreneurship:FinalEvaluationofProjectGATE.WashingtonDC,USDepartmentofLaborEmploymentandTrainingAdministration.

165 Breinlich,H.,Mion,G.,Nolen,P.,andNovy,D.(2012)IntellectualProperty,OverseasSales,andtheImpactofUKTIAssistanceinEnteringNewOverseasMarkets(UKTI).London,UKTI.

166 Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2011)Broaderordeeper?Exploringthemosteffectiveinterventionprofileforpublicsmallbusinesssupport.EnvironmentandPlanningA,43,87–105.

167 Hart,M.,andRoper,S.(2003)ModellingtheEffectsofPublicSupporttoSmallFirmsintheUK-ParadiseGained?In:EuropeanRegionalScienceAssociation,EuropeanRegionalScienceAssociationConference.Jyväskylä,Finland,August2003.

168 Cumming,D.J.,andFischer,E.(2012)PubliclyFundedBusinessAdvisoryServicesandEntrepreneurialOutcomes.ResearchPolicy41,467–481.

169 Girma,S.,Görg,H.,andPisu,M.(2005).Quantitativeanalysisandlinkedmicro-datastudyofUKTIservices-Finalreport.London,UKTI.

170 Mansson,J.,andWiderstedt,B.(2012).TheSwedishBusinessDevelopmentProgram:Evaluationandsomemethodologicalandpracticalnotes.In:EuropeanRegionalScienceAssociation,EuropeanSocietyofRegionalAnalysis.Bratislava,Slovakia,2012.

172 Kosters,S.,andObschonka,M.(2011)PublicBusinessAdviceintheFoundingProcess:AnEmpiricalEvaluationofSubjectiveandEconomicEffects.EnvironmentandPlanningC,29,577–604.

174 VanCauwenberge,P.,VanderBauwhede,H.,andSchoonjans,B.(2013)Anevaluationofpublicspending:theeffectivenessofagovernment-supportednetworkingprograminFlanders.EnvironmentandPlanningC,31,24–38.

177 Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2009)AssessingtheEffectivenessofBusinessSupportServicesinEnglandEvidencefromaTheory-BasedEvaluation.InternationalSmallBusinessJournal,27,557–582.

182 Wren,C.,andStorey,D.J.(2002)Evaluatingtheeffectofsoftbusinesssupportuponsmallfirmperformance.OxfordEconomicPapers,54,334–365.

183 Godel,M.,andMantovani,I.(2012)EvaluationoftheimpactandcosteffectivenessofUKTI’sregionalnetworksupport.London,UKTI.

184 Rogers,M.,andHelmers,C.(2008)IntellectualpropertyandUKTIPassportfirms-Stage2Report.London,UKTI.

195 Robson,P.J.A.,andBennett,R.J.(2000)SMEGrowth:TheRelationshipwithBusinessAdviceandExternalCollaboration.SmallBusinessEconomics,15,193–208.

217 Oberschachtsiek,D.,Scioch,P.(2011)Theoutcomeofcoachingandtrainingforself-employment:astatisticalevaluationofnon-financialsupportschemesforunemployedbusinessfoundersinGermany.Nuremberg,InstituteforEmploymentResearchoftheFederalEmploymentAgency(IAB).

269 Bartle,D.,andMorris,M.(2010)Evaluatingtheimpactsofgovernmentbusinessassistanceprogrammes:approachestotestingadditionality.ResearchEvaluation,19,275–280.


Number Reference

276 Sanders,C.(2002)TheImpactofMicroenterpriseAssistancePrograms:AComparativeStudyofProgramParticipants,Nonparticipants,andOtherLow‐WageWorkers.SocialScienceReview,76,321-340.

282 Fairlie,R.W.,Karlan,D.,andZinman,J.(2012)BehindtheGATEExperiment:EvidenceonEffectsofandRationalesforSubsidizedEntrepreneurshipTraining.Cambridge,Mass.,NationalBureauofEconomicResearch.

284 Moleetal(2007)EconomicImpactStudyofBusinessLinkLocalService.London,DeprtmentforBusinessEnterpriseandRegulatoryReform.

285 Moleetal(2008)DifferentialgainsfromBusinessLinksupportandadvice:atreatmenteffectsapproach.EnvironmentandPlanningC,26,315–334.

286 Schoonjans,B.,Cauwenberge,P.V.,andBauwhede,H.V.(2013)Knowledgenetworkingandgrowthinservicefirms.TheServiceIndustriesJournal,33(11),1051-1067.

287 Benus,J.M.,Wood,M.L.,Grover,N.,andAbtAssociates(1994)AComparativeAnalysisoftheWashingtonandMassachusettsUISelf-EmploymentDemonstrations.Cambridge,Mass,AbtAssociates.


Appendix B: Search Terms and Sources

Source Search Terms

EconLit “businesssupport”ANDjobs

EconLit “businesssupport”ANDemployment

EconLit “businesssupport”ANDturnover

EconLit “businesssupport”ANDproductivity

EconLit “businesssupport”ANDmeta

EconLit incubatorANDjobs

EconLit incubatorANDemployment

EconLit incubatorANDturnover

EconLit incubatorANDproductivity

EconLit incubatorANDmeta

EconLit business*ANDmentoringANDjobs

EconLit business*ANDmentoringANDemployment

EconLit business*ANDmentoringANDturnover

EconLit business*ANDmentoringANDproductivity

EconLit business*ANDmentoringANDmeta

EconLit business*ANDnetwork*ANDjobs

EconLit business*ANDnetwork*ANDemployment

EconLit business*ANDnetwork*ANDturnover

EconLit business*ANDnetwork*ANDproductivity

EconLit business*ANDnetwork*ANDmeta

WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDjobs

WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDemployment

WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDturnover

WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDproductivity

WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDmeta

WebofScience(SCCI)viaEndnoteweb incubatorANDjobs

WebofScience(SCCI)viaEndnoteweb incubatorANDemployment

WebofScience(SCCI)viaEndnoteweb incubatorANDturnover

WebofScience(SCCI)viaEndnoteweb incubatorANDproductivity

WebofScience(SCCI)viaEndnoteweb incubatorANDmeta


Source Search Terms

WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDjobs

WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDemployment

WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDturnover

WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDproductivity

WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDmeta

WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDjobs

WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDemployment

WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDturnover

WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDproductivity

WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDmeta

WebofScience(SCCI)viaEndnoteweb mentor*ANDbusiness*ANDevaluat*

WebofScience(SCCI)viaEndnoteweb compan*ANDmentor*ANDevaluat*

WebofScience(SCCI)viaEndnoteweb mentor*andbusiness*ANDgrowth

GoogleScholar businessANDsupport

GoogleScholar “smallbusiness”ANDsupport

GoogleScholar businessANDmentoring

GoogleScholar entrepreneurANDsupport

GoogleScholar “localeconomicgrowth”AND“smallbusiness”

GoogleScholar SMEANDsupport

GoogleScholar micro-enterpriseANDsupport

GoogleScholar micro-firmANDsupport

GoogleScholar SMEANDmentoring

GoogleScholar businessadvi*

GoogleScholar “businesssupport”ANDmeta

www.gov.uk/publications(BIS,DCLG,HMTreasury,CabinetOffice,EconomicandSocialResearchCouncil,UKCommissionforEmploymentandSkills)

“smallbusiness”

www.gov.uk/publications “smallbusiness”ANDsupport

www.gov.uk/publications SME

www.gov.uk/publications SMME

www.gov.uk/publications micro-business

www.gov.uk/publications micro-enterprise


Source Search Terms

www.gov.uk/publications micro-firm

www.gov.uk/publications businessANDmentor

NationalAuditOffice-PublicationsSearch businesssupportjobs

NationalAuditOffice-PublicationsSearch businesssupportemployment

NationalAuditOffice-PublicationsSearch businesssupportturnover

NationalAuditOffice-PublicationsSearch businesssupportproductivity

NationalAuditOffice-PublicationsSearch incubator

NationalAuditOffice-PublicationsSearch businessmentoring

NationalAuditOffice-PublicationsSearch businessnetwork

IZAJournalofLaborEconomics Visualscanoffullpublicationslist

IZAJournalofLaborPolicy Visualscanoffullpublicationslist

IZAJournalofLaborandDevelopment Visualscanoffullpublicationslist

IZAJournalofEuropeanLaborStudies Visualscanoffullpublicationslist

IZA,ResearchinLaborEconomics Visualscanoffullpublicationslist

CEPR,DiscussionPapersonDevelopmentEconomics

Visualscanoffullpublicationslist

CEPR,DiscussionPapersonEconomicHistory Visualscanoffullpublicationslist

CEPR,DiscussionPapersonFinancialEconomics

Visualscanoffullpublicationslist

CentreforCities-Publications “businesssupport”

CentreforCities-Publications businessmentoring

CentreforCities-Publications network(incategory:businessandenterprise)

TheWorkFoundation Visualscanofreportssection

OECD businesssupport

OECD incubatorANDjobs

OECD incubatorANDemployment

OECD incubatorANDproductivity

OECD business+networkORnetworking+job

OECD business+networkORnetworking+employment

OECD business+networkORnetworking+productivity

OECD business+mentorORmentoring+jobs

OECD “businessmentoring”

REPECviaEconPapers “businesssupport”ANDjobs


Source Search Terms

REPECviaEconPapers “businesssupport”ANDemployment

REPECviaEconPapers “businesssupport”ANDturnover

REPECviaEconPapers “businesssupport”ANDproductivity

REPECviaEconPapers business*ANDmentor*ANDjobs

REPECviaEconPapers business*ANDadviceANDjobs

REPECviaEconPapers business*ANDadviceORadvis*ANDjobs

REPECviaEconPapers business*ANDadviceORadvis*ANDgrowth

REPECviaEconPapers SMEANDjobsANDsupport

REPECviaEconPapers SMEANDgrowthANDsupportORadvis*

REPECviaEconPapers SMEANDmentor*ANDemployment

REPECviaEconPapers SMEANDmentor*ANDjobs

REPECviaEconPapers SMEANDmentor*ANDgrowth

REPECviaEconPapers firm*ANDmentor*ANDemployment

REPECviaEconPapers firm*ANDmentor*ANDjobs

REPECviaEconPapers firm*ANDmentor*ANDgrowth

REPECviaEconPapers entrepreneur*ANDmentor*ANDproductivity

REPECviaEconPapers entrepreneur*ANDmentor*ANDjobs

REPECviaEconPapers entrepreneur*ANDmentor*ANDgrowth

REPECviaEconPapers business*ANDnetwork*ANDjobs

REPECviaEconPapers “smallbusiness”ANDnetwork*ANDjob*

REPECviaEconPapers business*ANDnetwork*ANDemploymentANDimpact*

REPECviaEconPapers business*ANDnetwork*AND“economicgrowth”ANDimpact*

REPECviaEconPapers business*ANDnetwork*ANDproductivityANDimpact*

REPECviaEconPapers SMEANDnetwork*ANDjob*ANDimpact*

REPECviaEconPapers SMEANDnetwork*ANDgrowth

REPECviaEconPapers SMEANDnetwork*ANDjob*

REPECviaEconPapers SMEANDnetwork*ANDproductivity

REPECviaEconPapers firm*ANDnetwork*ANDproductivityANDimpact

REPECviaEconPapers firm*ANDnetwork*ANDjobsANDevaluat*

REPECviaEconPapers firm*ANDnetwork*ANDjobsANDimpact*

REPECviaEconPapers firm*ANDnetwork*ANDgrowthANDimpact*


Source Search Terms

LEPwebsites(all) growthORjobs“businesssupport”

LEPwebsites(all) growthORjobsmentoring

IPPR “businesssupport”

IPPR mentoring

IPPR “businessadvice”

RDAarchives VisualscanofRDAwebsitesonNationalArchivewebsite

Eurofound “businesssupport”

Eurofound mentoring

Eurofound businessadvice

NESTA Visualscan,followedbysearchusingtermssuchas“businesssupport”&mentoring

UKCES Searchbytheme,keyterms(“businesssupport”)andvisualscanofevidencereports

CORDIS “businesssupport”

CORDIS “businesssupport”+impactjobsORgrowth

CORDIS mentoring+impactjobsORgrowth

CORDIS mentoring+businessjobsORgrowth

EuropeanCommission Visualscanwithsomesearches(“businesssupport”),butappearedtobeproblemwithsearchfunction

UNESCO Visualscan

TechnologyStrategyBoard(www.innovateuk.org)

smallANDbusiness

IPPR Visualscanoffullpublicationslist


Appendix C: Findings for RCT studiesAmongstthe23shortlistedevaluations,fourwereRandomisedControlTrialswhichscoredthemaximumfiveontheSMSscale.Theresultsofthetopscoringevaluationsaremixed.Twoofthetopfourscoringstudiesfoundpositiveresults,andtheothertwo(whichbothevaluatethesameprogramme)foundmixedresultswithminimalimpactonbusinessgrowthoutcomes.72

Study162looksataprogrammeinMexicotoprovidesubsidisedbusinessconsultingservicestomicro-,smallandmediumsizedbusinesses.Supportincludeddevelopingmissionstatements,visions,targetsaswellasadviceonaccounts,recordkeeping,pricing,teamworkandleadership.Volunteerswererandomlyselectedeithertoparticipateortoformpartofthecontrolgroupandfollow-upinterviewswereundertakentwoyearsafterprogrammeentry.Thestudyfoundverylargeproductivitygainsinfirmswhoreceivedtheadvicecomparedwiththecontrolgroup.Thosewhoparticipatedexperiencedincreasedprofits,andthereissomeevidenceofincreasedsales.Specifically,monthlysalesincreasedbyaround80%andprofitsincreasedby120%inthetreatmentgroupcomparedwiththecontrolgroup.Theaverageincreaseinprofitsisestimatedtoliebetween$7,600and$11,000permonth,comparedwithacostof$988permonthfortheconsultingservices.

Study287evaluatestwoUSprogrammesaimedatencouragingtheunemployedintoself-employment.TwoRandomisedControlTrialswerecarriedoutonfederally-fundeddemonstratorprojectsinWashingtonStateandMassachusetts.Thoseapplicantswhowererandomlyassignedtotheprogrammewereofferedentrepreneurialtraining,businesssupportandfinancialassistanceanalogoustounemploymentinsurancefortheearlyperiodofbusinessset-up.Thetwoprogrammesweredifferentintermsofdetail,butbothincludedelementsofclassroom-basedgrouplearningandinone-on-oneindividualadvicewithspecialistbusinesscounsellors.Followupsurveyswerecarriedoutaftertwoyears.Theevaluationfoundthatbothprogrammesincreasedthelikelihoodofentryintoself-employment21monthslater,andacceleratedmovesfromunemploymentintoself-employment(5.9monthsearlierfortheWashingtonprogramme,2.4monthsfortheMassachusettsprogramme).Bothprogrammesalsosignificantlyincreasedthechancesofbeingemployed/self-employed,whichwas14%higherforWashingtonparticipantsand5%higherforthoseintheMassachusettsinitiative.However,neitherprogrammesignificantlyincreasedparticipants’totalearnings,inpartbecausepeoplewereless likelytomoveintoaregularjob.Thedifferencesbetweenthetwoprogrammesseempartlyrelatedtowidereconomicconditionsinthetwostates;mostoftheUSwasinrecessionduringthedeliveryperiod,butWashington’seconomywaslargelyinsulatedfromthesemacrotrends.

TwoevaluationslookattheimpactsofProjectGATE(GrowingAmericaThroughEntrepreneurship),amicroenterprisesupportprogrammewhichaimedtosupportthedevelopmentoffledglingbusinesses,supportentrepreneurshipandmovepeoplefromunemploymenttoself-employmentaspartofawiderActiveLabourMarketProgramme.

Theprojectwassupportedbythefederalgovernmentanddeliveredlocallythrough14smallbusinessdevelopmentcentreswhichwerepubliclyownedanddeliveredanddeliveredasnon-profitorganisations.TheevaluationwascarriedoutbymeansofaRandomisedControlTrialatsevenofthe14locations(foururban,threerural)acrossthreeUSstates.Over4,000qualifyingapplicantswithabusinessideawereallocatedintoeithertreatmentorcontrolgroupsandtrackedbymeansoffollow-upsurveysforfiveyearsafterenteringtheprogramme.

Theofficialprojectevaluationfoundthatparticipantsweremorelikelytostartabusinessthanthe

72 Studies162,287


controlgroup,althoughthisimpactwasonlystatisticallysignificantduringthefirstfewquartersafterrandomassignment.73Businessesstartedbythetreatedgroupexperiencedsimilargrowthandprofitabilitytothosestartedbycontrolgroupmembers.Self-employmentandhoursworkedbyself-employedincreased(butthattheself-employmenteffectwasoffsetbyareductioninemploymentandemploymenthourswhichsuggeststhatindividualswere,onthewhole,transferringfromemploymenttoself-employmentratherthanmovingfromunemploymenttoself-employment.

Afollow-upevaluationbydifferentauthors,usingthesamedatasetaimstoprovideadditionalanalysisofProjectGATE’simpacts. 74Extendingtheoriginalstudy’sexplorationofself-employmentoutcomes,thisresearchconcludedthatmarginalbusinessescreatedasaresultofProjectGATEdonotsurviveinthelongrun.Self-employmentwas5.2%higherinthetreatmentgroup,butthisresultwasnolongerstatisticallysignificantafter18months.Theresearchersalsolookedatthesalesandemploymentoutcomesofnewbusinesses,findingnotreatmenteffect.Theprogrammealsohadnosignificanteffectonthelikelihoodofcreatinghigh-revenueorhigh-employment‘gazelle’firms.

Ingeneral,thesehigherqualityevaluationstendtofindmorepositiveprogrammeimpactsthanthelessrobustevaluations.TwooffouroftheRCTevaluationsfindpositiveresultscomparedto7of19SMS3studies.8oftheseSMS3studiesshowmixedresultscomparedtotwoRCTs(althoughtheseconsiderthesameprogramme).NoneoftheRCTsfindzeroorharmfuleffectsoverall,comparedwith4of19SMS3studies.Althoughtheseconclusionsarebasedonaverysmallnumberofstudies,theydosuggestthathighqualityevaluationmaybeimportantfordetectinganypositiveimpactofbusinessadviceprogrammes.

73 Study163.74 Study282.


Appendix D: Case Studies

Study 169: Matching; SMS level 3This2005studybySourafelGirmaandcolleaguesevaluatestwoprogrammesrunbytheUKTradeandInvestmentagency(UKTI).AftercareprovidessupportforinwardinvestorsalreadyintheUKandaimstoattractmoreinvestmentbytheseforeignfirmsbyhelpingthemintegratewithinthelocaleconomy.ThePassporttoExportservicesupportsdomesticfirmstoincreasetheirexportactivityandoffersarangeofservicessuchastraining,accesstoforeigncontracts,andexportdevelopmentservices.

ThefactthatUKTIchooseswhichoftheapplyingfirmsparticipatemakesithardertoassesstheprogramme’simpact.Therearetwosourcesofselectionbias:firstly,onlyfirmsthatexpecttobenefitwilllikelyapplyfortheprogram.Secondly,givenalimitedbudgetUKTImaybetemptedto‘pickwinners’byselectingfirmsthatlooklikelytosucceed.Overall,thismeansthatparticipatingfirmswilllikelybedifferentinperformancetootherfirms–andthismaybiastheestimatedeffectsoftheprogramme.Forexample,ifparticipantsweregenerallylargerfirms(observable)orhadmoreambitiousCEOs(unobservable)thendifferencesinperformancecomparedwithnon-participantscouldsimplybeaneffectofthesefactors.

Theauthorsusedadifference-in-differencematchingapproachtoestimatetheimpactofthepolicy.Acontrolgroupwasconstructedbycarefullymatchingprogramparticipantswithotherfirmsonthebasisofvariablessuchassize(employment),productivity(valueaddedperworker),assetsandexportingintensity.Thedifference-in-differenceestimateisthebefore-and-afterchangeinoutcomevariableforthetreatmentgroup(i.e.thosethatreceiveadvice)relativetothiscontrolgroup.Thiskindofmatchingbasedon‘observable’(orrecorded)characteristicscanneverbeperfect.Participantsandnon-participantswillalwaysdifferinsomeunobservedrespects.Becausetheevaluationisn’tabletodoanythingtoaddressthisconcern,wescorethismethoda‘3’ontheScientificMarylandScale.

Toimplementthisapproach,firmleveldatawastakenfromtheFAMEdatabasefrom1994to2003andlinkedwithUKTImanagementdata.Thereweretwootherexportpromotionprogrammesthatweredroppedfromtheanalysissincethelinkingbetweendatasetsforparticipantsoftheseprogrammeswasconsideredunsuccessfulbytheresearchers.Theirfinaltreatmentgroupsinclude299firmswhoreceivedPassportsupportatsomepointoverthistimeframeand509firmswhousedAftercare.

TheevaluationexaminestheeffectofAftercareandPassportonanumberofoutcomevariables,coveringnotonlycoreobjectives(exportingandFDI)butalsowideroutcomessuchasemployment,productivityandprofits.ForthisreviewwearemostinterestedinthePassportscheme.Specifically,participationinPassportisfoundtohavesuccessfullyincreasedthelevelofexportactivityforbothmanufacturingandservicesfirms–thecoreaimoftheprogramme.Passportisalsofoundtoincreasetotalfactorproductivity,comparedwiththecontrolgroup,butthereisnosignificanteffectonemployment.

Whatdotheseresultsmeanforpolicymakers?Onthefaceofit,theseresultsimplythattheUKTIprogrammeshavebeenverysuccessfulinreachingtheirgoalsandpromotingfirmperformanceingeneral.However,sincetheauthorsonlycontrolforobservablecharacteristics(SMS3)itmaybethattheseresultsaredrivenbythefirms’owngoals(whichareunobservable)aligningwiththepolicyobjectives.Forexample,firmsonthePassportprogrammearelikelytohavebeenconsideringincreasingtheirexportactivityinthefutureandmayhaveshownincreasescomparedwiththecontrolgroupeveniftheyhadnotbeenselected.Furtherresearchcontrollingforunobservables(throughrandomisation,instrumentsorquasi-experiments)wouldhelppindownprogrammeimpacts.


Study 170: Matching; SMS level 3InthispaperJonasMånssonandBarbroWiderstedtevaluatetheRegionalBusinessDevelopmentProgramme(RBDP)inSweden.TheRBDPprovidessupportforSMEsoperatinginruralareasofSweden–especiallyinthesparselypopulatednorth.Thepolicyaimstostimulatefirmperformancethroughacombinationofgrantsand‘consultancycheques’,whichfirmsspendonbusinessadviceandmentoring(thefocusoftheanalysis).Theprogrammeisvoluntary:firmsprepareabusinesscaseandapplyforfunding,withtheregionaldevelopmentagencymakingthedecision.In2009,theaverageawardwasabout€7,000(£5,700).

Thereareseveralchallengesinidentifyingtheeffectsofabusinesssupportprogrammelikethisone.First,firms’characteristicsratherthantheinterventionmaydriveperformance,andweneedsomewaytocontrolforthese.Second,becausetheprogrammeisvoluntaryfirmsmay‘select’intoit.Forinstance,ifthefirmswholeastneedsupportdominateapplications,thiswillbiasresultsupwards.Third,agencystaffmaynotmakeobjectivedecisions–reinforcingtheself-selectionproblem.Finally,eveniffirmsdon’treceiveRBDPfundingtheymaystillgetotherformsofbusinesssupport,whichmaycontaminatetheevaluationresults.

Todealwiththeseissues,theauthorsuseamatchingapproach,usingavailabledatatoconstructacontrolgroupthatlookssimilartothetreatmentgroup.Theyalsovarythematchingprocesstocontrolforcontaminationandself-selection–forexample,todealwithself-selectiontheycompareoutcomesfortreatmentfirmswithacontrolgroupofunsuccessfulRBDPapplicants.Theyavoidasimplecomparisonoftreatmentandcontrolgroupsafterthefirmsreceiveadvicebecausedifferencesinperformancemaybedrivenbyunobservablefirmcharacteristics.Instead,theylookatthechangeinfirms’performancebeforeandaftertreatment,andcheckifthedifferencebetweentreatmentandcontrolgroupsismeaningful.(Statisticianscallthisa‘differenceindifference’approach.)

Thisstrategyisstrongbutnotperfect.Matchingbasedon‘observable’characteristicsisneverperfect,asunobservablefactorsmaystillexplaindifferencesinfirmperformance.Theresearchersarealsounabletoseethegrant-makingdecision,sothatsourceofselectionbiasmaystillbepresent.Forthesereasons,wescorethismethoda‘3’ontheScientificMarylandScale.

Theresearchersuserichadministrativedataon1,010RBDPrecipientsand52,613controlfirmsbetween2004and2007,withsmallersamplesforcontaminationandselectiontests.Theyfindthatinthebasicanalysis,consultancycheques‘work’:valueaddedwas14.3%higherfortreatmentfirms,andemployment12.6%higher.Allowingforcontamination,effectsaresmallerbutstillstatisticallysignificant.However,theresultsbecameinsignificantoncetreatmentfirmsarecomparedwithunsuccessfulapplicants(ratherthanallotherfirmsintheregion).

Sodoestheprogrammehavezeroeffect?Theauthorssuggestthatit’sthetimefirmsspendthinkingaboutbusinessdevelopmentthatgeneratesimpacts–andthisispartoftheapplicationprocess,socanbeconsideredpartofthetreatment.Ineffect,theresultssuggestakindofplaceboeffectwheretheRBDPapplicationdoesthejobratherthanthesupportitself.Thatimpliesthatpolicymakersshouldlookforother,lesscostlywaysofdeliveringsupporttofirms–perhapsthroughoutreachratherthanstructuredadvice.

WeshouldbecarefulinusingtheseresultsinaUKcontext.Asnoted,therearesomeunresolvedissuesinidentifyingtrueeffects,andthefindingsmaynottranslatefromruralSwedentoruralBritain.TheevaluationcouldbereplicatedintheUK,forexampletestingcontrolgroupsagainstfirmsreceivingapplicationadviceandasecondgroupreceivingfullsupport.Ideally,thesegroupswouldberandomlyselected.


Study 162: Randomised Control Trial; SMS level 5ThisresearchbyMiriamBruhnandcolleaguesteststheimpactofsubsidisedconsultingservicesonoutcomesforsmallandmediumenterprises(SMEs)usingaRandomisedControlledTrialinPuebla,Mexico.TheprogrammeranfromMarch2008toFebruary2009andwasimplementedbytheStateGovernmentofPuebla.ItwasaimedatincreasingthesizeofSMEs,therebycreatingjobsandeconomicgrowthintheregion.Participatingbusinesseswerematchedwithanappropriateconsultingfirmthatwasaskedtodiagnoseproblemsthatpreventedgrowth,suggestsolutionsandassistwithimplementation.TheStateofPueblathencovered90%ofthecostofservicesformicroenterprises(definedhereaslessthan10employees),80%forsmallenterprises(11-30employees)and70%formediumenterprises(31-100employees).

Fromthe432SMEswhoappliedfortheprogramme(followingstate-wideadvertisements),150wererandomlyselectedintothetreatmentgroup.Theremaining282firmsmadeupthecontrolgroupanddidnotreceiveanysubsidisedservices.Thefactthattreatmentwasrandomisedmakesitmucheasiertoassesstheprogramme’simpact.Specifically,randomisingreceiptallowedtheresearchersto‘control’forbothobservablecharacteristics(e.g.ageoffirm)andunobservablecharacteristics(e.g.theCEO’sleadershipqualities)thatmightaffectoutcomes.Thisalsoavoids‘selectionintotreatment’issuesthatmightariseif(forexample)governmentofficialsdecidedwhogotsubsidisedconsulting,allowingustobeconfidentthatdifferencesinoutcomesforthetreatedgroupreallyarecausedbythefactthattheyreceivedbusinessgrowthadvice.

Inordertoexaminetheeffectsofthetreatment,theresearchersconductedabaselinesurveyofthefirmsinOctober-December2007andafollowupsurveyinMarch-May2009.Theresultsofthebaselinesurveydemonstratedthatthetreatmentandcontrolgroupswerelargelysimilaronobservablecharacteristics,suggestingthatrandomisationwassuccessful.Thebaselinecharacteristicsalsoshowedthatmoreproductive/profitablefirmsinthetreatmentgroup(i.e.thosereceivingadvice)weremorelikelytotakeuptheofferedservices(80ofthe150‘treated’firmsdecidedtotakeuptheservices).Solongasthesefirmsarekeptinthetreatmentgroupandsurveyed,thisdoesnotbiastheresultsbutsimplymeanstheevaluationcapturestheeffectsofbeingofferedratherthangettingsubsidisedconsulting.Therateofresponseforthefollowupsurveywasaround88%,withnodifferencesacrosstreatmentandcontrolgroups,norevidenceofcompositionalshiftsduetofirmsdroppingout(‘attrition’).Sincethetreatmentwassuccessfullyrandomisedandattritionwasnotabigproblem,wescorethisstudyasa5ontheMarylandScale.

Thisfieldexperimentfoundhugereturnstosubsidisedconsulting.Monthlysalesincreasedbyaround80%andprofitsincreasedby120%inthetreatmentgroupcomparedwiththecontrolgroup.Theaverageincreaseinprofitsisestimatedtoliebetween$7,600and$11,000permonth,comparedwithacostof$988permonthfortheconsultingservices.Inourmainreviewwefindthatbusinessadviceprogrammeshaveamixedrecordonimprovingemployment,andinthiscasetheresearchersdonotfindanyincreaseinthenumberofworkersemployedbytreatedfirms–althoughperhapsemploymentmaytakelongerthantwoyearstoadjust.Thecost-effectivenessoftheprogrammeposesthequestionofwhyfirmsdidn’tinvestintheseservicesthemselvesinthefirstplace.Theresearcherssuggestthemostlikelysourceofmarketfailureinthiscaseisfundingconstraints.Anotherpossibleexplanationisthattheysimplyneverconsideredtakingupconsultancyservicesordidnotknowsuchserviceswereavailable.

TowhatextentaretheresultsfromthistrialrelevanttoUKpolicymakers?Theresultshighlightthepotentialforworthwhileinterventionintheareaofbusinesssupport.However,suchapolicywouldbesuccessfulintheUKonlyifitwereaddressingasimilarmarketfailuresuchasalackoffunding/informationforSMEs.SincetheMexicaneconomyissignificantlydifferenttotheUKeconomyitmaybethatthepolicywouldnotbesosuccessful.TheonlywaytoknowforsureifitwouldbeeffectiveintheUKistoreplicatetheprogrammehereasarandomisedtrial.


Study 287: Randomised Control Trial; SMS level 5ThisresearchbyJohnBenusandcolleagues(1994)evaluatestwoUSprojectsdesignedtosupportunemployedpeopletostarttheirownbusinesses:theWashingtonStateSelf-EmploymentandEnterpriseDevelopmentProject(SEED),andtheMassachusettsUISelf-EmploymentDemonstration.Bothprogrammesranfrom1989to1991,andtestedwhetherpublicassistancecouldhelpparticipantsmoveintoemploymentorself-employment,raiseindividualincomesandincreasebusinessstart-uprates.Bothprogrammesprovidedamixofexperttrainingsessions,one-to-onesessionstodevelopbusinessideasandworkshopscoveringbusinessplanning,marketing,accountingandmanagement.Inaddition,theSEEDprogrammegaveparticipantscompletingthecoursealumpsumtohelpstarttheirnewfirm.

BothprogrammesweredesignedasRandomisedControlTrials(RCTs),withparticipantsrecruitedfrompeoplereceivingunemploymentinsurance.Arandomnumbergeneratorwasusedtoassignindividualstotreatmentorcontrolgroups,acrossanumberofsitesineachstate.(InthecaseofSEED,755peoplewereassignedtotreatment,withafurther752inthecontrolgroup;numberswasslightlylowerfortheMassachusettsprogramme.)

Randomisationmakesitmucheasiertoassesstheseprogrammes’impact.Specifically,itallowsresearcherstocontrolforparticipants’observablecharacteristics(suchasgenderorqualifications)andunobservablecharacteristics(suchasmotivation)thatmightaffectoutcomes.Italsoavoidsselectionissuesthatmightariseif(say)programmemanagersassignedonlythestrongestparticipantstotreatment.Thisallowsustobeconfidentthatahigherrateofself-employmentinthetreatedgroupisreallycausedbyprogrammeparticipation.OnepotentialproblemwithRCTsispeopledroppingoutofthestudy–theresearchersdealwiththisissueusingadministrativedata,whichallowstrackingevenifparticipantsdon’trespondtofollow-upsurveys.Wethereforescorethisstudy‘5’ontheScientificMarylandScale.

Theresearchfindsthatbothprogrammesincreasedthelikelihoodofentryintoself-employment21monthslater,andacceleratedmovesfromunemploymentintoself-employment(5.9monthsearlierforSEED,2.4monthsfortheMassachusettsprogramme).Bothprogrammesalsosignificantlyincreasedthechancesofbeingemployed/self-employed,whichwas14%higherforSEEDparticipantsand5%higherforthoseintheMassachusettsinitiative.However,neitherprogrammesignificantlyincreasedparticipants’totalearnings,inpartbecausepeoplewerelesslikelytomoveintoaregularjob.Thedifferencesbetweenthetwoprogrammesseempartlyrelatedtowidereconomicconditionsinthetwostates;mostoftheUSwasinrecessionduringthedeliveryperiod,butWashington’seconomywaslargelyinsulatedfromthesemacrotrends.

Whatdoesthismeanforpolicymakers?Thestudiesprovideveryhighqualityevidencethatbusinesssupportinterventionsareeffectiveinraisingemployment,butlesseffectiveiftheaimistoraisewagesandincomes.Whatislessclearisexactlywhichelementsofthetreatmentmixaremosteffective,andhowitcouldbefurtherfine-tuned.Thisisimportanttoknow,giventhestrongerperformanceofSEEDmayberelatedtoprogrammedesigndifferencesaswellasexternaleconomicconditions.Furthertrialswhichrandomisedtreatmentstrandswouldhelpanswerthesequestions.

The What Works Centre for Local Economic Growth is a collaboration between the London School of Economics and Political Science (LSE), Centre for Cities and Arup.

www.whatworksgrowth.org


This work is published by the What Works Centre for Local Economic Growth, which is funded by a grant from the Economic and Social Research Council, the Department for Business, Innovation and Skills and the Department of Communities and Local Government. The support of the Funders is acknowledged. The views expressed are those of the Centre and do not represent the views of the Funders.

May2014

WhatWorksCentreforLocalEconomicGrowth

[email protected]@whatworksgrowth


©WhatWorksCentreforLocalEconomicGrowth2014

mailto:info%40whatworksgrowth.org%0D?subject=

https://twitter.com/whatworksgrowth

http://whatworksgrowth.org/

evidence review business advice - whatworksgrowth.org

Documents