Reducing Biases in Economics and Business Research

T.D. Stanley (Tom)Honorary Professor, Alfred Deakin Research Institute

Julia Mobley Professor of Economics,

Hendrix College, USA Stanley@Hendrix.edu

“When reviewing an empirical literature, manyeconomists seem to become Bayesian, holdingstrong priors formed on the basis of theoreticalconsiderations and raising high barriers to anycontrary empirical evidence” –Stanley (2001)

The market is efficient and gov’t intervention only makes things worse

Global Financial Crisis: 2007-2008

Meta-Analysis of Economics Research-Network (MAER-Net)

•Prague, CZ—2015 (Sept 10-12)• Athens, Greece (2014)• Greenwich, UK (2013)• Perth, Australia (2012)• Cambridge University, UK (2011)• Hendrix College, US (2010)• Corvallis, Oregon US (09)• Nancy, France (08)• Sonderborg, Denmark (07)

to join: Stanley@Hendrix.edu

MAER-Net Colloquium at Prague-Sept 10-12

• http://ies.fsv.cuni.cz/en/node/516/index.htm

Conflicting Empirical Findings• Rarely do single studies provide definitive

answers upon which to base policy or to settle the theoretical disputes.

• Very large research variation is the norm.

The Problems of Economics Research

Econometric Misspecification BiasPublication Selection BiasLow Power- {Chris D., John Ioannidis & I for EJ}Researcher Commitment to Ruling Theory {Doucouliagos & Stanley (2013): the stronger the consensus about theory, the larger the bias.}Ideological Bias

Is Meta-Regression Analysis the Answer?

MRA:• Uses replicable methods to identify and

code all relevant research. • Employs objective statistical methods to

summarize and explain all research results. • Accommodates and corrects selection and

misspecification biases routinely found among reported economic findings.

• Provides a rigorous basis for evidence-based practice.

Our Book“In our view, the central task of meta-regression analysis is to filter out systematic biases, largely due to misspecification and selection, already contained in economics research.”

–Stanley & Doucouliagos (2012, p.16, Meta-Regression Analysis in Economics and Business, Routledge.

“Believing is seeing”–Demsetz (1974, p. 164).

• In Psychology, this is called: confirmation (or experimenter) bias

• For example, when asked “Are you happy with your social life?”—Y(73%); N(22%); Undec(5%)

• When asked: “Are you unhappy with your social life?” —Y(65%); N(27%); Undecided (8%)

• In Economics, Ideology and Conventional Theory frames and shapes how evidence is interpreted.

Believing is Seeing IIPublication Selection Bias• Reviewers and editors are predisposed to

accept papers consistent with the conventional view.

• Researchers may treat the conventionally expected result as a model selection test.

• Everyone may have a preference for ‘statistically significant’ findings.

—Card and Krueger (1995)

Picturing Research and Its Bias: Funnel Graphs

• A funnel graph is a scatter diagram of precision (1/SE) vs. estimated effect.

• In the absence of publication selection, estimates will vary randomly and symmetrically around true effect (β).

• The expected inverted funnel shape is dictated by predictable heteroskedasticity.

0

10

20

30

40

50

60

70

80

90

100

1/S

e

-.8 -.6 -.4 -.2 0 .2 .4 .6r

“Picture this: A simple graph that reveals much ado about research.” (Stanley and Doucouliagos, 2010, Journal of Economic Surveys)

Figure 1: Funnel Plot of Union-ProductivityPartial Correlations

Data: 73 partial correlation coeff’s of union membership and worker productivity

source: Doucouliagos, C. and Laroche, P. (2003)

Funnel Graph is a scatter diagram of all estimates and their precisions (precision= 1/SE)

Figure 2: Value of Statistical Life(in millions of 2000 US $s)

Data: 39 estimates of VSL from hedonic wage equations.

Source: Doucouliagos, Stanley, and Giles (2012), JHE.

Figure 3: Antidepressant Trials

Source: Turner et al. (2008).

Data: 74 effect sizes, Hedges g, from the US-FDA registry, considered the goldstandard. 50 of these were published.

2

3

4

5

6

7

8

9

10

11

Pre

cisi

on

-.4 -.2 0 .2 .4 .6 .8 1

g

FDAPublished

Conventional Neoclassical theory:

• predicts a strict LAW of DEMAND for labor {i,e., an inverse relation between wages and employment}.• Therefore, raising the minimum wage must

reduce employment.• All real economists know that government

intervention into the free market always has negative consequences. . . .

• Recall the controversy in the 1990s around Card and Krueger’s ‘Myth and Measurement.”

Figure 4: A graph seen around the world:Minimum Wage Employment Elasticities

Data: 1,474 US minimum-wage elasticities of employment. (Doucouliagos and Stanley, 2009)

Source: Economic Report of the President (2013)

-50

0

50

100

150

200

250

300

Prec

ision

(1/S

E

)

-20 -15 -10 -5 0 5

Employment Elasticity

Elasticity = -1

Testing for Publication Selection bias and Identifying Genuine Empirical Effect

FAT-PET-MRAej= β + β0Sej + εj (j=1, 2, …L) (1)

{Comment: Always use WLS; 1/Se2 as the weight}

FAT (funnel-asymmetry test): t-test of β0 is a test for publication bias (Egger et al., 1997)PET (precision-effect test): t-test of β is a test of authentic effect, beyond publication bias (Stanley, 2005; 2008; S&D, 2014)

FAT-PET-MRA-WLS– Eq. (1)Table 1: MRA Tests of PB & Genuine Effect

*Robust t-values in parentheses

Variable Union-Productivity

VSL Anti-Depressants

Minimum Wage

Se ( β0 ) .65 (1.72)* 3.20(6.67)* 1.84 (5.47)* -1.6(-4.49)*

Intercept β -.0179(-1.06)* 0.81(3.56)* .13 (2.50)* -.01(-1.09)*

Ave. Effect .03 $9.5 mil .47 -.19Top10 -.0003 $2.00 mil .32 -0.022

PEESE . 0009 $1.67mil .29 -.036n 73 39 50 1,474

Putting it all together with Multiple MRA

effecti=(β +ΣαkZik ) + (β0Sei +ΣγjKijSei )+εi (3)Heterogeneity & Publication BiasMisspecification

Biases

Table 2: Minimum-Wage Moderator VariablesK & Z

VariableDefinition Mean (standard

deviation)

Elasticity is the estimated effect, the dependent variable, and an elasticity -1.69 (2.83)

Se is the elasticity’s standard error 22.76 (28.32)

Panel =1, if estimate relates to panel data with time-series as the base 0.45 (0.50)

Cross =1, if estimate relates to cross-sectional data with time-series as the base 0.13 (0.34)

Adults =1, if estimate relates to young adults (20-24) rather than teenagers (16-19) 0.14 (0.35)

Male =1, if estimate relates to male employees 0.07 (0.26)

Non-white =1, if estimate relates to non-white employees 0.05 (0.22)

Region =1, if estimate relates to region specific data 0.10 (0.30)

Lag =1, if estimate relates to a lagged minimum wage effect 0.13 (0.34)

Hours =1, if the dependent variable is hours worked 0.07 (0.25)

Double =1, if estimate comes from a double log specification 0.42 (0.49)

AveYear is the average year of the data used, with 2000 as the base year -19.17 (11.90)

Agriculture =1, if estimates are for the agriculture industry 0.01 (0.11)

Retail =1, if estimates are for the retail industry 0.08 (0.27)

Food =1, if estimates are for the food industry 0.13 (0.34)

Time =1, if time trend is included 0.37 (0.48)

Yeareffect =1, if year specific fixed effects are used 0.30 (0.46)

Regioneffect =1, if region/State fixed effects are used 0.34 (0.47)

Un =1, if a model includes unemployment 0.56 (0.50)

School =1, if model includes a schooling variable 0.15 (0.35)

Kaitz =1, if the Kaitz measure of the minimum wage is used 0.40 (0.49)

Dummy =1, if a dummy variable measure of the minimum wage is used 0.17 (0.38)

Published =1, if the estimate comes from a published study 0.85 (0.35)

22 Z-var’s + 22 K-var’s = 44 Var’s

Table 3: General-to-Specific MRA Minimum-Wage

Elasticity

Variables: Cluster-Robust FE-Panel

Genuine empirical effects (Z-variables)Intercept ( β ) 0.120 (4.39) 0.102 (6.04)Panel -0.182 (-4.72) -0.149 (-10.5)Double 0.064 (3.20) 0.041 (5.42)Region 0.040 (0.92) 0.090 (6.34)Adult 0.024 (2.68) 0.021 (3.72)Lag 0.026 (1.60) 0.010 (1.59)AveYear 0.004 (4.34) 0.003 (6.38)Un -0.042 (-3.04) -0.042 (-5.79)Kaitz 0.052 (3.06) 0.032 (3.88)Yeareffect 0.069 (1.98) 0.067 (7.44)Published -0.041 (-2.69) -0.037 (-4.89)Time -0.022 (-2.08) -0.017 (-2.46)

Publication bias (K-variables)Se ( β0 ) -0.359 (-0.11) -1.374 (-5.94)Double x Se -1.482 (-3.23) -1.073 (-3.90)Un x Se -0.840 (-1.87) 1.164 (3.08)

Heterogeneity

Publication Selection

Doucouliagos and Stanley (2009), British Journal of

Industrial Relations.

‘Prediction’ and Implications

• Once publication selection is accommodated, No evidence of anadverse employment effect remains.

• Substituting any defensible notion of ‘Best Practice Research’ into these estimated MRA coefficients (Table 3) finds: Nosupport for a practically significantadverse employment effect!

Robustness Checks

• Many sets of robustness checks were investigated. • robust regressions• unbalanced panel MRA and cluster-robust

standard errors to control for potential within study dependence

• the ‘best-set’ (i.e., one estimate per study)• Card and Krueger’s (1995) data• omitting all positive elasticities

{All confirm an absence of an adverse employment effect.}

Replicated by 2 Independent Teams &

A Meta-Analysis of UK’s Minimum Wage:• Followed MAER-NET (2013) guidelines

• Comprehensive search, 2 coders, etc.• Confirmed our US findings

• No evidence of adverse employment• But no evidence of publication bias in UK{Leonard et al. (2014), “Does the UK minimum wage reduce employment? BJIR}

Policy Implications

• 2013 US Economic Report of the President quoted us and reproduced our funnel graph.

• President Obama proposed an increase in the US minimum wage—from $7.25 to $9/hour.

• Many US cities and states (including Arkansas) have already raised their minimum wage.

• Minimum wage will be a campaign issue in the next US election.

• UK’s Low Pay Commission has funded an update of our meta-analysis.

Doucouliagos, Haman, and Stanley (2012): Industrial Relations, 51: 670-703.

Pay for Performance and Corporate Governance Reform

Too often, executive compensation in the U.S. is ridiculously out of line with performance. That won't change, moreover, because the deck is stacked against investors when it comes to the CEO's pay. . . Getting fired can produce a particularly bountiful payday for a CEO. Indeed, he can “earn” more in that single day, while cleaning out his desk, than an American worker earns in a lifetime of cleaning toilets. Forget the old maxim about nothing succeeding like success: Today, in the executive suite, the all-too-prevalent rule is that nothing succeeds like failure.

− Warren E. Buffett, Berkshire Hathaway Report, February 28, 2006.

US corporate scandals, e.g., Bernie Ebbers (WorldCom);

Ken Lay and Jeff Skillings (ENRON), . . . . . . . The 2007 global financial meltdown. . . Large bonuses at Wall-Street banks { often the

same ones that caused the 2007 financial crisis and had to be bailed out}

Predictably, media reports of egregious corporate excesses lead to public calls for greater corporate regulation and reform.

Those Buzzards. . . . .

‘Comply or Explain’ “The essence of the principle is that compliance with the codes is not mandatory, but that disclosure relating to compliance is.”

– (MacNeil and Li; 2006, p.486)

Every few years, the UK has adopted new corporate governance reforms, attempting to strengthen the link between CEO pay and corporate performance and thereby to protect shareholders. Cadbury Report (1992); the Greenbury Report (1995); the Combined

Code of Corporate Governance (1998); the revised Combined Code of Corporate Governance (2003); Companies Act 2006; . . .

CEO remuneration committees are now ‘required’ to include independent non-executive directors

shareholders get to vote on directors’ pay at annual meetings

FAT-PET-MRA-WLS– Eq. (5)Table 4: MRA Tests of PB & Genuine Effect

MRA equation (1): Dependent Variable = r (partial correlations) CEO Pay-

Performance Intercept (β) 0.077(8.09)*

SEi (β0) 0.097(0.33)*

Simple Mean 0.093

WLS weighted average

0.079

Top10 0. 078

PEESE 0.077

n 511

*t-values are reported in parenthesis are from cluster-robust standard errors.

Table 5: Selected MRA Results There is evidence that:

‘Comply and Explain’ has been effective.

The pay-performance link is declining over time.

Higher economic freedom leads to stronger pay-performance linkage.

{RE: greater reliance on markets should lead to stronger pay-performance links}

The ways in which both pay and performance are measured matter.

Bad News: ‘Size’ matters; performance: not so much.

Variable Base model General to specific

Constant 25.337 (2.87) 8.177 (2.85) AverageYear -0.013(-2.84) -0.004 (-2.79) Cadbury 0.058 (3.69) 0.037 (2.43) CombinedCode 0.079 (2.16) - EcoFree 0.102 (1.98) - Cash - -0.082 (-2.46) Changepay - -0.053 (-3.12) Returns - 0.058 (3.41) ROE - 0.205 (2.52) Sales - 0.143 (22.98) Lagperform - -0.032 (-2.62) Panel - -0.048 (-2.11) Education - -0.105 (-1.72) Lagpay - 0.065 (3.78) Time effects - -0.025 (-1.75) Composition - -0.055 (-2.19) Adjusted R2 0.38 0.82

A Medical Application:Nicotine Replacement Therapy (NRT)

Stead et al. (2012) The Cochrane Library:

• 122 placebo-controlled clinical trials (RCTs)• large effects from NRT {quitting by 50-70%}

• coded for ‘risks of bias’ for each RCT• did not control for publication bias.

Our MRAs that control for both Stead et al.’s (2012) risks of bias and publication bias find :

No evidence a NRT Effect

Funnel Graph: High quality NRT trials

0

1

2

3

4

5

6

7

8

9Pr

ecis

ion

(1/s

)

-.25 0 .25 .5 .75 1 1.25 1.5 1.75 2 2.25lnRR

Table 6: MRA of NRT trialsVariable/Estimate 1: Q2 2: Q3 3: Q4 4: Full Sample

Intercept: β̂ -0.02 (-0.23)

0.04 (0.30)

-0.05 (-0.25) -0.09 (-0.64)

Sei : 0β̂ 1.68* (4.96)

1.52* (2.76)

1.84* (2.20) 2.04* (3.89)

Riski: 1α̂ — — — 0.11* (2.93)

si ·Riski : 2α̂ — — — -0.39* (-2.64)

Sample size n 46 23 12 121 Adj R2 25.2% 23.1% 25.8% 12.6% Homogeneity: Q 36.8 19.1 11.2 145.1*

Note: Q2, Q3, and Q4 represent decreasing risk of bias, defined by Stead et al. (2012)

Funnel Symmetry Exceptions{aside from pub’bias of course}

Very large, partial correlation coefficients. Estimating AR(1) coefficients, Time Series If unexplained systematic heterogeneity is skewed and

correlated with SE. Non-Market environmental values when calculated from

an estimated price coefficient. {and any other nonlinear transformation of an regression coefficient}

Due to power analysis? Large effects might be associated with smaller samples; hence smaller precision?

Publication Selection Paradox• Many economists have difficulty seeing that the

suppression of positive price elasticities (or negative VSLs) is somehow ‘biased.’

• It may be individually rational for economists to ‘validate’ their estimate– i.e., to suppress positive price elasticities or negative VSL.

• However, when most researchers suppress these ‘wrong’ signs, our aggregate knowledge worsens.

• Fallacy of Composition {ecological fallacy}

ReferencesCard, D. and Krueger, A.B. 1995. Time-series minimum-wage studies: A meta-analysis, American Economic Review 85: 238-243.Demsetz, H. (1974) ‘Two systems of belief about monopoly’, in H.J. Goldschmid, H.M. Mann and Weston, J. F. (eds.) Industrial

Concentration: The New Learning, Boston: Little, Brown and Company, pp.164–84.Doucouliagos, C.(H) and Stanley, T.D. 2009. Publication selection bias in minimum-wage research? A meta-regression analysis,”

British Journal of Industrial Relations 47: 406-29. Doucouliagos, C.(H), Stanley, T.D. and Giles, M. 2012. ‘Are Estimates of the Value of a Statistical Life Exaggerated?’ Journal of

Health Economics, 31: 197-206..Egger, M., Smith, G.D., Scheider, M., and Minder, C. 1997. Bias in meta-analysis detected by a simple, graphical test, British

Medical Journal 316: 629-34.Glass, G.V. 1976. Primary, secondary, and meta-analysis of research. Educational Researcher 5: 3-8.Stanley, T.D. 2001. Wheat from chaff: Meta-analysis as quantitative literature review. Journal of Economic Perspectives 15: 131-

50.Stanley, T.D. 2004. ‘Does unemployment hysteresis falsify the natural rate hypothesis? A meta-regression analysis’, Journal of

Economic Surveys, 18: 589-612. Stanley, T.D., 2005a. Beyond publication bias, Journal of Economic Surveys 19: 309-45.Stanley, T.D. 2005b. ‘Integrating the empirical tests of the natural rate hypothesis: A meta-regression analysis’, Kyklos, 58: 611-

34. Stanley, T.D. 2008. Meta-regression methods for detecting and estimating empirical effect in the presence of publication bias.

Oxford Bulletin of Economics and Statistics 70:103-127.Stanley, T.D., Jarrell, S.B., 1989. Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic

Surveys 3: 54-67.Stanley, T.D., and Doucouliagos, H(C) 2010. Picture this: A simple graph that reveals much ado about research. Journal of

Economic Surveys 24: 170-191.