the balance‐sample size frontier in matching … balance-sample size frontier in matching ......

The Balance-Sample Size Frontier in MatchingMethods for Causal Inference

Gary King Harvard UniversityChristopher Lucas Harvard UniversityRichard A. Nielsen Massachusetts Institute of Technology

Abstract: We propose a simplified approach to matching for causal inference that simultaneously optimizes balance(similarity between the treated and control groups) and matched sample size. Existing approaches either fix the matchedsample size and maximize balance or fix balance and maximize sample size, leaving analysts to settle for suboptimalsolutions or attempt manual optimization by iteratively tweaking their matching method and rechecking balance. To jointlymaximize balance and sample size, we introduce the matching frontier, the set of matching solutions with maximumpossible balance for each sample size. Rather than iterating, researchers can choose matching solutions from the frontierfor analysis in one step. We derive fast algorithms that calculate the matching frontier for several commonly used balancemetrics. We demonstrate this approach with analyses of the effect of sex on judging and job training programs that showhow the methods we introduce can extract new knowledge from existing data sets.

Replication Materials: The data, code, and additional materials required to replicate all analyses in this articleare available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at:doi:10.7910/DVN/SURSEO. See also King, Lucas, and Nielsen (2016).

Matching is a statistically powerful and con-ceptually simple method of improving causalinferences in observational data analysis. It

is especially easy to use in applications when thoughtof as a nonparametric preprocessing step that identifiesdata subsets from which causal inferences can be drawnwith greatly reduced levels of model dependence (Hoet al. 2007). The popularity of this approach is increasingquickly across disciplinary areas and subfields. Indeed,the proportion of articles using matching in the Amer-ican Journal of Political Science has increased by about500% from 2005 to 2015, with matching used in about aquarter of all articles.

Although successful applications of matching requireboth reduced imbalance (increased similarity betweenthe treated and control groups) and a sufficiently largematched sample, existing matching methods optimizewith respect to only one of these two factors. Typically,

Gary King is Albert J. Weatherhead III University Professor, Institute for Quantitative Social Science, 1737 Cambridge Street, HarvardUniversity, Cambridge, MA 02138 ([email protected]). Christopher Lucas is Ph.D. Candidate, Institute for Quantitative Social Science,1737 Cambridge Street, Harvard University, Cambridge, MA 02138 ([email protected]). Richard A. Nielsen is Assistant Professor,Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139 ([email protected]).

Our thanks to Carter Coberley, Stefano Iacus, Walter Mebane, Giuseppe Porro, Molly Roberts, Rocio Titiunik, Aaron Wells, and Yu Xie.Jeff Lewis’s comments on an early version were especially helpful. Software to implement the methods proposed in this article can be foundat http://projects.iq.harvard.edu/frontier.

the required joint optimization is performed by manuallytweaking existing methods or is ignored altogether. How-ever, jointly optimizing balance and sample size is crucialsince, if the subset identified by the matching method istoo small, the reduction in model dependence (and hencebias) achieved will be counterbalanced by an unaccept-ably high variance. Similarly, the small variance associatedwith a large matched data subset may be counterbalancedby unacceptably high levels of imbalance (and thus modeldependence and bias). Some of these problems may alsobe exacerbated by current matching methods, which op-timize with respect to one balance metric but encourageresearchers to check the level of balance achieved withrespect to a completely different metric for which themethod was not designed and does not optimize.

To remedy these problems, we introduce a procedurethat enables researchers to define, estimate, visualize, andthen choose from what we call the matching frontier, a set

American Journal of Political Science, Vol. 00, No. 0, xxxx 2016, Pp. 1–17

C©2016, Midwest Political Science Association DOI: 10.1111/ajps.12272

1

2 GARY KING, CHRISTOPHER LUCAS, AND RICHARD A. NIELSEN

of matched samples that fully characterize the trade-offbetween imbalance and the matched sample size. Unlikeother approaches, we allow researchers to evaluate howmuch balance is achieved by pruning observations and si-multaneously trade balance off against the lower varianceproduced by larger matched sample sizes. At each location(denoted by the matched sample size) along the match-ing frontier, our approach offers a matched subset of thecomplete data such that no other possible subset of thesame size has lower imbalance. Any matching solution noton this frontier is suboptimal, meaning that a lower levelof imbalance can be achieved using an alternative datasubset of the same size. As a result, no matching methodcan outperform the matching frontier, provided that bothuse the same imbalance metric. Thus, constructing andanalyzing a matching frontier achieves all of the benefitsof any individual matching method, allows researchers toextract maximal causal information from their observa-tional data, is considerably easier to apply appropriatelythan manual optimization of balance and sample size,and avoids many of the pitfalls and difficulties that leadresearchers to ignore best practices in applications.

Unfortunately, finding the matching frontier for anarbitrary imbalance metric by examining each possiblematched sample is an exponentially difficult computa-tional task that is infeasible for all but the smallest datasets, even if it were possible to simultaneously use everycomputer ever built and run them for as long as the uni-verse has existed. We show how to calculate the frontierfor a set of commonly used imbalance metrics, for whichwe are able to derive a set of algorithms that run in a fewminutes on one personal computer. We prove that thesealgorithms compute the (optimal) frontier exactly, thatis, without any approximation.

We begin by introducing the trade-off betweenpruning observations to reduce model dependence andretaining observations to reduce variance that exists inall matching methods for causal inference. We then detailour mathematical notation, goals, and assumptions;the choices required for defining a matching frontier;and a formal definition of and algorithms for calcu-lating the frontier. Finally, we offer several empiricalexamples and conclude. Software to implement all thealgorithms proposed in this article can be found athttp://projects.iq.harvard.edu/frontier (King, Lucas, andNielsen 2016).

The Matching Frontier Trade-Off

Matching methods selectively prune observations from adata set to reduce imbalance. A reduction in imbalance

reduces, or reduces the bound on, the degree of modeldependence, a result that has been shown both formally(Iacus, King, and Porro 2011a; Imai, King, and Stuart2008; King and Zeng 2006) and in real data (Ho et al.2007). However, matching has a potential cost in thatobservations pruned from the data may increase the vari-ance of the causal effect estimate. Although researchersusing matching confront the same bias-variance trade-off as in most of statistics, two issues prevent one fromoptimizing on this scale directly. First, since matching iscommonly treated as a preprocessing step, rather than astatistical estimator, particular points on the bias-variancefrontier cannot be computed without also simultaneouslyevaluating the estimation procedure applied to the result-ing matched data set. Second, best practice in matchinginvolves avoiding selection bias by ignoring the outcomevariable while matching (Rubin 2008), which requires giv-ing up the ability to control either bias or variance directly.

Thus, instead of bias, matching researchers focus onreducing the closely related quantity, imbalance. The spe-cific mathematical relationship between the two is givenby Imai, King, and Stuart (2008), but conceptually, imbal-ance combines with the relative importance of individualcovariates to determine bias. Researchers exclude rela-tive importance because it cannot be estimated withoutthe outcome variable (although scaling the covariates byprior expectations of importance is a common and valu-able step). Similarly, instead of variance, researchers focuson the matched sample size. The variance is determinedby the matched sample size along with the heterogeneity(i.e., residual unit-level variance) in the data. Researchersexclude heterogeneity because it can only be estimated byusing the outcome variable.

Consequently, the goal of matching involves the jointoptimization of balance and matched sample size. Op-timizing with respect to one, but not both, would be amistake. Existing methods address the joint optimiza-tion by combining machine optimization of one of thesefactors with manual (human) optimization of the other.These attempts to optimize by hand are time-consumingand usually result in suboptimal matched samples be-cause human data analysts are incapable of evaluating allpossible matching solutions.

Many good suggestions for ad hoc approaches tomanual optimization of matching methods have ap-peared in the methodological literature (e.g., Austin 2008;Caliendo and Kopeinig 2008; Rosenbaum, Ross, andSilber 2007; Stuart 2008). For example, Rosenbaum andRubin (1984) detail their gradual refinement of an initialmodel by including and excluding covariates until theyobtain a final model with 45 covariates, including seveninteraction degrees of freedom and one quadratic term.

THE BALANCE-SAMPLE SIZE FRONTIER 3

Ho et al. (2007, 216) recommend trying as many match-ing solutions as possible and choosing the one with thebest balance. Imbens and Rubin (2009) propose runningpropensity score matching, checking imbalance, adjust-ing the specification, and iterating until convergence, aswell as manual adjustments. Applying most of these meth-ods can be inconvenient, difficult to use optimally, andhard to replicate. With nearest neighbor and propensityscore matching in particular, tweaking the procedure toimprove imbalance with respect to one variable will oftenmake it worse on others, and so the iterative process canbe frustrating to apply in practice.

Because manual tweaking is time-consuming andfrustrating, applied researchers using matching rarely fol-low suggested best practices, such as the procedures justlisted. In what follows, we replace the existing machine-human procedures for optimizing both balance andsample size with a machine-machine optimization pro-cedure, thus guaranteeing optimal results in considerablyless time.

Causal Inference Objectives

We define here our notation and choices for the causalquantity of interest. We separate discussion of the neces-sary assumptions into those that are logically part of thenotation and assumptions that become necessary whentrying to learn about the quantities of interest from thedata.

Notation and Basic Assumptions

For unit i , let Ti denote a treatment variable coded 1 forunits in the treated group and 0 in the control group. LetYi (t) (for t = 0, 1) be the (potential) value the outcomevariable would take if Ti = t. Denote the treatment effectof T on Y for unit i as TEi = Yi (1) − Yi (0). However,for each i , either Yi (1) or Yi (0) is observed, but neverboth (which is known as the fundamental problem ofcausal inference; Holland 1986). This means we observeYi = Ti Yi (1) + (1 − Ti )Yi (0). Finally, define a vector ofk pretreatment control variables Xi .

We simplify this general framework by restrictingourselves to consideration of TEs only for treated units.1

In this situation, the only unobservables are Yi (0) forunits that received treatment Ti = 1 (since Yi (1) ≡ Yi isobserved for these units).

1Since the definition of which group is labeled “treated” is arbitrary,this does not restrict us in practice.

A coherent interpretation of this notation implies twoassumptions (Imbens 2004). The first is overlap (some-times called “common support”): Pr(Ti = 1|X) < 1 forall i (see also Heckman, Ichimura, and Todd 1998, 263).The idea here is that for treated units, where Ti = 1, itmust be conceivable that before treatment assignment,an intervention could have taken place that would haveassigned unit i instead to the control group, while hold-ing constant the values of X . If this could not have beenpossible, then Yi (0), which we need to define TEi , doesnot even logically exist.

A second assumption implied by the notation is stableunit treatment value (SUTVA), which can be thought ofas logical consistency, so that each potential value is fixedeven if T changes (or conceptualized differently, this as-sumption requires no interference between units, and nohidden versions of treatment; VanderWeele and Hernan2012). If overlap holds but SUTVA does not, then Yi (0)and TEi exist but are not fixed quantities to be estimated.

Quantities of Interest

From the basic definitions given above, we can computemany quantities, based on the average of TE over givensubsets of units. We focus on two in this article. We firstdefine these theoretically and then explain how they workin practice, followed by the assumptions necessary foridentification.

First is the sample average treatment effect on thetreated, SATT = meani∈{T=1}(TEi ), which is TE averagedover the set of all treated units {T = 1} (Imbens 2004).2 Ifmatching only prunes data from the control group, SATTis fixed throughout the analysis.

Second, since many real observational data sets con-tain some treated units without good matches, analystsoften choose to compute a causal effect among only thosetreated observations for which good matches exist. Wedesignate this as the feasible sample average treatment ef-fect on the treated (FSATT).3 Other possibilities include

2Formally, for set S with cardinality #S, define the average over iof function g (i) as meani∈S [g (i)] = 1

#S

∑#Si=1 g (i).

3Using FSATT is common in the matching literature but may beseen as unusual elsewhere because the quantity of interest is de-fined by the statistical procedure. In fact, this approach follows theusual practice in observational data analysis of collecting data andmaking inferences only where it is possible to learn something. Theadvantage here is that the methodology makes a contribution to astep previously considered outside the statistical framework (e.g.,Crump et al. 2009; Iacus, King, and Porro 2011a), just as mea-surement error, missing data, selection bias, and other issues oncewere. As Rubin (2010, 1993) puts it, “In many cases, this search forbalance will reveal that there are members of each treatment arm


TE averaged over all observations, as well as populationaverage treatment effects.

Although the distinction between SATT and FSATTis clear given a data set, the distinction can blur in practicebecause in observational data analysis there often exists no“correct” or canonical definition of the target population.Usually, if observational data analysts have access to morerelevant data, they use it; if a portion of the data is nothelpful or too difficult to use, because of measurementproblems or the extreme nature of the counterfactualinferences required, they drop it. If we have a new, morepowerful telescope, we observe more—for that reason.Thus, since observational data analysis is, in practice, anopportunist endeavor, we must recognize first that eventhe choice of SATT as a quantity of interest always involvessome feasibility restriction (quite like FSATT), eitherexplicitly where we choose to make a SATT inferencein a chosen subset of our data, or implicitly due to thechoice of our data set to begin with. Thus, regardless ofthe definition of the units over which an average will takeplace, researchers must always be careful to characterizethe resulting new estimand, for which we offer some toolsbelow.

Suppose we insist on choosing to estimate SATT eventhough some counterfactuals are so far from our data thatthey have no reasonable matches and require model-basedextrapolation. If SATT really is the quantity of interest,this situation cannot be avoided, except when it is possibleto collect more data. To understand this problem, wefollow Iacus, King, and Porro (2011a) and partition theN treated units into N f treated units that can be wellmatched with controls (a “feasible estimation set”) andNn f remaining treated units that cannot be well matched(a “nonfeasible estimation set”), such that N = N f +Nn f . In this case, we express SATT as a weighted averageof an estimator applied to each subset separately:

SATT = FSATT · N f + NFSATT · Nn f

N. (1)

When estimating SATT in this unfortunate, but com-mon, situation, it is often worthwhile to compute its twosubcomponents separately since only FSATT will be esti-matable without (much) model dependence. We refer tothe subsets of observations corresponding to FSATT and

who are so unlike any member of the other treatment arm that theycannot serve as points of comparison for the two treatments. Thisis often the rule rather than the exception, and then such units mustbe discarded . . . . Discarding such units is the correct choice: A gen-eral answer whose estimated precision is high, but whose validityrests on unwarranted and unstated assumptions, is worse than aless precise but plausible answer to a more restricted question.”

NFSATT, respectively, as the overlap set and nonoverlapset.4

Statistical Assumptions

To establish statistical properties for estimators of thesequantities, statisticians typically posit an extra layerof complication by imagining a superpopulation fromwhich the observed data are drawn repeatedly and thenattempting to infer fixed population quantities from theaverage over hypothetical applications of an estimator torepeated hypothetical draws from the population.5 Wefirst explain the assumption necessary in this hypothet-ical situation, simplified for SATT, but then go a stepfurther and establish conditions under which we wouldget the correct answer in our one sample, without an extralayer of complication. As it turns out, these conditions aresimpler and easier to understand.

First, for formal statistical identification, we makean ignorable treatment assignment (ITA) assumption,which for SATT requires that the mechanism that pro-duced the treatment assignment (i.e., the values of T)be independent of the potential outcome Yi (0) givenX : Ti⊥Yi (0)|X for all treated units (Barnow, Cain, andGoldberger 1980; Rosenbaum and Rubin 1983). This in-dependence assumption can be weakened in ways that arenot usually crucial distinctions in practice (Imbens 2004).(This assumption has also been referred to as “selectionon observables,” “unconfoundedness,” and “conditional

4Although these definitions borrow language from the related over-lap assumption introduced in the previous subsection, the two aredistinct: Regardless of whether a matching counterfactual obser-vation exists to estimate Yi (0), we need to ensure that the ex anteprobability of an alternative treatment assignment would have beenpossible for observation i . However, if the overlap assumption isviolated, it would be impossible to find a suitable counterfactualobservation.In all cases, ways of estimating the overlap set (see above) necessarilydepend on substantive characteristics of the data, but methodol-ogists usually attempt to offer some guidance on the basis of thedata alone. The simplest and most stringent existing definition forthe overlap region is exact matching (Manski 1995). However, inpart because in most applications this definition would result inalmost all observations being in the nonoverlap set and in partbecause reasonable smoothness assumptions make extrapolatingsmall distances over continuous space relatively safe (Zhao 2004),most scholars choose more relaxed definitions. Some others includedefinitions based on nonparametric estimation of the propensityscore, the quality of the worst individual matches (Imbens 2004),and the convex hull (King and Zeng 2006). In the next subsection,we offer approaches that seem naturally implied by each imbalancemetric. In this way, we reduce the number of adjustable parametersto be chosen or assumed while using our methodology.

5Other, alternative sampling and modeling frameworks are some-times suggested instead, but all add an extra imaginary layer ofsome kind.


independence”; special cases of it are referred to as “exo-geneity” and “no omitted variable bias,” among others.)Perhaps the simplest way to satisfy this assumption is toinclude in X any variable that from prior research or the-ory is known to cause either Y or T , since if any subset ofthese variables satisfies ITA, this set will too (VanderWeeleand Shpitser 2011).

Second, for clarity, we pare down ITA and its su-perpopulation sampling framework to its essentials nec-essary for getting the correct answer in the sample:̂SATT = SATT. We retain the counterfactual frameworkinherent in the definition of causality so that Yi (0) isunobserved but defined for treated units. However, forpoint estimates, imagining infinite random draws froman invented population is unnecessary. And if we wishto consider this (e.g., for some types of uncertainty es-timates), we shall follow the principle of privileging thesample in hand so that SATT is defined over the treatedunits in the only data set we actually observe.

The idea of matching is to replace the unob-served Yi (0) for each treated unit i with an observedY j (0) ≡ Y j for a control unit (i.e., Tj = 0) with match-ing covariate values (Xi = X j ). A sufficient (but not

necessary) condition for ̂SATT = SATT is (Yi (0)|Ti =1, Xi ) = (Y j (0)|Tj = 0, Xi ) for all treated units i (withmatching controls j ). However, if any Y j (0) from indi-vidual matches does not equal Yi (0), we can still esti-mate SATT correctly so long as the estimates of Yi (0)are right on average over treated units. Note that thissounds like, but is distinct from, the concept of “un-biasedness,” which refers to averages over hypotheti-cal repeated draws from an imaginary superpopulation,rather than our need for being correct on average overthe real in-sample treated units. We formalize this ideawith the less restrictive uncorrelated treatment assignment(UTA) assumption, which is that Y (0) is uncorrelatedwith T , within strata of X , or equivalently but more sim-ply: meani (Yi (0)|T = 0, X) = mean j (Y j (0)|T = 1, X),which means that within strata defined by X , the averageof Yi (0) for (unobserved) treated and (observed) controlunits is the same.

An even easier way to understand the UTA assump-tion is to consider the case with a data set composedof one-to-one exact matches. Exact matching is equiv-alent to conditioning on X , and one-to-one matchingmeans that weighting within strata is not required (seethe next section). In this simpler situation, X is irrele-vant and the assumption is simply mean(Yi (0)|T = 1) =mean(Y j (0)|T = 0), which reduces to meani (Yi (0)|Ti =1) = mean j (Y j |Tj = 0) since the second term is fullyobserved.

When used in practice, applying UTA (or ITA) re-quires both (a) choosing and measuring the correct vari-ables in X and (b) using an analysis method that controlssufficiently for the measured X so that T and Y (0) aresufficiently unrelated that any biases that a relationshipgenerates can be ignored (e.g., such as if they are muchsmaller than the size of the quantities being estimated).Virtually all observational data analysis approaches, in-cluding matching and modeling methods, assume (a)holds as a result of choices by the investigator. This in-cludes defining which variables are included in X andensuring that the definition of each of the variables has ameaningful scaling or metric. Then, given the choice ofX , the methods distinguish themselves by how they im-plement approximations to (b) given the choice for thedefinition of X .

Matching Frontier Components

The matching frontier (defined formally in the next sec-tion) requires the choice of options for four separate com-ponents. In addition to the quantity of interest, they in-clude and we now describe fixed- versus variable-ratiomatching, a definition for the units to be dropped, andthe imbalance metric.

Fixed- or Variable-Ratio Matching

Some matching methods allow the ratio of treated to con-trol units to vary, whereas others restrict them to have afixed ratio throughout a matched data set. Matching withreplacement usually generates variable-ratio matching.Examples of fixed-ratio matching include simple 1-to-1,1-to- p (for integer p ≥ 1), or j -to-k (for integer j ≥ 1and k ≥ 1) matching. Fixed-ratio matching can be less ef-ficient than variable-ratio matching because some prun-ing usually occurs solely to meet this restriction. However,an important goal of matching is simplicity, so the abilityto match without having to modify subsequent analysisprocedures remains popular. Fixed-ratio matching is alsouseful in large data sets where observations are plentifuland the primary goal is reducing bias. In practice, mostanalysts opt for the even more restrictive requirement ofone-to-one matching.

In fixed-ratio matching, SATT can be estimated bya simple difference in means between the treated andcontrol groups: meani∈{T=1}(Yi ) − mean j∈{T=0}(Y j ).

In variable-ratio matching, we can estimate the TEwithin each matched stratum s by a simple differ-ence in means: meani∈s ,{T=1}(Yi ) − mean j∈s ,{T=0}(Y j ).


However, aggregating up to SATT requires weighting,with the stratum-level TE weighted according to the num-ber of treated units. Equivalently, a weighted difference inmeans can be computed, with weights W such that eachtreated unit i receives a weight of Wi = 1, and each controlunit j receives a weight of Wj = (m0/m1)[(ms1 )/(ms0 )],where m0 and m1 are, respectively, the number of controland treated units in the data set, and ms1 and ms0 arethe number of treated and control units in the stratumcontaining observation j .6

Defining the Number of Units

The size and construction of a matched data set influ-ence the variance of the causal effect estimated from it.Under SATT, the number of treated units remains fixed,and so we measure the data set size by the number ofcontrol units. For FSATT, we measure the total numberof observations.

For both quantities of interest, we will ultimately usean estimator equal to or a function of the difference inmeans of Y between the treated and control groups. Thevariance of this estimator is proportional to 1

nT+ 1

nC,

where nT and nC are the number of treated and con-trol units in the matched set. Thus, the variance of theestimator is largely driven by min(nT , nC ), and so wewill also consult this as an indicator of the size of thedata set.

To simplify notation in these different situations, wechoose a method of counting from those above and letN denote the number of these units in the original dataand n the number in a matched set, with n ≤ N. In ourgraphs, we will represent this information as the numberof units pruned, which is scaled in the same direction asthe variance.

Imbalance Metrics

An imbalance measure is a (nondegenerate) indicator ofthe difference between the multivariate empirical densi-ties of the k-dimensional covariate vectors of treated X1

and control X0 units for any data set (i.e., before or af-ter matching). Our concept of a matching frontier, whichwe define more precisely below, applies to any imbalancemeasure a researcher may choose. We ease this choice bynarrowing down the reasonable possibilities from mea-sures to metrics and then discuss five examples of con-tinuous and discrete families of these metrics. For each

6See j.mp/CEMweights for further explanation of these weights.Also, for simplicity, we define any reuse of control units to matchmore than one control as variable-ratio matching.

example, we give metrics most appropriate for FSATTand SATT when feasible.

Measures versus Metrics. To narrow down the possiblemeasures to study, we restrict ourselves to the more spe-cific concept of an imbalance metric, which is a functiond : [(m0 × k) × (m1 × k)] → [0, ∞] with three proper-ties, required for a generic semi-distance:

1. Nonnegativeness: d(X0, X1) ≥ 0.2. Symmetry: d(X0, X1) = d(X1, X0) (i.e., replac-

ing T with 1 − T will not affect the calculation).3. Triangle inequality: d(X0, X1) + d(X1, Z) ≥

d(X0, Z), given any k-vector Z.

Imbalance measures that are not metrics have been pro-posed and are sometimes used, but they add complica-tions such as logical inconsistencies without conferringobvious benefits. Fortunately, numerous imbalance met-rics have been proposed or could be constructed.

Adjusting Imbalance Metrics for Relative Importance.Although one can always define a data set that will pro-duce large differences between any two imbalance met-rics, in practice the differences among the choice ofthese metrics are usually not large or at least not themost influential choice in most data analysis problems(Imbens 2004; Zhao 2004). Although we describe contin-uous and discrete metrics below, more important is thechoice of how to scale the variables that go into it. In-deed, every imbalance metric is conditional on a defini-tion of the variables in X , and so researchers should thinkcarefully about what variables may be sufficient in theirapplication.

Analysts should carefully define the measurement ofeach covariate so that it makes logical sense (e.g., ensuringinterval-level measurement for continuous distance met-rics) and, most crucially, reflects prior information aboutits “importance” in terms of its relationship with Y |T . Inalmost all areas of applied research, the most importantcovariates are well known to researchers—such as age,sex, and education in public health, or partisan identi-fication and ideology in political science. Since bias is afunction of both importance and imbalance, and match-ing is intended to affect the latter (Imai, King, and Stuart2008), researchers should seek to reduce imbalance morefor covariates known to be important. This is easy to doby choosing the right scale for the variables or, equiva-lently, adjusting weights used in continuous metrics (e.g.,the standardization in average Mahalanobis distance orthe raw variable scaling in Euclidean distance; see Greevyet al. 2012) or the degree of variable-specific coarsening


used in discrete metrics (e.g., H in L 1). We now offermore detail on these continuous and discrete metrics.

Continuous Imbalance Metrics. The core buildingblock of a continuous imbalance metric is a (semi-)distance D(Xi , X j ) between two k-dimensional vectorsXi and X j , corresponding to observations i and j .For example, the Mahalanobis distance is D(Xi , X j ) =√

(Xi − X j )S−1(Xi − X j ), where S is the sample co-variance matrix of the original data X . The Euclideandistance would result from redefining S as the identitymatrix. Numerous other existing definitions of continu-ous metrics could be used instead. In real applications,scholars should choose variables, the coding for the vari-ables, and the imbalance metric together to reflect theirsubstantive concerns. With this method as with most oth-ers, the more substantive knowledge one encodes in theprocedure, the better the result will be.

For example, consider data designed to predict theDemocratic proportion of the two-party vote in a cross-section of Senate elections (“vote”), with incumbencystatus as the treatment, controlling for covariates popu-lation and the vote in the prior election (“lagged vote”).Clearly, lagged vote will be highly predictive of the currentvote, whereas the effect of population will be tiny. Thus,in designing an imbalance metric, we want to be sure tomatch lagged vote very well, and should only be willingto prune observations based on population when we seegross disparities between treatment and control. For ex-ample, if we used Euclidean distance, we could code voteand lagged vote on a scale from 0 to 100 and populationon a smaller scale, such as log population. In computingthe Euclidean distance between two observations, then,population differences would count as equivalent to rel-atively small vote differences.

To produce the overall imbalance metric, wemust move from the distance between two individualobservations to a comparison between two sets ofobservations, by aggregating binary distance calculationsover all the observations. One way to do this is withthe average Mahalanobis imbalance (AMI) metric,the distance between each unit i and the closestunit in the opposite group, averaged over all units:D = meani [D(Xi , X j (i))], where the closest unit in theopposite group is X j (i) = arg minX j | j∈{1−Ti }[D(Xi , X j )]and {1 − Ti } is the set of units in the (treatment orcontrol) group that does not contain i .

Finally, for SATT, it is helpful to have a way to identifythe overlap and nonoverlap sets. A natural way to dothis for continuous metrics is to define the nonoverlapregion as the set of treated units for which no controlunit has chosen it as a match. More precisely, denote the

(closest) treated unit i that control unit j matches toby j (i) ≡ arg mini |i∈{T=1}[D(Xi , X j )]. Then define theoverlap and nonoverlap sets, respectively, as

O ≡ { j (i) | j ∈ {T = 0}} (2)

and

NO ≡ {i | i ∈ {T = 1} ∧ {i �∈ O}}, (3)

where ∧ means “and,” connecting two statements re-quired to hold.

Discrete Imbalance Metrics. Discrete imbalance met-rics indicate the difference between the multivariate his-tograms of the treated and control groups, defined byfixed bin sizes H . The results of this calculation dependon the scale of the input covariates, and so the scale canbe used to adjust the variables for their likely impact onthe outcome variable.

To define these metrics, let f�1···�k be the relativeempirical frequency of treated units in a bin with co-ordinates on each of the X variables as �1 · · · �k sothat f�1···�k = nT�1 ···�k

/nT , where nT�1 ···�kis the number of

treated units in stratum �1 · · · �k and nT is the numberof treated units in all strata. We define g�1···�k similarlyamong control units. Then, among the many possiblemetrics built from these components, we consider two:

L 1(H) = 1

2

∑(�1···�k )∈H

| f�1···�k − g�1···�k | (4)

and

L 2(H) = 1

2

√ ∑(�1···�k )∈H

( f�1···�k − g�1···�k )2. (5)

To remove the dependence on H , Iacus, King, and Porro(2011a) define L 1 as the median value of L 1(H) fromall possible bin sizes H in the original unmatched data(approximated by random simulation); we use the samevalue of H to define L 2. The typically numerous emptycells of each of the multivariate histograms do not affectL 1 and L 2, and so the summation in (4) and (5) each haveat most only n nonzero terms.

When used for creating SATT frontiers, these discretemetrics suggest a natural indicator of the nonoverlap re-gion: all observations in bins with either one or moretreated units and no controls or one or more controlunits and no treateds.

With variable-ratio matching, and the correspond-ing weights allowed in the calculation of L 1, the metricis by definition 0 in the overlap region. With fixed-ratiomatching, L 1 will improve as the heights of the treatedand control histogram bars within each bin in the over-lap region equalize. (In other words, what the weights,


included for variable-ratio matching, do is to equalize theheights of the histograms without pruning observations.)

Constructing Frontiers

Now that we have given our notation and discussedvariable- versus fixed-ratio matching, the number ofunits, and imbalance metrics, we can formally define thefrontier and describe algorithms for calculating it.

Definition

Begin by choosing an imbalance metric d(x0, x1), a quan-tity of interest Q (SATT or FSATT), whether to useweights (to allow variable-ratio matching) or no weights(as in fixed-ratio matching) R, and a definition for thenumber of units U . We will consider all matched dataset sizes from the original N, all the way down fromn = N, N − 1, N − 2 . . . , 2.

For quantity of interest SATT, where only control

units are pruned, denote Xn as the set of all(

Nn

)pos-

sible data sets formed by taking every combination ofn rows (observations) from the (N × k) control groupmatrix X0. Then denote the combined set of all sets Xn

asX ≡ {Xn | n ∈ {N, N − 1, . . . , 1}}. This combined setX is (by adding the null set) known as the power set ofrows of X0, containing (a gargantuan) 2N elements. Forexample, if the original data set contains merely N = 300observations, the number of elements of this set exceedscurrent estimates of the number of elementary particlesin the universe. The task of finding the frontier requiresidentifying a particular optimum over the entire powerset. Using a brute force approach of trying them all andchoosing the best is obviously infeasible even with all theworld’s computers turned to the task. A key contributionof this article is a set of new algorithms that make possiblefinding this exact same frontier in only a few minutes onan ordinary personal computer.

To be more specific, first identify an element (i.e., dataset) of Xn with the lowest imbalance for a given matchedsample size n, and the choices of Q, U , and R:

xn = arg minx0 ∈ Xn

d(x0, x1), given Q, R, and U (6)

where for convenience when necessary we define thearg min function in the case of nonunique minima asa random draw from the set of data sets with the sameminimum imbalance. We then create a set of all theseminima {xN, xN−1, . . . , 1}, and finally define the match-ing frontier as the subsetF of these minima after imposing

monotonicity, which involves eliminating any elementthat has higher imbalance with fewer observations:

F ≡ {xn | (n ∈ {N, N − 1, . . . , 1}) ∧ (dn−1 ≤ dn)},(7)

where dn = d(xn, x1). We represent a frontier by plottingthe number of observations pruned N − n horizontallyand dn vertically.

For simplicity, we will focus on SATT here, but ourdescription also generalizes to FSATT by defining Xn

as the set of all combinations of the entire data matrix(X ′

0, X ′1)′ taken n at a time.

Algorithms

Calculating the frontier requires finding a data subset ofsize n with the lowest imbalance possible, chosen fromthe original data of size N for each possible n (N > n)—given choices of the quantity of interest (and thus thedefinition of the units to be pruned), fixed- or variable-ratio matching, and an imbalance metric, along with themonotonicity restriction.

Adapting existing approaches to algorithms for thistask is impractical. The most straightforward would in-volve directly evaluating the imbalance of the power setof all possible subsets of observations. For even moderatedata set sizes, this would take considerably longer thanthe expected lives of most researchers (and for reasonablysized data sets, far longer than life has been on the planet).Another approach could involve evaluating only a sampleof all possible subsets, but this would be biased, usuallynot reaching the optimum (for the same reason that amaximum in a random sample is an underestimate of themaximum in the population). Finally, adapting generalpurpose numerical optimization procedures designed forsimilar but different purposes would take many years, andthey are not guaranteed to reach the optimum.7

The contribution of our algorithms, then, is not thatthey can find the optimum, but that they can find it fast.Our solution is to offer analytical rather than numericalsolutions to this optimization problem by leveraging theproperties of specific imbalance metrics.

7We are indebted to Diamond and Sekhon (2012) for their idea of“genetic matching,” which seeks to approximate one point on thefrontier (i.e., one matched sample size). For this point, imbalancewill usually be higher and never lower than the frontier, and so theapproach proposed here dominates. We also find, using data fromSekhon (2011) as an example, that computing the entire (optimal)frontier with our approach is about 100 times faster than it takesgenetic matching to compute one point (approximating, but notguaranteed to be on, the frontier). For each individual point, ourapproach is about 1,000,000 times faster.


FIGURE 1 A Demonstration of the Continuous Metric FSATTAlgorithm

6 8 10 12 14 16 18

4

6

8

10

12

14

Covariate #1

Cov

aria

te #

2C7

C3

C6

C8

T3

T1

T5 T8

T2●

●

●

●

●

● ●

Number of Observations Pruned

Mea

n M

ahal

anob

is D

ista

nce

1 2 3 4 5 6 7 8 9

2.50

2.75

3.00

3.25

3.50

3.75

4.00

4.25

4.50

C3,T3

C6C7 C8,T8

T1

T2

T5

Note: The left panel gives an example data set with two covariates, one on each axis. The num-bers indicate the order in which observations are pruned, the arrows denote the observation inthe opposite treatment regime to which each unit is matched, and “T” or “C” denote treatmentand control, respectively. The right panel displays the frontier for this data set, where pointsare labeled to correspond with the left plot.

We now outline algorithms we developed for calcu-lating each of four families of matching frontiers, withmany possible members of each. We leave to future re-search the derivation of algorithms for other families ofmatching frontiers (defined by choosing among permu-tations of the choices defined in the previous subsection,and finding a feasible algorithm). In all cases with SATTfrontiers, we first remove the nonoverlap set and thencompute the remaining frontier within the overlap set.FSATT frontiers do not require this separate step.

Continuous, FSATT, Variable Ratio. Our first family ofalgorithms is based on the average continuous imbalancemetric for FSATT with variable-ratio matching. We il-lustrate this with the AMI metric, although any othercontinuous metric could be substituted. Define N as thenumber of units in the original data, n as the number thathave not yet been pruned, and Dn as the current matcheddata set. Then our algorithm requires five steps:

1. Begin with N = n.2. Match each observation i (i = 1, . . . , N) to the

nearest observation j (i) in the oppositetreatment condition {1 − Ti }; thismatched observation has index j (i) =arg min j∈{1−Ti }[D(Xi , X j )], and distanced j (i) ≡ D(Xi , X j (i)).

3. Calculate AMI for Dn.4. Prune from Dn the unit or units with distance

d j (i) equal to max(d j (i)|i ∈ Dn). Redefine n asthe number of units remaining in newly pruneddata set Dn.

5. If n > 2 and AMI > 0, go to Step 3; else stop.

This is a greedy algorithm, but we now prove that it isoptimal by proving two claims that, if true, are sufficientto establish optimality. Below, we offer a simple example.

Claim 1. For each unit i remaining in the data set, thenearest unit (or units) in the opposite treatment regime j(i)(defined in Step 2) remains unchanged as units are prunedin Step 4.

Claim 2. The subset Dn with the smallest AMI, among all(Nn

)subsets of DN, is that defined in Step 4 of the algorithm.

Given Claim 1, Claim 2 is true by definition since Dn

includes the n smallest distances within DN .We prove Claim 1 by contradiction. First assume that

the next unit to be pruned is the nearest match for aunit not yet pruned (hence requiring the data set to berematched). However, this is impossible because the unitnot yet pruned would have had to have a distance to thepruned unit lower than the maximum, in which case itwould have been matched to that unit originally, which isnot the case. This proves Claim 1. By induction, we thenhave a proof of optimality.

To illustrate, consider the simple data set in the leftpanel of Figure 1, with five treated units (denoted by theletter “T”) and four control units (denoted by the letter“C”), each measured on two covariates (one on each axis).We denote, with an arrow coming from each unit, theclosest unit (measured in Mahalanobis distance) in theopposite treatment regime.

The algorithm is illustrated by removing the obser-vations in numerical order in the left panel of Figure 1,starting with observation T1. The right panel gives the


frontier for these data, numbered with the observation tobe removed next.

The figure also illustrates the special case of “mutualminima,” which is a treated unit that has as its nearestmatch a control unit that, in turn, has as its closest matchthe same treated unit. In this situation, we remove both(or more if they are also tied) units at the same time, asper Step 4. This is illustrated in the left panel of Figure 1by two arrows between the pair of observations marked3 (and also 8, when only two units remain). Technically,this means that some points on the frontier are not rep-resented, such as if we wished to prune exactly three ob-servations in this simple example. Although we could fillin these missing points by enumerating and checking allpossible data sets this size, we find that omitting them,and thus saving substantial computational time, is almostcostless from a practical point of view: A researcher whowants a data set of a particular sample size would almostalways be satisfied with a very slightly different samplesize. With a realistically sized data set, the missing pointson the frontier, like discrete points in a (near) continu-ous space, are graphically invisible and for all practicalpurposes substantively irrelevant.8

Put differently, the intuition as to why the greedy al-gorithm is also optimal is that at every point along thefrontier, the closest match for each observation in the re-maining set is the same as that in the full set, which impliesthat (1) each observation contributes a fixed distance tothe average distance for the entire portion of the frontierin which it remains in the sample and (2) observationsdo not need to be rematched as others are pruned.

Continuous, SATT, Variable Ratio. The SATT frontieris identical to the FSATT frontier when the SATT require-ment of keeping all treated units is not binding. Thisoccurs when the maxi d j (i) calculation in the FSATT algo-rithm leads us to prune only control units. Usually, this isthe case for some portion of the frontier, but not the wholefrontier. When the calculable part of this SATT frontier isnot sufficient, part of each matched data set along the restof the frontier will have a nonoverlap region, requiringextrapolation. In these situations, we recommend usingthe FSATT algorithm for the overlap region, extrapolat-ing to the remaining nonoverlap region and combiningthe results as per Equation (1).

8It is important to note that the missing points on the frontiercannot violate the monotonicity constraint. If the point after the gapcontains n observations, those n observations are the n observationswith the closest matches. Therefore, there can be no set of n + 1observations (say, in the gap) with a lower mean distance than thatin the n observations contained at the defined point because theadditional observation must have a distance equal to or greaterthan the greatest distance in the existing n observations.

Discrete, FSATT, Variable Ratio. As a third family offrontiers, we can easily construct a discrete algorithm forFSATT with variable-ratio matching, using L 1 for ex-position. With variable-ratio matching in discrete met-rics, weighting eliminates all imbalance in the overlap set,making this frontier a simple step function with only twosteps. The first step is defined by L 1 for the original data,and the second is L 1 after the nonoverlap region has beenremoved. Within each step, we could remove observa-tions randomly one at a time, but since imbalance doesnot decline as a result, it makes more sense to define thefrontier for only these two points on the horizontal axis.

If the binning H is chosen to be the same as thecoarsening in coarsened exact matching (CEM), the sec-ond step corresponds exactly to the observations retainedby CEM (Iacus, King, and Porro 2011a, 2011b).

Discrete, SATT, Fixed Ratio. A final family of frontiersis for discrete metrics, such as L 1 or L 2, for quantityof interest SATT with fixed-ratio matching. To definethe algorithm, first let bi T and biC be the numbers, andpi T = bi T/nT and piC = biC /nC be the proportions, oftreated and control units in bin i (i ∈ {1, . . . , B}) inthe L 1 multivariate histogram, where nT = ∑B

i=1 bi T and

nC = ∑Bi=1 biC are the total numbers of treated and con-

trol units.To prune k observations optimally (i.e., with mini-

mum L 1 imbalance) from a data set of size N, we offerthis algorithm:

1. Define p′iC = biC /(nC − k).

2. Prune up to k units from any bin i where afterpruning p′

iC ≥ pi T holds.3. If k units have not been pruned in Step 2, prune

the remaining k′ units from the bins with the k′

largest differences piC − p′i T .

An optimal frontier can then be formed by applying thisalgorithm with k = 1 pruned and increasing until smallnumbers of observations result in nonmonotonicities inL 1.

The discreteness of the L 1 imbalance metric meansthat multiple data sets have equivalent values of the im-balance metric for each number of units pruned. Indeed,it is possible with this general algorithm to generate onedata set with k − 1 pruned and another with k prunedthat differ with respect to many more than one unit. Infact, even more complicated is that units can be added,removed, and added back for different adjacent points onthe frontier. This, of course, does not invalidate the algo-rithm, but it would make the resulting frontier difficultto use in applied research. Thus, to make this approach


easier to use, we choose a greedy algorithm that is a spe-cial case of this general optimal algorithm. The greedyalgorithm is faster, but more importantly, it by definitionnever needs to put a unit back in a data set once it hasbeen pruned.

Our greedy algorithm, which we show to be optimal,is as follows. Starting with the full data set N = n,

1. Compute and record the value of L 1 and thenumber of units n.

2. Prune a control unit from the bin with themaximum difference between the proportionsof treated and control units, such that there aremore controls than treateds. That is, prune a unitfrom bin f (i), where

f (i) = arg maxi∈{nC >nT }

|pic − pit |. (8)

3. If L 1 is larger than the previous iteration, stop;otherwise, go to Step 1.

To understand this algorithm intuitively, first notethat deleting a control unit from any bin with more con-trols than treateds changes L 1 by an equal amount (be-cause we are summing over bins normalized by the totalnumber of controls, rather than by the number of con-trols in any particular bin). When we delete a control unitin one bin, the relative size of all the other bins increasesslightly because all the bins must always sum to 1. Deletingcontrols from bins with the greatest relative difference, aswe do, prevents the relative number of treated units fromever overtaking the relative number of controls in any binand guarantees that this greedy algorithm is optimal.

To illustrate the greedy version of this optimal algo-rithm, Figure 2 gives a simple univariate example. Panel0 in the top left represents the original data set with a his-togram in light gray for controls and dark gray for treatedunits. The L 1 imbalance metric for Panel 0 is reported inthe frontier (Figure 2, bottom right) marked as “0.” Theunit marked in black in Panel 0 is the next control unit tobe removed, in this case because it is in a bin without anytreated units.

Then Panel 1 (i.e., where one observation has beenpruned) removes the black unit from Panel 0 and renor-malizes the height of all the bars with at least some con-trol units so that they still sum to 1. As indicated by theextra horizontal lines, reflecting the heights of controlhistogram bars from the previous panel, the heights ofeach of the remaining light gray control histogram barshave increased slightly. The “1” point in the bottom-rightpanel in Figure 2 plots this point on the frontier. Theblack piece of Bin 5 in Panel 1 refers to the next observa-tion to be removed, in this case because this bin has the

largest difference between control and treateds, amongthose with more controls than treateds (i.e., Equation 8).Panels 2–4 repeat the same process as in Panel 1, until weget to Panel 4, where no additional progress can be madeand the frontier is complete.

Applications

We now demonstrate our approach in two substantiveapplications. The first uses a SATT frontier, so a directcomparison can be made between experimental and ob-servational data sets. The second uses an FSATT frontierin a purely observational study, and thus we focus onunderstanding the new causal quantity being estimated.

Job Training

We begin with a data set compiled from the Na-tional Supported Work Demonstration (NSWD) and theCurrent Population Survey (Dehejia and Wahba 2002;Lalonde 1986). The first was an experimental interven-tion, whereas the second is a large observational data col-lection appended to the 185 experimental treated unitsand used in the methodological literature as a mock con-trol group to test a method’s ability to recover the ex-perimental effect from observational data. Although ourpurpose is only to illustrate the use of the matching fron-tier, we use the data in the same way.

As is standard in the use of these data, we match onage, education, race (black or Hispanic), marital status,whether or not the subject has a college degree, earningsin 1974, earnings in 1975, and indicators for whether ornot the subject was unemployed in 1974 and 1975. Earn-ings in 1978 is the outcome variable. To ensure a directcomparison, we estimate the SATT fixed-ratio frontier,and thus prune only control units. We give the fullmatching frontier in the top-left panel of Figure 3, and, inthe two lower panels, the effect estimates for every pointon the frontier (the lower-left panel displays estimatesover the full frontier, and the lower-right panel zooms inon the final portion of the frontier). There is no risk ofselection bias in choosing a point on the frontier basedon the estimates, so long as the entire set is presented, aswe do here.

These data include 185 treated (experiment) unitsand 16,252 control units, so pruning even the vastmajority of the control units will not reduce the vari-ance much and could help reduce imbalance (see thesection “Defining the Number of Units”). In the presentdata, this insight is especially important given that the


FIGURE 2 A Demonstration of the L 1 SATT Algorithm

Bin 1 Bin 2 Bin 3 Bin 4 Bin 50.0

0.1

0.2

0.3

0.4

Next Unit to Remove

Control

Treatment

0


0.1

0.2

0.3

0.4

Horizontal lines mark the heights of bars in

the previous graph

1


0.1

0.2

0.3

0.4

2


0.1

0.2

0.3

0.4

3


0.1

0.2

0.3

0.4

4●

●

●

●

●

1 2 3 4 5

0.04

0.06

0.08

0.10

0.12

0.14


0

1

2

3

4

L1 D

iffer

ence

largest trade-off between pruning and imbalance occursafter most of the observations are pruned; this can be seenat the right of the left panel of Figure 3, where the frontierdrops precipitously.

The bottom left panel gives the causal effect esti-mates, where the apparent advantages of pruning most ofthe cases can be seen. Over most of the range of the fron-tier, causal estimates from the (badly matched) data do

not move much from the original unmatched estimate,−$8,334, which would indicate that this job training pro-gram was a disaster. Then after the fast drop in imbal-ance (after pruning about 15,000 mostly irrelevant obser-vations), the estimates rise fast and ultimately intersectwith the experimental estimate of the training program,producing a benefit of $1,794 per trainee (denoted bythe red horizontal line). Overall, the frontier reveals the


FIGURE 3 The Effect of a Job Training Program on Wages

0 5000 10000 15000

0.5

0.6

0.7

0.8

0.9

1.0


L1


Scale

d M

eans

0.0

0.2

0.4

0.6

0.8

0 5000 10000 15000

Age

Education

BlackHispanic

Married

No degreeIncome 1974

Income 1975

Unemployment 1974Unemployment 1975

0 5000 10000 15000

−10000

−8000

−6000

−4000

−2000

0

2000


Estim

ate

Experimental Estimate

Matching Estimate

16000 16050 16100 16150

−10000

−8000

−6000

−4000

−2000

0

2000


Estim

ate

Experimental Estimate

Matching Estimate

Note: The top-left panel is the L 1 frontier for the job training data, beginning at the left before pruning with n = 16, 252(nonexperimental) control units. The top-right panel displays covariate means across the frontier. The bottom-left paneldisplays causal effect estimates along that frontier, with Athey-Imbens model dependence intervals calculated at each point.The bottom-right panel displays the final points on the frontier, effectively “zooming” in on the portion where imbalanceimproves rapidly. The horizontal dashed line is the estimated treatment effect from the experimental sample.

whole range of possible conclusions as a function of thefull bias-variance trade-off. The bottom panels also givesAthey-Imbens model dependence intervals (Athey andImbens 2015) around the point estimates;9 the widths of

9Athey and Imbens (2015) propose “a scalar measure of the sensi-tivity of the estimates to a range of alternative models.” To computethis measure, investigators estimate the quantity of interest with abase model, after which the quantity of interest is estimated insubsamples divided according to covariate values. The deviation ofthese subsample estimates from the base estimate is then a measureof model dependence.

these are controlled by the model dependence remainingin the data and so decrease as balance improves across thefrontier. Correspondingly, the largest change in modeldependence occurs near the end of the frontier, whereimbalance improves the most.

Sex and Judging

For our second example, we replicate Boyd, Epstein,and Martin (2010), who offer a rigorous analysis of the


effect of sex on judicial decision making. They first reviewthe large number of theoretical and empirical articles ad-dressing this question and write that “roughly one-thirdpurport to demonstrate clear panel or individual effects,a third report mixed results, and the final third find nosex-based differences whatsoever.” These prior articles alluse similar parametric regression models (usually logitor probit) and related data sets. To tame this remarkablelevel of model dependence, they introduce nearest neigh-bor propensity score matching to this literature, findingno effects of sex on judging in every one of 13 policy areasexcept for sex discrimination, which makes good sensesubstantively. The authors also offer a spirited argumentfor bringing “promising developments in the statisticalsciences” to important substantive questions in judicialpolitics, and so we follow their lead here, too. We thusfollow their inclinations but with methods developed af-ter the publication of their article, by seeing whether ourmore powerful approach can detect results not previouslypossible.

Boyd, Epstein, and Martin (2010) motivate theirstudy by clarifying four different mechanisms throughwhich sex might influence judicial outcomes, each withdistinct empirical implications. First, different voice im-plies that “males and females develop distinct worldviewsand see themselves as differentially connected to society.”This account suggests that males and females should ruledifferently across a broad range of issues, even those withno clear connection to sex. Second, the representationalmechanism posits that “female judges serve as representa-tives of their class and so work toward its protection in lit-igation of direct interest.” This theory predicts that malesand females judge differently on issues of immediate con-cern to women. Third, the informational account arguesthat “women possess unique and valuable informationemanating from shared professional experiences.” Here,women judge differently on the basis of their unique in-formation and experience, and so they might differ frommen on issues over which they have distinct experiences,even if not related to sex. Finally, the organizational the-ory posits that “male and female judges undergo identicalprofessional training, obtain their jobs through the sameprocedures, and confront similar constraints once on thebench.” This theory predicts than men and women donot judge differentially.

Boyd, Epstein, and Martin (2010) argue that theirresults—that males and females judge differently only insex discrimination cases—are “consistent with an infor-mation account of gendered judging.” Of course, their re-sults are also consistent with representational theories. In-deed, as Boyd, Epstein, and Martin (2010) argue, womenmight judge differently on sex discrimination because

they have different information as a result of shared ex-periences with discrimination. But it is also possible thatwomen judge differently on sex discrimination as a wayto protect other women, consistent with representationalaccounts.

One way to use our new methodological approach isto attempt to distinguish between these conflicting inter-pretations of the effects of sex. To do so, we analyze caseson an issue for which we expect to observe a differenceif and only if the informational account is true—that is,an issue area where the unique experiences of womenmight lead to informational differences between men andwomen but that nonetheless does not directly concern theinterests of women. For this analysis, we consider casesrelated to discrimination on the basis of race. Becausewomen have shared experiences with discrimination,they have informational differences from men relevant tothis issue area. However, judgments on racial discrimi-nation do not have direct consequences for women morebroadly. This issue area is the only such issue area thatallows us to distinguish between these two accounts andfor which we also have a suitable amount of available data.

Thus, using their data, we reanalyze race discrimina-tion cases made on the basis of Title VII. In their originalanalysis, Boyd, Epstein, and Martin (2010) found a nulleffect with this issue area, both before and after match-ing. We now show, with our method, which enables us toanalyze data with less dependence and bias than previousmatching approaches, that female judges rule differentlyon race discrimination cases. We show that differencesin male and female judgments are at least in part due toinformational differences.

We arranged the data from Boyd, Epstein, and Martin(2010) so that the unit of analysis is the appellate courtcase, the level at which decisions are made. For exam-ple, the fourth observation in our data set is Swinton v.Potomac Corporation, which was decided in 2001 by theNinth Circuit Court of Appeals, at the time composedof Judges William A. Fletcher, Johnnie B. Rawlinson, andMargaret M. McKeown. Our treatment is whether or notat least one female judge was included in the three-judgepanel. In Swinton v. Potomac Corporation, Judges Rawl-inson and McKeown are female, and so this observationis in the treatment group. For each appellate court case,we use the following covariates: (1) median ideology asmeasured by Judicial Common Space scores (Epstein et al.2007; Giles, Hettinger, and Peppers 2001), (2) median age,(3) an indicator for at least one racial minority among thethree judges, (4) an indicator for ideological direction ofthe lower court’s decision, (5) an indicator for whether amajority of the judges on the three-judge panel were nom-inated by Republicans, and (6) an indicator for whether a


FIGURE 4 The Effect of Sex on Judges in Title VII Race Discrimination Cases

0 50 100 150 200 250 300

0.0

0.2

0.4

0.6

0.8


Mahala

nobis

Dis

cre

pancy

(a) Title VII Race Imbalance-n Frontier

0 50 100 150 200 250

0.00

0.05

0.10

0.15

0.20

0.25

0.30


Estim

ate

(b) Title VII Race Causal Effects Estimates

Note: Panel (a) displays the imbalance-n frontier, and Panel (b) shows estimates of the causal effect acrossthat same frontier. The shaded region is the Athey-Imbens model dependence interval.

majority of the judges on the panel had judicial experienceprior to their nomination. In Swinton v. Potomac Corpora-tion, for example, the median ideology was at the 20th per-centile in the distribution of judges who ruled on a TitleVII race discrimination case and in the 10th percentile inthe distribution of median ideologies, where lower scoresare more liberal and higher scores are more conservative.This is unsurprising, as all three judges were nominated byPresident Clinton in either 1998 or 2000. Our outcome isthe ideological direction of the decision—either liberal orconservative—and unsurprisingly, the ruling on Swintonv. Potomac Corporation was a liberal one. For our analy-sis, we use these six covariates to construct a Mahalanobisfrontier for the estimation of FSATT.

We present the matching frontier in Figure 4, Panel(a); in contrast to our previous example, most of the re-duction in imbalance for observations pruned happensearly on, but substantial imbalance reduction continuesthrough the entire range. Our substantive results can befound in Panel (b), which indicates that having a femalejudge increases the probability of a liberal decision, overthe entire range. The vertical axis quantifies the substan-tial effect we see in terms of the reduction in probability,from about 0.05 with few observations pruned and higherlevels of model dependence to 0.25 with many pruned andlower levels. Most importantly in this case, the point es-timate of the causal effect increases as balance improvesand we zero in on a data subset more closely approximat-ing a randomized experiment. Correspondingly, modeldependence, indicated by the shaded interval, decreases.

Because we pruned both treated and control units inthis example, we must carefully consider how the quantityof interest changes as balance improves. Interestingly, thebest balance exists within more conservative courts (andthus the effect we find in Figure 4 is among these courts).To see this, note that in Figure 5, which plots the meansof each covariate as observations are pruned, variablesassociated with the court ideology changed the most, andall moved in a conservative direction.

More specifically, Republican majority has the largestdifference in means, followed by median ideology and me-dian age. That makes sense, as we expect all-male panels(those assigned to control) to be more conservative onaverage, if only because Democratic presidents are morelikely to appoint women than Republicans. In our data,7% of judges appointed by Republicans are women, com-pared with 25% of judges appointed by Democrats. Thekey issue is that liberal courts with no female judges arerare, and so the data do not admit reasonable inferencesamong this subset of data (e.g., 31% of courts with aRepublican majority have at least one woman comparedwith 50% of courts without a Republican majority).

Our matching technique thus successfully identifiesa subset of data with balance, and this subset is wherewomen are assigned to conservative courts as well as asuitable comparison group of conservative courts with-out a woman. The causal effect of sex on judging in con-servative courts is not the only question of interest inthe literature, but it is certainly one question of interest,and, as it happens, it is one in these data that offers a


FIGURE 5 Covariate Means Across the TitleVII Race Discrimination Frontier


Scale

d M

eans

0.0

0.2

0.4

0.6

0.8

0 100 200 300

Median Ideology

Median Age

Republican Majority

Has Minority

Majority Experienced

Liberal Lower Direction

(a) Title VII Race Covariate Means

Note: The figure displays the means of each covariate as ob-servations are pruned and balance improves. Note the largechange in variables measuring the ideology of the court.

reasonably secure answer. Thus, for example, after prun-ing 210 appellate court cases, the difference in the me-dian Judicial Common Space score for treated and controlunits is 0.02, compared to 0.1 in the full data. However, in-terestingly, the average median score for controls is essen-tially constant as we prune observations, changing from0.123 to 0.163. In contrast, the average median score fortreated units (courts with at least one woman) goes from0.02 to 0.143, which is to say it rises to the level of conser-vatism displayed by courts assigned to the control group.

These results suggest an especially interesting conclu-sion to supplement the original findings in Boyd, Epstein,and Martin (2010), namely, that the mechanism for dif-ferences in judicial decision making is at least in partinformational, perhaps in addition to being representa-tional, even among the most conservative courts.

Concluding Remarks

Matching methods provide a simple yet powerful wayto control for measured covariates. However, in practice,implementing matching methods can be frustrating be-cause they require analysts to jointly optimize balanceand sample size through a complicated, poorly guidedprocess of tweaking matching algorithms to obtain bet-ter matched samples. The matching frontier we describe

here offers the first simultaneous optimization of bothbalance and sample size while retaining much of the sim-plicity that made matching attractive in the first place.

With the approach offered here, once a researcherchooses an imbalance metric and set of covariates, allanalysis is automatic. This is in clear distinction to the bestpractices recommendations for prior matching methods.However, although the choice of a particular imbalancemetric does not usually matter that much, the methodsoffered here do not free one from the still crucial task ofchoosing the right set of pretreatment control variables,coding them appropriately so that they measure what isnecessary to achieve ignorability or, in the case of FSATT,from understanding and conveying clearly to readers thequantity being estimated.

Our approach generalizes directly for multivaluedtreatments without major changes in any of the technol-ogy described. A larger change, and potentially fruitfultopic for future research, would be combining our ap-proach to the frontier with formal sensitivity testing forcausal inference.

References

Athey, Susan, and Guido Imbens. 2015. “A Measure of Robust-ness to Misspecification.” American Economic Review Papersand Proceedings 105(5), 476–480.

Austin, Peter C. 2008. “A Critical Appraisal of Propensity-ScoreMatching in the Medical Literature between 1996 and 2003.”Journal of the American Statistical Association 27(12): 2037–49.

Barnow, Burt S., Glen G. Cain, and Arthur S. Goldberger. 1980.“Issues in the Analysis of Selectivity Bias.” In EvaluationStudies, ed. E. Stromsdorfer and G. Farkas. Vol. 5. San Fran-cisco: Sage.

Boyd, Christina L., Lee Epstein, and Andrew D. Martin. 2010.“Untangling the Causal Effects of Sex on Judging.” AmericanJournal of Political Science 54(2): 389–411.

Caliendo, Marco, and Sabine Kopeinig. 2008. “Some Practi-cal Guidance for the Implementation of Propensity ScoreMatching.” Journal of Economic Surveys 22(1): 31–72.

Crump, Richard K., V. Joseph Hotz, Guido W. Imbens, andOscar Mitnik. 2009. “Dealing with Limited Overlap in Es-timation of Average Treatment Effects.” Biometrika 96(1):187–99.

Dehejia, Rajeev H., and Sadek Wahba. 2002. “Propensity Score-Matching Methods for Nonexperimental Causal Studies.”Review of Economics and Statistics 84(1): 151–61.

Diamond, Alexis, and Jasjeet S. Sekhon. 2012. “Genetic Match-ing for Estimating Causal Effects: A General MultivariateMatching Method for Achieving Balance in ObservationalStudies.” Review of Economics and Statistics 95(3): 932–45.

Epstein, Lee, Andrew D. Martin, Jeffrey A. Segal, and ChadWesterland. 2007. “The Judicial Common Space.” Journal ofLaw, Economics, and Organization 23(2): 303–25.


Giles, Micheal W., Virginia A. Hettinger, and Todd Peppers.2001. “Picking Federal Judges: A Note on Policy and Parti-san Selection Agendas.” Political Research Quarterly 54(3):623–41.

Greevy, Robert A., Carlos G. Grijalva, Christianne L. Roumie,Cole Beck, Adriana M. Hung, Harvey J. Murff, Xulei Liu,and Marie R. Griffin. 2012. “Reweighted Mahalanobis Dis-tance Matching for Cluster-Randomized Trials with Miss-ing Data.” Pharmacoepidemiology and Drug Safety 21(S2):148–54.

Heckman, James, Hidehiko Ichimura, and Petra Todd. 1998.“Matching as an Econometric Evaluation Estimator: Evi-dence from Evaluating a Job Training Program.” Review ofEconomic Studies Evaluation of Training and Other SocialProgrammes (Oct., 1997), 64(4): 605–654.

Ho, Daniel, Kosuke Imai, Gary King, and Elizabeth Stuart.2007. “Matching as Nonparametric Preprocessing for Re-ducing Model Dependence in Parametric Causal Inference.”Political Analysis 15(3): 199–236. http://gking.harvard.edu/files/abs/matchp-abs.shtml.

Holland, Paul W. 1986. “Statistics and Causal Inference.” Jour-nal of the American Statistical Association 81(396): 945–60.

Iacus, Stefano M., Gary King, and Giuseppe Porro. 2011a. “Mul-tivariate Matching Methods That Are Monotonic ImbalanceBounding.” Journal of the American Statistical Association106(493): 345–61. http://gking.harvard.edu/files/abs/cem-math-abs.shtml.

Iacus, Stefano M., Gary King, and Giuseppe Porro. 2011b.“Replication Data for Causal Inference without BalanceChecking: Coarsened Exact Matching.” http://hdl.handle.net/1902.1/15601. Murray Research Archive [Distributor],V1 [Version].

Imai, Kosuke, Gary King, and Elizabeth Stuart. 2008. “Mis-understandings among Experimentalists and Observation-alists about Causal Inference.” Journal of the Royal Sta-tistical Society, Series A 171(2): 481–502. http://gking.harvard.edu/files/abs/matchse-abs.shtml.

Imbens, Guido W. 2004. “Nonparametric Estimation of AverageTreatment Effects under Exogeneity: A Review.” Review ofEconomics and Statistics 86(1): 4–29.

Imbens, Guido W., and Donald B. Rubin. 2009. “Causal In-ference for Statistics, Social, and Biomedical Sciences: AnIntroduction.” 2015. New York: Cambridge University Press.

King, Gary, Christopher Lucas, and Richard A. Nielsen. 2016.MatchingFrontier: Automated Matching for Causal Inference.R package version 2.0.0.

King, Gary, Lucas Christopher, Richard A. Nielsen2016. “Replication Data for: The Balance-Sample Size

Frontier in Matching Methods for Causal Infer-ence”, doi:10.7910/DVN/SURSEO, Harvard Dataverse, V1[UNF:6:N+6bvznjSZ2ZDl/OMFDocQ==]

King, Gary, and Langche Zeng. 2006. “The Dangers of Ex-treme Counterfactuals.” Political Analysis 14(2): 131–59.http://gking.harvard.edu/files/abs/counterft-abs.shtml.

Lalonde, Robert. 1986. “Evaluating the Econometric Evalua-tions of Training Programs.” American Economic Review76(4): 604–20.

Manski, Charles F. 1995. Identification Problems in the SocialSciences. Cambridge, MA: Harvard University Press.

Rosenbaum, Paul R., Richard N. Ross, and Jeffrey H. Silber.2007. “Minimum Distance Matched Sampling with FineBalance in an Observational Study of Treatment for Ovar-ian Cancer.” Journal of the American Statistical Association102(477): 75–83.

Rosenbaum, Paul R., and Donald B. Rubin. 1983. “The CentralRole of the Propensity Score in Observational Studies forCausal Effects.” Biometrika 70(1): 41–55.

Rosenbaum, Paul R., and Donald B. Rubin. 1984. “ReducingBias in Observational Studies Using Subclassification on thePropensity Score.” Journal of the American Statistical Associ-ation 79(387): 515–24.

Rubin, Donald B. 2008. “For Objective Causal Inference, De-sign Trumps Analysis.” Annals of Applied Statistics 2(3):808–40.

Rubin, Donald B. 2010. “On the Limitations of ComparativeEffectiveness Research.” Statistics in Medicine 29(19): 1991–95.

Sekhon, Jasjeet S. 2011. “Multivariate and Propensity ScoreMatching Software with Automated Balance Optimization:The Matching Package for R.” Journal of Statistical Software42.

Stuart, Elizabeth A. 2008. “Developing Practical Recommen-dations for the Use of Propensity Scores: Discussion of ‘ACritical Appraisal of Propensity Score Matching in the Med-ical Literature between 1996 and 2003’.” Statistics in Medicine27(12): 2062–65.

VanderWeele, Tyler J., and Miguel A. Hernan. 2012. “CausalInference under Multiple Versions of Treatment.” Journal ofCausal Inference 1: 1–20.

VanderWeele, Tyler J., and Ilya Shpitser. 2011. “A New Criterionfor Confounder Selection.” Biometrics 67(4): 1406–13.

Zhao, Zhong. 2004. “Using Matching to Estimate TreatmentEffects: Data Requirements, Matching Metrics, and MonteCarlo Evidence.” Review of Economics and Statistics 86(1):91–107.

the balance‐sample size frontier in matching … balance-sample size frontier in matching ......

Documents