European Journal of Operational Research 206 (2010) 667–675

Contents lists available at ScienceDirect

European Journal of Operational Research

journal homepage: www.elsevier .com/locate /e jor

Innovative Applications of O.R.

The selection efficiency of tournaments

Dmitry Ryvkin *

Department of Economics, Florida State University, Tallahassee, FL 32306-2180, USA

a r t i c l e i n f o a b s t r a c t

Article history:Received 20 April 2009Accepted 5 March 2010Available online 15 March 2010

Keywords:Human resourcesApplied probabilitySelectionTournamentSimulation

0377-2217/$ - see front matter � 2010 Elsevier B.V. Adoi:10.1016/j.ejor.2010.03.016

* Tel.: +1 850 644 7209; fax: +1 850 644 4535.E-mail address: dryvkin@fsu.edu

We discuss tournaments in terms of their efficiency as probabilistic mechanisms that select high-qualityalternatives (‘‘players”) in a noisy environment. We characterize the selection efficiency of three suchmechanisms – contests, binary elimination tournaments, and round-robin tournaments – dependingon the shape of the distribution of players’ quality, the number of players, and noise level. The resultshave implications as to how, and under what circumstances, the efficiency of tournament-based selectioncan be manipulated.

� 2010 Elsevier B.V. All rights reserved.

1. Introduction

Consider recruitment of employees by a firm. After the initialscreening of applications, those who meet the minimum qualifica-tions form a set from which further selection is made by comparingapplicants with one another.

Recruitment is an example of a selection problem. Generally, adecision maker (DM) faces a selection problem when one or sev-eral ‘‘best” alternatives must be chosen from a set. Nontrivial in-stances of this problem are, for example, selecting a populationwith the highest unobserved mean (e.g., Gibbons et al., 1977),the secretary problem (e.g., Bearden et al., 2006; Bearden andMurphy, 2007), or a multi-attribute choice problem (e.g., Leskinenet al., 2004; Baucells et al., 2008).

In this paper, we study the following selection problem: The DMfaces a random sample of N alternatives (‘‘players”). Each alterna-tive is characterized by a single attribute (‘‘ability”) – an indepen-dent draw from the population with a known distribution. The DMcannot observe abilities, but can observe the results of (possiblymultiple) ordinal comparisons of randomly perturbed abilitiesamong the alternatives. The DM is interested in choosing the alter-native with the highest ability, or, equivalently, the lowest rankorder. Multi-stage recruitment is an example of such an environ-ment: applicants’ abilities are unknown, and their overall perfor-mance difficult to quantify on a numerical scale, but it may berelatively easy to compare one applicant with another. This selec-tion problem is different from the ones discussed in Gibbons et al.(1977), where alternatives are populations with unknown (but

ll rights reserved.

deterministic) parameters (e.g., means), and multiple observationsfrom each population are available to the DM.

Selection in such an environment can be made through a tour-nament – a scheme that uses (possibly multiple) comparisons ofalternatives (‘‘players”) to produce one player as the winner. In anoisy environment, the best player wins a tournament with someprobability different from one. A DM, therefore, can be interestedin the efficiency of different selection schemes, and how it dependson parameters such as the number of players, the noise level, andthe distribution of players’ abilities in the population.

In the economics and management literature, tournaments aremainly discussed in relation to incentive provision in firms (e.g.,Prendergast, 1999; Orrison et al., 2004; Gerchak and He, 2003),sports (e.g., Szymanski, 2003), research competition (e.g., Taylor,1995), and rent seeking (e.g., Lockard and Tullock, 2001). Undertournament incentives, players choose the supply of costly effortor other resources, and the principal’s objective is, typically, themaximization of total output.

An alternative view of tournaments as selection mechanismsthat help identify better players was introduced to the economicsliterature by Hvide and Kristiansen (2003). The authors considera contest in which players can choose the level of risk pertainingto their output and find that selection efficiency, defined there asthe probability of a high-ability type player winning the contest,may be a nonmonotonic function of the number of competitorsand of the proportion of high-ability types in the population.

The concept of selection efficiency of tournaments as quality-enhancing selection mechanisms is also discussed in the statisticaldecision theory literature (see, e.g., Gibbons et al., 1977; Narayana,1979; David, 1988). Here, unlike in the economics and manage-ment literature, it is typically assumed that tournament ‘‘players”

668 D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675

do not make strategic choices, i.e. their performance is pre-deter-mined by their intrinsic quality and, possibly, random factors be-yond their control. A DM’s objective then is to choose theappropriate selection scheme to most reliably identify players withbetter quality by ‘‘filtering out” the noise through multipleobservations.

In this paper, similar to the statistical decision theory literature,we assume that players’ effort levels are not a choice variable. Thisassumption restricts the applicability of our results in managerialsettings to situations when the incentives in a tournament gameare set in such a way that all players always choose to performat their best. Albeit restrictive, this assumption is realistic for animportant class of selection situations involving human subjects,such as final stages of recruitment tournaments and other environ-ments where stakes are high, and significant prior investment hasalready been made by competitors. Other examples include forma-tion of the Olympic team, innovation races, elections, or high levelsports tournaments. Our results, of course, are also applicable tochoice situations where alternatives are not humans, such as, forexample, choice among different technologies.

Tournament selection schemes can be constructed using differ-ent matching and/or elimination rules, or formats. The simplesttournament format is the contest format, in which all players per-form only once and the player with top performance is the winner.Recruitment, however, is often done in several stages, which sug-gests that a one-shot format, such as a contest, is too noisy to makereliable inference in the selection problem discussed here. A multi-stage elimination format, for example, the binary elimination (alsoknown as knock-out) format, can be used instead. In the economicsand management literature, such formats are discussed in the con-text of promotions in hierarchical organizations (see, e.g., Rosen,1986; O’Flaherty and Siow, 1995; Devaro, 2006). In the statisticaldecision theory literature, the selection efficiency of knock-outtournaments is studied for various configurations of winning prob-abilities and seeding (see Hartigan, 1968; Knuth, 1987; Marchand,2002; Israel, 1981; Hwang, 1982; Horen and Reizman, 1985,among others).

Another prominent example of a nontrivial tournament selec-tion scheme is the round-robin format, in which multiple binarycomparisons determine the score of each player, and the playerwith the top score is the winner (see, e.g., Harary and Moser,1966; Rubinstein, 1980; Mendonça and Raghavachari, 2000). Anumber of authors compare round-robin and knock-out tourna-ments in some special cases (David, 1959; Glenn, 1960; Searls,1963; Appleton, 1995; McGarry and Schutz, 1997). In this litera-ture, tournaments involving multiple binary comparisons (suchas knock-out and round-robin) are typically parameterized by amatrix of deterministic winning probabilities. This setting doesnot allow one to explore the parametric dependence of tournamentoutcomes on the number of players, noise level, or the distributionof players’ abilities, therefore in this paper we use a differentapproach.

The efficiency of a tournament selection scheme can be charac-terized by several criteria. In the context of the selection problemdiscussed here, Ryvkin and Ortmann (2008) explore the predictivepower criterion – the probability of selecting the best player asthe winner. One of the central findings of Ryvkin and Ortmann(2008) is that predictive power exhibits nonmonotonicity as afunction of the number of players for fat-tailed distributions ofplayers’ abilities.

In this paper, we explore two alternative measures of selectionefficiency of tournaments: the expected ability of the winner, andthe expected rank of the winner. From a DM’s perspective, thesemeasures are more ‘‘balanced” than the predictive power measurein that they refer to the characteristics of the chosen alternativenot requiring necessarily that it be the best one. For example, a

recruiting committee interested in hiring the candidate with thehighest possible ability, regardless of ranking, can choose to max-imize the expected ability of the winner. At the same time, arecruiting committee whose goal is to get an edge in a race againstother firms will tend to minimize the expected rank (see, e.g., Assafand Samuel-Cahn, 1996).

Ryvkin and Ortmann (2008) provide the results of exploratorysimulations for the expected ability and expected rank of the win-ner. Simulations suggest that, similarly to predictive power, thetwo efficiency criteria can exhibit nonmonotonic behavior. No the-oretical foundation is provided, however. In this paper, we analyzethe expected ability and expected rank of the winner for threetournament formats – contests, binary elimination and round-ro-bin tournaments. We provide a theory, a comprehensive set of sim-ulation results, and a detailed discussion for the two criteria. Ourmajor contribution is in showing that (i) the expected ability ofthe winner always increases in the number of players; and, (ii) likepredictive power, the expected rank of the winner exhibits nonmo-notonicity as a function of the number of players for fat-tailed dis-tributions of abilities. Also, both the expected ability and expectedrank of the winner become nonmonotonic as a function of noise le-vel for round-robin tournaments when the number of players issufficiently large. Our results have important implications forDMs facing the selection problem in noisy environments, such asrecruitment committees in organizations, or Olympic committees.

The rest of the paper is organized as follows. In Section 2, wepresent a general model. In Sections 3–5, the selection efficiencyof three tournament formats – contests, binary elimination, andround-robin tournaments – is analyzed. Specifically, in Section 3we provide a theory for the expected ability and expected rankof the winner in contests, and numerically illustrate our results.A theory for the expected ability of the winner in binary elimina-tion tournaments is developed in Section 4. In Sections 4 and 5,we provide the results of numerical simulations for binary elimina-tion and round-robin tournaments. Section 6 contains a discussionof our findings and concluding remarks.

2. The model

Let N ¼ f1; . . . ;Ng be a set of N alternatives (‘‘players”). Eachplayer i ¼ 1; . . . ;N is characterized by an attribute Xi 2 R (‘‘abil-ity”). Abilities X ¼ ðX1; . . . ;XNÞ are independently and identicallydistributed (i.i.d.) with a probability density function (pdf) f(x)and the corresponding cumulative density function (cdf) F(x). Itis assumed, for simplicity, that f(x) is continuous on its support,and has a finite second moment.

The selection problem is to identify the ‘‘best” player, i.e. theplayer with the highest ability. Albeit straightforward when abili-ties Xi are directly observable, selection becomes nontrivial whenthe abilities are perturbed by noise. In this case, a selection schemehas to be employed that can only give the ‘‘right” answer withsome probability less than one.

We consider a special class of selection schemes we call tourna-ments, which use ordinal comparisons of perturbed abilities toidentify the best player as the ‘‘winner.” Specifically, we considerthree prominent tournament formats: contests, binary eliminationtournaments, and round-robin tournaments.

A tournament selection scheme can involve one or severalstages. At each stage t, player i’s output, Yit , is her perturbed ability:Yit ¼ Xi þ �it , where �it are zero-mean, i.i.d. across players andacross stages, with a symmetric pdf /ð�Þ, cdf Uð�Þ, and a finite sec-ond moment. The overall level of noise can be characterized byparameter r2, the variance of �it . According to the tournamentscheme used, output levels Yit are compared, and the winner ofthe tournament, player iw, is determined. Let Xw � Xiw denote the

D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675 669

winner’s ability, and Rw – the winner’s rank (i.e. the rank order ofXw in the set of abilities fX1; . . . ;XNg, with rank 1 correspondingto the highest ability and rank N to the lowest ability).

In this paper, we focus on two statistical properties of iw: the ex-pected ability of the winner, s ¼ EðXwÞ, and the expected rank ofthe winner, r ¼ EðRwÞ. Each can serve as an efficiency criterion fora tournament selection scheme. In what follows, we explore hows and r depend on the number of players, N, and the properties ofthe distributions of players’ abilities, f ð�Þ, and noise, /ð�Þ, for thethree tournament formats.

3. Contests

In contests, players’ perturbed abilities are observed only once,and the player with the highest perturbed ability is the winner. LetYi ¼ Xi þ �i be the players’ output levels. The winner of the contestis player iw ¼ arg max16j6NYj.

Consider a contest of N players with fixed abilities X ¼ x ¼ðx1; . . . ; xNÞ. The pdf of Yi conditional on Xi ¼ xi is /ðy� xiÞ. Theprobability for player i to win the contest conditional on X ¼ x is

qiðxÞ ¼ PrfYi P Yj for all j – ijX ¼ xg ¼ PrfZj P 0for all j – ijX ¼ xg; ð1Þ

where for a given i new variables Zj are defined as Zj ¼ Yi�Yjð1� dijÞ, with the inverse transformation being Yj ¼ Zi � Zjð1�dijÞ. Here dij is the Kronecker delta-symbol.

The joint pdf of Z ¼ ðZ1; . . . ; ZNÞ conditional on X ¼ x is/ðzi � xiÞ

Qj – i/ðzi � zj � xjÞ. Eq. (1) then gives

qiðxÞ ¼Z

dzi/ðzi � xiÞYj – i

Z 1

0dzj/ðzi � zj � xjÞ

¼Z

dz/ðz� xiÞYj – i

Uðz� xjÞ: ð2Þ

3.1. The expected ability of the winner

The expected ability of the winner conditional on X ¼ x isEðXwjX ¼ xÞ ¼

PiqiðxÞxi. The unconditional expected ability of

the winner s ¼ EðXwÞ can be found by integration over the joint dis-tribution of X, using Eq. (2):

s ¼Z

dFðx1Þ � � �Z

dFðxNÞEðXwjX ¼ xÞ

¼X

i

ZdzZ

dFðxiÞxi/ðz� xiÞYj–i

ZdFðxjÞUðz� xjÞ: ð3Þ

In Eq. (3), all N terms in the sum are the same, and within each termi, integration over dFðxjÞ produces the same expression for all j – i.Finally, we obtain,

s ¼Z

dx xfwðxÞ; f wðxÞ ¼ Nf ðxÞZ

dz/ðz� xÞZ

dFðtÞUðz� tÞ� �N�1

:

ð4Þ

Eq. (4) gives, in a closed form, the expected ability of the winnerand the distribution of the winner’s ability, fwðxÞ. We use FwðxÞ todenote the corresponding cdf. In the absence of noise ðr! 0Þ, with/ð�Þ ¼ dð�Þ and Uð�Þ ¼ Ið�P 0Þ, Eq. (4) gives fwðxÞ ¼ Nf ðxÞFðxÞN�1,which is the pdf of the Nth order statistic of f(x). Here, dð�Þ is theDirac delta-function, and Ið�Þ is the indicator function. In the oppo-site limit of large noise ðr!1Þ we have /ðz� xÞ � /ðzÞ andUðz� tÞ � UðzÞ under the integrals in Eq. (4), which givesfwðxÞ ¼ f ðxÞ. Thus, with increasing noise intensity the distribution

of the winner’s ability changes from the pdf of the Nth order statis-tic for small noise to the original distribution of abilities for largenoise.

For the dependence of fwðxÞ on N, the following property holds(all proofs are relegated to the Appendix A).

Proposition 3.1. Let XðnÞw denote the winner’s ability in a contest of nplayers. Then XðnÞw first-order stochastically dominates XðmÞw for n > m.

Corollary 3.1. The expected ability of the winner s in contestsincreases monotonically with N.

The properties of fwðxÞ are illustrated in Fig. 1, which shows thecdfs of the winner’s ability for different values of N for a fixedr ðr ¼ 1, left panel), and for different values of r for a fixedN (N = 10, right panel). The Figure is obtained using Eq. (4) withthe normal distribution of abilities f(x) with variance r2

X ¼ 1, andthe normal distribution of noise. The left panel illustrates thestochastic dominance result of Proposition 3.1. In the rightpanel, the original distribution of abilities and the distribution ofthe Nth order statistic are shown to illustrate how FwðxÞ changeswith r.

3.2. The expected rank of the winner

Assume, without loss of generality, that the players are rankedso that X1 P X2 P � � �P XN , then iw introduced earlier identifiesthe rank of the winner, and ability Xi can be thought of as theðN þ 1� iÞth order statistic of f(x).

The probability for a player ranked i to win the contest is

qi ¼ N!

Z 1

�1dFðx1Þ

Z x1

�1dFðx2Þ � � �

Z xN�1

�1dFðxNÞqiðxÞ; ð5Þ

with qiðxÞ given by Eq. (2). Note that we can change the order ofintegration in Eq. (5) and re-write it for each i in the form

qi ¼ N!

Z 1

�1dFðxiÞ

Z 1

xi

dFðxi�1ÞZ 1

xi�1

dFðxi�2Þ � � �Z 1

x2

dFðx1Þ

�Z xi

�1dFðxiþ1Þ

Z xiþ1

�1dFðxiþ2Þ � � �

Z xN�1

�1dFðxNÞqiðxÞ: ð6Þ

The probability for a player ranked i to win the contest thenbecomes

qi ¼N!

ðN � iÞ!ði� 1Þ!

ZdzZ

dFðxÞ/ðz� xÞ

�Z x

�1dFðtÞUðz� tÞ

� �N�i Z 1

xdFðtÞUðz� tÞ

� �i�1

: ð7Þ

The expected rank of the winner is r ¼P

iiqi. Using Eq. (7), it can bewritten as

r ¼ 1þ NðN � 1ÞZ

dzZ

dFðxÞ/ðz� xÞ

�Z 1

xdFðtÞUðz� tÞ

ZdFðtÞUðz� tÞ

� �N�2

: ð8Þ

In the absence of noise ðr! 0Þ, with /ð�Þ ¼ dð�Þ andUð�Þ ¼ Ið�P 0Þ, Eq. (8) immediately gives r = 1. In the opposite lim-it of large noise ðr!1Þ, with /ðz� xÞ � /ðzÞ and Uðz� tÞ � UðzÞ,Eq. (8) gives r ¼ ðN þ 1Þ=2, the rank of the median player.

The dependence of r on the number of players N is nontrivial. Itwas shown previously (Ryvkin and Ortmann, 2008) that q1, theprobability of player ranked 1 winning, tends to zero (one) asN !1 for narrow-tailed (fat-tailed) distributions of abilities f(x).The distinction between the two types of distributions is basedon the limit km ¼ limx!1f ðxþ mÞ=f ðxÞ for some m > 0. A distributionis characterized as fat-tailed if km ¼ 1, and as narrow-tailed if

Fig. 1. Left: The cdf of the winner’s ability, FwðxÞ, for a contest of N ¼ 4; 16;64 and 128 players. The distribution of noise, /ð�Þ, is normal with variance r2 ¼ 1. Right: The cdf ofthe winner’s ability for a contest of N = 10 players and noise intensity r ¼ 0:1;1 and 5. The cdf of the Nth order statistic for N = 10 is shown with empty squares. The originalcdf of abilities (normal, with r2

X ¼ 1Þ is shown with solid squares in both panels.

670 D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675

km ¼ 0. The exponential distribution, f ðxÞ ¼ e�x, is the boundarycase for which km ¼ e�m > 0 is a function of m.

As a consequence, for fat-tailed distributions r ! 1 as N !1.Since r = 1 also for N = 1, we conclude that r should exhibit a non-monotonic dependence on N for fat-tailed distributions f(x).

Proposition 3.2. Suppose that (i) for all m > 0; limx!1½f ðxþ mÞ=f ðxÞ� ¼ 1;(ii) for all m > 0; limx!1½f 0ðxþ mÞ=f ðxÞ� ¼ 0. Then limN!1r ¼ 1, i.e. ris nonmonotonic in N.

3.3. A numerical illustration

To illustrate our findings, we calculated the expected ability ofthe winner, s, and the expected rank of the winner, r, for contestsusing three distributions of players’ abilities f (x): the normal, theexponential, and the Pareto distribution. The normal distributionhas a narrow tail, and is an empirically important benchmark.The exponential distribution is the boundary case between nar-row-tailed and fat-tailed distributions. Finally, the Pareto distribu-tion is fat-tailed, and has been documented to describe highlyselected populations, such as the distribution of students’ gradesin top universities, or the distribution of the number of citationsof journal articles (see, e.g., Reed, 2001).

For noise, we used the zero-mean normal distribution charac-terized by variance r2, the noise intensity parameter. The parame-ters of all the ability distributions have been chosen to normalizethe variance of ability to one. Thus, by varying r from 0.1 to 10,we covered the small-noise ðr� 1Þ, intermediate-noise ðr 1Þ,and large-noise ðr 1Þ regimes. The small-noise (large-noise) re-gime corresponds to large (small) signal-to-noise ratio (SNR), whennoise is small (large) compared to the variation in players’ abilities.The intermediate-noise regime describes the case when the SNR is

Fig. 2. The normalized expected ability of the winner in a contest, ~s, as a function of noiseThe curves correspond to the values of N ¼ 2;4;8; . . . ;256.

of order one; it is the most interesting from the inference point ofview.

Let ~s denote the normalized expected ability of the winner,~s ¼ ðs� lXÞ=rX , with lX ¼ EðXiÞ being the expected ability, andrX ¼ 1 – the standard deviation of Xi. Fig. 2 shows the normalizedexpected ability of the winner in a contest, ~s, as a function of noiseintensity, r, for the three ability distributions. Different curves inthe figure correspond to the values of the number of playersN ¼ 2;4;8; . . . ;256, i.e. the powers of 2 from 1 to 8. Each curve isobtained by smoothly connecting discrete points calculated usingEq. (4) for the values of r ¼ 0:1;0:2; . . . ;2:0;3;4;5 and 10.

As seen from Fig. 2, and predicted above, the expected ability ofthe winner decreases monotonically with r for all values of N andability distributions considered. It starts with the expected value ofthe Nth order statistic of the corresponding distribution f(x) forr! 0, and tends towards lX in the large-noise regime.

It is also seen from Fig. 2 that the expected ability of the winnerincreases with N for all values of r and ability distributions consid-ered. This is a consequence of the first-order stochastic dominanceresult (Proposition 3.1).

The values of ~s depend significantly on the distribution of play-ers’ abilities. For the narrow-tailed normal distribution, ~s is of orderone and decays most rapidly with r, while for the fat-tailed Paretodistribution it significantly exceeds one in a wide range ofparameters.

Fig. 3 shows the expected rank of the winner in a contest, r, as afunction of the number of players, N, for the three ability distribu-tions. The curves in the Figure correspond to different values of rand are obtained by smoothly connecting discrete points calcu-lated using Eq. (8) for the same values of N as in Fig. 2. For betterexposition, the dependence of r on N is shown on the log2—log2

scale.

intensity r, for the normal, exponential, and Pareto distributions of players’ abilities.

Fig. 3. The expected rank of the winner in a contest, r, as a function of the number of players N, for the normal, exponential, and Pareto distributions of players’ abilitiesðlog2—log2 scale). The curves correspond to the values of r ¼ 0:1; 0:2; . . . ;2:0;3;4;5;10.

D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675 671

As seen from Fig. 3, the expected rank of the winner increasesmonotonically with r. At the same time, in accordance with thetheoretical prediction above, the dependence of r on N exhibitsnonmonotonicity for the Pareto distribution of abilities f(x). Inter-estingly, for the exponential distribution r tends to a constant asN increases. This is an illustration of the fact that the exponentialdistribution is the boundary case between narrow-tailed and fat-tailed distributions.

For narrow-tailed distributions (e.g., the normal distribution inFig. 3), r increases with N. Distributions with a finite support, suchas the uniform distribution, are a special case of narrow-taileddistributions.

4. Binary elimination tournaments

A binary elimination tournament is a format that initially re-quires N ¼ 2R players ðR ¼ 1;2; . . .Þ. The tournament consists of Rstages enumerated by t ¼ 1; . . . ;R. It is also convenient to introduce‘‘stage Rþ 1,” at which there is only one player left.

This selection format uses multiple two-player contests withelimination. At stage t, the players with the distribution of abilitiesftðxÞ are randomly matched in pairs. Within each pair, the rankingof perturbed abilities is observed, with the winner advancing toround t þ 1. The winner of stage R, the ‘‘final” match, is the overallwinner iw.

We can use Eq. (4) with N = 2 to write a recursion relation forthe distribution of players’ abilities ftðxÞ:

ftþ1ðxÞ ¼ 2f tðxÞZ

dz/ðz� xÞZ

dyUðz� yÞftðyÞ; f 1ðxÞ ¼ f ðxÞ: ð9Þ

As Eq. (9) shows, the distribution of abilities starts with f(x) at stage1 and then gets refined at each stage due to elimination. The distri-

Fig. 4. Left: The cdf of a player’s ability at stage 2 of a binary elimination tournament forshown with solid squares; the cdf of the Nth order statistic for N = 2 is shown with emptyN ¼ 2;4 and 8 players. Also shown is the original cdf of abilities FðxÞ � F1ðxÞ (normal w

bution of the winner’s ability will be fwðxÞ ¼ fRþ1ðxÞ, and the ex-pected ability of the winner s ¼

RxfwðxÞdx.

Proposition 4.1. Let XðtÞ denote the random variable correspondingto a player’s ability at stage t of a binary elimination tournament,distributed with pdf ftðxÞ. Then XðtÞ first-order stochastically domi-nates Xðt

0 Þ for t > t0.

Corollary 4.1. The expected ability of the winner s in binary elimina-tion tournaments increases monotonically with N.

Fig. 4 shows the relevant changes in the distribution of abilitiesdue to elimination. The normal distribution of abilities with r2

X ¼ 1was chosen for illustration. The Figure was obtained using Eq. (9).In the left panel, second-stage refinement of the distribution isshown for different values of noise level r. In the large-noise re-gime (illustrated by r ¼ 5Þ, the distribution practically does notchange after one round of elimination. In the small-noise regime(illustrated by r ¼ 0:1Þ, the distribution becomes that of the Nthorder statistic for N = 2.

The right panel in Fig. 4 shows how the distribution of the win-ner’s ability changes depending on the number of refinementstages, for a fixed noise level (an intermediate-noise level r ¼ 1was chosen for illustration). The distribution refinement is in linewith the stochastic dominance result of Proposition 4.1.

4.1. A numerical illustration

We used numerical simulations to calculate the expected abilityof the winner and the expected rank of the winner for the binaryelimination format. (For the expected ability of the winner, it ispossible, alternatively, to use Eq. (9).) We used the same set ofparameterizations as in Section 3.3 for contests: three distributions

different values of noise level r. The original cdf of abilities (normal, with r2X ¼ 1Þ is

squares. Right: The cdfs of the winner’s ability in a binary elimination tournament ofith r2

X ¼ 1Þ.

672 D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675

of players’ abilities (normal, exponential, and Pareto) with rX ¼ 1,noise levels r ¼ 0:1;0:2; . . . ;2:0;3;4;5;10, and N ¼ 2;4;8; . . . ;256.

For each constellation of parameters ðf ð�Þ;r;NÞ we simulatedK ¼ 105 realizations of the elimination tournament and recordedthe values of sk and rk, the ability and the ranking of the winner,for each realization k ¼ 1; . . . ;K . The expected ability and expectedrank of the winner then were consistently estimated as sampleaverages s ¼ K�1P

ksk and r ¼ K�1Pkrk. We also calculated sample

variances S2s ¼ ðK � 1Þ�1P

kðsk � sÞ2 and S2r ¼ ðK � 1Þ�1P

kðrk � rÞ2

to estimate standard errors rs ¼ K�1=2Ss and rr ¼ K�1=2Sr . The esti-mates of standard errors did not exceed 1%.

Figs. 5 and 6 show the results for the normalized expected abil-ity of the winner ~s ¼ ðs� lXÞ=rX and the expected rank of the win-ner, r. The curves are obtained by smoothly connecting discretepoints estimated through simulations. In Fig. 5, the normalized ex-pected ability of the winner is shown as a function of noise level rfor different values of the number of players N. As seen from theFigure, s decreases monotonically with r and increases monotoni-cally with N for all three ability distributions considered. Similarlyto contests (Section 3.3), there are significant differences in themagnitudes of the normalized expected ability ~s acrossdistributions.

In Fig. 6, the expected rank of the winner, r, is shown as a func-tion of the number of players, N, for different values of r. Similarlyto contests, r depends on N nonmonotonically for the Pareto distri-bution of abilities. This result parallels the earlier findings (Ryvkinand Ortmann, 2008) for q1, the probability of the best player win-ning. Indeed, if q1 ! 1 as N !1 for fat-tailed distributions, theexpected rank of the winner must tend to one as well. Note that,similarly to contests (see Section 3.3), for the exponential distribu-tion of abilities, r tends to a constant as N !1.

Fig. 5. The normalized expected ability of the winner in a binary elimination tournamdistributions of players’ abilities. The curves correspond to the values of N ¼ 2;4;8; . . . ;

Fig. 6. The expected rank of the winner in a binary elimination tournament, r, as a functioplayers’ abilities ðlog2—log2 scale). The curves correspond to the values of r ¼ 0:1;0:2; .

5. Round-robin tournaments

In a round-robin tournament, N players are matched pairwise inall possible M ¼ NðN � 1Þ=2 matchings. Each match (i,j) betweenplayers i and j is a two-player contest with two possible outcomes:player i wins, or player j wins. The score of player i is the number ofwins she has. The player with the maximal score is the winner ofthe tournament. There can be a number n P 1 of players withthe maximal score, in which case the winner is determined ran-domly among them with probability 1=n for each.

We used a simulation procedure and parameterizations similarto those discussed in Section 4.1 to calculate the expected ability ofthe winner and the expected rank of the winner in a round-robintournament. The results of the simulations are shown in Figs. 7and 8. In simulations, we generated K ¼ 105 realizations of around-robin tournament for each parameterization except for thecases of N = 128 and 256. For N so large, we only generatedK ¼ 104 realizations, due to computational time constraints. Thus,for the curves corresponding to N = 128 and 256 in Fig. 7 and theupper panels of Fig. 8, the accuracy of simulations is approximatelythree times worse than for all other curves, hence the artifactualsmall oscillations. Nevertheless, the sample standard deviationfor both s and r did not exceed 1% of the value even for those curves.

As seen From Fig. 7, the expected ability of the winner increasesmonotonically with N for all parameterizations considered. This isin line with the generally established results for contests and bin-ary elimination tournaments. It is also seen from the Figure thatthe dependence of s on the noise level r undergoes a bifurcationfor some critical value of N ¼ Nc: for N < Nc; s decreases monoton-ically with r, while for N > Nc it can exhibit nonmonotonicity as afunction of r in the range of intermediate-noise levels. The

ent, ~s, as a function of noise intensity r, for the normal, exponential, and Pareto256.

n of the number of players N, for the normal, exponential, and Pareto distributions of. . ;2:0;3;4;5;10.

Fig. 8. The expected rank of the winner in a round-robin tournament, r, as a function of the noise level r, and of the number of players N, for the normal, exponential, andPareto distributions of players’ abilities (log2—log2 scale). The curves in the upper row correspond to the values of N ¼ 2;4;8 . . . ;256. The curves in the lower row correspondto the values of r ¼ 0:1;0:2; . . . ;2:0;3;4;5;10.

Fig. 7. The normalized expected ability of the winner in a round-robin tournament, ~s, as a function of noise intensity r, for the normal, exponential, and Pareto distributionsof players’ abilities. The curves correspond to the values of N ¼ 2;4;8; . . . ;256.

D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675 673

upward-sloping part of the dependence is not very pronounced,but the vertical distance between the minimum and the maximumsignificantly exceeds the simulation accuracy. For example, for theexponential distribution of abilities, the distance between thebottom of the well and the top of the hump of the N = 256 curveis � 0:07, whereas the standard error of s in this region is �0.013.

It is also worth noticing that the magnitude of the winner’s ex-pected ability produced by the round-robin tournament is very dif-ferent from the one produced by contests or binary eliminationtournaments. For r! 0, all three formats lead to the same valuesof s corresponding to the expected value of the Nth order statisticsof f(x). However, as noise increases, s decays much slower for theround-robin format than for the other two formats (compare Figs.2, 5 and 7).

Fig. 8 shows the dependence of the expected rank of the winneron r and N. As seen from the upper panels, the dependence of r onr undergoes a bifurcation for some critical value of N similar tothat identified for the expected ability of winner. Here, the result-

ing nonmonotonic dependence of r on r is very pronounced, espe-cially for the exponential distribution of abilities.

The counterintuitive nonmonotonic dependence of s and r on rfor the round-robin format is reminiscent of the behavior of predic-tive power q1 (Ryvkin and Ortmann, 2008). We propose the follow-ing explanation. When the distribution of players’ abilities has adecaying upper tail, the players are affected differentially by an in-crease in the noise level. Noise increases the probability of upsets,but less so for players with higher abilities. In a round-robin tour-nament, winning is based on point counting. Thus, it is not neces-sary to win all matches to be the winner; one should only winmore matches than others. Therefore, if lower ranked player upsetone another more intensively, there is a chance that, as noise in-creases, higher ranked players will win more often. This mecha-nism works in a limited range of parameters (sufficiently large N,not too large rÞ.

The lower three panels in Fig. 8 show the dependence of ron N. The nonmonotonic dependence is observed for all three

674 D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675

distributions of abilities. This is consistent with the nonmonotonicbehavior of predictive power q1 (Ryvkin and Ortmann, 2008).

The magnitudes of r for the round-robin format are much lowerthan those for the other two formats (compare Figs. 3, 6, and 8).This shows, in line with the results of other authors (e.g., David,1959), that there is an overall efficiency gain from using a round-robin tournament as a selection mechanism. Of course, it is alsomore costly and not always institutionally admissible.

6. Discussion and concluding remarks

In this paper, we address the problem of selection of the bestalternative (‘‘player”) in a noisy environment. Acknowledging thatselection can possibly have strategic aspects (see Hvide andKristiansen, 2003), we abstract from those and focus entirely onthe efficiency of selection algorithms. This restricts the applicabil-ity of our findings to situations when the players’ unobserved attri-butes (‘‘abilities”) are revealed truthfully. (This is not to say thattrue abilities are directly observed. Rather, the players act accord-ing to their true abilities.) We argue that this, somewhat restrictedfrom a managerial perspective, class of situations still includespractically interesting cases, such as top-level recruitment and pro-motion tournaments, sporting events, R&D races, competition forpublic finance projects, and, more generally, competitive environ-ments with high stakes.

In this paper, we have analyzed the efficiency of three prominenttournament selection schemes corresponding to three tournamentformats: contests, binary elimination and round-robin tourna-ments. The contest format represents a class of simple one-shot for-mats and is an important benchmark. The binary elimination formatrepresents a class of dynamic formats with multiple matchings andgradual elimination of weaker players. The round-robin format rep-resents a class of formats using Condorcet-type point counting.

The contest and binary elimination formats have the features ofreal-life selection processes in organizations. The contest formatcorresponds to a simple one-stage interview, which is usual for rel-atively low-paid jobs. Hiring for professional positions typically oc-curs in several stages involving preliminary screening ofapplications followed by one or several rounds of interviews and/or office visits. At each stage, a significant portion of the applicantsare eliminated. The round-robin format is somewhat more exoticin application to hiring, albeit used extensively in sports.

For each of the three tournament formats, we calculated twocriteria of selection efficiency: the expected ability of the winner,s, and the expected rank of the winner, r. We explored the depen-dence of s and r on the number of players, N, the distribution ofplayers’ abilities in the population, and the level of noise, r.

Our first major finding is that s is increasing in the number ofplayers involved in the selection process. This result is rather intu-itive. For contests and binary elimination tournaments, we pro-vided a general proof [We note that the proof can be triviallyextended to arbitrary elimination tournaments, i.e. those involvingcontests of more than two players, and/or more than one winnerpromoted from each contest to the next stage (see, e.g., Gradsteinand Konrad, 1999; Fu and Lu, 2006)], whereas for round-robin tour-naments our conclusion relies on extensive numerical simulations.

Our second major finding is that r can be a nonmonotonic func-tion of N. Consequently, even though an increase in the number ofplayers unambiguously leads to an increase in the expected qualityof the winner, it may or may not do so at a cost of missing the bestcandidate. Specifically, for narrow-tailed distributions of players’abilities, r increases as a function of N. At the same time, for fat-tailed distributions of abilities, r depends on N nonmonotonically,and tends to one for sufficiently large N. Thus, by expanding thepool of potential candidates, the DM can reach a point where each

additional candidate increases both the expected quality of thewinner and the likelihood of the best player winning. We predictthat it should occur in highly selected populations characterizedby fat-tailed distributions of skills, such as graduates of top univer-sities, which is the relevant subject pool for professional jobs.

The round-robin tournament format is special in that r(N) canbe nonmonotonic even for narrow-tailed distributions providedthey have an infinite support (such as the normal distribution).Additionally, s and r can be nonmonotonic as a function of r. Wedo not emphasize these results per se because round-robin tourna-ments are not typically used for hiring. At the same time, we be-lieve that these results will survive a generalization to othersimilar formats using point counting. Our findings, thus, suggestthat such selection schemes can potentially be useful.

Practically it is, of course, impossible to have a distribution ofabilities with an infinite support, therefore, for sufficiently largeN, the expected rank of the winner will always be increasing inN. However, if the distribution has a decaying fat tail in a boundedregion (e.g., a truncated Pareto distribution), r(N) still can be adecreasing function in a wide range of not too large N, as long asthe expected value of the Nth order statistic of the ability distribu-tion is sufficiently far from the support’s upper bound.

Our findings have interesting implications for managerial deci-sions regarding recruitment strategies (Breaugh and Starke, 2000).Apart from the somewhat obvious difference in selection efficiencybetween simple contest-like schemes and more complex multi-stage elimination schemes, with the latter producing, on average,more qualified and higher ranked hires (albeit at a higher cost),we demonstrate the importance of the interplay between the num-ber of candidates and the distribution of skills. For populationswith fat-tailed distributions of skills, expanding the pool of candi-dates helps increase both the quality and the relative ranking of thehire. For populations with narrow-tailed distributions of skills,additional candidates improve the expected quality of the hirebut also increase the chance of missing the best candidate. Theshape of the distribution of skills and the noise level can also bechanged: for example, requiring an advanced degree or equivalentwork experience may serve as a stratifying device to decrease thelevel of noise and make the distribution of skills fat-tailed.

For practitioners, the problem of choosing the optimal tourna-ment format is also tied to costs. Ryvkin and Ortmann (2006) intro-duced two types of costs a tournament format entails: time costsand measurement costs. Both are related in a simple way to thetournament format and the number of players, N. The tournamentorganizer’s decision problem then can be discussed in terms ofchoosing the optimal number of players and tournament formatwith the objective function equal to selection efficiency less thecosts. As expected, for the most part, contests are the least costlyand the least accurate, and round-robin tournaments are the mostcostly and the most accurate. Thus, when marginal time and mea-surement costs are low, the round-robin format is preferred, whilewhen marginal costs are high enough, the contest format is pre-ferred, with the binary elimination format somewhere in between.

To conclude, our study suggests that tournament-based selec-tion is more efficient in some populations than in others, and helpsidentify the relevant population properties – the size of the pool,the distribution of skills, and the noise level – that can be manip-ulated by the decision maker. We also show that, depending onthe environment, more complex selection strategies may or maynot yield nontrivial efficiency gains.

Acknowledgments

I am grateful to three anonymous referees and the editor, Dr.Robert G. Dyson, for their valuable comments. Special thanks toAnton Tyutin.

D. Ryvkin / European Journal of Operational Research 206 (2010) 667–675 675

Appendix A. Proofs of propositions

Proof of Proposition 3.1. Let X1; . . . ;Xn be i.i.d. random vari-ables (‘‘abilities”); let also �1; . . . ; �n be i.i.d. random variables(‘‘noise terms”), and define Yi ¼ Xi þ �i for i ¼ 1; . . . ;n (‘‘outputlevels”). Further let wðnÞ ¼ arg max16j6nYj denote the index of the‘‘winner” (the player with the highest output level). We need toshow that for any x 2 R and any n > m; PrðXwðnÞ 6 xÞ 6PrðXwðmÞ 6 xÞ.

Fix a vector ðy1; . . . ; ynÞ 2 Rn, and let m < n. Then

An ¼ PrðXwðnÞ 6 xjY1 ¼ y1; . . . ; Yn ¼ ynÞ ¼ 1�Uðyarg maxfy1 ;...;yng � xÞ;

where Uð�Þ is the cdf of noise terms �i (a nondecreasing function).Clearly, yarg maxfy1 ;...;yng P yarg maxfy1 ;...;ymg whenever n > m, thereforeAn 6 Am. Integrating over ðy1; . . . ; ynÞ with the corresponding proba-bility density, we obtain the result. h

Proof of Proposition 3.2. It is sufficient to show thatlimN!1q1 ¼ 1. Let X1; . . . ;XN be i.i.d. random variables with pdf f(x). Without loss of generality, assume that X1 P Xk for all k > 1,i.e. X1 is the Nth order statistic. Further, let �1; . . . ; �N be i.i.d. ran-dom variables with pdf /ð�Þ, and define Yi ¼ Xi þ �i fori ¼ 1; . . . ;N. Recall that q1 is defined as q1 ¼ PrfY1 P Yk;8k > 1g.

Introduce the probability of Y1 being greater than Yk for all k > 1conditional on a realization of noise � ¼ ð�1; . . . ; �NÞ:

q1ð�Þ ¼ PrfY1 P Yk;8k > 1j�g ¼ PrfZk P mk;8k > 1j�g:

Here, the new variables are defined as Z1 ¼ X1; Zk ¼ X1 � Xk ðk >1Þ, and mk ¼ �k � �1 ðk > 1Þ. Given that X1 is the Nth order statistic,the joint pdf of X1; . . . ;XN is N

Qkf ðxkÞIðx1 P xkÞ, therefore the joint

pdf of Z1; . . . ; ZN is Nf ðz1ÞQ

k>1f ðz1 � zkÞIðzk P 0Þ. This gives

q1ð�Þ ¼ NZ

dFðz1ÞYk>1

Z f0;mkg1

maxdzkf ðz1 � zkÞ

¼ NZ

dFðzÞYk>1

Fðzþminf0;�mkgÞ:

Let mðNÞ ¼maxk>1mk, then

q1ð�ÞP NZ

dzf ðzÞFðz� mðNÞÞN�1 ¼ NZ

dzf ðzÞ f ðzþ mðNÞÞf ðzÞ FðzÞN�1

:

Integrating by parts, obtain, using condition (i) of the proposition,

q1ð�ÞP 1�Z

dzddz

f ðzþ mðNÞÞf ðzÞ

� �FðzÞN :

Let E� denote the averaging operator over the realizations of vector�. This gives

q1 P 1� E�

Zdz

ddz

f ðzþ mðNÞÞf ðzÞ

� �FðzÞN:

For N !1, the second term on the right-hand side tends to zeroprovided f ðzþ mðNÞÞ=f ðzÞ is sufficiently flat at z!1 (ensured bycondition (ii) of the proposition), and we obtain the result. h

Proof of Proposition 4.1. It is sufficient to prove that Xðtþ1Þ

FOSD XðtÞ for any t. The result then follows by transitivity of theFOSD relation. But Xðtþ1Þ is related to XðtÞ in the same way as Xð2Þ

is related to Xi, which is the same as the way in which Xwð2Þ is re-lated to Xwð1Þ from Proposition 3.1. The result, therefore, followsfrom Proposition 3.1. h

References

Appleton, D., 1995. May the best man win? The Statistician 4, 529–538.

Assaf, D., Samuel-Cahn, E., 1996. The secretary problem: Minimizing the expectedrank with i.i.d. random variables. Advances in Applied Probability 28, 828–852.

Baucells, M., Carrasco, J., Hogarth, R., 2008. Cumulative dominance and heuristicperformance in binary multi-attribute choice. Operations Research 56, 1289–1304.

Bearden, J., Murphy, R., 2007. On generalized secretary problems. In: Abdellaoui, M.,Luce, R.D., Machina, M.J., Munier, B. (Eds.), Risk and Uncertainty: Mental, Formaland Experimental Representations. Springer, New York, pp. 187–206.

Bearden, J., Rapoport, A., Murphy, R., 2006. Sequential observation and selectionwith rank-dependent payoffs: An experimental study. Management Science 52,1437–1449.

Breaugh, J., Starke, M., 2000. Research on employee recruitment: So many studies,so many remaining questions. Journal of Management 26, 405–434.

David, H., 1959. Tournaments and paired comparisons. Biometrika 46, 139–149.David, H., 1988. The Method of Paired Comparisons. Oxford Press, New York.Devaro, J., 2006. Strategic promotion tournaments and worker performance.

Strategic Management Journal 27, 721–740.Fu, Q., Lu, J., 2006. The Optimal Multi-stage Contest. MPRA Paper 946, University

Library of Munich, Germany.Gerchak, Y., He, Q.-M., 2003. When will the range of prizes in tournaments increase

in the noise or the number of players? International Journal of Game Theory 5,151–165.

Gibbons, J., Olkin, I., Sobel, M., 1977. Selecting and Ordering Populations: A NewStatistical Methodology. Wiley, New York.

Glenn, W., 1960. A comparison of the effectiveness of tournaments. Biometrika 47,253–262.

Gradstein, M., Konrad, K., 1999. Orchestrating rent-seeking contests. The EconomicJournal 109, 536–545.

Harary, E., Moser, L., 1966. The theory of round robin tournaments. AmericanMathematical Monthly 73, 231–246.

Hartigan, J., 1968. Inference from a knockout tournament. The Annals ofMathematical Statistics 39, 583–592.

Horen, J., Reizman, R., 1985. Comparing draws for single elimination tournaments.Operations Research 33, 249–262.

Hvide, H., Kristiansen, E., 2003. Risk taking in selection contests. Games andEconomic Behavior 42, 172–179.

Hwang, F., 1982. New concepts in seeding knockout tournaments. AmericanMathematical Monthly 89, 235–239.

Israel, R., 1981. Stronger players need not win more knockout tournaments. Journalof American Statistical Association 76, 950–951.

Knuth, D., 1987. Random knockout tournament: Problem 86-2. SIAM Review 29,127–129.

Leskinen, P., Kangas, A., Kangas, J., 2004. Rank-based modelling of preferences inmulti-criteria decision-making. European Journal of Operational Research 158,721–733.

Lockard, A., Tullock, G. (Eds.), 2001. Efficient Rent-seeking: Chronicle of anIntellectual Quagmire. Kluwer Academic Publishers, Boston.

Marchand, E., 2002. On the comparison between standard and random knockouttournaments. Journal of the Royal Statistical Society. Series D (The Statistician)51, 169–178.

McGarry, T., Schutz, R., 1997. Efficacy of traditional sport tournament structures.Journal of Operational Research Society 48, 65–74.

Mendonça, D., Raghavachari, M., 2000. Comparing the efficacy of ranking methodsfor multiple round-robin tournaments. European Journal of OperationalResearch 123, 593–605.

Narayana, T., 1979. Lattice Path Combinatorics with Statistical Applications.University Toronto Press, Toronto.

O’Flaherty, B., Siow, A., 1995. Up-or-out rules in the market for lawyers. Journal ofLabor Economics 13, 709–735.

Orrison, A., Schotter, A., Weigelt, K., 2004. Multiperson tournaments: Anexperimental examination. Management Science 50, 268–279.

Prendergast, C., 1999. The provision of incentives in firms. Journal of EconomicLiterature 37, 7–63.

Reed, W., 2001. The Pareto, Zipf, and other power laws. Economics Letters 74, 15–19.

Rosen, S., 1986. Prizes and incentives in elimination tournaments. The AmericanEconomic Review 76, 701–715.

Rubinstein, A., 1980. Ranking the participants in a tournament. SIAM Journal ofApplied Mathematics 38, 108–111.

Ryvkin, D., Ortmann, A., 2006. Three Prominent Tournament Formats: PredictivePower and Costs. Working Paper Available at <http://ssrn.com/abstract=1132408>.

Ryvkin, D., Ortmann, A., 2008. The predictive power of three prominent tournamentformats. Management Science 54, 492–504.

Searls, D., 1963. On the probability of winning with different tournamentprocedures. Journal of the American Statistical Association 58, 1064–1081.

Szymanski, S., 2003. The economic design of sporting contests: A review. Journal ofEconomic Literature 41, 1137–1187.

Taylor, C., 1995. Digging for golden carrots: An analysis of research tournaments.The American Economic Review 85, 872–890.