NOTE

Neutrality Tests for Sequences with Missing DataLuca Ferretti,*,1,2 Emanuele Raineri,†,2 and Sebastian Ramos-Onsins*

*Centre for Research in Agricultural Genomics, 08193 Bellaterra, Spain and †Centro Nacional de AnálisisGenómico, 08028 Barcelona, Spain

ABSTRACTMissing data are common in DNA sequences obtained through high-throughput sequencing. Furthermore, samples of lowquality or problems in the experimental protocol often cause a loss of data even with traditional sequencing technologies. Here wepropose modified estimators of variability and neutrality tests that can be naturally applied to sequences with missing data, without theneed to remove bases or individuals from the analysis. Modified statistics include the Watterson estimator uW, Tajima’s D, Fay and Wu’sH, and HKA. We develop a general framework to take missing data into account in frequency spectrum-based neutrality tests andwe derive the exact expression for the variance of these statistics under the neutral model. The neutrality tests proposed here can alsobe used as summary statistics to describe the information contained in other classes of data like DNA microarrays.

NEUTRALITY tests are among the most widely used toolsin population genetics. Many neutrality tests have been

developed based on the levels and the patterns extractedfrom segregating sites, and in particular to be applied tobiallelic SNP data. The simplest information that can beextracted from SNP data are the allele frequency spectrum;therefore, many tests focus on the difference between theobserved and expected spectrum under the neutral Wright–Fisher model. Widespread tests of this kind include Tajima’sD (Tajima 1989), Fu and Li’s F and D (Fu and Li 1993), andFay and Wu’s H (Fay and Wu 2000). However, this class oftests is much larger, as recently shown by Achaz (2009)following an idea of Nawa and Tajima (2008), and includesamong the others the tests by Fu (1997), Zeng et al. (2006),and Achaz (2009). A subclass of optimal neutrality testsagainst specific alternative scenarios was described by Ferrettiet al. (2010). Some general results on the variances of thesetests were provided by Fu (1995) and Pluzhnikov andDonnelly (1996).

All these statistics assume a complete knowledge of thealleles present in the n sequenced individuals for all theL positions genotyped. However, this is rarely the case: ex-

perimental problems in sample preparation or genotypingoften result in missing data; i.e., some individual alleles atsome positions are actually unknown.

At present, most packages for population genetics ana-lyses like DNAsp (Librado and Rozas 2009) deal with missingdata simply by removing individuals and/or positions af-fected with incomplete data. This is a good strategy as longas missing data represent a very minor fraction of the alleles,since in this case they do not affect the power of the anal-ysis. However, there could be situations in which a largeamount of missing data are unavoidable. For example, insamples taken from natural populations the quality of thesamples could be low or the amount of DNA available perindividual could be insufficient; therefore, genotyping thesesamples could miss a significant fraction of the alleles.

There is another important reason to consider sequenceswith missing data. Many of the sequences that are beingproduced currently are not obtained through Sanger se-quencing, but from next-generation sequencing (NGS) tech-nologies. These technologies sequence a large amount ofshort reads that are then realigned to reconstruct theoriginal sequence. The coverage of these reads is stronglyinhomogeneous along the genome and there is often a largefraction of bases that is not covered by a sufficient number ofreads, unless the coverage is very high. Missing data aretherefore inherent to these technologies: hence, removingindividuals or bases with missing alleles would imply a hugeloss of information. Given the growing relevance of NGStechnologies for population genetics studies, a differentstrategy is needed to deal with this circumstance. Several

Copyright © 2012 by the Genetics Society of Americadoi: 10.1534/genetics.112.139949Manuscript received February 22, 2012; accepted for publication May 24, 2012Supporting information is available online at http://www.genetics.org/content/suppl/2012/06/01/genetics.112.139949.DC1.1Corresponding author: Centre de Recerca en Agrigenòmica (CRAG), CampusUniversitat Autònoma de Barcelona, 08193 Bellaterra, Spain.E-mail: luca.ferretti@uab.cat

2These authors contributed equally to this work.

Genetics, Vol. 191, 1397–1401 August 2012 1397

estimators of variability can be applied directly to sequencedreads (Lynch 2008; Hellmann et al. 2008; Jiang et al. 2009;Futschik and Schlötterer 2010; Kang and Marjoram 2011);however, no estimator is available for the sequences ob-tained after genotype call has been completed for each in-dividual in each position. The difference between thesetwo situations is that, once the genotype has been deter-mined, all the information about the single read bases align-ing on a given position and their qualities is (for our purposes)lost.

In this article we present a simple generalization of someestimators and tests that take missing data into account. Inparticular we consider the Watterson estimator of geneticvariability (Watterson 1975), the Tajima estimator of nucle-otide diversity (Tajima 1983), neutrality tests based on thefrequency spectrum like Tajima’s D and Fay and Wu’s H, andthe HKA test (Hudson et al. 1987) for neutral evolutionbased on the pattern of polymorphism and divergence.The most important result of this article is the general ex-pression for the covariance between the frequency spectrumat two sites CovðjiðxÞ; jjðyÞÞ, which is the basis for the com-putation of the variances of the estimators and tests pre-sented here.

Note that in sequence data, missing data (usually rep-resented by N’s located in the same position as the missingalleles) are not equivalent to gaps (represented by whitespaces). Gaps correspond to insertions or deletions (indels)in some of the sequences. In this article we do not addressindels, even if very short biallelic indels (a few bases long)are similar to SNPs as genetic variants and therefore couldbe analyzed by similar methods. In practice it is difficult todifferentiate indels from missing data if the rate of missingdata are high, and this is especially true for sequencesobtained from NGS data. Here we consider sequences with-out indels.

Neutrality Tests Including Missing Data

In this article we consider estimators and tests based on thefrequency spectrum. The basic population parameter in-volved in these tests is the nucleotide variability u = 2Nem,where Ne is the haploid effective population size and m is themutation rate per base per generation. We assume a smallvariability u 1 and a large window length L 1, such thatuL Oð1Þ.

All the tests and estimators belong to a general class ofneutrality tests that can be parametrized in terms of weightsvi, Vi (Achaz 2009),

u ¼ 1L

Xn21

i¼1

iviji;Xn21

i¼1

vi ¼ 1 (1)

T ¼ u2 u9

Varðu2 u9Þ ¼Pn21

i¼1 iViji

VarPn21

i¼1 iViji

; Xn21

i¼1

Vi ¼ 0; (2)

where ji indicates the number of variants with frequency ifor the derived allele. The weights vi, Vi multiply the nor-malized frequency spectrum ui = iji (Nawa and Tajima2008). The estimators are unbiased estimators of u, whilethe tests are normalized to have mean 0 and variance 1under the standard neutral model without recombination.

Our definition of neutrality tests based on the frequencyspectrum is the most general parametrization compatiblewith Equations 1–2 that takes explicitly into account thecoverage for each site. We denote by n the total numberof individuals sequenced and by nx the number of individu-als for which the allele at position x is known. The estima-tors and tests are defined as

u ¼ 1L

XLx¼1

Xnx21

i¼1

ivi;nxjiðxÞ;1L

XLx¼1

Xnx21

i¼1

vi;nx ¼ 1 (3)

T ¼ u2 u9

Varðu2 u9Þ

¼PL

x¼1Pnx21

i¼1 iVi;nxjiðxÞVarPL

x¼1Pnx21

i¼1 iVi;nxjiðxÞ; XL

x¼1

Xnx 21

i¼1

Vi;nx ¼ 0;

(4)

where ji(x) is an index variable that is 1 if there is a segre-gating site with i derived alleles in position x and 0 other-wise. The estimators are unbiased, i.e., EðuÞ ¼ u, while thetests are normalized to E(T) = 0, Var(T) = 1 as in the usualframework.

The weights vi;nx , Vi;nx define the specific estimatoror test (Achaz 2009). The most important estimator of var-iability is the Watterson estimator uW ¼ S=anL; where wedenote by S the total number of segregating sites andan ¼Pn21

i¼1 1=i. This estimator can be obtained as a maximumcomposite likelihood estimator (MCLE) (Hellmann et al.2008). Its natural generalization is the MCLE with missingdata,

vWi;nx

¼ 1iP L

x¼1anx=L⇒ uW ¼ SP L

x¼1anx

; (5)

which depends only on S. For most of the other estimatorslike Tajima’s P, we choose the weights vi;nx to be simply thesame as the weights vi for the estimators (1), where n issubstituted by nx, that is, vP

i;nx ¼ 2ðnx2iÞ=nxðnx21Þ. Thisdefinition is equivalent to uP ¼ P=L, where P is the averagepairwise diversity per base, which is naturally defined evenwith inhomogeneous coverage.

As for neutrality tests, Tajima’s D (Tajima 1989) corre-sponds to uP2uW, that is, to the weights

VDi;nx

¼ 2ðnx 2 iÞnxðnx 2 1Þ2

1iP L

y¼1any=L; (6)

and Fay and Wu’s H (Fay and Wu 2000) corresponds to

1398 L. Ferretti et al.

VHi;nx

¼ 2ðnx 2 iÞnxðnx 2 1Þ2

2inxðnx 2 1Þ: (7)

The other tests can be generalized in a similar way. All thetests and estimators reduce to their usual expressions if nodata are missing; i.e., nx = n for all sites.

In this framework it is also possible to implement errorcorrections for error-prone data: for example, removingsingletons (Achaz 2008) is equivalent to the choice ofv1;nx ¼ vnx21;nx ¼ 0 and a rescaling of the other vi;nx tomatch the normalization in Equation 3. For NGS data, in-formation on base or SNP qualities is usually available; hence,a more refined error correction strategy consists in weightingeach SNP in Equations 3–4 by the probability that it has beencorrectly identified. A detailed treatment can be found inSupporting Information, File S1. For NGS data it is also usefulto filter out the bases with low coverage, i.e., the ones forwhich information from most individuals is missing. If weassume that the minimum number of individuals coveringreliable positions is nmin, this filter can be easily implementedby removing all positions with nx , nmin from the analysis.

To evaluate the tests (4), we need the variances in thedenominators. Our basic result for these variances (leavingout subleading terms in u and 1/L) is

Var

XLx¼1

Xnx21

i¼1

iVi;nx jiðxÞ!

¼XLx¼1

Xnx21

i¼1

iV2i;nx

u þXLx; y¼1x 6¼y

Xnx21

i¼1

Xny21

j¼1

ijVi;nxVj;nyCovjiðxÞ; jjðyÞ

(8)

since ji(x) are index variables with mean E(ji(x)) = u/i andji(x), jj(x) are mutually exclusive for i 6¼ j, so E(ji(x)jj(x)) = 0.The covariance Cov(ji(x), jj(y)) for the standard neutral modelwithout recombination is presented in the next section.

The HKA (Hudson, Kreitman, Aguadé) test (Hudson et al.1987) and the formulae for estimators and tests based onthe folded spectrum are treated in File S1.

Covariance of the Frequency Spectrumat Different Sites

Since ji(x), jj(y) are index variables, their covariance underthe standard neutral model without recombination is

CovjiðxÞ; jjðyÞ

¼ Pijðnx ;ny ;nxyÞ2u2

ij; (9)

where Pijðnx ;ny ;nxyÞ is the probability of observing SNPs of fre-quency i and j, nx and ny are the numbers of individuals withknown alleles at the two sites, and nxy is the number ofindividuals for which both alleles are known. This probabil-ity can be obtained as

Pijðnx ;ny;nxyÞ ¼Xnxþny2nxy21

k;l¼1

CSij;klðnx ;ny;nxyÞP

Sklðnxþny2nxyÞ

þ CEij;klðnx ;ny ;nxyÞP

Eklðnxþny2nxyÞ

; (10)

where PSklðnÞ and PEklðnÞ are the probabilities of shared (S) orexclusive (E) pairs of mutations of frequency k and l in ncomplete sequences. (We define a pair of mutations asshared if there are individuals with derived alleles in bothloci and as exclusive if no individual sequence contains boththe derived alleles.) The sum PSkl þ PEkl gives the probabilityfor complete sequences Pkl = u2(1/kl + skl), where thematrix skl is defined in Fu (1995, Equations 2–3).

The coefficients CS;Eij;klðnx ;ny ;nxyÞ represent the probabilities

that, given a pair of shared or exclusive mutations withfrequencies k and l in nx + ny 2 nxy complete sequences,i and j derived alleles are found among the nx, ny alleles inx and y, respectively, assuming that the nx, ny individuals(with nxy in common between the two sets) are randomlyextracted from the complete set of nx + ny 2 nxy individuals.The combinatorial formulae for these probabilities are

CSij;klðnx ;ny ;nxyÞ ¼

nx 2nxyl2 j

ny 2nxyk2 i

nx þ ny 2 nxyl; k2 l;nx þ ny 2nxy 2 k

Xminði;nx 2nxy ;k2 jÞ

kx¼maxði2nxy ;l2 jÞ

nxy

i2 kx

nx 2nxy þ j2 l

kx þ j2 l

k2 kx

j

(11)

if k$ l; otherwise use the identity CSij;klðnx ;ny ;nxyÞ ¼ CS

ji;lkðny ;nx ;nxyÞ,

CEij;klðnx ;ny ;nxyÞ ¼

nx 2 nxyl2 j

ny 2 nxyk2 i

nx þ ny 2 nxyk; l; nx þ ny 2 nxy 2 k2 l

Xminði;nx 2 nxyþj2 lÞ

kx¼maxð0;i2nxy ;k2 nyþjÞ

nxy

i2 kx

nx 2 nxy þ j2 l

kx

ny 2 kþ kx

j

;

(12)

where ð ab; c; a2b2c Þ is the multinomial coefficient a!=b!c!

ða2b2cÞ!. We define CSij;klðnx ;ny ;nxyÞ or C

Eij;klðnx ;ny ;nxyÞ to be zero

if there are negative arguments in the binomial or multino-mial coefficients in the above Equations 11 or 12.

The formulae for the probabilities PS;EklðnÞ can be obtainedby breaking the derivation of E(jkjl) by Fu (1995) into thecontributions from shared mutations (Fu 1995, Equations24 and 28) and exclusive mutations (Equations 25, 29,and 30):

PSklðnÞ ¼ u2dklbnðkÞ þ u2ð12 dklÞbnðminðk; lÞÞ2bnðminðk; lÞ þ 1Þ

2

(13)

PEklðnÞ ¼

8>>>>>><>>>>>>:

u21kl2

bnðkÞ2bnðkþ 1Þ þ bnðlÞ2bnðlþ 1Þ2

  for kþ l, n

u2an 2 akn2 k

þ an 2 aln2 l

þ bnðkÞ þ bnðlÞ2

      for kþ l ¼ n

0           for kþ l.n

(14)

Neutrality Tests with Missing Data 1399

with

an ¼Xn21

i¼1

1i; bnðiÞ ¼

2nðn2 iþ 1Þðn2 iÞ ðanþ1 2 aiÞ2 2

n2 i:

(15)

Some special cases of these formulae are treated in File S1,Figure S1, and Figure S2.

Finally, note that the computation of the variances re-quires an estimate of u and u

2. These estimates are usually

obtained by the method of moments (MM). In our approach,u is given by the Watterson estimator (5), while the MMestimate of u

2is given by

u2 ¼ S2 2 SPL

x¼1anx

2þPLx;y¼1  

Pnx21i¼1  

Pny21j¼1 Cov

jiðxÞ; jjðyÞ

u2:

(16)

Discussion

In this article we presented a general framework for esti-mators of variability and neutrality tests based on thefrequency spectrum that take into account missing data ina natural way. This is particularly interesting in the light ofsequences obtained from NGS data, since for these technol-ogies a relevant fraction of bases is often not sequenced orsequenced at very low read depth.

The approach discussed here is based on results that areconditional on the distribution of the missing data, assummarized by the distribution of all the triples (nx, ny,nxy). An effective way of implementing numerically theabove variances is to sample Ns random values of (nx, ny,nxy) from the empirical distribution and compute the cova-riances using only these values and then rescale the secondterm in Equation 8 by a factor L2/Ns.

The modifications presented in this article can be appliedto all estimators and tests included in the framework ofAchaz (2009) and represent, therefore, a complete tool withwhich to deal with missing data. However, it would be in-teresting to know the impact of the missing data on theperformance of the estimators and tests. If we fix the samplesize, an increase in the amount of missing data leads to anincrease in the variance of the estimators (Figure 1), as is tobe expected given that this is equivalent to loss of informa-tion. On the other hand, if the loss of information associatedwith missing data is compensated by sequencing more indi-viduals, the performances of the estimators actually increase(i.e., their variances decrease) with respect to completesequences with the same coverage (Figure 1). A similar ef-fect can be observed for neutrality tests (Figure 2). Theexplanation for this counterintuitive behavior lies in the factthat in the case of complete sequences, all individuals sharethe same genealogical tree at all positions, i.e., the same evo-lutionary history, while in this case different positions are cov-ered by different sets of individuals with partly independenthistories in the same population; therefore, the number ofavailable histories is actually larger and the variance is re-duced similarly to what happens with recombination. Ourresults imply that with the same amount of information perbase, missing data could improve the power of neutrality tests.

Figure 1 Variance of the Watterson estimator uW on a window of L =100 bases for u = 0.1. Computed by drawing out randomly Ns = 100triples (nx, ny, nxy) in two different ways. First, we fix the sample size n =20 and remove alleles randomly according to the probability pm of miss-ing an allele (solid blue line). In this case the number of individuals is fixedbut the actual depth may vary along the sequence. Second, the numberof individuals sequenced in each position is adjusted to keep the averagedepth constant at n(1 2 pm) ’ 20 (dashed green line) and then weremove alleles with probability pm. In this case the sample size is n ’20/(1 2 pm).

Figure 2 Variance of Tajima’s D (bottom line) and Fay and Wu’s H(top line) on a window of L = 100 bases for u = 0.1. Computed as inFigure 1 for fixed average depth n(1 2 pm) ’ 20. (Note that Fayand Wu’s H variance is divided by 4 to appear in scale with Tajima’sD variance.)

1400 L. Ferretti et al.

Acknowledgments

Work funded by grant CGL2009-09346 to S.R.O., grantAG2010-14822 to Miguel Pérez-Enciso, and Consolidergrant CSD2007-00036 “Centre for Research in Agrigeno-mics” (Ministerio de Ciencia e Innovación, Spain). S.R.O.is recipient of a Ramón y Cajal position (Ministerio de Cien-cia e Innovación, Spain). L.F. acknowledges support fromConsejo Superior de Investigaciones Científicas (Spain) underthe JAE-doc program.

Literature Cited

Achaz, G., 2008 Testing for neutrality in samples with sequencingerrors. Genetics 179: 1409.

Achaz, G., 2009 Frequency spectrum neutrality tests: one for alland all for one. Genetics 183: 249.

Fay, J., and C.-I. Wu, 2000 Hitchhiking under positive Darwinianselection. Genetics 155: 1405.

Ferretti, L., M. Perez-Enciso, and S. Ramos-Onsins, 2010 Optimalneutrality tests based on the frequency spectrum. Genetics 186:353.

Fu, Y., and W.-H. Li, 1993 Statistical tests of neutrality of muta-tions. Genetics 133: 693.

Fu, Y.-X., 1995 Statistical properties of segregating sites. Theor.Popul. Biol. 48: 172–197.

Fu, Y.-X., 1997 Statistical tests of neutrality of mutations againstpopulation growth, hitchhiking and background selection. Ge-netics 147: 915.

Futschik, A., and C. Schlötterer, 2010 The next generation of mo-lecular markers from massively parallel sequencing of pooleddna samples. Genetics 186: 207.

Hellmann, I., Y. Mang, Z. Gu, P. Li, M. Francisco et al.,2008 Population genetic analysis of shotgun assemblies of ge-nomic sequences from multiple individuals. Genome Res. 18:1020–1029.

Hudson, R., M. Kreitman, and M. Aguadé, 1987 A test of neutralmolecular evolution based on nucleotide data. Genetics 116:153.

Jiang, R., S. Tavaré, and P. Marjoram, 2009 Population geneticinference from resequencing data. Genetics 181: 187.

Kang, C., and P. Marjoram, 2011 Inference of population muta-tion rate and detection of segregating sites from next-generationsequence data. Genetics 189: 595–605.

Librado, P., and J. Rozas, 2009 DnaSP v5: a software for compre-hensive analysis of DNA polymorphism data. Bioinformatics 25:1451.

Lynch, M., 2008 Estimation of nucleotide diversity, disequilibriumcoefficients, and mutation rates from high-coverage genome-sequencing projects. Mol. Biol. Evol. 25: 2409–2419.

Nawa, N., and F. Tajima, 2008 Simple method for analyzing thepattern of dna polymorphism and its application to snp data ofhuman. Genes Genet. Syst. 83: 353–360.

Pluzhnikov, A., and P. Donnelly, 1996 Optimal sequencing strate-gies for surveying molecular genetic diversity. Genetics 144:1247.

Tajima, F., 1983 Evolutionary relationship of DNA sequences infinite populations. Genetics 105: 437.

Tajima, F., 1989 Statistical method for testing the neutral muta-tion hypothesis by DNA polymorphism. Genetics 123: 585.

Watterson, G., 1975 On the number of segregating sites in genet-ical models without recombination. Theor. Popul. Biol. 7: 256.

Zeng, K., Y.-X. Fu, S. Shi, and C.-I. Wu, 2006 Statistical tests fordetecting positive selection by utilizing high-frequency variants.Genetics 174: 1431–1439.

Communicating editor: N. A. Rosenberg

Neutrality Tests with Missing Data 1401

GENETICSSupporting Information

http://www.genetics.org/content/suppl/2012/06/01/genetics.112.139949.DC1

Neutrality Tests for Sequences with Missing DataLuca Ferretti, Emanuele Raineri, and Sebastian Ramos-Onsins

Copyright © 2012 by the Genetics Society of AmericaDOI: 10.1534/genetics.112.139949

FILE S1

SUPPORTING INFORMATION

General framework for tests with missing data:

The general framework proposed by ACHAZ (2009) for estimators θ and neutrality tests T based on the frequency

spectrum ξi is based on these assumptions:

1. the estimator/test statistics is a linear function of the frequency spectrum ξi and a general function of the vari-

ability θ;

2. the expected value of the statistics under the standard neutral model (SNM) is E(θ|θ) = θ for the estimators and

E(T |θ) = 0 for the tests;

3. the tests are normalized such that their variance under the SNM without recombination is Var(T |θ) = 1.

Finally, the actual values of θ, θ2 in the statistics are estimated from the Watterson estimator and the MM estimator for

θ2. It is easy to check that these conditions imply the equations (1), (2) for general estimators and tests.

In the framework of sequences with missing data, the estimators and tests should be actually based on the site

frequency spectrum ξi(x). We propose a set of assumptions which is a slight generalization of the one above:

1. the estimator/test statistics is a linear function of the frequency spectrum ξi(x) and a general function of the

variability θ;

2. the expected value of the statistics under the standard neutral model (SNM) is E(θ|θ) = θ for the estimators and

E(T |θ) = 0 for the tests;

3. the tests are normalized such that their variance under the SNM without recombination is Var(T |θ) = 1;

4. the relative weight of the site frequency spectrum ξi(x) for a given position x depends on local information only.

Assumption 4 is not compulsory (in fact, more general tests can be obtained), but it helps to reduce considerably the

complexity of the class of tests without a sensible reduction of their power. The assumptions 1-4 imply immediately

the general form of equations (3), (4) for estimators and tests.

Accounting for base/SNP calling errors in sequences from NGS:

Sequences called from NGS data could contain a relatively high number of incorrectly called bases. As a result

of these errors, false SNPs could appear and affect the statistics. (In principle, these base errors could change SNP

frequencies or avoid detection of true SNPs; however, the fraction of SNPs in a sequence is generally low enough that

these effects are rare and not relevant.)

Ferretti, Raineri and Ramos-Onsins 2 SI

Depending on the way the sequences have been obtained, two kind of quality data could be available: base qualities

(often available when all sequences have been called separately) and SNP qualities (available as an output of SNP

callers). These qualities are actually given in terms of error probabilities; for example, if the qualities are Phred scaled,

the error probability is 10−quality/10, so quality 10 means error probability 0.1, quality 20 means error probability

0.01, quality 30 means error probability 0.001, etc.

We assume that all the sites are biallelic (this can be done by SNP calling, or by taking only the two most abundant

alleles, or the two alleles with lowest product of base error probabilities). For each position x where multiples alleles

are present in the data, we want to obtain the probability of true SNP pSNP (x). The way to do it depends on the

available data:

• SNP qualities: pSNP (x) is simply 1 minus the SNP error probability;

• base qualities: for each allele in position x compute the product of the base error probabilities, then take pSNP (x)

to be 1 minus the higher of the two products.

Once pSNP (x) has been obtained, the estimators and tests can be corrected for sequencing errors as follows:

θ =1

L

L∑x=1

nx−1∑i=1

iωi,nxpSNP (x)ξi(x) (SI-1)

T =

∑Lx=1

∑nx−1i=1 iΩi,nxpSNP (x)ξi(x)

Var(∑L

x=1

∑nx−1i=1 iΩi,nx

ξi(x)) (SI-2)

where in the denominator of T we neglect terms of order pSNP (1− pSNP ) since we assume pSNP ' 1.

HKA test with missing data:

We propose also a modified version of the HKA test (HUDSON et al. 1987) that deals with missing data. The HKA

test is a widely used multi-locus test for neutral sequence evolution, based on the statistics

X2 =∑l

(Sl − E(Sl))2

Var(Sl)+∑l

(S′l − E(S′l))2

Var(S′l)+∑l

(Dl − E(Dl))2

Var(Dl)(SI-3)

which has an approximate χ2 distribution in the neutral case. In the above equation, Dl is the divergence between

the two species for the lth locus (i.e. the number of fixed differences) and Sl, S′l denote the numbers of segregating

sites of the two species. This statistics can be applied to incomplete sequences by substituting the correct values

for E(S), E(D), Var(S) and Var(D). For sequences with missing data, E(S) = θ∑Lx=1 anx while Var(S) =

Var(θW

)(∑Lx=1 anx

)2. The expected value and variance of the divergence are given by the standard formulae,

taking into account that sites with no coverage in one or both populations must be discarded and do not count in E(D)

or Var(D).

Ferretti, Raineri and Ramos-Onsins 3 SI

The variance of the Watterson estimator Var(θW

), as well as the variance of all the estimators (3), can be obtained

from equation (8) by substituting Ωi,nx with ωi,nx/L. In particular, Var(θW

)is given by

Var(θW

)=

θ∑Lx=1 anx

+1(∑L

x=1 anx

)2 L∑x,y=1

nx−1∑i=1

ny−1∑j=1

Cov(ξi(x), ξj(y)) (SI-4)

Covariance formulae - special cases:

There are two special cases of the formulae for the covariances (10-14) given in the Main Text. The first case occurs

when the allele in y is known for all individuals with known allele in x, i.e. ny = nxy . In this case Pij(nx,ny,nxy)

reduces to a simpler expression in terms of an hypergeometric distribution:

Pij(nx,nxy,nxy) =

nx−nxy+j∑l=j

Pil(nx)

(nx−nxy

l−j)(nxy

j

)(nx

l

) (SI-5)

where Pil(nx) = θ2(1/il + σil) is the probability obtained by FU (1995) for nx complete sequences. The second

special case corresponds to nxy = 0, i.e. there are no individuals for which both alleles at x and y are known. In this

case both CS and CE reduce to generalized hypergeometric distributions:

CSij,kl(nx,ny,0)=

(nx

l−j,i+j−l,nx−i)(

ny

j,k−i−j,ny−k+i)(

nx+ny

l,k−l,nx+ny−k) , k ≥ l (SI-6)

CEij,kl(nx,ny,0)=

(nx

i,l−j,nx−l+j−i)(

ny

k−i,j,ny−k+i−j)(

nx+ny

k,l,nx+ny−k−l) (SI-7)

Formulae for folded spectrum:

The results in Main Text have been obtained for the case where the ancestral allele is known, for example from an

outgroup sequence, and therefore the frequency spectrum is unfolded. In many situations an outgroup is not available

and it is not possible to discriminate between derived and ancestral alleles; in this case, the estimators and tests should

be based on the folded frequency spectrum ηi(x) = (ξi(x) + ξnx−i(x))/(1 + δnx,2i). As discussed by ACHAZ

(2009), estimators and tests for folded data should satisfy additional conditions, which in our framework read iωi,nx =

(nx − i)ωnx−i,nxand iΩi,nx

= (nx − i)Ωnx−i,nx. We explain here how to obtain the estimators and tests based on

the folded spectrum.

In our framework, all estimators and tests depend on the folded site frequency spectrum ηi(x) = (ξi(x) +

ξnx−i(x))/(1 + δbnx/2c,i). The general form for estimators and tests is

θ =1

L

L∑x=1

bnx/2c∑i=1

i(nx − i)nx(1 + δbnx/2c,i)

ωi,nxηi(x) ,

1

L

L∑x=1

bnx/2c∑i=1

ωi,nx= 1 (SI-8)

Ferretti, Raineri and Ramos-Onsins 4 SI

T =θ − θ′

Var(θ − θ′

) =

∑Lx=1

∑bnx/2ci=1

i(nx−i)(1+δbnx/2c,i)

nxΩi,nx

ηi(x)

Var(∑L

x=1

∑bnx/2ci=1

i(nx−i)(1+δbnx/2c,i)

nxΩi,nx

ηi(x)) ,

L∑x=1

bnx/2c∑i=1

Ωi,nx= 0 (SI-9)

The variances are

Var

L∑x=1

bnx/2c∑i=1

i(nx − i)(1 + δbnx/2c,i)

nxΩi,nxξi(x)

=

L∑x=1

bnx/2c∑i=1

i(nx − i)(1 + δbnx/2c,i)

nxΩ2i,nx

θ+ (SI-10)

+

L∑x,y=1

x 6=y

bnx/2c∑i=1

bny/2c−1∑j=1

i(nx − i)(1 + δbnx/2c,i)

nx

j(ny − j)(1 + δbny/2c,j)

nyΩi,nx

Ωj,nyCov(ηi(x), ηj(y))

in terms of the covariance Cov(ηi(x), ηj(y)) between different sites, which can be obtained as

Cov(ηi(x), ηj(y)) =Cov(ξi(x), ξj(y)) + Cov(ξnx−i(x), ξj(y)) + Cov(ξi(x), ξny−j(y)) + Cov(ξnx−i(x), ξny−j(y))

(1 + δbnx/2c,i)(1 + δbny/2c,j)

(SI-11)

Numerical results and discussion:

In the main text we provide evidence of an increase in performance when the read depth is fixed and more indi-

viduals are sequenced, both for Watterson estimator (Figure 1 in the main text) and for neutrality tests (Figure S1).

This decrease in variance (i.e., the increase in performance) is apparent again if we compare these variances with fixed

sample size and pm > 0 with the variances at the same average depth but without missing data, as in Figure S2. The

effect is stronger at lower depth. Interestingly, missing data could therefore result in a loss of power for haplotype

tests, but they increase the performance of tests and estimators based on the frequency spectrum as long as they are

compensated by an higher number of sequences.

LITERATURE CITED

ACHAZ, G., 2009 Frequency Spectrum Neutrality Tests: One for All and All for One. Genetics 183: 249.

FU, Y.-X., 1995 Statistical properties of segregating sites. Theoretical Population Biology 48: 172–197.

HUDSON, R., M. KREITMAN, and M. AGUADÉ, 1987 A test of neutral molecular evolution based on nucleotide data.

Genetics 116: 153.

Ferretti, Raineri and Ramos-Onsins 5 SI

Figure S1: Variance of Tajima’s D (lower lines) and Fay and Wu’s H (upper lines) on a window of L = 100 basesfor θ = 0.1. Computed as in Figure 1 of the main text for fixed sample size n = 20 (solid lines) and fixed averagedepth n(1− pm) ' 20 (dashed lines). The decrease in variance for fixed sample size is due to the the reduced effectivesample size 20(1 − pm). (Note that Fay and Wu’s H variance is divided by 4 to appear in scale with Tajima’s Dvariance.)

Ferretti, Raineri and Ramos-Onsins 6 SI

Figure S2: Ratio of the variances of Tajima’s D (blue line) and Fay and Wu’s H (green line) between two cases withthe same average depth 20(1 − pm): first, with missing data (pm > 0) and fixed sample size n = 20, second, withsample size ' 20(1 − pm) but without missing data (pm = 0). Computed as in Figure S1 on a window of L = 100bases for θ = 0.1.

Ferretti, Raineri and Ramos-Onsins 7 SI