G. Cowan, RHUL Physics Statistics for early physics page 1

Statistics jump-start for early physics

ATLAS Statistics Forum

EVO/Phone, 4 May, 2010

Glen Cowan (RHUL)Eilam Gross (Weizmann Institute)

G. Cowan, RHUL Physics Statistics for early physics page 2

IssuesFor early physics we need to finalize methodologies forsetting limits (and quoting discovery significance).

For today focus mainly on limits

Need to agree with CMS what procedures we will use(a) for comparison (also with e.g. Tevatron)(b) for combination

Review of methods in useProfile likelihoodCLsBayesian

Status of software tools (RooStats, …)

Look-elsewhere-effect (an issue for discovery, not limits).

G. Cowan, RHUL Physics Statistics for early physics page 3

Frequentist discovery and limitsFor frequentist methods, focus on p-value

or equivalent significance Z = (1 – p)

“Discovery” if p-value of background-only < 2.9 × 10(Z > 5)

Can compute e.g. median discovery significance assuming different signal models.

(In some cases may not have well-defined signal model.)

Exclusion at CL = 1 – if p < (e.g. = 0.05) Can compute e.g. median limit assuming background-only.For exclusion, parameter (model) being tested is the null;compute power relative to background-only alternative:

Power = P(reject model(parameter) | background only)

G. Cowan, RHUL Physics Statistics for early physics page 4

Bayesian discoveryFor Bayesian discovery, compute Bayes factor to comparee.g. model i (Higgs) to j (no Higgs):

Gives posterior odds if prior odds were 50-50.

Avoids dependence on unlikely data outcomes that werenever seen (cf. tail probabilities for p-values.)

Work needed here, mainly on computational issues.

G. Cowan, RHUL Physics Statistics for early physics page 5

Bayesian limitsJust integrate the posterior probability obtained from Bayes thm,

Not widely used so far in ATLAS (?) but will need anywayfor comparison with Tevatron.

Despite recent discussion on reference priors (a la Bernardo, Jeffereys…), not aware of realistic applications in HEP.

Need survey of code and of who is using this in ATLAS.

Await outcome of CMS discussion on priors.

G. Cowan, RHUL Physics Statistics for early physics page 6

SystematicsConnect systematic to nuisance parameters . Then form e.g.

Profile likelihood:

Marginal likelihood:

and use these to construct e.g. likelihood ratios for tests.Coverage not guaranteed for all values of the nuisance params.

Literature contains some variants that we do not recommend,e.g., forming a likelihood ratio and integrating it with a prior.

Need to decide what to do when systematic cannot be dealtwith using nuisance parameters (e.g. corrections using PYTHIA vs. HERWIG).

G. Cowan, RHUL Physics Statistics for early physics page 7

ZBi, Z, ZN, etc.Several authors have studied various ways to obtain thesignificance in the simple counting experiment with systematicuncertainty on the background, e.g., Cranmer (PHYSTAT 03,05) and Cousins, Tucker, Linnemann (physics/0702156).

Our usual profile likelihood method with a subsidiary measurementfor the background gives ZBi, also preferred by CMS.

ZN assumes a Gaussian prior for the background (truncated atzero) and in many cases is not a realistic model. Should only beused if the analyst actually believes that an average with respectto the truncated Gaussian prior is the appropriate model (not likely).

CMS has flagged this as an area where we should reach agreement;do not anticipate trouble here (does ATLAS use ZN?)

G. Cowan, RHUL Physics Statistics for early physics page 8

ATLAS practiceWe need to complete our survey of methods used in ATLAS.

In StatForum we have made important progress in usingprofile likelihood ratio tests.

Can get significance, limits without any toy MC (validfor large samples, in practice can even be smallish).

For 95% CL limits for very low lumi, can always turnto toy MC to calibrate method.

G. Cowan, RHUL Physics Statistics for early physics page 9

CLsIf we test parameter values to which we have no sensitivity(e.g. very large Higgs mass), then there is a probability of1 – CL (e.g. 5%) that we will reject.

In the CLs method the p-value is reduced according to therecipe

Statistics community does not smile upon ratio of p-values;would prefer to regard parameter m as excluded if:

(a) p-value of m < 0.05(b) power of test of m with respect to background-only > some threshold (0.5?)

Needs study. In any case should produce CLs result for purposesof comparison with CMS/Tevatron.

G. Cowan, RHUL Physics Statistics for early physics page 10

Choice of likelihood ratio statistic

Ongoing discussion as to whether best to use LEP-stylelikleihood ratio

or

and in both cases how to deal with the nuisance parameters.

In simple cases one obtains the same test from both statistics.

G. Cowan, RHUL Physics Statistics for early physics page 11

Questions to discuss

Need to decide about CLs (CMS have not yet decided).

Decide on methods for incorporating systematics (Zbi, ZN, …?)

Decide on ATLAS recommended methods to be used in presentation of results

Check the status of Roostats as a tool for hypothesis testing and for combinations.