Available on CMS information server CMS CR -2015/334

The Compact Muon Solenoid Experiment

Mailing address: CMS CERN, CH-1211 GENEVA 23, Switzerland

Conference Report07 December 2015 (v3, 12 December 2015)

Rare and Exotic decays of the Higgs boson at theLHC

Yuta Takahashi for the ATLAS, CMS and Higgs collaborations.

Abstract

The recent LHC searches on rare and exotic Higgs boson decays are presented. The analyses areindividually performed by the ATLAS and CMS collaborations, using LHC run-1 dataset with anintegrated luminosity of 20 fb−1. The standard model rare Higgs decays, such as H→ µµ, ee, andH → Z/γ(→ ``) + γ are reviewed first, followed by exotic decays, such as lepton-flavour violatingdecays (H → µτ , eτ , eµ) and decays into a pair of pseudoscalar Higgs bosons (H→ a1a1).

Presented at LHCP2015 The 3rd Conference on Large Hadron Collider Physics

Rare and Exotic decays of the Higgs boson at the LHC

Yuta Takahashi1on behalf of the ATLAS and CMS collaborations

1European Organization for Nuclear Research (CERN)

Yuta.Takahashi@cern.ch

Abstract. The recent LHC searches on rare and exotic Higgs boson decays are presented. The analyses are individually performedby the ATLAS and CMS collaborations, using LHC run-1 dataset with an integrated luminosity of ∼ 20 fb−1. The standard modelrare Higgs boson decays, such as H → µµ, H → ee, and H → Z/γ∗(→ ℓℓ) + γ are reviewed first, followed by exotic decays, suchas lepton-flavour violating decays (H → µτ, eτ, eµ) and decays into a pair of pseudoscalar Higgs bosons (H → 2a1).

Introduction

The Higgs boson was discovered in 2012 by the ATLAS [1] and CMS [2] collaborations [3, 4]. The observed propertiesof this particle, e.g. its couplings to fermions and bosons and its spin and parity, are consistent with those of theStandard Model (SM) Higgs boson with a mass near 125 GeV [5, 6, 7, 8]. However, as implied by the currentconstraints on the Higgs boson branching ratio to non-SM particles, B(H →BSM) < 34% @ 2σ [9], there is stillplenty of room left for the possible contributions from physics beyond the SM. This motivates direct searches for rareand exotic Higgs boson decays at the LHC.

In this note, searches for the rare SM Higgs boson decays (H→ ee, µµ and H → Z/γ∗(→ ℓℓ)γ) are reviewed first,followed by recent searches for exotic decays such as lepton-flavor violating Higgs boson decays (H→ µτ, eτ andeµ) and decays to a pair of light pseudoscalar neutral Higgs bosons, as predicted by next-to-minimal supersymmetricstandard model [10]. References corresponding to the analyses described here can be found in [11, 12].

Rare Higgs boson decays

H → ee, µµIn the SM, the decay rate of the Higgs boson into fermions is proportional to m2

f , where m f is the fermion mass. Assuch, the branching ratio of the Higgs boson decaying into ττ final state is by far the largest among Higgs leptonicdecay (6%), while µµ (0.02%) and ee (5 × 10−7%) are much smaller. Given the fact that the ττ final state is alreadyobserved with 5.5σ significance by combining the ATLAS and CMS results [9], non-observation of the µµ and eefinal states will be the direct evidence that the Higgs leptonic coupling is not flavour universal, unlike Z → ℓℓ.

The basic strategy of the analysis [13, 14] is to reconstruct di-lepton invariant mass (mll) and look for a narrow di-lepton resonance on top of the large Drell-Yan background, whose contribution is typically three orders of magnitudelarger than the signal. In order to increase the analysis sensitivity, the event categorization is performed based onnumber of jets, transverse momentum (pT ) of the di-lepton system and the pseudo-rapidity (η) of the lepton. Figure 1shows the di-muon invariant mass distribution in one of the most sensitive category, the VBF (Vector Boson Fusion)category with more than 2 jets satisfying the invariant mass, m j j > 500 GeV, |η j1 − η j2| > 3 and η j1 × η j2 < 0, wherej1 ( j2) is the (sub)leading jet.

The obtained mℓℓ distributions are fitted by signal and background templates, where an analytic function is usedfor the background modeling. Since no excess of events is observed, 95% C.L upper limits are set on the H → µ+µ−

signal strength in units of the signal expected in the SM: 7.0 (obs) / 7.2 (exp) in the ATLAS analysis (assuming

mH = 125.5 GeV), while 7.4 (obs) / 6.5 (exp) in the CMS analysis (assuming mH = 125 GeV). The CMS collaborationalso puts the upper limit on the cross section times H → ee branching ratio: 0.0041 pb. These results imply that theHiggs leptonic coupling is indeed not flavour universal. The H → µµ final state is expected to become accessible with400 fb−1 of data at

√s = 13 TeV, which will be the scope of high-luminosity LHC.

FIGURE 1. (left) di-muon invariant mass distribution in the VBF category, i.e. events with more than 2 jets with m j j > 500 GeV,|η j1 −η j2| > 3 and η j1 ×η j2 < 0. (right) observed (solid) and expected (dashed) 95% confidence level upper limits on the H → µ+µ−

signal strength as a function of mH . The green and yellow band shows ±1σ, ±2σ, respectively [13, 14].

H → Z/γ∗(→ ℓℓ)γ

The other important rare decay of the Higgs boson is H → Z/γ∗ + γ [15, 16, 17]. Because of the exploitation of theZ/γ∗ → ℓℓ decay in the analysis, the relative production rate of the Zγ → ℓℓγ (γ∗γ → ℓℓγ) is merely 2.2% (4.0%)compared to the H → γγ decay. However, observation of this final state is motivated, as a large enhancement of thebranching ratio can be expected by heavy charged particles in the loop, just as for the H → γγ decay.

The analysis proceeds by selecting events with opposite-sign lepton pairs (ee or µµ) with one isolated photon.In order to increase the analysis sensitivity, an event categorization based on the η of the lepton, η of the photon, andshower shape variable are used in the CMS analysis. ATLAS exploits the categorization based on the ∆η(Z, γ) andHiggs boson pT with respect to the thrust axis of the event. Further improvement has been obtained, in the ATLASanalysis, by using Z boson mass constraint fit to the lepton kinematics, which leads to 20% better ℓℓγ mass resolution.

As a discriminant variable, the invariant mass of the ℓℓγ system was used. Concerning the H → Zγ analysis, anadditional requirement on the di-lepton mass is applied, e.g. mℓℓ > mZ − 10 GeV. The dominant backgrounds comefrom irreducible initial-state radiation SM Zγ production, which is modeled by an analytic function. Since no excessof events above the backgrounds was found, the upper limit on the signal strength is set to be µ < 11 (obs) / 9 (exp) inATLAS, and µ < 9.5 (obs) / 10 (exp) in CMS analysis.

CMS also performed searches, targeting H → γ∗(→ ℓℓ)γ decay. To ensure that the di-lepton comes from low-mass γ∗, mℓℓ < 20 GeV is additionally required. Events with di-muon mass in the range 2.9 < mµµ < 3.3 GeV and9.3 < mµµ < 9.7 GeV, however, are rejected to avoid J/ψ → µµ and Υ → µµ contributions. Figure 2 shows themµµγ spectra for the 8 TeV data and shows no excess of events above the expected backgrounds. The upper limit isset on the µ value as a function of mH and at 125 GeV, µ < 7.7 (exp. 6.4) is obtained. It should be noted that eventsconsistent with J/ψ → µµ in di-muon invariant mass are also used to set a 95% C.L limit on the branching ratio,B(H → (J/ψ)γ) < 1.5 × 10−3, although that is 540 times the SM prediction for mH = 125 GeV [16].

FIGURE 2. (left) mµµγ spectra at 8TeV, together with the result of a background-only fit to the data. (right) 95% C.L upper limiton the µ-value as a function of mH [16].

Exotic Higgs boson decays

Lepton Flavour Violating decay : H → eτ, µτ, eµ

Within the SM, the leptonic-flavour violating decays of the Higgs boson are forbidden. However, such decays cannaturally occur in supersymmetric models [18], composite Higgs models [19], models with flavour symmetries [20],Randall-Sundrum models [21], and many others. The pre-LHC limits on such decays areB(H → eµ) < 10−8, obtainedfrom null search results on µ → eγ, and B(H → µτ, eτ) < O(10%), obtained from τ → µγ, eγ, other rare τ decays,and muon g − 2 experiments. The weak constraints on H → µτ and H → eτ motivate direct searches at the LHC.

The analysis strategy is similar to that of the SM H → ττ search. However, as depicted in Figure 3, there arestriking differences in kinematic variables which can be exploited to select the events. For example, one can expectharder lepton pT spectrum and the collinearity between the missing transverse momentum (Emiss

T ) and the lepton inthe lepton-flavour violating decay.

After a set of event selections, CMS adopts the collinear mass as the final discriminant for the H → µτ andH → eτ analysis, while the eµ invariant mass is used for the eµ channel. ATLAS analyzed H → µτhad final state anduses a mass estimator, mMMC

µτ , reconstructed by the observed muon, hadronic τ and EmissT . The dominant backgrounds

depend on the categories, but in general, come from Drell-Yan Z → ττ. To increase the analysis sensitivity, CMScategorizes the events based on the number of reconstructed jets, while ATLAS uses the transverse mass, mT (µ, Emiss

T )and mT (τ, Emiss

T ) to define the signal region. Figure 4 shows discriminant variable distributions in recent papers [22,23, 24].

FIGURE 3. Schematic difference in the (left) SM H → τµτe decay and (right) lepton-flavour violating H → µτedecay.

FIGURE 4. From left to right, post-fit combined mMMCµτhad

distribution obtained by adding individual distributions in signal regions(ATLAS [24]), collinear eτ mass distribution in the 2 jet category (CMS), and invariant eµ mass distribution (CMS) [23].

Table 1 summarizes the obtained upper limit on the branching ratio for the lepton-flavour violating decays.The obtained limits in eτ and µτ channels are 0.7-1.9% level, which improves current indirect bounds by an or-der of magnitude. These limits are subsequently used to constrain the µτ, eτ and eµ Yukawa coupling, which is√|Yµτ|2 + |Yτµ|2 < 3.6 × 10−3,

√|Yeτ|2 + |Yτe|2 < 2.41 × 10−3, and

√|Yeµ|2 + |Yµe|2 < 5.43 × 10−4, respectively. Given

the slight excess in the H → µτ channel in both ATLAS (1.3σ) and CMS (2.4σ) analysis, it will be interesting topursue with LHC run-2 data.

TABLE 1. The observed / expected upper limit on the lepton-flavour violating Higgs boson decay and its best fitvalue.

ATLAS CMSchannel µτhad µτ eτ eµ

obs. (exp.) limit < 1.85%(1.24%) < 1.51%(0.75%) < 0.69%(0.75%) < 0.036%(0.048%)Best fit 0.77 ± 0.62% 0.84+0.39

−0.37% −0.10+0.37−0.36% —

Light pseudoscalar decay : H → a1a1 → 4µIn the next-to-minimal supersymmetric standard model (NMSSM), the Higgs sector is extended to have three CP-even Higgs bosons h1,2,3, two CP-odd neutral Higgs bosons a1,2 and a pair of charged Higgs bosons H±. The h1 orh2 can decay via h1,2 → 2a1. where either the h1 or h2 can be the observed Higgs boson. If the a1 mass lies within2mµ < ma1 < 2mτ, the decay, a1 → µ+µ− will be the dominant one. This motivates searches for 4µ final states [25].

The events are selected if there is a pair of low-mass di-muons with similar invariant mass. Figure 5 shows the2D correlation plot between the 1st and 2nd di-muon invariant mass. The signal region is defined as the diagonalcomponent of this plot, denoted by the white dashed lines. The dominant backgrounds come from SM bb̄ production.

The off-diagonal sideband region, which lies outside the white dashed lines, is used to estimate the expectedbackground in the signal region. Together with small backgrounds arising from electroweak production of four muonsand direct Jψ production, the expected background in the signal region is estimated to be 2.2 ± 0.7, while 1 event isobserved at m1,µµ = 0.33 GeV, m2,µµ = 0.22 GeV. Since the observed yield is consistent with background expectation,the limit was set in the context of the next-to-minimal supersymmetric standard model, and in scenarios containing ahidden sector, including those predicting a non-negligible light boson lifetime.

Light pseudoscalar decay : H → a1a1 → µµττ

Similarly to the 4µ analysis, one can also target the a1 mass range 2mτ < ma1 [26]. ATLAS performed such searcheswith the µµττ final state. Although the production rate of µµττ final state is roughly 1/100, compared to the 4τ finalstate, one can benefit from a larger S/B ratio in this channel. In this analysis, at least one of the τ-leptons is requiredto decay leptonically.

FIGURE 5. Distribution of the invariant masses m1,µµ and m2,µµ for the isolated di-muon pair events [25]. The signal region isoutlined as a white dashed lines, containing one data event (triangle) at m1,µµ = 0.33 GeV, m2,µµ = 0.22 GeV.

The analysis proceeds by selecting events with opposite-charge muons, with pT (µµ) > 40 GeV and 2.8 < mµµ <70 GeV. The high pT requirement on the di-muon system stems from the fact that the two a1 bosons are expectedto be produced back-to-back in the transverse plane. The event must contain a third lepton (muon or electron) that iscoming from the τ-lepton decay.

FIGURE 6. observed mµµ distribution in the signal region and the background-only fit [26]. The expected signal distribution froma signal with B(h→ aa) = 10% is shown for three different ma hypothesis (5, 10, and 20 GeV).

After a set of event selections, the di-muon invariant mass distribution is used to evaluate the possible excess,as shown in Figure 6. The full background model consists of six SM resonances (J/ψ, ψ

′,Υ1S ,Υ2S ,Υ3S and Z), tt̄

components and continuum Drell-Yan backgrounds. The signal and each SM resonances are modeled by double-sided Crystal Ball functions, and an un-binned log-likelihood fit is performed on the observed di-muon invariant mass

spectra to a combination of background and signal models. No excess of data is observed in the di-muon mass rangefrom 3.7 GeV to 50 GeV. Upper limits are placed on the production of H → a1a1 relative to the SM gg → Hproduction. Assuming no coupling of the a boson to quarks, the most stringent limit is placed at 3.5% for ma =

3.75 GeV.

Summary

Since the discovery of a Higgs-like boson, searches have been made for rare and exotic Higgs boson decays by theATLAS and CMS collaborations. The rare Higgs boson decays, such as H → ee, µµ, H → Z/γ∗ + γ, have not beenobserved yet with the run-1 dataset (19.7 fb−1), as expected by the standard model. The current upper limit on thesignal strength lies 7-11 times the SM values, and these channels are expected to become visible with high-luminosityLHC data. The exotic Higgs boson decays, e.g. lepton-flavour violating decays, decays into a pair of light pseudoscalarin the µµµµ or µµττ final states, depending on the a1 mass range, have been also explored using LHC run-1 dataset.Although there is a mild excess in the H → µτ decay, there are, in general, no surprises up to now.

It should be noted that there is still plenty of room left for the contributions from physics beyond the SM, thatis compatible with the observed Higgs boson. The current weak constraint on the Higgs boson branching ratio to thenon-SM particles, B(H → BS M) < 34% (2σ), encourages to continue these interesting searches throughout LHCrun-2.

REFERENCES

[1] ATLAS Collaboration, JINST 3 S08003 (2008)[2] CMS Collaboration, JINST 3 S08004 (2008)[3] ATLAS Collaboration, Phys. Lett. B 716 (2012) 1-29[4] CMS Collaboration, Phys. Lett. B 716 (2012) 30-61[5] ATLAS Collaboration, Phys. Lett. B 726 (2013) 88-119[6] ATLAS Collaboration, Phys. Lett. B 726 (2013) 120-144[7] CMS Collaboration, Phys. Rev. D 92 (2015) 012004[8] CMS Collaboration, Eur. Phys. J. C 75 (2015) 212[9] ATLAS and CMS Collaborations, Measurements of the Higgs boson production and decay rates and con-

straints on its couplings from a combined ATLAS and CMS analysis of the LHC pp collision data at√

s = 7and 8 TeV, ATLAS-CONF-2015-044, CMS-PAS-HIG-15-002

[10] U. Ellwanger, M. Rausch de Traubenberg, and C. A. Savoy, Phenomenology of supersymmetric models witha singlet, Nucl. Phys. B 492 (1997) 21,

[11] All ATLAS Higgs Physics Public Results, presented in this note, can be found athttps://twiki.cern.ch/twiki/bin/view/AtlasPublic/HiggsPublicResults

[12] All CMS Higgs Physics Public Results, presented in this note, can be found athttp://cms-results.web.cern.ch/cms-results/public-results/publications/HIG/index.html

[13] CMS Collaboration, Phys. Lett. B 744 (2015) 184-207[14] ATLAS Collaboration, Phys. Lett. B 738 (2014) 68-86[15] CMS Collaboration, Phys. Lett. B 726 (2013) 587-609[16] CMS Collaboration, Search for a Higgs boson decaying into γ∗γ → ℓℓγ with low dilepton mass in pp

collisions at√

s = 8 TeV, arXiv:1507.03031, CMS-HIG-14-003[17] ATLAS Collaboration, Phys. Lett. B 732C (2014), 8-27[18] J. L. Diaz-Cruz and J. J. Toscano, Phys. Rev. D 62 (2000) 116005[19] K. Agashe and R. Contino, Phys. Rev. D 80 (2009) 075016[20] H. Ishimori et al., Prog. Theor. Phys. Suppl. 183 (2010) 1[21] S. Casagrande et al., JHEP 10 (2008) 094[22] CMS Collaboration, Phys. Lett. B 749 (2015) 337[23] CMS Collaboration, Search for lepton-flavour-violating decays of the Higgs boson to eτ and eµ at

√s =

8 TeV, CMS-PAS-HIG-14-040[24] ATLAS Collaboration, Search for lepton-flavour-violating H → µτ decays of the Higgs boson with the

ATLAS detector, arXiv:1508.03372, CERN-PH-EP-2015-184[25] CMS Collaboration, A search for pair production of new light bosons decaying into muons,

arXiv:1506.00424, CMS-HIG-13-010[26] ATLAS Collaboration, Phys. Rev. D92 (2015) 052002