ZCh, TE-MPE-TM 1

nQPS trips due to EMC transients

30/06/2011

Z.Charifoulline

- Thunderstorm@22/06/2011: # nQPS.BS signals in A12 & A34 # statistics over all sectors – how many sectors would be tripped?- EMC Transient example in P8- Brief statistics for nQPS trips 2011 (hyperlinks to logbooks)- Current status of BS thresholds for long quad buses (TS#3)

ZCh, TE-MPE-TM 2

ev_110622-161742.469_RQ10.R1

ev_110622-161742.468_RQ8.L1 ev_110622-161742.480_RD2.R1 ev_110622-161742.468_RQ8.R1 ev_110622-161742.468_RQ10.L1 ev_110622-161742.477_RQ6.R1 ev_110622-161742.492_RD2.L1 ev_110622-161742.468_RQ9.L1 ev_110622-161742.467_RQ4.R1 # ev_110622-172405.414_RQ8.L5 ev_110622-172405.407_RQ10.R5

ev_110622-172405.418_RQ10.L2 ev_110622-172405.397_RQ10.L4 ev_110622-172405.403_RQ10.R4

ev_110622-172405.401_RQ10.R2

ev_110622-172405.409_RQ10.L1 ev_110622-172405.420_RQ10.L6 ev_110622-172405.401_RQ10.L5 ev_110622-172405.403_RQ8.R4 ev_110622-172405.401_RQ8.L4 ev_110622-172405.409_RQ9.L1 ev_110622-172405.401_RQ9.R2 # ev_110622-174525.904_RQ10.L4 # ev_110622-175508.837_RQ10.L8 ev_110622-175508.836_RQ10.R5

ev_110622-175508.836_RQ8.R6 ev_110622-175508.834_RQ9.L6 # ev_110622-175641.117_RQ10.R8

ev_110622-175641.115_RQ10.L1 ev_110622-175641.115_RQ9.L8 ev_110622-175641.115_RQ9.L1 ev_110622-175641.116_RQ9.R8 # ev_110622-181706.748_RQ10.L2 ev_110622-181706.729_RQ10.L4 ev_110622-181706.730_RQ10.R4

ev_110622-181706.729_RQ10.R2

ev_110622-181706.752_RQ10.L5 ev_110622-181706.730_RQ9.R2

#22/06/2011 16h17: P1LHC FPA@Injection

#22/06/2011 17h24: P1,P2,P4,P5

#22/06/2011 17h55: P5,P6,P8

#22/06/2011 17h57: P1,P8

#22/06/2011 18h17: P2,P4,P5

#22/06/2011 17h45: P4

22/06/2011 Thunderstorm in LHC tunnel

30/06/2011

LHCOP 16h24: Power cut global in the 400 V, all kickers off.

LHCOP 17h25: We lost again 4 sectors

ZCh, TE-MPE-TM 3

22/06/2011 Thunderstorm: Sector A12

B33R1_RQFB31R1_RQDB27R1_RQFB25R1_RQFB15R1_RQD…B32L2_RQFB30L2_RQFB28L2_RQDB24L2_RQDB22R1_RQDB34R1_RQD

A12.RQD/RQFNo significant EMC effects in the first shot. Why?

#22/06/2011 17h24: P1,P2,P4,P5- more than 12 channels above 500uV;- EMC effect is stronger in the middle of the arc;- EE switches were already open;

Most likely it would trip the circuit.

30/06/2011

500uV

ZCh, TE-MPE-TM 4

22/06/2011 Thunderstorm: Sector A34

B34R3_RQDB28L4_RQDB26R3_RQFB30R3_RQFB12R3_RQF…B31L4_RQFB19L4_RQFB21L4_RQDB25L4_RQDB15R3_RQDB17R3_RQD

A34.RQD/RQFNo significant EMC effects in the first shot.

#22/06/2011 17h24: P1,P2,P4,P5- more than 12 channels above 500uV;- EMC effect is more uniform over the arc;- EE switches were closed;

Most likely it would trip the circuit.

30/06/2011

U_RESmax

500uV

ZCh, TE-MPE-TM 5

22/06/2011 Thunderstorm: overall view & conclusion

#22/06/2011 17h24

#22/06/2011 17h55U_RESmax

U_RESmax

Thunderstorm => 400V Circuit BreakersBut no detailed information found

Many sectors would be tripped due to the huge spikes on RQD/RQF-circuits even if IPQs will be stable, but without firing the sleeping quench heaters!

Relatively easy solution is to increase the U_RES threshold to 1mV (170nΩ@6kA)?What else? It might be more sophisticated BS-board firmware with Jens experience in the Thermal Amplifier Project (mBS-board).

30/06/2011

ZCh, TE-MPE-TM 630/06/2011

LHCOP 16/06/2011 14:49:Global Post Mortem Event Confirmation

Dump Classification: Other Operator / Comment: stefano / Trip of the main quads in S81 - MP3 investigating the source of the trip.

---------------------------No information found from SPS&Injection team – ALL OK.

UFO or Tramp AUG?

A81: 98 -> 99A78.RQ

A81.RQ

500uV

600uV

Last nQPS.BS trip at Point8, 16/06/2011 “1 fb-1”

ZCh, TE-MPE-TM 730/06/2011

Time Stamp & Where Comments

2011.06.22 **** Thunderstorm => 400V Breaker

2011.06.16 A81.RQD/RQF (B11R8) Glitch in Point8, but it is not known why and where? 98 => 99 (2011.06.16)

2011.05.27 A67.RQD/RQF (B15L7) Trip on RQF due to nQPS (Same as yesterday evening). Proposal to increase the threshold to 600 uV and postpone investigation… DQQBS board #9 B15L7 successfully exchanged by Joaquim --> ok. Problem solved.

2011.05.17 A78.RQD/RQF (B11L8) •Circuit tripped always at the end of a pre-cycle at around 1400A. This happened only if all the circuits are pre-cycled together. … We're just pretty close to the threshold. Further tests are required to find out which circuits causes the xtalk on the B11L8_8 card. JS (precyles&ramps view)

98=>99

2011.05/17 A81.RQD/RQF BS boards tripped during SPS magnet tests at 11:39:11 in RQ81 (overall sector view)

2011.05.02 A81.RQD/RQF (B11R8) Signal B11R8_4 passed the threshold of 500uV.All the nQPS crates signals pick up a noise which is bigger than usual (with an amplitude of about 100uV and a frequency of about 40mHz or 25 seconds)Probably there is some emf coupling with the SPS. See also the event on 30 march 2010 at 10h22!!! AW

98 => 99 ???

2011.02.19 A12.RB (B9L2) Replacement of the two BS boards of B9.L2 slots 3 and 4 (exceeded 500uV earlier this morning.

98???

2011 nQPS.BS Trip Statistics from logbooks

Not so many cases, but not negligible. All in Point 8 so far.Increasing the BS thresholds for all long buses and checking it bypower cycling of the concerned units might help in many cases.Now it is very right moment to do so: TS#3.

ZCh, TE-MPE-TM 830/06/2011

A67.B15L7_RQD(A)

A78.B11L8_RQD(A)A78.B11L8_RQF(A)

A81.B11R8_RQD(B)A81.B11R8_RQF(B)A81.B11L1_RQD(A)A81.B11L1_RQF(A)

Current status of BS-thresholds for the RQ long buses (30/06/2011)

ZCh, TE-MPE-TM 930/06/2011

Point Crate Bus Bar Segment Magnet 1 Magnet 2 splNum thrs A thrs B1

DQAMGS.B11R1 3 DCQDE.11R1.R MQ.12R1.B1 <-> MQ.11R1.B2 32 98DQAMGS.B11R1 4 DCQFE.11R1.L MQ.12R1.B2 <-> MQ.11R1.B1 32 98DQAMGS.B10L2 3 DCQDD.7L2.L MQ.11L2.B2 <-> DFLAS.7L2.2 16 98DQAMGS.B10L2 4 DCQFD.7L2.R MQ.11L2.B1 <-> DFLAS.7L2.4 16 98DQAMGS.B11L2 3 DCQDQ.12L2.R DFLAS.7L2.1 <-> MQ.12L2.B1 20 98DQAMGS.B11L2 4 DCQFQ.12L2.L DFLAS.7L2.3 <-> MQ.12L2.B2 20 98

2DQAMGS.B11R2 3 DCQDB.C12R2.R DFLAS.7R2.1 <-> MQ.12R2.B2 20 98DQAMGS.B11R2 4 DCQFB.C12R2.L DFLAS.7R2.3 <-> MQ.12R2.B1 20 98DQAMGS.B10R2 3 DCQDD.7R2.L MQ.11R2.B1 <-> DFLAS.7R2.2 16 98DQAMGS.B10R2 4 DCQFD.7R2.R MQ.11R2.B2 <-> DFLAS.7R2.4 16 98

3DQAMGS.B10L4 3 DCQDD.7L4.R MQ.11L4.B2 <-> DFLAS.7L4.1 16 98DQAMGS.B10L4 4 DCQFD.7L4.L MQ.11L4.B1 <-> DFLAS.7L4.3 16 98DQAMGS.B11L4 3 DCQDQ.12L4.L DFLAS.7L4.2 <-> MQ.12L4.B1 20 98DQAMGS.B11L4 4 DCQFQ.12L4.R DFLAS.7L4.4 <-> MQ.12L4.B2 20 98

4DQAMGS.B11R4 3 DCQDB.C12R4.R DFLAS.7R4.1 <-> MQ.12R4.B2 20 98DQAMGS.B11R4 4 DCQFB.C12R4.L DFLAS.7R4.3 <-> MQ.12R4.B1 20 98DQAMGS.B10R4 3 DCQDD.7R4.L MQ.11R4.B1 <-> DFLAS.7R4.2 16 98DQAMGS.B10R4 4 DCQFD.7R4.R MQ.11R4.B2 <-> DFLAS.7R4.4 16 98DQAMGS.B11L5 3 DCQDQ.11L5.L MQ.12L5.B2 <-> MQ.11L5.B1 32 98DQAMGS.B11L5 4 DCQFQ.11L5.R MQ.12L5.B1 <-> MQ.11L5.B2 32 98

5DQAMGS.B11R5 3 DCQDE.11R5.L MQ.12R5.B1 <-> MQ.11R5.B2 32 98DQAMGS.B11R5 4 DCQFE.11R5.R MQ.12R5.B2 <-> MQ.11R5.B1 32 98DQAMGS.B10L6 3 DCQDD.5L6.L MQ.11L6.B2 <-> DFLAS.5L6.2 17 98DQAMGS.B10L6 4 DCQFD.5L6.R MQ.11L6.B1 <-> DFLAS.5L6.4 17 98DQAMGS.B11L6 3 DCQDQ.12L6.R DFLAS.5L6.1 <-> MQ.12L6.B1 21 98DQAMGS.B11L6 4 DCQFQ.12L6.L DFLAS.5L6.3 <-> MQ.12L6.B2 21 98

6DQAMGS.B11R6 3 DCQDB.C12R6.L DFLAS.5R6.2 <-> MQ.12R6.B2 19 98DQAMGS.B11R6 4 DCQFB.C12R6.R DFLAS.5R6.4 <-> MQ.12R6.B1 19 98DQAMGS.B10R6 3 DCQDD.5R6.R MQ.11R6.B1 <-> DFLAS.5R6.1 15 98DQAMGS.B10R6 4 DCQFD.5R6.L MQ.11R6.B2 <-> DFLAS.5R6.3 15 98

7DQAMGS.B10L8 3 DCQDD.7L8.L MQ.11L8.B2 <-> DFLAS.7L8.2 16 98DQAMGS.B10L8 4 DCQFD.7L8.R MQ.11L8.B1 <-> DFLAS.7L8.4 16 98DQAMGS.B11L8 3 DCQDQ.12L8.R DFLAS.7L8.1 <-> MQ.12L8.B1 20 99DQAMGS.B11L8 4 DCQFQ.12L8.L DFLAS.7L8.3 <-> MQ.12L8.B2 20 99

8DQAMGS.B11R8 3 DCQDB.C12R8.R DFLAS.7R8.1 <-> MQ.12R8.B2 20 99DQAMGS.B11R8 4 DCQFB.C12R8.L DFLAS.7R8.3 <-> MQ.12R8.B1 20 99DQAMGS.B10R8 3 DCQFD.7R8.R MQ.11R8.B2 <-> DFLAS.7R8.4 18 98DQAMGS.B10R8 4 DCQDD.7R8.L MQ.11R8.B1 <-> DFLAS.7R8.2 16 98DQAMGS.B11L1 3 DCQDQ.11L1.L MQ.12L1.B2 <-> MQ.11L1.B1 32 99DQAMGS.B11L1 4 DCQFQ.11L1.R MQ.12L1.B1 <-> MQ.11L1.B2 32 99

1

Current status of BS-thresholds for the RQ long buses (30/06/2011)

A81.B11R8_RQD(B)A81.B11R8_RQF(B)A81.B11L1_RQD(A)A81.B11L1_RQF(A)

A78.B11L8_RQD(A)A78.B11L8_RQF(A)

ZCh, TE-MPE-TM 1030/06/2011

Thanks!