1CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Review of CMN Problem/StudiesReview of CMN Problem/Studies
• Ariella requested me to review the current understanding of the CMN problem What are the symptoms What studies have been done What we think the most likely source of the problem is
• Also discuss test results of the problem modules test in the current ARCS/LT analysis software How/will the software flag the CMN problem modules Will the software be able to find all of these problems prior to
installation in rods/CMS
2CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Symptoms of CMN ProblemSymptoms of CMN Problem• A channel develops an extremely high noise• When the channel’s noise is higher than 20-25 ADC, the entire chip begins
to oscillate Common mode noise seen, which is not always correctly subtracted At this point, excess bias current always seen
• As low as 0.5 A excess has been seen to cause the problem• Excess current needed to start an oscillation has varied between 0.5-5 A
• Consistent with micro-discharge; dependent on frequency of current spectra of discharge
Problem has (almost) always occurs on first test during voltage ramp up• 1 module at FNAL developed problem during module burn-in with no sign of problems
on effected channel Increased bias current (almost) always seen in IV reprobing
• 1 module at UCSB developed high current after assembly • Rules out module production and assembly as cause of problem
The voltage at which the problem begins has had a wide distribution• Between 50-450 ADC
• No obvious visible damage seen on channels• On three modules probed at Karlsruhe, the bulk of the excess current seen
on channel with the increased noise
3CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Current CMN StatusCurrent CMN Status
• 15 of 73 modules produced at UCSB have CMN problem
• Once notified of problem, FNAL found 6 of 15 previously tested module also had CMN problem In addition, 1 module developed problem during module LT
testing
• 1 TEC module module tested at Karlsruhe also found to have problem
• 2 test beam modules also shown to have CMN problem See L. Borrello’s talk in sensor meeting 1 of 6 ceramic hybrid modules in test beam 1 of 6 flex hybrid modules in test beam
• Both built and test prior to knowledge of problem
4CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN Turn-on VoltageCMN Turn-on Voltage
CMN Turn-on Voltage
0
0.5
1
1.5
2
2.5
3
3.5
50 100 150 200 250 300 350 400 450
Voltage
Nu
mb
er o
f M
od
ule
s
5CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN problem vs. voltageCMN problem vs. voltage
Module 1011
0
5000
10000
15000
20000
25000
30000
35000
40000
0 100 200 300 400 500
Voltage
Cu
rren
t(n
A)
Sensor 1
Sensor 2
Sensor 1 +2
Module Unbonded
•Once, IV diverges from QTC expections, noise on channel increases rapidly causing CMN at 20-60 V above the divergence point
•IN NO CASE WAS THERE AN INDICATION OF NOISE BELOW THE DIVERGENCE POINT OR IN QTC PROBING
6CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Are the faults caused by assembly?Are the faults caused by assembly?
Extensive program of sensor re-probing and additional module IV measurement undertaken Sensors probed prior to assembly in modules
• Sensors with >5 A extra current relative to sensor QTC measurement separated from others
Module then assembled and bias bonded to first sensor• IV measured
Bias is bonded to second sensor• IV re-measured
Module is then fully bonded and tested
• During all measurements, environment controlled Temperature between 23-24 C RH <30%
7CMN Problem Review-Anthony AffolderTPO, December 11, 2003
IV Correlation with CMN problemsIV Correlation with CMN problems
• Significant differences from QTC sensor probing have been found ~7% of sensors have current increases >5 A from QTC prior to
module assembly• Roughly consistent with the rate of occurrence of the CMN problem (aka
micro-discharge) observed at various production sites• The increased current occurs during ramp up during IV probing
• Production Results with IV Pre-Screening Of the 39 modules produced with sensors whose IV curves in the QTC
database matched those obtained in UCSB re-probing, only 1 showed any change in current
• This module showed regular current in some tests afterwards so the problem appears to be intermittent
• Another showed CMN problem with only 0.5 A extra bias current Of the 5 modules with sensors whose IV curves in the QTC database
with 5 extra A of current from those obtained in UCSB re-probing, 4 had serious CMN problems
• Rules out hypothesis that problems due to mishandling in US
• Indicates any change in IV curve relative to original QTC measured a good predictor for sensors that will cause this problem
8CMN Problem Review-Anthony AffolderTPO, December 11, 2003
IV Re-ProbingIV Re-Probing
•Pisa, Perugia, UCSB, FNAL and UR all have begun extensive re-probing program
See sensor meeting Plan on re-probing all sensors not in
modules yet
•6-8% of sensors re-measured from 2001-2002 at all four sites have a 5 A increase in current•Most modules built with these sensors will have CMN problems
If we had not re-probed, we would have 10-20% modules with this problem now
We have NO understanding of the cause of the change in the bias current
• We DO NOT know the time constant/rate for the development of increased
•Therefore, we do not know if more Therefore, we do not know if more sensors would develop higher currents sensors would develop higher currents once built into modules, rods, detector once built into modules, rods, detector
Idb vs Ifermi @ 400V - 2002
0
5
10
15
20
25
30
0 5 10 15 20 25 30
I Production DB (microA)
I F
erm
i (m
icro
A)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
FNAL
PISA
9CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN vs BatchCMN vs Batch
0
5
10
15
20
25
30
35
Number of Sensors
138 143 147 148 150 202 203 216 218 219 220 318 323
Production Week
•Sensors which cause CMN are fairly evenly distributed throughout production years 2001-2002
•Early indications are that 2003 may be better Extremely low statistics Only low bias current sensors used
10CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Common Mode Subtracted Noise (Peak Off)Common Mode Subtracted Noise (Peak Off)
25 ADC
869 881 1010 1011 1013 1014 1015 1016 1030 1031 1038 1042
6.5 ADC
Modules with CMN (micro-discharge problem)Common mode subtracted noise in blue
For majority of modules with problems, the common mode subtraction is imperfect.7 of 12 have >2.0 ADC noise
3 of 12 have >3.0 ADC noise (Two times regular noise)
11CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Common Mode Subtraction VariationCommon Mode Subtraction Variation
• Common mode subtraction results inconsistent Answer differs mode-to-mode or test-to-test Would yield varying signal efficiency/noise during data taking Not clear how this will evolve with time/radiation
Module 1016 Module 1010
12CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Study of Common ModeStudy of Common Mode
• The common mode point is calculated event-by-event for groupings of 32 channels
The spectra of the common mode is fit for groupings within a chip with CMN problems
• Excluding the grouping with high noise channel
• Spectra is fit with two Gaussians Central core plus tail Fit parameters are:
• Fraction of events in tail
• Width of central core
• Width of tail
• Study how parameters vary with current
13CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN Problem Module After Re-ProbingCMN Problem Module After Re-Probing
• Last SS6 module built using one sensor with 1.2 A extra current (450 nA vs 1700 nA) in UCSB re-probing at 450 V.
Well within old selection criteria No large addition increase in
current during module assembly Old sensors
• 30210320274206• 30210320274214
• CMN seen in chip 46 with extremely high noise in channels 423-424
Sensor flaw seen between two channels
• Not clear if flaw cause of problem
Begins at 400 V where database and measured bias current diverge
• ~0.5 A difference
30200020001051
0
500
1000
1500
2000
2500
3000
0 100 200 300 400 500Voltage
Bia
s C
urr
en
t (n
A)
Current(DB)
Current(probing)
Current(Bias Only)
Current (Bonded)
14CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN Problem Module After Re-ProbingCMN Problem Module After Re-Probing
• Module tested at slightly elevated voltage to measure effect as function of current
Bias current 3.7 A, < 2 A more than expected from database
• For first half of chip, CM subtracted noise a factor of ~1.75 higher than typical noise.
A very little amount of micro-discharge can cause the CM subtraction algorithm not to work properly
• CM subtraction algorithm used is same as LT, and test beam software
15CMN Problem Review-Anthony AffolderTPO, December 11, 2003
CMN Problem Module After Re-ProbingCMN Problem Module After Re-Probing
30200020001038
0
10002000
3000
4000
50006000
7000
80009000
10000
0 200 400 600
Voltage
Bia
s C
urr
en
t (n
A) Current(DB)
Current(probing)
Current (Bias Only)
Current(Bonded)
Current (Retest)
Current (Retest 2)
• Micro-discharging strip dis-appears/appears randomly
• But always with the same 2 IV curves
•It IS NOT clear if module which is tested today will be good tommorrow
16CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Module 705 before LT (FNAL)Module 705 before LT (FNAL)• After assembly module was tested (09/08) on ARCS at 400 V and graded “B” (6 faulty channels). No problems observed.
17CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Module 705: LT data (FNAL)Module 705: LT data (FNAL)
Data was taken at 20°C on Sept.19 (10 days after the first test). A group of high noise channels is seen around channel 219 and increased CMN is seen in chip #2
18CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Comparison of IV curves (FNAL) Comparison of IV curves (FNAL)
”before” measurement is taken on 09/08 on ARCS before LT “after” measurement is taken on 09/23 on ARCS after LT green curve is a measurement done using Keithley on 09/24
with 1 minute interval between steps
0
5
10
15
20
applied voltage (V)
lea
ka
ge
cu
rre
nt
(um
)
ARC beforeARC afterkeithley
No visual defects are observed on the sensors around noisy channel #219
We know it is not a humidity effect: 3 weeks in dry air did not cure it
19CMN Problem Review-Anthony AffolderTPO, December 11, 2003
“Fix” For CMN Problem Modules“Fix” For CMN Problem Modules
•In most cases, the CMN can be removed from chip by:
Removing bond from effected sensor
Adding Bond between AC pad (AL strip) and bias ring
•Uses the coupling capacitor as a high-frequency shunt of the increased current
Thus, neighbors do not see noise Increases noise on sensor edges
due to increased current on bias ring
•Does not increase (or decrease) current drawn by module•The long term stability of this fix is not known
Cannot apply this fix once installed on rods, petals, or in detector
20CMN Problem Review-Anthony AffolderTPO, December 11, 2003
How Does ARCS React to CMN ProblemHow Does ARCS React to CMN Problem25 ADC
869 881 1010 1011 1013 1014 1015 1016 1030 1031 1038 1042
6.5 ADC
Common mode subtracted noise in blue
• With standard fault finding, only CMS noise would flag problem
• Of 15 modules, 7 would be graded A or B
• Since CMN varies with time and mode, grading varies with time
•Module’s high current would generally indicate a problem though.
21CMN Problem Review-Anthony AffolderTPO, December 11, 2003
How To Modify Programs To Increase Sensitivity of CMN
How To Modify Programs To Increase Sensitivity of CMN
• Add grading due to bias current directly into program• Add flag of major problem if noise of any single channel
above 20 ADC• Add flag of major problem if average raw noise of a chip
above 2 ADC (Peak) and 2.5 ADC (Deconvolution) • The average raw noise already in output can quantify size of
CMN• RMS of CMS noise per chip could be used as an indicator
of how well the CM subtraction works on module • BUT AS A REMINDER, THERE IS NO GUARANTEE
THAT THE PROBLEM WILL BE THERE AT THE TIME OF THE FIRST TEST !!! 1 module at UCSB has the problem coming and going randomly 1 module at FNAL developed the problem during LT test
22CMN Problem Review-Anthony AffolderTPO, December 11, 2003
ConclusionsConclusions
• We are seeing time evolution of the sensors
• These sensors cause CMN noise on a chip with a turn-on distribution between 50-450 V The noise is not always subtractable The noise varies with time significantly It would be a nightmare to commission/operate a detector of
this size with ~5% modules with this effect
• With pre-probing, the rate of the problem is reduced BUT NO way to know if the sensors will continue to evolve
• The current testing protocol will find the problem if it exists at the time of the test But there are many good reason to believe that many module
WILL NOT have the problem at the time of testing, but will develop it later
23CMN Problem Review-Anthony AffolderTPO, December 11, 2003
IV Test Results (UCSB)IV Test Results (UCSB)
• Environmental conditions tightly controlled Temperature 23.1-23.8 C RH < 30% at all times
• An increase greater than 5 A can cause CMN• Much better results with newer OB2 sensors (2002)• None of the 20 newest (2003) OB2 sensor show any increase in bias
current!!!
Sensors > 2 A
> 5 A
>10 A
>20 A
>100 A
< -2 A
<-5 A
<-10 A
OB2 (’00-01) 15% 9% 8% 5% 1% 8% 3% 1%
OB1 (’00-01) 6% 3% 3% 3% 3% 3% 0% 0%
OB2 (’02) 3% 3% 0% 0% 0% 2% 2% 0%
OB2 (’03) 0% 0% 0% 0% 0% 0% 0% 0%
Probed Current @ UCSB (400 V) – QTC Measurement (400 V)
24CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Could Grounding Cause The Problem?Could Grounding Cause The Problem?
• It is extremely unlikely that grounding could create or enhance the CMN problem
• LV and HV supplies floating Same as the final detector
• Clamshell Module holding plate in clamshell but isolated
• > 1cm from metal shell Grounding achieved with large gauge wire to hybrid-to-
utri adaptors
• Four grounding schemes studied Grounding clamshell/module carrier
• Grounding used chosen because it minimizes the CM noise and sensitivity to environment
• Only changes made relative to standard test stands at time were: Grounding hybrid-to-utri adaptor to test box instead of
module testing plate Use of a thick, continuous, metal shield
• Most centers now have use the same grounding scheme
25CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Noise vs. GroundingNoise vs. Grounding
• CMN seen in both ARCS LT Answer differs mode-to-mode, test-to-test, test stand-to-test stand
• Grounding vastly different ARCS use floating power supplies and “star” grounding
• Only one common point LT have non-floating power supplies and everything is grounded to everything else
• Multiple ground loop
Module 1016 (ARCS) Module 1016 (LT)
Raw Noise CMS Noise
26CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Why isn’t problem seen at test beam?Why isn’t problem seen at test beam?
• Most of modules pre-screened against the CMN problem prior to shipment to CERN
All of UCSB modules sent either do not have problem or have had enough channels pulled in order to remove problem
Most of FNAL modules sent after they began pre-screening for problem• Only 6 modules made with flex hybrids sent without testing at 400 V
• ~50% of the time, 0 of 6 modules would not have this problem• Assumes that the rate of problem and distribution of CMN turn-on voltage constant
• We know that many sensor effects are severely batch dependent
• Many possible reasons why not seen in old prototype modules Many circuits in prototype ceramic hybrids have been changed Testing methods have changed Maybe batch of sensors used in production did not have the problem
• Well, it was in test beam. 2 with turn-on voltage of 400 V See L. Borrello’s talk in sensor meeting
27CMN Problem Review-Anthony AffolderTPO, December 11, 2003
Fit Result of Common Mode PointFit Result of Common Mode Point
• Fraction of events is flat with bias current (~strip current)
• Width of central core increases with bias current (~strip current)
• Width of tail increases with bias current (~strip current) and may flattens out at some current
Fraction of Events in Tail
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0 5 10 15 20 25 30
Bias Current (nA)
Frac
tion
1011
1010
1014
1015
Width of Central Core
0.00
0.20
0.40
0.60
0.80
1.00
1.20
0 5 10 15 20 25 30
Bias Current (nA)
Wid
th (
AD
C) 1011
1010
1014
1015
Width of Tail
0.00
2.004.00
6.008.00
10.00
12.0014.00
16.0018.00
20.00
0 5 10 15 20 25 30
Bias Current (nA)
Wid
th (
AD
C) 1011
1010
1014
1015