PEP Run 4 Post-run Review

Operations IssuesM. Stanek

5-Aug-2004

PEPII Run 4 Time Accounting

65%5%

17%

11%

2%

BaBar

PEP Mach. Dev.

Tuning / Filling

Unsched. Down

Sched. Off

9/15/2003 - 7/31/2004 (excluding holday shutdown)

PEPII Run 4 Both Rings Trickle

73%

3%

11%

12%1%

BaBarPEP Mach. Dev.Tuning / FillingUnsched. DownSched. Off

3/12/2004 - 7/31/2004

Operations De-briefing

• Monday, Aug 2 – Ops and EOIC’s met to discuss the run:– General comments– What needs improvement?

• Technical• Safety• Administrative

– What went well?

General notes

• Ops and EOIC’s were “on-shift” for 22 straight months (except Dec holiday). A record!

• Aug/Sep down is a training opportunity.– We will be arranging talks from various PEP

experts to review key systems and present future up-grades.

Positives

• Betatron Tune management– Much easier with constant IHER/ILER ratios (trickle).– Less frequent tune tweaking meant more time for

bump tuning Higher Lumi.• Quicker Diagnosis of RF stuck combs and stuck

tuners.– Ops learned a lot from Dmitry (class, and other

interactions).– Ops (and others) developed tools to deal with some

chronic problems.

Positives (cont.)

• SIT-Match Launch feedback works well.– (go NIT!)

• BPM Orbit overhead displays (CUDs)– More attention to orbits, steering– More display features coming…

• BPM offset, and Status history buffers.

Issues to be worked on

• Injection & BIC• BPM’s / Orbits• RF setup and Abort diagnosis• Luminosity tuning• PPS• General Instrumentation

Injection• NIT-Match beam loss problem

– Beam loss – PLIC signal, correctors max’ed out, orbit offsets required to inject.

– Optics or Magnet problem? Obstruction?

• Energy & E-spread setup for different injection “quanta” too easily lost. Ops ended up manually adjusting e- gun current to fill from scratch.– Some of this solved by more consistent use of

existing tools by ops. – Improvements to Energy feedbacks – independently

measure & control each “quantum” energy.

Injection (cont.)

• Septa bumps tuned for injection are usually not consistent with saved Gold Orbit – hard to diagnose changes.– Diagnostic feedbacks need commissioning

• Injection Kickers – usually able to diagnose timing problems with

existing system. (infrequent)– ops are curious about past and future

improvements, new diagnostics will be needed.

Injection (cont.)

• BaBar trickle background plots (Turner’s talk)– Designed for trickle commissioning, very useful.– Need improvements to serve as a “production”

system:• Color coding – HER (red), LER (green)• EPICS variable names should be more intuitive.• Single display with a few key plots for both rings. • Develop an injection “goodness” parameter – single number

for each ring – history buffered.

Injection (cont.)

• Clarification of when to start “trickle” at end of fills from scratch. – To maximize integrated Lumi, start << peak

current.– BaBar has issues with high trickle rates, but

not consistently expressed, and not understood by ops.

– Ideally, an on-line calculation should aid the decision.

BIC issues

• CPU crash problems, slow-downs, 80% CPU usage with empty rings (!?)– Hopefully being addressed this downtime

• Trickle algorithm does not maintain flat bunch current profile when “trickling up” at top of fill.– Fills one bucket continually, then another

• BIC “maximum trickle rate” is not obeyed when available rate >10.– If 15 hz on tune-up dump, set Max trickle rate = 10,

Injection rate may go to 15.

BPM’s,Orbits & GOF Steering

• New BPM software tools more consistent identification of problems, but...– Putting “repaired” BPM’s back in service takes

coordinated effort:• Should I update the Gold orbit file?• Is it in a Feedback?• Has the GOF setup been edited?• Does it need to be re-timed, and do we have a procedure?• How long will this take, and will it cost lumi?• Will I be able to sort it all out before the next abort?

• BPM auto-calibration issues:– When Lumi drops coincident with new public

calibration - what to do? Was it a step change (feedback?), or a slow change (GOF?)

• Try to tune it back with feedback setpoints?• Back out to an old calibration?• Time/effort is spent restoring lumi – NOT figuring out what

happened.

– How long should we wait after stable beams before allowing new calibration?

• Ops discussion

– How much current is required for “refine calibration”?– Some open questions on software for accept/reject

calibration data. (Ops/SWE working on)

BPM’s (cont.)

• Need a process for evaluating these questions– Maybe a BPM czar?– can he coexist with an Orbit czar?– Ops need the ability to make changes at 2am.

• More discussion needed between Ops, PEP Physicists, Controls.– Spring ’04 meetings were fruitful.

GOF issues

• Problem: A corrector inside the GOF range, but OFFLINE in the GOF setup CANNOT be used in bump tuning.– If an “OFFLINE” Corrector BDES gets moved

(eg. a bump knob) after GOF calculates it’s next changes, but before the changes are implemented, GOF sets the “OFFLINE” corrector back to its pre-calc value.

• Multiple ARTEMIS written, not fixed.

GOF improvement suggestions• Beam current limits for GOF actions

– Would help hold-off GOF during “trickle up”• Slow “ramp” option for large GOF trims.

– Occasionally see lumi drops coincident with GOF trim.– If large changes, slow ramp gives orbit feedbacks a chance to

keep up.• Get GOF information on CUD (BPM orbit CUD).

– SWE work has started.• Changing from a button macro to a stand-alone process

or SCP application or feedback would improve monitoring and manipulation of GOF parameters.

RF Setup and Abort diagnosis

• Real cause of any RF Abort still difficult to determine from fault files. – Immediate remedy is to reduce current.– RF task force evaluates these aborts, but

analysis and practical advice not consistently getting back to Ops.

• We want to help, but what should we be looking for? We aren’t learning the signatures of new fault types.

• What are the “practical” limits on Forward power for each station?

Crippling Instabilities and chronic aborts – i.e. “getting stuck”

• Need more comprehensive Call-In help for “first line” diagnosis of longitudinal instability or repeated RF aborts.– Eg. Long. motion at relatively low current.

• What is causing it? LLRF configuration? LFB hardware? One RF station?

• Call-In book has AMRF and Controls Hardware support.

• Maybe a flow-chart diagnostic aide to narrow it down?

Minor RF issues• 4-3 comb filter module resets at high current cause

aborts – why?• Phase tracking cron jobs

– Effective “band aids”, but difficult to monitor what they are doing.– Which stations are being changed?– How much are they moving?– (can it be done with EPICS?)

• LFB EPICS panels have minor problems.– Amp labels inconsistent, file load gives no confirmation

message.– should be fixed, but panels use old version of EPICS.

• Switching 78 RF stations – Much improved, almost routine, but need to maintain configs for

7 stations when RF is changed in 8 station mode. (Ops)

Luminosity Tuning issues• Luminosity oscillations (10-30 min.) – not

understood.• IP Y-angle feedback “thing”

– IP BPMs measure HER Y-angle (“monitor only” feedback).

• Too positive, flange thermocouple trips• Too negative, lumi suffers

– IP XY Dither Feedback controls Y-angle for local lumi maximum.

• If LER Y-angle drifts, HER tracks it (dither), but may wander too positive or negative.

– Ops manually move combined HER-LER Y-angle until HER Y-angle back to nominal.

• New IP Angle “KNOB” feedback helps– Allows ops to move HER-LER angle with

LERO and IPXY still on – fewer aborts from beams out of collision.

– But – Ops still spend significant time repeatedly tracking the HER Y-angle back within range. Can a feedback be constructed that does what the ops do?

PPS issues• Occasional trips on IR8 BSOIC on routine LER aborts

(estimate ~3%).– May become an issue as currents are raised in the future.

• ROD access for large equipment through Zone 2 and 12 Equipment doors.– Complicated procedure, with non-standard use of PPS controls

to maintain search and avoid searching two zones.– A relatively simple, locally controlled door bypass switch would

save many hours on RODs.• Linac SPPS Chicane BCS Ion chamber trips from HE-

Extraction line losses (or Profile monitor IN)– Relocate it?

General Instrumentation• Bunch-by-bunch “deconvolution” needs work.

– Current ramps are used to compensate errors.• An audible Warning channel for Abort Temps in “Warning” state.• Fast (scrolling) scope for DCCT signals?• The new LER LFB phase scope is useful for instability diagnosis.

– Need to be able to see both rings, need remotely controllable scope.• LFB & TFB diagnostic to tell how close to run-away condition.• More rack space for monitors and scopes.

– BCS hardware is using prime real estate…• More Sun Workstations with multiple “desktops”

– Can handle more displays per monitor, better management of high information density.

• Keyboard/Mouse to control multiple NCD workstations.

What to do with these issues?• Ops will meet and estimate cost/benefit ratios,

and prioritize.– Some become new Software requests.– Some become Ops projects.– Some become classes/lectures from experts.

• Ops will arrange small group meetings with PEP Physicists, Software & Hardware Engineering, others. to discuss possible solutions and next steps.

• If you have ideas for solutions, let us know, we’ll do what we can to help...