review of hardware commissioning may 11-13, 2005 hardware commissioning co-ordination organisation...
TRANSCRIPT
Review of Hardware CommissioningMay 11-13, 2005
Hardware Commissioning Co-ordination
Organisation
Roberto Saban the logic day to day organization safety decision making failure scenarios/fall-back solutions my worries
Roberto Saban TS/HDO HC Review 050511
subsystems ready
the logic
all systems ready for machine check out
individualsystem tests
hardwarecommissioning
InjectionSystem
RF CollimatorSystem
SuperconductingCircuits
ShortCircuitTests
we assume everything is installed before the Hardware Commissioning starts
two types of documents : as designedas tested
impact on equipment safetyimpact on equipment/system performanceimpact on machine performance
Roberto Saban TS/HDO HC Review 050511
preparation
the scopeall the accelerator systems in the LHC tunnel with particular emphasis to superconducting circuits
documentationdocumentation the programthe program the schedulethe schedule the manpowerthe manpower
AB/CO
...
AB/PO
AB/BT
TS/LEA
TS/EL
TS/CV
TS/CSE
AB/ATB
AT/VAC
AT/ACR
AT/MEL
HCC
individual system tests
commissioning
failure scenarios
x 48
Roberto Saban TS/HDO HC Review 050511
preparation : software
requirementsrequirements prototypesprototypes
procedures
commissioning scenario
x 10
sacec : software applications for the commissioning of the electrical circuits
AB/PO
AB/CO HCC
AT/ACR
AT/MEL
AB/CO
QPS Power Converters Powering Interlock
Roberto Saban TS/HDO HC Review 050511
subsystems ready
the as designed documentation
all systems ready for machine check out
defines the pre-requisites to start the tests in terms of conditions and availability of systems
describes in what conditions the system should be delivered to the next step
specifies the objective of the test and either describes each step or summarizes the test procedure
... although special meetings were devoted by the HC WG, many documents describing the individual system tests are missing
Roberto Saban TS/HDO HC Review 050511
preparation : essential time savers
user friendly application suite tool to aid the interpretation of the
test results programmed procedures for battery
tests but also all the classical tools like the
logging system, the alarm system, the post-mortem system, the electronic logbook, etc.
optimization of the MTF configuration for hardware commissioning
exchange of data between the central database and the specialist owned databases
support from the Controls Group
support from the MTF and Database Team
Roberto Saban TS/HDO HC Review 050511
day to day organization: the field
RH87
RE78
RE82
RE12RE88
Point 7
RR73
RR77
UJ 76
PM76
TZ76
RA83
UA83
UJ 83
UJ 84
PM85
PX84
PZ85
UX85
TX84
UL86
UA87
RA87 UJ 86
UJ 87
UW85
US85
UL84
TI 8UJ 88
PGC8
TJ 8
RR13
UJ 13
RT12
UJ 14
US15
TI 12
PM15
PX14
UX15
UL14
UJ 12
LSS4
UJ 17
UJ 18
UJ 16TI 18
RR17
PX16 PX15
USA15
UL16
LHCb
ATLAS
SPS Point 8
Point 1
minor control centres
field control room
arc and even matching section
inner triplets and odd matching section
warm magnets
~210 cold circuits 190 orbit corrector circuits~10 warm circuits
~210 cold circuits 190 orbit corrector circuits~10 warm circuits
Roberto Saban TS/HDO HC Review 050511
day to day organization: the field control room
DFBs
fieldcontrolroom
UA 83 – Left of Point 8
PowerConverters
EnergyExtractionSystems
+ wi-fi coverage of the UA+ printer
+ social corner + water fountain
+ desks+ meeting table+ lockable closets
cryogenics powering tests interlocks post-mortem
Roberto Saban TS/HDO HC Review 050511
day to day organization: the tunnel and the odd extremities
+ wi-fi coverage of the UJ and RR+ desk+ lockable closets
106 m
mobile wi-fi coverage of the tunnel
will it work with the magnet assemblies in place ?
Roberto Saban TS/HDO HC Review 050511
the sequence of tests around an even pointPowering Tests
Roberto Saban TS/HDO HC Review 050511
organisation
communicationcommunication
follow up ofdata entry
follow up ofdata entry
analysisanalysis
commissioningof warm
components
commissioningof warm
components
individualsystem tests
and cool down of sector B’
individualsystem tests
and cool down of sector B’
individualsystem tests
and cool down of sector A’
individualsystem tests
and cool down of sector A’
commissioning of sector B
commissioning of sector B
commissioning of sector A
commissioning of sector A
presence in the fieldpresence in the field
coordinationcoordination
Roberto Saban TS/HDO HC Review 050511
day to day ... coordination : the human resources
x 6
x 9x 9
x 1x 1x 1x 1x 1
operationsupport to operationdata recordingreporting
Field Control Room
Sector A1 Engineer1 Technician
1 AB Operator
Sector B1 Engineer1 Technician
5 people during two shifts
Progress TrackingPlanningRe-schedulingMeetingsCrisis ManagementMTFDatabaseAnalysisInformation processingPublications
Analysis
Project Management
1 Project Leader1 Mr.Circuit & DPL1 Expert Data Analysis1 Database/Web Expert1 Administrative Assistant
5 people during one shift
hardware orientedremoval of FCRsmall repairmissing detailsaccess/balisage
Field Preparation
on request
LHC Safety Coordinationadvise on safety mattersenforce safety in the field
Roberto Saban TS/HDO HC Review 050511
Job Descriptions
Roberto Saban TS/HDO HC Review 050511
resource loading : Hardware Commissioning Coordination
team 1
team 2
team 1
team 2
team 1
team 2
team 1
team 2
team 3
team 3
team 3
team 3
1
2
3
4
[team
s]
Roberto Saban TS/HDO HC Review 050511
personnel safety: assumptions
1. the safety systems (fire and smoke detectors, ODH system, evacuation sirens, emergency lighting, arrêts d’urgence, etc.) are operational
2. posts, signs, flashing lights are in place to announce the ongoing hardware commissioning activity in the tunnel sector or sub sector
3. the access system at the surface allows in only personnel involved in hardware commissioning
4. the panels at the entrance indicating tests in progress will not be installed on time
5. the tunnel and underground areas are ventilated in nominal conditions
Roberto Saban TS/HDO HC Review 050511
personnel safety: the field control room
the issue of the danger related to electricity was discussed on two occasions at the HC WG meetings, but failed to obtain clear directives on the training level of the people present in the FCR; many followed an ad hoc course with dissatisfaction
other issues related to helium venting in the tunnel, fire, number of people present undeground, etc. are known to worry some who are expected to spend a significant amount of their time in the UAs, UJ and RRs.
Roberto Saban TS/HDO HC Review 050511
safety: zonage & balisage
RH87
RE78
RE82
RE12RE88
Point 7
RR73
RR77
UJ 76
PM76
TZ76
RA83
UA83
UJ 83
UJ 84
PM85
PX84
PZ85
UX85
TX84
UL86
UA87
RA87 UJ 86
UJ 87
UW85
US85
UL84
TI 8UJ 88
PGC8
TJ 8
RR13
UJ 13
RT12
UJ 14
US15
TI 12
PM15
PX14
UX15
UL14
UJ 12
LSS4
UJ 17
UJ 18
UJ 16TI 18
RR17
PX16 PX15
USA15
UL16
LHCb
ATLAS
SPS Point 8
Point 1
+ riot barriers+ red & white plastic chains
HC Coordination delimits the areas where particular safety conditions must be enforced during the tests in conjunction with LHC Safety Coordination
Roberto Saban TS/HDO HC Review 050511
safety : personnel underground
Front 1Front 2 Front 1 Front 2
Left Right
6.6 km
in the field on call2t + 2o 1e+1o cryogenics operation1e+1t cryogenics
instrumentation1x+2e+2t quench protection2x+2t 1x+1t+ ... machine interlocks1x+2t 2e+1t+ ... power converters2e+2t coordination1t 1e controls1o operation
4x+5e+10t+3o 1x+4e+2t+1o
o operatorx experte engineert technician
22 present in the field
Front 1 arc and the matching section on the even side
Front 2 arc, the matching section on the odd side and the inner triplets on both sides
two shifts = 15 hour days resources deployed for the powering tests
Roberto Saban TS/HDO HC Review 050511
safety : access during hardware commissioning 1/2
assumptions 1. the sectorization (into access sectors) of the machine is not possible
2. installation is ongoing in other sectors3. the access system will not be operational
states 1. access is not allowed2. access is regulated
• the activity is stopped during an access - if required ...
• for the passage of a convoy, the equipment for the tests is folded so that it is not in the way
regulatedregulated
Roberto Saban TS/HDO HC Review 050511
safety: access during hardware commissioning 2/2
activityaccess
restrictionsarea concerned
first cool down of a sector from room temperature to 80K
no access whole sector
electrical quality assurance (hi-pot)
regulated arc, subsector
short circuits tests no access subsector (RA, RR, UJ)
powering tests1 no access whole sector
instrument checks regulated arc, subsector
equipment checks regulated arc, subsector
validation of systems regulated arc, subsector
(1) above minimum current as during normal operation of the collider
Roberto Saban TS/HDO HC Review 050511
decisions
the commissioning must
be carried-out in safety for the personnel and the hardware,
be according to the approved procedures,
be carried-out in the time expected,
be traceable,
remain within budget.
schedulingimpact on commissioning time
impact on other systems budget, resources
go/no-gorules personnel safety, procedures, traceability
equipment integritycircuit integrity
HCC
HCC + HC Teamowners
HCC
operation & organisationimpact on commissioning time
HCC
Roberto Saban TS/HDO HC Review 050511
communication
electronic media web pages progress reports (logbook, MTF, etc.) e-mails for major milestones/events
regular meetings daily operations meeting in the FCR where every system is
represented including the experiments via LEA weekly reports to GLMs, Team Meeting weekly meetings of HC monthly TCC
extraordinary meetings in case of crisis with concerned Groups, the Project and Department Leaders
Roberto Saban TS/HDO HC Review 050511
failure scenarios
following the suggestion of the MAC in December
2003 and the endorsement of MARIC, failure scenarios during the commissioning - both for Individual System Tests and Hardware Commissioning - were studied
the objective was to define strategies for at least a few case studies per major system in order to gain in reactivity at the time of failures
evaluate the impact on the program and
suggest alternatives to carry-on
six sessions of the HCWG were devoted to this study:
Superconducting Electrical Circuits Cooling and Ventilation System Cryogenic System Vacuum System & Leak Tests Electrical Power Distribution Power Interlock Controllers
can it becircumvented
do itaccordingto definedprocedures
cleardiagnostic
applydiagnosticprotocols
decisioncrash
programme
failure
yes
yes
decision
applydiagnosticprotocols
Roberto Saban TS/HDO HC Review 050511
failure scenarios
Superconducting Electrical Circuits earth fault diagnostic short circuit configuration demineralised water circuit air cooling of the 600 A cables
in the ducts loss of quench detection
instrumentation bad splices start-up with missing circuits
Cooling and Ventilation System redundancy of components blowing of water cooled cables
Cryogenic System redundancy of cooling through
the five feedpoints higher heat loads redundant instrumentation but
no redundant cabling contamination of the helium
inventory helium loss in the tunnel via
safety valves impact of a utility (water,
electricity) failure
Vacuum System & Leak Tests overhead for leaks detected
after the pressure test Electrical Power Distribution
over-voltages (40 to 50% amplitude) for some tens of ms
short spikes Power Interlock Controllers
electro-magnetic compatibility the download of a wrong
configuration file to a controller or a software bug
swapping cables failures at the level of SCADA
failure scenario document in preparation
*
*
*
*
Roberto Saban TS/HDO HC Review 050511
impact of a fault requiring a repair
any failure requiring a repair of a cold part of the collider takes between 10 and 35 days [1]
qualification tests (e.g. elqa) will have to be repeated
commissioning program will have to be rerun: how much depends on where we were when the fault was encountered
during most of this time, the commissioning of the warm systems can continue
5 w
eeks
warm
-up
repair
cool-
dow
n
qu
alif
y
between 15 and 40 days out of ~72
[1] LHC-PM-ES-0002 General Parameters for Equipment installed in the LHC, April 1999
Roberto Saban TS/HDO HC Review 050511
consequence of a failure
Roberto Saban TS/HDO HC Review 050511
... and then there is an exception !
a core team made of a few specialists/experts will meet on short notice to study the problem recommend additional
diagnostics take a decision
go no-go & crisis meeting with
Project and Department Management
something the specialists or the hardware commissioning team do not understand.
Roberto Saban TS/HDO HC Review 050511
can it be circumvented/can it be accepted as is ?
what is the impact of a non-conformity on the performance of
the equipment ? the system ? the machine ?
what is the budget ? have we reached the limit of the budget ?
AB-ABP
LHCOP
Equipment Groups
Hardware Commissioning Core Team
Roberto Saban TS/HDO HC Review 050511
my worries
coordination missing documentation transport & logistics access restrictions not respected safety of tunnel, of field control room, of the other
areas availability of components
major components is not the problem of HC sub-components like instrumentation is
operation thefts noisy environment system testing infrastructure
hardware (e.g. field busses) software (e.g. post-mortem system)
applications utilities (e.g. e-logbook)
LSS8L tests not given the importance it deserves accelerator component failure during the
commissioning