elder matias canadian light source university of saskatchewan
DESCRIPTION
System Integration and QA. Elder Matias Canadian Light Source University of Saskatchewan. Agenda. The CLS Facility System Engineering Approach Control System Design Instrumentation Design Conclusion. Where is Saskatoon?. Why Saskatoon?. - PowerPoint PPT PresentationTRANSCRIPT
Elder MatiasCanadian Light Source
University of Saskatchewan
System Integration and QA
Agenda
• The CLS Facility• System Engineering Approach• Control System Design• Instrumentation Design• Conclusion
Where is Saskatoon?
Why Saskatoon?
• 1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research.
• Saskatoon was chosen for the CLS due to existing complement of staff and facilities
What are the CLS Objectives?
170.88 m circumference 2.9 GeV DBA lattice with 12-fold periodNominal Tune:
x = 10.22 y = 3.26Eloss per turn: > 0.876 MeVBend magnet radiation: c = 1.6 Å Ec = 7.6 keVx = 18.1 nm•radDamping times:
x = 2.4 ms, y = 3.8 ms, E = 2.7 ms~10 mm bunch length
MAX-IITLS-I
BESSY-IIALS
ELETTRA
PLS
SLS(240m)
ANKA
ESRF APSSpring-8
CLS (171m)SPEAR3 (240m)
Super-SOR LSB BOOMERANGTLS-II
SOLEIL(2006,354m)
NSLS-II
DIAMOND(2007,562m)
ESRFAPS Spring-8ELETTRA
1
10
100
0 1 2 3 4 5 6 7 8 9Energy(GeV)
Em
ittan
ce(n
m·r
ad)
KSRS (124m)
The CLS Project
• 1999-2004– LTB1 (Transfer Line)– BR1 (Booster Ring)– BTS1 (Transfer Line)– SR1 (Storage Ring)– Diagnostic Beamlines
• OSR• XSR
– Scientific Beamline• SGM, PGM (Soft-X-ray)• SM• Mid IR, Far IR• HXMA (Hard X-ray)• CMCF (PX)
• 2005-– Additional 7 beamlines
• 2008-– Additional 6 beamlines
CLS Project Structure
• Nine Work packages:– 0. Administrative Support and Project Planning– 1. Linac Refurbishment– 2. LTB – Transfer Line to Booster– 3. BR1 – Booster Ring– 4. BTS – Transfer Line to Storage Ring– 5. SR1 – Storage Ring– 6. Phase 1 Beamlines– 7. System Integration (Controls and
Diagnostics)– 8. Facility (Building and Mechanical Services)
Collaborate and Reuse
• Internal Collaboration/Reuse– Standardize equipment where it does not impact
scientific capability to reduce development and maintenance time
– Standardize Design Approach Toolkits and Methods across beamlines and accelerator
– Common toolkit and tools across all projects• External Collaboration/Reuse
– Based on analysis of requirements versus available systems
e.g., EPICS, RTEMS, IRMIS, ScienceStudio
Design Package
• A Design Package Includes:– PFD Drawings used by Mechanical Engineering
to capture system layout and critical parameters, e.g., water flow rates etc.
– P&ID Drawings used by Controls to define the inputs and outputs of the system and basic relationships• Partially based on American Instrumentation Society
– Wiring diagrams– Requirements Document (Developed as
required)– PLC and EPICS Software
Drawings
• All Drawings have a unique drawing number• AutoCAD, Inventor, Eagle, Visio• Draft Drawings have letter numbering• Approved Drawings Alpha Numbering• Drawings Review and Approval Process• Sketches have Sketch numbers• As-built captured on master print• Master print in control room, updated by
CAD as time permits
PID Example
PID Example
Change Control (Major Changes)• Major Changes handled through ECR/ECO Process• Minor changes handled through MKS Integrity database
Change Control (Minor Changes)
• Central Database• MKS Integrity
MKS Source– Includes source control– Web and IDE Based
• Implements the concept of “Sandboxes” and “Change Packages”
• Exploring options for using:– MKS Requirements – MKS Deploy
Sample Problem Report
Sample Change Package
Design for Maintainability
• Design systems for the long term
• Phase out and replace older equipment whenchanging standards
• Examples– Replacing 68360 IOC
with Moxa IOCs– Linac upgrade– PS upgrade
Conventional Software EngineeringUsed for Web Services Projects
Con
tent
Time
Slide prepared by IBM for CLS CANARIE Project
Comparison with the Unified Process?
Inception Elaboration Construction Transition
Requirements Analysis Design Implementation Test
Unified Process
Core Workflows In Each Phase
- Establish Feasibility- Establish Business/Scientific Case- Capture Essential Requirements- Identify Critical Risks- Establish initial budget & schedule
Deliverables (as required):- Project Plan- Risk Assessment- Initial Requirements (10-20%)
Requirements: - Refine System ScopeAnalysis: - Establish what to buildDesign:- Create an ArchitectureImplementation- Build an architecture baseline- Build any prototypesTest- Test the architecture baseline- Test any prototypes
Deliverables (as required):- PID Drawings- Wiring Diagrams- Updated Requirements Document- System breakdown
Requirements: - Uncover missing requirementsAnalysis: - Finish the analyis Design:- Finish detailed designImplementation- Build and install the systemTest:- Test and ring-out the system
Deliverables (as required):- Running System
- Correct any defected- Provide support for commissioning - Prepare final documentation
Deliverables (as required):- Final documentation- Working system
CLSI System (Beamline) Engineering ProcessProposal Conceptual
DesignPreliminary
DesignDetailedDesign Build Commissioning
Safety Critical Software
• Applications:– lockup system (ACIS)– Oxygen monitoring– BMIT human studies (under development)
• IEC 61508 – SIL 3 based system• Subject to CNSC Approval• Siemens S7/400 F• Redundant Second Chain• Fail-safe design• Independent Verification• ALARP Hazard Analysis
Hazard Analysis
Risk Class I
Risk Class II
Risk Class IV
Risk Class III
Risk cannot betolerated except in extraordinary circumstances.
Tolerable only if further risk reduction is impractical or if the cost is grossly disproportionate to the improvement gain.
It is not necessary to maintain assurance the risk remains at this level.
As the risk is reduced, the less, proportionately, it is required to spend to reduce it further to satisfy ALARP. The concept of diminishing proportion is shown in the triangle. Based on IEC 61508-5 and IEC 1564
ConsequenceCatastrophic Critical Margina
lNegligib
leFrequency1
Frequent (10-3) I I I IIProbable (10-4) I I II IIIOccasional (10-5) I II III IIIRemote (10-6) II III III IVImprobable (10-7) III III IV IVIncredible (10-8) IV IV IV IV
Frequency Consequence Risk Classification
Initial Risk Probable Catastrophic I Residual Risk Incredible Catastrophic II
No. Constraint/Requirement Allocation
AM 4.1.1.a A lockup (search) procedure shall be developed requiring trained personnel or users to inspect the hutch prior to beam operation.
Procedure
AM 4.1.1.b The procedure shall define a specific configuration of the hutch perimeter doors prior to, during and after the inspection and the inspector is to follow a prescribed search patch. These controls are in place to ensure that on one may enter behind the inspector un-noticed.
Procedure
1.Hutch is not Searched and Secured Prior to Beam OperationAnalysis: The level of radiation present in a beamline hutch when the safety shutters are open is of a potentially lethal level and can not be easily detected by a worker in the hutch, it is conceivable that multiple workers could be harmed. To mitigate this hazard it is necessary to introduce an E/E/PE system (called the Access Control and Interlock System (ACIS)) that requires each hutch to be search and secured prior to beam operation through a lockup sequence.
Mitigation:
• System design based on highly distributed control.• Extensive use of single board computers (originally used in
SAL).• Target lifetime of 15+ years.• Data communication over Ethernet when possible.• System must be user-friendly. • The accelerator and beamline systems must be maintainable by
a small team. • Reliability and availability of beam are critical to the success of
the facility.• Building an open source control system was not the initial goal, it
was the outcome.
• Accelerator complex must be complete by Dec. 2003 and the first phase of beamlines by Dec. 2004. The project must come in on budget.
Control System Design Principles
Distributed Control Systems• The options: (1) EPICS or (2) Isagraph/Virgo.• EPICS was selected, since it had:
– large built up accelerator and beamline user community;– availability of suitable drivers and utilities;– credibility with the CLS user community; and– good design.
• EPICS Extensions selected include:– EDM,– Accelerator Toolbox, – Gateway and– Data Archiver.
• EPICS extensions that were locally developed:– assortment of drivers,– IOC Auto-Save-Restore,– simple beamline scanning program, and– SQL Alarm Management Database.
Control Room/Areas
• Quad Headed Scientific Linux workstations in the accelerator control room and Dual Headed workstations on the beamlines.
• Scientific Linux (CERN/Fermilab)(https://www.scientificlinux.org/)
• Human Factors Engineering
• EPICS Tools– EDM (Display Manager)– Strip Tool (Data Trending)
• CLS Specific– Audio Alarm Annunciation– Legacy hard-wired controls
from older Linac Equipment
• The Options: (1) RTEMS and (2) VxWorks.
• RTEMS was selected, since it had:– good experience from SAL,– additional flexibility with single
board computers, and– high level of reliability.
• IOCs are CLS/SIL embedded controllers (approx 150) based on the MC68360 25 MHz. Processor.
Pros and Cons:– No dynamically loaded libraries;
must be linked prior to download.
– Large number of IOCs (separation of function but more points of failure)
Note: EROCS now replaced with MOXA Linux computers.
Selecting a Real-time OS
Moxa
• Transitioning from SAL single-board-computers to MOXA based IOC
• Linux based• EPICS with the asyn driver and older CLS
serial drivers• Used extensively for
RS-232/422/485
• VME hardware connected to a Linux PC.
• SIS1100 PCI card <-> fiber optic link <-> SIS3100 VME module
• Maps VME backplane to IOC memory.
• Advantages:– PC can be physically separated from VME
crate.– More than one VME crate per PC.– Multiple applications can access the same
crate.– High throughput 25 to 80 Mbytes/sec block
transfer.
• Using RTEMS Real-time operating system.(www.rtems.org)
/dev/SIS1100_2(descriptor 2)
VME CRATE 1:Hardware
mapped memory
VME CRATE 2:Hardware
VME CRATE 3:Hardware
Fiber Optic link
FiberOptic Link
Fiber Optic Link
/dev/SIS1100_1(descriptor 1)
/dev/SIS1100_3(descriptor 3)
EPICSApplication
PV record read
routines
PV record write
routines
Linux or RTEMS IOC
sis1100 PCI card
sis1100 PCI card
sis1100 PCI card
VME
VME CrateFiber Optic Cable (up to 450 m long)
Data Acquisition StationSIS3100
SIS1100
Data Acquisition Computer
• The options: (1) Matlab, (2) SciLab, or (3) root.
• Matlab was selected primarily because of the availability of the accelerator toolbox and staff experience.
• Matlab is commercial, the accelerator toolbox is open source.
• Software originates from ALS and SPEAR III.
• Augmented with other CLS specific utilities.
• Also being used as a commissioning tool for beamlines.
• Special care is required to maintain consistency with other parts of the control system.
Online scripting environment
• Provides fiber optic signal distribution of triggers.
• VXI based hardware• IOC running EPICS on
RTEMS.• Operator
Interface implemented using Glade.
• Glade wasselected forthe table andfile handlingcapabilities.
Timing System
• Single board computers (EPICS/RTEMS) used for:– stepper motors,– power supply control,– vacuum equipment
monitoring,– radiation monitors, and– other RS-232 devices.
• PLC hardware/software used for machine protection.
• Industrial PCs with VME used for diagnostics.
• Linux servers used for high-level control, network services and EPICS/PLC interface.
• MOXA RS-232 Computers
Implementation Strategy
EPICS
ProfibusTCP/IP
Siemens S7/300 PLC
ModbusTCP/IP
GPIB
RS-232
Channel A
ccess Protocol
IOC
IOC
IOC
IOC
State MachineEngine
CA
CA
CA
CA
CA
Single BoardComputer
CA
Operator WorkstationUser Applications
TouchPanels
CA
CA
Telemecanique Momentum PLC
VME
IOCCA
Linac Controls
• Machine Protection– Telemecanique Momentum PLC
• RF– Hardwired + Telemecanique Momentum PLC
• Power Supplies– Old (20+ year) power supplies upgraded (Danfysik +
Brooker)– Now being replaced (IE Power + Agilent)
• Diagnostics– FCT, ICT etc. (Scope)– Spill Monitors (CBLM)– Pop-up Viewers (CCTV + Line Generators)– TRM (Computer based image processing)– Isolated Beam-dumps
BR1 Controls
• Turn-key Danfysik booster• Machine Protection (CLS Design)
– Telemecanique Momentum PLC• RF (Danfysik/ACCEL Design)
– Siemens S7/300– ANKA based electronics– ramped with trigger
• Power Supplies– Danfysik (RS-232)– Ramped Power Supplies, with trigger– Kickers PPT
• Diagnostics– Bergoz BPMs– Bergoz FCT, ICT, PCT– Bergoz Spill Monitors– Striplines– CLS CBLM Spill Monitors– CLS Spill Monitors– Synchrotron Light Monitors (3)– Pop-up viewers (4)
SR1 RF
• Amplifier (Thales)– Siemens S7/400
• Cavity (ACCEL)– Siemens S7/300
• Low Level RF (CLS)– Siemens S7/300
• Cryo Plant (Linde)– Siemens S7/400
SR1 Machine Protection
• Vacuum, Water Flow, Thermal Switches– Telemecanique Momentum PLC
• Vacuum Chamber Temperature– National Instruments FieldPoint
(should have used Momentum)• Fast Orbit Protection
– Custom electronics, – PLC provides thresholds
for comparison– Trip when current < 10mA
based on RF power
SR1 Power Supplies
• IE Power– Ring Lattice Power Supplies– RS-232/485 Slow Control– Special/Custom Interface
for Fast Correctors• Danfysik/PPT
– Kicker Power Supplies– RS-485 + Trigger
• Significant Time Needs to be allocated to tuning new power supplies
SR1 Diagnostics
• Bergoz BPM• Bergoz PCT• CBLMs• OSR & XSR• Agilent VSA• Agilent Remote Scopes• Matlab Toolbox• Envelop Detector• Transient Recorder?• Diagnostic Kicker (under development)
BPM Electronics Selection
• Studies were done on the Bergoz, and Libera Electron units
• This summer we will test Libera Brilliance
• Beamline Controls are based on the same software and hardware as the accelerator systems.
• Each beamline is on a separate virtual network.
• The EPICS Gateway provides links between the different networks.
• Matlab is used for scripting.
Beamlines
Remote Beamline Access Prototype Architecture
BrowserJavascript/Ajax
Netw
ork
Web Server
Other Services- SOAP, etc.
RDBMS
ApplicationLogic SOAP
Beamline - EPICS
J2EE - SPRING
restUI - jaxscript
Persistence - Spring DB
(DERBY)
Spring MVCEPICS Spring
Bridge
POJOs
Login
Selecting a Scan Region
Lightpath Accelerator controls a software virtual cross-connect that commands UCLP.
User Configurable Light Paths
In effect, CA*Net4 is treated as a single lightpath cross-connect
real device real devicevirtual device
domain manager process
UCLPcommands
Mechanical Services
• Telemecanique MomentumPLCs
• Ring temperature stabilityrequirement +/- 0.1 C.
• Geographically Distributed• Legacy Systems:
– Most 1960s equipment upgradedin 2005
– Most 1980s equipment upgradedin 2004
– Limited number of systemsstill using Invensys DMS DCS
Fire Protection
• Notifier System• Smoke Detectors• Laser Detection• VESDA• CO2 Near Oil RF
Systems• Power Trip
– Two Zones Trip– Pull Station
Electrical Services
• MCC (Siemens)– SR1/BR1 - 600 V– Linac - 480V
• Panels– 120 V, 208 V
• Conduit used extensively• For control applications
each rack cluster is on the same phase
• Early morning grid adjustments were problem at times for some power supplies
Grounding
• Grounding routed back to the main transformer yard
• Beamline have isolated grounds, with mixed results.
• Beamline have two separate transformers– Convenience (Dirty)– Isolated (Clean)
• Mechanical System from a Separate Transformer
The End