Elder MatiasCanadian Light Source

University of Saskatchewan

System Integration and QA

Where is Saskatoon?

Why Saskatoon?

• 1964 Saskatchewan Accelerator Lab (SAL) was established for chemistry and nuclear physics research.

• Saskatoon was chosen for the CLS due to existing complement of staff and facilities

What are the CLS Objectives?

170.88 m circumference 2.9 GeV DBA lattice with 12-fold periodNominal Tune:

x = 10.22 y = 3.26Eloss per turn: > 0.876 MeVBend magnet radiation: c = 1.6 Å Ec = 7.6 keVx = 18.1 nm•radDamping times:

x = 2.4 ms, y = 3.8 ms, E = 2.7 ms~10 mm bunch length

MAX-II

TLS-I

BESSY-IIALS

ELETTRA

PLS

SLS(240m)

ANKA

ESRFAPS

Spring-8

CLS (171m)SPEAR3 (240m)

Super-SORLSB

BOOMERANGTLS-II

SOLEIL(2006,354m)

NSLS-II

DIAMOND(2007,562m)

ESRFAPS Spring-8ELETTRA

1

10

100

0 1 2 3 4 5 6 7 8 9Energy(GeV)

Em

ittan

ce(n

m·r

ad)

KSRS (124m)

The Science

CLS Operation - 2006

0

50

100

150

200

250

300

350

400

450

Run 21 Run 22 Run 23 Run 24 Run 25 Run 26 Run 27 Run 28 Run 29

Ho

urs

per

Ru

n

0

200

400

600

800

1000

1200

1400

1600

1800

Cu

mu

lati

vel

Ho

urs

Development Beamlines Total Operating Time Total Beamlines

Ring Performance - 2006

0

20

40

60

80

100

120

140

160

Run 21 Run 22 Run 23 Run 24 Run 25 Run 26 Run 27 Run 28 Run 29

Bea

m C

urr

ent

(mA

)

0

20

40

60

80

100

120

140

160

180

200

Inte

gra

ted

Cu

rren

t (A

-h)

Average Run Current Average User Current Amp-Hours User Amp-Hours

Life Sciences11%

Environmental and Earth Sciences

39%

Material & Chemical Sciences

48%

Protein Crystallography

2%

Researchers vs Discipline

0

50

100

150

200

250

Jan-June/05 July-Dec/05 Jan-June/06

General User

Beam Team Members

Purchased Access

The CLS Project

• 1999-2004– LTB1 (Transfer Line)– BR1 (Booster Ring)– BTS1 (Transfer Line)– SR1 (Storage Ring)– Diagnostic Beamlines

• OSR• XSR

– Scientific Beamline• SGM, PGM (Soft-X-ray)• SM• Mid and Far IR• HXMA (Hard X-ray)

• 2005-– Additional 7 beamlines

Drawings

• All Drawings have a unique drawing number• AutoCAD, Inventor, Eagle, Visio• Draft Drawings have letter numbering• Approved Drawings Alpha Numbering• Drawings Review and Approval Process• Sketches have Sketch numbers• As-built captured on master print• Master print in control room, updated by

CAD as time permits

Documents

• Documents – Numbered and Traceable– Under Revision Control– Review and Approval Process– Centrally Stored and Distributed

Agenda

• The CLS Facility• Quality Assurance Program• System Engineering Approach• Control and Instrumentation Design

PID Example

PID Example

Safety Critical Software

• Applications:– lockup system (ACIS)– Oxygen monitoring– BMIT human studies (under development)

• IEC 61508 – SIL 3 based system• Siemens S7/400 F• Redundant Second Chain• Fail-safe design• Independent

Verification

Agenda

• The CLS Facility• Quality Assurance Program• System Engineering Approach• Control and Instrumentation Design

• System design based on highly distributed control.• Extensive use of single board computers (originally used in

SAL).• Target lifetime of 15+ years.• Data communication over Ethernet when possible.• System must be user-friendly. • The accelerator and beamline systems must be maintainable by

a small team. • Reliability and availability of beam are critical to the success of

the facility.• Building an open source control system was not the initial goal, it

was the outcome.

• Accelerator complex must be complete by Dec. 2003 and the first phase of beamlines by Dec. 2004. The project must come in on budget.

Control System Design Principles

Distributed Control Systems• The options: (1) EPICS or (2) Isagraph/Virgo.• EPICS was selected, since it had:

– large built up accelerator and beamline user community;– availability of suitable drivers and utilities;– credibility with the CLS user community; and– good design.

• EPICS Extensions selected include:– EDM,– Accelerator Toolbox, – Gateway and– Data Archiver.

• EPICS extensions that were locally developed:– assortment of drivers,– IOC Auto-Save-Restore,– simple beamline scanning program, and– SQL Alarm Management Database.

• The options: (1) Sun or (2) Linux.

• Linux was selected, since it had:– better hardware

availability, and– fairly equivalent reliability

levels.

• EPICS Extensions selected include:– EDM,– Array Display Tool,– StripTool, and– Knob Manager.

• We are now starting to deploy touch screens running Linux/EDM.

The Operator Interface

Accelerator Dash board

Accelerator Dashboard

Related displays: links to other EDM screens

Implementation

• A group status PV is created to indicate the collective status of a group of components in a sub-system.

Eg. PV LTB1:IOP:status gives the collective status for all the ion pumps in the LTB

• The group status PV monitors all individual PVs in this group and calculate the group status. Different algorithms are used for different types of PVs.

Eg. The most common algorithm finds out the highest alarm level in all the individual PVs and pass it to the group PV.

Linac Screen

Design – LINAC Master Display

Process Screens

• Some screens are done in Visio or AutoCAD then imported into EDM

• The Options: (1) RTEMS and (2) VxWorks.

• RTEMS was selected, since it had:– good experience from SAL,– additional flexibility with single

board computers, and– high level of reliability.

• IOCs are CLS/SIL embedded controllers (approx 150) based on the MC68360 25 MHz. Processor.

Pros and Cons:– No dynamically loaded libraries;

must be linked prior to download.

– Large number of IOCs (separation of function but more points of failure)

Note: EROCS now replaced with MOXA Linux computers.

Selecting a Real-time OS

Moxa

• Transitioning from SAL single-board-computers to MOXA based IOC

• Linux based• EPICS with the asyn driver and older CLS

serial drivers• Used extensively for

RS-232/422/485

• Using VME hardware connected to a Linux PC.

• SIS1100 PCI card <-> fiber optic link <-> SIS3100 VME module

• Maps VME backplane to IOC memory.

• Advantages:– PC can be physically separated from VME

crate.– More than one VME crate per PC.– Multiple applications can access the same

crate.– High throughput 25 to 80 Mbytes/sec block

transfer.

• Work ongoing on RTEMS support.

/dev/SIS1100_2(descriptor 2)

VME CRATE 1:Hardware

mapped memory

VME CRATE 2:Hardware

VME CRATE 3:Hardware

Fiber Optic link

FiberOpticLink

Fiber Optic Link

/dev/SIS1100_1(descriptor 1)

/dev/SIS1100_3(descriptor 3)

EPICSApplication

PV recordread

routines

PV recordwrite

routines

Linux IOCsis1100 PCI

card

sis1100 PCIcard

sis1100 PCIcard

VME

VME Crate

Fiber Optic Cable (up to 450 m long)

Data Acquisition Station

SIS3100

SIS1100

Data Acquisition Computer

• The options: (1) Matlab, (2) SciLab, or (3) root.

• Matlab was selected primarily because of the availability of the accelerator toolbox and staff experience.

• Matlab is commercial, the accelerator toolbox is open source.

• Software originates from ALS and SPEAR III.

• Augmented with other CLS specific utilities.

• Also being used as a commissioning tool for beamlines.

• Special care is required to maintain consistency with other parts of the control system.

Online scripting environment

• Provides fiber optic signal distribution of triggers.

• VXI based hardware• IOC running EPICS on

RTEMS.• Operator

Interface implemented using Glade.

• Glade wasselected forthe table andfile handlingcapabilities.

Timing System

• Single board computers (EPICS/RTEMS) used for:– stepper motors,– power supply control,– vacuum equipment

monitoring,– radiation monitors, and– other RS-232 devices.

• PLC hardware/software used for machine protection.

• Industrial PCs with VME used for diagnostics.

• Linux servers used for high-level control, network services and EPICS/PLC interface.

• MOXA RS-232 Computers

Implementation Strategy

• Beamline Controls are based on the same software and hardware as the accelerator systems.

• Each beamline is on a separate virtual network.

• The EPICS Gateway provides links between the different networks.

• Matlab is used for scripting.

Beamlines

VLANs for: each beamline, machine control, development, office, visitors

VME Crate

(Reflective Memory)

MicroStep

EROCIOC

RTEMS

FieldDev. RS-232

Devices

OPI

Linux

IOCStep Controller

RTEMS

Motors

MicroStep

OPI

Linux

OPI

Linux

Touch PanelOPI Linux

NetworkServer

(bootp, dhcp,auto restore)

Linux

DataArchiveServer

Linux

AlarmServer

MS-Win

MS-SQLServer

MS-Win

PowerEdgeIOC

Linux

PS BoardsIOC

RTEMS

PowerSupplies

EROCIOC

RTEMS

FieldDev.

EthernetDevices

PLC & GPIB

FieldDev.

MagnetsMotors

1GigBridge

IOC

Linux

FieldDev.

ProfibusPLC

Linac Controls

• Machine Protection– Telemecanique Momentum PLC

• RF– Hardwired + Telemecanique Momentum PLC

• Power Supplies– Old (20+ year) power supplies upgraded (Danfysik +

Brooker)– Now being replaced (IE Power + Agilent)

• Diagnostics– FCT, ICT etc. (Scope)– Spill Monitors (CBLM)– Pop-up Viewers (CCTV + Line Generators)– TRM (Computer based image processing)– Isolated Beam-dumps

BR1 Controls

• Turn-key Danfysik booster• Machine Protection (CLS Design)

– Telemecanique Momentum PLC• RF (Danfysik/ACCEL Design)

– Siemens S7/300– ANKA based electronics– ramped with trigger

• Power Supplies– Danfysik (RS-232)– Ramped Power Supplies, with trigger– Kickers PPT

• Diagnostics– Bergoz BPMs– Bergoz FCT, ICT, PCT– Bergoz Spill Monitors– Striplines– CLS CBLM Spill Monitors– CLS Spill Monitors– Synchrotron Light Monitors (3)– Pop-up viewers (4)

SR1 Machine Protection

• Vacuum, Water Flow, Thermal Switches– Telemecanique Momentum PLC

• Vacuum Chamber Temperature– National Instruments FieldPoint

(should have used Momentum)• Fast Orbit Protection

– Custom electronics, – PLC provides thresholds

for comparison– Trip when current < 10mA

based on RF power

SR1 RF

• Amplifier (Thales)– Siemens S7/400

• Cavity (ACCEL)– Siemens S7/300

• Low Level RF (CLS)– Siemens S7/300

• Cryo Plant (Linde)– Siemens S7/400

SR1 Power Supplies

• IE Power– Ring Lattice Power Supplies– RS-232/485 Slow Control– Special/Custom Interface

for Fast Correctors• Danfysik/PPT

– Kicker Power Supplies– RS-485 + Trigger

• Significant Time Needs to be allocated to tuning new power supplies

SR1 Diagnostics

• Bergoz BPM• Bergoz PCT• CBLMs• OSR & XSR• Agilent VSA• Agilent Remote Scopes• Matlab Toolbox• Envelop Detector• Transient Recorder?• Diagnostic Kicker (under development)

Mechanical Services

• Telemecanique Momentum PLC• Geographically Distributed• 1960s equipment upgraded

in 2005• 1980s equipment upgraded

in 2004 Limited legacy systemusing Invensys DMS

Fire Protection

• Notifier System• Smoke Detectors• Laser Detection• VESDA• CO2 Near Oil RF

Systems• Power Trip

– Two Zones Trip– Pull Station

Electrical Services

• MCC (Siemens)– SR1/BR1 - 600 V– Linac - 480V

• Panels– 120 V, 208 V

• Conduit used extensively• For control applications

each rack cluster is on the same phase

• Early morning grid adjustments were problem at times for some power supplies

Useful Commissioning Tools

• MKS Integrity• Data Archiver• Strip Tool• Matlab Accelerator Toolbox

(for storage ring)

The End