2014 epics collaboration meeting - presentation on beast

Managed by UT-Battellefor the Department of Energy

Kay Kasemir

ORNL/SNS

kasemirk@ornl.gov

May 2014

Alarm System Guidelines

2 Managed by UT-Battellefor the Department of Energy

B. Hollifield, E. Habibi,"Alarm Management: Seven Effective Methods for Optimum Performance", ISA, 2007

It’s not about adding 1000ns of PVs to the alarm system

3 Managed by UT-Battellefor the Department of Energy

Help operators take correct actions

Alarms with guidance, related displays

Manageable alarm rate

Operators will respond to every alarm

4 Managed by UT-Battellefor the Department of Energy

Help operators take correct actions

5 Managed by UT-Battellefor the Department of Energy

Help operators take correct actions

Alarms with guidance, related displays

Link to Subsystem displays,operating procedures, telephone numbers, ..

Alarm handled by knowledgeable operators

Help them make an informed decision

No alarms:

Known solution - Automate via records, SNL

Safety - Add (hardware) interlock

6 Managed by UT-Battellefor the Department of Energy

Help operators take correct actions

Managable Rate

Less than 150 alarms/day

Even knowledgeable operator needs time to investigate

Tools should help:Sort alarms by severity, region of the machineOpen displays related to alarmSend emails

7 Managed by UT-Battellefor the Department of Energy

Help operators take correct actions

Operators will respond to every alarm

Corollary to ‘manageable rate’

Don’t..

.. add ‘MAJOR’ alarm in case operators ignore ‘MINOR’.

.. add both ‘Voltage’ and ‘Current’ alarm if one suffices.

8 Managed by UT-Battellefor the Department of Energy

DOES IT REQUIRE IMMEDIATE OPERATOR ACTION?

– What action?

– Consequence of not reacting?

– How much time to react?

Is it the best alarm?– Which PV? Create new PV?

Discuss:Operator, Subsystem expert, EPICS programmer

What’s a valid alarm?

9 Managed by UT-Battellefor the Department of Energy

CSS Integrates Alarms

CS-Studio, 2014 Lanzhou EPICS

9

Context-Menu

Complete Alarm Perspective:Tree view, Table of current alarms

10 Managed by UT-Battellefor the Department of Energy

Alarm: Guidance, Releated Displays

CS-Studio, 2014 Lanzhou EPICS

10

11 Managed by UT-Battellefor the Department of Energy

Integration: Alarm…

CS-Studio, 2014 Lanzhou EPICS

11

12 Managed by UT-Battellefor the Department of Energy

Integration: Alarm…

CS-Studio, 2014 Lanzhou EPICS

12

13 Managed by UT-Battellefor the Department of Energy

Easy: Add Alarm PV to CSS

Open CSS Alarm Tree

Select location in alarm hierarchy, “Add”– Enter PV name

– Guidance: How to respond

– Related screen: Reason for alarm (limits, …), link to screens mentioned in guidance

– Link to rationalization, procedures, .. (wiki)

Anybody authorized can change any of this at runtime– Typo in guidance; wrong telephone number;

additional related display;..

14 Managed by UT-Battellefor the Department of Energy

Harder: Find Good Alarm PV

More than just setting HIGH, HIHI, HSV, HHSV– Screen to show the limit

Dynamic limits, enable based on machine state,…– Screen to show which limit is active and why

HYST is good idea

Maybe use CALC to create alarm trigger from PVs with basic machine readings

15 Managed by UT-Battellefor the Department of Energy

How many “Alarms”?

16 Managed by UT-Battellefor the Department of Energy

Avoid Multiple Alarm Levels

17 Managed by UT-Battellefor the Department of Energy

Alarms for Redundant Pumps

18 Managed by UT-Battellefor the Department of Energy

Alarm Generation: Redundant Pumps the wrong way

Control System– Pump1 on/off status

– Pump2 on/off status

Simple Config setting: Pump Off => Alarm:– It’s normal for the ‘backup’ to be off

– Both running is usually bad as well Except during tests or switchover

– During maintenance, both can be off

19 Managed by UT-Battellefor the Department of Energy

Redundant Pumps

Control System– Pump1 on/off status

– Pump2 on/off status

– Number of running pumps

– Configurable number of desired pumps

Alarm System: Running == Desired?– … with delay to handle tests, switchover

Same applies to devices that are only needed on-demand

11Required Pumps:Required Pumps:

20 Managed by UT-Battellefor the Department of Energy

Basic Alarm Log

1. PV triggers,clears, triggers again

2. Alarm Server latches alarm

4. Problem fixed

3. Alarm Server annunciates

5. Ack’ed by operator

6. All OK

21 Managed by UT-Battellefor the Department of Energy

Web Reports: Managable Rate?

22 Managed by UT-Battellefor the Department of Energy

Bad Example: Old SNS ‘MEBT’ Alarms

Each amplifier trip:≥ 3 ~identicalalarms, no guidance

Rethought w/ subsystemengineer, IOC programmerand operators: 1 better alarm

23 Managed by UT-Battellefor the Department of Energy

Related

ANKA alarm filtering – Igor KriznarMay be easier to configure than “calc record” layers.

‘kryten’ - Andrew Starritt (Australian Synch.)Useful for temporary “Email me if Temp > 10”.

24 Managed by UT-Battellefor the Department of Energy

Summary

CSS Alarm Tools– Check alarms in Table, Tree, Panel

– Fix it: Read Guidance, use Display Links

– ✔Acknowledge

– Configuration can be changed online

Good configuration needs time and effort

Easy

Hard