case study: critical facilities monitoring system

Critical Facilities Monitoring System for

Bank of AmericaProper alarm configuration isn’t an option. It’s a

requirement.

PRESENTED BYAllan Evora, PresidentDan Curlin, Sr. Project ManagerAffinity Energy

Affinity Energy: SCADA Experts• 450 mission critical

projects• 175 data center

projects• In-depth system

control, management, and performance monitoring

• SCADA, EPMS, PLC, submetering, etc.

Learning Objectives• Alarming is applicable to all critical monitoring

systems, from data centers to medical campus central utility plants.

• How non-customized communications architecture and alarm configuration can contribute to excess alarming.

• Following alarm handling best practices can provide the operator with a tool rather than a deterrent to effective problem resolution.

• Alarm management can be used as both a real-time and historical tool to support O&M planning, reduce costs, and increase reliability.

Alarming… What’s the Point?Alert via view screen/

text/email/page when a value goes outside a

normal range.

Should signal an abnormal condition.

Should never confirm normally running

processes.

Alarming is a call to action for the operator.

5

Smart Alarming & Customization

with Bank of America

Their Problem • 2002: Bank of America

needed a Critical Facilities Monitoring System (CFMS) for real-time monitoring and alarming

• Challenges:– Identified problems by (literally)

walking through all 13 buildings– Large workforce labor costs– Many manufacturers; how do

you bring all equipment types under one monitoring umbrella?

Challenge: Differing Manufacturers

• Emerson Network Power (Liebert, Alber)

• Schneider Electric (SquareD, Modicon, PowerLogic, APC ION)

• ISO/CAT• Russelectric• GE• Eaton• Siemens• PDI• LayerZero• Trane• Caterpillar• Stulz• Mitsubishi• Carrier• Dynasonics• CellWatch• MultiStack

• 44 UPSs, 149 PDUs, 228 AC units...

8

Affinity Energy’s Critical Facilities

Monitoring System Plan

Typical DevicesCommunication

Gateways

Ethernet Switches &

Network

Low Voltage (>600 V)

Power Meter

Medium Voltage Switches

UPS (4x500kW)

ADAM DiscreteI/O Module

RS485 Serial to Ethernet

gateway

Lantronix uds1100ConneXium TSXETG100

RS485 Serial to Ethernet

gatewayDataModbus RTU/RS485

TCP/IP for Web Interface

Q: How can we bring this all together? A: Create a common network data protocol.

Let's Talk...

Modbus TCP to CFMS Virtual

Servers & Software

CFMS Virtual Servers

Email/Text Paging

HOST2 Windows Server

HOST1 Windows Server

Mail server

Data Server:• Collects data from

devices via Modbus TCP

• Supplies data to View app

Alarm Paging Software:• Monitors alarms

from View app• Emails/texts alarm

messages

Data Historian:• Stores historical data from app• Passes history data to View app for trend charts,

reports• History of alarms, disabled/enabled

HMI View ApplicationEthernet Switches &

Network

Monitoring System Alarming• Alarms = fast

response time without investing heavily on labor

• Text and email alarming

• Redundant alarm paging applications to ensure up-time

• Critical Facilities Team can pinpoint errors quickly before alarms turn into disasters

Avoiding Nuisance Alarms• CFMS only sends

alerts per device, not per alarm

• System notifies when further investigation is needed

• Helps team obtain data to conduct proactive system maintenance

Alarm Hierarchy and Categories• System health (system

wide)• Building• Floor• Area• Device

– Communication– Alarms

Suppressing Alarms

• Enabling and disabling of alarms by building, area, floor, device, or specific point

• Reports to identify what’s enabled and disabled (tied to user’s log-in)

Alarm Reports• User tracking and

timestamp for alarm acknowledgment and disable/enable

• Can be sorted for building, floor, area, device type, and specific device

• Easy to identify reoccurring alarms and perform root cause analysis

• Alarms will prompt team to dig through historical records

Successful Monitoring ofCritical Facilities Portfolio

• 1,160 devices, 70,000 points• Integrated mission critical

assets in 17 buildings in Charlotte, NC (originally 13, at one time 21)

• Capability to easily add or remove devices, areas, floors, or buildings in future

• 13 workforce 2 on-call staff• Saves hundreds of

thousands in manpower, overtime, labor

• Redundant systems ensure up-time

The critical facilities monitoring system Affinity Energy installed has over 70,000 points and integrates mission critical assets in 17 buildings in Charlotte, NC and the surrounding area. Most importantly, Affinity Energy professionally supports what they put in 24/7/365.

-Walter PerrymanOperating Engineer, Jones Lang LaSalle

(JLL)

18

Stop Alarm Apathy: Practical Ideas for

Effective Alarm Management

Stop Alarm Apathy• Categorize and

prioritize alarms• Use custom alarm

messages • Eliminate nuisance

alarms using logic and suppression

• Track who acknowledges alarms

• Alarms should only be for actionable conditions

Which Situations Warrant Alarms?

• Room temperature high• Inverter fault• Circuit breaker tripped• Battery low• Leak detected

• Low fuel level• Hydrogen gas detected• ATS locked out• Generator low coolant

level• Emergency stop

Please Stop Alarming on Events

• High outside air temp• Automatic throw-over

(ATO) in manual *• Generator started *• Circuit breaker open *

• High grid frequency• UPS battery discharging

* Could be alarm candidates based on additional conditions

FYIIf your system does not support events,configure non-actionable conditions as

low priority alarms

Designer/Operator Disconnect

#1 reason events are classified as alarms?

In the absence of an alarm point list in the

plans or specifications, the integrator takes a

conservative approach.

X&$@*Y^??????

KNOW OPERATOR LANGUAGEWRITE ALARMS IN THE SAME LANGUAGE

Well-Formed Alarm Messages• What does operator/alarm recipient need to know?• Messages should be consistent and based on standard,

agreed upon terminology known and used by all operators

T1 LL Alarm vs. Tank 1 LowLow Limit Alarm vs. Fuel Oil Tank #1 approaching LL Limit.

• Messages should tell the operator the problem, then guide towards appropriate action

• Consider all different devices operator will use to receive and process alarm messages

Prioritizing and Grouping Alarms• Categorize based on

severity• Typically, less than 5%

should be classified as top priority

• Proper alarm grouping allows routing to appropriate personnel and facilitates filtering

Alarm Coordination and Scheduling

• Not every alarm needs to go to every operator

• Alarm handling system can route alarms to individuals based on schedule and group– EPSS group– Boiler plant group– Chiller plant group

• Large systems: reduce detailed alarms, use a common alarm

• Deadbands and time delays are additional strategies for reducing nuisance limit alarms

Vs.

Also, make sure your alarms are configured to be retentive in the event of a system reset/reboot.

Alarm Cascading Effect• Single event may spawn a

large number of resulting alarms and events

• Use of conditional alarming and logic can greatly reduce frustration of dealing with tens, hundreds, thousands of alarms flooding monitors, inboxes, or mobile devices

Biggest risk with alarm “avalanches?” The potential

for operators to miss an alarm because it gets buried

in the endless stream of messages.

Alarm Suppression• Great tool in battle against

alarm apathy• If not used properly, could

represent significant risk• 2 types

– Reactive– Proactive

• Requires audit trail, reporting, and high visibility to alarms placed in suppression

Reactive vs. Proactive Examples

Operator prepares to perform preventive

maintenance on a UPS and

suppresses alarms during maintenance

window.

Instrument out of calibration.

Operator receiving nuisance limit

alarms.

Wrong: change limit

Right: suppress alarm until

instrument can be re-calibrated

Recap

• Alarms vs. events• Priorities and groups• Coordination & logic• Clear, consistent

messages• Suppression• Use routing, schedules

and escalation

Any Questions?Allan Evoraaevora@affinityenergy.com704-766-2060 x110

Dan Curlindcurlin@affinityenergy.com704.766.2060 x114