case study: critical facilities monitoring system
TRANSCRIPT
Critical Facilities Monitoring System for
Bank of AmericaProper alarm configuration isn’t an option. It’s a
requirement.
PRESENTED BYAllan Evora, PresidentDan Curlin, Sr. Project ManagerAffinity Energy
Affinity Energy: SCADA Experts• 450 mission critical
projects• 175 data center
projects• In-depth system
control, management, and performance monitoring
• SCADA, EPMS, PLC, submetering, etc.
Learning Objectives• Alarming is applicable to all critical monitoring
systems, from data centers to medical campus central utility plants.
• How non-customized communications architecture and alarm configuration can contribute to excess alarming.
• Following alarm handling best practices can provide the operator with a tool rather than a deterrent to effective problem resolution.
• Alarm management can be used as both a real-time and historical tool to support O&M planning, reduce costs, and increase reliability.
Alarming… What’s the Point?Alert via view screen/
text/email/page when a value goes outside a
normal range.
Should signal an abnormal condition.
Should never confirm normally running
processes.
Alarming is a call to action for the operator.
5
Smart Alarming & Customization
with Bank of America
Their Problem • 2002: Bank of America
needed a Critical Facilities Monitoring System (CFMS) for real-time monitoring and alarming
• Challenges:– Identified problems by (literally)
walking through all 13 buildings– Large workforce labor costs– Many manufacturers; how do
you bring all equipment types under one monitoring umbrella?
Challenge: Differing Manufacturers
• Emerson Network Power (Liebert, Alber)
• Schneider Electric (SquareD, Modicon, PowerLogic, APC ION)
• ISO/CAT• Russelectric• GE• Eaton• Siemens• PDI• LayerZero• Trane• Caterpillar• Stulz• Mitsubishi• Carrier• Dynasonics• CellWatch• MultiStack
• 44 UPSs, 149 PDUs, 228 AC units...
8
Affinity Energy’s Critical Facilities
Monitoring System Plan
Typical DevicesCommunication
Gateways
Ethernet Switches &
Network
Low Voltage (>600 V)
Power Meter
Medium Voltage Switches
UPS (4x500kW)
ADAM DiscreteI/O Module
RS485 Serial to Ethernet
gateway
Lantronix uds1100ConneXium TSXETG100
RS485 Serial to Ethernet
gatewayDataModbus RTU/RS485
TCP/IP for Web Interface
Q: How can we bring this all together? A: Create a common network data protocol.
Let's Talk...
Modbus TCP to CFMS Virtual
Servers & Software
CFMS Virtual Servers
Email/Text Paging
HOST2 Windows Server
HOST1 Windows Server
Mail server
Data Server:• Collects data from
devices via Modbus TCP
• Supplies data to View app
Alarm Paging Software:• Monitors alarms
from View app• Emails/texts alarm
messages
Data Historian:• Stores historical data from app• Passes history data to View app for trend charts,
reports• History of alarms, disabled/enabled
HMI View ApplicationEthernet Switches &
Network
Monitoring System Alarming• Alarms = fast
response time without investing heavily on labor
• Text and email alarming
• Redundant alarm paging applications to ensure up-time
• Critical Facilities Team can pinpoint errors quickly before alarms turn into disasters
Avoiding Nuisance Alarms• CFMS only sends
alerts per device, not per alarm
• System notifies when further investigation is needed
• Helps team obtain data to conduct proactive system maintenance
Alarm Hierarchy and Categories• System health (system
wide)• Building• Floor• Area• Device
– Communication– Alarms
Suppressing Alarms
• Enabling and disabling of alarms by building, area, floor, device, or specific point
• Reports to identify what’s enabled and disabled (tied to user’s log-in)
Alarm Reports• User tracking and
timestamp for alarm acknowledgment and disable/enable
• Can be sorted for building, floor, area, device type, and specific device
• Easy to identify reoccurring alarms and perform root cause analysis
• Alarms will prompt team to dig through historical records
Successful Monitoring ofCritical Facilities Portfolio
• 1,160 devices, 70,000 points• Integrated mission critical
assets in 17 buildings in Charlotte, NC (originally 13, at one time 21)
• Capability to easily add or remove devices, areas, floors, or buildings in future
• 13 workforce 2 on-call staff• Saves hundreds of
thousands in manpower, overtime, labor
• Redundant systems ensure up-time
The critical facilities monitoring system Affinity Energy installed has over 70,000 points and integrates mission critical assets in 17 buildings in Charlotte, NC and the surrounding area. Most importantly, Affinity Energy professionally supports what they put in 24/7/365.
-Walter PerrymanOperating Engineer, Jones Lang LaSalle
(JLL)
18
Stop Alarm Apathy: Practical Ideas for
Effective Alarm Management
Stop Alarm Apathy• Categorize and
prioritize alarms• Use custom alarm
messages • Eliminate nuisance
alarms using logic and suppression
• Track who acknowledges alarms
• Alarms should only be for actionable conditions
Which Situations Warrant Alarms?
• Room temperature high• Inverter fault• Circuit breaker tripped• Battery low• Leak detected
• Low fuel level• Hydrogen gas detected• ATS locked out• Generator low coolant
level• Emergency stop
Please Stop Alarming on Events
• High outside air temp• Automatic throw-over
(ATO) in manual *• Generator started *• Circuit breaker open *
• High grid frequency• UPS battery discharging
* Could be alarm candidates based on additional conditions
FYIIf your system does not support events,configure non-actionable conditions as
low priority alarms
Designer/Operator Disconnect
#1 reason events are classified as alarms?
In the absence of an alarm point list in the
plans or specifications, the integrator takes a
conservative approach.
X&$@*Y^??????
KNOW OPERATOR LANGUAGEWRITE ALARMS IN THE SAME LANGUAGE
Well-Formed Alarm Messages• What does operator/alarm recipient need to know?• Messages should be consistent and based on standard,
agreed upon terminology known and used by all operators
T1 LL Alarm vs. Tank 1 LowLow Limit Alarm vs. Fuel Oil Tank #1 approaching LL Limit.
• Messages should tell the operator the problem, then guide towards appropriate action
• Consider all different devices operator will use to receive and process alarm messages
Prioritizing and Grouping Alarms• Categorize based on
severity• Typically, less than 5%
should be classified as top priority
• Proper alarm grouping allows routing to appropriate personnel and facilitates filtering
Alarm Coordination and Scheduling
• Not every alarm needs to go to every operator
• Alarm handling system can route alarms to individuals based on schedule and group– EPSS group– Boiler plant group– Chiller plant group
• Large systems: reduce detailed alarms, use a common alarm
• Deadbands and time delays are additional strategies for reducing nuisance limit alarms
Vs.
Also, make sure your alarms are configured to be retentive in the event of a system reset/reboot.
Alarm Cascading Effect• Single event may spawn a
large number of resulting alarms and events
• Use of conditional alarming and logic can greatly reduce frustration of dealing with tens, hundreds, thousands of alarms flooding monitors, inboxes, or mobile devices
Biggest risk with alarm “avalanches?” The potential
for operators to miss an alarm because it gets buried
in the endless stream of messages.
Alarm Suppression• Great tool in battle against
alarm apathy• If not used properly, could
represent significant risk• 2 types
– Reactive– Proactive
• Requires audit trail, reporting, and high visibility to alarms placed in suppression
Reactive vs. Proactive Examples
Operator prepares to perform preventive
maintenance on a UPS and
suppresses alarms during maintenance
window.
Instrument out of calibration.
Operator receiving nuisance limit
alarms.
Wrong: change limit
Right: suppress alarm until
instrument can be re-calibrated
Recap
• Alarms vs. events• Priorities and groups• Coordination & logic• Clear, consistent
messages• Suppression• Use routing, schedules
and escalation
