data center facility control systems commercial vs. industrial

Data Center Facility Control Systems Confidential – Commercial vs. Industrial of 28 9/10/2007

DATA CENTER FACILITY CONTROL SYSTEMS COMMERCIAL VS. INDUSTRIAL

SUMMARY

Data centers strive for high availability but unlike other mission-critical facilities, data centers are typically built with commercial DDC control systems. Control systems have been identified as the most frequent source of HVAC system problems. Industrial control systems differ from commercial systems in their use of more accurate and rugged sensors and devices, signal types and wiring methods. Industrial controllers are more robust, have higher performance, faster networks and more flexible programming capability. Redundancy options with industrial controls can address the most difficult control issues without relying on “passive automation”. A data center design is presented which includes industrial grade sensors, redundant PLC processors, redundant PLC I/O network, redundant SCADA servers and remote programming and monitoring capability over the internet.

INTRODUCTION

System uptime is the crucial objective of data center operations. To help meet this objective, the Uptime Institute has provided four tier classifications defining site infrastructure performance:

Data Center Tier Classification

Design Features

Availability Percentage

Annual Allowable Downtime Hours

I Single path for power and

cooling distribution, no redundant components

99.67 % 28.8

II Single path for power and

cooling distribution, redundant components

99.75 % 22.0

III

Multiple power and cooling distribution paths, but only one path active, redundant components, concurrently

maintainable

99.98 % 1.6

IV

Multiple active power and cooling distribution paths, redundant components,

fault tolerant

99.99 % 0.8

Table 1: Data Center Tier Classifications - From the Uptime Institute (1)


Guidelines from the Institute propose design topologies and attributes for critical mechanical and electrical systems to attain these tiers. These guidelines are quite useful to facility designers but they do not specifically address one significant component of overall data center system reliability: the control systems that run this critical equipment.

Availability is measured as the percentage of time a system is performing its proper function. Threats from many sources endanger availability. To counter these risks, higher tiers turn to more elaborate designs. For example, a Tier II data center design may include redundant UPS power supplies and redundant chillers. A Tier IV design will include those items plus dual power sources, thermal storage and even dual piping systems. But consider this: a study at Lawrence Berkeley National Laboratories identified controls as the single largest source of HVAC system problems (see below). Its therefore possible that the biggest threat to data center availability may actually be a malfunction or problem in the control system.

Figure 1: Frequency of common problems encountered in a 60 building study performed

by Lawrence Berkeley National Laboratories (2)

Building automation controls are generally regarded as a simpler subset of process control. In most buildings, thermal mass is large, response times are slow and the consequences of system failure are usually not severe. But in data centers, this is not true. If the control system does not respond quickly and appropriately, a data center may experience a destructive and rapid failure - even if redundant chillers, air handlers and power sources have been installed.

Compared to most factories, the physical environment in a data center is benign. Nevertheless, a data center has unique and demanding HVAC requirements. One study by ComputerSite Engineering showed that during a cooling failure, the air temperature in a modestly loaded data center could see a 25 deg F temperature rise in only 10 minutes (see below). As heat densities increase, the response time will decrease to just


a few minutes with an even higher temperature rise. This is enough to seriously disrupt or damage sensitive computer equipment.

Figure 2: Continuous Cooling is Required for Continuous Availability - From the Uptime Institute (3)

In spite of these stringent requirements and the serious consequences of failure, most data centers are built with the same commercial DDC (Direct Digital Control) style control systems used in office buildings. This is in contrast to other mission-critical environments such as semiconductor cleanrooms, or pharmaceutical laboratories, where industrial controls such as a combination of PLCs (Programmable Logic Controllers) with SCADA (Supervisory Control and Data Acquisition) computers or even DCS (Distributed Control Systems) systems perform many of the same functions.

Industrial controls generally cost more than commercial. A rule of thumb for control systems is this: industrial controls total installed cost is approximately $2000/point. Commercial systems cost approximately $1000/point. For reference, a recent data center project was completed with 500 I/O points. This represents a difference of $1M vs. $500K. This does not consider the difference in maintenance and service contract costs (which is typically higher for commercial controls) but is a reasonable idea of the difference in up-front costs. So, besides price, what differentiates industrial from commercial style controls?

Following is an overview of the five main areas where commercial and industrial systems differ:


• Quality of Devices

• Controllers and Software

• Programming Capability

• System Performance

• System Architecture and Redundancy

This paper will describe these differences and how they may be important in the control of a data center. An example of a data center control system built with industrial controls is also presented.

QUALITY OF DEVICES

Automated control starts with the measurement of ambient and system parameters. The measurement process is a chain of sensors, transducers, analog to digital conversion and software processing. Errors and uncertainties at any point in the chain affect the accuracy of measurement and ultimately, the capability of the control system. For both DDC and industrial control systems, the largest source of inaccuracy is typically the sensor itself.

Signal Types & Resolution

DDC systems can use directly connected resistance measurements for temperature, and 0-10 VDC or 4-20 ma for other input signals. Industrial systems nearly always specify 4-20 ma current loops which are most impervious to electrical noise and wiring degradation. Current loops also easily distinguish “loss of signal” from “zero signal”. In commercial installations, sensor wiring is not always placed in conduit. Industrial sensor wiring is almost always in conduit where it is further protected from physical damage and electrical interference. At the controller, input signals are converted from analog to digital with different levels of precision. Commercial controllers typically have 10 or 12 bit resolution (1024 or 4096 bits). Industrial controllers have 14 or 16 bit resolution (16384 or 65536). While not always significant for environmental parameters, higher resolution coupled with more accurate sensors and lower noise signals means better measurement.

Sensors for temperature, humidity, flow, pressure, voltage and current are all used in data centers. Commercial sensors have a minimal accuracy requirement but are normally chosen for their low cost and sometimes, architectural attractiveness. Industrial controls generally use more accurate and robustly packaged devices. Of course, in either setting, there must be enough sensors installed in the right locations to meet requirements for control capability and redundancy.


Temperature Sensors

The most numerous input devices in a data center are temperature sensors. Commercial systems use either thermistors or RTDs (Resistance Temperature Devices) – often with the sensing element wired directly to a controller. Both of these sensor types rely on a change in resistance to indicate a change in temperature. Since there are no transmitters, sensors of this type are less expensive. However, the affect of lead length between the sensor and the controller can impact measurement accuracy.

Figure 3: Commercial temperature sensor - from MINCO website (4)

Industrial systems normally use RTD sensors packaged with built-in transmitters. Overall accuracy can be similar for both thermistors and RTDs (typically +/- 0.5 deg F) but the combination of an RTD with a transmitter provides a standard 4-20ma signal that does not lose accuracy due to lead length differences. RTD themselves have excellent stability, repeatability, and sensitivity. The construction of these sensors can also be more rugged with separate compartments for electronics and terminals.

Figure 4: Industrial temperature sensor - from Rosemount website (5)


Humidity Sensors

Maintaining an optimal humidity level is important in a data center. Typically, data centers are designed to meet the recommendations of ASHRAE TC9.9 which recommends a control range of 40% to 55%. Most important is that relative humidity is maintained above 30% where electro-static discharge can occur and below 55% where computer equipment operates most reliably. Sensors controlling humidification equipment must be accurate, fast-acting and stable. Otherwise, humidifying and de-humidifying may be attempted too late or simultaneously – leading to no control at all.

Commercial grade relative humidity sensors are typically of the resistive type. Normal accuracy rating for these devices is 5%. The better versions of these devices may attain 3% accuracy but still suffer from slow response time and considerable drift. They are sold in plastic cases with little environmental protection. Industrial humidity sensors are usually capacitive type. These are rated at 2% accuracy or better and come with more rugged sensor enclosures and even conformally coated circuit boards. The best sensors of this type have quick response time and little or no drift.

Resistive Type Humidity Sensor Capacitive Type Humidity Sensor used in commercial control systems used in industrial control systems

Figure 5: Humidity Sensor Construction – From NBCIP Product Testing Report: Duct-Mounted Relative Humidity Transmitters (6)

Other Types of Sensors

Pressure, flow and power sensors are also used in data centers. Air pressure sensors are used to reset makeup air fan VFDs for room pressurization control. Chilled water pressure sensors may be used to directly control distribution pump speed. High quality sensors and good installation practice is important for these devices as well.

Output Devices

Output devices are the “muscle” of the control system – the control system elements that actually make something happen. Output devices include motor starters, solenoids, air dampers and control valves. There are both commercial and industrial versions for most of these devices.


Control valves

The most essential HVAC function in a data center is cooling. In data centers that use water cooling, water flow through cooling coils is normally controlled by a modulating globe valve. Commercial versions of this device typically have a bronze body and an electric actuator.

Commercial Control Valve Industrial Control Valve Johnson Controls VG 700 (7) Bauman Little Scotty (8)

Figure 6: Commercial and Industrial Control Valves

Industrial control valves can be very different from their commercial counterparts. Most industrial valves are carbon steel and use a pneumatic actuator complete with a positioner. The positioner senses the actual location of the valve plug and, through a feed-back loop, continually adjusts the output to attain setpoint. Without a positioner, a valve is essentially operating in an “open-loop” mode. This can be much less accurate particularly over time as the valve wears. Additionally, the latest “smart” positioners often have built-in diagnostic capabilities. These capabilities allow maintenance staff to monitor for impending problems or adjust a valve without taking it out of service thus reducing or eliminating downtime.

Air Control Dampers

The motive force for valves and dampers can be either electric or pneumatic. Assuming there is a choice (because not all facility designs include a source of instrument air), each has advantages. Pneumatic actuators typically move much faster. Pneumatic devices require an instrument air connection with all the necessary appurtenances (filters, regulators, gauges, etc.) but electric actuators also must also have a separate source of voltage.


These days, most commercial systems use electric actuators. Industrial systems generally use pneumatics. Unless spring returns are specified, electric actuators have the feature of failing to the last state – i.e. if the control signal or power is lost, the device will simply stop moving. With pneumatic actuators, careful consideration of failure response is necessary including both loss of control signal and loss of air pressure – two potentially different scenarios.

Commercial Electric Damper Actuator Industrial Pneumatic Damper Actuator Belimo AFA24-SR US (9) HyTork XL (10)

Figure 7: Commercial and Industrial Actuators

CONTROLLERS AND SOFTWARE

All input and output signals eventually connect to some sort of controller - the central element of any control system. Commercial systems use a mix of “unitary” controllers typically to control a single piece of equipment and larger building controllers used for facility wide programming tasks or monitoring general I/O points. Industrial systems use PLCs which also come in a in a range of sizes and intended applications. The differences between these controllers can be discussed in terms of form factor and physical robustness, I/O type and capacity, and processor programming capability and flexibility.

DDC Unitary Controllers

These are the basic building blocks of most DDC systems. Unitary DDC controllers are characteristically complete with built-in I/O, network connections and sometimes small operator interface displays. They are built to operate a particular piece of HVAC equipment and can come pre-programmed for specific applications - e.g. a cooling only air handler with an economizer, a fan coil unit with a reheat coil, etc. They normally accept 24 VAC power and are packaged for typical HVAC installations with provision for field wiring termination directly on the controller. Unitary controllers have network capability so they can receive plant-wide data (e.g. outside air temperature) and respond to polling requests for data from a higher level controller.


Siemens Power MEC (11) Johnson Controls FEC (12)

Figure 8: Unitary DDC Controllers

DDC Building Controllers

Larger primary or building controllers perform both a control and a supervisory function. Just like unitary controllers, this equipment may have I/O capability, network connections and can perform local DDC control from a library of pre-programmed HVAC applications. Primary controllers have more memory and the ability to perform more complex controls. They may even provide trending and alarming. More importantly, they carry out site wide scheduling and reset functions and are the communicating device acting as routers that connects unitary DDC controllers to the operator interface computers. At least one building controller is at the heart of most DDC systems.

Siemens MBC (13) Trane Tracer v 17 (14)

Figure 9: Primary Building DDC Controllers


PLC Controllers

PLCs are built for industrial facilities and have many differences from DDCs. These differences start with more rugged packaging and a variety of form factors. Small PLCs, which can be comparable to unitary DDC controllers in function, have a limited amount of program memory and a fixed number of built in inputs and outputs. Typically, expansions are available if the base model does not have enough I/O. Larger PLCs have more memory with expandable I/O connections and can handle a greater amount of I/O. Large PLCs are generally modular which means they have a chassis (or rack) with space for different I/O and communication modules.

Small PLC Large PLC

Allen Bradley CompactLogix (15) Modicon Quantum (16)

Figure 10: Example of PLC Controllers

PLCs offer a wide assortment of I/O card types and densities. Cards can be specified with separate pre-wired terminal blocks, individual channel fusing and the ability to remove and insert cards while under power. As mentioned previously, analog PLC cards typically have greater resolution than DDCs.

Sometimes, a single large PLC can control an entire facility. Several racks can be controlled by a single processor, with capacity for up to thousands of inputs and outputs. A special high speed communication link is used so that racks can be distributed away from the processor, reducing the wiring costs for large plants. This provides possibilities for system architecture not typically available with DDC systems.

Controller Programming

DDC programming languages have evolved from text based versions of high level computer languages like BASIC and PASCAL into graphical drawing versions. A DDC programmer creates a program or control strategy by placing a box representing the function block on the screen and connecting the inputs and outputs appropriately.


Figure 11: Sample of graphical DDC controller programming - from Johnson Controls controller configuration tool website (17)

Once these graphical representations are complete, they are translated or compiled into actual machine readable code and downloaded to the controller. Each vendor has their own programming languages that are specific to their equipment – sometimes several different software products for different controllers. DDC vendors often supply control program templates optimized for specific HVAC functions. The templates can match standard HVAC applications quite perfectly.

Programming a PLC is very different from programming a DDC controller. Like DDC manufacturers, PLC vendors each have their own programming software. In contrast to DDC programs, templates are not normally provided by PLC manufacturers. However, most PLC manufacturers do offer a common software product that programs all of the PLCs they sell. There has also been a significant effort to standardize the programming languages used by all PLCs. IEC 1131-3 is the international standard for programmable controller programming languages. This standard specifies the syntax, semantics and display for a suite of PLC programming languages:

• Ladder diagram (LD)

• Sequential Function Charts (SFC)

• Function Block Diagram (FBD)

• Structured Text (ST)

• Instruction List (IL) The result of this effort is that today, most PLC programs share a common look and feel regardless of the manufacturer. In the USA, PLCs are typically programmed in ladder logic. This visual language is quite familiar to electricians. In fact, its name comes from the hardwired relay control diagrams used to run machinery that look like ladders.


Figure 12: Sample of ladder logic programming

The difference between PLC and DDC programs is essentially one of flexibility. The programming functions in a PLC are more numerous and powerful. There is a richer instruction set for math, logic and bit manipulation. Many PLCs allow encapsulation of instructions to create user defined function blocks. This is a powerful tool that sophisticated users leverage to create simple, re-usable code. These differences allow more sophisticated and powerful programs to be created. Finally, modification of PLC programs can be done “on-line” which means the controllers do not need to be stopped if the program needs to be changed.

Network Connections

Both commercial and industrial control systems generally use a layered network hierarchy. For each layer, there are a wide variety of protocols and media available. The choice of network protocol is influenced by the need for speed, data capacity and number of connections. The type of network media (optical fiber, shielded twisted pair, coax cable, etc.) is chosen based on system topology, distance between nodes, noise immunity and requirements for redundancy. The choice of network type generally dictates what media is required.


Commercial Control Networks

A simplified picture of a typical commercial control system architecture with sensors connected to both building and unitary controllers might look like this:

Figure 13: Sample DDC commercial control system block diagram

A high level network connects the larger primary building controllers and perhaps some unitary controllers to the operator workstation monitoring computers. This network connection is necessary for operators to view data and remotely command changes. Trending and alarming data is received from this network link. Ethernet is the typical choice for this function today.

Lower level networks connect the unitary and building controllers to allow coordinated control functions to occur de-coupled from the system monitoring requirements. If a sensor is located in one area but the control requirement takes place elsewhere, its likely that equipment on different subnets must pass control data over the monitoring network. If this data is delayed, control operation will suffer.

In the past, commercial control system vendors developed network protocols that were particular to their equipment. Today, the HVAC industry has (arguably) settled on the BACnet protocol which allows equipment and devices from different manufacturers to

BACnet over Ethernet

BACnet over MS/TP

Operator Workstation /

BACnet

Building

Controller

S

S

Unitary Controller

S Unitary

Controller

S Unitary

Controller

S Unitary

Controller

S Unitary

Controller

S Unitary

Controller

S Unitary

Controller

S Unitary

Controller

Building

Controller S S Unitary

Controller


function “interoperably”. For some vendors this is still more of a goal than a reality as not all products speak “native” BACnet. These items rely on routers or gateways to allow interconnection of different company’s equipment. While still uncommon, field sensors and devices may also communicate using BACnet.

The BACnet protocol can be transmitted “over” many types of networks including Ethernet, ARCNet, LonTalk and MS/TP. MS/TP (Master Slave/Token Passing) a four wire RS-485 system operating at 76.8 kb/sec is the most common network for unitary controllers. While this is a relatively slow implementation, one positive feature of MS/TP is its deterministic nature – i.e. each network node is guaranteed some time to receive the token and transmit a message. Because there is a known maximum amount of time for all nodes to receive the token, communication is deterministic regardless of network traffic.

PLC Control Networks

PLCs generally have faster, more secure and more numerous network options than DDCs.

Figure 14: Sample PLC/SCADA control system block diagram

Remote I/O Network (ControlNet, ProfiBUS, Modbus Plus or Ethernet)

Large PLC

OPC via Ethernet

S

S

Remote I/O

S Remote

I/O

S Remote

I/O

S Remote

I/O

S Remote

I/O

S Remote

I/O

S Remote

I/O

S Remote

I/O

Large PLC S S

Small PLC

Operator

Workstation / OPC Server


At the controller level, most PLC systems are designed with their own high-speed, deterministic, error-checking protocols. Some typical PLC communication standards are Allen Bradley ControlNet (5Mb/Sec), Siemens Profibus (12Mb/Sec) and Modicon ModbusPlus (2Mbit/Sec). This speed difference is a major reason why PLC system performance can exceed DDC systems.

At the supervisory level, there is an even more powerful method to connect equipment from multiple vendors: OPC (Object Linking & Embedding for Process Control). OPC is a client/server protocol that has rapidly become the standard for industrial controllers to interface with supervisory systems. It works like this: an OPC data server program exposes the available data from a subsystem or seprate controller (e.g. points in a PLC). The client application (e.g. SCADA software) can read these data points as well as secondary information such as data quality to determine if the system is communicating properly and the data is valid. This abstraction provides a simple, interoperable method for computers from different vendors to “talk”. In fact OPC has been so successful it is becoming available as a viable communications method in the DDC world as well.

SYSTEM PERFORMANCE

The two types of systems conceptually can look very similar. So what is the difference? In a word: performance. Industrial systems are designed for “real-time” control. Like a DDC, a PLC program looks at sensor data input, performs logic or calculations and writes outputs. However, the speed of processing and communication in PLC systems is faster than DDC systems. This allows inputs to be read from anywhere in the system, logic solved, and outputs to be written to anywhere else in the system - in real-time.

The time it takes for a PLC to read inputs, solve logic, write outputs and perform overhead functions is called “scan rate”. Scan rates for PLCs, even in large programs with distributed I/O, are generally measured in milliseconds. DDCs have program execution frequencies measures in seconds.

Read Inputs

Solve Logic

Write Outputs

Perform Housekeeping

PLC Scan Rate = Time Required for One Cycle


Figure 15: Explanation of PLC Scan Rate

A temperature control loop may not normally need such a fast update time - but others like exhaust fan or chilled water distribution pressure certainly do. Because of its faster scan rate, the system response to a process upset or setpoint change can be dramatically better in a PLC than a DDC.

Supervisory System Performance

For an operator, the window into the system is the Operator Workstation or SCADA terminal. For either a DDC or PLC based system the supervisory platform provides these functions: representation of system status, ability to change setpoints or commands, presentation of alarms, historical and data trending.

DDC vendors generally offer their own supervisory software package. In the industrial segment, several leading vendors offer SCADA software packages such as Wonderware, Cimplicity/Intellution, Citect, RSView and others. The graphical and user interface capabilities of most of these packages are quite powerful. Today, both types of systems have similar features. In fact, looking at the screens, it may be difficult to tell them apart. But there can be differences.

Supervisory System Update Rates

The faster data is presented, the better. The update rate of a well designed PLC/SCADA system will easily display a new screen and refresh data in less than one second. Some DDC systems can come close to this rate but more typical update rates are 30 seconds or longer. Commands written by an operator should trigger an immediate system response. PLC/SCADA systems can effectively respond immediately but depending on system architecture, DDC systems may take up to 1 minute. Devices that go into alarm in a PLC/SCADA system will be seen within 1 second. DDC system alarms may again take up to 1 minute to appear. These differences are mostly due to the underlying network communication speed and the slower program execution rate in DDCs.

Most PLC/SCADA systems today rely on the OPC protocol for communications between the SCADA computers and PLCs. The OPC interface lets users define “topics” and browse a list of “items” available from each connected device. The update rate of each topic can be set individually. This is a powerful method for tuning the performance of SCADA systems but is not generally available in DDC systems.

OPC itself can be used bi-directionally, that is, an application can sometimes be a data server or a data client. There are occasions when either is necessary. If a facility has both a DDC supervisory and a SCADA program they both must be able to address the question of which application is “on top”.


Data Storage

Most modern SCADA and some DDC systems take advantage of SQL database technology for data storage. The frequency and trigger for storage of data can be defined for each point to optimize speed and memory usage. For instance, a point can be defined to be stored every minute or whenever its value changes outside of a certain dead-band. “Denser” data is important for more meaningful and accurate event reconstruction. Internal or even third party query tools are available to display this data within a trend window or some other type of report. Note that some DDC systems still use proprietary data file types which are not as fast, efficient or accessible as SQL.

Integration with other functions

In a data center there may be other systems that should interface with the supervisory system. Lighting, security and fire alarm programs all may need to do this. Here the historical capabilities of DDC systems gives them a decided advantage. Some SCADA systems can provide this link, particularly if the subsystem can communicate via OPC.

SYSTEM ARCHITECTURE AND REDUNDANCY

Reliability should consider the dependability of individual items but also a system in which a failure in one part doesn’t affect others. With careful engineering, control systems can be designed for fault tolerance, i.e. so that no single failure will prevent the control system from working.

Distributed Control

One common method of achieving controller fault tolerance is to provide distributed control. Valid for either commercial or industrial systems, small inexpensive controllers can be dedicated to individual machines or processes. In this case, the loss of a single controller cannot shutdown the entire facility – if there are redundant pieces of equipment installed each with their own controller.

This type of design may be called “passive automation”. This implies that the system will continue to operate properly even if the automation system is not actively performing its function. For most systems in a datacenter this is certainly true but there are many situations in site wide control that should always function continuously. Some functions cannot easily be duplicated or shared between controllers. In these cases, a single controller must make decisions that require inputs from or outputs to various systems. Here are a few of these cases:

Site Wide Control Issues

• Temperature Control - the output from several sensors may need to be averaged and the output used to reset control loops within several air handlers. At no time must these air handlers be allowed to fight each other (i.e. attempt to provide heating and cooling simultaneously).


• Power restart – the restart of electrical equipment must be coordinated after a power failure. If the system is on generator power, the status of the generator(s) may need to be considered when allowing equipment to restart. A site wide control system that receives and propagates this information in real-time provides a more secure restart.

• Load Shedding – similar to power restart, the ability to shed electrical load is best handled by a control system with fast site wide capability. As the cost of electricity rises the necessity of advanced energy saving schemes increases.

• Humidity control – sensors for room humidity may need to be averaged and used to control humidifying coils or makeup air handlers for de-humidfying. This control typically involves sensors and equipment in various parts of the facility.

• Chiller sequencing interface – chillers usually represent the largest electrical loads in the facility. Direct hardwired control is often used to ensure these critical pieces of equipment start and stop immediately as required. A network link to read operational data is useful but not necessarily recommended for control purposes.

• Pressurization – coordination of exhaust and makeup air to maintain static pressure may require inputs from and outputs to different controllers. These items are usually in several different locations with controllers spread out accordingly.

Redundancy

Instead of distributed control, another method of achieving high reliability is to build a fault-tolerant, redundant control system. With this approach, just a few or even one single controller pair can run the entire facility but no single failure can prevent continuous operation. A good design of this type requires careful consideration of each component of the system. These include:

• Redundant Controllers - for PLC applications that require fault tolerance and high availability, hot standby processors are often used. A hot standby system provides “bumpless” standby control in case of a component failure or power source interruption. The scans of the primary and standby controllers are synchronized. If a failure occurs in the primary controller, the I/O modules hold their last state for one scan while the backup controller takes over. This is a very effective means of control that is not available with DDCs.


Figure 16: Redundant PLC architecture showing 1 pair of hot-standby processors connected to 2 remote I/O drops (15)

• Redundant Network Media – the control network in PLC systems is normally installed in the rugged environment of a factory floor. To prevent loss of control in the event of a damaged cable or failed component, PLC vendors offer dual media capability for their systems. This is not available with DDC systems.

Figure 17: Sample Redundant I/O network cabling (18)

• Redundant Power Supplies – separate power supplies can be provided for PLC controllers. (Note that externally powered devices may also need a separate pair of power supplies.) Two 24 VDC power supplies are connected through a diode redundancy module. PLC manufacturers offer these supplies with factory supplied connectors and built-in monitoring capability. This is a commonly used feature in PLCs that is not typically available with DDCs.


Figure 18: Sample Redundant PLC power supply installation (19)

• Dual Power feeds – where fault tolerance is essential, the redundant power supplies described above can be fed from two separate power feeds. This is quite simple in PLC control panels but not generally feasible with DDC controllers. This concept is quite analogous to supplying dual-power feeds to data center loads.

• Redundant SCADA Servers – While not as critical as the controllers themselves, provision of separate SCADA servers can guarantee access to the control system even if one of the servers fails. This is a feature offered in most SCADA systems and possible with some DDC systems. Note that the SCADA system itself may be inoperable without disrupting the facility itself.


Figure 20: Sample Redundant SCADA server installation (20)

• Separate Supervisory Networks – Most SCADA systems separate networks for data collection (Servers – PLCs) from networks for viewing data (Clients – Servers). This architecture is shown in the diagram above. Segmenting allows uninterrupted data collection no matter how many clients are on-line or what they are doing. This also means that the SCADA servers are not necessarily functioning as operator workstations. DDC systems generally have one computer that functions both as a data collecting server and the operator workstation.

Other Control System Considerations

The following is a list of programming feature or operational tasks that should be compared when considering a particular architecture or controller for data center applications. DDC functionality has increased tremendously in the last few years and the latest systems from the leading suppliers can provide most or all of these features. It still must be said that some of these items are more difficult or impossible to accomplish with DDC systems:

• Ability to “hold last state” - during communication loss or programming downloads, this ability can prevent loss of control or a lengthy recovery period.

• Master/ backup loops – critical control loops are sometimes programmed so that one controller is the master but a second is the backup. In case of a controller failure, the loop continues to operate.

• Hot swap of modules – PLC modules are often designed to be replaced under power. This feature prevents the necessity of powering down a controller to perform a repair.

Commercial Issues

Service contracts – most DDC systems are maintained by the manufacturer that programmed and installed the system. A service contract with the supplier generally covers all aspects of system maintenance including programming.

Whether this is a positive or a negative feature is particular to each individual project. Some facilities prefer to have one entity accountable for the entire control system. Other organizations assume different responsibilities and are comfortable either performing this programming internally or hiring and overseeing proficient experts. PLC’s are broadly supported by system integrators. Facilities are not locked in to one service provider.


Controller Programmers

In contrast to DDC systems, the control system contractors do not normally provide PLC system programmers. A third party system integrator typically fills this role. Interestingly, the same study that identified control systems as the most common source of HVAC system errors also found that programming errors where the largest cause of control system faults.

Figure 21: Categorization of control problems encountered in a 60 building study performed by Lawrence Berkeley National Laboratories (2)

No matter what type of controller is used, there is no substitute for a knowledgeable and dedicated control system programmer. The largest portion of the biggest problem threatening trouble-free operation of a data center may be the control system programmer. Choose them wisely (and pay them well)!

CONCLUSION

Both DDC and PLC/SCADA systems are capable of controlling the facility equipment found in a typical data center. There are differences in performance, flexibility and reliability. Owners and system designers should be aware of the differences and not expect to achieve industrial control system performance on a commercial control system budget.


EXAMPLE OF AN INDUSTRIAL CONTROL SYSTEM FOR A DATA CENTER

Telecarrier, a telecommunications company based in Panama City, Panama, built a data center to supply their clients with a secure and reliable computer co-locating facility in Panama. The overall facility was designed to meet Tier II to Tier III system criteria with the capability of extension to Tier IV.

Equipment Controlled

The electrical system included one power feed, two emergency generators, and one uninterruptible power supply. The mechanical system included 2 air cooled chillers, 2 makeup air handlers, 8 recirculation air handlers for the co-location computer space and several other systems with different levels of redundancy.

A summary of the equipment controlled is shown in the following facility status screen shot:

Sample Facility Status SCADA Screen from Telecarrier data center

Control System Features

The control system design had to reliably control the mechanical equipment and comprehending the variations in redundancy, ensure that the appropriate systems and equipment responded no matter what the failure. The control system was built with the following features:

• 1 PLC with Hot Standby Processors


• 6 Remote I/O Cabinets

• Dual Power supplies with separate feeds to each control panel

• Dual Redundant Modbus Plus Network Cables

• Individually Fused I/O channels

• Industrial grade instruments and valves

• Redundant SCADA servers

• Industrial SQL Server for Historical Trending

• Terminal Server for multiple client access

• Remote Internet Monitoring and Programming Capability

A block diagram of the control system is shown in the following “System Diagnostics” screen shot from the facility SCADA system:

Sample Facility System Diagnostics SCADA Screen from Telecarrier data center

Communications between the remote I/O panels and the PLCs take place over redundant cables to ensure that failure of one network will not stop communications. The hot-standby PLCs are installed so that detection of a failed component on the


primary controller will cause the backup unit to assume control of the system bumplessly within one scan.

The SCADA system was designed with local access provided from a rack mounted operator workstation or from remote clients via a terminal server. The terminal server also allows remote clients to access the system from within the facility or from anywhere in the world via the internet.

Redundant Industrial Sensors

Industrial grade sensors and valves were used throughout the facility. In certain cases, such as the chilled water system, redundant pressure and temperature sensors were installed. These sensors were connected to different I/O cards, sometimes in different control panels. The system was programmed so that operators can choose which sensors to include in the system averaging scheme. As long as one or more sensors is working properly, the PLC can accurately modulate the control loop.

Sample Chilled Water SCADA Screen from Telecarrier data center

PLC Programming

The PLC program was created using Modicon Concept software. This program was written in both ladder and function block formats. It took advantage of user defined function blocks to create standardized, easy to read blocks that were re-used throughout the system.


Sample custom user defined PLC function block from Telecarrier data center

Changing conditions may require the software to be modified over time. The PLC programming software is also configured so that a properly authorized programmer can view or modify programs – even on line.

System Performance

This system has been in operation for since 2003 with no control system issues.

BIOGRAPHICAL INFORMATION:

Steve Blaine is an instrumentation and controls specialist with IDC Architects, a CH2M HILL company. He has more than 25 years experience designing control systems for processes and facilities - including data centers. Mr. Blaine holds a B.S. degree in Electrical Engineering from the University of Pennsylvania and an M.S. degree in Engineering Management from Portland State University. He is a registered professional engineer in the states of Oregon, New Mexico, Arizona, Florida, Texas and Utah, and is also a

licensed electrician in Oregon.


REFERENCES

1. Turner, W. P, Seader, John H. and Brill, Kenneth 1996. Tier Classifications Define Site Infrastructure Performance. The Uptime Institute. Santa Fe, NM.

2. NBCIP: Building Energy Use and Control Problems: Defining the Connection (revised) 2002. National Building Controls Information Program Ames, Iowa

3. Menuet, Rob and Turner, W. Pitt 2006. Continuous Cooling is Required for Continuous Availability: Data Center Managers Need to Match Their Availability Expectations to the Appropriate Class of Continuous Cooling. The Uptime Institute. Santa Fe, NM.

4. MINCO temperature sensor website: http://www.minco.com/uploadedFiles/Products/Temperature_Sensors_and_Instruments/aa20-hvacr_installation.pdf

5. Rosemount temperature sensor website: http://www.emersonprocess.com/rosemount/products/temperature/transmitters_mounting_options.html#field

6. NBCIP: Product Testing Report: Duct – Mounted Relative Humidity Transmitters 2004. National Building Controls Information Program Ames, Iowa

7. Johnson Controls VG700 control valve website: http://cgproducts.johnsoncontrols.com//MET_PDF/977140.PDF

8. Emerson - Baumann Little Scotty control valve website: http://www.documentation.emersonprocess.com/groups/public/documents/bulletins/24.1.ls.pdf

9. Belimo Damper Actuator website: https://www.belimo.us/bellib/Damper_Actuators/AFA24_SR_US.pdf

10. Hytork XL series Actuator website: http://www.emersonprocess.com/valveautomation/hytork/Products/HTML_Pages/Excel_Actuator.htm

11. Siemens Modular Equipment Controller website: http://www.sbt.siemens.com/sbttemplates/library/pdf/149344.pdf

12. Johnson Controls Field Equipment Controller website: http://cgproducts.johnsoncontrols.com//CAT_PDF/1900346.PDF

13. Siemens Modular Building Controller website: http://www.sbt.siemens.com/sbttemplates/library/pdf/149251.pdf


14. Trane Summit v17 website: http://www.trane.com/Commercial/Uploads/Pdf/966/basprc001en0706.pdf

15. Allen Bradley Compact Logix Website: http://literature.rockwellautomation.com/idc/groups/literature/documents/br/1768-br001_-en-p.pdf

16. Modicon Quantum PLC website: http://download.telemecanique.com/85256E540062EA48/all/B47E3BAFEE3C38B885256EB500267C9D/$File/modiconquantum__broc_en_200408.pdf

17. Johnson Controls Controller Configuration Tool website: http://cgproducts.johnsoncontrols.com/MET_PDF/12011147.PDF

18. Modicon Modbus Plus cabling website: http://www.alamedaelectric.com/Modicon%20Documents/PLC%20MB+%20Networking%20IO%20Servicing%20Guide%20v2.0.pdf

19. Allen Bradley power ControLogix website: http://literature.rockwellautomation.com/idc/groups/literature/documents/um/1756-um523_-en-p.pdf

20. Citect SCADA software website: http://www.controsys.hu/anyagok/Redundancy.pdf

data center facility control systems commercial vs. industrial

Business