reliable data centers guideline

Reliable Data Centers

GUIDELINE

Disclaimer

Publisher: BITKOM Bundesverband Informationswirtschaft, Telekommunikation und neue Medien e.V.

Albrechtstraße 10 10117 Berlin-Mitte

Telefon: 030/27576-0 Fax: 030/27576-400

[email protected] www.bitkom.org

Editor: Dr. Ralph Hintemann

Responsible BITKOM body: Working group on operationally secure data centers & infrastructure

Assistant editor Leila Ambrosio

Contact person: Dr. Ralph Hintemann 030/27576-250 [email protected]

Version : November 2006

The information contained in this guide has been carefully researched. It reflects the views of BITKOM at the time of going to press. However, this publication does not claim to be exhaustive. Despite the greatest care on our part, we do not accept liability for the content. The latest guide can be obtained free of charge from www.bitkom.org/publikationen. All rights, including duplication of excerpts, lie with BITKOM.

3

Table of contents

1 Introduction 5

2 Data Center Availability 6

3 The influence of security standards on data center design 8

3.1 ISO 27001 / ISO 17799 8

3.2 ITIL 9

3.3 Sarbanes Oxley Act and SAS 70 9

3.4 Evaluation of standards 9

4 Server Cabinet or Data Center 10

4.1 Secure Server Cabinets 10

4.2 Reliable Data Centers 11

5 Power, Climate Control and Cooling 12

5.1 Power supply companies – Power distribution and feed-in 12

5.1.1 Initial situation 12

5.1.2 Infrastructure operating principles 12

5.1.3 Recommended configurations for different downtimes 13

5.2 Distribution within the facility 14

5.2.1 Initial situation 14



5.3 Uninterruptible Power Supply (UPS) 16

5.3.1 Initial Situation 16

5.3.2 Operating Principles 16

5.3.3 Recommended Configurations for different downtimes 17

5.3.4 Special Considerations 18

5.4 Emergency Power 19

5.4.1 Power generating systems for stand-by power supplies (emergency power) in case of power failure 19

4

5.4.2 Emergency Power Supplies 19

5.4.3 Emergency Power System Configuration 20

5.4.4 Recommended Emergency Power Supply in Relation to Permitted Downtimes 21

5.5 Climate Control 23

5.5.1 Initial Situation 23

5.5.2 Climate Control: A Challenge 23

5.5.3 Infrastructure Operating Principles 24

5.5.4 Air Cooling 25


6 Fire Protection 30

6.1 Active fire protection 30



6.2 Structural Fire Protection 33

6.2.1 Fire Protection Objectives 34

6.2.2 Operating Principles and Room Requirements 34

6.2.3 Recommended Configurations for Different Downtimes 35

6.2.4 Special Considerations 35

7 Floor space design and security zones for data centers 36

8 Appendix 38

9 Glossary 41

10 Acknowledgement 42

11 Notes 43

5

1 Introduction

The planning, construction and operation of IT infrastructure for mission-critical applications in reliab-le data centers represents a real challenge. It is not only choosing the right IT equipment to satisfy an individual company’s requirements which is of the greatest importance. There are other factors which also need to be considered: those involving building design and size, electric cabling, waste heat, venti-lation equipment, availability as well as acquisition and operating costs.

This guide offers support for the planning and implementation of a data center, thus supplementing the existing standards and regulations which one can turn to for support. These often contain only very general requirements, whereas this guide goes into greater depth, to give you specific advice on how to design a data center. It complements the matrix “Planungshilfe Betriebssicheres Rechenhent-rum” (Planning aid for reliable data centers) which, like this guide, can be freely downloaded from the BITKOM website. Excerpts from the matrix are contained in the guide’s subsections. However, both guide and planning aid are no substitute for expert advice and support from competent consultants and planners.

6

2 Data Center Availability

The ongoing development and integration of information technology in all areas of business means that today, no company can afford this technology to fail. Almost without exception, a power supply failure in industry and business leads to collective incapacity.

Only a few years ago, many organisations were able to “survive” an IT Infrastructure outage of several hours, but today the number of companies for whom continuous IT availability is essential is growing steadily. According to a study by the Meta Group, a 10 day breakdown of key IT systems can cause such lasting damage to an organisation that there is a 50% chance it will disappear from the market within the next five years.

Uptime Tier Classification – US Uptime Institute

Tier Classification Introduction DescriptionTier I 1960s Single power distribution path, simple cooling equipment, no

redundant capacity components, 99,671% availabilityTier II 1970s Single power distribution path, simple cooling equipment, red-

undant capacity components, 99.741% availabilityTier III Late ‘80s Multiple power distribution paths, but only one actively red-

undant capacity component, maintenance work possible without disruption, 99.982% availability

Tier IV 1994 Multiple active power and chilled water distribution paths, fault tolerant redundant capacity components 99.995% availability

Source: US Uptime Institute: Industry Standards Tier Classification

These days, whenever an IT plan is drafted, upgraded or even reviewed, a vital factor is the assessment of the company’s IT infrastructure availability. The basic question is:

“What are the customer’s maximum tolerable IT downtimes?”

The following chart gives an outline of the relationship between downtimes and the resulting availabi-lity requirements:

7

% availability in time% Days Hours Minutes Seconds

99.0% 3.650 87.60 5256.0 315360.0099.1% 3.285 78.84 4730.4 263824.0099.2% 2.920 70.08 4204.8 252288.0099.3% 2.555 61.32 3679.2 220752.0099.4% 2.190 52.65 3153.6 189216.0099.5% 1.825 43.80 2628.0 157680.0099.6% 1.460 35.04 2102.4 126144.0099.7% 1.095 126.28 1576.8 94608.0099.8% 0.730 17.52 1051.2 63072.0099.9% 0.365 8.76 526.6 31536.00

The specified time periods refer to downtime

99.99% 0.0365 0.876 52.56 3153.6099.999% 0.00365 0.0876 5.265 315.4

99.9999% 0.000365 0.00876 0.5265 31.54

As a result of growing availability demands on IT infrastructure, it is not only the demands on the IT systems themselves which are increasing, but above all the need for fault-tolerant environmental con-ditions and services. Redundant climate control and power supply configurations, double power feeds and concurrent systems maintenance have become the established standards for high-availability IT facilities.

In future, the central factor in guaranteed continuous availability will be the development and imple-mentation of comprehensive security concepts which take account of all areas of IT security, including the physical security of IT infrastructure. This brings us to the key criterion for companies competing in the market.

The close links between IT infrastructure availability and a company’s survival is one of the reasons behind the trend towards the increase in regulations and statutory guidelines. Therefore, we find the highest growth rates within the IT services sector in the area of security and privacy services. They set the standards for the development and quality of service of the facilities and in doing so, they also defi-ne availability standards.

8

3 The influence of security standards on data center design

A large number of security standards are applicable when data centers are planned and designed. Th-ey provide those responsible with a valuable aid, but they also present a challenge.

The most important standards from ISMS (Information Security Management Systems) and ITIL (IT In-frastructure Library) and from the Sarbanes-Oxley-Act are presented here.

You can find further information on IT security standards in the BITKOM guide “Kompass der IT-Sicher-heitsstandards” (Compendium of IT safety standards) available as a PDF-download from www.bitkom.org/de/publikationen.

3.1 ISO 27001 / ISO 17799

The ISO/IEC 27001 series of standards, which has been in effect since October 2005, has been created to protect information assets from security threats. It is becoming increasingly relevant, as it provides a basis from which companies are able to meet the demands of third parties. These can take the form of statutory requirements (KonTraG), for example, contractual or other requirements. This standard re-places the previously applicable British Standard BS 7799-2, which was withdrawn in February 2006.

The ISO/IEC 27001 provides a framework for the implementation and operation of a security manage-ment system. The process works on the well-known four step principle: “Plan, Do, Check, Act”.

During the planning phase (PLAN-phase) of the process, a risk analysis is carried out in order, among other things, to identify those systems and applications which are critical to a company’s survival and the level of dependence on these respective systems and applications. The results help to identify se-curity requirements and to determine the availability demands on systems and applications.

The implementation phase (DO phase) introduces specific measures for risk minimization and identifi-cation. During this process, the IT security objectives and measures as defined by ISO/IEC 17799:2005, come into play. This outlines the objectives and best practices which are recommended for the design of secure physical infrastructure for IT security areas.

The implemented measures come under regular scrutiny during routine monitoring and periodic audits (CHECK-phase) to identify improvement potential (e.g. fire protection monitoring systems or fire pro-tection tests).

During the fourth phase (ACT-phase), those measures that have been identified as improvements are put into practice.

9

3.2 ITIL

IT service management is an important factor in the planning and operation of an “reliable data cen-ter”. Best practice recommendations for IT service management evolved in the late ‘80s when the Bri-tish government’s Central Computer and Telecommunications Agency (formerly CCTA, now OGC) pub-lished the first elements of the IT Infrastructure Library (ITIL). This set of written guidelines includes detailed advice on individual processes within the ITIL, as well as codes of practice and the recently published ISO 20000 standard.

Customers with existing data centers are turning increasingly to ITIL service management strategies. Service data centers, for example, are noticing that invitations to tender are now stipulating that ten-derer have implemented ITIL procedures. ITIL has two important core areas:

Service Support

Service Delivery

This code of practice is applicable to IT organizations of all sizes.

3.3 Sarbanes Oxley Act and SAS 70

The Sarbanes Oxley Act (SOX) is a US federal law for the improvement of company accounting and re-porting practices. It was passed in response to a number of major accounting scandals involving com-panies such as Enron and Worldcom. The law, which came into force on July 30 2002, not only affects financial data, but it also requires security measures in the IT sector.

The Sarbanes-Oxley Act requires that business procedures be described and defined and that control mechanisms be implemented to minimize the risk of false accounts. Companies are subject to inspec-tion by certified auditors who use the “SAS 70” question catalog.

Where a company affected by SOX has outsourced either individual systems or its entire IT depart-ment, for example, the SAS 70 review will also affect the respective providers. In such cases it is possible either for the customer’s auditor to carry out the SAS 70 review in the service data center or for the data center to have its own audit performed. The auditor’s report must date from within six months of the customer’s fiscal year end. For this reason SOX audits are typically carried out twice yearly, consuming a considerable amount of time and effort.

Possible conflicts between the Sarbanes Oxley Act and national legislation have been debated on an international level. Solutions to most of these conflicts are still pending at the present time.

3.4 Evaluation of standards

These standards are often called into question or scrutinized by customers, certification bodies, audi-ting firms and other interested parties. It is debatable whether Sarbanes Oxley and SAS 70 have made data centers more operationally secure, but the general requirements for improved security measures contained in ISO/IEC 17799:2005 and ISO/IEC 27001:2005 are entirely justified and they make sense. Evidence shows that ITIL and ISO 20000 enhance data center processes and render them more secure.

10

4 Server Cabinet or Data Center

There are two ways of accommodating IT systems securely: either in a separate room or in a secure ser-ver cabinet. Both alternatives do have some similarities, even though they differ in terms of size and location. As their function is to protect the ITC systems and data they house, both are subject to the same fire protection and other safety standards.

4.1 Secure Server Cabinets

Where possible, a secure server cabinet should be of modular design. This offers companies adequate security at a reasonable price. A modular cabinet can be dismantled, modified or reinstalled at another location as required. If the company moves its location, flexible systems of this kind are easier to trans-port and reinstall.

When planning a secure server cabinet – just as in the case of a secure data center – the following cha-racteristics are recommended to ensure continuous system security and availability:

maintain constant temperature and humidity through climate control

maintain power supply through uninterruptible power supply (UPS) and emergency power system where appropriate

safeguards against misuse (locking systems, network surveillance of rack access, biometric data collection)

fire prevention and suppression

modules can be integrated in a central monitoring and management system

Stability is an important aspect of all rack solutions. The high packing density of modern server sys-tems and storage solutions with their network components calls for server racks with load capacities of up to 1000 kg, depending on the application. This means that the base plates and slide rails also need to be dimensioned to support heavy loads. At present, load capacities of up to 150kg per base plate or rail can be dealt with.

Another important factor is cable entry. The trend towards increasingly high-speed networks using copper cabling makes it essential to have separate points of entry for power and data cables to redu-ce the risk of interference between them. When choosing a racking system it is vital to make sure it has easy-to-integrate power distribution, because in the end, it is the power supply which dictates IT availability. There should also be a protected low-voltage distribution panel, as well as a flexible rack power supply system which can be connected to both the mains power and an uninterruptible power supply (UPS). Modern solutions can provide up to 200-amps in a single rack unit. This is made possible through several independent three-phase power input feeds which guarantee a constant power sup-ply, despite rising demands.

11

4.2 Reliable Data Centers

Apart from the aforementioned basic requirements for a secure data center (SDC) there are many de-tails to be clarified concerning the constructional measures of a project.

First the company should carry out an exact risk and weak-point analysis to identify possible risks to the IT systems. Its scope covers those responsible for the planning and construction of a data center, access authorization as well as periodic security checks by independent auditors.

A number of different coordinators are involved in the planning, construction and operation of a data center. Apart from the IT specialists, these can be construction experts such as architects and structu-ral engineers, specialists in the fields of climate control, power or safety, the organization department and last but not least, the management. The physical requirements of a data center do not consist en-tirely of IT issues such as the number and type of servers, networks and storage devices to be installed, but also of threat reporting and defense.

Possible features of such a data center include modular, and thus upgradeable or modifiable, fire-proof and preferably certified security cells. Equally obligatory is the installation of a solid, laminated fire door for protection against intrusion and vandalism. Today there are also other, state-of-the-art components available, such as hermetically sealed wall, ceiling and floor systems offering protection against infiltration by smoke or water and multi-level, very early smoke detection alarms (VESDA) with multiple aspiration tubes also located in raised floors. Furthermore, you will require an appropriately dimensioned, independent fire-suppression system with pressure relief and ventilation duct dampers, a personalized access control procedure using card-readers or biometrics and peripheral LAN video sur-veillance of the data center.

For a more flexible data center design, it is in your interest to work together with those planners and suppliers who can guarantee the long-term availability of their products.

12

5 Power, Climate Control and Cooling

5.1 Power supply companies – Power distribution and feed-in

5.1.1 Initial situation

The power supply plays a vital role in the operation of both server cabinets or entire data centers.

The power supply chain starts at the power supply companies - the producers and suppliers of primary energy. The primary energy producers send the electricity through cables across high voltage pylons to the medium-voltage substations. From here, the power is routed through underground cables to the transformer stations. Transformer stations can usually be found in larger buildings, as well as at pur-pose-built roadside locations.

Large data centers with several thousand square meters of floorspace often have an additional pow-er feed from another, separate, medium-voltage substation. This ensures 100% redundancy – i.e. the duplication of critical components of a system with the intention of increasing its availability – which reaches right up to the power stations themselves.

Possible causes of power outages:

technical faults in equipment (e.g. servers)

technical faults in the power distribution system (e.g. cables, distribution panels)

faults in the backup power system (e.g. emergency power generators emergency diesel generators, battery backed uninterruptible power sources (UPS systems))

process-related faults (e.g. faults in the power system design, logistical faults)

Examples from the recent past show how situations can dramatically escalate when there is a general power failure and no power backup system in place. In the winter of 2005/2006, iced up high-voltage cables in Germany’s Münsterland area caused power transmission pylons to buckle under their weight. The general power supply was paralyzed for several days over large areas. Events such as this demons-trate the necessity of an independent power supply, particularly in highly sensitive sectors such as IT.

In the field of data center construction there are no ready-made power supply solutions on hand. But there are certain power supply principles which need to be individually applied. The challenge for plan-ners is to match these principles to the customer, his needs and requirements and not least, to his bud-get.

5.1.2 Infrastructure operating principles

The ring mains power supply supplies voltage which is then stepped down by transformers to 400 V. It reaches the data center through service cables or conduits, crossing the low voltage main distribution panel and utility power supply network. The network sub panel also provides electricity for the unin-terruptive power supply systems (UPS).

13

The UPS outlet leads to the UPS distribution panel and from there to the individual server cabinets. For this purpose, distribution boxes and connection boxes are designed into the raised floor system. The power is then directed through further wires from the distribution or connection boxes to the server power supply unit (PSU) in the cabinet. Where there is only one UPS system, both A and B in-feeds should be jointly supplied. Where there are two UPS systems, they should be fed separately.

5.1.3 Recommended configurations for different downtimes

RZ K

ateg

orie

Permitted

Data Center

Dow

ntime

300 to 1.000 Watt/qmFrom 5 kW to 24 kWUp to 5 kW

Data Center/ Server Room

Server CabinetServer Cabinet

Distribution

Approx. Requirements

Infrastructure

0 min

E

Redundant configuration

PowerTechnical infrastructureto Tier II / Tier IIV** standard

10 min

D


PowerTechnical infrastructureto Tier III / Tier IV ** standard

1 hC

Redundant configurationPowerTechnical infrastructureto Tier III ** standard

24 h

B

Standard, Qualified electrical personnel on call

PowerTechnical infrastructureto Tier I / Tier II ** standard

Data

Cen

ter C

ateg

ory

A

Standard, dependent on power supply units


Standard 72 h

Fig. 1: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” Utility Power Supplies

Categories A and B are currently in operation in many companies within the small business sector. Of-ten, they do not even have a connection for a mobile emergency power unit (EPU). On closer inspec-tion, this alternative offers no real security, as it relies solely upon the power supply companies. Time and again you hear people say, “…. it’ll be all right. We’ve never had any problems up to now …..”. But if only one link in the supply chain breaks, the whole utility power input fails and the UPS unit then has to provide the power.

But the performance of UPS units is often very modest. It depends on the capacity of the batteries and the demand. Several hours of downtime cannot normally be bridged by a UPS unit. In such cases, a functioning computer shutdown routine should be automatically triggered, issuing notifications, sa-ving data, closing down applications and finally shutting down the computers themselves.

It is thus important for planners to ensure that the performance time of the UPS system is greater than the time needed for the transportation and connection of a mobile EPU. Under the operating condi-tions mentioned above, batteries with a backup time capacity at full load of up to four hours and more are usually employed.

14

Category C offers greater security potential. Here the redundant power supply starts right from the main distribution panel. Should a supply route behind the distribution panel fail, the power supply is automatically routed through a second, redundant channel. In case of primary power provider break-down, the electricity supply is still guaranteed by the mobile emergency power system.

Unlike Category A, Category C has a second, additional UPS with its own two UPS sub-distribution pa-nels. This ensures a redundant supply from the UPS system to the servers’ power supply units.

The striking feature of Category D, as compared to Category C, is that the power supply is fed from two separate transformer substations with their respective downstream infrastructures. Additionally, should there be a utility power failure, the permanently installed emergency power system will take over power production and supply. This category represents a very high level of security.

Category E is the ultimate in reliability. Not only is there additional redundancy from a second emer-gency power system, but there is also an additional power feed from another, independent medium voltage substation. It is, however, almost always the case that the primary power company first has to install this second supply line from a different medium voltage substation. This can mean that several kilometers of new cable have to be laid to the new data center location, a factor which is highly cost-in-tensive and which should be allowed for in the project cost estimate.

Key factors in maintaining availability are the regular servicing of the entire infrastructure by qualified personnel and adherence to the systems’ operational guidelines and procedures.

5.2 Distribution within the facility

5.2.1 Initial situation

Power from the utility network, the generator and the UPS is supplied to equipment, systems and lighting across the current distribution panel. Greater levels of availability can be guaranteed by using two distribution panels.


The utility network provides the power for the building infrastructure including elevators, lighting (apart from emergency lighting installations to VDE0108 standard), compressors in DX-air conditioning units (DX = direct expansion), water chillers and other installations. If there is a mains failure, this pow-er supply is interrupted until an in-house backup generator kicks in and an automatic transfer switch restores the electricity supply.

All power distribution units must have an input fuse. The size and design of the PDU depends upon the required power output, the number of circuits and the current capacity per circuit. There are both sing-le-phase and three-phase circuits. Typical capacities are 16A, single-phase (approx. 3.5 kW), 32A single-phase (approx. 7 kW) or 32A three-phase (22 kW) for high-performance cabinets. A particularly difficult topic is the so-called “selective fuse co-ordination”, which makes it possible to safely isolate an IT unit in a cabinet if it short-circuits, while keeping the remainder of the cabinets and IT units operational. Fu-se tripping times of a maximum of 10 milliseconds must not be exceeded in the process.

15

In today’s data centers, most IT units are installed in 19” cabinets. This raises the question of where to locate the distribution panels and how to route the power cable to the 19” cabinets. Distribution pa-nels come in the form of recessed or surface-mounted models, separate cabinets as well as integrated versions built into 19” cabinets. The cables are often routed under the raised floor, which is also used to channel cold air. The air flow can be affected and access to the cables is more difficult. This can be avoided by having cables feed up through one of the raised floor panels into the bottom of the19” cabi-nets. Alternatively, cables can be run in overhead or vertical trays, which means they enter the 19” ca-binets from above. Integrated distribution panels have the advantage that they are in the immediate vicinity of the point of use and thus in easy reach of the 19” cabinets. It is also possible to run cabling on top of the 19” cabinets as long as provision has been made for data and power cables to be run sepa-rately.

Particular attention must be paid to the power distribution strips inside the cabinets. Today, large numbers of IT units can be installed into any one cabinet. One of the most elaborate examples is a ca-binet with 42 rack units (U) which can house 42 1U servers with two power supply units per server. This requires a total of 84 power sockets.


RZ K

ateg

orie

Permitted

Data Center

Dow

ntime




Distribution


Infrastructure

0 min

E



10 min

D



1 hC

Redundant configurationPowerTechnical infrastructureto Tier III ** standard

24 h

B

Standard, Qualified electrical personnel on call


Data

Cen

ter C

ateg

ory

A

Standard, dependent on power supply units


Standard 72 h

Fig. 2: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” - Distribution

The level of redundancy depends upon the number of power supply units in the IT equipment. To en-sure high availability, it is advisable to have two redundantly configured power supply units per IT unit, so that if one fails, the remaining PSU is able to continue providing the IT unit with a normal power supply. However, it is important that the power distribution system should feed these two power sup-ply units per IT unit via two separate power strips on two separate circuits.

16

Even higher availability can be achieved by using two separate distribution panels which are in turn served by two separate UPS units via two separate transformers and two separate generators.

5.3 Uninterruptible Power Supply (UPS)

5.3.1 Initial Situation

Power outages often happen for banal reasons: even simple voltage fluctuations or short power losses in the power grid can be enough to damage hard or software or to cause such disturbance that major faults occur. Mains perturbations abnormalities may be rare occurrences, but they happen more often than is commonly assumed.

Number of power outages and short interruptions per annum

Most of the dangers lurkin the millisecond range

0 – 10 ms 10 – 20 ms 20 ms – 1 s 1 s – 1 h > 1h

60

50

40

30

20

10

0

Duration of power outages

60

50

40

30

20

10

Duration of power outages

Fig. 3: Frequency of electricity network disturbances in relation to their average duration

UPS systems are used to help avoid possible negative consequences of short power outages of this kind. They filter out aberrations such as voltage surges or dips and bridge gaps in the power supply network. In this way, transmission errors, system crashes, program errors and data loss can be avoided.

5.3.2 Operating Principles

Static UPS models are grouped in three categories. The classification of static UPS systems and the cor-responding classification methods are defined and described in the European Standard EN62040-3.

17

VFD

10.

9.

8.

7.

6.

5.

4.

3.

2.

1.

continuousVoltage harmonics

Voltage+FrequencyIndependent

sporadicFrequencyfluctuation

< 4 msVoltage surges

periodicVoltage Distortion(Burst)

continuousOver-voltage

EN 62040-3e.gDurationGrid Disturbances

4…16 msVoltage transients

< 16 msVoltage fluctuation

VI *)

VFD

sporadicLightning effects

continuousUnder-voltage

…………..

…………..

…………..

…………..

…………..

Surge DiverterSolution

…………..

…………..

Lightningdiverted and voltage surgeprotectionIEC 60364-5-534)

…………..

…………..

UPS Solution

Classification 2

Classification 3> 10 msUtility power failures

Grid Distubances and UPS Solutions

Classification 1Double-Conversion-Betrieb(Online)

PassiverStandby Mode(Offline)

Line-Interactive-Mode

Voltage+Independent

Voltage+FrequencyDependent

*) Alternative methods are able to deal with grid disturbances nos. 1 – 9 Fig. 4: Types of mains disturbance and the appropriate UPS solutions according to the EN62040-3 Standard (Ref.: “Uninterruptible Power Supply – European Guide”; Editor: ZVEI, 2004)

5.3.3 Recommended Configurations for different downtimes

UPS systems are basically dependent upon the electric power requirement of the critical loads to which they are connected, and upon the deployment situation. Other important factors are the redundancy concept and the input and output voltage supply. When UPS systems are installed in server cabinets or as individual UPS racks in rooms that also house IT equipment, the alarm and fire prevention system must make allowance for the additional thermal load from the batteries.

18

RZ K

ateg

orie

Permitted

Data Center Dow

ntime




UPS


Infrastructure

E

Redundant configuration, 10 – 30 min. back up


D

Redundant configuration, 10 – 20 min. back up


CRedundant configuration, 10 – 20 min. back up

PowerTechnical infrastructureto Tier III ** standard

Data

Cen

ter C

ateg

ory

A StandardA minimum 10 min. bridging period. Maximum backup time dependent upon the controlledserver shutdown period


StandardA minimum 1 h bridgingperiod. Maximum backup time dependent upon the controlledserver shutdown period

B StandardA minimum 10 min. bridging period. Maximum backup time dependent upon the controlledserver shutdown period


StandardA minimum 1 h bridging period. Maximum backup time dependent upon the controlledserver shutdown period

0 min

10 min

1 h

72 h

24 h

Fig. 5: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” - UPS

5.3.4 Special Considerations

Additional important project planning criteria for the dimensioning and installation of a UPS system are:

Nominal power rating at the required power factor (today almost 1)

Connection specifications such as input and output voltage, frequency

Currents, wiring cross-sections and connection options for UPS input and output

Efficiency and power loss

Feedback effects on power supply and input power factor

Available battery backup time at actual load

Maximum available battery backup time at nominal load

Battery system and battery charge/discharge specifications

Permitted ambient parameters such as operating temperature and humidity; effective degree of pro-tection; fire safety and climate control requirements

Noise generation

Electromagnetic compatibility (EMC) protection

Weights and dimensions

19

This guide does not aim to provide an exact analysis of individual criteria. The following points are in-cluded as examples:

The implications of the connected battery for backup time in case of power failure

Consideration of the input power factor when designing an emergency power system

The effect of the UPS output power factor on the ability to deliver power to modern switch mode power supplies (SMPS), even at full load

Reduced performance in high altitude operations

The cost of a UPS unit depends on configuration details such as filters, transformers, fans, electronic bypass, integrated or external manual bypass, different circuit designs. The costing of best practise solutions for UPS systems is highly complex and requires a complex analysis of circumstances, cons-traints, dependencies and the consideration of a multitude of individual parameters.

5.4 Emergency Power

5.4.1 Power generating systems for stand-by power supplies (emergency power) in case of power failure

Power producers cannot guarantee an uninterrupted electrical power supply and in standard con-tracts, the power supply companies (PSCs) disclaim any liability. Brief interruptions or longer power ou-tages have therefore to be bridged by emergency power systems to ensure the continued operation of data centers with all their related technical systems such as climate control, power and security.

Maximum permitted downtimes must be given top priority when emergency power systems are plan-ned. Consequently, emergency power units are classified in different groups:

Devices without stipulated load transfer time. The units have manual start-up.

Units designed for a stipulated load transfer time. In this case, there is a longer interruption which can exceed 15 seconds before the unit automatically starts up and takes over the power supply.

Rapid standby units with short interruption times, typically of less than 1 second’s duration.

No-break emergency generator sets providing an uninterruptible power supply. In the event of utili-ty power failure, the load is transferred without interruption.

5.4.2 Emergency Power Supplies

In the last two cases, specially designed power generation units are necessary. As standby units, they must be equipped with energy storage devices. These have to be continuously charged. The ensuing operating costs are the price the consumer pays for increased supply security.

There are various different types of standby unit with combinations of diesel-engine, fly wheel, electric motor and corresponding coupling devices.

Standby units are needed wherever the interruption periods which arise when simpler emergency ge-nerators are used, would be untenable for the reliable continuation of a user’s operations.

20

5.4.3 Emergency Power System Configuration

The following factors play a decisive role in the configuration of the system’s capacity:

The size of load being served

Coincidence factor

Load start-up

Uninterruptible power supply (UPS) with battery charger

Acceptable dynamic performance

Load response

Reserve capacity for future system upgrades

Allowance for varying ambient conditions

System Load Requirement

When calculating the load requirement, both apparent power and effective power must be taken into account.

Coincidence Factor

In data centers, the power rating for generators must have a coincidence factor of 1, as all loads have to keep the data center operational in summer and winter alike.

Start-up characteristics

The start-up and switch-on characteristics of electric motors, transformers, and major lighting installa-tions with incandescent lamps will all affect the generator power rating.

In asynchronous motors, apparent power can reach up to six times and effective power 2-3 times the nominal rating. A sequential power-up system can considerably reduce the required generator power rating. All available measures for minimizing the start-up load should be taken.

Dynamic Performance

The dynamic performance of the generator unit operating at full load with anticipated load changes should be adjusted to the permitted values of the power consuming devices.

In meeting these requirements it may be necessary to overdimension the engine, the generator or both.

Ambient Conditions

The engine reference temperature is 27° in accordance with DIN 6271. If higher operating temperatures are expected, then the engine size must increased accordingly. The reduction factors of the engines must be known.

21

5.4.4 Recommended Emergency Power Supply in Relation to Permitted Downtimes

RZ K

ateg

orie

Permitted

Data Center

Dow

ntime




Emergency Power


Infrastructure

0 min

EEmergency power generatorsfully operational in 15 secondsFuel supply: 72 hours


10 min

D Emergency power generator: 1 unitfully operational in 15 secondsFuel supply: 24 hoursDuring maintenace work: mobile replacement unit


1 h

CEmergency power generator: 1 unitfully operational in 15 secondsFuel supply: 24 hours

PowerTechnical infrastructureto Tier III ** standard

24 h

B

Optional


A

Optional


Data

Cen

ter C

ateg

ory

72 h

Fig. 6: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” – Emergency Power

It is possible to lease generators from power companies. These provide an emergency power supply via an external connection during routine maintenance and repair work. These leased generators, however, are no solution in case of unexpected outages, as one can never be sure whether there will be any available at the critical moment.

Spatial Layout

The room or container for an emergency power generator needs to fulfil the following criteria: A simp-le transportation route and easy access must be guaranteed. The room or container should be adequa-tely dimensioned, so that all components can be easily installed. A one to two metre clearance area must be maintained around the unit to allow for operation and maintenance. In addition, the genera-tor should be located as far away as possible from inhabited rooms.

22

Main Storage TankMain Storage Tank

M

Transfer Pump Set

Rupture Basin

Day Tank

Fig.7: Diagram of a fuel supply system with main storage tank, day tank and transfer pump set

The regulations for the storage of inflammable liquids must be adhered to. The stored fuel supply should be sufficient for 24 – 72 hours of operation at full load. Larger fuel storage tanks should be loca-ted outside the generator room if possible.

Depending on their capacity, non-soundproofed generators produce acoustic emissions of between 90 and 120 dB(A). In Germany, maximum noise emission levels are dictated by the TA Lärm regulations (Technical Guidelines on Noise Pollution) or by the regional environmental agencies.

Daytime Night-time In accordance with these guidelines, rooms, ventilation units and exhaust pipes must be sound-proofed.

Industrial Areas 70 dB (A)Mixed Areas 60 45 dB (A)Residential Areas 50 35 dB (A)

Switchgear

Air IntakeM

uffler

Tank

Strip foundation

Exha

ustM

uffle

r

Fig. 8: Enclosed Generator Set

23

Electrical System

Every generator requires electrical switchgear, designed either for manual or automatic operation.

Fully automatic systems have the following functions:

Line voltage monitoring

Generator Switchgear

Control Panel Power Switching

Mai

ns

Load

Power unit with switching device

Generator protection and control

Power supply for auxiliary drives

Battery charger

If several generators are to be operated in parallel, the switchgear will need to be designed accordingly. For parallel operation with the mains grid, prior consultation with the relevant power supply compa-nies is obligatory.

5.5 Climate Control

5.5.1 Initial Situation

Each kilowatt (kW) of electrical power consumed by IT devices is given off in the form of heat. This heat has to be transported out of the device, the cabinet and the room in order to maintain constant operating temperatures. For heat removal there are climate control units and systems with various different operating principles and capacities available.

5.5.2 Climate Control: A Challenge

Climate control is essential for maintaining the performance and security of data center operations. Increasing levels of integration and packing density in processors and server systems are now causing levels of waste heat which would have been inconceivable in such limited spaces a few years ago.

24

There are different climate control solutions on the market, depending upon the individual power con-sumption and loss levels – i.e. waste heat – of the IT components in question. It is not only the range of climate control components that counts, but their compatibility with the cabinet or server rack sys-tems in use. An optimal system is one that is easily upgradeable and whose components complement each other in an ideal way.

Today it is possible for a single CPU to produce more than 130 W/cm∞ – the equivalent of two standard incandescent light bulbs per cm². This gives a whole new meaning to the task of data centre climate control. Despite the wide range of cooling systems available, it is becoming increasingly difficult to dis-sipate heat losses.

Data center cooling presents a new challenge. Test results and practical experience indicate that a ma-ximum of 5 kW heat loss per rack can be handled using the conventional raised floor cooling systems installed in many data centers. The airflow characteristics of the raised floor systems found in typi-cal mainframe data centers are no longer able to cope with today’s extremely heavy demands. This is hardly surprising, as the mainframes had relatively low heat loss in proportion to their performance and size and the cold air stream came from underneath.

For decades, a cooling capacity of between 1 and 3 kW per 19” cabinet has been sufficient, but today the cooling capacity per rack needs to be drastically increased. Modern servers in 42U 19” rack cabinets can consume over 20 kW of electric power and produce over 20 kW of heat. This figure will double in the foreseeable future due to increased performance coupled with shrinking installation sizes, or to put it more simply: power density is increasing.

Important criteria for climate control solutions are: the maximum expected power loss, installation conditions, acquisition costs, operating costs, upgrade costs, future-proof design, downtime costs, physical safety etc.

5.5.3 Infrastructure Operating Principles

The recommended ambient temperature is between 22°C and 26°C with between 30 and 50% relati-ve humidity (RH). However, the IT component manufacturers’ requirements must be followed on this. Room temperature should not be unnecessarily low, as operating costs increase considerably with each degree’s Celsius drop in temperature.

If humidity is too low there will be electrostatic build-up, if it is too high, the result will be corrosion in electrical and electronic components. Operating environments with very low temperatures below 12°C and high humidity, which can lead to a build-up of condensation on IT equipment, should be avoided under all circumstances. Climate control systems normally consist of at least one indoor unit located inside the server room as well as an outdoor unit on the exterior of the building which gives off the heat to the surrounding environment.

In climate control technology there are two basic types of system. Those which evaporate a refrigerant directly (DX = direct expansion) and systems which use cold water (CW = chilled water) to remove heat. In DX systems, each inside unit has its own external unit and compressors are used for the transfer of both the refrigerant and the heat.

Their performance is slightly below that of comparable CW systems. These use chilled water from a closed supply loop to which several indoor units can be connected. The water is pumped through the

25

system. CW installations require elaborate outdoor units where the heat is removed from the water during the cooling cycle. In case of damage, refrigerant can leak from DX systems in the form of gas and water can leak from CW systems. Provisions for water detection are recommended.

5.5.4 Air Cooling

Almost all the commercially available 19”-compatible IT devices use air to dissipate thermal power loss.

The simplest method of heat dissipation is through the cabinet walls, as long as the ambient air tempe-rature is below that inside the cabinet.

Alternatively, cabinets can be passively ventilated. Air exchange takes place using a mixture of slots and perforations in the bases, roof panels, doors and possibly the sides, whereby the airflow draws cooler air in from the front and base and exhausts the heated air at the top and rear. This design is in-adequate in high density data center environments, however. Here there must be one active ventilati-on unit per cabinet. These include versions with supplementary fans inside the rack to draw in cold air or to exhaust heated air.

In both cases, the layout of racks and the configuration of the technology inside them play an impor-tant role. The data center is laid out with hot and cold-air zones or aisles in order to make optimal use of the cooling qualities of the air. Care must be taken to prevent the rack cabinets in the front row from drawing in the hot exhaust air from preceding rows of cabinets with their intake air, as this would impair cooling.

Fig. 10: Raised floor in combination with a dropped ceiling

If none of these solutions are feasible, special ventilation and cooling components will need to be in-stalled to ensure adequate climate control. These are models which can deliver in excess of 20 kW coo-ling capacity per rack, while providing simultaneous redundancy.

There are various different approaches being taken to push climate control to increasingly higher levels of performance. The most well-known and widely-used is the installation of water-cooled racks. Here there are one or water coolers installed inside a fully sealed cabinet. Redundant fan-blown air circu-lates within the cabinet, passing through the coolers. This system is capable of dealing with the heat loads produced by today’s ultra high-density blade servers.

26

Liquid Cooling

Rack-integrated air-water heat exchangers are suitable for the removal of heat where it is produced. They generate a microclimate which can be applied far more selectively than computer room air condi-tioning systems (CRACs).

For high-performance clusters, there is also the option of direct liquid cooling of CPUs, as this is where the bulk of the thermal power loss occurs, and at the same time, heat loss from memories, hard disks and PSUs should not be overlooked.

Direct cooling at a glance

Selection Criteria: expected maximum power loss, installation conditions, acquisition costs, opera-ting costs, upgrade costs, future-proof design, downtime costs, physical safety etc.

“IT cooling” per se doesn’t exist. After a careful requirements analysis and future planning have been completed, the chosen cooling strategy should be implemented using high-quality components

Water is without doubt a more effective medium for heat transportation than air.

Thermal loads in excess of 40 kW per rack can be simply, safely and efficiently removed using air-to-water heat exchangers

Liquid cooling - even on CPUs – is now a tried and tested technology

Only liquid cooling technology will be able to satisfy the cooling capacity requirements of the next few years

Adjustment to existing data center structures will be necessary – a change from room to rack cli-mate control

Thanks to liquid cooling in the rack, there is no need to gear the entire computer room air conditio-ning system to meet the requirements of a high performance server rack

There are also some cutting-edge, “piggy-back solutions”, where individual, rack-rear-mounted, air-to-water heat exchangers supplement and reduce the burden on the computer room air conditio-ning system by releasing only cooled exhaust air into the room

27


Data

Cen

ter C

ateg

ory

0 min

Permitted

Data Center

Dow

ntime




Climate Control


Infrastructure

E PowerTechnical infrastructureto Tier II / Tier IIV** standard

10 min

D PowerTechnical infrastructureto Tier III / Tier IV ** standard

24 h

B PowerTechnical infrastructureto Tier I / Tier II ** standard

A PowerTechnical infrastructureto Tier I / Tier II ** standard

72 h

Precision cooling

Precision cooling

Precision coolingExperts on callUPS back-up recommended

Precision cooling

Precision cooling withredundant configurationUPS support

Precision cooling withredundant configurationUPS support

Precision cooling withredundant configuration


High-performanceor liquid cooling,Redundant configuration


High-performanceor liquid cooling,Expert on call,Redundancy required forhigh-density cabinets

High-performanceor liquid cooling

1 h

CPowerTechnical infrastructureto Tier III ** standard




Fig. 11: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” – Climate Control

Controlling room climate with precision cooling

In server rooms smaller than about 25 m², one often finds inadequate ceiling or wall-mounted, “com-fort” air conditioning units in use which are not suited to the requirements of IT equipment. These units dehumidify the room air, they have a low air flow rate and they are not designed to be used all year round.

Climate control of data centers with a floor space of over 25 m² is often accomplished using so-called raised-floor systems or pressure floors. These precision air conditioning units are located on the peri-meter walls of the room, from where they blow cold air into the raised floor system. This chilled air is delivered directly to the front of the 19” cabinets, exiting through specially perforated tiles. This type of system can provide between 3 and 5 kW of cooling capacity per rack, depending on the size of the room and the height of the raised floor.

In “low-density” data center installations (up to 3kW per rack), this constitutes an acceptable climate control solution. At the same time, the arrangement of server racks in hot and cold aisles, the correct deployment of perforated floor tiles as chilled air exits and the use of cable cut-outs with well-fitting coverplates are all factors which must be taken into account. Some floor-mounted devices such as mainframes or storage cabinets may require vertical bottom-to-top air-flow solutions.

28

High-performance or liquid cooling

Unlike precision cooling, high-performance or liquid cooling solutions have the following advantages when used in water-cooled, closed rack systems:

completely independent of the surrounding room

no heat loss to the room

noticeably reduced cooling load for the room

guaranteed cooling capacity per rack

uniform cooling temperature from bottom to top

high-density deployment of air-cooled 19” servers of any type

excellent operational reliability

fire detection and suppression at rack level, thus easy localisation and damage containment

protection of IT equipment from outside influences (smoke, firefighting water)

Redundant configuration of the cooling system

With both systems it is important to ensure redundant configuration for tolerable downtimes of un-der an hour’s duration. For this purpose it is necessary to have fail-safe or redundant configuration of both the chilled water supply and the rack cooling system.

Special Considerations

Considerable constructional measures are often necessary when air conditioning units are installed in server rooms. The following items need to be clarified beforehand:

the delivery and installation of the indoor and outdoor units (ceiling load capacity)

where will wall penetrations be needed?

which cable runs can be used for the supply lines?

voltage supply for the climate control infrastructure (possibly via utility back-up)

lightning protection for outside units

a comprehensive water alarm system (for the climate control system at least)

In addition, it should be borne in mind that at present there are differing philosophies on the market regarding climate control technology: water in the server room - yes or no?

From a technological point of view, liquid cooling is no problem today.

29

Recommendations

cabinets should be arranged so that their fronts face each other across a ‚cold’ aisle, and their backs face each other across a ‚hot‘ aisle

vacant 19” slots should be sealed with blanking panels to prevent recirculation/bypass airflow

integrated rack air distribution units for optimal cooling air flow to critical components

cooling capacity calculations should be based on realistic assumptions. The nameplate ratings spe-cified on IT equipment are never reached under actual operating conditions. In practice, the maxi-mum required cooling capacity is often 30% lower or more.

cooling capacity for sealed cooling systems should be based on the racks and not the room

avoid hot-spots, aim for even distribution of the thermal load in the room

if possible, rack configuration should not be on an exclusively functional basis, but should also make allowance for power loss

30

6 Fire Protection

“Experience has shown that fire can break out at almost any time. The fact that in many buildings the-re hasn’t been a fire outbreak for decades does not prove that no danger exists. If anything, it should be seen as a lucky streak for those concerned and its end should be reckoned with any time. “

This 1987 statement by a German high court needs nothing added, which is why a reliable, rapid fire detection and suppression system is an essential prerequisite for reliable fire protection in data cen-ters.

Reliable data centers are a great challenge to the hardware. An extinguishing agent such as water is inappropriate in a data center environment. Today, specialist firms offer appropriate suppression sys-tems for every eventuality. In new data centers or when fire protection is retro-fitted, it is important to plan the system carefully.

6.1 Active fire protection

Fire, smoke, aggressive gases or firefighting water are all potential hazards in data centers. To ensure safety, it is necessary to install a technically high-grade fire detection system in conjunction with first class fire suppression technology. An alternative solution is fire avoidance by lowering the oxygen con-tent of the air. Through the controlled introduction of nitrogen, the oxygen concentration can be redu-ced to a precise, preset level. In spite of this, such rooms can still be entered by personnel.

Foam fire extinguishers, on the other hand, cannot be used, as they would damage computer systems or their power supply units. Powder-based fire fighting systems are equally unsuitable for use in data centers with all their sensitive equipment, as the fire suppression measures could possibly cause more damage than the fire itself. Today, therefore, it is almost always gaseous suppression systems which are employed.


The fire suppression system infrastructure consists of smoke and fire detection devices, the suppressi-on systems themselves and the fire alarm control panels (FACP).

Smoke and fire detection systems

Smoke alarms are widely used for smoke detection. The systems of the various different manufactu-rers all operate on broadly the same principles. Another option is the so-called ionisation alarm, but because of its radioactivity, it is now only installed in a very limited number of special cases.

Because of the risk of false alarm, two stage systems are recommended that trigger only when two zone sensors go off – in single-sensor configurations this is not possible. For this reason, single sensors only set off technical, internal alarms.

Inside data centers, layers of warm air up to a meter thick can build up under the server room ceilings. These are created by air turbulence from air conditioners, PSU fans or rack ventilation units. This could

31

lead to situations where smoke is prevented from reaching the spot smoke detectors, with the result that an outbreak of fire is difficult to detect, and by the time it is detected, it is too late. Spot-type smoke detectors generally reach their performance limits above certain heights, in high air-flow rate environments or at extreme temperatures.

Under such conditions, very early smoke detection apparatus (VESDA) provides the best protection. These systems function on aspiration principles, a lot like vacuum cleaners. As a rule, the smoke detec-tors are highly sensitive optical devices but increasingly, digital particle counters based on laser techno-logy are being used.

For several years now, manufacturers of fire-detection and suppression equipment have also been pro-ducing systems which can detect and suppress fires in individual racks – however, these individual rack solutions are only suitable for a very limited range of rack types.

Fire suppression systems

The efficiency and reliability of a fire protection system depends on risk-adjusted project planning, quantitative analysis and dimensioning.

Fire suppression by oxygen reduction

Gaseous fire suppression is the appropriate technology for data centers and their equipment. It works on the principle of oxygen reduction. The extinguishing agent reduces the oxygen content of the air to such an extent that a combustion process is prevented. This type of fire suppression system employs either inert or chemical gases.

Carbon Dioxide (CO2)

Carbon dioxide is present in our atmosphere. When used as a fire suppression agent, it is compressed and stored in high pressure cylinders or low pressure bulk tanks. High concentrations of carbon dio-xide represent a health hazard and for this reason, special safety precautions must be taken. When a data center is flooded with carbon dioxide, a sudden drop in room temperature will occur and this can be detrimental to highly sensitive systems.

Argon(Ar)

Argon is an inert gas obtained from ambient air. Argon itself is non-toxic, but at the concentration ne-cessary for fire suppression it can present a health hazard from drastically reduced oxygen levels and effluent gases.

Nitrogen (N2)

Nitrogen is also present in the atmosphere. It is colorless, odorless and tasteless. Nitrogen is non-toxic but can present a danger to health from oxygen depletion and effluent gases.

Extinguishing Gases FM-200, HFC 227ea, HFC125, NOVEC

These extinguishing agents suppress fires by absorbing heat from the flame zone – a combination of physical heat removal and, to a lesser extent, chemical interaction. Independent scientific studies ha-ve shown that FM-200 and NOVEC do not represent a health hazard in themselves and, more impor-

32

tantly, that rooms flooded with the agents can still be frequented by people. This means that after a data center has been flooded, personnel can manually shut off the systems causing the fire. Another important factor is that the extinguishing agent containers can be stored in the room which is to be protected. This can be crucial when a fire suppression system is retrofitted in an existing data center which was previously without one. In addition, when rooms are flooded with these gases there is no significant drop in the ambient temperature, a fact which benefits the IT systems.

Above all, gaseous suppression technology is an important element in object, room and equipment protection. There is no water damage and no residue from powder or foam. Fires are extinguished ra-pidly within 10 to 12 seconds and any damage is limited to the equipment which caused the fire. These extinguishing agents are electrically non-conductive so that there is no risk of short circuiting during or after the suppression process. Normal operations can be quickly resumed when extinguishing gases are employed. Extinguishing gases can also be used for individual racks.

Oxygen reduction fire suppression systems

This alternative doesn’t wait until a fire starts – it prevents it from breaking out in the first place. A top quality, highly accurate control system keeps the reduced oxygen levels constant. At the same time, nitrogen is introduced into the protected areas creating an atmosphere in which a fire cannot occur. The protected areas can still be occupied by humans, although physicians believe that highly sensitive members of staff could suffer temporary health problems.


Surveillance system with Very Early Fire Detection

Surveillance system with Very Early Fire DetectionFire alarm system,Surveillance system withVery Early Fire Detection

24 hB Fire Protection

A Fire Protection 72 h

Active Fire Protection

Data

Cen

ter C

ateg

ory Perm

ittedD

ata Center D

owntim

e





Infrastructure

Surveillance system with Very Early Fire Detectionand fire suppression system 1 hC Fire Protection

Surveillance system with Very Early Fire Detectionand fire suppression system

Surveillance system with VeryEarly Fire Detection/ Oxygenreduction system

10 minD Fire Protection

Surveillance system with Very Early Fire Detectionand fire suppression system/ Oxygen reduction system, each in redundant configuration

0 minE Fire Protection

Fig.12: from BITKOM-Matrix “Planning Aid for Reliable Data Centers” – Active Fire Protection

33

Server Cabinets

A fire detection and alarm system usually consists of at least 2 smoke sensors plus the correspon-ding analysis component. If smoke is detected, the system will react by sending an alert to a manned alarm-receiving center or by generating an automatic emergency phone call.

If the maximum tolerable downtime is less than one hour, the detection equipment must be supple-mented by a fire suppression system. If the tolerable downtime is under 10 minutes, a combination of Very Early Smoke Detection Apparatus (VESDA) and a suitable fire suppression system in redundant configuration is necessary.

Data Centers

If there is a maximum 24 hour tolerable downtime, then smoke detection and analysis units without a follow-up fire suppression system are sufficient. If there is a maximum downtime tolerance of less than one hour, a follow-up fire suppression system – gaseous if possible – is absolutely vital. In this case, fire avoidance by means of oxygen reduction is a viable alternative.

The decision on whether to use gaseous fire suppression or oxygen reduction can only be taken by ex-perts after they have carried out a thorough analysis. Project planning features for the dimensioning and installation of a controlled fire suppression system are as follows:

Definition of system(s) requirements – tolerable downtimes etc.

Definition of the type of detection unit (spot sensors, ASD [aspiration smoke detectors])

Definition of the fire suppression system (type of gas)

Possible planning of an oxygen reduction system with nitrogen feed

6.2 Structural Fire Protection

The aim of structural fire protection is the saving of human life. This calls for the highest quality both in terms of materials and workmanship, as well as strict adherence to regulations and guidelines.

The basic principles of structural fire protection are laid down in building codes and regulations on fire protection measures, fire safety planning, fire walls and emergency escape routes. The flammability characteristics of building materials and structural elements is regulated by the German national stan-dard DIN 4102 , although this makes absolutely no allowance for the particular fire protection objec-tives, especially in IT data centers.

The fire rating of load-bearing structures, fire prevention measures in the electrical installations and utility supply systems must all be considered. When planning a data center, it is also necessary to cla-rify fire safety options relating to fire resistance levels and escape routes. Fire service elevators and enclosed stairways should be allowed for here. In addition, there are sector-specific fire safety regula-tions for data centers, for example the German Verband der Sachversicherer (Association of Property Insurers) guideline 2381 (VdS 2381).

Fire suppression, extinguishing agents and smoke ventilation must also be included in planning. Amongst other things, this includes fire extinguishers and, possibly, extinguishing-agent retention.

34

6.2.1 Fire Protection Objectives

When data centers are planned, it is vital to identify the fire protection objectives. During the plan-ning phase companies must clarify whether they are in a position to implement regulations, guidelines and protection objectives themselves. It is advisable to employ experienced planners, as structural and active fire protection measures need to be harmonized with the requirements of a 24/7/365 data cen-ter operation. Subsequent installations and building alterations consume immense sums of money or lead to a drastic rise in insurance premiums for fire and electronics cover.

6.2.2 Operating Principles and Room Requirements

Structural elements are classified in fire resistance classes according to their flammability characteris-tics. The usual fire resistance ratings are for 30, 60, 90 and 120 minutes.

F 30 means that in flammability tests, a wall, for example, will resist the flames for at least 30 minutes. In building legislation, class F 60 is “flame-retardant”, and class F 90 “fire-resistant”.

Walls, floors and ceilings must all conform to fire resistance class F 90 at minimum. Doors need to be designed to at least T90 specifications, which means that they can withstand fire for 90 minutes. Pro-tection against fumes and water spray is also recommended.

Cable and installation conduits should be efficiently protected. Cable conduits should be protected with functionality to E30 or even E90. Installation conduits should be designed to conform to I 30 or I 90 and automatic ventilation ducts to L 90. Where electric cables pass through fire-resistant ceilings and walls, the cable penetration ducts must also be fitted with fire and smoke-resistant seals. In certa-in cases, penetrations can be sealed with fire pillows.

In case of fire, cable trays represent a major risk and should have a water and moisture-resistant coa-ting. In this way they act as an intumescent layer to prevent fire spreading along the cable. The cables themselves should be of fire-retardant material which also minimizes the formation of aggressive fu-mes.

Fire can spread rapidly and uncontrollably through inflammable piping on walls and ceilings. These can be protected by special pipe insulation which acts as a fire-retardant and smoke-tight barrier.

However, for complex and operationally secure data center facilities, testing only the structural ele-ments is by no means sufficient. The proposed rooms or modular security cells, as well as the compo-nents used in ceiling-wall and floor-wall joints, cable entries, the excess pressure discharge system and doorways must all undergo a system audit in accordance with the EN 1047-2 standard. This European standard for data center infrastructure lays down and certifies the limits of resilience to stresses (e.g. fire, water, shock). The user thus has the guarantee that an entire system is fire-resistant, and not just a wall or a door.

35

6.2.3 Recommended Configurations for Different Downtimes

Structural Fire Protection

Room requirements:Walls, floor,ceiling:Minimum fire rating class F90Protection against smoke and water spray,at least T90-doors

Walls, floor,ceiling:Min. fire rating class F90Protection against smoke and water spray,at least T90-doors



Walls, floor,ceiling:Min. fire rating class F90Protection against smoke and water spray,at least T90-doors


Data

Cen

ter C

ateg

ory Perm

ittedD

ata Center D

owntim

e





Infrastructure


Walls, floor,ceiling:Min. fire rating class F9030 minutes‘ protection againstsmoke and water spray,at least T90-doors



Walls, floor,ceiling:Min. fire rating class F9030 minutes‘ protection againstsmoke and water spray,at least T90-doors


Room requirements:Walls, floor,ceiling, doorsto European Standard EN 1047-2

30 minutes‘ protection against smoke and water

Walls, floor,ceiling, doorsto European Standard EN 1047-2

60 minutes‘ protection againstsmoke and water

1 hC Fire Protection





1 hC Fire Protection











Room requirementsWalls, floor,ceiling:Minimum fire rating class F90Protection against smoke and water spray,at least T90-doors


Room requirementsWalls, floor,ceiling:Minimum fire rating class F90Protection against smoke and water spray,at least T90-doors


Fig.13: from BITKOM-Matrix “Planning Aid for Reliable Secure Data Centers” – Structural Fire Protection

6.2.4 Special Considerations

The following project planning features should be considered:

Definition of protection objectives with respect to the special requirements of IT infrastructure

Definition of structural layout

Planning of building construction – preferably by professional planners

Production of requirements specification for individual components for the invitation to tender

Collation, examination and evaluation of incoming offers

Production of a contract award recommendation for the decision makers

36

7 Floor space design and security zones for data centers

The term IT security covers a broad spectrum, including logical data security, the physical security of systems and the organizational security of processes. The goal of a comprehensive security con-cept is to survey all areas, identify risks early on, assess them and then take measures to ensure that a company’s market competitiveness is not compromised.

By reviewing the IT infrastructure and the different functional areas of the IT, it is possible – with a well thought-out concept – to reduce, or even eliminate, significant threats to physical security. The loca-tions of IT facilities on the one hand, and the spatial relationship between the different functional are-as on the other, both have a key role to play.

Location of IT facilities

The design of an IT infrastructure and therefore also the choice of location for a data center are based on the data security concept of the company in question, which in turn reflects the availability require-ments and corporate policy alignment.

The following criteria should be considered when appraising the physical security of a location:

minimal risk from neighbouring installations, adjacent structures or operations

avoidance of risk from telecommunications and utility supply lines, vibrations or chemicals which could threaten the physical security of the IT systems

avoidance of potential danger from natural disasters (flooding, storms, lightning, earthquakes) – as-sessment of regional peculiarities

the data center as a separate, autonomous functional area

protection from sabotage through “sheltered” location

assessment of potential threat to customer on account of the social status of the company

If all the risk factors and customer-specific parameters are taken into account, then risks can be exclu-ded and costs and effort avoided right from the start, during the conceptual design of the IT infrastruc-ture.

Data center layout

During the design and planning of a data center, the different functional areas are arranged according to their security requirements and their mission-critical value.

37

The different functional areas can be classified as follows:

Security Zones Function Identification (example)1 Site White2 Semi-public area, adjacent office space Green3 Operational areas, IT side rooms Yellow4 Technical systems for IT operations Blue5 IT and network infrastructure Red

Layout of Security Zones

When the different security zones are presented in diagram form, it could look like this illustration: The IT area (red) is at the centre and is protected by the adjacent zones 3 and 4 (yellow/blue). Security zones 1 and 2 (white/green) form the outer layer. The individual security zones are separated by security boundaries.

The security boundaries are the controlled and protected transition points between the zones and are configured to meet the security re-quirements of the customer.

To avoid the danger of possible sabotage, it is advisable to separate the different functional areas al-lowing only limited access to sensitive areas. In this way, for example, a service engineer responsible for air-conditioning systems or UPS would only receive access authorization for the company’s techni-cal areas (blue) and not for the IT area (red).

The location of the different functional areas and the lay-out of the security zones are both important factors in maintaining the security of IT infrastructure. Yet continuous IT-availability can only be achie-ved within the overall context of a comprehensive security concept embracing all areas of IT security.

38

8 Appendix

A selection of relevant guidelines and legislation

DIN 6280, Parts 1-15 Power generating sets with reciprocating internal combustion engines

Part 1 Basic terms

Part 2 Performance interpretation and rating plates

Part 3 Operational limits for engine, generator and power unit performance characteristics

Part 4 Speed control and speed characteristics of reciprocating internal combustion engines - terms

Part 5 Performance characteristics of synchronous generators for genset operation

Part 6 Performance characteristics of asynchronous generators for genset operation

Part 7 Switching gear and control equipment for genset operation

Part 8 Performance characteristics during genset operation - terms

Part 9 Approval test

Part 10 Small power generating sets, requirements and test

Part 11 Measurement and assessment of vibration stressing in power generating sets with reciprocating internal combustion engines

Part 12 Power generating sets – Uninterruptible Power Supply - Dynamic UPS systems with and without reciprocating internal combustion engines

Part 13 Power generating sets – power generating sets with reciprocating internal combustion engines for emergency power supply in hospitals and public buildings

Part 14 Combined heat and power systems (CHPS) with reciprocating internal combustion engines – Basics, requirements, components and applications

Part 15 Combined heat and power systems (CHPS) with reciprocating internal combustion engines – tests

ISO 8528 Reciprocating internal combustion engine driven alternating current genera-ting sets.

39

Federal Emission Control Act (BimSchG) (Germany)

4. Ordinance on the Implementa- tion of the Federal Emission Control Act

Ordinance on installations requiring a license

9. Ordinance on the Implementati-on of the Federal Emission

Basic principles of the licensing procedure

TA Luft Technical Instructions on Air Quality Control

TA Lärm Technical Instructions for the Protection against Noise

DIN / VDE 0107 High voltage power installations in hospitals and locations for medical use outside hospitals

Supplement 1 Extracts from Federal German building and industrial safety regulations

Supplement 2 Interpretation, commentaries

DIN / VDE 0108 High voltage power installations and safety power supply in communal facilities

Supplement 1 Building regulations

Part 2 Meeting places

Part 3 Office buildings and exhibition sites

Part 4 Multi-storey buildings

Part 5 Restaurants

Part 6 Enclosed car parks

Part 7 Workplaces and business premises

Part 8 Temporary buildings

DIN / VDE 0100 Part 728 Emergency power supply systems

TAB General Technical Requirements of the utility companies

VDEW German Power Suppliers’ Association - Guidelines for emergency power generating sets

40

VDEW Concurrent operation with the low-voltage grid

EltBauVO Electrical Building Regulations

VDS German Insurance Association regulations

WHG Federal Water Act

Mineralölsteuergesetz Law on the Taxation of Mineral Oils (Operation of stationary generators with fuel oil)

DIN 31051 Maintenance and repair

41

9 Glossary

19”-Cabinet Rack with approx. 40 U, overall height approx. 2 metres. Installation height is measured in rack units (U), 1 U = 44.45 mm

CW Chilled Water; air conditioning units which use cold water

Data Center Server room and/or data center

DX Direct eXpansion; refrigerant-type air conditioning units

PDU Power Distribution Unit, also main distribution board

emission A substance discharged by an appliance into and affecting the environ-ment

EMC Electromagnetic Compatibility

IT Information Technology (formerly EDP = Electronic Data Processing)

modular System constructed of several modules

EPU Emergency Power Unit (usually a back-up diesel generator)

parallel operation Two or more output devices which combine current to supply connected loads

precision cooling Air conditioning units which keep both temperature and humidity at constant levels. Air parameters at IT equipment intake vents should be between 20 and 25°C and between 40 and 50% RH.

ASD Aspiration Smoke Detector

redundant Multiple configuration to increase availability (fault tolerance) scalable ca-pable of adapting to demand

scalable capable of adapting to demand

UPS Uninterruptible Power Supply

42

10 Acknowledgement

This guide to “Secure Data Centers” was produced in coordination with the BITKOM technical commit-tee on “Operationally Secure Data Centers”.

We would like to express our sincere thanks to all members of the technical committee for valuable discussions and suggestions, and also to all authors,

Dieter Henze (Rittal GmbH & Co.KG),

Siegbert Hopf (Masterguard GmbH),

Peter Koch (Knürr AG), Helmut Göhl (O2 GmbH),

Kurt Krabbes (TDS Informationstechnologie AG),

Matthias Lohmann (TÜV Secure),

Ingo Lojewski (Emerson Network Power GmbH),

Achim Pfleiderer (Stulz GmbH)

Jörg Richter (I.T.E.N.O.S GmbH),

Harry Schnabel (Schnabel AG),

Michael Schumacher (APC Deutschland GmbH)

Jürgen Strate (IBM Deutschland GmbH)

Judith Wagener (Bull GmbH),

Ralph Wölpert (Lampertz GmbH & Co.KG),

Eckhard Wolf (AEG Power Supply Systems GmbH),

and

Sandra Schulz (Giesecke & Devrient GmbH) for her committed collaboration and her unceasing enthusiasm for this subject.

Our special thanks go to Harry Schnabel, head of the technical committee and to Lisa Röseler for the proofreading and editorial revision of the guide.

Translation by Caroline Taunt.

43

11 Notes

German Association for Information Technology Telecommunications and New Media e.V.

Albrechtstraße 10 10117 Berlin

Phone.: 030/27 576-0 Fax: 030/27 576-400

www.bitkom.org [email protected]

The German Association for Information Technology, Telecommunications and New Media (BITKOM) represents a total of more than 1,000 companies. Its 850 regular members employ some 700,000 people and generate revenues of 120 billion Euro. They include manufacturers of ICT equipment and providers of software, IT services, telecommunication services and content. BITKOM is working, in particular, to improve the regulatory framework in Germany, for modernization of the education system and for an economic policy which encourages innovation.

reliable data centers guideline

Business