Janus: Co-Designing HPC Systems and Facilities

Henry M. TufoUniversity of Colorado,

Boulder430 UCB - ECOT 623

Boulder, CO 80309-0430tufo@cs.colorado.edu

Michael K. PattersonIntel Corporation

2111 NE 25th AvenueHillsboro, OR 97124

michael.k.patterson@intel.com

Michael ObergNational Center for

Atmospheric Research1850 Table Mesa Drive

Boulder, CO 80305oberg@ucar.edu

Matthew WoitaszekNational Center for

Atmospheric Research1850 Table Mesa Drive

Boulder, CO 80305mattheww@ucar.edu

Guy CobbUniversity of Colorado,

Boulder430 UCB - ECOT 717

Boulder, CO 80309-0430guy.cobb@colorado.edu

Robert StrongCritical Facilities Technology6380 W. 54th Ave. Suite 100

Arvada, CO 80002rstrong@cftrm.com

Jim GutowskiDell, Inc.

One Dell WayRound Rock, TX 78682

james_gutowski@dell.com

ABSTRACTThe design and procurement of supercomputers may requiremonths, but the construction of a facility to house asupercomputer can extend to years. This paper describesthe design and construction of a Top-50 supercomputersystem and a fully-customized pre-fabricated facility tohouse it. The use of a co-design process reduced the timefrom conception to delivery to three months, commensuratewith the amount of time it currently takes to deliver thecomputer system alone. Moreover, the facility was designedto provide efficient datacenter space for a 15-year lifespan.The design targets an expected yearly average power usageeffectiveness (PUE) of 1.2, with a measured PUE of 1.1to date. Leveraging the rapid deployment technologies inuse by industry allowed the procurement of the completeenvironment, including the facility and the resource, insignificantly less time than a machine room renovation andyears less than a new building.

Categories and Subject DescriptorsC.5.1 [Large and Medium Computers]: Super (verylarge) computers

Keywordscomputer facilities, modular data centers, system integra-tion, computer and facility co-design

Copyright is held by the author/owner(s).SC ’11, November 12-18, 2011, Seattle, Washington, USA.ACM 978-1-4503-0771-0/11/11.

1. INTRODUCTIONThe Front Range Computing Consortium, a collaborative

initiative between the University of Colorado at Boul-der (CU), the National Center for Atmospheric Research(NCAR), and the University of Colorado at Denver (UCD),has been deploying large computational infrastructure tosupport the needs of Boulder-area and national computa-tional researchers since 2002 [16]. As the systems procuredfor these collaborative research campaigns increase in size,the complexity of the logistics and planning required toconstruct computer room facilities have exceeded the com-plexity of designing the computers. Whereas the smallwhite-box Linux clusters of the early 2000s could be easilyhidden in a sub-basement of a university building, multi-rackBlue Gene/L systems and order-megawatt supercomputersrequire specialized computer room facilities with power,cooling, and space closely matched to the system’s uniquerequirements – space not easily obtainable from a largeorganization without procedural hurdles such as facilitycosting models, construction procedures, and occasionalbudget-induced capital construction moratoriums.

In this paper, we present our experiences with the co-design and deployment of the University of Colorado’s Top-31 (June 2010) supercomputer system, Janus [17], and thepre-fabricated data center facility to house it. While mod-ular data centers using components such as prefabricatedshipping containers have been widely used in industry overthe past decade, the Janus deployment is (to the best of ourknowledge) the first Top-50 supercomputer system intendedfor scientific computing deployed in a pre-fabricated facility[14]. Rather than a pre-established solution such as ashipping container, the Janus facility was designed-to-order,fabricated, and fully tested off-site in a three-month period,and then installed in a former parking lot at the university ina single day. The custom facility was designed to house theJanus system, provide room for additional expansion overthe building’s 15-year lifespan, support expansion with the

addition modular components, and achieve a power usageeffectiveness (PUE) of 1.2 under a representative operationalload.

The use of a co-design process to procure a systemalongside its facility provides numerous advantages over al-ternatives such as renovating or retrofitting existing facilitiesor constructing new facilities. In particular, renovatingor retrofitting existing facilities to support new compu-tational systems can be a time-consuming process. Forexample, NCAR selected an IBM POWER6-based systemfeaturing liquid cooling as part of its Integrated Comput-ing Environment for Scientific Simulation procurement in2008. Supporting the system’s heat load through chillerinterruptions required the installation of two 1,500-gallonwater storage tanks, and due to facility constraints thesetanks were placed in the computer room in an area formerlycovered by raised floor. Facility renovations required overnine months starting from the delivery of facility specifi-cations from IBM [15]. Even more challenging and time-consuming than renovation is constructing new general-purpose computational facilities with open-ended servicelifetimes. For example, development of the NCAR-WyomingSupercomputer Center (NWSC), the new permanent homefor NCAR’s supercomputing systems, officially commencedin March 2009 and the facility is expected to enter serviceby June 2012 [12]. The reduced time required to construct apre-fabricated facility, and the ability to more rapidly tuneit, allows the construction of facilities balanced to computersystems in a timeframe consistent with the technologylifecycle.

In addition to reducing the facility construction timeto more closely match the technology lifecycle, the useof modularized solutions can provide economic benefits aswell. For example, the Uptime Institute suggests a cost of$10,000/kW - $25,000/kW for data center space dependingon the facility’s “tier” rating [19], and an Intel benchmarksurvey suggested around $10,000/kW [18]. The Janusfacility came in significantly less than those targets.

The remainder of this paper is organized as follows: Sec-tion 2 highlights related work in the areas of containerizedand modular computing facilities. Section 3 describes ourquest to house a supercomputer, emphasizing the finalco-design process used to create the Janus facility andcomputational system. Section 4 presents the facility’spredicted and measured PUE. The paper concludes witha discussion of the broader implications of our experiencesand future work.

2. BACKGROUND AND RELATED WORKThe traditional datacenter design and construction pro-

cess is increasingly incorporating efficient and eco-friendlytechnologies, with some of the newest designs approachinga PUE of 1.0. For example, the National RenewableEnergy Laboratory (NREL) recently constructed a net-zeroenergy office and datacenter with a design target PUE of1.1 and a measured PUE between 1.07 and 1.14 to date[5]. A variety of the latest energy–efficient techniqueswere used, including the free cooling provided by outsideambient air temperatures for most of the year. Similarly, thenew datacenters at the NCAR-Wyoming SupercomputingCenter (NWSC) [12, 13] and NCSA’s National PetascaleComputing Facility (NCPF) [11] achieve (or plan to achieve)PUE values around 1.1.

Facility PUE (est.) Impl. TimeNCAR NWSC < 1.1 - 1.3 3 yearsNCSA NPCF 1.1 - 1.2 18 monthsNREL RSF 1.07 - 1.14 –HP POD ≥ 1.25 6 weeksCirrascale FOREST 1.2 - 1.3 –IBM PMDC 1.3 12 weeksSGI ICE Cube 1.12 –AST MSS ≥ 1.05 –Bull mobull – ≤ 8 weeks

Table 1: Efficiency and Design / Build Time ofModern Datacenters

Many of the same techniques are now becoming availablein modular or containerized data center solutions that pro-vide similar PUE characteristics (see Table 1). Vendors ofmodular datacenter solutions span a spectrum from turnkeysolutions to component providers for integration with third–party vendors using standard 19” rack server equipment. A“containerized” modular datacenter is either a prefabricatedbuilding or built inside a standard ISO intermodal shippingcontainer of a standard length (typically 20’ or 40’, with amaximum of 53’), and a variety of other sizes is availablefor specialized applications. Solutions available from severalprominent vendors are surveyed in the next section.

2.1 Turn-key modular data centersTurn–key solutions are typically provided by mainstream

hardware vendors and consist of the container facility and as-sociated mechanical components (chilled water distribution,electrical, networking, etc.) along with server, storage, andnetworking equipment. Integration and testing is typicallyperformed at the vendor site and delivered intact to thecustomer location for final configuration. Example providersof such complete solutions are HP, Dell, and Cirrascale(formerly Verari).

The HP Performance Optimized Data Center (POD) [9]is sold in both 20’ and 40’ configurations in a traditionalhot/cold aisle configuration with 10 and 22 50U racks,respectively. The POD uses a “closely–coupled coolingdesign” with chilled water and a heat exchanger, providinga facility PUE of 1.07 and an overall PUE as low as 1.25as configured from HP. Power density for the POD is 700W/ft2, with a capacity of 27 kW per rack. POD systemsare optionally configured and tested by HP at their facilitybefore being shipped to the final location.

The Dell “Humidor” system [1] is somewhat unique inthat it consists of two containers, one stacked on top of theother. This has the benefit of separating the mechanicalcomponents from the compute components, each in its owncontainer and having separate access controls. This matcheswell with the common operational configuration of havingseparate personnel for facilities management and IT services.Currently the Humidor system is available through Dell’sData Center Solutions (DCS) unit on a limited basis. It isa highly customized product, and at the present time, thereappear to be no plans to make it more broadly available. Thelargest publicly–acknowledged user of the Humidor solutionis Microsoft, who is using it to power the Azure cloudcomputing platform in datacenters located near Quincy,Washington and Chicago, Illinois.

Another turn-key solution is the Cirrascale FOREST(Flexible, Open, Reliable, Energy-efficient, Scalable, andTransportable) [8] container. The FOREST container is soldas a complete solution based on the BladeRack 2 platformprovided by Cirrascale, with support for other standardthird–party equipment. The container itself is a standard40’ configuration with a maximum of 2,880 servers or 26PB of storage when used as a drop–in storage component.Cirrascale’s cooling approach for their blade configurationsis based on a “vertical cooling technology” where cool air isdrawn up from directly beneath the racks and exhausted outthe top instead of the standard hot/cold aisle arrangement.The base facility PUE is 1.05, increasing to approximately1.3 when the chiller load is included.

Sun Microsystems also offered a complete modular datacenter solution before their acquisition by Oracle, originallynamed Blackbox and later renamed to Sun Modular Data-center (SunMD) [3]. The system consisted of a set of 20’containers with 7 or 8 40U racks in one and mechanicaland electrical components spread among the others. A fewsystems were installed at locations such as the Stanford Lin-ear Accelerator Center in 2007 before the Oracle acquisitionand subsequent uncertainty about the future of HPC underOracle.

Other vendors such as IBM, APC, and Cisco have alsobegun offering products in this market, but unfortunatelydetailed information is not generally available. IBM providesa Portable Modular Data Center (PMDC) as part of their“Project Big Green” initiative [2], with a PUE range of1.35 to 1.5 advertised. Cisco’s product places the UnifiedComputing System (consisting of a data center architecturewith server hardware and software) into a container.

2.2 Component and Integration SolutionsA variety of vendors have emerged as providers of con-

tainer facilities and integrators of third–party equipment.Containerized, modular data centers are targeted at thecomplex and dynamic needs of a variety of entities, fromWeb 2.0 startups to large commercial and governmentorganizations. A rich ecosystem has emerged for thesevarious needs, allowing for detailed co–design with somevendors for those customers with precise requirements aswell as turnkey–based solutions for customers requiring rawcompute power for applications such as cloud computinginfrastructure. The rapid time to delivery (some vendorspromise 6 weeks from order to installation) allows for moreresponsive, adaptive, and expandable computing infrastruc-tures than was previously possible in a traditional monolithicdata center design and procurement strategy.

SGI provides a containerized data center in their ICECube [6] product line, formerly by Rackable systems. These20’ or 40’ containers are configured in a dual–row arrange-ment with up to 1920U of available space. A coolingcapacity of 1500W/ft2 is provided by in–row chilled water.Additional power savings are realized when SGI’s DC powertechnology is used, eliminating fans in rack equipment andleading to a PUE of less than 1.12.

The AST Modular Smart Shelter [10] container hosts 19racks of equipment in a 40’ container with a power densityfrom 5kW/rack to 30kW/rack. A variety of configurationsare available, supporting the standard ISO intermodalcontainer sizes. These containers can be stacked similar tothe Dell product and are rated to be fire resistant for 120

minutes. The cooling system has a condensation–avoidingdesign with optional air-side economizer (ASE).

The Bull mobull [7] container solution integrates theirwater–cooled rear cabinet doors to provide up to 40kW perrack cooling capacity while also helping eliminate equipmenthot spots.

Other companies have been using custom containerizedsolutions for their datacenter needs for some time. Googlehas been operating a containerized datacenter since 2005 [4],consisting of 45 containers on two levels of a warehouse–typefacility. Each container holds 1160 servers and is provisionedwith 250 kW of power for an overall power density of780W/ft2. Water cooling from a centralized cooling plantis used in conjunction with free cooling with the outside air,providing an overall facility PUE of 1.25.

3. DESIGNAs part of the National Science Foundation grant pro-

posal to acquire the computational system, the Universityadopted the solution of renovating an existing 1980s-eradata center facility. This pre-existing facility providedadequate floorspace (near 10,000 ft2), but shared power andcooling with a large amount of office space, a cafeteria, andconference facilities. While the facility renovation plan wasbeing developed, additional tenants rented office space in thebuilding, reducing the available power and cooling below theamount required to support the incoming system.

We then started an extensive analysis of the possibleoptions available to provide adequate facility infrastructureto support the supercomputer installation. We evaluatednearly a half-dozen possible locations for a new datacenter,mostly consisting of lightly used lab space, unused load-ing/shipping areas, large areas of office space, and severalroof areas possibly suitable for development. In the mean-time, our vendor partner proposed a modular containerizedsolution, consisting of three independent containers and atransformer located on a large concrete pad. The threetemporary structures housed the computational system,electrical distribution, and cooling equipment, and shippedindependently and coupled together onsite to provide anintegrated solution. Although we iterated over this proposedsolution for many months, the original vendor partnerdeclined to manufacture the solution after a corporatetakeover. This freed us to examine the facility and systemsolution space with greater knowledge of modular datacenters, resulting in the final co-design process describedbelow.

3.1 Computational System SummaryConstructing a facility to house the Janus system was

challenging due to the system’s size – and thus its powerand cooling requirements. The final Janus computationalsystem consists of 1,368 nodes, each containing two six-core Intel Xeon Westmere-EP chips at 2.8 GHz, for a totalof 2,736 processors and 16,416 cores with a theoreticalpeak performance of 184 TFLOP/s. Each core has 2GB of 1333 MHz DDR3 RAM for a total of 24 GB pernode and 32 TB of memory in the entire system. Nodesare connected using a fully-nonblocking QDR Infinibandnetwork comprised of three 648-port Mellanox spine chassisand 79 36-port leaf switches; a separate Ethernet network isused for management. Two Data Direct Networks SFA10000couplets with 300 disks per couplet provide an 860 TB

storage system with a combined I/O throughput of up to20 GB/s. The system’s footprint includes 17 48U racks ofcompute nodes, 3 racks dedicated to the Infiniband coreswitches, and 2 racks containing administrative, network,and storage equipment. The compute system has a sustainedload under HPL of just under 500 KW.

3.2 Co-design ProcessOur co-design process specifies the design of the facility

in parallel to the design of the HPC resource. The primarybenefit of this process is that the complete solution canbe designed and implemented in approximately the sameamount of time as the design and implementation of alarge-scale HPC resource alone. Not only are the designprocesses matched, but the design timelines are matched aswell. In addition, the co-design process allowed us to selectand size the facility parameters such as computer room airconditioning (CRAC) units, pumps, water pipes, and thecooling tower to match this resource and provide for thedesired expansion capability. This improved the efficiencyand allowed the facility to take special power and coolingrequirements into account at design and build time.

Co-designing and implementing the facility with the com-putational resource also allowed us to merge the imple-mentations into a single project with a single project man-ager and support staff, and coordinate the implementationschedules of each component at the granularity of a singleday (see Figure 1). The facility solution that we selectedis a pre-fabricated but made-to-order, full 15-year lifespandatacenter including all facility equipment required forpower distribution, HVAC, monitoring, fire suppression, etc.(see Figure 2). The facility was pre-built and tested bythe vendor, then shipped to arrive several days before thearrival of the system racks and hot-aisle containment. Theuniversity was solely responsible for routing the main utilitypower feed, utility fresh water supply and sanitary drains,as well as pouring the concrete pad and entrance ramp. Therest of the facility was provided as part of this project.

The key design consideration was to achieve a PUEthat meets or exceeds that of a number of comparable“free cooling” data centers that also utilize outside aireconomization, but while maintaining a sealed datacenterenvironment. While many datacenters use outside air econ-omization for significant efficiency gains, it requires largefiltering assemblies that are maintenance-intensive due tothe large volumes of outdoor air being continually exchangedwith the space. As the Janus system is located in ametropolitan areas and is close to a bus garage and servicingfacility, and to mitigate the affect of rapid changes inoutdoor humidity levels, maintaining traditional datacenterair integrity was a chief concern. We also sought to achievesignificant savings over an outside air system by reducing fanhorsepower to the absolute minimum required to handle theequipment regardless of the load with no wasted airflow. Theresulting Janus facility achieves efficiencies while eliminatingmany of the potential hazards, additional operating costs,or efficiency penalties that come with continually filteringor treating outside air in the large quantities requiredto cool a datacenter. It also fully integrates the freecooling source with the mechanical source with seamlessswitchover between several cooling modes while maintainingpeak efficiency at all times.

3.2.1 Holistic Design ApproachAnother benefit of our co-design process is that it allowed

the use of a holistic design approach that matched thefacility to the system. Instead of addressing each element inthe facility separately and then maximizing each componentbased on their individual budget, all resources were pooledand the approach focused on providing the most efficientsolution while staying within the total budgeted cost.

Typical design process. In the typical design process,the facility’s mechanical engineer receives the total heatrejection information from the system designer, which be-comes the design basis for the datacenter’s cooling system.The first optimization is to minimize the cost or maximize

Oct 22-28 Oct 29-Nov 12 Oct 10-14 Oct 4-10 Oct 15-21

Cable Vendor Test Connect Assemble Ship Build 1 week 2 weeks 1 week 1 week 1 week 3 months

Con

tain

er C

lust

er

Container build at Epsilon in Canada

(July 1-Oct 3)

System build and test at Dell in Texas

(June)

Finish container assembly

Container shipment Plumbing

and electrical

System installation

System shipment

Hot aisle construction

Infiniband cabling

Hardware and Infiniband testing;

LINPACK performance

Primary facility

integration

Contractual acceptance

testing

Con

tain

er B

uild

– O

ne D

ay

Figure 1: The co-design process used to deploy the Janus facility and system. The facility building wasassembled in Boulder in one day, with teams concurrently finishing the facility infrastructure and assemblingthe computational system.

the capacity from the datacenter cooling system, so theengineer selects CRAC units based on their ability to addressthe greatest heat load in the smallest footprint, cost, orquantity. The chilled water system is then designed tomeet the requirements of the CRAC units and their requiredtemperatures and flow rates. The result is usually highflow rates utilizing high-horsepower pumps with minimaldifferential water temperatures. Efficiency of the chilledwater system is typically maximized at this time (if at all).Often, this is where efficiency initiatives fail due to poorpayback or minimal overall efficiency improvement. Thisis typically due to the wrong component being selectedfor optimization, aggravated by segregated budgets. Theelectrical requirements to support this system are thenpassed on to the electrical engineer to be implemented withtheir distribution system design.

Employing this philosophy, the Janus project would haveused 45◦F chilled water with a 10◦ delta and a 55◦ returntemperature in order to maximize the coil surface area ofthe supplied in-row cooling units. The chiller would berequired to be a nominal 337 tons for complete build-out,with the pumps and piping systems large enough to supportthe required high flow rates to meet the needs of the in-row cooling units. While this system would still supportfree cooling and precooling, it would be at a reduced levelsas the required ambient wet bulb temperatures would beinfrequent.

Holistic design process. The holistic design processenabled by the co-design of the Janus system involves ex-amining every element available – structural, civil, electrical,mechanical, available cooling technology, local climate con-ditions, and IT technology – and not stopping at optimizingany one element, but as many as possible. In a holisticdesign, the engineering team has access to as many ofthe actual datacenter load data specifics as possible, suchas kW per Rack, CFM/kW, or server design differentialtemperatures on a rack-by-rack basis. This knowledge allowsthe engineer to maximize the efficiency of the space.

In the holistic design, total heat rejection (even be it afuture design limit, often established by the UPS or electricaldistribution size and measured in BTUs or in kW) remainsthe common metric used to design the space. However, rack-by-rack specifications can be used to establish additionalparameters on the cooling system. For the Janus facility, weexamined the affect of various chilled water temperatures onchiller and in-row CRAC performance, as well as their abilityto maximize free cooling. We settled on a 60◦F supplytemperature because it was the maximum that could besustained by the chiller without refrigeration issues. We thenselected in-row cooling units with hot-aisle containment dueto their performance with the elevated supply temperature.We concurrently investigated the chilled water flow rates forboth the chiller and CRAC units, selecting a 16◦F deltabecause beyond the 60◦F supply with 76◦F return (16◦Fdelta), the returns for efficiency versus the reduced capacitydid not fit our application.

The ability to examine and adjust all facility systems –and more importantly, the ability to allocate funds to thecomponents that would provide the most benefit – allowedus to optimize both initial cost and operational cost. A fewresulting affects on the system are as follows:

• By coupling heat rejection with cooling in an in-rowform factor in a hot-aisle containment system, we

Figure 2: Janus facility infrastructure and compu-tational system layout.

were able to eliminate mixing and maximize our coildifferential temperatures on the air side. This alsoallowed us to minimize the fan horsepower requiredfor the solution.

• Using elevated cold aisle temperatures allowed us toincrease the supplied chilled water temperatures andextend the differential on the water side to 16◦F.

• For capacity, the facility required additional coil sur-face area to make up for the increased water temper-atures and to support the elevated water differential.This resulted in an increase in the number of coolingunits by 43% – eight more CRAC units – to achieve therequired coil surface area. An added benefit was thatthis significantly reduced fan horsepower requirements,as the flow was now spread across more units allowingthem to operate at much lower fan speeds, furtherreducing power consumption.

• The 16◦F chilled water delta also allowed us to reducethe system pipe sizes due to reduced flow, and alsoreduce pump horsepower and therefore power con-sumption of the system.

• The elevated chilled water temperatures allowed usto maximize the number of free cooling hours ofoperation. It also increased the capacity of our chillerby approximately 35%; our chiller was rated by Traneat 337 tons in what is normally a 250-ton chiller frame.

Overall, the use of the holistic design required the additionof eight additional cooling units to provide adequate cooling,but those additional costs were offset by more than threeto one by the savings on the mechanical and electricalside of the project in significantly reduced material andinstallation costs. The end result was a project that cost lessto construct and less to operate as well. Overall, poolingall construction resources together in a holistic approachprovided the engineers the opportunity to communicate withone another and make the appropriate decisions to maximizethe system as a whole, rather than as separate parts to beassembled.

(a) Exterior of Janus facility. (b) Interior of Janus facility.

Figure 3: The Janus facility.

3.3 Facility DetailsThe facility (see Figure 3) provides two separate rooms

inside of the single structure: a mechanical room and araised floor datacenter. The entire facility is built using astructural steel insulated (R-16) framework complete withroof-mounted lifting lugs for transport. The mechanicalspace floor is a continuously seam welded 3/16” steel checkerplate, and the datacenter room uses a raised floor above thein-row cooling unit water piping supported by steel joists.The building is rated to 150-mph wind and 40 lb/ft2 snowloads, and the enclosure is factory painted and is rated fora 500-hour salt spray test.

Both the mechanical and the datacenter spaces are pro-tected by fire suppression systems. The mechanical roomis protected by an ANSUL Inergen fire suppression system,while the datacenter space is protected by a Sapphire (Novec1230) clean agent system. Both the Inergen and Sapphiresystems are halon alternatives suitable for total floodingapplications in occupied spaces. Moreover, they are arenon-toxic and non-conductive, making them a suitable non-destructive fire suppression mechanism for the computingequipment and supporting mechanical space. The datacen-ter room also has the Vesda Early Warning Smoke Detectionsystem, which is able to detect minute levels of smoke (belowlevels detectable by a human) to generate early alerts ofany datacenter fire and equipment events. Two externaland nine internal motion sensor detection cameras capturevideo and provides historical trending while the container isoccupied.

The mechanical room includes all of the componentsrequired to provide complete support of the datacenterspace. Power is provided by a 2000A (480V, 3-phase, 3-wire) utility transformer feed that was installed on siteto support the complete system build-out. (The facilityhas sufficient space to facilitate expansion for an additional66% of rack space.) Similarly, a fully-metered Square-D2000A main panel located in the facility mechanical spacefeeds the initial five in-row 150kW 480V-120/208V powerdistribution units (PDUs). The PDUs have branch circuitmonitoring to provide metered power distribution to thecompute cabinets and in-row cooling units. Incoming powertransients and spikes are conditioned by a Selenium-based

Transient Voltage Surge Suppression unit. Only a portion ofthe facility is protected by the 80kW N+1 Redundant UPSsystem, namely the administrative, storage and networkingequipment, as well as a few selected CRAC units.

Heat exchange is primarily provided by a Marley evapo-rative water tower, with the “free cooling” and pre-coolingoperational regimes utilizing a Mueller flat plate heat ex-changer. Mechanical cooling comes from a 337.10 ton Tranechiller. At installation time, the system was configuredto use a 60◦F supply temperature system, with a design16◦F delta-t (5-lbs delta-p) for a 76◦F return temperature.Redundant pumps were utilized on both the chilled waterand condenser water sides.

Inside the datacenter space, an APC InfraStruXure hot-isle containment system houses the computational system,including all compute, administrative and networking equip-ment using 48U server racks with overhead cable manage-ment, and uses 24 ACRC500 heat exchangers spread acrossfour rows of equipment in two hot aisle containment systems.

4. FACILITY PUE ANALYSISThe PUE design target of the facility is 1.2. Over the past

6 months we have gathered operational data on the efficiencyand performance of the coupled system. The facility hasconsistently provided a PUE of 1.059 under load, and 1.12while the resource has been idle (approximately 240kW).During this period, the facility has been operating in free-cooling mode, utilizing the external evaporative water towerand flat-plate heat-exchanger only. For much of this periodeven the cooling tower fan was operating at a minimal powerdraw, and the action of cycling the water through the towerwas sufficient to discharge the majority of the waste heat.

We performed a series of tests during a full-day facilityreview and analysis downtime with the following goals:

1. Verify the full operation and configuration of thefacility

2. Develop a model of the resulting PUE in operationover all of the regimes

3. Augment our operational experience

Min Wet Bulb Temp Operational Regime Expected Hours/Year Worst-Case Anticipated(◦F) PUE PUE67 Maximum Mechanical Cooling 39 1.340 1.28363 MMC with reduced Fans and Pumps 21 1.334 1.28062 Onset Precooling with MC 855 1.306 1.24656 Medium Precooling with MC 3001 1.286 1.20642 Max Precooling with Min MC 278 1.229 1.18341 Full Free Cooling, No MC 3324 1.143 1.117- FFC with reduced Fans and Pumps 1242 1.137 1.114

Yearly Average: 1.211 1.163

Table 2: HVAC Operational Regimes and Anticipated PUE for 350kW Load

4. Confirm our projected power utilization for each of themajor facility components

We systematically modified the facility parameters tostress individual components, allowing us to verify andmatch each component’s power draw and load to themanufacturer’s specifications. For example, we dropped thesupply water temperature from 60◦F to 55◦F, then to 50◦F,and then to 45◦F while monitoring the power draw of thechiller and other components. Because each branch circuitis monitored we were easily able to record the power drawof each component.

These tests verified our assumptions and holistic design.Table 2 summarizes the calculated PUE for the operationalregimes encountered over the course of a year given anominal load of 350kW, representing the approximate loadof the system under a broad array of user applications usedthus far.

Figures 4 through 6 show the relative proportion of timespent in each of the operational regimes along with theiraverage PUE in one figure and the contribution of each of themajor facility components in the other. Figure 4 shows theanticipated yearly PUE results for a 350kW load, and Figure5 shows the worst case scenario, also for a 350 kW load.These represent the anticipated operation of the system ascurrently installed. Figure 6 shows the predicted operationalcharacteritics if the system were expanded to a full 1 MWload.

5. DISCUSSION AND CONCLUSIONSDesigning and deploying a facility in this manner resulted

in a number of significant advantages and benefits over theother alternatives. The first major benefit was simply theamount of time that it saved in both the development anddeployment and the result of having both the facility andsystem deployed in the same quarter. The resulting synergyof the facility and system designs, operation and overallbalance has proven to have had a significant impact on theefficiency. This process also allowed us to avoid significanttechnical risk in the implementation of either component,and minimized the number of decision makers involved inthe development and review processes.

The solution described here emerged from our opportunityto redesign and was driven by our strong desire to developa tighter integration between the resource and the facility.Extensive coupling between the resource and facility pro-vides tremendous benefits to the efficiency of the overallfacility operation by: 1) balancing facility components and

system load and 2) incorporating information about thesystem’s operational status. Instead of treating the loadas an unpredictable, dynamic entity, this coupling providesthe ability for the system to interact with the facility inan integrated and cohesive way. This allows the systemto adjust the facility set points, in-row cooling unit fanspeeds, etc. to reliabliy and predictably set the facilityparameters, thus enabling the efficient, predictable, powerand performance-aware execution dictated by individualapplications, projects and the operational policies desiredby the administration.

This coupling also allows the administrative and manage-ment staff to couple the software systems for automatedpolicy modification based on scheduler and administrativerequirements, tune the solution for efficiency, performance,temporary power limitations and partial system downtimes,manage the facility during power constraints, and instantlyreact to facility events in a very agile manner. Forexample, the administrative staff may evaluate and deploy a“performance” period each week where the facility is run tooptimize the cooling of the nodes and sustain a higher per-core performance over the period via the Intel Turbo modebuilt into their newer-generation processors).

Perhaps the greatest risk in building a new facility is anunexpected paradigm shift in support infrastructure, suchas the change from air cooling to liquid cooling based onincreasing system power densities. Developing the buildingalongside the resources guarantees a parity between thefacility design and operational requirements and minimizesthe facility and system risks inherent in extremely densearchitectures, all while tuning the efficiency and minimizingthe long-term economic impact of the facility, reducing theamount of time to recoup investment, and supporting amission of investing in ”green” infrastructure.

Although the historical arc of this project took us througha fair number of facility and system designs, the end resulthas proven itself to be more flexible and cost-effectivethan we originally thought possible. We anticipate thatthis solution will provide the University of Colorado areliable and efficient datacenter space not only for the fulloperational life of the current resource, but also for the full15-year lifespan of the facility.

6. ACKNOWLEDGMENTSSupport for the Janus project was provided by NSF-MRI

Grant CNS-0821794, MRI-Consortium: Acquisition of aSupercomputer by the Front Range Computing Consortium(FRCC), with additional support from the University of

1.00!

1.05!

1.10!

1.15!

1.20!

1.25!

1.30!

1.35!

1.40!

0! 10! 20! 30! 40! 50! 60! 70! 80!

PUE!

Wet Bulb Temperature (Min)!

(a) Facility PUE by wet bulb temperature (surface areaindicates contribution to yearly average).

0! 50! 100! 150! 200! 250!

67!

63!

62!

56!

42!

41!

26!

Total Facility kW!

Wet

Bul

b Te

mp

(Min

)!

Chiller! Chilled Water Pump! Condensor Pump!Cooling Tower Fan! HACS! Conversion Losses!Misc/ Lighting!

(b) Facility power breakdown by component.

Figure 4: Anticipated PUE and power breakdown for 350KW load.

1.00!

1.05!

1.10!

1.15!

1.20!

1.25!

1.30!

1.35!

1.40!

0! 10! 20! 30! 40! 50! 60! 70! 80!

PUE!

Wet Bulb Temperature (Min)!

(a) Facility PUE by wet bulb temperature (surface areaindicates contribution to yearly average).

0! 50! 100! 150! 200! 250!

67!

63!

62!

56!

42!

41!

26!

Total Facility kW!

Wet

Bul

b Te

mp

(Min

)!

Chiller! Chilled Water Pump! Condensor Pump!Cooling Tower Fan! HACS! Conversion Losses!Misc/ Lighting!

(b) Facility power breakdown by component.

Figure 5: Worst case anticipated PUE and power breakdown for a 350KW load.

1.00!

1.05!

1.10!

1.15!

1.20!

1.25!

1.30!

1.35!

1.40!

0! 10! 20! 30! 40! 50! 60! 70! 80!

PUE!

Wet Bulb Temperature (Min)!

(a) Facility PUE by wet bulb temperature (surface areaindicates contribution to yearly average).

0! 50! 100! 150! 200! 250!

67!

63!

62!

56!

42!

41!

26!

Total Facility kW!

Wet

Bul

b Te

mp

(Min

)!

Chiller! Chilled Water Pump! Condensor Pump!Cooling Tower Fan! HACS! Conversion Losses!Misc/ Lighting!

(b) Facility power breakdown by component.

Figure 6: Anticipated PUE and power breakdown for 1MW load.

Colorado and NSF sponsorship of the National Center forAtmospheric Research. Thanks to the following individualsfor their assistance with the Janus system and facilitydevelopment and deployment, and for their assistance withthis paper: Paul Sorbo and Michael Groft (Dell Inc.);Benjamin Mayer and Jon Lusk (NCAR); and Theron Voranand Paul Marshall (CU Boulder).

The following institutions and individuals contributed tothe Janus project: APC: Ross Earley, Jean Kulasewski, andRyan Yeck; Arista: Tom Flaherty; Braconier HVAC:John Durant, Tom Gill, and Rich Hause; Criticial Fa-cilities Technology: Juston Vogt; Data Direct Net-works: Joseph Josephakis and Rick Scott; Dell: CarolynArredondo, Stephanie Baewer, Chris Bergen, Paul Betan-court, Neal Carr, Jeff Davis, Darren Estridge, MichaelFountaine, Chuck Gilbert, Joseph Guillory, Jeff Hall, DougHughes, Scott Jacobs, Russell Kelly, David Kewley, TravisMartin, Wes McCarty, Joey Meijer, Robert Norris, KurtOlsson, Renee Plemons, David Saim, Raghav Sood, JimStone, Brent Strickland, and Doug Taylor; Epsilon: TreyAustin, Alford Higgins, Bruce Johnston, and Karl Mc-Farland; Fire Protection Concepts: Paul Garcia andMark Neal; Intermountain Electric: Dave Burris andVince King; Ken Sons LLC: Ken Sons; Mellanox: GlennChurch, Matt Finlay, Pranav Patel, Todd Wilde, and SteveWilliams; NCAR: John Dennis and Jose Garcia; Trifu-sion: Thomas Hardt, Polo Jaimes, and Frank Veierstahler;University of Colorado: David Bodnar, Jazcek Braden,Chris Ewing, Joel Frahm, John Hanks, Thomas Hauser,Don Johnson, Andy Jordan, Paul Leef, Larry Levine, EricSchoeller, and Kimberly Stacey. The exterior photograph ofthe Janus facility was provided by University Communica-tions of the University of Colorado at Boulder.

7. REFERENCES[1] DELL Humidor Modular Data Center.

http://www.theregister.co.uk/2008/12/13/inside dell containers/, April 2008.

[2] IBM Project Big Green Tackles Global Energy Crisis.http://www-03.ibm.com/press/us/en/pressrelease/24395.wss, June2008.

[3] Sun Modular Datacenter.http://www.datacenterknowledge.com/archives/2008/01/29/sun-rebrands-blackbox-as-sun-md/, April 2008.

[4] Google Unveils Its Container Data Center.http://www.datacenterknowledge.com/archives/2009/04/01/google-unveils-its-container-data-center/, April2009.

[5] NREL’s plans and strategies for building green datacenters.http://info.ornl.gov/events/nlit09/Presentations/NRELS Plans and Strategies for Building Green DataCenters-Chuck Powers.pptx, March 2009.

[6] SGI ICE Cube. http://www.sgi.com/pdfs/4160.pdf,January 2009.

[7] Bull mobull Plug and boot data center.http://www.bull.com/extreme-computing/download/S-Mobull-en2Web.pdf, January2010.

[8] Cirrascale FOREST Container.http://www.cirrascale.com/documents/Cirrascale

FOREST Container.pdf, April 2010.

[9] HP Performance Optimized Datacenter (POD).http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-0122ENUS.pdf, 92010.

[10] AST Modular Smart Shelter Container.http://www.astmodular.com/pdf/AST-Modular Smart-Shelter-Container datasheet.pdf,April 2011.

[11] National Petascale Computing Facility (NPCF).http://www.ncsa.illinois.edu/AboutUs/Facilities/npcf.html,March 2011.

[12] NCAR-Wyoming Supercomputing Center (NWSC).http://nwsc.ucar.edu, April 2011.

[13] NCAR-Wyoming Supercomputing Center (NWSC)Specification Sheet.http://nwsc.ucar.edu/sites/default/themes/nwsc/shr/docs/NWSC crib sheet.pdf, April 2011.

[14] The Top500 List. http://top500.org/list/2010/11/100,April 2011.

[15] CISL FY2008 Annual Report. Production deploymentof bluefire supercomputer.http://www.nar.ucar.edu/2008/CISL/2ci/2.1.2.bluefire.php, 2008.

[16] E. R. Jessup, H. M. Tufo, and M. S. Woitaszek.Building an HPC watering hole for Boulder areacomputational science. In ICCS 2005: Proceedings ofthe Workshop on High Performance Computing inAcademia: Systems and Applications, pages 91–98,Atlanta, Georgia, USA, May 2005.

[17] M. Oberg, M. Woitaszek, T. Voran, and H. M. Tufo.A system architecture supporting high-performanceand cloud computing in an academic consortiumenvironment. In Proceedings of ISC’11: InternationalSupercomputing Conference 2011, Hamburg, Germany,June 2011. Accepted.

[18] M. K. Patterson, D. Costello, P. Grimm, andM.Loeffler. Data center TCO; a comparison ofhigh-density and low-density spaces. Technical report,Intel Corporation, 2007.http://www.intel.com/technology/eep/datacenter.pdf.

[19] W. P. Turner IV and K. G. Brill. Cost model: Dollarsper kW plus dollars per square foot of computer floor.Technical Report TUI3028A, The Uptime Institute,Building 100, 2904 Rodeo Park Drive East, Santa Fe,New Mexico, 87505, 2008.http://uptimeinstitute.org/wp pdf/(TUI3029A)CostModelDollarsperkWPlusDollars.pdf.