cern computer facilities evolution

Slide 1

CERN Computer Facilities EvolutionWayne SalterHEPiX May 2011Computing FacilitiesCERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCFCERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF1OverviewReminder on the current status of CERN Computer FacilitiesOverview of the current issues and anomaliesSummary and status of the various Evolution ProjectsClosing remarks

CERN Computer Facilities Evolution - HEPiX May 2011 - 2CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF2CERN Computer Facilities OverviewDesigned and built in early 1970sFully refurbished around 2000Increased power and improved coolingNominal current capacity: 2.5MW (including 240kW of critical power)Extended (usable) capacity: 2.9MW (including 340kW of critical power) forfeiting redundancy for all UPS systemsBut need to take action in the event of a UPS module failure!Small capacity at local Hosting Centre (17 racks and up to 100kW)CERN Computer Facilities Evolution - HEPiX May 2011 - 3CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF3Current Issues and AnomaliesCooling for critical UPS room insufficientMixing of critical and physics equipmentLocation and coolingNo cooling of CC when running on UPS and insufficient stored cooling capacity when on dieselInsufficient critical power availableApproaching the limit of available power for the buildingNo redundancy for critical UPSUsage of full available 2.9MW implies loss of redundancy for physics UPSA/C of CC coupled to adjacent office buildingHow to meet CERNs needs in longer-termCERN Computer Facilities Evolution - HEPiX May 2011 - 4CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF4Evolution OptionsLocal hostingProvide additional critical powerAllow some level of business continuityOn-going improvements of CCImprove efficiency and resilience of cooling systemUpgrade of current computer centreIncrease capacity from 2.9 to 3.5MWIncrease critical power from 340 to 600kWAddress a number of long-term issuesRemote hostingAddress the increasing capacity needs of CERNIncrease business continuity coverageCERN Computer Facilities Evolution - HEPiX May 2011 - 5CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF5 CERN Computer Facilities Evolution - HEPiX May 2011 - 6Local Hosting OverviewReason:Lack of critical powerProvide some level of BCGain experience with remote hostingHistoryPrice enquiry for hosting August 2009Tender for networking October 2009Planned start April 2010Actual local hosting contract start 15th June 2010Connectivity contract start 1st July 2010Contracts for local hosting and network connectivity renewed for a second yearCERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF6CERN Computer Facilities Evolution - HEPiX May 2011Summary of Current StatusAgreement for 40m (actually 36m) and 100kW (two power feeds)2 dark fibres on diverse paths17 IT racks + 2 network racks209 systems installed14 racks used and 57kW (average power density of 4.1kW/rack)3 racks remaining and 43kW!Interventions at Local Hosting site:Small number of Sys Admin interventions (2-3)*7-8 Service Manager interventions*33 Vendor interventionsMany installation interventions

* Worrying in terms of real remote hosting!CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF7

Layout CERN Computer Facilities Evolution - HEPiX May 2011 - 8CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF8IssuesEverything took longer than foreseen!Contracts with hosting company and network providerPreparation of CERN room at hosting companyPartitioning, power and fibre connections, access card reader, access to InsideEyes, Getting equipment into productionNetwork connectivity expensiveRoom smaller than expected (36 c.f. 40 m )Problems with rampHowever, no significant problems running systems remotely as far as we have seen so farCERN Computer Facilities Evolution - HEPiX May 2011 - 9CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF9Summary of Local HostingDespite a few teething problems the experience is generally goodHowever, due to proximity not everything done remotelyNeed to understand reasons and how to avoidStill not at full capacity Struggling to utilise full available powerLow average rack power density (~4kW)Good step towards remote Tier0 hosting CERN Computer Facilities Evolution - HEPiX May 2011 - 10CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF10INTEL suggested improvementsReport from INTEL received recently based on a review of CC done at end of 2010:Improve monitoring of power, temperature (inlet, outlet, return air), humidity, etc.Improve/balance the airflowUse variable speed fans for AHU (air flow is twice the required rate)Separation of cooling for critical/non-critical equipmentRun CC at higher temperatureImplement free cooling properlyImprove the exhaust route for hot air

CERN Computer Facilities Evolution - HEPiX May 2011 - 11

Wind direction11CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF11On-going improvementsWork already doneImproved monitoring (allows us to calculate PUE fairly accurately)Correction of air input temperatureReduce intake of hot exhaust airImprove mixing of outside and re-circulating airFull automation of air selection (% of outside vs. re-circulating)Change to anti-freeze protection (delay and better mixing)Above measures predicted to result in 80% saving for chiller powerAll control system components connected to UPSForeseen potential further improvementsHigher server inlet temperatureUnderstand the pressure drop from AHU to serversOptimize air flow (across all aisle)Mixing depending on RHUse variable speed fans for AHU to reduce air flow to only what needed CERN Computer Facilities Evolution - HEPiX May 2011 - 12CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF12Goals of the CC Upgrade ProjectSolve the cooling issue for the critical UPS roomNew UPS systems in a different locationIncrease critical capacity to 600kWIncrease overall power capacity to 3.5MWRestore N+1 redundancy for both critical and physics UPS systemsSecure cooling for critical equipment when running on UPS and extend stored cooling capacity for physics when on dieselDecouple the A/C for CC from the adjacent office buildingCERN Computer Facilities Evolution - HEPiX May 2011 - 13CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF13Power RequirementsCERN Computer Facilities Evolution - HEPiX May 2011 - 14IT Physics EquipmentIT Critical EquipmentPhysics UPS(4+1)Diesel Generator (N+1)Critical UPS(3+1)600kW600kW2.9MW3.5MWNormal NetworkCooling forIT Critical EquipmentCERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCFCritical = 4*400kVAPhysics = 4*5*400kVA14ApproachBarn area of CC building to be converted to house:3 new electrical roomsAn IT room with water cooled racks to house critical equipment (up to 450kW)New ventilation systems for the critical area of the main computer room (and also the critical UPS room in basement)New ventilation systems for the Telecoms rooms (CIXP) will be installed in adjacent officesNew critical UPS systems in basementA new partially sunken building for additional chillers and a storage tank for the cooling of the critical areasAn additional storage tank for extending the stored cooling capacity for physics equipmentOpportunity to install emergency evacuation stairs CERN Computer Facilities Evolution - HEPiX May 2011 - 15

CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF15


CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF16CERN Computer Facilities Evolution - HEPiX May 2011 - 17






CERN Computer Facilities Evolution - HEPiX May 2011 - 21CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF21Schedule and StatusBarn cleared of IT equipment end of OctoberRemoval of cabling and ducting finishedCivil engineering (CE) work just commencingDelayed due to difficulty in freeing officesCE works to be completed November 2011EL+CV installations Nov/2011-Nov/2012Increased physics power available Aug 2012 and increased critical power Nov 2012Project takes a long time and has high cost!Barn Video

CERN Computer Facilities Evolution - HEPiX May 2011 - 22CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF22Remote Tier0 Hosting Some HistoryHow to provide resources once CERN CC full?Studies for a new CC on Prvessin siteFour conceptual designs (2008/2009)Lack on site experienceExpensive!Lack of support from managementInterest from Norway to provide a remote hosting facilityInitial proposal not deemed suitableFormal offer not convincingInterest from other member statesCERN Computer Facilities Evolution - HEPiX May 2011 - 23CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF23StatusCall for interest at FC June 2010How much computing capacity for 4MCHF/year?Is such an approach technically feasible?Is such an approach financially interesting?Deadline end of November 2010ResponseSurprising level of interest 23+ proposalsWide variation of solutions and capacity offeredMany offering > 2MWAssumptions and offers not always clearly understoodWide variation in electricity tariffs (factor of 8!)CERN Computer Facilities Evolution - HEPiX May 2011 - 24CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF24Follow UpVisitsVisit to many sitesOthers invited to CERNGoalTo understand better the proposalsClarify CERNs needsBenefitsSee existing installationsTriggered us to reconsider some of our ideas/preconceptionsAllowed consortia to understand better our needsCollect information for technical specificationCERN Computer Facilities Evolution - HEPiX May 2011 - 25CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF25ScheduleProposed timelineOfficial decision on whether to go ahead spring 2011Official letter to be sent explaining procedureTender during summer 2011 for adjudication end 2011/early 2012Initial installation first half of 2013 to test operational modelGradual build up in capacity in-line with experiment needsCERN Computer Facilities Evolution - HEPiX May 2011 - 26CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF26SummaryCERN CC reaching end of capacityFurther improvements in CC are possible and being implemented in parallel (e.g. monitoring and cooling)Better monitoring capabilities Better efficiency, but We do not get additional computing capacity from this!Three options to address providing increased capacity:Local hostingLimited additional capacityCC Upgrade to 3.5 MWSlow and expensiveRemote hostingInteresting but introduces new challengesBut could allow us to address business continuity properly

CERN Computer Facilities Evolution - HEPiX May 2011 - 27CERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF27 Presentation Title - 28

BackCERN IT DepartmentCH-1211 Geneva 23Switzerlandwww.cern.ch/itCF28

cern computer facilities evolution

Documents