overview of sizing model and inputs into ldm-144€¦ · web viewthe structure and relationships...

Large Synoptic Survey Telescope (LSST)Site Specific Infrastructure Estimation

Explanation

Mike Freemon and Steve Pietrowicz

LDM-143

Latest Revision: July 25, 2014

This LSST document has been approved as a Content-Controlled Document by the LSST DM

Technical Control Team. If this document is changed or superseded, the new document will

retain the Handle designation shown above. The control is on the most recent digital

document with this Handle in the LSST digital archive and not printed versions. Additional

information may be found in the LSST DM TCT minutes.

LSST Site Specific Infrastructure Estimation Explanation LDM-143 7/25/2014

Change Record

Version Date Description Owner name

1 5/13/2006 Initial version (as Document-1684) Mike Freemon

2 9/27/2006 General updates (as Document-1684) Mike Freemon



5 4/11/2012 Modified rates for power, cooling, floorspace, shipping Mike Freemon

6 10/10/2013 TCT approved R Allsman

4 7/25/2014 Added Appendix A J. Kantor

The contents of this document are subject to configuration control by the LSST DM Technical Control Team.

i


Table of ContentsChange Record.............................................................................................................................................. i

1 Overview of Sizing Model and Inputs Into LDM-144............................................................................1

2 Data Flow Among the Sheets Within LDM-144....................................................................................2

3 DM-BaseSite ICD (LSE-77)....................................................................................................................3

3.1 DM Power Capacity......................................................................................................................3

3.2 DM Rack Space.............................................................................................................................3

4 Policies.................................................................................................................................................4

4.1 Ramp up.......................................................................................................................................4

4.2 Replacement Policy......................................................................................................................4

4.3 Storage Overheads.......................................................................................................................4

4.4 Spares (hardware failures)...........................................................................................................4

4.5 Extra Capacity..............................................................................................................................5

4.6 Multiple Copies for Data Protection and Disaster Recovery........................................................5

5 Key Formulas.......................................................................................................................................5

5.1 Compute Nodes: Teraflops Required...........................................................................................5

5.2 Compute Nodes: Bandwidth to Memory.....................................................................................5

5.3 Database Nodes: Teraflops Required...........................................................................................5

5.4 Database Nodes: Bandwidth to Memory.....................................................................................5

5.5 Database Nodes: Disk Bandwidth Per Node (Local Drives)..........................................................6

5.6 Disk Drives: Capacity....................................................................................................................6

5.7 Disk Drives and Controllers (Image Storage): Bandwidth to Disk.................................................6

5.8 GPFS NSDs....................................................................................................................................6

5.9 Disk Drives (Database Nodes): Aggregate Number of Local Drives..............................................6

5.10 Disk Drives (Database Nodes): Minimum 2 Local Drives..............................................................6

5.11 Tape Media: Capacity...................................................................................................................6

5.12 Tape Drives..................................................................................................................................7

5.13 HPSS Movers................................................................................................................................7

5.14 HPSS Core Servers........................................................................................................................7


ii


5.15 10GigE Switches...........................................................................................................................7

5.16 Power Cost...................................................................................................................................7

5.17 Cooling Cost.................................................................................................................................7

5.18 Cooling Connection Fee...............................................................................................................7

6 Selection of Disk Drive Types...............................................................................................................9

6.1 Image Storage..............................................................................................................................9

6.2 Database Storage.........................................................................................................................9

7 Rates and Discounts...........................................................................................................................10

7.1 Power and Cooling Rates...........................................................................................................10

7.2 Floorspace Leasing Rates...........................................................................................................12

7.3 Shipping Rates............................................................................................................................12

7.4 Academic and Non-Profit Discounts..........................................................................................12

8 DM Control System (DMCS) Servers..................................................................................................13

9 Additional Descriptions......................................................................................................................13

9.1 Description of Barebones Nodes................................................................................................13

10 Computing.....................................................................................................................................13

10.1 Gigaflops per Core (Peak)...........................................................................................................13

10.2 Cores per CPU Chip....................................................................................................................14

10.3 Bandwidth to Memory per Node...............................................................................................14

10.4 System Bus Bandwidth per Node...............................................................................................14

10.5 Disk Bandwidth per Node..........................................................................................................15

10.6 Cost per CPU..............................................................................................................................15

10.7 Power per CPU...........................................................................................................................16

10.8 Compute Nodes per Rack...........................................................................................................16

10.9 Database Nodes per Rack..........................................................................................................16

10.10 Power per Barebones Node...................................................................................................17

10.11 Cost per Barebones Node......................................................................................................17

11 Memory.........................................................................................................................................17

11.1 DIMMs per Node.......................................................................................................................17


iii


11.2 Capacity per DIMM....................................................................................................................18

11.3 Bandwidth per DIMM................................................................................................................19

11.4 Cost per DIMM...........................................................................................................................19

11.5 Power per DIMM.......................................................................................................................19

12 Disk Storage...................................................................................................................................20

12.1 Capacity per Drive (Consumer SATA).........................................................................................20

12.2 Sequential Bandwidth Per Drive (Consumer SATA)....................................................................20

12.3 IOPS Per Drive (Consumer SATA)...............................................................................................21

12.4 Cost Per Drive (Consumer SATA)................................................................................................21

12.5 Power Per Drive (Consumer SATA)............................................................................................21

12.6 Capacity Per Drive (Enterprise SATA).........................................................................................22

12.7 Sequential Bandwidth Per Drive (Enterprise SATA)...................................................................22

12.8 IOPS Per Drive (Enterprise SATA)...............................................................................................22

12.9 Cost Per Drive (Enterprise SATA)................................................................................................23

12.10 Power Per Drive (Enterprise SATA)........................................................................................23

12.11 Disk Drive per Rack................................................................................................................23

13 Disk Controllers..............................................................................................................................24

13.1 Bandwidth per Controller..........................................................................................................24

13.2 Drives Required per Controller..................................................................................................24

13.3 Cost per Controller.....................................................................................................................24

14 GPFS...............................................................................................................................................25

14.1 Capacity Supported per NSD......................................................................................................25

14.2 Hardware Cost per NSD.............................................................................................................25

14.3 Software Cost per NSD...............................................................................................................25

14.4 Software Cost per GPFS Client...................................................................................................26

15 Tape Storage..................................................................................................................................26

15.1 Capacity Per Tape......................................................................................................................26

15.2 Cost per Tape.............................................................................................................................26

15.3 Cost of Tape Library and HPSS...................................................................................................27


iv


15.4 Bandwidth Per Tape Drive.........................................................................................................27

15.5 Cost Per Tape Drive....................................................................................................................27

15.6 Tape Drives per HPSS Mover......................................................................................................28

15.7 Hardware Cost per HPSS Mover.................................................................................................28

15.8 Hardware Cost per HPSS Core Server.........................................................................................28

16 Networking....................................................................................................................................29

16.1 Bandwidth per Infiniband Port...................................................................................................29

16.2 Ports per Infiniband Edge Switch...............................................................................................30

16.3 Cost per Infiniband Edge Switch................................................................................................30

16.4 Cost per Infiniband Core Switch.................................................................................................30

16.5 Bandwidth per 10GigE Switch....................................................................................................31

16.6 Cost per 10GigE Switch..............................................................................................................31

16.7 Cost per UPS..............................................................................................................................31

17 Long Haul Network........................................................................................................................32

18 Development and Integration Clusters..........................................................................................32

19 Commissioning Cluster..................................................................................................................32

20 DM System Operations Center (DMSOC).......................................................................................32

21 Camera / DAQ Testbed..................................................................................................................32

22 Science User Interface (SUI) Servers / Visualization.......................................................................33

Appendix A: Treatment of Cost Deflation due to Technological Advancements in the DM Sizing Model and Computing Equipment Cost Estimate................................................................................................34

Purpose.................................................................................................................................................34

Task #1: Current Unit Prices..................................................................................................................34

Documents with Vendor Pricing...............................................................................................................34

http://www.intel.com/products/server/processor/xeon5000/index.htm................................................34

Document-11534 Switch Infiniband..........................................................................................................34

Document-11546 Intel Processor Pricing Oct 2013...................................................................................34

Document-11545 Sandy Bridge.................................................................................................................34

Task #2: Requirements and Design.......................................................................................................35


v


Task #3: Computing equipment power and capacity predictions base on technological advancement...............................................................................................................................................................35


vi


The LSST Site Specific Infrastructure Estimation Explanation

This document provides explanations and the basis for estimates for the technology predictions used in LDM-144 “Site Specific Infrastructure Estimation Model.”

The supporting materials referenced in this document are stored in Collection-974.

1 Overview of Sizing Model and Inputs Into LDM-144

Figure 1. The structure and relationships among the components of the DM Sizing Model


1


2 Data Flow Among the Sheets Within LDM-144


2


3 DM-BaseSite ICD (LSE-77)

LSE-77 defines and quantifies the DM infrastructure requirements for the BaseSite Facility in La Serena, Chile. This section provides additional details and justification for those requirements.

3.1 DM Power Capacity

The ICD specifies 440 kW.

Net Base CTR+DAC equipment power = 204 kWNet Base AP (or commissioning cluster) equipment power reservation = 60 kWNet replacement hardware power (10%) = 27 kWTotal net power for computing equipment = 291 kW

Adjustment for power utilization efficiency (1.5X) gives a total gross power including power for cooling of 437 kW.

3.2 DM Rack Space

The ICD specifies 64 racks.


3


Storage racks are 1.5 compute rack equivalents, and tape racks are 1.6 compute rack equivalents.

Base CTR = 2 compute racks + 4.5 compute rack equivalents for storage + 12.8 compute rack equivalents for tapeBase DAC = 13 compute racks + 1.5 compute rack equivalents for storage Base AP = 6 compute rack equivalentsReplacement hardware = 4 compute racks + 3 compute rack equivalents for storage + 12.8 rack equivalents for tape

Total compute rack equivalents = 60

4 Policies

4.1 Ramp upThe ramp up policy during the Commissioning phase of Construction is described in LDM-129. Briefly, in 2018, we acquire and install the computing infrastructure needed to support Commissioning, for which we use the same sizing as that for the first year of Operations.

4.2 Replacement Policy

Compute Nodes 5 YearsGPFS NSD Nodes 5 YearsDisk Drives 3 YearsTape Media 5 YearsTape Drives 3 YearsTape Library System Once at Year 5

4.3 Storage Overheads

RAID6 8+2 20%Filesystem 10%

4.4 Spares (hardware failures)

This is margin for hardware failures. This is what takes into account that at any given point in time, there will be some number of nodes and drives out of service due to hardware failures.

Compute Nodes 3% of nodes


4


Disk Drives 3% of drivesTape Media 3% of tapes

4.5 Extra Capacity

Disk 10% of TBTape 10% of TB

4.6 Multiple Copies for Data Protection and Disaster Recovery

Single tape copy at BaseSiteDual tape copies at ArchSite (one goes offsite for disaster recovery)

See LDM-129 for further details.

5 Key Formulas

This section describes the key formulas used in LDM-144.

Some of these formulas are interrelated. For example, the formulas used to establish minimum required nodes or drives will typically use multiple formulas based upon different potential constraining resources, and then take the maximum of the set in order to establish the minimum needed.

5.1 Compute Nodes: Teraflops Required(number of compute nodes) >= (sustained TF required) / (sustain TF per node)

5.2 Compute Nodes: Bandwidth to Memory(number of compute nodes) >=

(total memory bandwidth required) / (memory bandwidth per node)

5.3 Database Nodes: Teraflops Required(number of database nodes) >= (sustained TF required) / (sustain TF per node)

5.4 Database Nodes: Bandwidth to Memory(number of database nodes) >=

(total memory bandwidth required) / (memory bandwidth per node)


5


5.5 Database Nodes: Disk Bandwidth Per Node (Local Drives)(number of database nodes) >=

(total disk bandwidth required) / (disk bandwidth per node)

where the disk bandwidth per node is a scaled function of PCIe bandwidth

5.6 Disk Drives: Capacity(number of disk drives) >= (total capacity required) / (capacity per disk drive)

5.7 Disk Drives and Controllers (Image Storage): Bandwidth to Disk(number of disk controllers) = (total aggregate bandwidth required) /

(bandwidth per controller)

(number of disks) = MAX of A and Bwhere

A = (total aggregate bandwidth required) / (sequential bandwidth per drive)B = (number of controllers) * (drives required per controller)

5.8 GPFS NSDs(number of NSDs) = MAX of A and Bwhere

A = (total storage capacity required) / (capacity supported per NSD)B = (total bandwidth) / (bandwidth per NSD)

5.9 Disk Drives (Database Nodes): Aggregate Number of Local Drives(number of disk drives) >= A + Bwhere

A = (total disk bandwidth required) / (sequential disk bandwidth per drive)B = (total IOPS required) / (IOPS per drive)

5.10 Disk Drives (Database Nodes): Minimum 2 Local DrivesThere will be a minimum of at least two local drives per database node

5.11 Tape Media: Capacity(number of tapes) >= (total capacity required) / (capacity per tape)


6


5.12 Tape Drives(number of tape drives) = (total tape bandwidth required) /

(bandwidth per tape drive)

5.13 HPSS Movers(number of movers) = MAX of A and Bwhere

A = (number of tape drives) / (tape drives per mover)B = (total bandwidth required) / (bandwidth per mover)

5.14 HPSS Core Servers(number of core server) = 2

This is flat over time.

5.15 10GigE Switches(number of switches) = MAX of A and Bwhere

A = (total number of ports required) / (ports per switch)B = (total bandwidth required) / (bandwidth per switch)

5.16 Power Cost(cost for the year) = (kW on-the-floor) * (rate per kWh) * 24 * 365

5.17 Cooling Cost(cost for the year) = (mmbtu) * (rate per mmbtu) * 24 * 365where

mmbtu = btu / 1000000btu = watts * 3.412

5.18 Cooling Connection Fee(one-time cost) = ((high water MW) * 0.3412 / 12) * (rate per ton)where

high water MW = (high water watts) / 1000000high water watts = high water mark for watts over all the years of Operations

This is a one-time fee paid during Commissioning, and only applies at the Archive Site.


7



8


6 Selection of Disk Drive Types

At any particular point in time, disk drives are available in a range of capacities and prices. Optimizing for cost per TB requires selecting a different price point than optimizing for cost per drive. In LDM-144, the “InputTechPredictionsDiskDrives” sheet implements that logic using the technology prediction for disk drives based upon when leading edge drives become available. We assume a 15% drop in price each year for a particular type of drive at a particular capacity, and that drives at a particular capacity are only available for 5 years. The appropriate results are then used for the drives described in this section.

6.1 Image Storage

Disk drives for image storage are sitting behind disk controllers in a RAID configuration. Manufacturers warn against using commodity SATA drives in such environments, based on considerations such as failure rates caused by heavy duty cycles and time-limited error recovery (TLER) settings. Experience using such devices in RAID configurations support those warnings. Therefore, we select Enterprise SATA drives for image storage, and optimize for cheapest cost per unit of capacity.

SAS drives are not used as sequential bandwidth is the primary motivation for the drive selection, and SATA provides a more economical solution.

6.2 Database Storage

The disk drives for the database nodes are local, i.e. they are physically contained inside the database worker node and are directly attached. Unlike most database servers, where IOPS is the primary consideration, sequential bandwidth is the driving constraint in our qserv-based databases servers. Since these are local drives, and since they are running in a shared-nothing environment where the normal operating procedure is to take a failing node out of service without end-user impact, we do not require RAID or other fault-tolerant solutions at the physical infrastructure layer. Therefore, we strive to optimize for the cheapest cost per drive, and so select consumer SATA drives for the database nodes.

SAS drives are not used as sequential bandwidth is the primary motivation for the drive selection, and SATA provides a more economical solution.


9


7 Rates and Discounts

7.1 Power and Cooling Rates

7.1.1 Archive Site

The power rate for the University of Illinois for 2013 is $0.0746 per kWh.

The cooling rate for the University of Illinois for 2013 is $16.71 per mmbtu.

See Document-15107:

https://docushare.lsstcorp.org/docushare/dsweb/Get/Document-15107/FY13UtilityRates.pdf

which is also available at:

http://www.energymanagement.illinois.edu/pdfs/FY13UtilityRates.pdf

7.1.2 Base Site

The 2013 power rate for La Serena is $0.154 per kWh (USD).

The 2013 cooling rate for La Serena is $34.42 per mmbtu (USD).

Power Rate

See Document-14992.

Additional description:

On 10/2/2013 7:22 AM, Jeff Barr wrote:> ... *right now *the current electric rate at the > current exchange rate (October 2, 2013) is:> 71.79 CLP/kWh / 503.09 CLP/USD = 0.143 USD/kWH> > As previously noted there are transmission losses that are distributed > to all the users, both on the La Serena Recinto and on Cerro Pachón, so > for the final cost of effective kWH metered at the facility ~8% should > be added to that rate:> 0.143 x 1.08 = 0.154 USD/kWH


10


Cooling Rate

The cooling technology and power utilization efficiency (PUE) is not yet known for the La Serena facility. As an approximation, the cooling rates are assumed to be proportional to the power rates. In particular, the power rates at La Serena are 2.06 times the power rates at Champaign. Until the specific attributes of the La Serena facility are known, we assume the cooling rates at La Serena follow the same ratio, i.e. that the cooling rates at La Serena are 2.06 times the cooling rates in Champaign, IL. That represents a PUE of ~1.7.

Historical Trend for Power Rates

The DM Sizing Model is in Base Year 2013 dollars, so the following is historical trend is not used. It is provided only as a reference.

See Document-11758.

Figure 2. Historical cost for power in La Serena, Chile.


11


7.2 Floorspace Leasing Rates

7.2.1 Archive Site

Floorspace costs are imbedded within the University of Illinois’ campus F&A rate. There is no separate charge for this.

7.2.2 Base Site

Per Jeff Kantor and Ron Lambert, the lease costs for the Base Site are not part of Data Management.

7.3 Shipping Rates

The shipping rate for 2013 is $11.74 USD per pound.

See Document-15108 for a 2013 FedEx quote.

When comparing rate information from previous years, we find that rates have risen by 13-14% per year between 2007 and 2011. However, this model contains base year costs only -- all escalation is done in PMCS.

7.4 Academic and Non-Profit Discounts

For all monetary inputs to the DM Sizing Model, we use prices in BY2013 U.S. dollars that reflect what it will actually cost NCSA/University of Illinois. This includes any applicable academic and/or non-profit discounts.

The rule-of-thumb for discounts that NCSA can realize is 35% from list prices. This can vary widely for any particular acquisition. In most cases in the Sizing Model, we use discounted prices as they apply for the particular item in question.


12


8 DM Control System (DMCS) Servers

For fault tolerance, we plan to have one replicator per raft, including a separate one for the four wavefront sensors. This means that if one fails, we only need to retransmit (and potentially be late on) one raft's worth of data instead of (potentially) the entire focal plane. This means that there are 22 replicators. We add three spares, two for failures and one in case of performance issues for a total of 25. Each replicator at the Base is paired with a distributor at the Archive.

The catch-up replicator is sized to fill the data transfer pipe (in the time that the main replicators are not using it). Two nodes should be sufficient, with one extra spare for a total of three. Here again, each replicator at the Base is paired with a distributor at the Archive.

The Base requires a (separate) DMCS machine and a warm spare. The Archive is the same.

This adds up to a total of (22+3)+(2+1)+(1+1) = 30 machines at both the Base and the Archive.

9 Additional Descriptions

9.1 Description of Barebones Nodes

Includes smallish single local drive inside each node for O/S and swap.

Traditionally these are dual small 10K SAS drives in a RAID1 (mirror) plus another pair of local drives for swap on a separate disk controller, but we are studying whether that is necessary given our approach to failure scenarios. Our standard method of operation for these kinds of failures will be to simply take the node out of service, repair offline, and then place the node back into service. This will be a routine occurrence, and so we may be not need or want multiple drives with RAID1 within each node.

10 Computing

10.1 Gigaflops per Core (Peak)

10.1.1 TrendInitial (2013) value is 13 GF per core, and rises at 4% per year.

10.1.2 DescriptionCore speed is expected to remain constant at around 3 GHz, where it has remained since 2005. Vendors can be expected to make additional instructions and capabilities available, but we expect those to be incremental, and can’t automatically assume LSST codes will be able to leverage those new capabilities.


13


Taking all of these things into consideration, we model a 4% per year increase in per core performance.

10.1.3 References[1] Document-11526 An Overview of Exascale Architecture Challenges[2] Document-11570 Intel Pins Exascale Dreams To Knight Ferry

10.2 Cores per CPU Chip

10.2.1 TrendInitial (2013) value is 8 cores per chip, and double every 4 years.

10.2.2 DescriptionNumber of cores doubling of every four years. While extremely large numbers of cores in processors are expected in Knights Ferry from Intel, mainstream adoption isn't expected until 2018 [3].

10.2.3 References[1] Document-11510 International Solid-State Circuits Conference 2011 Trends Report.[2] Document-11511 Assessing Trends Over Time in Performance, Costs, and Energy Use For Servers.[3] Document-11570 Intel Pins Exascale Dreams to Knights Ferry

10.3 Bandwidth to Memory per Node

10.3.1 TrendInitial (2013) value is 32 GB/s, and doubles every 5 years.

10.3.2 DescriptionThis is Front-Side Bus (FSB) / QPI bandwidth.

10.4 System Bus Bandwidth per Node

10.4.1 TrendThe initial (2013) value is 8 GB/s, and doubles every 5 years.

10.4.2 DescriptionThis represent theoretical peak bandwidth for the system bus, and is used to scale the actual expected maximum bandwidth (see the next section).

10.4.3 References[1] Document-11542 PCI Express


14


10.5 Disk Bandwidth per Node

10.5.1 TrendInitial (2013) value is 4 GB/s, and doubles every 5 years.

10.5.2 DescriptionThe attached study [1] found that the system bus (PCIe) did not impose any bottleneck on I/O traffic. They achieved 3.2 GB/s disk bandwidth and speculate they should have gone higher if the RAID cards were better. Their bottleneck was the RAID cards they were using. However, it is unrealistic to believe that actual disk I/O could reach the theoretical peak in practice. Therefore, we adopt a model of using 1/2 of PCIe v2 theoretical peak for this system attribute.

10.5.3 References[1] Document-11675 Tom’s Hardware The 3GB Project Revisited

10.6 Cost per CPU

10.6.1 TrendThe cost per CPU chip is $996 and is invariant over time.

10.6.2 DescriptionUsing Xeon X5650 as an applicable reference model for the type of processor that would be most applicable to our systems. It is a 6 core, 3 GHz processor. Additional specifications, and comparison with other models, here:

Intel X5650 Intel L5640 Intel E5649 AMD 2439SE AMD 61406 cores 6 cores 6 cores 6 cores 8 cores3.06 GHz (turbo) 2.8 GHz (turbo) 2.93 GHz (turbo) 2.8 Ghz 2.6 GHz12M cache 12M cache 12M cache 3M cache 4M cache95W 60W 80W 105W 80W$996 $996 $774 $1229 $2300

10.6.3 References[1] http://www.intel.com/products/server/processor/xeon5000/index.htm[2] Document-11546 Intel Processor Pricing Oct 2013[3] Document-11545 Sandy Bridge


15


10.7 Power per CPU

10.7.1 TrendThe power per cpu chip is 95 watts, and is invariant over time.

10.7.2 DescriptionThe power per chip has been steady at 95W TDP. This is closely related to clockspeeds. Neither are expected to increase significantly given the physical properties of the materials and technology currently in use.

10.7.3 References[1] Document-11511 Assessing Trends Over Time in Performance, Costs, and Energy Use For Servers[2] Document-11545 Sandy Bridge

10.8 Compute Nodes per Rack

10.8.1 TrendThe number of compute nodes per rack is 48, and is invariant over time.

10.8.2 DescriptionA typical full size rack is 42U.

For compute nodes, we go with blade systems. Assuming we have a few U of power distribution unit, UPS, networking, etc., and further assuming we can install 3 blade chassis of 16 nodes each per 10U, with 30U available in each cabinet, we estimate 48 nodes per rack.

10.8.3 References[1] Document-11685 server-poweredge-m1000e-tech-guidebook.pdf

10.9 Database Nodes per Rack

10.9.1 TrendThe number of database nodes per rack is 34, and is invariant over time.

10.9.2 DescriptionA typical full size rack is 42U. Assuming we have a few U of power distribution unit, UPS, networking, etc., and further assuming we select 1U nodes, we estimate 34 nodes per rack. We cannot use a blade chassis due to the number of local disk drives in each database node.


16


10.10 Power per Barebones Node

10.10.1 TrendThe power per barebone node is 100 watts, and is invariant over time.

10.10.2 DescriptionPower for everything in a node except CPU chips, disk drives, and memory. This does include a single small local drive for O/S and swap space.

10.11 Cost per Barebones Node

10.11.1 TrendThe cost for a barebones node is $1500, and is invariant over time.

10.11.2 DescriptionCost for everything in a node except CPU chips, disk drives, and memory. This does include a single small local drive(s) for O/S and swap space. It also includes PCIe cards, such as for 10GigE or IB. A reference barebones system is shown in [1].

10.11.3 References[1] Document-11674 Intel R1304BTL Barebones System

11 Memory

11.1 DIMMs per Node

11.1.1 TrendThe number of DIMMs per node is 16, and is invariant over time.

11.1.2 DescriptionThe growth in memory per node comes from the growth in DIMM capacity, not the number of DIMMs. See the next section for Capacity per DIMM estimates.

11.1.3 References[1] Document-11547 You Probably Don't Need More DIMMs


17


11.2 Capacity per DIMM

11.2.1 TrendThe 2013 capacity per DIMM is 8 GB, and doubles every 3 years.

11.2.2 Description

From [1], the “# Core” line shows the projected trend of cores per socket, while the dynamic random access memory (DRAM) line shows the projected trend of capacity per socket.

The initial value chosen is 8GB for 2013. Although larger capacity DIMMs are available, the 8GB modules are the most cost effective solution that provides the required memory per core. The prices in section 2.4 match this capacity point.

11.2.3 References[1] Document-11670 Disaggregated Memory Architectures for Blade Servers[2] Document-11549 Trends in Memory Systems


18


11.3 Bandwidth per DIMM

11.3.1 TrendThe initial (2013) value for bandwidth per DIMM is 20 GB/s, and doubles every 4 years.

11.3.2 DescriptionThe initial value is based upon DDR3 dual-channel 128-bit memory. Although it has a higher theoretical bandwidth, real world performance is closer to that of single channel operation, which is reflected in our estimate. This doubles every 4 years [1].

11.3.3 References[1] Document-11571 List of Device Bit Rates

11.4 Cost per DIMM

11.4.1 TrendThe cost per DIMM is $120, and is invariant over time.

11.4.2 DescriptionMemory tends to start high, goes to a trough, and then rises again because of reduced production. Memory prices in the trough remain relatively constant, regardless of the memory size. Initial cost is for a representative server for a single DDR3-133 8GB DIMM in 2013[1].

11.4.3 References [1] Document-11671 DDR Memory Prices from Crucial.com Oct 2013.

11.5 Power per DIMM

11.5.1 TrendThe power per DIMM is 5 watts, and is invariant over time.

11.5.2 DescriptionThis is difficult to trend out for the reasons indicated in the references. We select a reasonable estimate of 5 watts and will continue to monitor trends.

11.5.3 References[1] Document-11590 DDR3 DIMM Memory Module.pdf[2] Document-11591 How many watts DDR2 memory module.pdf


19


12 Disk Storage

12.1 Capacity per Drive (Consumer SATA)

12.1.1 TrendThe capacity per drive is 4 TB, and doubles every 3.5 years.

12.1.2 DescriptionDisk space per drive doubles every 3 years for consumer SATA drives [1].

Although Kogge[2] in section 6.4.1.1 states that “10X growth over about 6 year periods seems to have been the standard for decades.” However, that report was published in 2008. Recent trends show a slowdown in the rate of growth. Alex Szalay (in 2011) and others , including the NCSA storage team running the Blue Waters system, now see a doubling every three years. We adopt a doubling every 3.5 years for our baseline, and will continue to monitor the trends.

12.1.3 References[1] Document-11568 History of Hard Disk Drives – Wikipedia[2] Document-11672 ExaScale Computing Study: Technology Challenges in Achieving Exascale Systems,

Peter Kogge, Editor & Study Lead

12.2 Sequential Bandwidth Per Drive (Consumer SATA)

12.2.1 TrendThe sequential bandwidth per drive is 80 MB/s, and increases at 40% every 3 years.

12.2.2 Description[1] gives a good description of the relationship between capacity and sequential bandwidth estimates. Basically, since bandwidth is proportional to linear density times rotation speed, assuming rotation speeds will stay constant over time, and that linear density is proportional to the square root of the areal density, we get:

2TB drives: 50 MB/s4TB drives: 71 MB/s

8TB drives: 100 MB/s16TB drives: 141 MB/s32TB drives: 200 MB/s64TB drives: 283 MB/s

128TB drives: 400 MB/s

These number are equivalent to a 40% increase every 3 years.


20


12.2.3 References[1] Document-11673 Sequential Transfer Rates: An examination of its effects on performance

http://www.storagereview.com/articles/9910/991014str.html[2] Document-11580 Seagate Barracuda Green 2TB Review

12.3 IOPS Per Drive (Consumer SATA)

12.3.1 TrendThe I/O per second per drive is 90, and is invariant over time.

12.3.2 DescriptionNeither rotational latency nor seek time can be significantly improved, so we expect IOPS to remain essentially flat over time.

12.3.3 References[1] Document-11569 Calculate IOPS in a Storage Array

12.4 Cost Per Drive (Consumer SATA)

12.4.1 TrendThe cost for a mid-range consumer SATA drive is $80, and is invariant over time.

12.4.2 DescriptionDrive costs remain relatively constant for drives as capacity increases.

12.4.3 References[1] Document-11581 Seagate Barracuda Green ST2000DL003 Internal Hard Drive

12.5 Power Per Drive (Consumer SATA)

12.5.1 TrendPower Per Drive (watts) for representative consumer SATA drive is 5.8 Watts.

12.5.2 DescriptionPower consumption per drive constant, and is expected to continue this trend into the future. Reference drive is ST2000DL003, which consumes about 5.8 watts, which is due to power management for this "green" drive; otherwise it would be higher.

12.5.3 References[1] Document-11579 Seagate Barracuda Green Consumer SATA Specification


21


12.6 Capacity Per Drive (Enterprise SATA)

12.6.1 TrendSame as Consumer SATA. See comments above.

12.6.2 DescriptionSame as Consumer SATA. See comments above.

12.6.3 ReferencesSame as Consumer SATA. See comments above.

12.7 Sequential Bandwidth Per Drive (Enterprise SATA)




12.8 IOPS Per Drive (Enterprise SATA)





22


12.9 Cost Per Drive (Enterprise SATA)

12.9.1 TrendThe cost for a mid-range enterprise SATA drive is $220, and is invariant over time.

12.9.2 DescriptionDrive costs remain relatively constant drives as capacity increases. Representative drive is the Hitachi Ultrastar 7K4000, with a 24x7 duty cycle [1]. Note that this price includes an expected academic discount not listed in the reference document.

12.9.3 References[1] Document-11560 HITACHI Ultrastar 7K4000 Hard Drive

12.10 Power Per Drive (Enterprise SATA)

12.10.1 TrendThe power per drive is 11 watts, and is invariant over time.

12.10.2 DescriptionReference is Hitachi Ultrastar A7K2000[1].

12.10.3 References[1] Document-11561 Ultrastar A7K2000 Specification

12.11 Disk Drive per Rack

12.11.1 TrendThe number of disk drives per rack is 360, and is invariant over time.

12.11.2 Description36 SATA drives per 4U space.

12.11.3 References[1] Document-11839 LSI NetApp IBM DS3500[2] Sun X4540


23


13 Disk Controllers

13.1 Bandwidth per Controller

13.1.1 TrendThe bandwidth per disk controller is 1.3 GB/s, and doubles every 5 years.

13.1.2 DescriptionThe DS3500 is a reasonable balance between low-end controllers with unproven reliability and high-end, very expensive controllers.

13.1.3 References[1] Document-11839 LSI NetApp IBM DS3500[2] DDN SFA10K Inifiniband

13.2 Drives Required per Controller

13.2.1 TrendThe number of drives required per controller to achieve the rated bandwidth is 24, and is invariant over time.


13.2.3 References[1] Document-11839 LSI NetApp IBM DS3500

13.3 Cost per Controller

13.3.1 TrendThe cost per disk controller is $4K, and is invariant over time.


13.3.3 References[1] Document-11839 LSI NetApp IBM DS3500


24


14 GPFS

14.1 Capacity Supported per NSD

14.1.1 TrendThe capacity supported per GPFS NSD server is 500 TB, and doubles every 3.5 years.

14.1.2 DescriptionThis is useable capacity, not raw capacity. The growth curve follows the same rate as disk drives.

14.1.3 ReferencesSee section 5.6 in the following document:http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs_faqs%2Fgpfsclustersfaq.html

14.2 Hardware Cost per NSD

14.2.1 TrendThe hardware costs per NSD server is $16K, and is invariant over time.

14.2.2 DescriptionThis is the estimated price for the hardware needed to serve as a GPFS NSD based upon actual purchases recently at NCSA.

14.3 Software Cost per NSD

14.3.1 TrendThe software costs per NSD server is $4K, and is invariant over time.

14.3.2 DescriptionThis is special pricing from IBM due to the University of Illinois’ campus licensing agreement. This pricing is nearly the same as what we would get support of a Lustre installation. Licenses are priced by IBM on per processor core. A dual CPU system with 8 cores in each CPU, but with only 4 cores in each CPU dedicated to GPFS would require 8 client licenses.

14.3.3 References[1] See section 1.7 of:http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs_faqs%2Fgpfsclustersfaq.html


25


14.4 Software Cost per GPFS Client

14.4.1 TrendGPFS clients are free.

14.4.2 DescriptionThere are no software licensing costs associated with GPFS clients. This is special pricing from IBM due to the University of Illinois’ campus licensing agreement.

15 Tape Storage

15.1 Capacity Per Tape

15.1.1 TrendThe capacity per tape is 2.5 TB, and doubles every 5 years.

15.1.2 DescriptionThe base year technology is LTO-6, which hold 2.5 TB per tape media [1]. In the past, tape capacity has doubled every 2 years[2][3]. Since we have no reliable information for dates beyond 2015, we adopted the less aggressive performance curve of tape capacity doubling every 5 years. These are uncompressed capacities.

15.1.3 References[1] Document-15115 Linear Tape-Open[2] Document-11533 Two new LTO tape gens announced 2010[3] Sun roadmap, Oct 2005[4] NCSA’s T2 proposal

15.2 Cost per Tape

15.2.1 TrendThe cost per tape is $70, in is invariant over time.

15.2.2 DescriptionThis is based on in October 2011 pricing from a discount retailer[1], and it also matches our academic discounted price from existing vendors at NCSA.

15.2.3 References[1] Document-11564 LTO-6 Data Cartridge


26


15.3 Cost of Tape Library and HPSS

15.3.1 TrendThe one-time acquisition cost for the tape library system is $375K, with annual maintenance at 15% of the purchase price.

The one-time software licensing for HPSS is $500K, which includes the software for the movers, core servers, and all clients. The annual software licensing for HPSS is $150K per year.

15.3.2 DescriptionThis library includes 8000 slots, no media, and no drives. Media and drives are purchased separately. The basis of estimate for these items are actual vendor quotes from a different project at NCSA. Due to confidentiality concerns, those are not copied into the LSST document repository.

15.4 Bandwidth Per Tape Drive

15.4.1 TrendThe bandwidth per tape drive is 160 MB/s, and increase 50% every 5 years.

15.4.2 DescriptionSee [1]. Note for those who may read vendor specifications: Vendors typically assume a 50% compression ratio. We assume no compression, as the LSST data will already be compressed. The vendor estimates should be adjusted accordingly.

15.4.3 References[1] Document-15115 Linear Tape-Open

15.5 Cost Per Tape Drive

15.5.1 TrendThe cost per tape drive is $8K, and is invariant over time.

15.5.2 DescriptionSee [1] for a reference drive compatible with requirements.

15.5.3 References[1] Document-15116 Quantum Scalar i40/i80 Tape Drive Module, LTO-6


27


15.6 Tape Drives per HPSS Mover

15.6.1 TrendThe number of tape drives per HPSS mover is 8, and is invariant over time.

15.6.2 DescriptionBased upon our current design, we estimate that we’ll need 7-8 drives per mover. The outside range is around 5-10.

15.7 Hardware Cost per HPSS Mover

15.7.1 TrendThe hardware cost per HPSS mover is $15K, and is invariant over time.

15.7.2 DescriptionThis is an estimate of the hardware needed to serve as an HPSS mover, and includes 10Gbps network interface cards.

15.8 Hardware Cost per HPSS Core Server

15.8.1 TrendThe hardware cost per HPSS core server is $80K, and is invariant over time.

15.8.2 DescriptionThese are high-end machines that manage the HPSS metadata and control the robots. Our computing environment will need 2 of these (which is accounted for elsewhere in the sizing model). The software license costs are bundled with the HPSS software licensing above.


28


16 Networking

16.1 Bandwidth per Infiniband Port

16.1.1 TrendThe bandwidth per IB port is 10 Gbps, or 1 GB/s (QDR), and increases at the rate of 6x every 8 years.

16.1.2 Description

Note: Although the DM sizing model used “smoothed” values for the out years, the actual equipment with be rated at a specific step level, e.g. 10Gbps, 40Gbps, etc.

16.1.3 References[1] Document-15536 IBTA – Infiniband Trade Association


29


16.2 Ports per Infiniband Edge Switch

16.2.1 TrendThe number of ports per IB edge switch is 18, and is invariant over time.

16.2.2 DescriptionThis is the estimate for the number of ports per switch.

16.2.3 References[1] Document-11534 Infiniband Switch Price – The Technology and its Cost of Ownership

16.3 Cost per Infiniband Edge Switch

16.3.1 TrendThe cost per IB edge switch is $12K, and is invariant over time.

16.3.2 DescriptionAssumes 36 unoptimized port switches, $340 per IB Edge Port, $1000 per Core Switch.


16.4 Cost per Infiniband Core Switch

16.4.1 TrendThe cost per IB core switch is $36K, and is invariant over time.

16.4.2 DescriptionAssumes 36 unoptimized port switches, $340 per IB Edge Port, $1000 per Core Switch



30


16.5 Bandwidth per 10GigE Switch

16.5.1 TrendThe bandwidth per 10Gbps ethernet switch is 80 GB/s, and doubles every 5 years.

16.5.2 DescriptionBased on Juniper Ex4500 40-port switch, which does 900 Mpps, or, at 90 bytes per frame, about 80 GB/s.

16.5.3 References[1] Document-11575 10gb-switch-compare-6-2011.xlsx

16.6 Cost per 10GigE Switch

16.6.1 TrendThe cost per 10Gbps ethernet switch is $30K, and is invariant over time.

16.6.2 DescriptionJuniper Ex4500 40-port switch: $29,000 list. As reference, a mid-range switch is Juniper 8200 64-port switch is $380,000, and a high-end Juniper Ex8216 128-port switch is $730,000. Our estimate is based on the Juniper Ex4500 40-port switch. This is lower throughput than the next class higher, which is the Juniper 8200, but the cost is over 10 times as much. We expect the throughput rates to go higher over time, but this price point should remain relatively steady.

16.6.3 References[1] Document-11575 10gb-switch-compare-6-2011.xlsx

16.7 Cost per UPS

16.7.1 TrendThe cost per rack-based UPS unit is $3K, and is invariant over time.

16.7.2 DescriptionThis is the estimated cost for each rack-based UPS unit, to ensure a controlled shutdown (and flush of data buffers to disk) in the event of a facility power outage.


31


17 Long Haul NetworkMany of the documents in Collection-648 provide the Basis of Estimate for the Long Haul Network, including:

LSE-78 LSST Observatory Network DesignDocument-14938 Axys Estimate AURA 2013doc Document-14939 Axys Fibre installation updated BOE Document-11811 Feasibility of Redundant Fiber from Mountain to BaseDocument-14313 Shipping Media analysisDocument-14575 Mountain Base Fiber Cost Evaluation 20130605

18 Development and Integration ClustersThe Development and Integrations Clusters are scaled at 20% of the size of the Year 1 Operational System at the Archive Site. This includes both the compute clusters and the data access facilities. This is sufficient to test at close to production scale (especially for the Integration Cluster) and to allow rapid turnaround of experiments at smaller scales (e.g. 20 times faster for 1% scale).

19 Commissioning ClusterThe Commissioning Cluster was originally sized at 10% of the Alert Production capacity in order to support the Commissioning Camera, which is 5% of the size of the full Camera, assuming that in Commissioning multiple reductions of data will be necessary. The hardware capacity has been turned into dollars to allow flexibility in the balance between compute and storage resources for Commissioning, which is likely to be different than that for Operations.

20 DM System Operations Center (DMSOC)The Basis of Estimate for the Data Management System Operations Center are these two documents:

Document-9265 DM System Operations Center ConceptDocument-14490 DMSOC Software and Installation Costs

21 Camera / DAQ TestbedThe Basis of Estimate for the Camera / DAQ Testbed is from the Camera Team, and captured as Document-15127, “DAQ testbed costs”.


32


22 Science User Interface (SUI) Servers / VisualizationThe estimate for the Science User Interface (SUI) server cluster is a framework of 20 web/application servers, at $20K per server for a total of $400K, 30 TB of user storage/query workspace, at $1K/TB for a total of $30K, file service infrastructure (servers, backups, etc.) at $100K, networking infrastructure (switches, etc.) at $70K, and computing support, i.e., 20 cluster nodes for download generation, at $10K/node for a total of $200K. Our estimates are partially based on current usage statistics of the NASA/Infrared Science Archive and partially based on the Science Project Requirements, assuming an average 1-hour SUI user session, with low- and high-volume simultaneous image and catalog queries, which result in a need for 29.8 TB storage to accommodate all concurrent sessions.


33

LSST Site Specific Infrastructure Estimation Explanation LDM-143

7/25/2014

Appendix A: Treatment of Cost Deflation due to Technological Advancements in the DM Sizing Model and Computing Equipment Cost Estimate

PurposeFor all DM computing equipment, the DM Sizing Model was used to develop the estimate. The Sizing Model exists to answer a fundamental question: How do we estimate cost of equipment that will be purchased several years in the future, for which no vendor will provide quotes, since those specific models do not yet exist?

We do this with the Sizing Model by performing 3 tasks:

1. Acquiring current unit costs for similar classes of equipment, as deployed in current data centers

2. Estimating what we will need in terms of computing equipment and when we will need it

3. Estimating how power and capacity per unit price will increase over time, i.e. cost deflation due to technological advances (e.g. Moore's Law)

All of the above elements of the sizing model have been reviewed multiple times during the LSST R&D phase, and a summary of review recommendations, findings, and comments is provided in Addendum 3 Appendix B.

Task #1: Current Unit PricesTask #1 above can be directly compared to quotes from vendors, and we have provided documents with comparative vendor pricing, which are referenced in LDM-143 the Sizing Model explanation document, and repeated below:

Documents with Vendor Pricing

http://www.intel.com/products/server/processor/xeon5000/index.htm

Document-11534 Switch Infiniband

Document-11546 Intel Processor Pricing Oct 2013

Document-11545 Sandy Bridge

Document-11560 Hard Drive Hitachi Ultrastar

Document-11564 Data Cartridge Fujifilm LTO6

Document-11575 Switch 10 gb

Document-11581 Internal Hard Drive Barracuda

Document-11671 DDR Memory Prices from Crucial.com Oct 2013

Document-11674 Barebones Intel System

Document-11839 System Storage


34


7/25/2014

Document-15116 Tape Drive Quantum LTO6

Note that the equipment pricing is not for consumer-grade desktop or laptop computing but rather for enterprise-grade equipment suitable for deployment in a data center.

Task #2: Requirements and DesignTask #2 requires detailed knowledge of LSST and DM System Requirements and Design. As can be seen in Addendum 3 Appendix B, the DM System Requirements, Design, and Sizing Model have been reviewed approximately 20 times by over 50 academic, government, and industry experts to provide a level of assurance that this has been done in an appropriate manner and level of detail and rigor.

Task #3: Computing equipment power and capacity predictions base on technological advancementTask #3 involves engineering judgment and historical input from industry experts regarding technology trends. In this document, we describe how these predictions are developed, captured in the DM Sizing Model, and applied to the DM computing equipment estimate. Additional details on this topic can be found in LDM-143.

Purchases are planned and estimated with granularity of one year, with just in time acquisition to take maximum advantage to cost deflation in computing infrastructure due to technological advancements. Each annual purchase is for the equipment needed in the next year of operations.

Each estimate was calculated from individual estimates of the computing, disk storage, tape storage, local area network, and workstations required at that time to meet the operational requirements of LSST. Operating costs for power, cooling, and maintenance are included to cover the MREFC period.

LDM-143 also documents assumptions built into the Sizing Model about sparing and replacement rates for failed equipment.

The sizing model and this document extrapolate from current prices to future prices using technology predictions for 7 computing technology areas, as summarized in Table 1 below, with a more detailed description for each area in the remaining sections. Each cost deflation percentage was developed by reviewing at least a decade of historical price/performance data in that equipment category, as well as supplier and industry and trade association predictions for future trends.

Computing Equipment Category Sizing Model Cost Deflation (annual)Computing (Central Processing Unit) -24%

Memory -21%

Hard Drive -18%

Tape -13%

Local Area Network (ethernet) -13%

Local Area Network (infiniband) -20%


35


7/25/2014

Workstations 0%Table 1: Summary of Cost Deflation Percentages in the DM Sizing Model

Computing

Computing was estimated individually for 3 types of server nodes: Pipeline Server nodes, Database Server nodes, and Support Server nodes, each with somewhat different requirements for CPU power per server node. Each server node is estimated as a combination of CPU cost and bare bones node cost (motherboard, power supply, local disk for operating system and software) as documented in LDM-143. The cost of a bare bones node was assumed a constant $1500/node.

While the unit cost for CPU in each type of server node is the same over time ($996), the computing power in trillions of floating-point operations per second (TFLOPS) per server node increases over time (modeled as an increase in the number of cores per CPU and GFLOPS/core), per the Input Technology Predictions in the Sizing Model, consistent with historical data and industry expectations. This increase is such that the TFLOPS/CPU is doubled approximately every 4 years. Thus, the cost/TFLOPS decreases over time, inversely.

This can be verified by examining the InputTechnologyPredictions sheet within LDM-144. Table 2 below is extracted from LDM-144, and documents the growth of cores/CPU chip (2x every 4 years, as well as a smaller 4%/year increase in billions of floating-point operations (GFLOPS) per core. This nets to an annual cost deflation of 24%. Thus, in FY13 a CPU chip with 8 cores provides 0.275 TFLOPS at a cost of $996, and therefore the cost is $36174.72/TFLOPS. By contrast, in FY20, a CPU chip with 29 cores provides 0.1219 TFLOPS at a cost of $996, and thus the cost is $8172.77/TFLOPS, a cost deflation of approximately a factor of 4.

Survey Year

Fiscal Year

GF per core

(peak)Cores per CPU chip

TF per CPU chip

(sustained)

Cost per CPU Chip

($)

Cost/TFAnnual Cost

Deflation

2013 13.0 8 0.0275 $996$36,174.7

2

2014 13.5 10 0.0341 $996$29,249.2

2 -24%

2015 14.0 12 0.0421 $996$23,649.5

8 -24%

2016 14.6 14 0.0521 $996$19,121.9

7 -24%

2017 15.2 17 0.0644 $996$15,461.1

5 -24%

2018 15.8 20 0.0797 $996$12,501.1

8 -24%

2019 16.4 24 0.0985 $996 $10,107.8 -24%


36


7/25/2014

8

2020 17.1 29 0.1219 $996 $8,172.77 -24%

1 2021 17.8 34 0.1507 $996 $6,608.13 -24%

2 2022 18.5 40 0.1864 $996 $5,343.03 -24%

3 2023 19.2 48 0.2305 $996 $4,320.13 -24%

4 2024 20.0 57 0.2851 $996 $3,493.06 -24%

5 2025 20.8 68 0.3527 $996 $2,824.33 -24%

6 2026 21.6 81 0.4361 $996 $2,283.62 -24%

7 2027 22.5 96 0.5394 $996 $1,846.43 -24%

8 2028 23.4 114 0.6671 $996 $1,492.94 -24%

9 2029 24.3 136 0.8251 $996 $1,207.12 -24%

10 2030 25.3 161 1.0205 $996 $976.02 -24%

2031 26.3 192 1.2621 $996 $789.17

-24%

Table 2: CPU Cost Deflation

Memory

Memory is estimated in numbers of gigabytes (GB) and number of GB/Dual Inline Memory Module (DIMM). The unit cost per DIMM is fixed over time, but the capacity of the DIMM in GB/DIMM increases over time, consistent with historical data and industry expectations. This increase is such that the GB/DIMM is doubled approximately every 3 years. Thus, the cost/GB decreases over time, inversely. Similarly, the input/output bandwidth of DIMMs increases over time, which is an important factor when verifying that we not only provide sufficient memory capacity, but for meeting our processing throughput requirements.

This can be verified by examining the InputTechnologyPredictions sheet within LDM-144. Table 3 below is extracted from LDM-144, and documents the growth of GB/DIMM (2x every 3 years, as well as bandwidth per DIMM in GB/s. This nets to an annual cost deflation of 21%. Thus, in FY13 a DIMM provides 0.8 GB at a cost of $120, and therefore the cost is $15/GB. By contrast, in FY20, a DIMM provides 40.3 GB at a cost of $120, and thus the cost is $2.98/GB, a cost deflation of approximately a factor of 5.

Survey Year

Fiscal Year

DIMMs per

Node

Capacity per

DIMM

Annual Cost

Cost per

DIMM

Power per

DIMM

Bandwidth per DIMM

Cost per GB


37


7/25/2014

(GB) Deflation ($) (watts) (GB/s) ($)

2013 16 8.0 $120 5.0 20.0 $15.00

2014 16 10.1 21% $120 5.0 23.8 $11.91

2015 16 12.7 21% $120 5.0 28.3 $9.45

2016 16 16.0 21% $120 5.0 33.6 $7.50

2017 16 20.2 21% $120 5.0 40.0 $5.95

2018 16 25.4 21% $120 5.0 47.6 $4.72

2019 16 32.0 21% $120 5.0 56.6 $3.75

2020 16 40.3 21% $120 5.0 67.3 $2.98

1 2021 16 50.8 21% $120 5.0 80.0 $2.36

2 2022 16 64.0 21% $120 5.0 95.1 $1.88

3 2023 16 80.6 21% $120 5.0 113.1 $1.49

4 2024 16 101.6 21% $120 5.0 134.5 $1.18

5 2025 16 128.0 21% $120 5.0 160.0 $0.94

6 2026 16 161.3 21% $120 5.0 190.3 $0.74

7 2027 16 203.2 21% $120 5.0 226.3 $0.59

8 2028 16 256.0 21% $120 5.0 269.1 $0.47

9 2029 16 322.5 21% $120 5.0 320.0 $0.37

10 2030 16 406.4 21% $120 5.0 380.5 $0.30

2031 16 512.0 21% $120 5.0 452.5 $0.23

Table 3: Memory Cost Deflation


38


7/25/2014

Disk Storage

Disk storage is estimated in two parts: local disk drives contained within database servers (consumer grade SATA) and network attached shared storage for file servers (Enterprise grade SATA drives, drive trays, and disk controllers).

While the unit cost for each type of drive, trays, and controllers is the same over time (Consumer SATA $80, Enterprise SATA $220, controller $4000, tray $2000), the capacity/drive in TB increases over time, per the Input Technology Predictions in the Sizing Model, consistent with historical data and industry expectations. This increase is such that the TB/drive is doubled approximately every 3.5 years. Thus, the cost/TB decreases over time, inversely. There is also an increase in bandwidth/controller, to ensure that the controllers are sized to adequately handle the i/o rate of the aggregate of drives controlled. Drive trays are assumed to be of fixed capacity (24 drives/tray) and rack profile.

This can be verified by examining the InputTechnologyPredictions sheet within LDM-144. Tables 4 and 5 below are extracted from LDM-144, and document the growth of TB/drive (2x every 3.5 years), for each of the types (Consumer SATA, Enterprise SATA). This nets to an annual cost deflation of 18%. Thus in Table 4, in FY13 a consumer SATA drive provides 4 TB at a cost of $80, and therefore the cost is $20/TB. By contrast, in FY20, a consumer SATA drive provides 16 TB at a cost of $80, and therefore the cost is $5/TB, a cost deflation of a factor of 4.

Survey Year

Fiscal Year

Capacity per Drive

(TB) Cost/TB

Annnual Cost

Deflation %

Seq Bandwidth per Drive

(MB/s)

IOPS per

DriveCost per Drive ($)

20134 $20.00 80 90 $80

20145 $16.41 -18% 89 90 $80

20156 $13.46 -18% 100 90 $80

20167 $11.04 -18% 112 90 $80

20179 $9.06 -18% 125 90 $80

201811 $7.43 -18% 140 90 $80

201913 $6.10 -18% 157 90 $80

202016 $5.00 -18% 175 90 $80

1 202120 $4.10 -18% 196 90 $80


39


7/25/2014

2 202224 $3.36 -18% 220 90 $80

3 202329 $2.76 -18% 246 90 $80

4 202435 $2.26 -18% 275 90 $80

5 202543 $1.86 -18% 307 90 $80

6 202653 $1.52 -18% 344 90 $80

7 202764 $1.25 -18% 385 90 $80

8 202878 $1.03 -18% 430 90 $80

9 202995 $0.84 -18% 481 90 $80

10 2030116 $0.69 -18% 538 90 $80

2031141 $0.57 -18% 602 90 $80

Table 4: Consumer Grade SATA Drive Cost Deflation

Similarly per Table 5, in FY13 a Enterprise SATA drive provides 4 TB at a cost of $220, and therefore the cost is $55/TB. By contrast, in FY20, and Enterprise SATA drive provides 16 TB at a cost of $220, and therefore the cost is $13.75/TB, a cost deflation of a factor of 4.

Survey Year Fiscal Year

Capacity per

Drive (TB)

Cost/TB

Annnual Cost

Deflation %

Seq Bandwidt

h per Drive

(MB/s)

IOPS per

Drive

Cost per

Drive ($)

20134 $55.00 80 90 $220

20145 $45.12 -18% 89 90 $220

20156 $37.01 -18% 100 90 $220

20167 $30.36 -18% 112 90 $220

20179 $24.91 -18% 125 90 $220

201811 $20.43 -18% 140 90 $220


40


7/25/2014

201913 $16.76 -18% 157 90 $220

202016 $13.75 -18% 175 90 $220

1 202120 $11.28 -18% 196 90 $220

2 202224 $9.25 -18% 220 90 $220

3 202329 $7.59 -18% 246 90 $220

4 202435 $6.23 -18% 275 90 $220

5 202543 $5.11 -18% 307 90 $220

6 202653 $4.19 -18% 344 90 $220

7 202764 $3.44 -18% 385 90 $220

8 202878 $2.82 -18% 430 90 $220

9 202995 $2.31 -18% 481 90 $220

10 2030116 $1.90 -18% 538 90 $220

2031141 $1.56 -18% 602 90 $220

Table 5: Enterprise SATA Drive Cost Deflation

Tape Storage

Tape storage was estimated in a fashion analogous to disk storage, but with tape storage a robotic tape farm was presumed. There are fixed costs for the Tape Library system hardware ($375,000), hardware maintenance ($56,250), HPSS software ($500,000), and HPSS software maintenance ($150,000). The cost deflation is represented in the tape media.

While the unit cost for each tape is the same over time ($70), the capacity/tape in TB increases over time, per the Input Technology Predictions in the Sizing Model, consistent with historical data and industry expectations. This increase is such that the TB/tape is doubled approximately every 5 years. Thus, the cost/TB decreases over time, inversely.

This can be verified by examining the InputTechnologyPredictions sheet within LDM-144. Table 6 below is extracted from LDM-144, and documents the growth of TB/tape (2x every 5 years. This nets to an annual cost deflation of 13%. Thus in Table 6, in FY13 a tape provides 2.5 TB at a cost of $70, and therefore the cost is $28/TB. By contrast, in FY20, a tape provides 6.6 TB at a cost of $70, and therefore the cost is $10.61/TB, a cost deflation of a factor of 2.64.


41


7/25/2014

Survey Year Fiscal Year

Capacity per Tape

(TB) Cost/TB

Annnual Cost

Deflation %Cost per Tape ($)

20132.5 $28.00 $70

20142.9 $24.38 -13% $70

20153.3 $21.22 -13% $70

20163.8 $18.47 -13% $70

20174.4 $16.08 -13% $70

20185.0 $14.00 -13% $70

20195.7 $12.19 -13% $70

20206.6 $10.61 -13% $70

1 20217.6 $9.24 -13% $70

2 20228.7 $8.04 -13% $70

3 202310.0 $7.00 -13% $70

4 202411.5 $6.09 -13% $70

5 202513.2 $5.31 -13% $70

6 202615.2 $4.62 -13% $70

7 202717.4 $4.02 -13% $70

8 202820.0 $3.50 -13% $70

9 202923.0 $3.05 -13% $70

10 203026.4 $2.65 -13% $70

203130.3 $2.31 -13% $70

Table 6: Tape Cost Deflation


42


7/25/2014

Local Area Networks

Local Area Network was estimated in terms of a representative building-wide ethernet network connecting individual workstations, and an inside-the-data-center infiniband network connecting the computing and storage.

While the unit cost per switch remains constant over time ($30,000 for ethernet switch, $36,000 for infiniband core switch and $12,000 per infiniband edge switch) the bandwidth in GB/s increases over time, per the Input Technology Predictions in the Sizing Model, consistent with historical data and industry expectations. This increase is such that the GB/s is doubled approximately every 5 years for ethernet switches and increases by a factor 6 every 8 years for infiniband switches. Thus, the cost/GB/s decreases over time, inversely.

This can be verified by examining the InputTechnologyPredictions sheet within LDM-144. Tables 7 and 8 below are extracted from LDM-144, and document the growth of GB/s for these switch types. This nets to an annual cost deflation of 20% for infiniband switches and 13% for ethernet switches. Thus in Table 7, in FY13 an infiniband switch (with core and edges) provides 1 GB/s at a cost of $48,000, and therefore the cost is $48,000/GB/s. By contrast, in FY20, an infiniband switch provides 4.8 GB/s at a cost of $48,000, and therefore the cost is $10,000 GB/s, a cost deflation of a factor of 4.8.

Survey Year

Fiscal Year

Bandwidth per IB

Port (GB/s) Cost/GB/s

Annnual Cost

Deflation %

Ports per IB Edge

Switch

Cost per Infiniband

Edge Switch

($)

Cost per Infiniband

Core Switch

($)

20131.0

$48,000.00 18 $12,000 $36,000

20141.3

$38,368.28 -20% 18 $12,000 $36,000

20151.6

$30,669.27 -20% 18 $12,000 $36,000

20162.0

$24,515.15 -20% 18 $12,000 $36,000

20172.4

$19,595.92 -20% 18 $12,000 $36,000

20183.1

$15,663.78 -20% 18 $12,000 $36,000

20193.8

$12,520.68 -20% 18 $12,000 $36,000


43


7/25/2014

20204.8

$10,008.27 -20% 18 $12,000 $36,000

1 20216.0 $8,000.00 -20% 18 $12,000 $36,000

2 20227.5 $6,394.71 -20% 18 $12,000 $36,000

3 20239.4 $5,111.54 -20% 18 $12,000 $36,000

4 202411.7 $4,085.86 -20% 18 $12,000 $36,000

5 202514.7 $3,265.99 -20% 18 $12,000 $36,000

6 202618.4 $2,610.63 -20% 18 $12,000 $36,000

7 202723.0 $2,086.78 -20% 18 $12,000 $36,000

8 202828.8 $1,668.04 -20% 18 $12,000 $36,000

9 202936.0 $1,333.33 -20% 18 $12,000 $36,000

10 203045.0 $1,065.79 -20% 18 $12,000 $36,000

203156.3 $851.92 -20% 18 $12,000 $36,000

Table 7: Infiniband Switch Cost Deflation

Similarly in Table 8, in FY13 an ethernet switch provides 80 GB/s at a cost of $30,000, and therefore the cost is $375.00/GB/s. By contrast, in FY20, an ethernet switch provides 211 GB/s at a cost of $30,000, and therefore the cost is $142.10GB/s, a cost deflation of a factor of 2.64.

Survey Year

Fiscal Year

Bandwidth per

10GigE Switch (GB/s) Cost/GB/s

Annnual Cost

Deflation %

Cost per 10GigE Switch

($)

201380 $375.00 $30,000

201492 $326.46 -13% $30,000

2015106 $284.20 -13% $30,000

2016 121 $247.41 -13% $30,000


44


7/25/2014

2017139 $215.38 -13% $30,000

2018160 $187.50 -13% $30,000

2019184 $163.23 -13% $30,000

2020211 $142.10 -13% $30,000

1 2021243 $123.70 -13% $30,000

2 2022279 $107.69 -13% $30,000

3 2023320 $93.75 -13% $30,000

4 2024368 $81.61 -13% $30,000

5 2025422 $71.05 -13% $30,000

6 2026485 $61.85 -13% $30,000

7 2027557 $53.85 -13% $30,000

8 2028640 $46.87 -13% $30,000

9 2029735 $40.81 -13% $30,000

10 2030844 $35.52 -13% $30,000

2031970 $30.93 -13% $30,000

Table 8: Ethernet Swtich Cost Deflation

Workstations

Workstations were estimated at a unit cost of $2500 fixed over time. While it is expected that the capability of the workstations will increase over time (CPU in terms of GFLOPS, memory capacity in terms of GB, and local disk capacity in terms of TB), these increases will not result in the need for fewer workstations, since the number of workstations needed is driven the anticipated number of operations staff. Nor will the cost of each purchased workstation decrease, rather more capable workstations will be purchased for the same fixed unit cost.


45