Prof Ian F Bitterlin CEng PhD BSc(Hons) BA DipDesInn

FIET MCIBSE MBCS

Visiting Professor, School of Mechanical Engineering, University of Leeds

Data Centre Infrastructure Consulting Engineer

Data Centre Specialist Group

Moore’s Law

PUE, CUE, WUE...

ISO/IEC 30134 KPIs for Resource Effective Data Centres

2011 Thermal Guidelines

Moore’s Law

What is driving power growth?

Is growth a problem?

What can moderate growth?

Where could it all end?

• Gordon Moore, founder of Intel, wrote his Law in 1965 when he

predicted the doubling of the number of transistors on a

microprocessor every two years

• It was a physical photo-etching process improvement cycle that has

held true ever since, latterly by other means

• Directly applies to • Doubling compute capacity

• Halving the Watts/FLOP

• Halving kWh per unit of compute load

• Law ‘updated’ by Intel to 18 months for clock-rate increase

• Kurzweil suggests that the doubling is ‘now’ every 14 months

• Encourages ever-shorter hardware refresh rates • Facebook <12 months – slows down their data centre growth

• Google was <36 months refresh rate, now shorter?

• Keeping ICT hardware >3 years is now energy profligate

1947, Bell Labs, the first transistor was hand built 2012, Intel 22nm tri-gate transistor 100 million into a 1.8mm pin-head 6 million onto a typed full-stop, 0.1mm2

3rd Generation quad-core has 1.48 Billion transistors Each can switch on/off 100 Billion times per second Pundits predict that 22nm is the end of Moore’s Law

Intel make 5 Billion transistors every second 20 million per person on the planet, per year

1971-2012 4,000x faster, 1/5,000th energy, 1/50,000th cost

Data traffic growth • At Photonics West 2009 in San Jose, Cisco correctly predicted for

2012 that ‘20 US homes with FTTH will generate more traffic than the

entire internet backbone carried in 1995’

• Japanese average home with FTTH - download rate is >500MB per

day, dominated by HD-Video • One Sky movie download = 1.3GB

• 4K UHD-TV will just increase traffic and power consumption

• More video content is uploaded to YouTube every month than a TV

station can broadcast 24h/day for 400 years

• Phones with 4G are huge data-generators. Even with 3G in 2011

Vodafone reported a 79% data-growth in one year – was that all

social networking, photo’s and video?

2001: 690 TB/month 2015: 680,000 TB/month

1000x growth 54% CAGR* 4% CGR per month Peak 3.34 TB/s Capacity 12.85 TB/s *CAGR = Compound Annual Growth Rate

Information generation

• 2009 = 50GB/s

• 2020 = 500GB/µs

• 10,000,000x increase

Ray Kurzweil has been described as “the restless genius” by the Wall Street Journal, and “the ultimate thinking machine” by Forbes magazine, ranking him #8 among entrepreneurs in the United States and calling him the “rightful heir to Thomas Edison”. PBS included Ray as one of 16 “revolutionaries who made America,” along with other inventors of the past two centuries.

The Singularity is Near Raymond Kurzweil, 2005, Viking

The Law of Accelerating Returns, Kurzweil

Jevons Paradox ‘It is a confusion of ideas to suppose that the

economical use of fuel is equivalent to diminished

consumption. The very contrary is the truth’ William Stanley Jevons, 1865

The Coal Question, Published 1865, London, Macmillan & Co

Newcomen’ s engine was c2% thermally efficient and coal supplies in the UK were highly strained Watt’s engine replaced it with c5% efficiency - but the result was rapid increase in coal consumption

Can the same be said of internet usage?

Sex

Wi-Fi

Digital Agenda

The EU has a digital agenda that involves 20MB/s super-fast broadband for

all citizens at an affordable price

Faster access will, according to Jevons Paradox, generate a power demand

increase but no government has yet appeared to understand the direct

linkage mechanism between data-generation and power demand…

Faster access used for education, security & medical services may be key to

many poorer nations development

7 Billion global population, c2 Billion internet connections?

‘Internet access will become a privilege, not a right’ Vint Cerf, 2011 Inventor of the IP address and often regarded as one of the ‘Fathers of the Internet’

Now VP and Chief Internet Evangelist, Google – working on inter-Galactic IP addresses

What is this driving?

The UK-Gov’s TV campaign for super-fast roll-out... Applications? Whilst DECC & DEFRA are trying to reduce growth in ICT power consumption... Does the left-hand know what the right-hand is doing? Has Jevons taught us nothing?

TIME magazine reported that it

takes 0.0002kWh to stream 1

minute of video from the YouTube

data centre…

Based on Jay Walker’s recent

TED talk, 0.01kWh of energy is

consumed on average in carrying

1MB over the Internet.

The average home device energy

consumption is around 0.001kWh

for 1 minute of video streaming

For 1.7B downloads of this 17MB

file streaming for 4.13 mins gives

the overall energy for this one

viral pop video in just one year…

• 35MW of 24/7/365 generation

• 100 million Litres of fuel oil

• 250,000 Tons CO2

• 80,000 UK car years

• Viral videos appear every day, numerous examples at >2Bn annual rate

• Volvo (JC van Damme)

• First Kiss

298GWh = more than the annual electricity consumption of the

nine million population of the Republic of Burundi (East Africa)

who used 273GWh in 2008

• 2001: LD photos and pornography • 2015: HD photos, YouTube, Gaming, Gambling

and Films • Limits to growth? 7Bn people (not growing

exponentially) each with a maximum viewing time of 24h/day

• Exponential growth is coming from The-Internet-of-Things with multi-quadrillion sensors, each transmitting a stream of short data packets

• Underpinned by growth from the third world

• ICT in total is generally regarded as consuming 6-9% of the utility and data centres 1/3rd of that, so…

• Data centres consume 2-3% of our grid capacity and that is currently growing at c20% CAGR

• Clearly NOT sustainable...

• But what happens if Moore’s Law runs out of steam?

• Efficiency = Work output ÷ Work input

• What is the output of a data centre? • Digital services and 100% heat , no ‘work done’

• Better to use the terminology of Effectiveness • PUE, WUE, CUE...

• Productivity metrics are the hardest to define

• ITEE+ITEU for servers, storage & I/O

• Sustainability metrics like RWH, REF and waste recycling...

• PUE is critical but NOT to be used in isolation

• True sustainability is more complicated...

• Reduce consumption • What is the social & economic ‘value’ of the data processed?

• Virtualization, high utilisation, frequent ICT refresh

• Improve: 2014 average server runs at 10% utilisation on 40% power

• Restrict access for certain applications? Political suicide...

• Improve efficiency (effectiveness) • Moore’s Law plus a paradigm shift in network photonics?

• PUE, WUE, REF, CUE, RWH, ITEE & ITEU + ....

• Power from renewable energy sources • Building a ‘legacy’ data-centre next to a hydro-electric scheme is NOT a

sustainable design, it could be a waste of renewable energy on an ineffective data centre if the first two steps are not taken

• EU (DG Connect) is starting down the road of regulating data centre power • Fixated by renewables for ICT

• People have discussed (but will reject as political suicide) the throttling-back of broadband for certain applications – but regulation or taxation? • Remove HD-cameras from mobile-phones?

• Natural limits on network capacity will place ‘Jevons’ type barriers upon ‘unlimited’ data bundles

• ISO/IEC 30134 - KPIs for Resource Effective Data Centres

• Better ICT hardware – high efficiency SMPS - ITEU

• Better software, more efficient code

• Higher utilisation (more virtualisation) - ITEE

• Best practice data centres – lower PUE

• Low WUE

• Low CUE

• High RWH

• The EU is fixated in using ICT as an energy exemplar • Renewables are a finite and variable resource • There is no logical reason why ICT should be

powered by renewable power more than any other application like research, medicine, education or social services…

• On-site solar-PV power can only contribute <1.5% to a data centres power demand, everything else has to be imported

• ISO/IEC 30134-3 REF (currently out for public comment)

100% in 2001 0.08% in 2015 Moore's Law Clock-rate PUE, 3 to 1.5 Virtualization, 0-40% Utilization, 5-40%

1,250x

What’s the difference between a ‘mega’ data centre and a ‘micro’ data centre?

The answer is not ‘size’...

The answer is ‘time’

• Best-in-class, today 1250x 2001

• Best practice, today 700x 2001

• Best practice, 4 year old base 100x 2001

2001 IDC 5MW site 2MW IT load 2000 cabinets ~3000m2

2015 Best Practice/4y 40kW site 30kW IT load 5 cabinets ~16m2

2015 Best-in-class, today 2kW site <2kW IT load <1 cabinet ~3m2

Grid Power

3%

Years

• For high-penetration internet use in Europe there is a natural upper limit to the current avalanche of HD-video data: The number of people each with 24h/day

• The Internet-Of-Things will grow to dominate traffic volume – multi-trillions of sensors, each sending a small packet of data every second

• The next paradigm shift (Kurzweil’s 5th) will replace silicon in microprocessors and impact network photonics and servers • Graphene? Arsenide's? E.Coli? • Setting off on another 50 years of Moore’s Law?

Green = All ICT, 9% with 20% CAGR + 5th paradigm shift + IoT Red = 3% Data Centre, 20% CAGR 1-6y, 10% CAGR 7-12y Blue = 3% Data Centre + 5th paradigm shift + IoT 0.03% with 60% CAGR

Waiting for the 6th paradigm shift?

Years Years

% of utility capacity % of utility capacity

The 5th paradigm shift

• Reduce consumption of ICT? No chance!

• Increase ICT hardware effectiveness and dramatically improve utilisation

• Minimise PUE to meet business case • Cooling pPUE is still the low hanging fruit (ASHRAE)

• Eco-mode UPS (EU CoC)

• Re-use waste heat • Hard to do in ‘bulk’

• All of these, and more, will be in ISO/IEC 30134

ASHRAE - widening the envelope • The critical change has been to concentrate on server inlet

temperatures, maximising the return-air temperature

• Rigorous air-containment is ‘best practice’

• Choice of Class determines minimum pPUE

Will ASHRAE go further and expand the ‘Recommend’, not just the ‘Allowable’? Probably not… why?

Spec_Power: Server OEMs input the data

Utility servers

In this small extract from the web-site, HP ProLiant models average 41% idle power and vary from 24% to 79% Pundits predict that 23% is the minimum for silicon http://www.spec.org/power_ssj2008/

The real ‘efficiency’ battleground…

Average utilisation must increase

The IT load will become dynamic and overall energy consumption will reduce The above illustrates a >90% reduction in energy consumption per compute

• The industry talks a lot about energy ‘efficiency’ but many end-users don’t ‘do’ much, why? • Not enough server sales are with 80+ & 90+ power supplies

• The cost of energy is very often a minor issue with respect to the cost of ICT failure and the business enabled by ICT • 1kWh in a data centre drives £125 of UK business turnover • At £0.1/kWh = 0.08% cost • Cutting data centre power by 25% = 0.02% impact

• Paranoia and fear of business failure dominates • Would you want to be treated in a cut-cost ‘effective’

operating theatre?

Data Centres are at the heart of the internet and enable the digital

economy, including the concepts of smart-cities and smart-grids

Data-centres will expand as our demands, in both social and

business contexts, for digital content and services grow ◦ Facilities will become storage dominant and footprint will increase…

◦ Loads will become more load:power linear and, as a result, more dynamic

◦ Thermal management will become adopted and PUE’s will fall to c1.2

◦ Only larger, highly virtualised and heavily loaded facilities will enable low-cost

digital services as the cost of power escalates (but these are the hardest for

RWH to be applied)

Despite industry best efforts power consumption will rise, not fall ◦ Best forecasts show a growth rate at 15-20% CAGR for the foreseeable future –

often in connected locations where energy is cheap and taxes are low…

◦ Utilisation is the single largest factor in moderating power growth

◦ Data growth continues to outstrip Moore’s Law and a paradigm shift in network

photonics and devices will be required but, even then, a change in usage

behaviour will probably be required …

Predicting the future of IT is hard...

1997 – the world’s fastest super-computer

SANDIA National Laboratories ‘ASCI RED’ 1.8 teraflops 150m² raised floor 800kW

2006 Sony Playstation3 1.8 teraflops 0.08m² 190W

Top500 Nov 2013, China Milky Way-2 33.86 PetaFLOPS (55 peak) 20,000x the speed of 1997 17MW Source: www.top500.org

Data Centre Specialist Group