cloud engineering
DESCRIPTION
An introduction to datacenter and cloud computing for master students in engineeringTRANSCRIPT
![Page 1: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/1.jpg)
Cloud EngineeringSoftware in DatacentersGwendal SimonDepartment of Computer ScienceInstitut Telecom2009
![Page 2: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/2.jpg)
Gartner Hype Cycle 2009
2 / 43 Gwendal Simon Cloud Engineering
![Page 3: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/3.jpg)
Literature
A book:“The Datacenter as a Computer”, Luiz AndréBarroso and Urs Hölzle
and scientific publications including:ACM Sigops (SOSP) and Usenix OSDIHPDC, EuroPar, OOPSLA, etc.blogs (e.g. http://perspectives.mvdirona.com/)
3 / 43 Gwendal Simon Cloud Engineering
![Page 4: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/4.jpg)
Disclaimer
No discussion about the impact of cloud computing:net neutralityinteractions between CDNs and ISPsprivacy and electronic human rights. . .
Focus here on how cloud computing works
4 / 43 Gwendal Simon Cloud Engineering
![Page 5: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/5.jpg)
Disclaimer
No discussion about the impact of cloud computing:net neutralityinteractions between CDNs and ISPsprivacy and electronic human rights. . .
Focus here on how cloud computing works
4 / 43 Gwendal Simon Cloud Engineering
![Page 6: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/6.jpg)
Introduction
5 / 43 Gwendal Simon Cloud Engineering
![Page 7: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/7.jpg)
In a nutshell
Cloud computing is a model for enabling convenient,on-demand network access to a shared pool ofconfigurable computing resources (e.g., networks,servers, storage, applications, and services) that canbe rapidly provisioned and released with minimalmanagement effort or service provider interaction
NIST: http://csrc.nist.gov/groups/SNS/cloud-computing/index.html
6 / 43 Gwendal Simon Cloud Engineering
![Page 8: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/8.jpg)
Why Cloud Computing
The ∗aaS paradigm:SaaS: Software as a Service
applications for end-users (salesforce.com, Google, etc.)- email, office suite, photos sharing, video storage
PaaS: Platform as a Serviceservices for web app developers (Azure, Google, etc.)- workflow facilities and various basic services (http, database)
IaaS: Infrastructure as a Serviceresources for developers (Amazon, Joyent, etc.)- servers, network equipment, memory, CPU
7 / 43 Gwendal Simon Cloud Engineering
![Page 9: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/9.jpg)
Why Cloud Computing
The ∗aaS paradigm:SaaS: Software as a Service
applications for end-users (salesforce.com, Google, etc.)- email, office suite, photos sharing, video storage
PaaS: Platform as a Serviceservices for web app developers (Azure, Google, etc.)- workflow facilities and various basic services (http, database)
IaaS: Infrastructure as a Serviceresources for developers (Amazon, Joyent, etc.)- servers, network equipment, memory, CPU
7 / 43 Gwendal Simon Cloud Engineering
![Page 10: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/10.jpg)
Why Cloud Computing
The ∗aaS paradigm:SaaS: Software as a Service
applications for end-users (salesforce.com, Google, etc.)- email, office suite, photos sharing, video storage
PaaS: Platform as a Serviceservices for web app developers (Azure, Google, etc.)- workflow facilities and various basic services (http, database)
IaaS: Infrastructure as a Serviceresources for developers (Amazon, Joyent, etc.)- servers, network equipment, memory, CPU
7 / 43 Gwendal Simon Cloud Engineering
![Page 11: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/11.jpg)
An Engineer Vision
Benefits: Outsourcing Infrastructurereduce run time and response timeminimize infrastructure riskease deployment and upgrading
Challenge: Warehouse-Scale Computersthousands of individual computing nodescostly equipments (power, conditioning, cooling)large buildings with engineering teams
8 / 43 Gwendal Simon Cloud Engineering
![Page 12: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/12.jpg)
An Engineer Vision
Benefits: Outsourcing Infrastructurereduce run time and response timeminimize infrastructure riskease deployment and upgrading
Challenge: Warehouse-Scale Computersthousands of individual computing nodescostly equipments (power, conditioning, cooling)large buildings with engineering teams
8 / 43 Gwendal Simon Cloud Engineering
![Page 13: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/13.jpg)
Few Pictures
9 / 43 Gwendal Simon Cloud Engineering
![Page 14: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/14.jpg)
Few Pictures
9 / 43 Gwendal Simon Cloud Engineering
![Page 15: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/15.jpg)
Few Pictures
9 / 43 Gwendal Simon Cloud Engineering
![Page 16: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/16.jpg)
Few Pictures
9 / 43 Gwendal Simon Cloud Engineering
![Page 17: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/17.jpg)
Datacenter is Different
Datacenter vs. Desktop Software:inherent parallelismsoftware controlplatform homogeneityfault-free requirement
Datacenter vs. High-Performance Computingunpredictable inputhigh volume of datanot only computing
10 / 43 Gwendal Simon Cloud Engineering
![Page 18: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/18.jpg)
Datacenter is Different
Datacenter vs. Desktop Software:inherent parallelismsoftware controlplatform homogeneityfault-free requirement
Datacenter vs. High-Performance Computingunpredictable inputhigh volume of datanot only computing
10 / 43 Gwendal Simon Cloud Engineering
![Page 19: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/19.jpg)
Basic Elements ofa Datacenter
Computing ArchitectureEnergyDealing with Failures
11 / 43 Gwendal Simon Cloud Engineering
![Page 20: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/20.jpg)
Basic Elements ofa Datacenter
Computing ArchitectureEnergyDealing with Failures
12 / 43 Gwendal Simon Cloud Engineering
![Page 21: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/21.jpg)
Architecture Basics
13 / 43 Gwendal Simon Cloud Engineering
![Page 22: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/22.jpg)
Main Elements
Storage: distributed file sys. (e.g. GFS) or NAS ?GFS is cheaper and faster for read operations
Network: 1-Gbps switch with 48 ports in a rack1 port per server, 8 ports for cluster rack
→ Oversubscription factor greater than 5scarce cluster-level bandwidthattractive rack-level networking
14 / 43 Gwendal Simon Cloud Engineering
![Page 23: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/23.jpg)
Main Elements
Storage: distributed file sys. (e.g. GFS) or NAS ?GFS is cheaper and faster for read operations
Network: 1-Gbps switch with 48 ports in a rack1 port per server, 8 ports for cluster rack
→ Oversubscription factor greater than 5scarce cluster-level bandwidthattractive rack-level networking
14 / 43 Gwendal Simon Cloud Engineering
![Page 24: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/24.jpg)
System Overview (in 2009)
15 / 43 Gwendal Simon Cloud Engineering
![Page 25: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/25.jpg)
Basic Elements ofa Datacenter
Computing ArchitectureEnergyDealing with Failures
16 / 43 Gwendal Simon Cloud Engineering
![Page 26: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/26.jpg)
Main Electric Components
17 / 43 Gwendal Simon Cloud Engineering
![Page 27: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/27.jpg)
Cooling Challenge
18 / 43 Gwendal Simon Cloud Engineering
![Page 28: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/28.jpg)
Evaluating Energy Efficiency
efficiency =computationtotal energy
Power Usage Effectiveness = total energyenergy in equipment
first generation datacenter PUE was poor (often ≥ 3.0)toward PUE around 1.2
Server PUE = critical component powertotal server power
basic SPUE is 1.7state-of-the-art servers reach 1.2
and computing efficiency
19 / 43 Gwendal Simon Cloud Engineering
![Page 29: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/29.jpg)
Evaluating Energy Efficiency
efficiency =computationtotal energy
Power Usage Effectiveness = total energyenergy in equipment
first generation datacenter PUE was poor (often ≥ 3.0)toward PUE around 1.2
Server PUE = critical component powertotal server power
basic SPUE is 1.7state-of-the-art servers reach 1.2
and computing efficiency
19 / 43 Gwendal Simon Cloud Engineering
![Page 30: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/30.jpg)
Evaluating Energy Efficiency
efficiency =computationtotal energy
Power Usage Effectiveness = total energyenergy in equipment
first generation datacenter PUE was poor (often ≥ 3.0)toward PUE around 1.2
Server PUE = critical component powertotal server power
basic SPUE is 1.7state-of-the-art servers reach 1.2
and computing efficiency19 / 43 Gwendal Simon Cloud Engineering
![Page 31: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/31.jpg)
Computing Efficiency
Benchmarking cluster-level efficiency: on-going work
Benchmarking individual computer is easier
20 / 43 Gwendal Simon Cloud Engineering
![Page 32: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/32.jpg)
Computing Efficiency
Benchmarking cluster-level efficiency: on-going workBenchmarking individual computer is easier
20 / 43 Gwendal Simon Cloud Engineering
![Page 33: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/33.jpg)
Computing Efficiency
Benchmarking cluster-level efficiency: on-going workBenchmarking individual computer is easier
20 / 43 Gwendal Simon Cloud Engineering
![Page 34: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/34.jpg)
Toward Energy-Proportional Servers
21 / 43 Gwendal Simon Cloud Engineering
![Page 35: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/35.jpg)
Basic Elements ofa Datacenter
Computing ArchitectureEnergyDealing with Failures
22 / 43 Gwendal Simon Cloud Engineering
![Page 36: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/36.jpg)
Fault-Tolerant Application-Level
Fault-Tolerant software infrastructure layer:masks failures of lower-layer levelsreduces hardware costeases operational procedures (e.g., upgrade)
But application-level still experiences failuresservice is in degraded modeservice is unreachableservice is corrupted (loss of data)
23 / 43 Gwendal Simon Cloud Engineering
![Page 37: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/37.jpg)
Fault-Tolerant Application-Level
Fault-Tolerant software infrastructure layer:masks failures of lower-layer levelsreduces hardware costeases operational procedures (e.g., upgrade)
But application-level still experiences failuresservice is in degraded modeservice is unreachableservice is corrupted (loss of data)
23 / 43 Gwendal Simon Cloud Engineering
![Page 38: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/38.jpg)
Origin of Impacting Failures
Cause % of eventssoftware 33 %
configuration 28 %human 13 %network 12 %hardware 11 %
other 3 %
hardware faults are masked by fault-tolerant software
24 / 43 Gwendal Simon Cloud Engineering
![Page 39: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/39.jpg)
Origin of Impacting Failures
Cause % of eventssoftware 33 %
configuration 28 %human 13 %network 12 %hardware 11 %
other 3 %
hardware faults are masked by fault-tolerant software
24 / 43 Gwendal Simon Cloud Engineering
![Page 40: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/40.jpg)
Failures and Crash
Average machine availability is ' 99.9%95% of machines restart less than once a month80% of restart events last less than 10 minutes
Software most frequent faults (in one year):DRAM soft-errors: 1% experience uncorrectable errdisk soft-errors: 3% of drives see corrupted sectors
25 / 43 Gwendal Simon Cloud Engineering
![Page 41: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/41.jpg)
Failures and Crash
Average machine availability is ' 99.9%95% of machines restart less than once a month80% of restart events last less than 10 minutes
Software most frequent faults (in one year):DRAM soft-errors: 1% experience uncorrectable errdisk soft-errors: 3% of drives see corrupted sectors
25 / 43 Gwendal Simon Cloud Engineering
![Page 42: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/42.jpg)
SoftwareInfrastructure
FundamentalsCluster-Level: MapReduceApplication-Level: Web Search
26 / 43 Gwendal Simon Cloud Engineering
![Page 43: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/43.jpg)
SoftwareInfrastructure
FundamentalsCluster-Level: MapReduceApplication-Level: Web Search
27 / 43 Gwendal Simon Cloud Engineering
![Page 44: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/44.jpg)
Three Layers
Infrastructure-level software:kernel, operating systems, networking libraries
Cluster-level software (middleware):specific software operating a pool of servers
Application-level software:implementation of the Internet services
28 / 43 Gwendal Simon Cloud Engineering
![Page 45: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/45.jpg)
Three Layers
Infrastructure-level software:kernel, operating systems, networking libraries
Cluster-level software (middleware):specific software operating a pool of servers
Application-level software:implementation of the Internet services
28 / 43 Gwendal Simon Cloud Engineering
![Page 46: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/46.jpg)
Three Layers
Infrastructure-level software:kernel, operating systems, networking libraries
Cluster-level software (middleware):specific software operating a pool of servers
Application-level software:implementation of the Internet services
28 / 43 Gwendal Simon Cloud Engineering
![Page 47: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/47.jpg)
Main Software Components
replication
partitioningload-balancinghealth checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 48: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/48.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 49: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/49.jpg)
Main Software Components
replicationpartitioning
load-balancing
health checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 50: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/50.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checking
integrity checkcompression
weak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 51: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/51.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 52: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/52.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checkingintegrity check
compression
weak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 53: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/53.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 54: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/54.jpg)
Main Software Components
replicationpartitioning
load-balancinghealth checkingintegrity check
compressionweak consistency
MapReduceDynamoBigTableHadoopSawzallChubbyDryad
29 / 43 Gwendal Simon Cloud Engineering
![Page 55: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/55.jpg)
OS at a Cluster-Level Scale
Resource Management: mapping tasks to resourcesshould optimize energy usage
Hardware Abstraction: handling hardware elementsshould optimize performances
Deployment Maintenance: upgrading and monitoringshould reduce manual tasks
Programming Frameworks: easing implementationshould increase programmer productivity
30 / 43 Gwendal Simon Cloud Engineering
![Page 56: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/56.jpg)
OS at a Cluster-Level Scale
Resource Management: mapping tasks to resourcesshould optimize energy usage
Hardware Abstraction: handling hardware elementsshould optimize performances
Deployment Maintenance: upgrading and monitoringshould reduce manual tasks
Programming Frameworks: easing implementationshould increase programmer productivity
30 / 43 Gwendal Simon Cloud Engineering
![Page 57: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/57.jpg)
OS at a Cluster-Level Scale
Resource Management: mapping tasks to resourcesshould optimize energy usage
Hardware Abstraction: handling hardware elementsshould optimize performances
Deployment Maintenance: upgrading and monitoringshould reduce manual tasks
Programming Frameworks: easing implementationshould increase programmer productivity
30 / 43 Gwendal Simon Cloud Engineering
![Page 58: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/58.jpg)
OS at a Cluster-Level Scale
Resource Management: mapping tasks to resourcesshould optimize energy usage
Hardware Abstraction: handling hardware elementsshould optimize performances
Deployment Maintenance: upgrading and monitoringshould reduce manual tasks
Programming Frameworks: easing implementationshould increase programmer productivity
30 / 43 Gwendal Simon Cloud Engineering
![Page 59: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/59.jpg)
SoftwareInfrastructure
FundamentalsCluster-Level: MapReduceApplication-Level: Web Search
31 / 43 Gwendal Simon Cloud Engineering
![Page 60: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/60.jpg)
Motivation
Map/Reduce is a software framework for easily writing applications whichprocess vast amounts of data (multi-terabyte data-sets) in-parallel on largeclusters (thousands of nodes) of commodity hardware in a reliable,fault-tolerant manner.
32 / 43 Gwendal Simon Cloud Engineering
![Page 61: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/61.jpg)
Functional Programming
Two fundamentals functions:map: apply a function to a list of elements
map f [] = []| map f [x::xs] = (f x) :: (map f xs)
map square [1,2,5] → [1,4,25]
reduce: build a value from a function and a listreduce f a [] = a| reduce f a [x::xs] = reduce f (f x a) xs
reduce add 0 [1,3,6] → 10
33 / 43 Gwendal Simon Cloud Engineering
![Page 62: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/62.jpg)
Functional Programming
Two fundamentals functions:map: apply a function to a list of elements
map f [] = []| map f [x::xs] = (f x) :: (map f xs)
map square [1,2,5] → [1,4,25]
reduce: build a value from a function and a listreduce f a [] = a| reduce f a [x::xs] = reduce f (f x a) xs
reduce add 0 [1,3,6] → 10
33 / 43 Gwendal Simon Cloud Engineering
![Page 63: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/63.jpg)
MapReduce
Implementing two functions w.r.t data (key,val)map: smaller sub-problems distributed to nodes
map (inKey, inVal) → list (outKey, v)produces intermediate values with an output key
reduce: combines results of sub-problemsreduce (outKey, list v) → outValproduces an output value from intermediate values
34 / 43 Gwendal Simon Cloud Engineering
![Page 64: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/64.jpg)
MapReduce
Implementing two functions w.r.t data (key,val)map: smaller sub-problems distributed to nodes
map (inKey, inVal) → list (outKey, v)produces intermediate values with an output key
reduce: combines results of sub-problemsreduce (outKey, list v) → outValproduces an output value from intermediate values
34 / 43 Gwendal Simon Cloud Engineering
![Page 65: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/65.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 66: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/66.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 67: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/67.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 68: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/68.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 69: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/69.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 70: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/70.jpg)
Example: Word Countmap(filename, content):
for each w in content:emitInt(w, 1)
reduce(word, partCount):int result = 0for pc in partCount:
result += pcemit(result)
map(file1, “hello me, goodbye me”)→<hello,1><me,1><goodbye,1><me,1>
map(file2, “hello you, bye you”)→<hello,1><you,1><bye,1><you,1>
a given key is allocated to a given server
reduce(hello,<1,1>) → 2. . .
<hello,2><me,2><you,2><goodbye,1><bye,1>
35 / 43 Gwendal Simon Cloud Engineering
![Page 71: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/71.jpg)
MapReduce Implemented
36 / 43 Gwendal Simon Cloud Engineering
![Page 72: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/72.jpg)
SoftwareInfrastructure
FundamentalsCluster-Level: MapReduceApplication-Level: Web Search
37 / 43 Gwendal Simon Cloud Engineering
![Page 73: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/73.jpg)
Basics
Input:the Web ' 100 billion file ' 400 terabytesthe pagerank algorithma query “w1 AND w2 AND · · · AND wn”
Output:a list of files containing all words wi , i ∈ [1, n]sorted by the pagerank algorithm
38 / 43 Gwendal Simon Cloud Engineering
![Page 74: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/74.jpg)
Basics
Input:the Web ' 100 billion file ' 400 terabytesthe pagerank algorithma query “w1 AND w2 AND · · · AND wn”
Output:a list of files containing all words wi , i ∈ [1, n]sorted by the pagerank algorithm
38 / 43 Gwendal Simon Cloud Engineering
![Page 75: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/75.jpg)
Implementation Overview
Offline task: index managementbased on keywords:
a word is associated with a tablea table contains all occurrences in the web
distributed on thousands of machinesmultiple copies and weak consistency
Online task: query managementfront-end Web servers to a subset of machines:
compute and rank their local resultsall best results are combined
intermediate servers to servers of file replicafrom file pointers to a set of metadata
39 / 43 Gwendal Simon Cloud Engineering
![Page 76: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/76.jpg)
Implementation Overview
Offline task: index managementbased on keywords:
a word is associated with a tablea table contains all occurrences in the web
distributed on thousands of machinesmultiple copies and weak consistency
Online task: query managementfront-end Web servers to a subset of machines:
compute and rank their local resultsall best results are combined
intermediate servers to servers of file replicafrom file pointers to a set of metadata
39 / 43 Gwendal Simon Cloud Engineering
![Page 77: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/77.jpg)
Discussion
The user-perceived latency is less than one second:read-only operationshigh parallelismmany thousands of queries per secondtraffic variations
Networking:tiny size of data exchangespossible packet loss around the front-end servers
40 / 43 Gwendal Simon Cloud Engineering
![Page 78: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/78.jpg)
Discussion
The user-perceived latency is less than one second:read-only operationshigh parallelismmany thousands of queries per secondtraffic variations
Networking:tiny size of data exchangespossible packet loss around the front-end servers
40 / 43 Gwendal Simon Cloud Engineering
![Page 79: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/79.jpg)
Conclusion
41 / 43 Gwendal Simon Cloud Engineering
![Page 80: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/80.jpg)
Key Challenges
Time-scale:datacenter are expected to last 10 yearsInternet apps gain popularity in weeks
Hardware components:processors are faster and more energy efficientmemory systems and networks are not
Server evolution:more cores mean more parallelism
42 / 43 Gwendal Simon Cloud Engineering
![Page 81: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/81.jpg)
Key Challenges
Time-scale:datacenter are expected to last 10 yearsInternet apps gain popularity in weeks
Hardware components:processors are faster and more energy efficientmemory systems and networks are not
Server evolution:more cores mean more parallelism
42 / 43 Gwendal Simon Cloud Engineering
![Page 82: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/82.jpg)
Key Challenges
Time-scale:datacenter are expected to last 10 yearsInternet apps gain popularity in weeks
Hardware components:processors are faster and more energy efficientmemory systems and networks are not
Server evolution:more cores mean more parallelism
42 / 43 Gwendal Simon Cloud Engineering
![Page 83: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/83.jpg)
Personal Thoughts
A new era in the Internet:the industrial era of applicationscomputer science does really matter
About nano-datacenterusing your always-on devices
43 / 43 Gwendal Simon Cloud Engineering
![Page 84: Cloud Engineering](https://reader033.vdocuments.us/reader033/viewer/2022051817/547ec2f9b379596a2b8b5530/html5/thumbnails/84.jpg)
Personal Thoughts
A new era in the Internet:the industrial era of applicationscomputer science does really matter
About nano-datacenterusing your always-on devices
43 / 43 Gwendal Simon Cloud Engineering