simulation of next generation systems - irisa€¦ · i a model for vm lifecycle management...

Simulation Of Next Generation SystemsSONGS (11 INFR 13)

Mid-term Project Evaluation

Bordeaux (Cepage): L. Eyraud, O. Beaumont, N. Bonichon, F. Mathieu,(Runtime): D. Barthou, B. Goglin, A. Guermouche

Grenoble (Mescal): A. Legrand, D. Kondo, J.-F. Mehaut, J.-M. Vincent

Nancy (Algorille): M. Quinson, L. Nussbaum

Nantes (Ascola): A. Lebre

Nice (Mascotte): O. Dalle, H. Renard

Villeurbanne (CCIN2P3): F. Suter, P.-E. Brinette, F. Desprez

Strasbourg (ICPS): S. Genaud, J. Gossa

ANRNovember 5, 2013. Paris.

Scientific ContextModern Computer Systems

I Grids, P2P, Clouds, HPC, . . .

I Hierarchical, complex and heterogeneous

I Very large and dynamic systems

Challenge: (correctness and) performance of these systems

I Reductionism is not satisfactory; experiences are mandatory

I We thus need Scientific Instruments, just as in physics or other

Idea: Computational Science of Computer Systems

I Computational Science use computers as scientific instruments

I It builds models to understand and conducts simulations to predict

I Can we reuse this approach to understand modern computer systems?2/19

SimGrid and the ANR SONGS project

SimGrid: Simulator of distributed applications

I Infancy (1999): Factorizing the code of some students

I Now: Versatile, extensible, verified predictive power, free and open

I Impact (2008-2012): ≈60 publications, ≈100 authors, 3 PhD

SONGS: Simulation Of Next Generation Systems ANR 11 INFR 13

I Platform project (1.8Me, 400 PM founded)

I 7 academic partners, 20+ researchers (420 PM)

I Modeling large-scale computer systems (+= 2 domains)I Task 1: [Data]GridI Task 2: Peer-to-Peer and Volunteer ComputingI Task 3: IaaS CloudsI Task 4: High Performance Computing

I Simulation methodology (more of our expertise)

I Task 5: Simulation KernelI Task 7: Analysis and Visualization

I Task 6: Concepts and ModelsI Task 8: Experimental Methodology

3/19

Use-Case Driven ResearchI Science pulled by users’ needs, not pushed by abilitiesI Scratch your own itches (more motivating, and leads to better results)I Longer term goal: Foster the emergence of a vivid research community

Work plan in each domainI Tx.1: Add models needed by the planned studies

I Grids: Storage modelingI P2P/VC: Scalable network modeling and ChurnI IaaS Clouds: VMs, hypervisorsI HPC: HPC networks, memory transfers

I Tx.2: Extend APIs to ease the planned studiesI Grids: High Performance Storage System APII P2P/VC: Higher level API such as catalog handlingI IaaS Clouds: Provider side, and client sideI HPC: MPI, OpenMP, Plasma & Magma

I Tx.3: Do planned studiesI Grids: Distributed Data mgmt for LHC; Hierarchical Storage SystemI P2P/VC: Replica Placement in VOD; Affinities in VCI IaaS Clouds: Study from client or provider POV; other metrics (energy)I HPC: Exascale; memory & energy models

4/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 1: [Data]Grids

Initial GoalsI SimGrid was intended to study Grid Scheduling, but without Storage

I Model: storage; APIs: Integrated Storage Servers

I Study: Data mgmt for LHC, Hierarchical Storage Dimensionning (CC-IN2P3)

Achievements during the first half

I An API capturing all relevant concepts is integrated (publication under review)

I Preliminary models proposed (by an associated PhD student at CERN)

Roadmap for the second half

I Validation and improvement of the storage models

I Conduct the planned studies

I Further push the collaboration with the CERN users

5/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 2: Peer-to-Peer/Volunteer Computing

Initial GoalsI Ultra-Scalable Simulation: (achieved) goal of USS-SimGrid ANR project

I Model: Scalable Network Modeling; Churn. APIs: High-level (DHT or similar)

I Study P2P: Replica Placement in VOD; VC: CPU/Network Affinities


I Theoretical study of data broadcast in NATed environments

I Shown the usefulness of dynamic scheduling according to affinities

I Random platform generation; Ability to specify random churn

I Framework to evaluate network tomography algorithms in practice (PlanetLab)

T1(i, j ) T2(i)

T3(j ) T4

j

i

i

j


I New scheduling strategies in BOINC (with U. Berkeley)

I Splay interface on top of SimGrid (with U. Neuchatel)

I Propose & run a large-scale network tomography

6/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 3: IaaS Clouds

Initial GoalsI Virtualization: Adapted framework for provider- and/or client- side studies

I Model: Tasks over VMs, cloud dynamics; APIs: VMs mgmt, AWS (EC2, S3)

I Provider-side Study: VM orchestration and migration in private clouds

I Client-side Study: Decision helper (the best performance at the best price)


I A model for VM lifecycle management including live migration with precopy

I Provider: VM mgmt optim; Client: strategies optimizing cost/makespan

I All-in-one initial modeling of the AWS infrastructure from the client POV

Physical Machine(CapacityC)

First solve

Then solve

VM(X1)

Task(X1,1)

Task(X1,2)

VM(X2)

Task(X2,1)

Task(X3)

VirtualM

achineLayer

PhysicalMachine

Layer

Eq2: X 2,1 < X2

Eq1:X1+X2+X3 < C

Eq2: X1,1 + X1,2 < X1


I Merge back all efforts into the released SimGrid

I (in)validated models for VM interactions/migrations

I Keep up with EC2 billing modeling (self-invalidating)

I Complete client- and provider-sides studies7/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 4: High Performance Computing

Initial GoalsI Challenging domain: high expectations on prediction accuracy, very large scale

I Model: HPC networks, Memory; APIs: MPI, OpenMP, Plasma & Magma

I Study: BigDFT, SPECFEM, MUMPS; Exascale ARM platform


I More MPI coverage (OMPI&MPICH collectives); Online and Offline; Testing

I Hybrid model (fluid+LogOP): Good BigDFT speedup predictions on tibidabo

I Memory modeling is even more challenging than expected ; slowing OpenMP

I Preliminary modeling of StarPU runtime (in collaboration with tool’s authors)


I Mixed online/offline for rapid testing of algorithms

I (in)Validation of StarPU models, more MPI apps

I Dimensionning tibidabo++ platform before building

I New models: IB networks, Memory, GPU, Xeon Phi

8/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 5: Simulation Kernel

Initial GoalsI Efficient simulation: Big enough, Fast enough (going distributed and parallel)

I Standard simulation: Plug SimGrid with other simulation tools with DEVS


I Exploratory work on distributed simulation, but gross performance loss

I Parallel simulation now works, but disappointing performance

I Kernel fully rewritten in C++ to ease the inclusion of users’ models

I Robust and Mature Framework, ready to use

0

10000

20000

30000

40000

0 500000 1e+06 1.5e+06 2e+06

Runnin

g t

ime in s

eco

nds

Number of nodes

PeerSimOverSimSimGrid

SimGrid (parallel) Roadmap for the second half

I Better Parallel and Distributed simulation

I Further ease the users’ models inclusion

I Eased interactions with NS3 and others

9/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 6: Concepts and Models

Initial Goals Grids P2P Clouds HPCPower Consumption • • •Storage Elements • • •

CPU and Memory •High Performance Networks • •Volatility & Dynamicity • • •


I Expertise hub for all modeling work and (in)validation studies

I Energy API and models (in collaboration with A.-C. Orgerie)

I Various models proposed; Network model thoroughly (in)validated

HypothesisHypothesisHypothesis

Experiments Campaign

H1' H2 Hn'

Analysis

Observations

Neglected observation

Sampled


I (in)validation of energy models, and othersThis is a never-ending, resource-hungry task

I Very large-scale wide-area network tomography?

10/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 7: Analysis and Visualization

Initial GoalsI Scalable visualization: fast enough, tractable, and without artifacts

I Trace comparison: to better understand how/why a given alternative is better


I Spatial and temporal aggregation to control artifacts (zooming effects)

I Entropy-based aggregation evaluation, helps selecting best aggregation

I Trace comparison used in (in)validations, but still very naive unfortunately

Clusters

Sites

Hosts

Grid


I Many new ideas on data aggregations to play with

I Trace comparison is slowly maturating

I Our main contributor got a permanent position; temporary slowdown, but new collaboration

11/19

P2PGrids

CloudsHPC

Ana

lysi

s

Models

Open

Science

CoreTask 8: Experimental Methodology

Initial GoalsI Design Of Experiments (DoE): choosing the right experiments to run

I Campaign Management: adapted batch schedulers for 106 jobs, DoE-aware

I Open Science: reproducible results, laying a basis for further works


I DoE: Increased proficiency in statistical tools (R, ggplot, org-mode)Factorial experience design and ANOVA used in several publications

I Campaign Mgmt: more oriented toward Grid’5000, not yet running simulations

I Open Science: our publications are (more) reproducible


I Package simulation experiments for reproducibilityConvenient runner as an incentive to the users

I Tool convergence with emulators and real platforms

I Foster a community on Open Science in our domainStill long way to go toward Reproducible Experiments

12/19

Work DistributionPartner WP1 WP2 WP3 WP4 WP5 WP6 WP7 WP8

Bordeaux • • •• • • •• ••Grenoble • •• • • •• •• •• •• ••

Nancy • • • • • • • • •• • • • • • •Nantes •• • •

Nice • • • • • • ••Strasbourg •• • • • • • •

Villeurbanne •• •• • • • • •

Planned Work Distribution Actual Investment at T24small • ; • large small • ; • large

I WP1: DQ2 team (CERN) on modeling of data storage serversI WP2: BOINC authors (U. Berkeley), Splay authors (U. Neuchatel)I WP3: AIST Japan on VM migrations over WAN; UCMadrid on multi-cloudsI WP4: StarPU authors (Inria Bordeaux), BigDFT authors (CEA INAC)I WP5: A. Giersch (U. Franche-Comte) on efficient simulationI WP6: A.-C. Orgerie (CNRS Rennes) on energy modelingI WP7: L. Schnorr (UFRGS, Brasil) on visualizationI WP8: M. Stillwell (U. Cranfield), S. Hunold (U. Vienna) on Open Science

13/19

Scientific Outcomes and Dissemination

OutcomesI 20 mono-site publications + 4 multi-site publications (many more under review)

I 10 invited talk and keynotes

I 3 major releases of SimGrid (150+ commits/month, 6+ contributors each month)

Dissemination: SimGrid Users’ Days

I 3 days conference gathering (potential) users to exchange news and feedback

I Takes place in remote location where there is nothing else to do

I 2010 and 2012 editions were rather classical (presentations)

I 2013 “working workshop”: hack your own SimGrid project, under our guidance

Efficient Networking

I We are at SuperComputing every year (on Inria booth); COMPAS in France

I Joint lab with Urbana Champain on HPC; Barcelona Supercomputer Center

I Connections to IETF toward an Informal RFC on P2P simulation

I Ongoing discussions for several European projects, Inria IPL, etc.

I Even recent interactions with science philosophers from Finland!14/19

Scientific Zoom: How do we model things?

“Original” epistemological stance

I Things are so complex that reductionism does not work anymore

Computer Systems ≈ Natural SystemsI Empirical measurements, hypothesis, modeling, (in)validation (ad eternam)

HypothesisHypothesisHypothesis

Experiments Campaign

H1' H2 Hn'

Analysis

Observations

Neglected observation

Sampled

Wanted ModelsI Explanatory and interpretable models: We model mainly to understand

I Quantitative: Computation or communication time

I Qualitative: Interactions between streams or between processes (or both)

I Semantic: Search for synchronization bugs15/19

What do we find when doing so?

Such models are possibleMeasurements MPI Send / MPI Recv

Small

Medium1Medium2

Detached

Small

Medium1Medium2

Detached

MPI_Send MPI_Recv

1e−04

1e−02

1e+01 1e+03 1e+05 1e+01 1e+03 1e+05Message size (bytes)

Dur

atio

n (s

econ

ds) group

Small

Medium1

Medium2

Detached

Large

SMPI Model

Asynchronous mode(k ≤ Sa)

T3

Pr

Ps

T1

T2

Detached mode(Sa < k ≤ Sd)

Ps

Pr

T2T4

T1

Synchronous mode(k > Sd)

Ps

Pr

T4 T2

(The SimGrid default model captures these effects, and much more)16/19

And this just works!

I Sweep3D: Simple (but not trivial) application predicted in all detailsI Graphene (16 procs), OpenMPI, TCP, Gigabit Ethernet without overfitting FX :)

17/19

Reality is sometimes . . . surprising

Hardware sucks

I BigDFT on Graphene

I Harware Bug (?)

I Packet drops;Timeouts

TCP sucks

LU

32

p(u

p:r

eal;

dow

nsi

m)

Congestion ; slows downSpeed = 0 ; timeout, and reset

LU

12

8p

We can incorporate these effects in our models (and others)

I But don’t you want to fix reality?

I We were modeling to understand, that’s a huge victory!

18/19

Conclusions on the SONGS project

I Goal: Computational Science of Computer SystemsI Systems are too large, dynamic and complex for a reductionist approachI We need models to understand, and simulations to predict

I Realistic models of Modern Systems (DataGrids, P2P, Clouds & HPC)I Efficient Methodology: Planning, Simulation and Analysis (with Open Science)

The project is Fully on Track

I Work factorization is really effective (; productivity gain)

I Many, many results. In all 8 WP.

I WP4: Sufficient models to predict MPI applicationsI Many dark areas remain, but this is unprecededI Non-trivial but correct predictions; Reality sometimes worse than Simulation :)

I WP8: Open Science opens a brave new world

There is much more to discover during the second half

I If things remain the same, all fixed goals should be reached

I The project is attracting many external contributors

I Are we experiencing the emergence of the vivid research community we need?19/19

Take Away Messages

SimGrid will prove helpful to your research

I Versatile: Used in several communities (scheduling, GridRPC, HPC, P2P, Clouds)

I Accurate: Model limits known thanks to validation studies

I Sound: Easy to use, extensible, fast to execute, scalable to death, well tested

I Open: User-community much larger than contributors group/ GPL120 publications (110 distinct authors, 5 continents), 4 PhD/25+commiters,5+ unaffiliated

I Around since over 10 years, and ready for at least 10 more years

Welcome to the Age of (Sound) Computational Science

I Discover: http://simgrid.gforge.inria.fr/

I Learn: 101 tutorials, user manuals and examples

I Join: user mailing list, #simgrid on irc.debian.org

We even have some open positions ;)

20/19

http://simgrid.gforge.inria.fr/

Other finding of the project: Open Science

The Devil is in the Details vs. Reproducibility Graal

I Experiment description (environment / protocol) not trivial(deluge de donnees)

I Very sensible experiments: impact macro of micro errors

I Statistical post-processing more and more advanced

But that works, too!I Grid’5000 is very precious: hardware, but also know-how

I Our tools (YMMV): git + org-mode + R

I Computational scientists already use them, btw

We just need to convince our community ;)

I I found the results section of this paper to be pretty weak.

I If less accurate models drive the user to the same conclusions(as Fig. 8 indicates), why we need more complex models?

21/19

Invalidating Simulators from the LitteratureNaive flow models documented as wrong

Setting Expected Output OutputB = 100 B = 100

B = 20

B = 100 B = 100

B = 20

B = 100 B = 100

B = 20

Known issue in Narses (2002), OptorSim (2003), GroudSim (2011).

Validation by general agreement“Since SimJava and GridSim have been extensively utilized in conductingcutting edge research in Grid resource management by severalresearchers, bugs that may compromise the validity of the simulationhave been already detected and fixed.” CloudSim, ICPP’09

Setting Expected Output OutputB B B

Buggy flow model (GridSim 5.2, Nov. 25, 2010). Similar issues with naivepacket-level models.

22/19

MapReduce on Grid’5000

I Huge CPU slowdown

I Due to the IDE disksDoes not happen in SATA

Can be modeled, but you have to know

23/19

SimGrid is an Operating Simulator

OS-like internal design, isolating user processes with simcalls

Functional ViewProcess Process Process

SimCall Interface

MaestroSimulation Modelske

rnel

Temporal View

M

U2U1

U3

Implementation

T1tn

T2

tn+1M

Efficient [parallel] simulation

0

10000

20000

30000

40000

0 500000 1e+06 1.5e+06 2e+06

Run

ning

tim

e in

sec

onds

Number of nodes

OverSim (OMNeT++)PeerSim

OverSim (simple underlay)SimGrid (sequential)SimGrid (4 threads)

dPeerSim: 2LP ; 4h / 16LP ; 1h

(but only 47s in sequential PeerSim, and 5s with SimGrid :)

Model-Checking (Safety & Liveness)1

2

iSend

3

iRecv

4

Wait

5

iRecv

1 5 3

Test TRUE

6

Test TRUE

7

iSend

8

Wait

9

iRecv

1 0

Test FALSE

1 1

MC_RANDOM (0)

1 1 5

MC_RANDOM (1)

1 2 2

iRecv

1 2

MC_RANDOM (0)

1 0 5

MC_RANDOM (1) 1 1 2

iRecv

1 3

MC_RANDOM (0)MC_RANDOM (1)

1 0 7

iRecv

MC_RANDOM (0)

1 5

MC_RANDOM (1)

2 4

iRecv

1 6

iSend

1 7

iRecv

1 8

Wait

1 9

Test TRUE

2 0

iSend

2 1

Wait

2 2

iRecv

Test FALSE

2 5

MC_RANDOM (0)

1 0 3

MC_RANDOM (1)

2 6

Test FALSE

2 7

MC_RANDOM (0)

6 6

MC_RANDOM (1)

2 8

MC_RANDOM (0)

6 2

MC_RANDOM (1)

2 9

MC_RANDOM (0)MC_RANDOM (1)

MC_RANDOM (0)

3 1

MC_RANDOM (1)

3 2

iSend

3 5

Test FALSE

3 3

Wait

Test TRUE

3 6

iSend

5 8

MC_RANDOM (0)

6 0

MC_RANDOM (1)

3 7

Wait

5 4

MC_RANDOM (0)

5 6

MC_RANDOM (1)

3 8

MC_RANDOM (0)

5 2

MC_RANDOM (1)

3 9

MC_RANDOM (0)

4 4

MC_RANDOM (1)

4 0

MC_RANDOM (0)

MC_RANDOM (1)

4 1

MC_RANDOM (0) MC_RANDOM (1)

4 2

Test TRUE

iSend

4 5

Test TRUE

4 6

iSend

4 7

Wait

4 8

iRecv

Test FALSE

Test TRUE

Wait Wait

iSend iSend

6 3

Test FALSE

Test FALSE

6 7

iSend

7 5

Test FALSE

6 8

Wait

6 9

Test TRUE

7 0

iSend

7 1

Wait

7 2

iSend

7 3

iRecv

Test FALSE

7 6

iSend

9 9

MC_RANDOM (0)

1 0 1

MC_RANDOM (1)

7 7

Wait

9 5

MC_RANDOM (0)

9 7

MC_RANDOM (1)

7 8

MC_RANDOM (0)

9 3

MC_RANDOM (1)

7 9

MC_RANDOM (0)

8 4

MC_RANDOM (1)

8 0

MC_RANDOM (0)

MC_RANDOM (1)

8 1


8 2

Test TRUE

iSend

8 5

Test TRUE

8 6

iSend

8 7

Wait

8 8

iSend

8 9

iRecv

Test FALSE

Test TRUE

Wait Wait

iSend iSend

iSend

Test FALSE

MC_RANDOM (0)

1 0 9

MC_RANDOM (1)

Test FALSE

MC_RANDOM (0)

MC_RANDOM (1)

1 1 6

iSend

1 1 7

iRecv

1 1 8

Wait

1 1 9

Test TRUE

1 2 0

iSend

Wait

MC_RANDOM (0)

1 2 4

MC_RANDOM (1)

1 2 5

iSend

1 2 8

Test FALSE

1 2 6

Wait

Test TRUE

1 2 9

iSend

1 4 9

MC_RANDOM (0)

1 5 1

MC_RANDOM (1)

1 3 0

Wait

1 4 5

MC_RANDOM (0)

1 4 7

MC_RANDOM (1)

1 3 1

MC_RANDOM (0)

1 4 3

MC_RANDOM (1)

1 3 2

MC_RANDOM (0)

1 3 7

MC_RANDOM (1)

1 3 3

MC_RANDOM (0)

MC_RANDOM (1)

1 3 4


1 3 5

Test TRUE

iSend

1 3 8

Test TRUE

1 3 9

iSend

Wait

Test TRUE

Wait Wait

iSend iSend

iRecv

Exhaustive Chord(2 processes)

I Aims at bug finding,not assessment

I System State Equality

I + DPOR Reduction

I Soon more parallelism

I Soon statistical MC

24/19

Toward an Integrated Scientific Workflow1. Prepare the experimental scenarios

2. Launch thousands of simulations

3. Post-processing and result analysisVisualization

Statistics

Textual logsApplicationUser

ParametersInput

SettingsExperimental

SimulatorScenario Outputs

SimulationKernel

Post-processing through Vizualization

time slice

time slice time slicetime slice

1st Space Aggregation 2nd Space AggregationGroupA

GroupB

Platform and Workload GenerationTopology

ModelTopological

Graph

Generation Promotion

Real WorldPlatform

PlatformBlank

Selection

Export

Graph

OtherOutputs

Setting

XML

Initial

PlatformSynthetic Experimental

OR

Description

Labelling

Extraction

Description

Cluster

Simulated

Execution

Time

Traces

Indepentdent

Replay

Inst

rum

enta

tio

n

Site 1 Site2 pro

cess

0.t

race

Execution

Traces

pro

cess

1.t

race

Instrumented

VersionApplication

SG_process0.trace

SG_process1.trace

SG_processN.trace

pro

cess

N.t

race

Execution

25/19

simulation of next generation systems - irisa€¦ · i a model for vm lifecycle management...

Documents