April 22, 20231

GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids

CCGrid 2006

A. Iosup, D.H.J. EpemaPDS Group, ST/EWI, TU Delft

April 22, 20232

Outline

• Introduction and Motivation• The GrenchMark Framework• Past and Current Experience with GrenchMark • A GrenchMark Success Story• Future Work• Conclusions

April 22, 20233

The Generic Problem of Analyzing, Testing, and Comparing Grids

• Use cases for automatically analyzing, testing, and comparing Grids• Comparisons for system design and procurement • Functionality testing and system tuning• Performance testing/analysis of grid applications• …

• For grids, this problem is hard !• Testing in real environments is difficult• Grids change rapidly• Validity of tests• …

April 22, 20234

A Generic Solution to Analyzing, Testing, and Comparing Grids• “ Generate and run synthetic grid workloads,

based on real and synthetic applications “

• Current alternatives (not covering all problems)• Benchmarking with real/synthetic applications

(representative?) • User-defined test management (statistically sound?)

• Advantages of using synthetic grid workloads• Statistically sound composition of benchmarks• Statistically sound test management• Generic: cover the use cases’ broad spectrum (to be shown)

April 22, 20236

GrenchMark: a Framework for Analyzing, Testing, and Comparing grids• What’s in a name?

grid benchmark → working towards a generic tool for the whole community: help standardizing the testing procedures, but benchmarks are too early; we use synthetic grid workloads instead

• What’s it about?A systematic approach to analyzing, testing, and comparing grid settings, based on synthetic workloads• A set of metrics for analyzing grid settings• A set of representative grid applications

• Both real and synthetic• Easy-to-use tools to create synthetic grid workloads• Flexible, extensible framework

April 22, 20237

GrenchMark: Iterative Research Roadmap

April 22, 20238

GrenchMark: Iterative Research RoadmapSimple functional systemA.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema,

Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005.

April 22, 20239

GrenchMark: Iterative Research Roadmap

Open-GrenchMark

CommunityEffort

Complex extensible systemThis work

April 22, 202311

GrenchMark Overview: Easy to Generate and Run Synthetic Workloads

April 22, 202312

… but More Complicated Than You Think• Workload structure

• User-defined and statistical models • Dynamic jobs arrival• Burstiness and self-similarity• Feedback, background load• Machine usage assumptions• Users, VOs

• Metrics• A(W) Run/Wait/Resp. Time • Efficiency, MakeSpan• Failure rate [!]

• (Grid) notions• Co-allocation, interactive jobs, malleable, moldable, …

• Measurement methods• Long workloads• Saturated / non-saturated system• Start-up, production, and cool-down scenarios• Scaling workload to system

• Applications• Synthetic• Real

• Workload definition language

• Base language layer• Extended language layer

• Other• Can use the same workload for both simulations and real environments

April 22, 202313

GrenchMark Overview: Unitary and Composite Applications

• Composite applications• Bag of tasks• Chain of jobs• Direct Acyclic Graph-based

(Standard Task Graph Archive)

• Unitary applications• sequential, MPI, Java RMI, Ibis, …

April 22, 202314

GrenchMark Overview: Workload Description Files

• Format:

Combining four workloads into oneNumber of jobsComposition and application

typesCo-allocation and number of components

Inter-arrival and start time

Language extensions

April 22, 202315

Using GrenchMark: Grid System Analysis

• Performance testing: test the performance of an application (for sequential, MPI, Ibis applications)• Report runtimes, waiting times, grid middleware

overhead• Automatic results analysis

• What-if analysis: evaluate potential situations• System change• Grid inter-operability• Special situations: spikes in demand

April 22, 202316

Using GrenchMark: Functionality Testing in Grid Environments• System functionality testing: show the ability

of the system to run various types of applications• Report failure rates

[ arguably, functionality in grids is even more important than performance ! 10% job failure rate in a controlled system like the DAS ]

• Periodic system testing: evaluate the current state of the grid• Replay workloads

April 22, 202317

Using GrenchMark: Comparing Grid Settings

• Single-site vs. co-allocated jobs: compare the success rate of single-site and co-allocated jobs, in a system without reservation capabilities• Single-site jobs 20% better vs. small co-allocated jobs (<32

CPUs), 30% better vs. large co-allocated jobs [setting and workload-dependent !]

• Unitary vs. composite jobs: compare the success rate of unitary and composite jobs, with and without failure handling mechanisms• Both 100% with simple retry mechanism

[setting and workload-dependent !]

April 22, 202318

A GrenchMark Success Story:Releasing the Koala Grid Scheduler on the DAS• Koala [ http://www.st.ewi.tudelft.nl/koala/ ]

• Grid Scheduler with co-allocation capabilities• DAS: The Dutch Grid, ~200 researchers• Initially

• Koala, a tested (!) scheduler, pre-release version• Test specifics

• 3 different job submission modules• Workloads with different jobs requirements,

inter-arrival rates, co-allocated v. single site jobs…• Evaluate: job success rate, Koala overhead and bottlenecks

• Results• 5,000+ jobs successfully run (all workloads); functionality

tests• 2 major bugs first day, 10+ bugs overall (all fixed)• KOALA is now officially released on the DAS

(full credit to KOALA developers, 10x for testing with GrenchMark)

April 22, 202319

A.Iosup, D.H.J.Epema (TU Delft), C. Franke, A. Papaspyrou, L. Schley, B. Song, R. Yahyapour (U Dortmund), On modeling synthetic workloads for Grid performance evaluation, 12th Workshop on Job Scheduling Strategies for Parallel Processing (JSSPP), held in conjunction with SIGMETRICS 2006, Saint Malo, France, June 2006 (accepted).

GrenchMark’s Current Status: pre-”Open-GrenchMark”• Already done in Python [http://www.python.org]• Workload Generator• Generic Workload Submitter (Koala, Globus GRAM,

option to extend for JSDL, Condor, PBS, LSF, SGE, …)

• Applications• Unitary, 3 types: sequential, MPI, Ibis (Java)• +35 real and synthetic applications• Composite applications: DAG-based

• Extending modeling capabilities

April 22, 202320

A. Iosup, C. Dumitrescu, D.H.J. Epema (TU Delft), H. Li, L. Wolters (U Leiden), How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications, (submitted).

Towards Open-GrenchMark: Grid traces, Simulators, Benchmarks

• Distributed testing• Integrate with DiPerF (C. Dumitrescu, I. Raicu, M. Ripeanu)

• Grid traces analysis• Automatic tools for grid traces analysis

• Use in conjunction with simulators• Ability to generate workloads which can be used in

simulated environments (e.g., GangSim, GridSim, …)• Grid benchmarks

• Analyze the requirements for domain-specific grid benchmarks

April 22, 202321

Conclusion

• GrenchMark generates diverse grid workloadseasy-to-use, flexible, portable, extensible, …

• Experienceused GrenchMark to test KOALA’s functionality and performance. used GrenchMark to analyze, test, and compare grid settings. 15,000+ jobs generated and run … and counting.

• (more) advertisementHave specific grid setting you would like to test? Test with GrenchMark!

April 22, 202322

Thank you!

Questions? Remarks? Observations? All welcome!

GrenchMark http://grenchmark.st.ewi.tudelft.nl/ [10x Paulo]

Alexandru IOSUPTU Delft

A.Iosup@tudelft.nlhttp://www.pds.ewi.tudelft.nl/~iosup/index.html [google: “iosup”]

Many thanks to Hashim Mohamed (Koala), Jason Maassen and Rob van Nieuwpoort (Ibis).