spec benchmarks for large systems
DESCRIPTION
SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group [email protected]. What is SPEC?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/1.jpg)
Matthias Müller Höchstleistungsrechenzentrum Stuttgart
SPEC Benchmarks for Large Systems
Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann,
R. Lieberman, M. van Waveren, and B. Whitney
SPEC High Performance Group
![Page 2: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/2.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
What is SPEC?
The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.
![Page 3: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/3.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC High-Performance Group
• Founded in 1994• Mission: To establish, maintain, and endorse a
suite of benchmarks that are representative of real-world high-performance computing applications.
• SPEC/HPG includes members from both industry and academia.
• Benchmark products:– SPEC OMP (OMPM2001, OMPL2001)– SPEC HPC2002 released at SC 2002
![Page 4: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/4.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Currently active SPEC HPG Members
• Fujitsu• HP• IBM• Intel• SGI• SUN• UNISYS• University of Purdue• University of Stuttgart
![Page 5: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/5.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Where is SPEC Relative to Other Benchmarks ?
There are many metrics, each one has its purpose
Raw machine performance: Tflops
Microbenchmarks: Stream
Algorithmic benchmarks: Linpack
Compact Apps/Kernels: NAS benchmarks
Application Suites: SPEC
User-specific applications: Custom benchmarks
![Page 6: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/6.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Comparison of different benchmark classes
coverage relevance Identify problems
Time evolution
portability
Micro 0 0 ++ + +
Algorithmic - 0 + ++ +
Kernels 0 0 + + +
SPEC + + + + +
Apps - ++ 0 0 0
![Page 7: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/7.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP
• Benchmark suite developed by SPEC HPG (High Performance Group)
• Benchmark suite for performance testing of shared memory processor systems
• Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates
![Page 8: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/8.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP/CPU2000 differences
• Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000
• Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium)
• Focus on SMP systems, and issued by HPG• SPEC OMP based on work for CPU2000, SPEC
OMP mixes integer and FP in one suite• OMPM is focused on 4-way to 16-way systems• OMPL is targeting 32-way and larger systems
![Page 9: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/9.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMP Results
• 66 submitted results for OMPM• 24 submitted results for OMPL
Vendor HP HP SUN SGI
Architecture Superdome Superdome Fire 15K O3800
CPU PA-8700+ Itanium2 UltraSPARC III
R12000
Speed 875 1500 1200 400
L1 Inst 0.75 MB 16 KB 32 KB 32 KB
L1 Data 1.5 MB 16 KB 64 KB 32 KB
L2 - 256 KB 8 MB 8 MB
L3 - 6144 KB - -
![Page 10: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/10.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Benchmark with scaling to 128
![Page 11: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/11.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Superlinear scaling of applu
![Page 12: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/12.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC OMPL Results: Benchmark with scaling to 64
![Page 13: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/13.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Benchmark
• Full Application benchmarks(including I/O) targeted at HPC platforms
• Serial and parallel (OpenMP and/or MPI)• Currently three applications:
– SPECenv: weather forecast – SPECseis: seismic processing, used
in the search for oil and gas– SPECchem: comp. chemistry, used
in chemical and pharmaceutical industries (gamess)
• All codes include several data sizes
![Page 14: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/14.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECenv execution models on a Sun Fire 6800
Medium scales betterOpenMP best for small sizeMPI best for medium size
![Page 15: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/15.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECenv scaling
![Page 16: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/16.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECseis execution models on a Sun Fire 6800
Medium scales betterOpenMP scales better than
MPI
![Page 17: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/17.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECseis scaling
![Page 18: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/18.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPECchem execution models on a Sun Fire 6800
Medium shows better scalability
MPI is better than OpenMP
![Page 19: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/19.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
SPEC HPC2002 Results: SPECchem scaling
![Page 20: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/20.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Hybrid Execution for SPECchem
![Page 21: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/21.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Current and Future Work
• SPEC HPC:– Update of SPECchem– Improving portability, including tools– Larger datasets
• New release of SPEC OMP:– Inclusion of alternative sources– Merge OMPM and OMPL on one CD
• Adoption of new benchmark codes:– Should represent a type of computation that is
regularly performed on HPC systems– We currently examine CPU2004 for candidates
![Page 22: SPEC Benchmarks for Large Systems](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814793550346895db4c4ca/html5/thumbnails/22.jpg)
Matthias MüllerHöchstleistungsrechenzentrum Stuttgart
Conclusion and Summary
• Results of OMPL and HPC2002:– Scalability of many programs to 128 CPUs
• Best choice of programming model (MPI,OpenMP, hybrid) depends on:– Hardware– Program– Data set size
• SPEC HPC will continue to update and improve the benchmark suites.