score-p and scalasca and scalasca... · portable open-source tools for scalable performance...

Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Score-P and ScalascaPortable open-source tools for scalableperformance analysis

February 1, 2014 Alexandre Otto Strube

Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Outline

Going Exascale

Scalasca

We’re not alone

Things got messy

Unification

Who uses/develops Score-P

What is ours

Extreme scalability

The future

February 1, 2014 Alexandre Otto Strube Slide 2

Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Going Exascale


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking

# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasing

Hybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasingHybrid environments

That affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOW

HPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearhead

We only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the others

Supercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

TL;DR

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

It doesn’t get easier

Increasing machine complexity (gpu, accelerators, etc)

Every doubling of scale reveals a new bottleneckPerturbation and data volumeDrawing insight from measurements


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneck

Perturbation and data volumeDrawing insight from measurements


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneckPerturbation and data volume

Drawing insight from measurements


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneckPerturbation and data volumeDrawing insight from measurements


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Example: Sweep3d Wait States on BG/P (2010)


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

This is an old song

Several performance tools exist, for many years

Most cease to work in huge processor/core countsKOJAK performance tool was created 16 years ago.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

This is an old song

Several performance tools exist, for many yearsMost cease to work in huge processor/core counts

KOJAK performance tool was created 16 years ago.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

This is an old song

Several performance tools exist, for many yearsMost cease to work in huge processor/core countsKOJAK performance tool was created 16 years ago.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca

Started in 2006 (following KOJAK from ’98)

Goals:

Scalable performance analysis toolsetSpecifically targeting large-scale parallel applications such asthose running on IBM Blue Gene or Cray XT with 10,000s or100,000s of processes


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca

Started in 2006 (following KOJAK from ’98)Goals:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca


Scalable performance analysis toolset

Specifically targeting large-scale parallel applications such asthose running on IBM Blue Gene or Cray XT with 10,000s or100,000s of processes


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca




Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features

Open source (New BSD license)

PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages

Fortran, C, C++MPI 2.2, basic OpenMP & hybrid MPI+OpenMP

Unique:

scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features

Open source (New BSD license)Portable

IBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages


Unique:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features

Open source (New BSD license)PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10

Supports common parallel programming paradigms &languages


Unique:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features

Open source (New BSD license)PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages


Unique:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features


Fortran, C, C++

MPI 2.2, basic OpenMP & hybrid MPI+OpenMPUnique:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features



Unique:



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features



Unique:scalable trace analysis

Automatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features



Unique:scalable trace analysisAutomatic wait-state search

Parallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features



Unique:scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalability

INSIGHTFUL


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca: Features



Unique:scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

This looks understandable...


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

... but this is a real code.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

and this.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

... it can get really confusing.


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Scalasca


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools exist

Different goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needs

Separate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formats

Complementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionality

Redundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenance

Limited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperability

Complications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Things got messy


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Unification


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Score-P project idea

Communnity project with common infrastructure

What we share:

Single instrumentation and measurement systemCommon data formats: Open Trace Format 2 (OTF2) fortracesPerformance report: Cube4

Single development effort, testing, supportSingle installation, interoperability, etc

So, Score-P is the base instrumentation/measurement forseveral projects


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Communnity project with common infrastructureWhat we share:





Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n



Single instrumentation and measurement system

Common data formats: Open Trace Format 2 (OTF2) fortracesPerformance report: Cube4




Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n



Single instrumentation and measurement systemCommon data formats: Open Trace Format 2 (OTF2) fortraces

Performance report: Cube4




Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n







Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n




Single development effort, testing, support

Single installation, interoperability, etc



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n







Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Who uses/develops Score-P?

Scalasca (Fz-Juelich, RTWH Aachen)

Vampir (TU Dresden)Periscope (Tu Munich)Tau (U. Oregon)


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)

Periscope (Tu Munich)Tau (U. Oregon)


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)Periscope (Tu Munich)

Tau (U. Oregon)


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n


Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)Periscope (Tu Munich)Tau (U. Oregon)


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

And why we did it?


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Cleaning the house


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?

Measurement of MPI, OpenMP, User-level functions

Generation of flat MPI profiles

Only relinkingMinimum overheadTimes each function was calledTime spent in each functionAmount of data transferred

Call-path profiles

Needs recompilationSome overhead - might need filtering

Trace analysis

Identifies inneficiency patterns in communication andsynchronizationTraces can quickly get huge - better filter that


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?

Measurement of MPI, OpenMP, User-level functionsGeneration of flat MPI profiles


Call-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?


Only relinking

Minimum overheadTimes each function was calledTime spent in each functionAmount of data transferred

Call-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?


Only relinkingMinimum overhead

Times each function was calledTime spent in each functionAmount of data transferred

Call-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?


Only relinkingMinimum overheadTimes each function was called

Time spent in each functionAmount of data transferred

Call-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?


Only relinkingMinimum overheadTimes each function was calledTime spent in each function

Amount of data transferredCall-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?



Call-path profiles


Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?



Call-path profilesNeeds recompilation

Some overhead - might need filteringTrace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?



Call-path profilesNeeds recompilationSome overhead - might need filtering

Trace analysis



Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?




Trace analysisIdentifies inneficiency patterns in communication andsynchronization

Traces can quickly get huge - better filter that


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

What do we measure?




Trace analysisIdentifies inneficiency patterns in communication andsynchronizationTraces can quickly get huge - better filter that


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Extreme scalability

All parallel:

Data collection/reduction

Analysis:

Pattern searchDelay analysisCritical-path analysis

Visualization


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Extreme scalability

All parallel:

Data collection/reductionAnalysis:


Visualization


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Extreme scalability

All parallel:


Pattern search

Delay analysisCritical-path analysis

Visualization


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Extreme scalability

All parallel:


Pattern searchDelay analysis

Critical-path analysis

Visualization


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Extreme scalability

All parallel:



Visualization


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Some MPI patterns

time

pro

cess

ENTER EXIT SEND RECV COLLEXIT

(a) Late Sendertime

pro

cess

(b) Late Receiver

time

pro

cess

(d) Wait at N x Ntime

pro

cess

(c) Late Sender / Wrong Order


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Late sender


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Late sender and application topology


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Direct wait time analysis

A

B

C Recv

Send

Send

foo

foo

foo

bar

bar Recv

bar

time

Recv

Recv

Direct wait


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Indirect wait time analysis

Recv

Send

Send

foo

foo

foo

bar

bar Recv

A

B

C

bar

time

Recv

Direct waitIndirect wait

Recv


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Direct wait time


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Indirect wait time analysis


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

Root cause analysis

A

B

C Recv

Send

Send

foo

foo

foo

bar

bar Recv

barDELAY

time

Recv

Recv

Direct waitIndirect wait

Recv

cause


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

6D Hardware topology


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future

Energy awareness

Bring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication [email protected]://www.scalasca.org


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future

Energy awarenessBring performance analysis to YOU!

There’s a bunch of experts craving for users and parallelapplication [email protected]://www.scalasca.org


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication developers!

[email protected]://www.scalasca.org


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication [email protected]

http://www.scalasca.org


Mem

bero

fthe

Hel

mho

ltz-A

ssoc

iatio

n

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication [email protected]://www.scalasca.org


score-p and scalasca and scalasca... · portable open-source tools for scalable performance...

Documents