score-p and scalasca and scalasca... · portable open-source tools for scalable performance...

Score-P and ScalascaPortable open-source tools for scalableperformance analysis

February 1, 2014 Alexandre Otto Strube

Outline

Going Exascale

Scalasca

We’re not alone

Things got messy

Unification

Who uses/develops Score-P

What is ours

Extreme scalability

The future

February 1, 2014 Alexandre Otto Strube Slide 2

Going Exascale

Single core perfomance peaking

# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasing

Hybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasingHybrid environments

That affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOW

HPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearhead

We only find the problems before the othersSupercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the others

Supercomputers of today → notebooks of tomorrow

Single core perfomance peaking# of cores increasingHybrid environmentsThat affects YOU - TODAY - RIGHT NOWHPC is just the spearheadWe only find the problems before the othersSupercomputers of today → notebooks of tomorrow

It doesn’t get easier

Increasing machine complexity (gpu, accelerators, etc)

Every doubling of scale reveals a new bottleneckPerturbation and data volumeDrawing insight from measurements

Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneck

Perturbation and data volumeDrawing insight from measurements

Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneckPerturbation and data volume

Drawing insight from measurements

Increasing machine complexity (gpu, accelerators, etc)Every doubling of scale reveals a new bottleneckPerturbation and data volumeDrawing insight from measurements

Example: Sweep3d Wait States on BG/P (2010)

This is an old song

Several performance tools exist, for many years

Most cease to work in huge processor/core countsKOJAK performance tool was created 16 years ago.

This is an old song

Several performance tools exist, for many yearsMost cease to work in huge processor/core counts

KOJAK performance tool was created 16 years ago.

This is an old song

Several performance tools exist, for many yearsMost cease to work in huge processor/core countsKOJAK performance tool was created 16 years ago.

Scalasca

Started in 2006 (following KOJAK from ’98)

Goals:

Scalable performance analysis toolsetSpecifically targeting large-scale parallel applications such asthose running on IBM Blue Gene or Cray XT with 10,000s or100,000s of processes

Scalasca

Started in 2006 (following KOJAK from ’98)Goals:

Scalasca

Scalable performance analysis toolset

Specifically targeting large-scale parallel applications such asthose running on IBM Blue Gene or Cray XT with 10,000s or100,000s of processes

Scalasca

Scalasca: Features

Open source (New BSD license)

PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages

Fortran, C, C++MPI 2.2, basic OpenMP & hybrid MPI+OpenMP

Unique:

scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL

Scalasca: Features

Open source (New BSD license)Portable

IBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages

Unique:

Scalasca: Features

Open source (New BSD license)PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10

Supports common parallel programming paradigms &languages

Unique:

Scalasca: Features

Open source (New BSD license)PortableIBM Blue Gene, Cray XT, SGI Altix, IBM SP, blade clusters,Solaris, Linux clusters, NEC SX, K Computer, Fujitsu FX10Supports common parallel programming paradigms &languages

Unique:

Scalasca: Features

Fortran, C, C++

MPI 2.2, basic OpenMP & hybrid MPI+OpenMPUnique:

Scalasca: Features

Unique:

Scalasca: Features

Unique:

Scalasca: Features

Unique:scalable trace analysis

Automatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL

Scalasca: Features

Unique:scalable trace analysisAutomatic wait-state search

Parallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL

Scalasca: Features

Unique:scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalability

INSIGHTFUL

Scalasca: Features

Unique:scalable trace analysisAutomatic wait-state searchParallel replay exploits memory & processors to deliverscalabilityINSIGHTFUL

This looks understandable...

... but this is a real code.

and this.

... it can get really confusing.

Scalasca

We’re not alone

Several tools exist

Different goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needs

Separate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formats

Complementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionality

Redundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenance

Limited or expensive interoperabilityComplications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperability

Complications for user experience, support, training

We’re not alone

Several tools existDifferent goals, similar needsSeparate measurement systems and output formatsComplementary features and overlapping functionalityRedundant effort for development and maintenanceLimited or expensive interoperabilityComplications for user experience, support, training

Things got messy

Unification

Score-P project idea

Communnity project with common infrastructure

What we share:

Single instrumentation and measurement systemCommon data formats: Open Trace Format 2 (OTF2) fortracesPerformance report: Cube4

Single development effort, testing, supportSingle installation, interoperability, etc

So, Score-P is the base instrumentation/measurement forseveral projects

Communnity project with common infrastructureWhat we share:

Single instrumentation and measurement system

Common data formats: Open Trace Format 2 (OTF2) fortracesPerformance report: Cube4

Single instrumentation and measurement systemCommon data formats: Open Trace Format 2 (OTF2) fortraces

Performance report: Cube4

Single development effort, testing, support

Single installation, interoperability, etc

Who uses/develops Score-P?

Scalasca (Fz-Juelich, RTWH Aachen)

Vampir (TU Dresden)Periscope (Tu Munich)Tau (U. Oregon)

Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)

Periscope (Tu Munich)Tau (U. Oregon)

Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)Periscope (Tu Munich)

Tau (U. Oregon)

Scalasca (Fz-Juelich, RTWH Aachen)Vampir (TU Dresden)Periscope (Tu Munich)Tau (U. Oregon)

And why we did it?

Cleaning the house

What do we measure?

Measurement of MPI, OpenMP, User-level functions

Generation of flat MPI profiles

Only relinkingMinimum overheadTimes each function was calledTime spent in each functionAmount of data transferred

Call-path profiles

Needs recompilationSome overhead - might need filtering

Trace analysis

Identifies inneficiency patterns in communication andsynchronizationTraces can quickly get huge - better filter that

What do we measure?

Measurement of MPI, OpenMP, User-level functionsGeneration of flat MPI profiles

Call-path profiles

Trace analysis

What do we measure?

Only relinking

Minimum overheadTimes each function was calledTime spent in each functionAmount of data transferred

Call-path profiles

Trace analysis

What do we measure?

Only relinkingMinimum overhead

Times each function was calledTime spent in each functionAmount of data transferred

Call-path profiles

Trace analysis

What do we measure?

Only relinkingMinimum overheadTimes each function was called

Time spent in each functionAmount of data transferred

Call-path profiles

Trace analysis

What do we measure?

Only relinkingMinimum overheadTimes each function was calledTime spent in each function

Amount of data transferredCall-path profiles

Trace analysis

What do we measure?

Call-path profiles

Trace analysis

What do we measure?

Call-path profiles

Trace analysis

What do we measure?

Call-path profilesNeeds recompilation

Some overhead - might need filteringTrace analysis

What do we measure?

Call-path profilesNeeds recompilationSome overhead - might need filtering

Trace analysis

What do we measure?

Trace analysis

What do we measure?

Trace analysisIdentifies inneficiency patterns in communication andsynchronization

Traces can quickly get huge - better filter that

What do we measure?

Trace analysisIdentifies inneficiency patterns in communication andsynchronizationTraces can quickly get huge - better filter that

Extreme scalability

All parallel:

Data collection/reduction

Analysis:

Pattern searchDelay analysisCritical-path analysis

Visualization

Extreme scalability

All parallel:

Data collection/reductionAnalysis:

Visualization

Extreme scalability

All parallel:

Pattern search

Delay analysisCritical-path analysis

Visualization

Extreme scalability

All parallel:

Pattern searchDelay analysis

Critical-path analysis

Visualization

Extreme scalability

All parallel:

Visualization

Extreme scalability

All parallel:

Visualization

Some MPI patterns

ENTER EXIT SEND RECV COLLEXIT

(a) Late Sendertime

(b) Late Receiver

(d) Wait at N x Ntime

(c) Late Sender / Wrong Order

Late sender

Late sender and application topology

Direct wait time analysis

C Recv

bar Recv

Direct wait

Indirect wait time analysis

bar Recv

Direct waitIndirect wait

Direct wait time

Indirect wait time analysis

Root cause analysis

C Recv

bar Recv

barDELAY

Direct waitIndirect wait

6D Hardware topology

The Future

Energy awareness

Bring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication developers!support@score-p.orghttp://www.scalasca.org

The Future

Energy awarenessBring performance analysis to YOU!

There’s a bunch of experts craving for users and parallelapplication developers!support@score-p.orghttp://www.scalasca.org

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication developers!

support@score-p.orghttp://www.scalasca.org

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication developers!support@score-p.org

http://www.scalasca.org

The Future

Energy awarenessBring performance analysis to YOU!There’s a bunch of experts craving for users and parallelapplication developers!support@score-p.orghttp://www.scalasca.org

score-p and scalasca and scalasca... · portable open-source tools for scalable performance...

Documents

exascale computing - hlrs.de · exascale computing 4...

sid optimization studies jan strube pacific northwest...

exascale radio astronomy

age of exascale

sos 14 challenges in exascale computingchallenges in...

the path to exascale computing -...

exascale computing research center (ecr) · exascale...

architectures for exascale computing

kojak and scalasca

adjusting to exascale computing

towards exascale direct numerical simulations of...

fusion energy sciences exascale requirements...

an evolutionary exascale programming model deserves ... ·...

scalasca: a scalable portable integrated performance

pflotran with scalasca - cug

waiting for exascale

heterogeneous exascale computing

mainissues - exascale

major challenges to achieve exascale performance · 1 major...

reference exascale - process project · process...