high performance computing : models, methods, & means introduction to libraries 1

CSC 7600 Lecture 19 : HPC Libraries 1Spring 2011

HIGH PERFORMANCE COMPUTING: MODELS, METHODS, & MEANS

INTRODUCTION TO LIBRARIES 1

Prof. Thomas SterlingDr. Hartmut Kaiser Department of Computer ScienceLouisiana State UniversityApril 7th , 2011

1


Outline

• Why libraries• What is a library• How to use a library• Standard library support• Linear Algebra Library : BLAS• Summary - Materials for Test

2


What's the expected output for the following program and why? Hint: What's the effect of the continue statement in a do-while loop?

enum { false = 0, true = 1 };

int main() { int i = 1; do { printf("%d\n",i); if(++i < 15) continue; } while (false); return 0; }

Puzzle of the Day

3References: [3]


Outline

• Why libraries• What is a library• How to use a library• Standard library support• Application domains• Linear Algebra Library: BLAS• Summary - Materials for Test

4


Why libraries?

• How can we dare to assume to be able to write correct code?

• Reuse, reuse, reuse!– Allows to concentrate on the science– Leverage knowledge and skills of others– Offload part of your work to library maintainers

• But: used libraries should be – High quality– Flexible and generic– Combinable– Preferrably have access to source code

• Use the right tool for the right job– Having a new and shiny hammer doesn‘t mean everything

is a nail

5


Outline


6


What is a library?

• Short history of software libraries• Different perspectives• Classification of libraries

7


Bouchons Loom (1725)

Basile Bouchon, Jean Falcon, Jacques Vaucanson, Joseph Marie Jacquard (1801):

An automated loom that transformed the 18th century textile industry and became the inspiration for future calculating and tabulating machines.

The binary principle embodied in the punched-card operation of the loom was inspiration for the data processing machines to come.

Picture of Jacquard loom (1830)

8

Short history of software libraries

References: [1]



9

Charles Babbage’s Analytical Machine (1830)

Every set of cards made for any formula will at any future time recalculate that formula with whatever constants may be required. Thus the Analytical Engine will possess a library of its own. Every set of cards once made will at any future time reproduce the calculations for which it was first arranged.

— Passages from the Life of a Philosopher, Charles Babbage (1864)References: [1]



Harvard Mark I: Grace Murray Hopper and Howard Aiken (1944)

Some sequences that were used again and again were permanently wired into the Mark I’s circuits... Since the Mark I was not a stored-program computer, Hopper had no choice for other sequences than to code the same pattern in successive pieces of tape. It did not take long for her to realize that if a way could be found to reuse the pieces of tape already coded for another problem, a lot of effort would be saved. The Mark I did not allow that to be easily done, but the idea had taken root and later modifications did permit multiple tape loops to be mounted.

10References: [1]



Alan Turing and the ACE (1946)

Subroutine, call stack, jump and return: When we wish to start on a subsidiary operation [subroutine] we need only make a note of where we left off the major operation [return address] and then apply the first instruction of the subsidiary. When the subsidiary is over we look up the note and continue with the major operation. Each subsidiary operation can end with instructions for this recovery of the note. How is the burying and disinterring [push and pop] of the note to be done? There are of course many ways. One is to keep a list of these notes in one or more standard size delay lines (1024), with the most recent last [a stack]... the burying being done through a standard instruction table BURY, and the disinterring by the table UNBURY.

— Proposals for the development in the Mathematics Division of an Automatic Computing Engine (ACE), Alan M. Turing (1946)

11References: [1]


Calling subroutines

• Hardware support for stack operations (LIFO: last in first out)

• Special stack pointer or general register

– Call:• Put parameters on top of

stack• Put return address on top of

stack• Jump to subroutine

– Return• Retrieve address from top of

stack• Jump to this address

• Modern compilers use stack for local data as well

12

Grows downwards


FORTRAN• Parameters put on stack from

left to right by reference• Called code is responsible for

cleaning parameters from stack (unwinding)

• Local data on stack is handled by called code

• Caller and callee must agree on number of arguments

• Names are not changed(sometimes all capital)

• Modern compilers have various options

C• Parameters put on stack from

right to left by value• Calling code is responsible for

cleaning parameters from stack (unwinding)

• Local data on stack is handled by called code

• Subroutines may have variable parameter count (printf)

• Prepended ‘_‘ for names(sometimes appended)

13

Calling conventions



EDSAC - Electronic Delay Storage Automatic Calculator (1951)

“The library of tapes on which subroutines are punched is contained in the steel cabinet shown on the left. The operator is punching a program tape on a keyboard perforator. She can copy mechanically tapes taken from the library on to the tape she is preparing by placing them in the tape reader shown in the center of the photograph.”

14References: [1]



• Key ideas of the EDSAC library (David Wheeler, Maurice Wilkes)

– Library of subroutines– Reuse of reliable components to shorten development time and

reduce defects– Linking, relocatable objects– Multiple versions of a subroutine, each with a clearly indicated

tradeoff of time, space, accuracy– Unit testing– Open subroutines (inline/intrinsic functions), nonstandard semantics

(‘interpretive subroutines’)– Pure vs. impure functions– Debugging interpreter– Passing functions as parameters (e.g., to integration subroutines)

15References: [1]


What is a library?

• No clear answer, could be many things:

– A library is a reuse repository• “A library is a bunch of code I don’t have to write.”

– A library is a knowledge base• A library is a knowledge base about a problem domain.

– A library is a language extension• Different languages have differently sharp borders between

language and libraries

• “In effect, designing a class library is like designing part of a programming language, and should be approached with commensurate respect.”

– A library is a notation

16References: [1]


What is a library?

– A library is an expert-in-a-box• Allows to concentrate on the science without having to worry about

implementation details

– A library is an abstraction• APIs hide the details; we can use libraries knowing what they do but

don’t need to know how they do it.

– A library is a de-facto standard• Widely used, open source, well tested

• Full standards: C99 Standard library, C++ Standard template library

– A library is a defect management strategy• “ The only error free code I ever write is the code I do not have to

write“

17References: [1]


What is a library?

– A library is a tool for software compression• Especially in the context of shared libraries

– A library is a stable platform• Implementation can change without breaking code relying on it

– A library is a vehicle for technology adoption• A certain technology (way of doing things) may be encapsulated

behind a API, simplifying it‘s adoption

• New technologies may be encapsulated behind old APIs

– A library is a communication medium• Allows to communicate on higher levels using conepts

18References: [1]


Libraries (by locality)

19


Libraries (by domain)

20


Application domains

• General parallelization, load balancing– MPI, Charm++

• Mesh manipulation and management– METIS, ParMETIS, Jostle

• Graph manipulation– (Parallel) Boost.Graph library, MTGL

• Vector/Signal/Image processing– VSIPL, PSSL

• Linear algebra– BLAS, ATLAS, LAPACK, LINPACK, Slatec, pim

• Ordinary and partial Differential Equations– PETSc

21


Outline


22


How to use a library?

• Compile single source file• Compile multiple source files• Create a library• Compile multiple source files written using different

languages• Combining different languages

23


Compile single source file

24


Compile single source file

25


Compile multiple source files

26


Create a library

• Static library (.a)– Created using ar (archiver)– Just collection of object

modules and a table of entry points

– Used by linker to add referred code to created executable

• Dynamic library (.so)– Created by ld (linker)– Executable binary code

with resolved externals– Used by linker to add

reference to created executable

27


Create a library

• Static library (.a)– No additional runtime

dependencies– Beneficial in simple

scenario‘s

– If used in more than one module code will be duplicated

• Dynamic library (.so)– Code loaded only once– Beneficial in complex

binary applications

– Additional runtime dependency

– Difficult version control

28



29

main.c

int main(){ return say_hello(“Hello“);}

say_hello.c

int say_hello(char const* msg)

{ return puts(msg);}

• Interface of subroutines must be known• Different languages have different means of

interface specification for modules, subroutines and functions



main.c

#include “say_hello.h“int main(){ return say_hello(“Hello“);}

say_hello.c

#include <stdio.h>#include “say_hello.h“int say_hello(char const*

msg){ return puts(msg);}

30

say_hello.h

int say_hello(char const* msg);


Multi language programming

• Need to account for– Different calling conventions

• C, FORTRAN calling conventions

• Parameter passing (by value/by reference)

• Parameter types (strings)

– Naming conventions• FORTRAN: all uppercase, C: case is significant

– Data types• Memory layout (row major, column major, strings)

• Most of this is done by providing a correct interface description to the FORTRAN and/or C compilers

• All of this is highly compiler specific, but GNU compiler suite (gcc, f77 etc.) are well suited

31


Why C++

• Multiparadigm language– Object orientation, functional programming, template meta-programming

• Better maintainability of programs• More frequent code re-use• More efficient software development in groups• Higher adaptability of software to new demands

– Huge amount of libraries, from simple data structures and algorithms to modules in highly specialized domains

– But you don’t pay for what you don’t use

• C++ is available and supported by vendors on almost all Supercomputers like Cray, NEC SX, Hitachi SR8000 …

• With a few minor exceptions C++ is a better C: this allows a smooth migration from C to C++

– C is a full subset of C++

32


C/C++

• Since C and C++ are ‚siblings‘ interfacing is easy: extern ”C” {…}– Adjusts naming and calling conventions

• C data types are generally compatible with C++• C++ data types (classes) are not generally compatible

with C except POD (plain old data) types

33


Calling C from C++

34

main.cppextern “C“ {#include “say_hello.h“}int main(){ return say_hello(“Hello“);}

say_hello.c#include <stdio.h>#include “say_hello.h“int say_hello(char const*


say_hello.h

int say_hello(char const* msg);


Calling C++ from C

35

say_hello.hpp#ifdef __cplusplusextern “C“ int say_hello(char const* msg);#elseint say_hello(char const* msg);#endif

main.c

#include “say_hello.hpp“int main(){ return say_hello(“Hello“);}

say_hello.cpp#include <stdio.h>#include “say_hello.hpp“int say_hello(char const*



Calling C++ from C

36

say_hello.hpp#ifdef __cplusplusextern “C“#endifint say_hello(char const* msg);

main.c

#include “say_hello.hpp“int main(){ return say_hello(“Hello“);}

say_hello.cpp#include <stdio.h>#include “say_hello.hpp“int say_hello(char const*



FORTRAN/C

• Data types:

37

FORTRAN C/C++

integer*2 short int

integer long int or int

integer iabc(2,3) int iabc[3][2];

logical long int or int

logical*1 bool (C++, one byte)

Real float

real*8 double

complex C: struct { float r, i; }C++: std::complex<float>

double complex C: struct { double dr, di; }, C++: std::complex<double>

character*6 abc char abc[6];

parameter #define PARAMETER value


Multi language programming

38

main.fINTERFACE TO SUBROUTINE SAY_HELLO [C.ALIAS: '_say_hello'] (msg) CHARACTER(*) msgEND PROGRAM MAINCALL SAY_HELLO(“Hello“)END

say_hello.c

int say_hello( char const* msg, int len){ return printf(“%*d”, len, msg);}

• Interface of subroutines must be declared in a language specific way• Tooling support available

• SWIG: http://www.swig.org• FLIB2C: http://www.mycplus.com/utilitiesdetail.asp?iPro=7

• More information: http://arnholm.org/software/cppf77/cppf77.htm


Outline


39


Standard library support

• Standard (runtime) libraries• C++ Standard template library

40


Standard (runtime) libraries

• Libraries needed by almost any application– Provide often used functions (data types, algorithms, I/O, support

routines)– No need to explicitly specify library

• Compiler and linker usually ‚know’ what runtime libraries to use

• Different languages have different level and amount of standard library support (system level and support)– F77: very small standards library, F90: more available

• Filesystem, math, auxiliary

– C99: large standards library aimed at portability over wide amount of platforms

• Operating system, filesystem, math, string handling, basic data types (complex, integer types)

– C++: everything in C99 plus Standards Template Library • Adds data structures, algorithms

41


C++ Standard (Template) Library

• C++0x: library ~700 pages description in Standard– Language support (exception handling, memory allocation, etc.)– Diagnostics (assertions, system errors, etc.)– General utilities (tuples, metaprogramming, type traits)– Strings (string classes, numeric conversions)– Localization (locales)– STL (Standard template Library)

• Containers: sequence, associative , unordered associative containers– (vector, list, set, map, deque etc.)

• Iterators– (forward, bidirectional, random_access etc.)

• Algorithms: non-modifying, modifying, sorting algorithms – (foreach, transform, copy, sort, uniq etc.)

– Numerics: complex numbers, random number generators, etc.– Input/output: streams, file I/O– Regex– Atomic operations– Thread support: threads, mutual exclusion, futures

42


C++ Standard Template Library

• Algorithms and data structures are generic and orthogonal– Each algorithm usable with any data structure– Algorithms usable with your data structures– Your algorithms can use standard data structures

• Iterators connect the two – General pointer concept, i.e. used by algorithms to

refer to the data items– Allow to abstract the algorithms from the data

structures

43


Simple example

• Copy a vector of integer’s into a list std::vector<int> v; // v = 1, 2, 3, 4; std::list<int> l;

std::copy(v.begin(), v.end(), std::back_inserter(l));

• Or v.v.: std::copy(l.begin(), l.end(), std::back_inserter(v));

• How is it implemented

template <typename InIter, typename OutIter> void copy(InIter f, InIter l, OutIter o) { for (/**/; f != l; ++f, ++o) *o = *f; }

44


Genericity leads to Concepts

• No strict interface anymore (as known from Fortran or C)• Rather components required to expose concepts, i.e.

satisfy set of requirements• In the copy example:

– First two parameters must be at least input iterators• implement operator++(), operator*()

– Last parameter must be at least an output iterator• Implement operator++(), operator*()

• Advantage:– Generic, simple, uniform, optimized

• Drawback:– Long compiler error messages if something is wrong

45


Iterator categories (concepts)

46

Input Output

forward

bi-directional

Random access

Read one item at a time, in forward direction only

Write one item at a time, in forward direction only

Read and write one item at a time, in forward direction only Write one item at a

time, either in forward or backward direction

Write one item at a time, either in forward or backward direction can jump any distance


Iterator behavior

• Common operations++i: Advance one element and return reference to ii++: Advance one element and return the previous value of i

• Input iterator operations*i: Return a read-only reference to the element at i‘s current positioni == j: Return true if i and j positioned at the same element

(i != j: at different elements)

• Output iterator operations*i: Return a write-only reference to the element at i‘s current positioni = j: set i‘s position to the same as j‘s

• Bidirectional iterator operations--i: Retreat one element and returns i‘s new valuei--: Retreat one element and returns i‘s previous value

• Random access iterator operationsi + n: Return an iterator positioned n elements ahead i‘s current positioni – n: Return an iterator positioned n elements behind i‘s current positioni[n]: return a reference to the n‘th element from i‘s current position

• A plain C pointer is a random access iterator

47


Containers and iterators

Container Iterator Container Iterator

vector random access map bidirectional

deque random access multimap bidirectional

list bidirectional stack none

set bidirectional queue none

multiset bidirectional priority_queue none

48

• Every container– Has typedefs for this (no need to remember above):

• iterator, const_iterator, reverse_iterator, const_reverse_iterator

– Exposes functions returning iterators:• begin(), end() (non-const and const variants)


What‘s that all about?

• Orthogonality: std::vector<int> v; // v = 3, 1, 4, 2; std::sort(v.begin(), v.end()); std::list<int> l; // l = 3, 1, 4, 2; std::sort(v.begin(), v.end());

• Any algorithm is usable with any container– Still optimal code, because STL contains optimal implementation

for each iterator type– Optimal code with your data structures as well

49


Outline

• Why libraries• What is a library• How to use a library• Standard library support• Application Domains• Linear Algebra Library: BLAS• Summary - Materials for Test

50


Application domains

• Linear algebra– BLAS, ATLAS, LAPACK, ScaLAPACK, Slatec, pim

• Ordinary and Partial Differential Equations– PETSc

• Mesh Manipulation and Load Balancing – METIS, ParMETIS, CHACO, JOSTLE, PARTY

• Graph Manipulation– Boost.Graph library

• Vector/Signal/Image Processing– VSIPL, PESSL (IBMs Parallel Engineering Scientific Subroutine Library

• General Parallelization– MPI, pthreads

• Other Domain Specific Libraries– NAMD, NWChem, Fluent, Gaussian, LS-DYNA

51


Application Domain Overview

• Linear Algebra Libraries – Provide optimized methods for constructing sets of linear equations,

performing operations on them (matrix-matrix products, matrix-vector products) and solving them (factoring, forward & backward substitution.

– Commonly used libraries include BLAS, ATLAS, LAPACK, ScaLAPACK, PaLAPACK

• PDE Solvers: – General-purpose, parallel numerical PDE libraries

– Usual toolsets include manipulation of sparse data structures, iterative linear system solvers, preconditioners, nonlinear solvers and time-stepping methods.

– Commonly used libraries for solving PDEs include SAMRAI, PETSc, PARASOL, Overture, among others.

52


Application Domain Overview

• Mesh manipulation and Load Balancing – These libraries help in partitioning meshes in roughly equal sizes

across processors, thereby balancing the workload while minimizing size of separators and communication costs.

– Commonly used libraries for this purpose include METIS, ParMetis, Chaco, JOSTLE among others.

• Other packages:– FFTW: features highly optimized Fourier transform package

including both real and complex multidimensional transforms in sequential, multithreaded, and parallel versions.

– NAMD: molecular dynamics library available for Unix/Linux, Windows, OS X

– Fluent: computational fluid dynamics package, used for such applications as environment control systems, propulsion, reactor modeling etc.

53


Outline


54


BLAS

• (Updated set of) Basic Linear Algebra Subprograms

• The BLAS functionality is divided into three levels: – Level 1: contains vector operations of the form:

as well as scalar dot products and vector norms

– Level 2: contains matrix-vector operations of the form

as well as Tx = y solving for x with T being triangular

– Level 3: contains matrix-matrix operations of the form

as well as solving for triangular matrices T. This level contains the widely used General Matrix Multiply operation.

55


BLAS

• Several implementations for different languages exist– Reference implementation (F77 and C-wrapper)

http://www.netlib.org/blas/– ATLAS, highly optimized for particular processor architectures– A generic C++ template class library providing BLAS

functionality: uBLAS http://www.boost.org

– Several vendors provide libraries optimized for their architecture (AMD, HP, IBM, Intel, NEC, NViDIA, Sun)

56


BLAS: F77 naming conventions

57


BLAS: C naming conventions

• F77 routine name is changed to lowercase and prefixed with cblas_

• All routines accepting two dimensional arrays have a new additional first parameter specifying the matrix memory layout (row major or column major)

• Character parameters are replaced by corresponding enum values

• Input arguments are declared const• Non-complex scalar input parameters are passed by value• Complex scalar input arguments are passed using a void*• Arrays are passed by address• Output scalar arguments are passed by address• Complex functions become subroutines which return the result

via an additional last parameter (void*), appending _sub to the name

58


BLAS Level 1 routines

• Vector operations(xROT, xSWAP, xCOPY etc.)

• Scalar dot products (xDOT etc.)

• Vector norms(IxAMX etc.)

59



• Matrix-vector operations(xGEMV, xGBMV, xHEMV, xHBMV etc.)

• Solving Tx = y for x, where T is triangular(xGER, xHER etc.)

60



• Matrix-matrix operations(xGEMM etc.)

• Solving for triangular matrices(xTRMM)

• Widely used matrix-matrix multiply (xSYMM, xGEMM)

61


Outline

• Why libraries• What is a library• How to use a library• Standard library support• Introduction to High Performance Libraries• Linear Algebra Libraries (BLAS, LAPACK)• Summary - Materials for Test

62


Summary – Material for the Test

• Why Libraries: (Slide 5)• Calling Subroutines: (Slide 12)• What is a Library: (Slides 16-18)• Library (by locality) : (Slide 19)• Library (by domain): (Slide 20)• Application domains: (Slide 21)• Creating a library: (Slides 27-28)• Multi language programming: (Slide 31)• Standard runtime libraries: (Slide 41)• Iterator Categories & Behavior: (Slides 46-47)• Containers & Iterators: (Slide 48)• What is all that about: (Slide 49)• High performance libraries (Slides 51-53)• Linear algebra libraries: BLAS (Slides 55-61)


References

1. Todd Vedhuizen, What is a Library?, Talk given at the Dagstuhl workshop Software Libraries: Design and Evaluation, Schloss Dagstuhl, Germany, March 9-11 2005

2. Rene Heinzl, Modern Application Design using Modern Programming Paradigms and a Library-Centric Software Approach, OOPSLA 2006, Workshop on Library Centric Software Design, Portland, Oregon, October 2006.

3. C puzzles, http://www.gowrikumar.com/c/index.html

64

high performance computing : models, methods, & means introduction to libraries 1

Documents