A  crash  course  on    (C-­‐)  CUDA  programming  for  (mul8)  GPU  

Massimo  Bernaschi  massimo.bernaschi@cnr.it  

h;p://www.iac.cnr.it/~massimo  

Single  GPU  part  of  the  class  based  on  

Code  samples  available  from  (wget)    h3p://twin.iac.rm.cnr.it/cccsample.tgz  Slides  available  from  (wget)  h3p://twin.iac.rm.cnr.it/CCC.pdf                

Let  us  start  with  few  basic  info  

A  Graphics  Processing  Unit  is  not  a  CPU!    

And  there  is  no  GPU  without  a  CPU…  

Cache

ALU Control

ALU

ALU

ALU

DRAM DRAM

CPU GPU

Graphics  Processing  Unit    ü Original ly   dedicated   to   specific  opera8ons   required   by   video   cards   for  accelera8ng  graphics,  GPUs  have  become    flexible   general-­‐purpose   computa8onal  engines  (GPGPU):  

ü Op8mized  for  high  memory  throughput  ü Very  suitable  to  data-­‐parallel  processing    

ü Tradi8onally   GPU   programming   has  been   tricky   but   CUDA,   OpenCL   and   now  OpenACC  made  it  affordable.    

What is CUDA? §  CUDA=Compute Unified Device Architecture

—  Expose general-purpose GPU computing as first-class capability

—  Retain traditional DirectX/OpenGL graphics performance

§  CUDA C —  Based on industry-standard C —  A handful of language extensions to allow heterogeneous

programs —  Straightforward APIs to manage devices, memory, etc.

CUDA  Programming  Model  •  The  GPU  is  viewed  as  a  compute  device  that:  

–  has  its  own  RAM  (device  memory)  –  runs  data-­‐parallel  por8ons  of  an  applica8on  as  kernels  

by  using  many  threads  •  GPU  vs.  CPU  threads    

–  GPU  threads  are  extremely  lightweight  •  Very  li;le  crea8on  overhead  

–  GPU  needs  1000s  of  threads  for  full  efficiency  •  A  mul8-­‐core  CPU  needs  only  a  few    

(basically  one  thread  per  core)  

CUDA C Jargon: The Basics

Host

§ Host – The CPU and its memory (host memory)

§ Device – The GPU and its memory (device memory)

Device

What  Programmer  Expresses  in  CUDA    

ü  Computa8on  par88oning  (where  does  computa8on  occur?)  ü  Declara8ons  on  func8ons  __host__,  __global__,  __device__  ü Mapping  of  thread  programs  to  device:  compute  <<<gs,  bs>>>(<args>)  

ü  Data  par88oning  (where  does  data  reside,  who  may  access  it  and  how?)  

ü Declara8ons  on  data  __shared__,  __device__,  __constant__,  …  ü  Data  management  and  orchestra8on  

ü Copying  to/from  host:    e.g.,  cudaMemcpy(h_obj,d_obj,  size,  cudaMemcpyDevicetoHost)  

ü  Concurrency  management  ü  e.g.  __synchthreads()  

P   P  

HO

ST

(CP

U)

M  

DE

VIC

E (G

PU

)

Interconnect  between  devices  and  memories  

M  

Code Samples  

1.   Connect  to  the  system  ssh  –X  XYZ@abcd.zz  

2.   Get  the  code  samples:    wget  h;p://twin.iac.rm.cnr.it/ccc.tgz  wget  h;p://twin.iac.rm.cnr.it/cccsample.tgz  

3.   Unpack  the  samples:  tar  zxvf  cccsample.tgz  

4.   Go  to  source  directory:  cd  cccsample/source  

5.   Get  the  CUDA  C  Quick  Reference  wget  h;p://twin.iac.rm.cnr.it/CUDA_C_QuickRef.pdf  

Hello, World! int main( void ) { printf( "Hello, World!\n" ); return 0; }    •  To  compile:  nvcc  –o  hello_world  hello_world.cu  

 •  To  execute:  ./hello_world  

 •  This  basic  program  is  just  standard  C  that  runs  on  the  host    

•  NVIDIA’s  compiler  (nvcc)  will  not  complain  about  CUDA  programs  with  no  device  code  

•  At  its  simplest,  CUDA  C  is  just  C!  

Hello, World! with Device Code __global__ void kernel( void ) { } int main( void ) { kernel<<<1,1>>>(); printf( "Hello, World!\n" ); return 0; }

•  Two  notable  addi8ons  to  the  original    “Hello,  World!”  

To  compile:  nvcc  –o  simple_kernel  simple_kernel.cu    To  execute:  ./simple_kernel  

Hello, World! with Device Code __global__ void kernel( void ) { }    •  CUDA  C  keyword    __global__ indicates  that  a  

func8on  –  Runs  on  the  device  –  Called  from  host  code  

•  nvcc  splits  source  file  into  host  and  device  components  –  NVIDIA’s  compiler  handles  device  func8ons  like  kernel() –  Standard  host  compiler  handles  host  func8ons  like  main()  

•  gcc, icc, … •  Microso]  Visual  C  

Hello, World! with Device Code int main( void ) { kernel<<< 1, 1 >>>(); printf( "Hello, World!\n" ); return 0; }

•  Triple  angle  brackets  mark  a  call  from  host  code  to  device  code  –  A  “kernel  launch”  in  CUDA  jargon    –  We’ll  discuss  the  parameters  inside  the  angle  brackets  later  

 •  This  is  all  that’s  required  to  execute  a  func8on  on  the  GPU!   •  The  func8on  kernel()  does  nothing,  so  let  us  run  something  a  

bit  more  useful:  

A kernel to make an addition __global__ void add(int a, int b, int *c) { *c = a + b; }

•  Compile:  nvcc  –o  addint  addint.cu  

•  Run:    ./addint  

•  Look  at  line  28  of  the  addint.cu    code  How  do  we  pass  the  first  two  arguments?  

A More Complex Example •  Another  kernel  to  add  two  integers:     __global__ void add( int *a, int *b, int *c ) { *c = *a + *b; }

 •  As  before,  __global__ is  a  CUDA  C  keyword  meaning  –  add()  will  execute  on  the  device  –  add()  will  be  called  from  the  host  

A More Complex Example •  No8ce  that  now  we  use  pointers  for  all  our  variables:  

  __global__ void add( int *a, int *b, int *c ) { *c = *a + *b; }  

•  add()  runs  on  the  device…so  a,  b,  and  c  must  point  to  device  memory  

•  How  do  we  allocate  memory  on  the  GPU?  

Memory Management •  Up  to  CUDA  4.0  host  and  device  memory  were  dis8nct  

en88es  from  the  programmers’  viewpoint  –  Device  pointers  point  to  GPU  memory  

•  May  be  passed  to  and  from  host  code  •  (In  general)  May  not  be  dereferenced  from  host  code  

–  Host  pointers  point  to  CPU  memory    •  May  be  passed  to  and  from  device  code  •  (In  general)  May  not  be  dereferenced  from  device  code  

 

Star8ng  on  CUDA  4.0  there  is  a  Unified  Virtual  Addressing  feature.  •  Basic  CUDA  API  for  dealing  with  device  memory  

–  cudaMalloc(&p, size),  cudaFree(p),    cudaMemcpy(t, s, size, direction)  

–  Similar  to  their  C  equivalents:  malloc(),  free(),  memcpy()

pointer  to  pointer  

A More Complex Example: main() int main( void ) { int a, b, c; // host copies of a, b, c int *dev_a, *dev_b, *dev_c; // device copies of a, b, c int size = sizeof( int ); // we need space for an integer // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, size ); a = 2; b = 7; // copy inputs to device cudaMemcpy( dev_a, &a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, &b, size, cudaMemcpyHostToDevice ); // launch add() kernel on GPU, passing parameters add<<< 1, 1 >>>( dev_a, dev_b, dev_c ); // copy device result back to host copy of c cudaMemcpy( &c, dev_c, size, cudaMemcpyDeviceToHost ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ) return 0; }  

Parallel Programming in CUDA C •  But  wait…GPU  compu8ng  is  about  massive  parallelism  

•  So  how  do  we  run  code  in  parallel  on  the  device?  

•  Solu8on  lies  in  the  parameters  between  the  triple  angle  brackets:  

add<<< 1, 1 >>>( dev_a, dev_b, dev_c ); add<<< N, 1 >>>( dev_a, dev_b, dev_c );

•  Instead  of  execu8ng  add()  once,  add()  executed  N  8mes  in  parallel  

Parallel Programming in CUDA C •  With  add()  running  in  parallel…let’s  do  vector  addi8on  

•  Terminology:  Each  parallel  invoca8on  of  add()  referred  to  as  a  block  

•  Kernel  can  refer  to  its  block’s  index  with  the  variable  blockIdx.x •  Each  block  adds  a  value  from  a[]  and  b[],  storing  the  result  in  c[]:

__global__ void add( int *a, int *b, int *c ) { c[blockIdx.x] = a[blockIdx.x]+b[blockIdx.x]; }

•  By  using  blockIdx.x  to  index  arrays,  each  block  handles  a  different  index    •  blockIdx.x is  the  first  example  of  a  CUDA  predefined  variable.  

Parallel Programming in CUDA C

Block  1  

c[1]=a[1]+b[1];

§ We write this code: __global__ void add( int *a, int *b, int *c ) {

c[blockIdx.x] = a[blockIdx.x]+b[blockIdx.x];

}

§ This is what runs in parallel on the device:

Block  0  

c[0]=a[0]+b[0];

Block  2  

c[2]=a[2]+b[2];

Block  3  

c[3]=a[3]+b[3];

Parallel Addition: main() #define N 512 int main( void ) { int *a, *b, *c; // host copies of a, b, c int *dev_a, *dev_b, *dev_c; // device copies of a, b, c int size = N * sizeof( int ); // we need space for 512

// integers // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, size ); a = (int*)malloc( size ); b = (int*)malloc( size ); c = (int*)malloc( size ); random_ints( a, N ); random_ints( b, N );

Parallel Addition: main() (cont) // copy inputs to device cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice ); // launch add() kernel with N parallel blocks add<<< N, 1 >>>( dev_a, dev_b, dev_c ); // copy device result back to host copy of c cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost ); free( a ); free( b ); free( c ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ); return 0; }  

Review §  Difference between “host” and “device”

—  Host = CPU —  Device = GPU

§  Using __global__ to declare a function as device code —  Runs on device —  Called from host

§  Passing parameters from host code to a device function

Review (cont) §  Basic device memory management

—  cudaMalloc() —  cudaMemcpy() —  cudaFree()

§  Launching parallel kernels —  Launch N copies of add() with: add<<< N, 1 >>>(); —  blockIdx.x allows to access block’s index

•  Exercise: look at, compile and run the add_simple_blocks.cu code

Threads §  Terminology: A block can be split into parallel threads

§  Let’s change vector addition to use parallel threads instead of parallel blocks:

__global__ void add( int *a, int *b, int *c ) { c[ ] = a[ ]+ b[ ]; }

§  We use threadIdx.x instead of blockIdx.x in add()

§  main() will require one change as well…

threadIdx.x threadIdx.x threadIdx.x blockIdx.x blockIdx.x blockIdx.x

Parallel Addition (Threads): main() #define N 512 int main( void ) { int *a, *b, *c; //host copies of a, b, c int *dev_a, *dev_b, *dev_c;//device copies of a, b, c int size = N * sizeof( int )//we need space for 512 integers // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, size ); a = (int*)malloc( size ); b = (int*)malloc( size ); c = (int*)malloc( size ); random_ints( a, N ); random_ints( b, N );

Parallel Addition (Threads): main() // copy inputs to device cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice ); // launch add() kernel with N add<<< >>>( dev_a, dev_b, dev_c ); // copy device result back to host copy of c cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost ); free( a ); free( b ); free( c ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ); return 0; }  

threads

1, N

blocks

N, 1

Exercise:  compile  and  run  the    add_simple_threads.cu  code  

Using Threads And Blocks §  We’ve seen parallel vector addition using

—  Many blocks with 1 thread apiece —  1 block with many threads

§  Let’s adapt vector addition to use lots of both blocks and

threads

§  After using threads and blocks together, we’ll talk about why threads

§  First let’s discuss data indexing…

Indexing Arrays With Threads & Blocks •  No  longer  as  simple  as  just  using  threadIdx.x  or  blockIdx.x  as  indices  

•  To  index  array  with  1  thread  per  entry  (using  8  threads/block)    

•  If  we  have  M  threads/block,  a  unique  array  index  for  each  entry  is  given  by   int index = threadIdx.x + blockIdx.x * M;

int index = x + y * width;

 

blockIdx.x = 0 blockIdx.x = 1 blockIdx.x = 2 blockIdx.x = 3

threadIdx.x

0 1 2 3 4 5 6 7

threadIdx.x

0 1 2 3 4 5 6 7

threadIdx.x

0 1 2 3 4 5 6 7

threadIdx.x

0 1 2 3 4 5 6 7

Indexing Arrays: Example •  In  this  example,  the  red  entry  would  have  an  index  of  21:  

int index = threadIdx.x + blockIdx.x * M;                                                  = 5 + 2 * 8; = 21;

 

blockIdx.x = 2

M = 8 threads/block

0 17 8 16 18 19 20 21 2 1 3 4 5 6 7 10 9 11 12 13 14 15

Indexing Arrays: other examples (4 blocks with 4 threads per block)

Addition with Threads and Blocks §  blockDim.x is a built-in variable for threads per block: int index= threadIdx.x + blockIdx.x * blockDim.x; §  gridDim.x is a built-in variable for blocks in a grid;

§  A combined version of our vector addition kernel to use blocks and

threads: __global__ void add( int *a, int *b, int *c ) { int index = threadIdx.x + blockIdx.x * blockDim.x; c[index] = a[index] + b[index]; }

§  So what changes in main() when we use both blocks and threads?

Parallel Addition (Blocks/Threads) #define N (2048*2048) #define THREADS_PER_BLOCK 512 int main( void ) { int *a, *b, *c; // host copies of a, b, c int *dev_a, *dev_b, *dev_c;//device copies of a, b, c int size = N * sizeof( int ); // we need space for N

// integers // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, size ); a = (int*)malloc( size ); b = (int*)malloc( size ); c = (int*)malloc( size );

random_ints( a, N ); random_ints( b, N );

Parallel Addition (Blocks/Threads) // copy inputs to device cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice ); // launch add() kernel with blocks and threads add<<< N/THREADS_PER_BLOCK, THREADS_PER_BLOCK >>>( dev_a, dev_b, dev_c); // copy device result back to host copy of c cudaMemcpy( c, dev_c, size, cudaMemcpyDeviceToHost ); free( a ); free( b ); free( c ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ); return 0; }  

Exercise:  compile  and  run  the    add_simple.cu  code  

Exercise:  array  reversal  

•  Start  from  the  arrayReversal.cu  code  The  code  must  fill  an  array  d_out  with  the  contents  of  an  input  array  d_in  in  reverse  order  so  that  if    d_in  is  [100,  110,  200,  220,  300]  then    d_out  must  be  [300,  220,  200,  110,  100].  

•  You  must  complete  line  13,  14  and  34.  •  Remember  that:    

– blockDim.x is the number of threads per block; – gridDim.x is the number of blocks in a grid;

Array  Reversal:  Soluaon  

__global__  void  reverseArrayBlock(int  *d_out,  int  *d_in)  {  

       int  in    =      blockDim.x  *  blockIdx.x    +  threadIdx.x  ;            int  out  =      (blockDim.x  *  gridDim.x  -­‐1)  –  in  ;          d_out[out]  =  d_in[in];  }    

Why Bother With Threads? §  Threads seem unnecessary

—  Added a level of abstraction and complexity —  What did we gain?

§  Unlike parallel blocks, parallel threads have mechanisms to —  Communicate —  Synchronize

§  Let’s see how…

Dot Product §  Unlike vector addition, dot product is a reduction from vectors to

a scalar

c = a · b

c = (a0, a1, a2, a3) · (b0, b1, b2, b3)

c = a0 b0 + a1 b1 + a2 b2 + a3 b3

Dot Product •  Parallel  threads  have  no  problem  compu8ng  the  pairwise  products:  

•  So  we  can  start  a  dot  product  CUDA  kernel  by  doing  just  that:   __global__ void dot( int *a, int *b, int *c ) { // Each thread computes a pairwise product int temp = a[threadIdx.x] * b[threadIdx.x];

Dot Product •  But  we  need  to  share  data  between  threads  to  compute  the  final  sum:  

  __global__ void dot( int *a, int *b, int *c ) { // Each thread computes a pairwise product int temp = a[threadIdx.x] * b[threadIdx.x];   // Can’t compute the final sum // Each thread’s copy of ‘temp’ is private!!! }

Sharing Data Between Threads §  Terminology: A block of threads shares memory called…

§  Extremely fast, on-chip memory (user-managed cache)

§  Declared with the __shared__ CUDA keyword

§  Not visible to threads in other blocks running in parallel

shared memory

Shared  Memory  

Threads  

Block  0  

Shared  Memory  

Threads  

Block  1  

Shared  Memory  

Threads  

Block  2  

…  

Parallel Dot Product: dot() §  We perform parallel multiplication, serial addition: #define N 512 __global__ void dot( int *a, int *b, int *c ) { // Shared memory for results of multiplication __shared__ int temp[N]; temp[threadIdx.x] = a[threadIdx.x] * b[threadIdx.x]; // Thread 0 sums the pairwise products if( 0 == threadIdx.x ) { int sum = 0; for( int i = N-1; i >= 0; i-- ){ sum += temp[i];

} *c = sum; } }

Parallel Dot Product Recap §  We perform parallel, pairwise multiplications

§  Shared memory stores each thread’s result

§  We sum these pairwise products from a single thread

§  Sounds good…

But…

Exercise:  Compile  and  run  dot_simple_threads.cu.    Does  it  work  as  expected?  

Faulty Dot Product Exposed! §  Step 1: In parallel, each thread writes a pairwise product

§  Step 2: Thread 0 reads and sums the products

§  But there’s an assumption hidden in Step 1…

__shared__ int temp  

__shared__ int temp  

Read-Before-Write Hazard §  Suppose thread 0 finishes its write in step 1

§  Then thread 0 reads index 12 in step 2

§  Before thread 12 writes to index 12 in step 1

This read returns garbage!

Synchronization §  We need threads to wait between the sections of dot(): __global__ void dot( int *a, int *b, int *c ) { __shared__ int temp[N]; temp[threadIdx.x] = a[threadIdx.x] * b[threadIdx.x]; // * NEED THREADS TO SYNCHRONIZE HERE * // No thread can advance until all threads // have reached this point in the code // Thread 0 sums the pairwise products if( 0 == threadIdx.x ) { int sum = 0; for( int i = N-1; i >= 0; i-- ){ sum += temp[i];

} *c = sum; } }

__syncthreads()

§ We can synchronize threads with the function __syncthreads()

§  Threads in the block wait until all threads have hit the __syncthreads()

§  Threads are only synchronized within a block!

__syncthreads()  

__syncthreads()  

__syncthreads()  

__syncthreads()  

__syncthreads()  

Thread 0  Thread 1  Thread 2  

Thread 3  

Thread 4  …  

Parallel Dot Product: dot() __global__ void dot( int *a, int *b, int *c ) { __shared__ int temp[N]; temp[threadIdx.x] = a[threadIdx.x] * b[threadIdx.x]; __syncthreads(); if( 0 == threadIdx.x ) { int sum = 0; for( int i = N-1; i >= 0; i-- ){ sum += temp[i];

} *c = sum; } }

§  With a properly synchronized dot() routine, let’s look at main()

Parallel Dot Product: main() #define N 512 int main( void ) { int *a, *b, *c; // copies of a, b, c int *dev_a, *dev_b, *dev_c; // device copies of a, b, c int size = N * sizeof( int ); // we need space for N integers // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, sizeof( int ) ); a = (int *)malloc( size ); b = (int *)malloc( size ); c = (int *)malloc( sizeof( int ) ); random_ints( a, N ); random_ints( b, N );

Parallel Dot Product: main() // copy inputs to device

cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice ); // launch dot() kernel with 1 block and N threads dot<<< 1, N >>>( dev_a, dev_b, dev_c ); // copy device result back to host copy of c cudaMemcpy( c, dev_c, sizeof( int ), cudaMemcpyDeviceToHost ); free( a ); free( b ); free( c ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ); return 0; }  

Exercise:  insert  __syncthreads()  in  the  dot  kernel.  Compile  and  run.  

Review §  Launching kernels with parallel threads

—  Launch add() with N threads: add<<< 1, N >>>(); —  Used threadIdx.x to access thread’s index

§  Using both blocks and threads —  Used (threadIdx.x + blockIdx.x * blockDim.x) to index input/output —  N/THREADS_PER_BLOCK blocks and THREADS_PER_BLOCK threads gave us N threads

total

Review (cont) §  Using __shared__ to declare memory as shared

memory —  Data shared among threads in a block —  Not visible to threads in other parallel blocks

§  Using __syncthreads() as a barrier —  No thread executes instructions after __syncthreads() until

all threads have reached the __syncthreads() —  Needs to be used to prevent data hazards

Multiblock Dot Product •  Recall  our  dot  product  launch:   // launch dot() kernel with 1 block and N threads dot<<< 1, N >>>( dev_a, dev_b, dev_c );

•  Launching  with  one  block  will  not  u8lize  much  of  the  GPU  

•  Let’s  write  a  mul9block  version  of  dot  product  

Multiblock Dot Product: Algorithm §  Each block computes a sum of its pairwise products like before:

Multiblock Dot Product: Algorithm §  And then contributes its sum to the final result:

Multiblock Dot Product: dot() #define THREADS_PER_BLOCK 512 #define N (1024*THREADS_PER_BLOCK) __global__ void dot( int *a, int *b, int *c ) { __shared__ int temp[THREADS_PER_BLOCK]; int index = threadIdx.x + blockIdx.x * blockDim.x; temp[threadIdx.x] = a[index] * b[index]; __syncthreads(); if( 0 == threadIdx.x ) { int sum = 0; for( int i = 0; i < THREADS_PER_BLOCK; i++ ) { sum += temp[i];

} } }

§  But  we  have  a  race  condi8on…  Compile  and  run  dot_simple_mul8block.cu  

§  We  can  fix  it  with  one  of  CUDA’s  atomic  opera8ons.    Use  atomicAdd  in  the  dot  kernel.  Compile  with  nvcc  –o  dot_simple_mulablock  dot_simple_mulablock.cu  –arch=sm_35  

*c += sum;  atomicAdd( c , sum );  

Race Conditions

§  Thread 0, Block 1 –  Read value at address c –  Add sum to value –  Write result to address c

§  Terminology: A race condition occurs when program behavior depends upon relative timing of two (or more) event sequences

§  What actually takes place to execute the line in question: *c += sum; —  Read value at address c —  Add sum to value —  Write result to address c

§  What if two threads are trying to do this at the same time?

§  Thread 0, Block 0 —  Read value at address c —  Add sum to value —  Write result to address c

Terminology: Read-Modify-Write

Global Memory Contention

c

Block 0 sum = 3

Block 1 sum = 4

Reads 0 0

Computes 0+3 0+3 = 3 3

Writes 3

Reads 3 3

Computes 3+4 3+4 = 7 7

Writes 7

0  

Read-Modify-Write

Read-Modify-Write

*c += sum   3  0   3   3   7  

Global Memory Contention

0  c

Block 0 sum = 3

Block 1 sum = 4

Reads 0 0

Computes 0+3 0+3 = 3 3

Writes 3

Reads 0 0

Computes 0+4 0+4 = 4 4

Writes 4

Read-Modify-Write

Read-Modify-Write

*c += sum   0   0   3   4  0  

Atomic Operations §  Terminology: Read-modify-write uninterruptible when atomic

§  Many atomic operations on memory available with CUDA C

§  Predictable result when simultaneous access to memory required

§ We need to atomically add sum to c in our multiblock dot product

§  atomicAdd() §  atomicSub() §  atomicMin() §  atomicMax()

§  atomicInc() §  atomicDec() §  atomicExch() §  atomicCAS()old == compare ? val : old

Multiblock Dot Product: dot() __global__ void dot( int *a, int *b, int *c ) { __shared__ int temp[THREADS_PER_BLOCK]; int index = threadIdx.x + blockIdx.x * blockDim.x; temp[threadIdx.x] = a[index] * b[index]; __syncthreads(); if( 0 == threadIdx.x ) { int sum = 0; for( int i = 0; i < THREADS_PER_BLOCK; i++ ){ sum += temp[i];

} atomicAdd( c , sum ); } }

§  Now  let’s  fix  up  main()  to  handle  a  mul9block  dot  product  

Parallel Dot Product: main() #define THREADS_PER_BLOCK 512 #define N (1024*THREADS_PER_BLOCK) int main( void ) { int *a, *b, *c; // host copies of a, b, c int *dev_a, *dev_b, *dev_c; // device copies of a, b, c int size = N * sizeof( int ); // we need space for N ints // allocate device copies of a, b, c cudaMalloc( (void**)&dev_a, size ); cudaMalloc( (void**)&dev_b, size ); cudaMalloc( (void**)&dev_c, sizeof( int ) ); a = (int *)malloc( size ); b = (int *)malloc( size ); c = (int *)malloc( sizeof( int ) ); random_ints( a, N ); random_ints( b, N );

Parallel Dot Product: main() // copy inputs to device cudaMemcpy( dev_a, a, size, cudaMemcpyHostToDevice ); cudaMemcpy( dev_b, b, size, cudaMemcpyHostToDevice ); // launch dot() kernel dot<<<N/THREADS_PER_BLOCK,THREADS_PER_BLOCK>>>(dev_a, dev_b, dev_c); // copy device result back to host copy of c cudaMemcpy( c, dev_c, sizeof( int ), cudaMemcpyDeviceToHost ); free( a ); free( b ); free( c ); cudaFree( dev_a ); cudaFree( dev_b ); cudaFree( dev_c ); return 0; }  

Review §  Race conditions

—  Behavior depends upon relative timing of multiple event sequences

—  Can occur when an implied read-modify-write is interruptible

§  Atomic operations —  CUDA provides read-modify-write operations guaranteed to be

atomic —  Atomics ensure correct results when multiple threads modify

memory —  To use atomic operations a new option (-arch=…) must be used

at compile time

Eratosthenes’  Sieve  

•  A  sieve  has  holes  in  it  and  is  used  to  filter  out  the  juice.  

•  Eratosthenes’s  sieve  filters  out  numbers  to  find  the  prime  numbers.  

Defini8on  

•   Prime  Number  –  a  number  that  has  only  two  factors,  itself  and  1.  

7 7 is prime because the only numbers that will divide into it evenly are 1 and 7.

The  Eratosthenes’Sieve  

The  Prime  Numbers  from  1  to  120  are  the  following:  

2,3,5,7,11,13,17,19,23,  29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97,  101,103,107,109,113    Download  (wget)  h3p://twin.iac.rm.cnr.it/eratostenePr.cu  

CUDA  Thread  organiza8on:  Grids  and  Blocks  

•  A  kernel  is  executed  as  a  1D,  2D  or  3D  grid  of  thread  blocks.    –  All  threads  share  the  global  memory  

•  A  thread  block  is  a  1D,  2D  or  3D  batch  of  threads  that  can  cooperate  with  each  other  by:  –  Synchronizing  their  execu8on  

•  For  hazard-­‐free  shared  memory  accesses  

–  Efficiently  sharing  data  through  a  low  latency  shared  memory  

•  Threads  blocks  are  independent  of  each  other  and  can  be  executed    in  any  order!  

Host

Kernel 1

Kernel 2

Device

Grid 1

Block (0, 0)

Block (1, 0)

Block (2, 0)

Block (0, 1)

Block (1, 1)

Block (2, 1)

Grid 2

Block (1, 1)

Thread (0, 1)

Thread (1, 1)

Thread (2, 1)

Thread (3, 1)

Thread (4, 1)

Thread (0, 2)

Thread (1, 2)

Thread (2, 2)

Thread (3, 2)

Thread (4, 2)

Thread (0, 0)

Thread (1, 0)

Thread (2, 0)

Thread (3, 0)

Thread (4, 0)

Courtesy: NVIDIA

Built-­‐in  Variables  to  manage  grids  and  blocks  

•  dim3 gridDim; –  Dimensions  of  the  grid  in  blocks    

•  dim3 blockDim; –  Dimensions  of  the  block  in  threads  

•  dim3 blockIdx; –  Block  index  within  the  grid  

•  dim3 threadIdx; –  Thread  index  within  the  block  

dim3:  a  new  datatype  defined  by  CUDA:  struct  dim3  {  unsigned  int  x,  y,  z  };  three  unsigned  ints  where  any  unspecified  component  defaults  to  1.  

int  main()  {          int  dimx  =  16;          int  dimy  =  16;          int  num_bytes  =  dimx*dimy*sizeof(int);            int  *d_a=0,  *h_a=0;  //  device  and  host  pointers            h_a  =  (int*)malloc(num_bytes);          cudaMalloc(  (void**)&d_a,  num_bytes  );            dim3  grid,  block;          block.x  =  4;          block.y  =  4;          grid.x    =  dimx  /  block.x;          grid.y    =  dimy  /  block.y;            kernel<<<grid,  block>>>(  d_a,  dimx,  dimy  );                  cudaMemcpy(h_a,d_a,num_bytes,                                                                cudaMemcpyDeviceToHost);            for(int  row=0;  row<dimy;  row++)  {                  for(int  col=0;  col<dimx;  col++)                          prin|("%d  ",  h_a[row*dimx+col]  );                  prin|("\n");          }            free(  h_a  );          cudaFree(  d_a  );          return  0;  }  

__global__ void kernel( int *a, int dimx, int dimy ) { int ix = blockIdx.x*blockDim.x + threadIdx.x; int iy = blockIdx.y*blockDim.y + threadIdx.y; int idx = iy*dimx + ix; a[idx] = idx+1; }

Bi-­‐dimensional  threads  configura8on  by  example:  

set  the  elements  of  a  square  matrix  

(assume  the  matrix  is  a  single  block  of  memory!)  

Exercise:  compile  and  run:  setmatrix.cu  

Matrix  mul8ply  with  one  thread  block  

•  One  block  of  threads  computes  matrix  dP  –  Each  thread  computes  one  element  

of  dP  •  Each  thread  

–  Loads  a  row  of  matrix  dM  –  Loads  a  column  of  matrix  dN  –  Perform  one  mul8ply  and  addi8on  

for  each  pair  of  dM  and  dN  elements  •  Size  of  matrix  limited  by  the  number  

of  threads  allowed  in  a  thread  block!    

Grid 1 Block 1

3 2 5 4

2

4

2

6

48

Thread )2, 2(‏

WIDTH

dM dP

dN

ü Start  from  MMGlob.cu  ü Complete  the  func8on  

MatrixMulOnDevice  and  the  kernel  MatrixMulKernel  

ü Use  one  bi-­‐dimensional  block  ü Threads  will  have  an  x    (threadIdx.x) and  an  y  iden8fier  (threadIdx.y)  

ü Max  size  of  the  matrix:  16  Compile:  nvcc  –o  MMGlob  MMglob.cu  Run:  ./MMGlob  N  

   

__global__ void MatrixMulKernel(DATA* dM, DATA* dN, DATA* dP, int Width) { DATA Pvalue = 0.0; for (int k = 0; k < Width; ++k) { DATA Melement = dM[ ]; DATA Nelement = dN[ ]; Pvalue += Melement * Nelement; } dP[ ] = Pvalue; }

Exercise:  matrix-­‐matrix    mul8ply  

CUDA  Compiler:  nvcc  basic  op8ons  ü  -­‐arch=sm_13    (or  sm_20,  sm_35)  Enable  double  precision  (on  compa8ble  hardware)  ü  -­‐G        Enable  debug  for  device  code  

ü  -­‐-­‐ptxas-­‐op8ons=-­‐v  Show  register  and  memory  usage  

ü  -­‐-­‐maxrregcount  <N>  Limit  the  number  of  registers  

ü  -­‐use_fast_math  Use  fast  math  library    

ü  -­‐O3  Enables  compiler  op8miza8on    

ü  -­‐ccbin  compiler_path  use  a  different  C  compiler  (e.g.,  Intel  instead  of  gcc)  Exercises:    

1.  re-­‐compile  the  MMGlob.cu  program  and  check  how  many  registers  the  kernel  uses:  nvcc  –o  MMGlob  MMGlob.cu  -­‐-­‐ptxas-­‐op8ons=-­‐v  

2.  Edit  MMGlob.cu  and  modify  DATA  from  float  to  double,  then  compile  for  double  precision  and  run  

CUDA  Error  Repor8ng  to  CPU  

•  All  CUDA  calls  return  an  error  code:  –  except  kernel  launches  –  cudaError_t  type    

•  cudaError_t  cudaGetLastError(void)  –  returns  the  code  for  the  last  error  (cudaSuccess  means  “no  error”)    

•  char*  cudaGetErrorString(cudaError_t  code)  –  returns  a  null-­‐terminated  character  string  describing  the  error    

prinn(“%s\n”,cudaGetErrorString(cudaGetLastError()  )  );  

CUDA  Event  API  •  Events  are  inserted  (recorded)  into  CUDA  call  streams  

•  Usage  scenarios:  –  measure  elapsed  8me  (in  milliseconds)  for  CUDA  calls  (resolu8on  ~0.5  microseconds)  –  query  the  status  of  an  asynchronous  CUDA  call  –  block  CPU  un8l  CUDA  calls  prior  to  the  event  are  completed  

 

cudaEvent_t  start,  stop;  float  et;  cudaEventCreate(&start);      cudaEventCreate(&stop);    

cudaEventRecord(start,  0);    

kernel<<<grid,  block>>>(...);    

cudaEventRecord(stop,  0);    

cudaEventSynchronize(stop);      

cudaEventElapsedTime(&et,  start,  stop);    

cudaEventDestroy(start);        cudaEventDestroy(stop);  prin|(“Kernel  execu8on  8me=%f\n”,et);  

Read  the  func8on  MatrixMulOnDevice  of  the  MMGlobWT.cu  program.  Compile  and  run      

Execution Model Software Hardware

Threads  are  executed  by  scalar  processors  

Thread  

Scalar    Processor  

Thread    Block   Mul8processor  

Thread  blocks  are  executed  on  mul8processors    Thread  blocks    do  not  migrate    Several  concurrent  thread  blocks  can  reside  on  one  mul8processor  -­‐  limited  by  mul8processor  resources  (shared  memory  and  register  file)  

...

Grid   Device  

A  kernel  is  launched  as  a  grid  of  thread  blocks    

86  

Transparent  Scalability  •  Hardware  is  free  to  assigns  blocks  to  any  

processor  at  any  8me  –  A  kernel  scales  across  any  number  of  parallel  

processors  Device

Block 0 Block 1

Block 2 Block 3

Block 4 Block 5

Block 6 Block 7

Kernel grid

Block 0 Block 1

Block 2 Block 3

Block 4 Block 5

Block 6 Block 7

Device

Block 0 Block 1 Block 2 Block 3

Block 4 Block 5 Block 6 Block 7

Each  block  can  execute  in  any  order  relaave  to  other  blocks!    

8me  

87!

Warps and Half Warps

Thread    Block   Mul8processor  

32  Threads  

32  Threads  

32  Threads  

...  

Warps  

16  

Half  Warps  

16  

DRAM  

 

 

Global

Local

A  thread  block  consists  of  32-­‐thread  warps    A  warp  is  executed  physically  in  parallel  (SIMD)  on  a  mul8processor  

Device    Memory  

=

A  half-­‐warp  of  16  threads  can  coordinate  global  memory  accesses  into  a  single  transac8on  

88  

Execu8ng  Thread  Blocks  

•  Threads  are  assigned  to  Streaming  Mulaprocessors  (SM)  in  block  granularity  –  Up  to  16  blocks  to  each  SM    

on  Kepler.  8  blocks  on  Fermi  SM.    –  A  Fermi  SM  can  take  up  to  1536  threads  

•  Could  be  256  (threads/block)  *  6  blocks    or  192  (threads/block)  *  8  blocks  but,  for  instance,  128  (threads/block)  *  12  blocks  not  allowed!  

–  A  Kepler  SM  can  take  up  to  2048  threads  •  Threads  run  concurrently  

–  SM  manages/schedules  thread  execu8on  

t0 t1 t2 … tm

Blocks

SP

Shared Memory

MT IU

SP

Shared Memory

MT IU

t0 t1 t2 … tm

Blocks

SM 1 SM 0

The  number  of  threads  in  a  block  depends  on  the  capability    (i.e.,  the  version)  of  the  GPU.  On  a  Fermi  or  Kepler  GPU  a  block  may  have  up  to  1024  threads.  

89  

Block  Granularity  Considera8ons  •  For  a  kernel  running  on  a  Fermi  and  using  mul8ple  blocks,  should  

we  use  8X8,  16X16  or  32X32  blocks?  

–  For  8X8,  we  have  64  threads  per  Block.  Since  each  Fermi  SM  can  take  up  to  1536  threads,  there  are  24  Blocks.  However,  each  SM  can  only  take  up  to  8  Blocks,  only  512  threads  will  go  into  each  SM!    

–  For  32X32,  we  have  1024  threads  per  Block.    Only  one  block  can  fit  into  a  SM!    

–  For  16X16,  we  have  256  threads  per  Block.  Since  each  Fermi  SM  can  take  up  to  1536  threads,  it  can  take  up  to  6  Blocks  and  achieve  full  capacity  unless  other  resource  considera8ons  overrule.  

Repeat  the  analysis  for  a  Kepler  card!  

90  

Programmer  View  of  Register  File  •  There  are  up  to  32768  (32  bit)  

registers  in  each  Fermi  SM  •  There  are  up  to  65536  (32  bit)  

registers  in  each  Kepler  SM  –  This  is  an  implementa8on  decision,  

not  part  of  CUDA  –  Registers  are  dynamically  par88oned  

across  all  blocks  assigned  to  the  SM  –  Once  assigned  to  a  block,  the  register  

is  NOT  accessible  by  threads  in  other  blocks  

–  Each  thread  in  the  same  block  only  access  registers  assigned  to  itself  

4  blocks   3  blocks  

Registers  “occupancy”  example  •  If  a  Block  has  16x16  threads  and  each  thread  uses  30  registers,  how  many  blocks  can  run  on  each  Fermi  SM?  – Each  block  requires  30*256  =  7680  registers  – 32768/7680  =  4  +  change  (2048)  – So,  4  blocks  can  run  on  an  SM  as  far  as  registers  are  concerned  

•  How  about  if  each  thread  increases  the  use  of  registers  by  10%?  – Each    Block  now  requires  33*256  =  8448  registers  – 32768/8448  =  3  +  change  (7424)  – Only  three  Blocks  can  run  on  an  SM,  25%  reducaon  of  parallelism!!!  

Repeat  the  analysis  for  a  Kepler  card!  

Optimizing threads per block ü  Choose threads per block as a multiple of warp size (32)

ü  Avoid wasting computation on under-populated warps ü  Facilitates efficient memory access (coalescing )

ü  Run as many warps as possible per multiprocessor (hide latency) ü  SM can run up to 16 (on Kepler) blocks at a time

ü  Heuristics ü Minimum: 64 threads per block ü  192 or 256 threads a better choice

ü Usually still enough registers to compile and invoke successfully

ü  The right tradeoff depends on your computation, so experiment, experiment, experiment!!!

Review  on  Memory  Hierarchy  •  Local  Memory:  per-­‐thread  

–  Private  per  thread  but  slow!  –  Auto  variables,  register  spill  

•  Shared  Memory:  per-­‐block  –  Shared  by  threads  of  the  same  block  –  Fast  inter-­‐thread  communica8on  

•  Global  Memory:  per-­‐applica8on  –  Shared  by  all  threads  –  Inter-­‐Grid  communica8on  

Thread

Local Memory

Grid 0

. . . Global

Memory

. . .

Grid 1 Sequential Grids in Time

Block

Shared Memory

Op8mizing  Global  Memory  Access  • Memory  Bandwidth  is  very  high  (~  150  Gbytes/sec.)  but…  

• Latency  is  also  very  high  (hundreds  of  cycles)!  • Local  memory  has  the  same  latency!  

• Reduce  number  of  memory  transac8ons  by  applying  a  proper        memory  access  pa;ern.  

Thread  locality  is  be3er  than  data  locality  on  GPU!  

Array  of  structures  (data  locality!)  

Structure  of  arrays  (thread  locality!)  

Global  Memory  Opera8ons  

•  Two  types  of  loads:  –  Caching  

•  Default  mode  •  A;empts  to  hit  in  L1,  then  L2,  then  GMEM  •  Load  granularity  is  128-­‐byte  line  

– Non-­‐caching  •  Compile  with  –Xptxas  –dlcm=cg  op8on  to  nvcc  •  A;empts  to  hit  in  L2,  then  GMEM  

–  Do not hit in L1, invalidate the line if it’s in L1 already •  Load  granularity  is  32-­‐bytes  

•  Stores:  –  Invalidate  L1,  write-­‐back  for  L2  

Caching  Load  •  Warp  requests  32  aligned,  consecu8ve  4-­‐byte  words  •  Addresses  fall  within  1  cache-­‐line  

– Warp  needs  128  bytes  – 128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  100%  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Non-­‐caching  Load  •  Warp  requests  32  aligned,  consecu8ve  4-­‐byte  words  •  Addresses  fall  within  4  segments  

– Warp  needs  128  bytes  – 128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  100%  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Caching  Load  

•  Warp  requests  32  aligned,  permuted  4-­‐byte  words  •  Addresses  fall  within  1  cache-­‐line  

– Warp  needs  128  bytes  – 128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  100%  

...  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

addresses  from  a  warp  

0  

Non-­‐caching  Load  

•  Warp  requests  32  aligned,  permuted  4-­‐byte  words  •  Addresses  fall  within  4  segments  

– Warp  needs  128  bytes  – 128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  100%  

...  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

addresses  from  a  warp  

0  

Caching  Load  •  Warp  requests  32  misaligned,  consecu8ve  4-­‐byte  words  

•  Addresses  fall  within  2  cache-­‐lines  – Warp  needs  128  bytes  – 256  bytes  move  across  the  bus  on  misses  – Bus  u8liza8on:  50%  

96   192  128   160   224   288  256  

...  addresses  from  a  warp  

32   64  0   352  320   384   448  416  Memory  addresses  

Non-­‐caching  Load  •  Warp  requests  32  misaligned,  consecu8ve  4-­‐byte  words  •  Addresses  fall  within  at  most  5  segments  

– Warp  needs  128  bytes  – 160  bytes  move  across  the  bus  on  misses  – Bus  u8liza8on:  at  least  80%  

•  Some  misaligned  pa;erns  will  fall  within  4  segments,  so  100%  u8liza8on  

96   192  128   160   224   288  256  

...  addresses  from  a  warp  

32   64  0   352  320   384   448  416  Memory  addresses  

Caching  Load  •  All  threads  in  a  warp  request  the  same  4-­‐byte  word  •  Addresses  fall  within  a  single  cache-­‐line  

– Warp  needs  4  bytes  – 128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  3.125%  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Non-­‐caching  Load  •  All  threads  in  a  warp  request  the  same  4-­‐byte  word  •  Addresses  fall  within  a  single  segment  

– Warp  needs  4  bytes  – 32  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:  12.5%  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Caching  Load  

•  Warp  requests  32  sca;ered  4-­‐byte  words  •  Addresses  fall  within  N  cache-­‐lines  

– Warp  needs  128  bytes  – N*128  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:    128  /  (N*128)  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Non-­‐caching  Load  

•  Warp  requests  32  sca;ered  4-­‐byte  words  •  Addresses  fall  within  N  segments  

– Warp  needs  128  bytes  – N*32  bytes  move  across  the  bus  on  a  miss  – Bus  u8liza8on:    128  /  (N*32)  

...  addresses  from  a  warp  

96   192  128   160   224   288  256  32   64   352  320   384   448  416  Memory  addresses  

0  

Shared  Memory  •  On-­‐chip  memory  

–  2  orders  of  magnitude  lower  latency  than  global  memory  – Much  higher  bandwidth  than  global  memory  –  Up  to  48  KByte  per  mul8processor  (on  Fermi  and  Kepler)  

•  Allocated  per  thread-­‐block  •  Accessible  by  any  thread  in  the  thread-­‐block  

–  Not  accessible  to  other  thread-­‐blocks  •  Several  uses:  

–  Sharing  data  among  threads  in  a  thread-­‐block  –  User-­‐managed  cache  (reducing  global  memory  accesses)        __shared__    float  As[16][16];  

Shared  Memory  Example  •  Compute  adjacent_difference  

–  Given  an  array  d_in  produce  a  d_out  array  containing  the  difference:  d_out[0]=d_in[0];    d_out[i]=d_in[i]-­‐d_in[i-­‐1];  /*  i>0  */  

Exercise:  Complete  the  shared_variables.cu  code  

N-­‐Body  problems  An  N-­‐body  simulaaon  numerically  approximates  the   evoluaon   of   a   system   of   bodies   in   which  each   body   conanuously   interacts   with   every  other  body.  The  all-­‐pairs  approach  to  N-­‐body  simulaaon  is  a  brute-­‐force   technique   that   evaluates   all   pair-­‐wise  interacaons  among  the  N  bodies.    

N-­‐Body  problems  We   use   the   gravitaaonal   potenaal   to   illustrate  the  basic  form  of  computaaon  in  an  all-­‐pairs  N-­‐body  simulaaon.  

N-­‐Body  problems  To   integrate   over   ame,   we   need   the  acceleraaon  a   i  =  F   i  /mi  to  update  the  posiaon  and  velocity  of  body  i.  

Download  h3p://twin.iac.rm.cnr.it/mini_nbody.tgz  and  unpack:  tar  zxvf  mini_nbody.tgz    look  at  nbody.c    

N-­‐Body:  main  loop  void  bodyForce(Body  *p,  float  dt,  int  n)  {  for  (int  i  =  0;  i  <  n;  i++)  {          float  Fx  =  0.0f;  float  Fy  =  0.0f;  float  Fz  =  0.0f;            for  (int  j  =  0;  j  <  n;  j++)  {              float  dx  =  p[j].x  -­‐  p[i].x;              float  dy  =  p[j].y  -­‐  p[i].y;              float  dz  =  p[j].z  -­‐  p[i].z;              float  distSqr  =  dx*dx  +  dy*dy  +  dz*dz  +  SOFTENING;              float  invDist  =  1.0f  /  sqrn(distSqr);              float  invDist3  =  invDist  *  invDist  *  invDist;                Fx  +=  dx  *  invDist3;  Fy  +=  dy  *  invDist3;  Fz  +=  dz  *  invDist3;          }            p[i].vx  +=  dt*Fx;  p[i].vy  +=  dt*Fy;  p[i].vz  +=  dt*Fz;      }  }    

N-­‐Body:  parallel  processing  •  We   may   think   of   the   all-­‐pairs   algorithm   as  calculaang  each  entry   fij   in   an  NxN  grid  of   all  pair-­‐wise  forces.  

•  Each  entry  can  be  computed  independently,  so  there  is  O(N  2)  available  parallelism  

•  We  may  assign  each  body  to  a  thread  – If   the   block   size   is,   say,   256   then   the   number   of  blocks  is:  (nBodies  +255)  /  256  

N-­‐Body:  a  first  CUDA  code  __global__  void  bodyForce(Body  *p,  float  dt,  int  n)  {      int  i  =  blockDim.x  *  blockIdx.x  +  threadIdx.x;      if  (i  <  n)  {          float  Fx  =  0.0f;  float  Fy  =  0.0f;  float  Fz  =  0.0f;          for  (int  j  =  0;  j  <  n;  j++)  {              float  dx  =  p[j].x  -­‐  p[i].x;              float  dy  =  p[j].y  -­‐  p[i].y;              float  dz  =  p[j].z  -­‐  p[i].z;              float  distSqr  =  dx*dx  +  dy*dy  +  dz*dz  +  SOFTENING;              float  invDist  =  rsqr|(distSqr);              float  invDist3  =  invDist  *  invDist  *  invDist;                Fx  +=  dx  *  invDist3;  Fy  +=  dy  *  invDist3;  Fz  +=  dz  *  invDist3;          }          p[i].vx  +=  dt*Fx;  p[i].vy  +=  dt*Fy;  p[i].vz  +=  dt*Fz;      }  }  

N-­‐Body:  a  first  CUDA  code  •  Do  we  expect  this  code  to  be  efficient?  

– All  threads  read  all  data  from  the  global  memory  

•  It   is   be3er   to   serialize   some   of   the  computaaons   to   achieve   the   data   reuse  needed   to   reach   peak   performance   of   the  arithmeac   units   and   to   reduce   the   memory  bandwidth  required.  

•  To  that  purpose  let  us  introduce  the  noaon  of  computaaonal  8le  

N-­‐Body:  Computa8onal  9le  

•  A  8le  is  a  square  region  of  the  grid  of  pair-­‐wise  forces  consisang  of  p  rows  and  p  columns.    –  Only   2p   body   descripaons   are   required   to   evaluate   all   p2  interacaons  in  the  ale  (p  of  which  can  be  reused  later).    

–  These  body  descripaons  can  be  stored  in  shared  memory  or  in  registers.  

N-­‐Body:  Computa8onal  9le  

•  Each  thread  in  a  block  loads  a  body  •  All  threads  in  a  block  compute  the  interacaons  with  the  bodies  that  have  been  loaded  

N-­‐Body:  using  Shared  Memory  (1)  __global__  void  bodyForce(float4  *p,  float4  *v,  float  dt,  int  n)  {  int  i  =  blockDim.x  *  blockIdx.x  +  threadIdx.x;      if  (i  <  n)  {          float  Fx  =  0.0f;  float  Fy  =  0.0f;  float  Fz  =  0.0f;          for  (int  8le  =  0;  8le  <  gridDim.x;  8le++)  {                __shared__  float3  spos[BLOCK_SIZE];                float4  tpos  =  p[8le  *  blockDim.x  +  threadIdx.x];                spos[threadIdx.x]  =  make_float3(tpos.x,  tpos.y,  tpos.z);            __syncthreads();  

         for  (int  j  =  0;  j  <  BLOCK_SIZE;  j++)  {  

                   …..                }                    __syncthreads();          }          v[i].x  +=  dt*Fx;  v[i].y  +=  dt*Fy;  v[i].z  +=  dt*Fz;      }  }  

N-­‐Body:  using  Shared  Memory  (2)  __global__  void  bodyForce(float4  *p,  float4  *v,  float  dt,  int  n)  {  int  i  =  blockDim.x  *  blockIdx.x  +  threadIdx.x;      if  (i  <  n)  {          ....              for  (int  j  =  0;  j  <  BLOCK_SIZE;  j++)  {                  float  dx  =  spos[j].x  -­‐  p[i].x;                  float  dy  =  spos[j].y  -­‐  p[i].y;                  float  dz  =  spos[j].z  -­‐  p[i].z;                  float  distSqr  =  dx*dx  +  dy*dy  +  dz*dz  +  SOFTENING;                  float  invDist  =  rsqr|(distSqr);                  float  invDist3  =  invDist  *  invDist  *  invDist;                    Fx  +=  dx  *  invDist3;  Fy  +=  dy  *  invDist3;  Fz  +=  dz  *  invDist3;              }          ….      }  }  

N-­‐Body:  Further  op8miza8ons  

•  Tuning  of  block  size  •  Unrolling  of  the  inner  loop  

for  (int  j  =  0;  j  <  BLOCK_SIZE;  j++)  {  }  

•  Test:    1.   shmoo-­‐cpu-­‐nbody.sh  2.   cuda/shmoo-­‐cuda-­‐nbody-­‐orig.sh  3.   cuda/shmoo-­‐cuda-­‐nbody-­‐block.sh  4.   cuda/shmoo-­‐cuda-­‐nbody-­‐unroll.sh  

Shared  Memory  •  Organiza8on:  

–  32  banks,  4-­‐byte  wide  banks  –  Successive  4-­‐byte  words  belong  to  different  banks  

•  Performance:  –  4  bytes  per  bank  per  2  clocks  per  mul8processor  –  smem  accesses  are  issued  per  32  threads  (warp)  –  serializa8on:  if  N  threads  of  32  access  different  4-­‐byte  words  in  the  same  bank,  N  accesses  are  executed  serially  

– mul8cast:  N  threads  access  the  same  word  in  one  fetch  •  Could  be  different  bytes  within  the  same  word  

Bank  Addressing  Examples  •  No  Bank  Conflicts   •  No  Bank  Conflicts  

Bank  31  

Bank  7  Bank  6  Bank  5  Bank  4  Bank  3  Bank  2  Bank  1  Bank  0  

Thread  31  

Thread  7  Thread  6  Thread  5  Thread  4  Thread  3  Thread  2  Thread  1  Thread  0  

Bank  31  

Bank  7  Bank  6  Bank  5  Bank  4  Bank  3  Bank  2  Bank  1  Bank  0  

Thread  31  

Thread  7  Thread  6  Thread  5  Thread  4  Thread  3  Thread  2  Thread  1  Thread  0  

Bank  Addressing  Examples  •  2-­‐way  Bank  Conflicts   •  x  

Thread  31  Thread  30  Thread  29  Thread  28  

Thread  4  Thread  3  Thread  2  Thread  1  Thread  0  

Bank  31  

Bank  7  Bank  6  Bank  5  Bank  4  Bank  3  Bank  2  Bank  1  Bank  0  

Thread  31  

Thread  7  Thread  6  Thread  5  Thread  4  Thread  3  Thread  2  Thread  1  Thread  0  

Bank  9  Bank  8  

Bank  31  

Bank  7  

Bank  2  Bank  1  Bank  0  x8  

x8  

Shared  Memory:  Avoiding  Bank  Conflicts  

•  32x32  SMEM  array  •  Warp  accesses  a  column:  

– 32-­‐way  bank  conflicts  (threads  in  a  warp  access  the  same  bank)  

31

2 1 0

31 2 1 0

31 2 1 0

warps: 0 1 2 31

Bank 0 Bank 1 … Bank 31 2 0 1

31

Shared  Memory:  Avoiding  Bank  Conflicts  

•  Add  a  column  for  padding:  – 32x33  SMEM  array  

•  Warp  accesses  a  column:  – 32  different  banks,  no  bank  conflicts  

31 2 1 0

31 2 1 0

31 2 1 0

warps: 0 1 2 31 padding

Bank 0 Bank 1 … Bank 31

31 2 0 1

Tree-­‐Based  Parallel  Reduc8ons  in  Shared  Memory  

•  CUDA  solves  this  by  exposing  on-­‐chip  shared  memory  – Reduce  blocks  of  data  in  shared  memory  to  save  bandwidth  

4 7 5 9

11 14

25

3 1 7 0 4 1 6 3

Parallel  Reduc8on:  Interleaved  Addressing  

10 1 8 -1 0 -2 3 5 -2 -3 2 7 0 11 0 2 Values (shared memory)

0 1 2 3 4 5 6 7

11 1 7 -1 -2 -2 8 5 -5 -3 9 7 11 11 2 2 Values

0 1 2 3

18 1 7 -1 6 -2 8 5 4 -3 9 7 13 11 2 2 Values

0 1

24 1 7 -1 6 -2 8 5 17 -3 9 7 13 11 2 2 Values

0

41 1 7 -1 6 -2 8 5 17 -3 9 7 13 11 2 2 Values

Thread IDs

Step 1 Stride 1

Step 2 Stride 2

Step 3 Stride 4

Step 4 Stride 8

Thread IDs

Thread IDs

Thread IDs

Interleaved addressing results in bank conflicts

•  Arbitrarily  bad  bank  conflicts  •  Requires  barriers  if  N  >  warpsize    •  Supports  non-­‐commutaave  operators  

Parallel  Reduc8on:  Sequen8al  Addressing  

10 1 8 -1 0 -2 3 5 -2 -3 2 7 0 11 0 2 Values (shared memory)

0 1 2 3 4 5 6 7

8 -2 10 6 0 9 3 7 -2 -3 2 7 0 11 0 2 Values

0 1 2 3

8 7 13 13 0 9 3 7 -2 -3 2 7 0 11 0 2 Values

0 1

21 20 13 13 0 9 3 7 -2 -3 2 7 0 11 0 2 Values

0

41 20 13 13 0 9 3 7 -2 -3 2 7 0 11 0 2 Values

Thread IDs

Step 1 Stride 8

Step 2 Stride 4

Step 3 Stride 2

Step 4 Stride 1

Thread IDs

Thread IDs

Thread IDs

Sequential addressing is conflict free

•  No  bank  conflicts  •  Requires  barriers  if  N  >  warpsize    •  Does  not  support  non-­‐commutaave  operators  

Parallel  Prefix  Sum  (Scan)  

•  Given  an  array  A  =  [a0,  a1,  …,  an-­‐1]    and  a  binary  associa8ve  operator  ⊕  with  iden8ty  I,    

•  scan(A)  =  [I,  a0,  (a0  ⊕  a1),  …,  (a0  ⊕  a1  ⊕  …  ⊕  an-­‐2)]  

•  Example:    if  ⊕  is  addi8on,  then  scan  on  the  set  –  [3  1  7  0  4  1  6  3]  

•  returns  the  set    –  [0  3  4  11  11  15  16  22]  

O(n  log  n)  Scan  

•  Step  efficient  (log  n  steps)  •  Not  work  efficient  (n  log  n  work)  •  Requires  barriers  at  each  step  (WAR  dependencies)  

132  

Constant  Memory  

ü  Both  scalar  and  array  values  can  be  stored  ü  Constants  stored  in  DRAM,  and  cached  on  chip  

–  L1  per  SM  ü  A  constant  value  can  be  broadcast  to  all  threads  in  

a  Warp  ü  Extremely  efficient  way  of  accessing  a  value  that  is  

common  for  all  threads  in  a  block!    

     

I $ L 1

Multithreaded Instruction Buffer

R F C $

L 1 Shared Mem

Operand Select

MAD SFU

cudaMemcpyToSymbol(ds,  &hs,  sizeof(int),  0,cudaMemcpyHostToDevice);  

Texture  Memory  (and  co-­‐stars)  

•  Another  type  of  memory  system,  featuring:  – Spa8ally-­‐cached  read-­‐only  access  – Avoid  coalescing  worries  –  Interpola8on  –  (Other)  fixed-­‐func8on  capabili8es  – Graphics  interoperability  

Tradi8onal  Caching  

•  When  reading,  cache  “nearby  elements”    –  (i.e.  cache  line)  

– Memory  is  linear!  

– Applies  to  CPU,  GPU  L1/L2  cache,  etc  

Texture-­‐Memory  Caching  

•  Can  cache  “spa8ally!”  (2D,  3D)  – Specify  dimensions  (1D,  2D,  3D)  on  crea8on  

•  1D  applica8ons:    –  Interpola8on,  clipping  (later)  – Caching  when,  e.g.,  coalesced  access  is  infeasible  

Texture  Memory  

•  “Memory  is  just  an  unshaped  bucket  of  bits”  (CUDA  Handbook)  

•  Need  texture  reference  in  order  to:  –  Interpret  data  – Deliver  to  registers  

Interpolaaon  

•  Can  “read  between  the  lines!”  

h;p://cuda-­‐programming.blogspot.com/2013/02/texture-­‐memory-­‐in-­‐cuda-­‐what-­‐is-­‐texture.html  

Download  h3p://twin.iac.rm.cnr.it/readTexels.cu  

•  Seamlessly  handle  reads  beyond  region!  

Clamping  

h;p://cuda-­‐programming.blogspot.com/2013/02/texture-­‐memory-­‐in-­‐cuda-­‐what-­‐is-­‐texture.html  

“CUDA  Arrays”  

•  So  far,  we’ve  used  standard  linear  arrays  

•  “CUDA  arrays”:  – Different  addressing  calcula8on  

•  Con8guous  addresses  have  2D/3D  locality!    

– Not  pointer-­‐addressable  –  (Designed  specifically  for  texturing)  

Texture  Memory  

•  Texture  reference  can  be  a;ached  to:  – Ordinary  device-­‐memory  array  – “CUDA  array”  

•  Many  more  capabili8es  

Texturing  Example  (2D)  

h;p://www.math.ntu.edu.tw/~wwang/mtxcomp2010/download/cuda_04_ykhung.pdf  h;p://www.math.ntu.edu.tw/~wwang/mtxcomp2010/download/cuda_04_ykhung.pdf  

Texturing  Example  (2D)  

h;p://www.math.ntu.edu.tw/~wwang/mtxcomp2010/download/cuda_04_ykhung.pdf  

Texturing  Example  (2D)  

h;p://www.math.ntu.edu.tw/~wwang/mtxcomp2010/download/cuda_04_ykhung.pdf  

Texturing  Example  (2D)  

146!

Arithmetic Instruction Throughput

Number  of  operaaons  per  clock  cycle  per  mulaprocessor  

147  

Control  Flow  Instruc8ons  •  Main  performance  concern  with  branching  is  divergence  

–  Threads  within  a  single  warp  (group  of  32)  take  different  paths  –  Different  execu8on  paths  are  serialized  

•  The  control  paths  taken  by  the  threads  in  a  warp  are  traversed  one  at  a  8me  un8l  there  is  no  more.  

•  Avoid  divergence  when  branch  condi8on  is  a  func8on  of  thread  ID  –  Example  with  divergence:    

•  if (threadIdx.x > 7) { } •  This  creates  two  different  control  paths  for  threads  in  a  block  •  Branch  granularity  <  warp  size;  threads  0,  1  ,  …  7  follow  different  path  than  the  

rest  of  the  threads  8,  9,  …,  31  in  the  same  warp.  –  Example  without  divergence:  

•  if (threadIdx.x / WARP_SIZE > 2) { } •  Also  creates  two  different  control  paths  for  threads  in  a  block •  Branch  granularity  is  a  whole  mul8ple  of  warp  size;  all  threads  in  any  given  

warp  follow  the  same  path  

148  

Warp  Vote  Func8ons  and  other  useful  primi8ves  

•  int __all(predicate) return  true  iff  the  predicate  evaluates  as  true  for  all  threads  of  a    warp    

•  int __any(predicate) return  true  iff  the  predicate  evaluates  as  true  for  any  threads  of  a    warp    

•  unsigned int __ballot(predicate) return  an  unsigned  int  whose  Nth  is  set  iff  predicate  evaluates  as  true  for  the  Nth  thread  of  the  warp.  

•  int __popc(int x) POPula8on  Count:  return  the  number  of  bits  that  are  set  to  1  in  the  32-­‐bit  integer  x.  

•  int __clz(int x) Count  Leading  Zero:  return  the  number  of  consecu8ve  zero  bits  beginning  at  the  most  significant  bit  of  the  32-­‐bit  integer  x.  

149  

Shuffle  Instruc8on  

•  Instruc8on  to  exchange  data  in  a  warp    

•  Threads  can  “read”  other  threads’  registers    

•  No  shared  memory  is  needed  

•  It  is  available  star8ng  from  SM  3.0    •  Kepler’s  shuffle  instruc8on  (SHFL)  enables  a  thread  to  directly  

read  a  register  from  another  thread  in  the  same  warp  

150  

Shuffle  Instruc8on  

•  4  Variants  (idx,  up,  down,  bfly):    

151  

A  simple  shuffle  example  (1)  

int  __shfl_down(int  var,  unsigned  int  delta,  int  width=warpSize);  

•  __shfl_down()  calculates  a  source  lane  ID  by  adding  delta  to  the  caller’s  lane  ID    –  the  lane  ID  is  a  thread’s  index  within  its  warp,  from  0  to  31.  Can  be  computed  as    threadIdx.x  %  32;  

•  The  value  of  var  held  by  the  resul8ng  lane  ID  is  returned:  this  has  the  effect  of  shi�ing  var  down  the  warp  by  delta  lanes  

•  If  the  source  lane  ID  is  out  of  range  or  the  source  thread  has  exited,  the  calling  thread’s  own  var  is  returned.  

152  

A  simple  shuffle  example  (2)  

int  i  =  threadIdx.x  %  32;  int  j  =  __shfl_down(i,  2,  8);  

153  

Using  shuffle  for  a  reduce  opera8on  

__inline__  __device__  int  warpReduceSum(int  val)  {        for  (int  offset  =  warpSize/2;  offset  >  0;  offset  /=  2)  {              val  +=  __shfl_down(val,  offset);  

 }                  return  val;  }  

154  

Using  shuffle  for  an  all  reduce  opera8on  

__inline__  __device__  int  warpAllReduceSum(int  val)  {        for  (int  mask  =  warpSize/2;  mask  >  0;  mask  /=  2)  {              val  +=  __shfl_xor(val,  mask);  

 }    return  val;  

}  

If   all   threads   in   the   warp   need   the   result,   it   possible   to  implement  an  “all  reduce”  within  a  warp  using  __shfl_xor()  instead  of  __shfl_down(),  like  the  following:  

155  

Block  reduce  by  using  shuffle  •  First  reduce  within  warps.    •  Then  the  first  thread  of  each  warp  writes  its  par8al  sum  to  shared  

memory.    •  Finally,  a]er  synchronizing,  the  first  warp  reads  from  shared  

memory  and  reduces  again.  __inline__  __device__  int  blockReduceSum(int  val)  {    

   staac  __shared__  int  shared[32];  //  Shared  mem  for  32  paraal  sums      int  lane  =  threadIdx.x  %  warpSize;      int  wid  =  threadIdx.x  /  warpSize;      val  =  warpReduceSum(val);                    //  Each  warp  performs  paraal  reducaon      if  (lane==0)  shared[wid]=val;  //  Write  reduced  value  to  shared  memory      __syncthreads();                            //  Wait  for  all  paraal  reducaons                //read  from  shared  memory  only  if  that  warp  existed      val  =  (threadIdx.x  <  blockDim.x  /  warpSize)  ?  shared[lane]  :  0;      if  (wid==0)  val  =  warpReduceSum(val);  //Final  reduce  within  first  warp      return  val;  

}  

156  

Reducing  Large  Arrays  •  Reducing  across  blocks  means  global  communica8on  which  

means  synchronizing  the  grid  •  Two  possible  approaches:  

–  Breaking  the  computa8on  into  two  separate  kernel  launches  –  Use  atomics  opera8ons  

•  Exercise  (home  work!):  write  a  deviceReduce  that  works  across  a  complete  grid  (many  thread  blocks).    

157  

Implement  SHFL  for  64  bits  Numbers    

Generic  SHFL:  h3ps://github.com/BryanCatanzaro/generics  

158  

Opamized  Filtering  Let   us   have   a   source   array,   src,   containing   n   elements,   and   a  predicate,   and   we   need   to   copy   all   elements   of   src   sa8sfying   the  predicate   into   the   des8na8on   array,   dst.   For   the   sake   of   simplicity,  assume   that   dst   has   length   of   at   least   n   and   that   the   order  of  elements  in  the  dst  array  does  not  ma;er.  If  the  predicate  is  src[i]>0,  a  simple  CPU  solu8on  is:  int  filter(int  *dst,  const  int  *src,  int  n)  {      int  nres  =  0;      for  (int  i  =  0;  i  <  n;  i++)          if  (src[i]  >  0)              dst[nres++]  =  src[i];      //  return  the  number  of  elements  copied      return  nres;  }  

159  

Opamized  Filtering  on  GPU:    global  memory  

A  straigh|orward  approach  is  to  use  a  single  global  counter  and   atomically   increment   it   for   each   new   element  wri;en  int  the  dst  array:  __global__    void  filter_k(int  *dst,  int  *nres,  const  int  *src,  int  n)  {      int  i  =  threadIdx.x  +  blockIdx.x  *  blockDim.x;      if(i  <  n  &&  src[i]  >  0)          dst[atomicAdd(nres,  1)]  =  src[i];  }  Here  atomic  opera8ons  are  clearly  a  bo;leneck,  and  need  to  be  removed  or  reduced  to  increase  performance  

Opamized  Filtering  on  GPU:    using  shared  memory  

_global__    void  filter_shared_k(int  *dst,  int  *nres,  const  int*  src,  int  n)  {      __shared__  int  l_n;      int  i  =  blockIdx.x  *  (NPER_THREAD  *  BS)  +  threadIdx.x;        for  (int  iter  =  0;  iter  <  NPER_THREAD;  iter++)  {          //  zero  the  counter          if  (threadIdx.x  ==  0)    l_n  =  0;          __syncthreads()          //  get  the  value,  evaluate  the  predicate,  and          //  increment  the  counter  if  needed          int  d,  pos;          if(i  <  n)  {              d  =  src[i];              if(d  >  0)  pos  =  atomicAdd(&l_n,  1);          }  

 __syncthreads();          //  leader  increments  the  global  counter          if(threadIdx.x  ==  0)    l_n  =  atomicAdd(nres,  l_n);          __syncthreads();            //  threads  with  true  predicates  write  their  elements          if(i  <  n  &&  d  >  0)  {              pos  +=  l_n;  //  increment  local  pos  by  global  counter              dst[pos]  =  d;          }          __syncthreads();            i  +=  BS;      }  }  

160  

Opamized  Filtering  on  GPU:    warp-­‐aggregated  atomics  

Warp   aggrega9on   is   the   process   of   combining   atomic  opera8ons  from  mul8ple  threads  in  a  warp  into  a  single  atomic.  1.  Threads  in  the  warp  elect  a  leader  thread.  2.  Threads  in  the  warp  compute  the  total  atomic  increment  for  

the  warp.  3.  The  leader  thread  performs  an  atomic  add  to  compute  the  

offset  for  the  warp.  4.  The  leader  thread  broadcasts  the  offset  to  all  other  threads  

in  the  warp.  5.  Each  thread  adds  its  own  index  within  the  warp  to  the  warp  

offset  to  get  its  posi8on  in  the  output  array.  

161  

Step  1:  elect  a  leader  

The  thread  iden8fier  within  a  warp  can  be  computed  as    #define  WARP_SZ  32  __device__  inline  int  lane_id(void)  {  return  threadIdx.x  %  WARP_SZ;  }  

A  reasonable  choice  is  to  select  as  leader  the  lowest  ac9ve  lane  int  mask  =  __ballot(1);    //  mask  of  acave  lanes  int  leader  =  __ffs(mask)  -­‐  1;    //  -­‐1  for  0-­‐based  indexing  __ffs(int  v)  returns  the  1-­‐based  posi8on  of  the  lowest  bit  that  is  set  in  v,  or  0  if  no  bit  is  set  (ffs  stands  for  find  first  set)  

162  

Step  2:  compute  the  total  increment  

The  total  increment  for  the  warp  is  equal  to  the  number  of  ac8ve  lanes  which  hold  true  predicates.  The  value  can  be  obtained  by  using  the  __popc  intrinsic  which   returns   the   number   of   bits   set   in   the   binary  representa8on  of  its  argument:  int  change  =  __popc(mask);  

163  

Step  3:  performing  the  atomic  

Only  the  leader  lane  performs  the  atomic  opera8on  int  res;  if(lane_id()  ==  leader)      res  =  atomicAdd(ctr,  change);    //  ctr  is  the  pointer  to  the  counter  

164  

Step  4:  broadcasang  the  result  

We  can   implement   the  broadcast  efficiently  using   the  shuffle  intrinsic  __device__    int  warp_bcast(int  v,  int  leader)  {  return  __shfl(v,  leader);  }      

 

165  

Step  5:  compuang  the  index    for  each  lane  

This   is   a   bit   tricky:   we   must   compute   the   output  index   for   each   lane,   by   adding   the   broadcast   counter  value  for  the  warp  to  the  lane’s  intra-­‐warp  offset  •  the   intra-­‐warp   offset   is   the   number   of   ac9ve   lanes  with  IDs  lower  than  current  lane  ID  

•  We   can   compute   it   by   applying   a  mask   that   selects  only   lower   lane   IDs,  and  then  applying  the  __popc()  intrinsic  to  the  result.  

res  +=  __popc(mask  &  ((1  <<  lane_id())  –  1));        

166  

Final  code    //  warp-­‐aggregated  atomic  increment  __device__  int  atomicAddInc(int  *ctr)  {      int  mask  =  __ballot(1);      //  select  the  leader      int  leader  =  __ffs(mask)  –  1;      //  leader  does  the  update      int  res;      if(lane_id()  ==  leader)    res  =  atomicAdd(ctr,  __popc(mask));      //  broadcast  result      res  =  warp_bcast(res,  leader);      //  each  thread  computes  its  own  value      return  res  +  __popc(mask  &  ((1  <<  lane_id())  –  1));  }  //  atomicAddInc  Download  h3p://twin.iac.rm.cnr.it/filter.cu  

167  

Exercise  Write   a   shuffle-­‐based   kernel   for   the   N-­‐body  problem.  Hint:  start  from  the  nbody-­‐unroll.cu  code    

168  

169

CUDA  Dynamic  Parallelism  •  A facility introduced by NVIDIA in their GK110 chip/architecture •  Allows a kernel to call another kernel from within it without returning

the host. •  Each such kernel call has the same calling construction - the grid

and block sizes and dimensions are set at the time of the call. •  Facility allows computations to be done with dynamically altering

grid structures and recursion, to suit the computation. •  For example allows a 2/3D simulation mesh to be non-uniform with

increased precision at places of interest, see previous slides.

170

Host code

kernel1<<< G1,B1>>>(…) ;

__global__ void kernel1 (…)

kernel2<<< G2,B2>>>(…) ;

return 0; Dynamic  Parallelism  

__global__ void kernel2 (…)

kernel3<<< G3,B3>>>(…) ;

return 0;

Kernels calling  other  kernels  

Notice the kernel call is a standard syntax and allows each kernel call to have different grid/block structures

Device code

Nested depth limited by memory and <= 63 or 64

171

Host code

kernel1<<< G1,B1>>>(…) ;

__global__ void kernel1 (…)

kernel1<<< G2,B2>>>(…) ;

return 0;

Dynamic  Parallelism  

Recursion  is  allowed  

Device code

172 Derived from Fig 20.4 of Kirk and Hwu 2nd Ed.

Grid A (Parent)

Grid B (Child)

Grid B launch

Grid A threads

Grid B completes

Host (CPU) thread

Parent-­‐Child  Launch  Nes8ng  

Grid A launch

Time

Grid B threads

Grid A completes

“Implicit synchronization between parent and child forcing parent to wait until all children exit before it can exit.”

173!

Host-Device Data Transfers

ü  Device-to-host memory bandwidth much lower than device-to-device bandwidth ü  8 GB/s peak (PCI-e x16 Gen 2) vs. ~ 150 GB/s peak

ü  Minimize transfers ü  Intermediate data can be allocated, operated on, and deallocated

without ever copying them to host memory

ü  Group transfers ü  One large transfer is usually much better than many small ones

Synchronizing  GPU  and  CPU  •  All  kernel  launches  are  asynchronous!  

–  control  returns  to  CPU  immediately  •  Be  careful  measuring  8mes!  

–  kernel  starts  execu8ng  once  all  previous  CUDA  calls  have  completed  •  Memcopies  are  synchronous  

–  control  returns  to  CPU  once  the  copy  is  complete  –  copy  starts  once  all  previous  CUDA  calls  have  completed  

•  cudaDeviceSynchronize()  –  CPU  code  –  blocks  un8l  all  previous  CUDA  calls  complete  

•  Asynchronous  CUDA  calls  provide:  –  non-­‐blocking  memcopies  –  ability  to  overlap  memcopies  and  kernel  execuaon  

175!

Page-Locked Data Transfers ü  cudaMallocHost() allows allocation of page-locked (“pinned”) host

memory ü  Same syntax of cudaMalloc() but works with CPU pointers and

memory

ü  Enables highest cudaMemcpy performance ü  3.2 GB/s on PCI-e x16 Gen1 ü  6.5 GB/s on PCI-e x16 Gen2

ü  Use with caution!!! ü  Alloca8ng  too  much  page-­‐locked  memory  can  reduce  overall  

system  performance    

See  and  test  the  bandwidth.cu sample  

176!

Overlapping Data Transfers and Computation

ü Async Copy and Stream API allow overlap of H2D or D2H data transfers with computation ü CPU computation can overlap data transfers on all CUDA capable

devices ü Kernel computation can overlap data transfers on devices with

“Concurrent copy and execution” (compute capability >= 1.1)

ü Stream = sequence of operations that execute in order on GPU ü Operations from different streams can be interleaved ü Stream ID used as argument to async calls and kernel launches

ü  Asynchronous host-device memory copy returns control immediately to CPU ü  cudaMemcpyAsync(dst, src, size, dir, stream); ü  requires pinned host memory

(allocated with cudaMallocHost)

ü  Overlap CPU computation with data transfer ü  0 = default stream

cudaMemcpyAsync(a_d, a_h, size, cudaMemcpyHostToDevice, 0);

kernel<<<grid, block>>>(a_d);

cpuFunction();

177!

Asynchronous Data Transfers overlapped

178!

Overlapping  kernel  and  data  transfer  

ü  Requires: ü  Kernel and transfer use different, non-zero streams

ü  For  the  crea8on  of  a  stream:  cudaStreamCreate ü  Remember that a CUDA call to stream 0 blocks until all previous calls

complete and cannot be overlapped

ü  Example: cudaStream_t stream1, stream2; cudaStreamCreate(&stream1); cudaStreamCreate(&stream2);

cudaMemcpyAsync(dst, src, size, dir,stream1); kernel<<<grid, block, 0, stream2>>>(…);

overlapped

Exercise:  read,  compile  and  run  simpleStreams.cu  

Device  Management  

•  CPU  can  query  and  select  GPU  devices  –  cudaGetDeviceCount(  int*  count  )  –  cudaSetDevice(  int  device  )  –  cudaGetDevice(  int    *current_device  )  –  cudaGetDeviceProper8es(  cudaDeviceProp*  prop,    int  device  )  –  cudaChooseDevice(  int  *device,  cudaDeviceProp*  prop  )  

•  Mul8-­‐GPU  setup  –  device  0  is  used  by  default  –  Star8ng  on  CUDA  4.0  a  CPU  thread  can  control  more  than  one  GPU  –  mul8ple  CPU  threads  can  control  the  same  GPU    

– the  driver  regulates  the  access.  

Device  Management  (sample  code)  

   int  cudadevice;      struct  cudaDeviceProp  prop;      cudaGetDevice(  &cudadevice  );      cudaGetDeviceProper8es    (&prop,  cudadevice);      mpc=prop.mul8ProcessorCount;      mtpb=prop.maxThreadsPerBlock;      shmsize=prop.sharedMemPerBlock;      prin|("Device  %d:  number  of  mul8processors  %d\n"                          "max  number  of  threads  per  block  %d\n"                          "shared  memory  per  block  %d\n",                              cudadevice,  mpc,  mtpb,  shmsize);  

Exercise:  compile  and  run  the  enum_gpu.cu  code  

CUDA  libraries    

ü  CUDA    includes  a  number  of  widely  used  libraries  –  CUBLAS:  BLAS  implementa8on  –  CUFFT:        FFT  implementa8on  –  CURAND:  Random  number  genera8on  –  Etc.  

ü  Thrust  h;p://docs.nvidia.com/cuda/thrust  –  Reduc8on  –  Scan  –  Sort  –  Etc.  

CUDA  libraries    

ü  There  are  two  types  of  run8me  math  opera8ons  –  __func():  direct  mapping  to  hardware  ISA  

•  Fast  but  low  accuracy  •  Examples:  __sin(x),  __exp(x),  __pow(x,y)  

–  func()  :  compile  to  mul8ple  instruc8ons  •  Slower  but  higher  accuracy  (5  ulp,  units  in  the  least  place,  or  

less)  •  Examples:  sin(x),  exp(x),  pow(x,y)    

ü The -use_fast_math compiler option forces every func() to compile to __func()

CURAND  Library  

•  Library  for  genera8ng  random  numbers  

•  Features:  –  XORWOW  pseudo-­‐random  generator  –  Sobol’  quasi-­‐random  number  generators  – Host  API  for  genera8ng  random  numbers  in  bulk  –  Inline  implementa8on  allows  use  inside  GPU  func8ons/kernels  

–  Single-­‐  and  double-­‐precision,  uniform,  normal  and  log-­‐normal  distribu8ons  

CURAND  Features  

–  Pseudo-­‐random  numbers  George  Marsaglia.  Xorshi�  RNGs.  Journal  of  Sta9s9cal  SoRware,  8(14),  2003.  Available  at  h;p://www.jstatso�.org/v08/i14/paper.  

–  Quasi-­‐random  32-­‐bit  and  64-­‐bit    Sobol’  sequences  with  up  to  20,000  dimensions.  

–  Host  API:  call  kernel  from  host,  generate  numbers  on  GPU,  consume  numbers  on  host  or  on  GPU.  

–  GPU  API:  generate  and  consume  numbers  during  kernel  execu8on.  

CURAND  use  1.  Create  a  generator:      

curandCreateGenerator()  2.  Set  a  seed:        

curandSetPseudoRandomGeneratorSeed()  3.  Generate  the  data  from  a  distribu8on:    

curandGenerateUniform()/(curandGenerateUniformDouble():  Uniform  curandGenerateNormal()/cuRandGenerateNormalDouble():  Gaussian  curandGenerateLogNormal/curandGenerateLogNormalDouble():  Log-­‐Normal  

4.  Destroy  the  generator:    curandDestroyGenerator()  

Example  CURAND  Program:    Compu8ng  π  with  MonteCarlo  method  

π = 4 * (Σ red  points)/(Σ points)    1.  Generate    0  <=  x  <  1  2.   Generate    0  <=  y  <  1  3.  Compute    z  =  x2  +  y2  4.  If  z  <=  1  the  point  is  red    Accuracy  increases  slowly…    

Example  CURAND  Program:    Compu8ng  π  with  MonteCarlo  method  

Compare  the  speed  of  pigpu  and  picpu  cd  Pi    gcc  –o  picpu  –O3  picpu.c  -­‐lm  nvcc  –o  pigpu  –O3  pigpu.cu  –arch=sm_13  -­‐lm  Run  for  100000  itera8ons  Look  at  pigpu  and  try  to  understand  why  is  it  so  SLOW…?    

Mul8-­‐GPU  Memory  

ü GPUs  do  not  share  global  memory  ü But  star8ng  on  CUDA  4.0  one  GPU  can  copy  data  from/to  another  GPU  memory  directly  if  the  GPU  are  connected  to  the  same  PCIe  switch  

ü Inter-­‐GPU  communica8on  ü Applica8on  code  is  responsible  for  copying/moving  data  between  GPU  

ü Data  travel  across  the  PCIe  bus  ü Even  when  GPUs  are  connected  to  the  same  PCIe  switch!  

Mul8-­‐GPU  Environment      

ü GPUs  have  consecu8ve  integer  IDs,  star8ng  with  0  ü Star8ng  on  CUDA  4.0,  a  host  thread  can  maintain  more  than  one  GPU  context  at  a  8me  ü CudaSetDevice  allows  to  change  the  “ac8ve”  GPU  (and  so  the  context)    

ü Device  0  is  chosen  when  cudaSetDevice  is  not  called  ü Remember  that  mul8ple  host  threads  can  establish  contexts  with  the  same  GPU  ü Driver  handles  8me-­‐sharing  and  resource  par88oning  unless  the  GPU  is  in  exclusive  mode  

Mul8-­‐GPU  Environment:  a  simple  approach      //  Run  independent  kernel  on  each  CUDA  device  

int  numDevs  =  0;    cudaGetNumDevices(&numDevs);  ...    for  (int  d  =  0;  d  <  numDevs;  d++)  {    cudaSetDevice(d);    kernel<<<blocks,  threads>>>(args);  

}  

Exercise:  Compare  the  dot_simple_mul8block.cu  and  dot_mul8gpu.cu  programs.  Compile  and  run.  A;en8on!  Add  the  –arch=sm_35  op8on:  nvcc  –o  dot_mulagpu  dot_mulagpu.cu  –arch=sm_35  

General  Mula-­‐GPU  Programming  Pa3ern  

•  The  goal  is  to  hide  communicaaon  cost  – Overlap  with  computa8on  

•  So,  every  ame-­‐step,  each  GPU  should:  –  Compute  parts  (halos)  to  be  sent  to  neighbors  –  Compute  the  internal  region  (bulk)  –  Exchange  halos  with  neighbors  

•  Linear  scaling  as  long  as  internal-­‐computaaon  takes  longer  than  halo  exchange  – Actually,  separate  halo  computa8on  adds  some  overhead  

overlapped  

Example:  Two  Subdomains  

Example:  Two  Subdomains  

send   compute  send  compute  

compute  compute  

Phase  1  

Phase  2  

GPU-­‐0:  green  subdomain  GPU-­‐1:  grey  subdomain  

CUDA  Features  Useful  for  Mula-­‐GPU  

•  Control  mulaple  GPUs  with  a  single  CPU  thread  – Simpler  coding:  no  need  for  CPU  mul8threading  

•  Streams:  – Enable  execu8ng  kernels  and  memcopies  concurrently  – Up  to  2  concurrent  memcopies:  to/from  GPU  

•  Peer-­‐to-­‐Peer  (P2P)  GPU  memory  copies  – Transfer  data  between  GPUs  using  PCIe  P2P  support  – Done  by  GPU  DMA  hardware  –  host  CPU  is  not  involved  

•  Data  traverses  PCIe  links,  without  touching  CPU  memory  – Disjoint  GPU-­‐pairs  can  communicate  simultaneously  

Peer-­‐to-­‐Peer  GPU  Communica8on  •  Up  to  CUDA  4.0,  GPU  could  not  

exchange  data  directly.              

Example:  read  p2pcopy.cu,  compile  nvcc  –o  p2pcopy  p2pcopy.cu  –arch=sm_35  

•  With  CUDA  >=  4.0  two  GPU  connected  to  the  same  PCI/express  switch  can  exchange  data  directly.              

cudaStreamCreate(&stream_on_gpu_0);  cudaSetDevice(0);  cudaDeviceEnablePeerAccess(  1,  0  );  cudaMalloc(d_0,  num_bytes);  /*  create  array  on  GPU  0  */  cudaSetDevice(1);  cudaMalloc(d_1,  num_bytes);    /*  create  array  on  GPU  1  */  cudaMemcpyPeerAsync(  d_0,  0,  d_1,  1,  num_bytes,  stream_on_gpu_0  );                                                                                    /*  copy  d_1  from  GPU  1  to  d_0  on  GPU  0:  pull  copy  */    

What  does  the  code  look  like?  

•  Arguments  to  the  cudaMemcpyPeerAsync()  call:  –  Dest  address,  dest  GPU,  src  addres,  src  GPU,  num  bytes,  stream  ID  

•  The  middle  loop  isn’t  necessary  for  correctness  –  Improves  performance  by  preven8ng  the  two  stages  from  interfering  with  

each  other  (15  vs.  11  GB/s  using  4-­‐GPU  example)  

for( int i=0; i<num_gpus-1; i++ ) // “right” stage cudaMemcpyPeerAsync( d_a[i+1], gpu[i+1], d_a[i], gpu[i], num_bytes, stream[i] );

for( int i=0; i<num_gpus; i++ ) cudaStreamSynchronize( stream[i] );

for( int i=1; i<num_gpus; i++ ) // “left” stage cudaMemcpyPeerAsync( d_b[i-1], gpu[i-1], d_b[i], gpu[i], num_bytes, stream[i] );

Code  Pa3ern  for  Mula-­‐GPU  

•  Every  ame-­‐step  each  GPU  will:  – Compute  halos  to  be  sent  to  neighbors  (stream  h)  – Compute  the  internal  region  (stream  i)  – Exchange  halos  with  neighbors  (stream  h)  

Code  For  Looping  through  Time  for( int istep=0; istep<nsteps; istep++) {

for( int i=0; i<num_gpus; i++ ) { cudaSetDevice( gpu[i] ); kernel<<<..., stream_halo[i]>>>( ... ); kernel<<<..., stream_halo[i]>>>( ... ); cudaStreamQuery( stream_halo[i] ); kernel<<<..., stream_internal[i]>>>( ... ); }

for( int i=0; i<num_gpus-1; i++ ) cudaMemcpyPeerAsync( ..., stream_halo[i] ); for( int i=0; i<num_gpus; i++ ) cudaStreamSynchronize( stream_halo[i] ); for( int i=1; i<num_gpus; i++ ) cudaMemcpyPeerAsync( ..., stream_halo[i] );

for( int i=0; i<num_gpus; i++ ) { cudaSetDevice( gpu[i] ); cudaDeviceSynchronize(); // swap input/output pointers }

}

Compute  halos  

Exchange  halos  

Synchronize  before  next  step  

Compute  internal  subdomain  

Code  For  Looping  through  Time  •  Two  CUDA  streams  per  GPU  

–  halo  and  internal  –  Each  in  an  array  indexed  by  GPU  

–  Calls  issued  to  different  streams  can  overlap  

•  Prior  to  ame-­‐loop  –  Create  streams  –  Enable  P2P  access  

   

for( int istep=0; istep<nsteps; istep++) {

for( int i=0; i<num_gpus; i++ ) { cudaSetDevice( gpu[i] ); kernel<<<..., stream_halo[i]>>>( ... ); kernel<<<..., stream_halo[i]>>>( ... ); cudaStreamQuery( stream_halo[i] ); kernel<<<..., stream_internal[i]>>>( ... ); }

for( int i=0; i<num_gpus-1; i++ ) cudaMemcpyPeerAsync( ..., stream_halo[i] ); for( int i=0; i<num_gpus; i++ ) cudaStreamSynchronize( stream_halo[i] ); for( int i=1; i<num_gpus; i++ ) cudaMemcpyPeerAsync( ..., stream_halo[i] );

for( int i=0; i<num_gpus; i++ ) { cudaSetDevice( gpu[i] ); cudaDeviceSynchronize(); // swap input/output pointers }

}

GPUs  in  Different  Cluster  Nodes  

•  Add  network  communicaaon  to  halo-­‐exchange  –  Transfer  data  from  “edge”  GPUs  of  each  node  to  CPU  – Hosts  exchange  via  MPI  (or  other  API)  –  Transfer  data  from  CPU  to  “edge”  GPUs  of  each  node  

•  With  CUDA  >=  4.0,  transparent  interoperability  between  CUDA  and  Infiniband    export  CUDA_NIC_INTEROP=1    Let  MPI  use,  for  Remote  Direct  Memory  Access  opera8ons,  memory  pinned  by  GPU    

Inter-­‐GPU  Communica8on  with  MPI  •  Example:  simpleMPI.c  

–  Generate  some  random  numbers  on  one  node.  –  Dispatch  them  to  all  nodes.  –  Compute  their  square  root  on  each  node's  GPU.  –  Compute  the  average  of  the  results  by  using  MPI.  To  compile:  

$nvcc  -­‐c  simpleCUDAMPI.cu      

         $mpic++  -­‐o  simpleMPI  simpleMPI.c  simpleCUDAMPI.o    

 -­‐L/usr/local/cuda/lib64/  -­‐lcudart    To  run:  

$mpirun  –np  2  simpleMPI  

On  a  single  line!  

GPU-­‐aware  MPI  

•  Support    GPU    to    GPU    communica8on    through    standard    MPI      interfaces:  –  e.g.    enable    MPI_Send,    MPI_Recv    from/to    GPU    memory  – Made  possible  by  Unified  Virtual  Addressing  (UVA)  in  CUDA  4.0  

•  Provide    high    performance    without    exposing    low    level    details      to    the    programmer:  –  Pipelined    data    transfer    which    automa8cally  provides    op8miza8ons      inside    MPI    library    without    user    tuning  

From:  MVAPICH2-­‐GPU:    Op8mized  GPU  to  GPU  Communica8on  for  InfiniBand  Clusters  H  Wang,  S.  Potluri,  M    Luo,  A.K.  Singh,    S.  Sur,  D.    K.    Panda  

Naive  code  with  “old  style”  MPI    

•  Sender:        cudaMemcpy(s_buf,    s_device,    size,  cudaMemcpyDeviceToHost);      MPI_Send(s_buf,    size,    MPI_CHAR,    1,    1,    MPI_COMM_WORLD);  

•  Receiver:    MPI_Recv(r_buf,    size,    MPI_CHAR,    0,    1,    MPI_COMM_WORLD,    &req);  cudaMemcpy(r_device,    r_buf,    size,    cudaMemcpyHostToDevice);  

Opamized  code  with  “old  style”  MPI    

•  Pipelining    at    user    level    with    non-­‐blocking    MPI    and    CUDA    func8ons  

•  Sender:        for    (j    =    0;    j    <    pipeline_len;    j++)        cudaMemcpyAsync(s_buf+j*block_sz,s_device+j*block_sz,…);    for    (j    =    0;    j    <    pipeline_len;    j++)  {      while    (result    !=    cudaSuccess)    {        results  =  cudaStreamQuery(..);        if  (j>0)  MPI_Test(…);        }      MPI_Isend(s_buf+j*block_sz,block_sz,MPI_CHAR,1,1,…)    }    MPI_Waitall();  

Code  with  GPU-­‐aware  MPI    

•  Standard  MPI  interfaces  from  device  buffers    Sender:        MPI_Send(s_dev,    size,    MPI_CHAR,    1,    1,    MPI_COMM_WORLD);  

 Receiver:    MPI_Recv(r_dev,    size,    MPI_CHAR,    0,    1,    MPI_COMM_WORLD,    &req);    

• MPI    library  can  automa8cally  differen8ate    between    device    memory    and      host    memory      • Overlap    CUDA    copy    and    RDMA    transfer  • Pipeline    DMA    of    data    from    GPU    and    InfiniBand    RDMA  

229  

Major  GPU  Performance  Detractors  

ü  Some8mes  it  is  be;er  to  recompute  than  to  cache  ü GPU  spends  its  transistors  on  ALUs,  not  on  cache!  

ü Do  more  computa8on  on  the  GPU  to  avoid  costly  data  transfers  ü Even   low  parallelism  computa8ons   can  at  8mes  be  faster  than  transferring  data  back  and  forth  to  host  

230  

Major  GPU  Performance  Detractors  

ü  Par88on  your  computa8on  to  keep  the  GPU  mul8processors  equally  busy  ü Many  threads,  many  thread  blocks  

ü  Keep  resource  usage  low  enough  to  support  mul8ple  ac8ve  thread  blocks  per  mul8processor  

•  Max  number  of  blocks  per  SM:  16  (Kepler)  •  Max  number  of  threads  per  SM:  2048  (Kepler)  •  Max  number  of  threads  per  block:  1024  (Fermi  &  Kepler)  •  Total  number  of  registers  per  SM:  65536  (Kepler)  •  Size  of  shared  memory  per  SM:  48  Kbyte  (Fermi  &  Kepler)  

A  useful  tool:  the  CUDA  Profiler  ü What  it  does:  

ü Provide  access  to  hardware  performance  monitors  ü Pinpoint  performance  issues  and  compare  across  implementa8ons  

ü Two  interfaces:  ü Text-­‐based:    

ü Built-­‐in  and  included  with  compiler  export  CUDA_PROFILE=1  (bash)  or  setenv  CUDA_PROFILE  1  (tcsh)  export    CUDA_PROFILE_LOG=XXXXXX  or  setenv  CUDA_PROFILE_LOG  XXXXX  where  XXXXX  is  any  file  name  you  want  

ü GUI:  specific  for  each  pla|orm  

Exercise:  run  the  MMGlob  program  under  the  control  of    the  profiler  

Final  CUDA  Highlights    

•  The  CUDA  API  is  a  minimal  extension  of  the  ANSI  C  programming  language                        Low  learning  curve  

•  The  memory  hierarchy  is  directly  exposed  to  the  programmer  to  allow  op8mal  control  of  data  flow                  Huge  differences  in  performance  with,  apparently,  8ny  changes  to  the  data  organiza8on.  

What  is  missing?  • ECC  effect  • Debugging  and  Advanced  Profiing  • Single  virtual  address  space  and  other  variants  of  memory  copy  opera8ons  • Many  GPU  libraries  

Thanks!