introduction to cuda (2 of 2)
DESCRIPTION
Introduction to CUDA (2 of 2). Patrick Cozzi University of Pennsylvania CIS 565 - Fall 2012. Announcements. Open pull request for Project 0 Project 1 released. Due Sunday 09/30 Not due Tuesday, 09/25 Code on GitHub Email grade to Karl. Agenda. Built-ins and functions - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/1.jpg)
Introduction to CUDA (2 of 2)
Patrick CozziUniversity of PennsylvaniaCIS 565 - Fall 2012
![Page 2: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/2.jpg)
Announcements
Open pull request for Project 0 Project 1 released. Due Sunday 09/30
Not due Tuesday, 09/25Code on GitHubEmail grade to Karl
2
![Page 3: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/3.jpg)
Agenda
Built-ins and functions Synchronizing threads Scheduling threads Memory model Matrix multiply revisited Atomic functions
3
![Page 4: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/4.jpg)
Functional Declarations
Executed on the: Only callable from the:
__global__ void KernelFunc() device host__device__ float DeviceFunc() device device__host__ float HostFunc() host host
See Appendix B.1 in the NVIDIA CUDA C Programming Guide for more details4
![Page 5: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/5.jpg)
Functional Declarations
__global__ Must return void
__device__ Inlined by default
See Appendix B.1 in the NVIDIA CUDA C Programming Guide for more details5
![Page 6: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/6.jpg)
Functional Declarations
What do these do? __global__ __host__ void func() __device__ __host__ void func()
6
![Page 7: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/7.jpg)
Functional Declarations
What do these do? __global__ __host__ void func() __device__ __host__ void func()
Code from http://developer.download.nvidia.com/compute/cuda/3_2_prod/toolkit/docs/CUDA_C_Programming_Guide.pdf7
![Page 8: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/8.jpg)
Functional Declarations
Global and device functions No recursion (except Fermi) No static variables No malloc() Careful with function calls through pointers
We’ll see similar constraints in GLSL
8
![Page 9: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/9.jpg)
Vector Types
char[1–4], uchar[1–4] short[1–4], ushort[1–4] int[1–4], uint[1–4] long[1–4], ulong[1–4] longlong[1–4], ulonglong[1–4] float[1–4] double1, double2
9
![Page 10: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/10.jpg)
Vector Types
Available in host and device code Construct with make_<type name>
int2 i2 = make_int2(1, 2);float4 f4 = make_float4( 1.0f, 2.0f, 3.0f, 4.0f);
10
![Page 11: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/11.jpg)
Vector Types
Access with .x, .y, .z, and .w
int2 i2 = make_int2(1, 2);int x = i2.x;int y = i2.y;
No .r, .g, .b, .a, etc. like GLSL
11
![Page 12: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/12.jpg)
Math Functions
Double and float overloadsNo vector overloads
On the host, functions use the C runtime implementation if available
See Appendix C in the NVIDIA CUDA C Programming Guide for a complete list of math functions12
![Page 13: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/13.jpg)
Math Functions
Partial list:sqrt, rsqrtexp, logsin, cos, tan, sincosasin, acos, atan2trunc, ceil, floor
See Appendix C in the NVIDIA CUDA C Programming Guide for a complete list of math functions13
![Page 14: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/14.jpg)
Math Functions
Intrinsic functionDevice onlyFaster, but less accuratePrefixed with ____exp, __log , __sin , __pow , …
See Appendix C in the NVIDIA CUDA C Programming Guide for a complete list of math functions14
![Page 15: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/15.jpg)
Review: Thread Hierarchies
Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf 15
![Page 16: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/16.jpg)
Review: Thread Hierarchies
int threadID = blockIdx.x * blockDim.x + threadIdx.x;
float x = input[threadID];float y = func(x);output[threadID] = y;
16
![Page 17: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/17.jpg)
Review: Thread Hierarchies
int threadID = blockIdx.x * blockDim.x + threadIdx.x;
float x = input[threadID];float y = func(x);output[threadID] = y;
Use grid and block position to compute a thread id
17
![Page 18: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/18.jpg)
Review: Thread Hierarchies
int threadID = blockIdx.x * blockDim.x + threadIdx.x;
float x = input[threadID];float y = func(x);output[threadID] = y;
Use thread id to read from input
18
![Page 19: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/19.jpg)
Review: Thread Hierarchies
int threadID = blockIdx.x * blockDim.x + threadIdx.x;
float x = input[threadID];float y = func(x);output[threadID] = y;
Run function on input: data-parallel!
19
![Page 20: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/20.jpg)
Review: Thread Hierarchies
int threadID = blockIdx.x * blockDim.x + threadIdx.x;
float x = input[threadID];float y = func(x);output[threadID] = y;
Use thread id to output result20
![Page 21: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/21.jpg)
Thread Synchronization
Threads in a block can synchronizecall __syncthreads to create a barrierA thread waits at this call until all threads in
the block reach it, then all threads continue
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i + 1]);
21
![Page 22: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/22.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 0
Thread 0 Thread 1
Thread 2 Thread 3
22
![Page 23: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/23.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 1
Thread 0 Thread 1
Thread 2 Thread 3
23
![Page 24: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/24.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 1
Thread 0 Thread 1
Thread 2 Thread 3
Threads 0 and 1 are blocked at barrier
24
![Page 25: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/25.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 2
Thread 0 Thread 1
Thread 2 Thread 3
25
![Page 26: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/26.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 3
Thread 0 Thread 1
Thread 2 Thread 3
26
![Page 27: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/27.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 3
Thread 0 Thread 1
Thread 2 Thread 3
All threads in block have reached barrier, any thread can continue
27
![Page 28: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/28.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 4
Thread 0 Thread 1
Thread 2 Thread 3
28
![Page 29: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/29.jpg)
Thread Synchronization
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Mds[i] = Md[j];__syncthreads();func(Mds[i], Mds[i+1]);
Time: 5
Thread 0 Thread 1
Thread 2 Thread 3
29
![Page 30: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/30.jpg)
Thread Synchronization
Why is it important that execution time be similar among threads?
Why does it only synchronize within a block?
30
![Page 31: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/31.jpg)
Thread Synchronization
Image from http://courses.engr.illinois.edu/ece498/al/textbook/Chapter3-CudaThreadingModel.pdf31
![Page 32: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/32.jpg)
Thread Synchronization
Can __syncthreads() cause a thread to hang?
32
![Page 33: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/33.jpg)
Thread Synchronization
if (someFunc()){ __syncthreads();}// ...
33
![Page 34: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/34.jpg)
Thread Synchronization
if (someFunc()){ __syncthreads();}else{ __syncthreads();}
34
![Page 35: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/35.jpg)
Scheduling Threads
Slide from David Luebke: http://s08.idav.ucdavis.edu/luebke-nvidia-gpu-architecture.pdf 35
![Page 36: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/36.jpg)
Scheduling Threads
36
![Page 37: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/37.jpg)
Scheduling Threads
Streaming Processing (SP)
37
![Page 38: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/38.jpg)
Scheduling Threads
Streaming Multi-Processor (SM)
38
![Page 39: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/39.jpg)
Scheduling Threads
Look familiar?39
![Page 40: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/40.jpg)
Scheduling Threads
Slide from David Luebke: http://s08.idav.ucdavis.edu/luebke-nvidia-gpu-architecture.pdf
G80 16 SMs Each with 8 SPs
128 total SPs Each SM hosts up
to 768 threads Up to 12,288
threads in flight
40
![Page 41: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/41.jpg)
Scheduling Threads
Slide from David Luebke: http://s08.idav.ucdavis.edu/luebke-nvidia-gpu-architecture.pdf
GT200 30 SMs Each with 8 SPs
240 total SPs Each SM hosts
up to 8 blocks, or 1024 threads
In flight, up to 240 blocks, or 30,720 threads
41
![Page 42: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/42.jpg)
Scheduling Threads
Warp – threads from a blockG80 / GT200 – 32 threadsRun on the same SMUnit of thread schedulingConsecutive threadIdx valuesAn implementation detail – in theory
warpSize
42
![Page 43: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/43.jpg)
Scheduling Threads
Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter3-CudaThreadingModel.pdf
Warps for three blocks scheduled on the same SM.
43
![Page 44: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/44.jpg)
Image from: http://bps10.idav.ucdavis.edu/talks/03-fatahalian_gpuArchTeraflop_BPS_SIGGRAPH2010.pdf
Scheduling Threads
Remember this:
44
![Page 45: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/45.jpg)
Scheduling Threads
Slide from: http://courses.engr.illinois.edu/ece498/al/Syllabus.html 45
![Page 46: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/46.jpg)
Scheduling Threads
What happens if branches in a warp diverge?
46
![Page 47: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/47.jpg)
Image from: http://bps10.idav.ucdavis.edu/talks/03-fatahalian_gpuArchTeraflop_BPS_SIGGRAPH2010.pdf
Scheduling Threads
Remember this:
47
![Page 48: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/48.jpg)
Scheduling Threads
If 3 blocks are assigned to an SM and each block has 256 threads, how many warps are there?
A SM on GT200 can host up to 1024 threads, how many warps is that?
48
![Page 49: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/49.jpg)
Scheduling Threads
32 threads per warp but 8 SPs per SM. What gives?
49
![Page 50: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/50.jpg)
Scheduling Threads
32 threads per warp but 8 SPs per SM. What gives?
When an SM schedules a warp: Its instruction is ready8 threads enter the SPs on the 1st cycle8 more on the 2nd, 3rd, and 4th cyclesTherefore, 4 cycles are required to
dispatch a warp
50
![Page 51: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/51.jpg)
Scheduling Threads
QuestionA kernel has
1 global memory read (200 cycles) 4 non-dependent multiples/adds
How many warps are required to hide the memory latency?
51
![Page 52: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/52.jpg)
Scheduling Threads
SolutionEach warp has 4 multiples/adds
16 cyclesWe need to cover 200 cycles
200 / 16 = 12.5 ceil(12.5) = 13
13 warps are required
52
![Page 53: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/53.jpg)
Memory Model
Image from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf
Recall:
53
![Page 54: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/54.jpg)
Memory Model
RegistersPer threadFast, on-chip,
read/write access Increasing the number
of registers used by a kernel has what affect?
54
![Page 55: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/55.jpg)
Memory Model
Registers - G80Per SM
Up to 768 threads 8K registers
How many registers per thread?
55
![Page 56: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/56.jpg)
Memory Model
Registers - G808K / 768 = 10 registers per threadExceeding limit reduces threads by the blockExample: Each thread uses 11 registers,
and each block has 256 threads How many threads can a SM host? How many warps can a SM host? What does having less warps mean?
56
![Page 57: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/57.jpg)
Memory Model
Local MemoryStored in global memory
Copy per threadUsed for automatic
arrays Unless all accessed with
only constant indices
57
![Page 58: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/58.jpg)
Memory Model
Shared MemoryPer blockFast, on-chip,
read/write accessFull speed random
access
58
![Page 59: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/59.jpg)
Memory Model
Shared Memory – G80Per SM
Up to 8 blocks 16 KB
How many KB per block
59
![Page 60: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/60.jpg)
Memory Model
Shared Memory – G8016 KB / 8 = 2 KB per blockExample
If each block uses 5 KB, how many blocks can a SM host?
60
![Page 61: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/61.jpg)
Memory Model
Global Memory Long latency (100s cycles) Off-chip, read/write access Random access causes
performance hit Host can read/write GT200
150 GB/s Up to 4 GB
G80 – 86.4 GB/s
61
![Page 62: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/62.jpg)
Memory Model
Constant MemoryShort latency, high
bandwidth, read only access when all threads access the same location
Stored in global memory but cached
Host can read/writeUp to 64 KB 62
![Page 63: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/63.jpg)
Memory Model
Variable Declaration Memory Scope LifetimeAutomatic variables other than arrays
register thread kernel
Automatic array variables local thread kernel__shared__ int sharedVar; shared block kernel__device__ int globalVar; global grid application__constant__ int constantVar; constant grid application
See Appendix B.2 in the NVIDIA CUDA C Programming Guide for more details63
![Page 64: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/64.jpg)
Memory Model
Global and constant variablesHost can access with
cudaGetSymbolAddress() cudaGetSymbolSize() cudaMemcpyToSymbol() cudaMemcpyFromSymbol()
Constants must be declared outside of a function body
64
![Page 65: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/65.jpg)
Let’s revisit matrix multiple
65
![Page 66: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/66.jpg)
Matrix Multiply: CPU Implementation
Code from: http://courses.engr.illinois.edu/ece498/al/lectures/lecture3%20cuda%20threads%20spring%202010.ppt
void MatrixMulOnHost(float* M, float* N, float* P, int width){ for (int i = 0; i < width; ++i) for (int j = 0; j < width; ++j) { float sum = 0; for (int k = 0; k < width; ++k) { float a = M[i * width + k]; float b = N[k * width + j]; sum += a * b; } P[i * width + j] = sum; }}
66
![Page 67: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/67.jpg)
Matrix Multiply: CUDA Kernel
Code from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf
Accessing a matrix, so using a 2D block
67
![Page 68: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/68.jpg)
Matrix Multiply: CUDA Kernel
Code from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf
Each kernel computes one output
68
![Page 69: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/69.jpg)
Matrix Multiply: CUDA Kernel
Code from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf
Where did the two outer for loops in the CPU implementation go?
69
![Page 70: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/70.jpg)
Matrix Multiply: CUDA Kernel
Code from: http://courses.engr.illinois.edu/ece498/al/textbook/Chapter2-CudaProgrammingModel.pdf
No locks or synchronization, why?
70
![Page 71: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/71.jpg)
Matrix Multiply
ProblemsLimited matrix size
Only uses one block G80 and GT200 – up to 512 threads per block
Lots of global memory access
71
![Page 72: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/72.jpg)
Matrix Multiply
Image from http://courses.engr.illinois.edu/ece498/al/textbook/Chapter3-CudaThreadingModel.pdf
Remove size limitation Break Pd matrix into tiles Assign each tile to a block Use threadIdx and blockIdx for indexing
72
![Page 73: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/73.jpg)
Matrix Multiply
Image from http://courses.engr.illinois.edu/ece498/al/textbook/Chapter3-CudaThreadingModel.pdf
Example Matrix: 4x4 TILE_WIDTH = 2 Block size: 2x2
73
![Page 74: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/74.jpg)
Matrix Multiply
Pd1,0Md2,0
Md1,1
Md1,0Md0,0
Md0,1
Md3,0
Md2,1
Pd0,0
Md3,1 Pd0,1
Pd2,0 Pd3,0
Nd0,3 Nd1,3
Nd1,2
Nd1,1
Nd1,0Nd0,0
Nd0,1
Nd0,2
Pd1,1
Pd0,2 Pd2,2 Pd3,2Pd1,2
Pd3,1Pd2,1
Pd0,3 Pd2,3 Pd3,3Pd1,3
Image from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Example Matrix: 4x4 TILE_WIDTH = 2 Block size: 2x2
74
![Page 75: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/75.jpg)
Matrix Multiply
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ int Row = blockIdx.y * blockDim.y + threadIdx.y; int Col = blockIdx.x * blockDim.x + threadIdx.x;
float Pvalue = 0; for (int k = 0; k < Width; ++k) Pvalue += Md[Row * Width + k] * Nd[k * Width + Col];
Pd[Row * Width + Col] = Pvalue;}
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html75
![Page 76: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/76.jpg)
Matrix Multiply
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ int Row = blockIdx.y * blockDim.y + threadIdx.y; int Col = blockIdx.x * blockDim.x + threadIdx.x;
float Pvalue = 0; for (int k = 0; k < Width; ++k) Pvalue += Md[Row * Width + k] * Nd[k * Width + Col];
Pd[Row * Width + Col] = Pvalue;}
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Calculate the row index of the Pd element and M
76
![Page 77: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/77.jpg)
Matrix Multiply
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ int Row = blockIdx.y * blockDim.y + threadIdx.y; int Col = blockIdx.x * blockDim.x + threadIdx.x;
float Pvalue = 0; for (int k = 0; k < Width; ++k) Pvalue += Md[Row * Width + k] * Nd[k * Width + Col];
Pd[Row * Width + Col] = Pvalue;}
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Calculate the column index of Pd and N
77
![Page 78: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/78.jpg)
Matrix Multiply
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ int Row = blockIdx.y * blockDim.y + threadIdx.y; int Col = blockIdx.x * blockDim.x + threadIdx.x;
float Pvalue = 0; for (int k = 0; k < Width; ++k) Pvalue += Md[Row * Width + k] * Nd[k * Width + Col];
Pd[Row * Width + Col] = Pvalue;}
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Each thread computes one element of the block sub-matrix
78
![Page 79: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/79.jpg)
Matrix Multiply
dim3 dimGrid(Width / TILE_WIDTH, Height / TILE_WIDTH);dim3 dimBlock(TILE_WIDTH, TILE_WIDTH);
MatrixMulKernel<<<dimGrid, dimBlock>>>( Md, Nd, Pd, TILE_WIDTH);
Invoke kernel:
79
![Page 80: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/80.jpg)
What about global memory
access?80
![Page 81: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/81.jpg)
Matrix Multiply
Limited by global memory bandwidthG80 peak GFLOPS: 346.5Require 1386 GB/s to achieve thisG80 memory bandwidth: 86.4 GB/s
Limits code to 21.6 GFLOPS In practice, code runs at 15 GFLOPS
Must drastically reduce global memory access
81
![Page 82: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/82.jpg)
Matrix Multiply
Image from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
82
M
N
P
WIDTH
WIDTH
WIDTH WIDTH
ty
tx
Each input element is read by Width threads
Use shared memory to reduce global memory bandwidth
![Page 83: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/83.jpg)
Matrix Multiply
Image from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Break kernel into phasesEach phase accumlates Pd
using a subset of Md and NdEach phase has good data
localityMd
Nd
Pd
Pdsub
TILE_WIDTH
WIDTHWIDTH
TILE_WIDTHTILE_WIDTH
bx
tx01 TILE_WIDTH-12
0 1 2
by ty 210
TILE_WIDTH-1
2
1
0
TILE_WIDTH
TILE_WIDTH
TILE_WIDTHE
WIDTH
WIDTH
83
![Page 84: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/84.jpg)
Matrix Multiply
Image from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
Each thread loads one element of Md and Nd
in the tile into shared memory
84
Md
Nd
Pd
Pdsub
TILE_WIDTH
WIDTHWIDTH
TILE_WIDTHTILE_WIDTH
bx
tx01 TILE_WIDTH-12
0 1 2
by ty 210
TILE_WIDTH-1
2
1
0
TILE_WIDTH
TILE_WIDTH
TILE_WIDTHE
WIDTH
WIDTH
m
kbx
by
k
m
![Page 85: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/85.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
85
![Page 86: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/86.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Shared memory for asubset of Md and Nd
86
![Page 87: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/87.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Width/TILE_WIDTH• Number of phasesm• Index for current
phase
87
![Page 88: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/88.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Bring one element each from Md and Nd into shared memory
88
![Page 89: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/89.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Wait for every thread in the block, i.e., wait for the tile to be in shared memory
89
![Page 90: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/90.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Accumulate subset of dot product
90
![Page 91: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/91.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Why?
91
![Page 92: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/92.jpg)
Code from http://courses.engr.illinois.edu/ece498/al/Syllabus.html
__global__ void MatrixMulKernel( float* Md, float* Nd, float* Pd, int Width){ __shared__ float Mds[TILE_WIDTH][TILE_WIDTH]; __shared__ float Nds[TILE_WIDTH][TILE_WIDTH];
int bx = blockIdx.x; int by = blockIdx.y; int tx = threadIdx.x; int ty = threadIdx.y;
int Row = by * TILE_WIDTH + ty; int Col = bx * TILE_WIDTH + tx;
float Pvalue = 0; for (int m = 0; m < Width/TILE_WIDTH; ++m) { Mds[ty][tx] = Md[Row*Width + (m*TILE_WIDTH + tx)]; Nds[ty][tx] = Nd[Col + (m*TILE_WIDTH + ty)*Width]; __syncthreads();
for (int k = 0; k < TILE_WIDTH; ++k) Pvalue += Mds[ty][k] * Nds[k][tx]; __synchthreads(); } Pd[Row*Width+Col] = Pvalue;}
Write final answer to global memory
92
![Page 93: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/93.jpg)
Matrix Multiply How do you pick TILE_WIDTH?
How can it be too large?
93
![Page 94: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/94.jpg)
Matrix Multiply How do you pick TILE_WIDTH?
How can it be too large? By exceeding the maximum number of threads/block
G80 and GT200 – 512 Fermi – 1024
94
![Page 95: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/95.jpg)
Matrix Multiply How do you pick TILE_WIDTH?
How can it be too large? By exceeding the maximum number of threads/block
G80 and GT200 – 512 Fermi – 1024
By exceeding the shared memory limitations G80: 16KB per SM and up to 8 blocks per SM
2 KB per block 1 KB for Nds and 1 KB for Mds (16 * 16 * 4) TILE_WIDTH = 16 A larger TILE_WIDTH will result in less blocks
95
![Page 96: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/96.jpg)
Matrix Multiply Shared memory tiling benefits
Reduces global memory access by a factor of TILE_WIDTH
16x16 tiles reduces by a factor of 16G80
Now global memory supports 345.6 GFLOPS Close to maximum of 346.5 GFLOPS
96
![Page 97: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/97.jpg)
First-order Size Considerations in G80
Each thread block should have many threads TILE_WIDTH of 16 gives 16*16 = 256 threads
There should be many thread blocks A 1024*1024 Pd gives 64*64 = 4K Thread Blocks
Each thread block perform 2*256 = 512 float loads from global memory for 256 * (2*16) = 8K mul/add operations. Memory bandwidth no longer a limiting factor
Slide from http://courses.engr.illinois.edu/ece498/al/Syllabus.html97
![Page 98: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/98.jpg)
Atomic Functions
__device__ unsigned int count = 0;// ...++count;
What is the value of count if 8 threads execute ++count?
98
![Page 99: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/99.jpg)
Atomic Functions
Read-modify-write atomic operationGuaranteed no interference from other threadsNo guarantee on order
Shared or global memory Requires compute capability 1.1 (> G80)
See G.1 in the NVIDIA CUDA C Programming Guide for full compute capability requirements99
![Page 100: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/100.jpg)
Atomic Functions
__device__ unsigned int count = 0;// ...// atomic ++countatomicAdd(&count, 1);
What is the value of count if 8 threads execute atomicAdd below?
100
![Page 101: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/101.jpg)
Atomic Functions
How do you implement atomicAdd?
__device__ int atomicAdd( int *address, int val);
101
![Page 102: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/102.jpg)
Atomic Functions
How do you implement atomicAdd?
__device__ int atomicAdd( int *address, int val){ // Made up keyword: __lock (address) { *address += val; }}
102
![Page 103: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/103.jpg)
Atomic Functions
How do you implement atomicAdd without locking?
103
![Page 104: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/104.jpg)
Atomic Functions
How do you implement atomicAdd without locking?
What if you were given an atomic compare and swap?
int atomicCAS(int *address, int compare, int val);
104
![Page 105: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/105.jpg)
Atomic Functions
atomicCAS pseudo implementation
int atomicCAS(int *address, int compare, int val){ // Made up keyword __lock(address) { int old = *address; *address = (old == compare) ? val : old; return old; }} 105
![Page 106: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/106.jpg)
Atomic Functions
atomicCAS pseudo implementation
int atomicCAS(int *address, int compare, int val){ // Made up keyword __lock(address) { int old = *address; *address = (old == compare) ? val : old; return old; }} 106
![Page 107: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/107.jpg)
Atomic Functions
atomicCAS pseudo implementation
int atomicCAS(int *address, int compare, int val){ // Made up keyword __lock(address) { int old = *address; *address = (old == compare) ? val : old; return old; }} 107
![Page 108: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/108.jpg)
Atomic Functions
Example:
*addr = 1;
atomicCAS(addr, 1, 2);atomicCAS(addr, 1, 3);atomicCAS(addr, 2, 3);
108
![Page 109: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/109.jpg)
Atomic Functions
Example:
*addr = 1;
atomicCAS(addr, 1, 2);atomicCAS(addr, 1, 3);atomicCAS(addr, 2, 3);
// returns 1// *addr = 2
109
![Page 110: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/110.jpg)
Atomic Functions
Example:
*addr = 1;
atomicCAS(addr, 1, 2);atomicCAS(addr, 1, 3);atomicCAS(addr, 2, 3);
// returns 2// *addr = 2
110
![Page 111: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/111.jpg)
Atomic Functions
Example:
*addr = 1;
atomicCAS(addr, 1, 2);atomicCAS(addr, 1, 3);atomicCAS(addr, 2, 3); // returns 2
// *addr = 3
111
![Page 112: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/112.jpg)
Atomic Functions
Again, how do you implement atomicAdd given atomicCAS?
__device__ int atomicAdd( int *address, int val);
112
![Page 113: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/113.jpg)
Atomic Functions__device__ int atomicAdd(int *address, int val){ int old = *address, assumed; do { assumed = old; old = atomicCAS(address, assumed, val + assumed); } while (assumed != old); return old;}
113
![Page 114: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/114.jpg)
Atomic Functions__device__ int atomicAdd(int *address, int val){ int old = *address, assumed; do { assumed = old; old = atomicCAS(address, assumed, val + assumed); } while (assumed != old); return old;}
Read original value at *address.
114
![Page 115: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/115.jpg)
Atomic Functions__device__ int atomicAdd(int *address, int val){ int old = *address, assumed; do { assumed = old; old = atomicCAS(address, assumed, val + assumed); } while (assumed != old); return old;}
If the value at *address didn’t change, increment it.
115
![Page 116: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/116.jpg)
Atomic Functions__device__ int atomicAdd(int *address, int val){ int old = *address, assumed; do { assumed = old; old = atomicCAS(address, assumed, assumed + val); } while (assumed != old); return old;}
Otherwise, loop until atomicCAS succeeds.
The value of *address after this function returns is not necessarily the original value of *address + val, why? 116
![Page 117: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/117.jpg)
Atomic Functions
Lots of atomics:
// Arithmetic // BitwiseatomicAdd() atomicAnd()atomicSub() atomicOr()atomicExch() atomicXor()atomicMin()atomicMax()atomicAdd()atomicDec()atomicCAS()
See B.10 in the NVIDIA CUDA C Programming Guide117
![Page 118: Introduction to CUDA (2 of 2)](https://reader036.vdocuments.us/reader036/viewer/2022062323/56815dc4550346895dcbf040/html5/thumbnails/118.jpg)
Atomic Functions
How can threads from different blocks work together?
Use atomics sparingly. Why?
118