GPUs and GPU ProgrammingBharadwaj Subramanian, Apollo Ellis, Keshav Pingali

Imagery taken fromNvidia Dawn Demo

Slide on GPUs, Cuda and Programming Models by Apollo Ellis

Slides on OpenCL by Bharadwaj Subramanian

A GPU is a Multi-core Architecture

• High throughput is prioritized over low latency single task execution

• Large collection of fixed function and software programmable resources

Graphics Pipeline

• Virtual scene Virtual camera used to render• Direct3D and OpenGL formulate the process as

a pipeline of operations on fundamental entities– Vertices– Primitives– Fragments– Pixels

• Data flows in entity streams between pipeline stages.

Graphics Pipeline

Graphics Pipeline

• GPU Front End– Otherwise known as Vertex Generator• Takes in vertex descriptors: Location plus Type (Line,

Triangle, Quad, Poly)– Attributes (Normal, Texture Coordinate, Color etc.)

• Performs a prefetch on the vertex data and constructs a vertex stream.

Graphics Pipeline

• Vertex Processing– Programmable Vertex Shader Execute• Typically converts from world space to camera space• Languages include Cg and HLSL

• Primitive Assembly– Convert form vertices to primitives

• Rasterization– Primitive Sampler in Screen space– Fragment Generator

Graphics Pipeline

• Fragment Processing– Programmable Fragment Shader Execute• Texture Lookup and Light Interaction Calculation• Cg and HLSL

• ROP– Raster Operations (Depth Buffer Cull, Alpha Blend)– Calculate each fragment’s contribution to given

pixels

Shader Programming

• Fragment or Vertex processing is defined by shader programs written in Cg or GLSL or HLSL

• Compiled at runtime to binary • Or compiled offline and then transformed at

runtime• C-like function that processes a single input and

output in isolation• Run in parallel on multiple shader cores• Wide SIMD instructions due to instruction streaming

Parallel Processing and Encapsulation

• Task Parallelism is available across stages– Eg. Vertices are processed while fragments processed

etc.• Data Parallelism is available across stream

entities.– Each entity is independ of each other because of the

task offloading onto the fixed function units• Fixed Function Units encapsulate hard to

parallelize work in optimized hardware components

Still A Scheduling Problem

• Processing and on-chip resources must be dynamically reallocated to pipeline stages

• Depends on the current loads at different stages

• How to determine if different stages get more cores or more cache becomes an issue.

• Hardware Multithreading provides a solution to thread stalls distributes resources more evenly

CUDA

• CUDA is a more general data parallel model– No Pipe

• Clusters of Threads• Scatter Operations (Multiple Write)• Gather Operations (Multiple Read)• Application based decomposition of threads• Threads can share data and communicate with

each other

CUDA Programming Model

• GPU is viewed as a coprocessor with DRAM and many parallel threads

• Data parallel portions of applications can be offloaded onto this coprocessor

• C on the GPU– Global and Shared Variables– Pointers and Explicit Memory Allocation– OpenGL and DirectX interoperability

Tesla Architecture

• Scalable array of multithreaded Streaming Multiprocessors or SMs 768 to 12,288 concurrent threads

Kernels

• C C++ Simple Functions or Full Programs• Consists of thread blocks and grids– Thread Block• Set of concurrent threads that cooperate through

barriers and shared memory.

– Grid• Set of thread blocks that are independent form each

other• Multiple Grids per Kernel

Syntax Example

• __global__ void my_par_func(float a){

do something with a

} int dimGrid = 256, dimBlock 256

my_par_func<<<dimGrid,dimBlock>>>(5.0f)

Execution

• SIMT Single Instruction Multiple Model Scheduler schedules Warps or sets of concurrent threads on SM units.

• Warp is scheduled independently of other warps• If a Warps threads diverge in control flow path the

paths are each executed turning off the threads that are not effected

• No recursion is allowed for stack space problems

SIMD vs SIMT

• CUDA utilizes the wide SIMD units• However SIMD is not exposed to the

programmer• Instead SIMD units are used by multiple

threads at once• SIMT utilizes of SIMD

CUDA Wrap Up

• More general model using same hardware• GPU is a CUDA coprocessor• Tesla Architecture 768 to 12000+ threads• C C++ syntax• Serial Branching• No recursion• SIMD used by SIMT

Another Model GRAMPS

• General Runtime Architecture for Multicore Parallel Systems

• A programming model for graphics pipelines• Allows for custom pipelines mixing fixed

function and programmable stages• Data is exchanged using queues and buffers• Motivation comes from hybrid applications– REYES Rasterization and Ray Tracing

Execution Graphs

• Analog of a GPU pipeline• Made up of Stages• Provides scheduling information• Not limited to execution DAGs– Cycles are not forbidden

• Forward progress is not guaranteed• Flexibility presumably outweighs the cost of

well behaved programs assurance

Stages

• Types SHADER THREAD FIXEDFUNCTION• Operate asynchronously exposing parallelism• Indicate similarities in data access and

execution characteristics for efficient processing

• Useful when benefits coherent execution outweigh deferred processing

Shader

• Short live run to completion computations• Per element programs• Push operation introduced for conditional

output• Otherwise queue inputs and outputs are

managed automatically• Shader instances are scheduled in packets

similar to GPU execution

Threads and Fixed Function

• Threads– Similar to CPU threads designed for task

parallelism– Must be manually parallelize by the application– Useful for repacking data between Shader stages

and processing bulk chunks of data where sharing or cross communication is needed

• Fixed Function– Hardware unit wrappers

Buffers and Queues

• Buffers– essentially shared memory across stages

• Queues– Packets are the primitive data format of the queue defined at

creation– Opaque packets: are for data chunks which need not be

interpreted– Collection packets: for shader group dispatch

• Queue Manipulation– Thread/Fixed Stages– Shader Stages

Thread Fixed Stages

• reserve-commit– reserve: returns called a reference to one or more

contiguous packets a reservation is also acquired– commit: is a notification that releases the

referenced data back to the system– Input commit means packet has been consumed– Output commit means packet can go downstream

Shader Stages

• Queue ops are transparent to the user• As input packets arrive output reservations are

attained• When all shader instances for a collection

packer are done the commits happen automatically

• Queue Sets are introduced– Groups of queues viewed as single queues for

sharing among shaders

Summary GRAMPS

• Application creates stages, queues, and buffers.

• Queues and buffer are bound to stages• Computation proceeds according to execution

graphs• Computation graphs are fully programmable• Dynamic aggregation of work at runtime