GPU PROGRAMMING WITH GPUIMAGE AND METALJANIE CLAYTON

About MeJanie Clayton

Co-Author of “iOS 8 SDK Development”

Software engineer at SonoPlot

@redqueencoder

http://redqueencoder.com

What is a GPU?A Graphics Processing Unit (GPU) is a small super computer that does one thing really really well. That one thing is processing floating point math in parallel.

There are several applications for being able to do really fast floating point math: Graphics processing, bioinformatics, molecular dynamics, etc…

Most people are going to primarily focus on graphics processing, as we will today. For GPGPU programming, go see Jeff Biggus speak about OpenCL!

What is Parallel Computing

The default processes in a project is serialized computing. One instruction is processed at a time and then the CPU moves on to the next one.

Parallel computing is the process of allowing multiple instructions to be carried out at once.

Can be done on different threads, cores, and even at the bit level.

But I Have Concurrency!

Concurrency is about dealing with a lot of things at once.

Parallelism is about doing lots of things at once.

SERIES CIRCUITLESS EFFICIENT

PARALLEL CIRCUITMORE EFFICIENT

Shader Basics

Shaders are the programs that determine what gets displayed on your screen.

Shaders determine shapes, colors, textures, lighting…

GRAPHICS ON IOS DEVICES

There are many levels of abstraction for graphics on iOS.

Some frameworks are more abstracted than others.

UIKit

Sprite Kit

Core Animation/Graphics

OpenGL ES/GLKit

A BRIEF HISTORY OF TIME, UH, OPENGL…

OpenGL Origins

First released in 1992

Was an attempt to formalize a 3D graphic specification across platforms

John Carmack was instrumental for the adoption of OpenGL as a cross-platform 3D graphic specification.

Problems with OpenGLWas created back when GPUs were not very powerful and existed on external graphics cards that could be swapped out

The computer system architecture was vastly different when OpenGL was created. Things that were not very efficient then, like the GPU, are vastly more efficient now.

Nothing is ever deprecated (Don’t ask Java programmers what that means, they don’t know)

Creation of OpenGL ES

ES: Embedded Systems

Wanted to strip out all of the dead code from OpenGL

Was specifically tailored to work on less powerful devices like mobile phones

We don’t need a dozen turtles that all do the same thing

OpenGL ES Specifics

Streamlined version of OpenGL

Everything you can do in OpenGL ES can directly be ported to OpenGL

Basically an optimized version of OpenGL

CPU VS GPU PROGRAMMING

CPU Expensive Tasks

Sending hardware commands to the GPU (Changing State Vectors)

Confirming that API usage is valid

Compiling the shaders

Interaction between the state and the shaders

How does the CPU Send tasks to the GPU?

Try to envision a client-server process. Instead of your program sending an instruction over the network to a server and getting data back, you are sending instructions from your CPU to your GPU to be executed. Since you are sending instructions away from your client to be done elsewhere, you want to minimize this as much as possible.

How does the CPU Send tasks to the GPU?

For example, in most Twitter client applications the client batches 20 or more Tweets in one call. This allows the application to feed tweets to the user without them having to wait for the network to deliver each and every tweet individually.

What Actually Sends Commands to the GPU?

glGenBuffers(): Creates the buffer

glBindBuffers(): Tells OpenGL to use this buffer.

glBufferData(): Allocate this much continuous memory

glVertexAttribPointer(): What kind of data do we have?

glDrawArrays(): Render the data in the buffer

glDeleteBuffer(): We don’t need the buffer anymore, get rid of it.

Fixed Function Pipeline

Present in OpenGL ES 1.1

Shaders were hard-coded into OpenGL

Easier to use, but were very limited

Programmable Pipeline

Introduced in OpenGL ES 2.0

Shaders are now the responsibility of the programmer

Harder to do, but provides far more flexibility and options for effects

OpenGL ES 1.1 vs 2.0

1.1 2.0

http://www.sunsetlakesoftware.com/molecules

What Frameworks are Hardware Accelerated?

Core Animation GLKitSpriteKitSceneKit

What About Core Graphics/Quartz?

Core Graphics/Quartz is NOT performed on the GPU. It is performed on the CPU.

Core Graphics is off on its own. UIKit is written on top of Core Animation, which is written on top of the GPU.

Core Graphics utilizes offscreen drawing. Anything using offscreen drawing is not hardware accelerated.

Offscreen DrawingCore Graphics (anything starting with “CG”)

Every “drawRect()” method

Anything using Core Text

CALayers using masks and shadows

CALayers with “shouldRasterize” set to YES

Do not animate anything using offscreen drawing! It is horribly inefficient!!

GLSL SHADER BUILDING

GLSL

OpenGL Shading Language (GLSL)

Introduced in OpenGL 2.0 in 2004

C-like language for building shaders, which are small, efficient programs to run on the GPU

Includes some specific data types and methods for processing geometry and graphics math that are not included in C

GLSL

Two shader components: Vertex and Fragment

Both are necessary to create a completed shader program

Vertex shaders deal with how geometry is handled on the screen

Fragment shaders calculate what each individual pixel will look like

Vertex Shaders

Fragment Shaders

GPU IMAGE

Creating GPUImage

GPUImage dates back to iOS 5.

Unlike Core Image (at the time), GPUImage utilized shaders more efficiently to make image processing faster. Core Image has been improved over the years and they are now comparable.

Why is GPUImage so Efficient?

OpenGL ES tasks must be performed on one thread

Many people utilize locks to manage the thread or, God forbid, only use the main thread. <shudder>

NSLock is expensive to the CPU

GPUImage utilizes a serial dispatch queue through GCD to manage anything that touches the GPU to keep everything happy and thread safe.

Demo

METAL: THE NEW KID IN TOWN

What does Metal Promise?Deep hardware integration between Apple chips and Apple frameworks

General Purpose GPU programming (GPGPU)

Precompiled Shaders

up to 10 times more draw calls per frame

Being able to perform draw calls on multiple threads

What Specifically are the CPU Expensive Tasks?

Compiling Shaders

Validating State

Start Work on the GPU

Life Before Metal

All three of these expensive tasks were done on each and every single draw call.

All of these tasks don’t have to be done thousands of times a frame. Many can be done once, as long as the program knows that it does not have to continually check them.

Life After Metal

Compiling Shaders: Now done when the applications builds

Validating State: Now done when the content loads

Start Work on the GPU: Still happens on each draw call. We can’t win them all…

Where Does Metal Help You?

Metal helps you when you have a lot of objects that need to work independently of one another.

Certain tasks, like image processing, do not involve a lot of objects, so you aren’t going to gain much with Metal.

Why is This Important?Before Metal, you would have to balance CPU time with GPU time. Tasks were so expensive that the GPU would usually not be used to capacity.

Now that the CPU tasks are less expensive, you can take that time to generate more AI and do more programming logic.

Also, when people were generating art assets, they had to make convoluted versions of assets to work around OpenGL ES limitations. Now the same assets can be used in all places. Not everything is about code.

ZEN GARDEN DEMOEPIC GAMES

SO, WHAT DO I THINK ABOUT METAL?

Why Metal is Scary

You have to control EVERYTHING!!!

You have to have a deep understanding of how the computer works that I have not seen demonstrated by a large number of people.

Metal assumes you know more than the computer does, which in my experience is usually a bad move.

BAD WOLF

PROJECT

DATE CLIENT3/27/15

WHAT HAPPENS WHEN YOU LOOKINTO THE HEART OF THE GPU

Why Metal is ExcitingMetal, along with Swift, signals a shift to figuring out how to do more parallel programming.

I believe Metal is not going anywhere. It will take a while for people to learn how to fully utilize it, but I believe it has the potential to be a game changer.

Metal, like Swift, is still partly baked. It gives early adopters an opportunity to master something extraordinary.

IS THERE ANY POINT IN LEARNING OPENGL ES ANYMORE?

“Easy things should be easy.

Hard things should be possible.”

–Larry Wall

Yes, absolutely. Metal’s API is very similar to OpenGL ES.

It will take a while for everyone to transition over to devices with A7 chips.

Apple will continue to support its developers who work with OpenGL ES, especially since the

Mac uses OpenGL and won’t be able to use Metal (yet).

Also, Metal is new. It usually takes Apple a few years to work the kinks out of their new frameworks.

Also, with Metal’s incredibly steep learning curve, very few people could work with it now.

Take AwaysWhether you learn GLSL or Metal Shading Language, the value comes from the algorithms. The languages are not complicated and are similar. If you don’t know how the math on a shader works, knowing the language won’t really help you.

There are lots of books on GPU programming out there explaining how to create effects, not to mention the shaders included in GPUImage. You will need to understand the math, but there are great resources online out there for this stuff.

Be tenacious. This takes a lot of time to master. It is worth it. Be patient.

Think Different.

The End

• http://www.objc.io/issue-21/gpu-accelerated-image-processing.html

• https://developer.apple.com/videos/wwdc/2011/#414

• https://developer.apple.com/videos/wwdc/2014/#603

• http://www.sunsetlakesoftware.com/2011/05/08/enhancing-molecules-using-opengl-es-20

Links