gpu programming 360idev
DESCRIPTION
These are the modified slides I created for 360|iDev in August 2014. These slides will continue to evolve from conference to conference.TRANSCRIPT
![Page 1: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/1.jpg)
GPU PROGRAMMING WITH GPUIMAGE AND METALJANIE CLAYTON-HASZ
![Page 2: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/2.jpg)
About MeJanie Clayton-Hasz
Co-Author of “iOS 8 SDK Development”
Software engineer at SonoPlot
@redqueencoder
http://redqueencoder.com
![Page 3: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/3.jpg)
What is a GPU?A Graphics Processing Unit (GPU) is a small super computer that does one thing really really well. That one thing is processing floating point math in parallel.
There are several applications for being able to do really fast floating point math: Graphics processing, bioinformatics, molecular dynamics, etc…
Most people are going to primarily focus on graphics processing, as we will today. For GPGPU programming, go see Jeff Biggus speak about OpenCL!
![Page 4: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/4.jpg)
What is Parallel Computing
The default processes in a project is serialized computing. One instruction is processed at a time and then the CPU moves on to the next one.
Parallel computing is the process of allowing multiple instructions to be carried out at once.
Can be done on different threads, cores, and even at the bit level.
![Page 5: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/5.jpg)
But I Have Concurrency!
Concurrency is about dealing with a lot of things at once.
Parallelism is about doing lots of things at once.
![Page 6: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/6.jpg)
Shader Basics
Shaders are the programs that determine what gets displayed on your screen.
Shaders determine shapes, colors, textures, lighting…
Everything you see and use comes down to shaders.
![Page 7: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/7.jpg)
GRAPHICS ON IOS DEVICES
![Page 8: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/8.jpg)
There are many levels of abstraction for graphics on iOS.
!
Some frameworks are more abstracted than others.
![Page 9: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/9.jpg)
UIKit
Sprite Kit
Core Animation/Graphics
OpenGL ES/GLKit
![Page 10: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/10.jpg)
A BRIEF HISTORY OF TIME, UH, OPENGL…
![Page 11: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/11.jpg)
OpenGL Origins
First released in 1992
Was an attempt to formalize a 3D graphic specification across platforms
![Page 12: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/12.jpg)
Problems with OpenGLWas created back when GPUs were not very powerful and existed on external graphics cards that could be swapped out
The computer system architecture was vastly different when OpenGL was created. Things that were not very efficient then, like the GPU, are vastly more efficient now.
Nothing is ever deprecated (Don’t ask Java programmers what that means, they don’t know)
![Page 13: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/13.jpg)
![Page 14: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/14.jpg)
Creation of OpenGL ES
ES: Embedded Systems
Wanted to strip out all of the dead code from OpenGL
Was specifically tailored to work on less powerful devices like mobile phones
![Page 15: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/15.jpg)
We don’t need a dozen turtles that all do the same thing
![Page 16: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/16.jpg)
OpenGL ES Specifics
Streamlined version of OpenGL
Everything you can do in OpenGL ES can directly be ported to OpenGL
Basically an optimized version of OpenGL
![Page 17: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/17.jpg)
CPU VS GPU PROGRAMMING
![Page 18: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/18.jpg)
CPU Expensive Tasks
Sending hardware commands to the GPU (Changing State Vectors)
Confirming that API usage is valid
Compiling the shaders
Interaction between the state and the shaders
![Page 19: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/19.jpg)
How does the CPU Send tasks to the GPU?
Try to envision a client-server process. Instead of your program sending an instruction over the network to a server and getting data back, you are sending instructions from your CPU to your GPU to be executed. Since you are sending instructions away from your client to be done elsewhere, you want to minimize this as much as possible.
![Page 20: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/20.jpg)
How does the CPU Send tasks to the GPU?
For example, in most Twitter client applications the client batches 20 or more Tweets in one call. This allows the application to feed tweets to the user without them having to wait for the network to deliver each and every tweet individually.
![Page 21: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/21.jpg)
![Page 22: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/22.jpg)
Fixed Function Pipeline
Present in OpenGL ES 1.1
Shaders were hard-coded into OpenGL
Easier to use, but were very limited
![Page 23: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/23.jpg)
Programmable Pipeline
Introduced in OpenGL ES 2.0
Shaders are now the responsibility of the programmer
Harder to do, but provides far more flexibility and options for effects
![Page 24: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/24.jpg)
GLSL SHADER BUILDING
![Page 25: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/25.jpg)
GLSL
OpenGL Shading Language (GLSL)
Introduced in OpenGL 2.0 in 2004
C-like language for building shaders, which are small, efficient programs to run on the GPU
Includes some specific data types and methods for processing geometry and graphics math that are not included in C
![Page 26: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/26.jpg)
Two Flavors of GLSL Shaders
VertexFragment
![Page 27: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/27.jpg)
Vertex Shaders
![Page 28: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/28.jpg)
Vertex Shaders
The Vertex Shader would record the vertices of the star (which would be broken down into a series of triangles)
The Vertex Shader would also specify that the area between the vertices is yellow. If there was a texture instead of a color, the shader would keep track of the texture coordinates.
![Page 29: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/29.jpg)
Fragment Shaders
![Page 30: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/30.jpg)
Fragment ShadersFragment Shaders determine what pixels receive which color.
If you look carefully at the star, there are areas outside the star that are yellow and areas inside that are white.
If there is a gradient, the Fragment Shader will calculate what specific color each individual pixel will be.
![Page 31: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/31.jpg)
Inputs and Outputs
Uniforms
Attributes
Varyings
![Page 32: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/32.jpg)
Uniforms
Values that don’t change during rendering
Read-Only
Available in both Vertex and Fragment Shaders
![Page 33: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/33.jpg)
Attributes
Vertex Shader only
Input values that change with every vertex, like their position, color, and texture coordinates
![Page 34: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/34.jpg)
Varyings
Used to pass data between the Vertex and the Fragment Shaders
Read-only in the Fragment Shader
Read-Write in the Vertex Shader
Varyings are the variables that determine the pixel color for the Fragment Shader
![Page 35: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/35.jpg)
GPU IMAGE
![Page 36: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/36.jpg)
Creating GPUImage
GPUImage dates back to iOS 5.
Unlike Core Image (at the time), GPUImage utilized shaders more efficiently to make image processing faster. Core Image has been improved over the years and they are now comparable.
![Page 37: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/37.jpg)
Why is GPUImage so Efficient?
OpenGL ES tasks must be performed on one thread
Many people utilize locks to manage the thread or, God forbid, only use the main thread. <shudder>
NSLock is expensive to the CPU
GPUImage utilizes a serial dispatch queue through GCD to manage anything that touches the GPU to keep everything happy and thread safe.
![Page 38: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/38.jpg)
Demo
![Page 39: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/39.jpg)
METAL: THE NEW KID IN TOWN
![Page 40: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/40.jpg)
What does Metal Promise?Deep hardware integration between Apple chips and Apple frameworks
General Purpose GPU programming (GPGPU)
Precompiled Shaders
up to 10 times more draw calls per frame
Being able to perform draw calls on multiple threads
![Page 41: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/41.jpg)
What Specifically are the CPU Expensive Tasks?
Compiling Shaders
Validating State
Start Work on the GPU
![Page 42: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/42.jpg)
Life Before Metal
All three of these expensive tasks were done on each and every single draw call.
All of these tasks don’t have to be done thousands of times a frame. Many can be done once, as long as the program knows that it does not have to continually check them.
![Page 43: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/43.jpg)
Life After Metal
Compiling Shaders: Now done when the applications builds
Validating State: Now done when the content loads
Start Work on the GPU: Still happens on each draw call. We can’t win them all…
![Page 44: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/44.jpg)
Why is This Important?
Before Metal, you would have to balance CPU time with GPU time. Tasks were so expensive that the GPU would usually not be used to capacity.
Now that the CPU tasks are less expensive, you can take that time to generate more AI and do more programming logic.
![Page 45: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/45.jpg)
Where Does Metal Help You?
Metal helps you when you have a lot of objects that need to work independently of one another.
Certain tasks, like image processing, do not involve a lot of objects, so you aren’t going to gain much with Metal.
![Page 46: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/46.jpg)
ZEN GARDEN DEMOEPIC GAMES
![Page 47: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/47.jpg)
SO, WHAT DO I THINK ABOUT METAL?
![Page 48: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/48.jpg)
![Page 49: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/49.jpg)
Why Metal is Scary
You have to control EVERYTHING!!!
You have to have a deep understanding of how the computer works that I have not seen demonstrated by a large number of people.
Metal assumes you know more than the computer does, which in my experience is usually a bad move.
![Page 50: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/50.jpg)
BAD WOLF
PROJECT
DATE CLIENT10/06/14
WHAT HAPPENS WHEN YOU LOOKINTO THE HEART OF THE GPU
![Page 51: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/51.jpg)
Why Metal is ExcitingMetal, along with Swift, signals a shift to figuring out how to do more parallel programming.
I believe Metal is not going anywhere. It will take a while for people to learn how to fully utilize it, but I believe it has the potential to be a game changer.
Metal, like Swift, is still partly baked. It gives early adopters an opportunity to master something extraordinary.
![Page 52: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/52.jpg)
IS THERE ANY POINT IN LEARNING OPENGL ES ANYMORE?
![Page 53: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/53.jpg)
“Easy things should be easy.
Hard things should be possible.”
–Larry Wall
![Page 54: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/54.jpg)
Yes, absolutely. Metal’s API is very similar to OpenGL ES. !
It will take a while for everyone to transition over to devices with A7 chips.
!
Apple will continue to support its developers who work with OpenGL ES, especially since the
Mac uses OpenGL and won’t be able to use Metal (yet). !
Also, Metal is new. It usually takes Apple a few years to work the kinks out of their new frameworks.
!
Also, with Metal’s incredibly stiff learning curve, very few people could work with it now.
![Page 55: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/55.jpg)
Take AwaysWhether you learn GLSL or Metal Shading Language, the value comes from the algorithms. The languages are not complicated and are similar. If you don’t know how the math on a shader works, knowing the language won’t really help you.
There are lots of books on GPU programming out there explaining how to create effects, not to mention the shaders included in GPUImage. You will need to understand the math, but there are great resources online out there for this stuff.
Be tenacious. This takes a lot of time to master. It is worth it. Be patient.
![Page 56: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/56.jpg)
![Page 57: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/57.jpg)
Think Different.
![Page 58: GPU Programming 360iDev](https://reader037.vdocuments.us/reader037/viewer/2022100309/556c571ed8b42acc228b49cd/html5/thumbnails/58.jpg)
The End