augmented reality on android with vuforia sdk by qualcomm

Augmented Reality on Android

What is Augmented Reality (AR)?

Virtual 2D and 3D content superimposed onto a live image

GPS and Compass-based Augmented Reality Experience

Limited to outdoor

experiences

Real World ViewLive camera view

Virtual Content

Information about points of interest

Good for rough location of items outside our field of view

Real World ViewLive camera view

Virtual ContentVision-based AR recognizes objects in field of view

and aligns graphics tightly to target object

Vision-based Augmented

Reality Experience

Enables a more immersive and interactive

experience

Pull Video Frame from Camera,

Scan for Known Objects or Features

Compare to Database of Known Images

Position and Orientation Determined

Z

X

Y

Graphics are Rendered

SCANS COMPARES POSITIONS RENDERS

times persecond30+

OYXGEN

HYDROGEN

HYDROGEN

OYXGEN

HYDROGEN

OYXGEN

HYDROGEN

Visualizing instructions

e.g. furniture assembly

Instructional

Applications

Gaming & Play

3D games in real world environments vs. virtual worlds

Turn Left on Main St.

Navigation & Discovery

Visualizing points of interest

Making print , outdoor, TV media, and product packaging “come alive”

Media / Advertising

Text

Translate words to multiple languages

Visual Search

Retrieving related informationfrom web for the object in view

Today

In the future…

AR is gaining momentum

16

�“According to Visiongain, in 2012, 25% of mobile apps will

feature augmented reality.”

�The firm forecasts the mobile AR industry will see $3 billion in

global revenue by 2016, up from $87 million (in 2011) and $21

million in 2010.”

�“Consumer engagement is the main reason that (augmented

reality) technology has been so successful. QR codes have

proven to be effective engagement tools, which has lead many

retailers, such as Macy’s, to incorporate them into their

marketing campaigns

17

Brands are using AR to engage consumers

AR deepens consumer interactions

with brands

10/18/2012Qualcomm Vuforia | Vuforia – Augmented

Reality Platform |18

�Point of Advertising – campaign has added AR element across media channels

�Point of Sale – product comes alive on the shelves

�Point of Use – added AR dimension when the product comes home

10/18/2012Qualcomm Vuforia | Vuforia –

Augmented Reality Platform |19

Vuforia – Qualcomm’s AR Platform

Vuforia Platform Traction

21,000+Registered

Developers from 130 Countries

400+ Models of

Smartphones and Tablets Supported

400+ Apps in Android

Market and iOS App Store

Accolades, Testimonials, and HonorsRECOGNITION FOR OUR AWARD WINNING AR TECHNOLOGY

“…your AR library is by far the best AR Library I’ve ever

used. It’s amazing, works incredibly fast, and really easy

to work with. Awesome job QCAR team!!”

2011 Future Mobile

Entertainment Award

for Mobile Augmented

Reality

Nicholas Rudolfsky, iOS Developer, Smule

“When it comes to mobile Augmented Reality technology,

Qualcomm is the top dog.”

Pocket-lint

“The opportunities to do amazing things—which we quite

literally couldn’t live without—are unparalleled with

augmented reality. This is Qualcomm’s secret weapon,

and it may assure the company thrives, rather than

simply survives the coming conflict.”

Rob Enderle, Analyst, TG Daily

“Qualcomm’s AR SDK has made it extremely easy

for us to prototype, design, and develop our ideas and

concepts.”

Morgan Jaffit, Co-Founder, Defiant Development

100s of articles in mainstream

technology and business press

Near Field Usage Environments

Retail ShelfRetail ShelfTable or FloorTable or Floor WallWall

Representations of real world elements that can be detected and tracked

“Augmentable” Objects

Image Target Frame Markers Simple 3D Objects

� Game boards

� Product packaging

� Posters

� Signs

� Greeting cards

� Business cards

� Books / magazine pages

� Game pieces

� Control cards

� Boxes

� Bottles

� Cans

Enables multiple cards to simultaneously trigger AR experiences

Frame Markers

Allows users to press “buttons” by touching certain areas on an image or object

Virtual Buttons

Features and Performance



Vision

• Ordinary Images

• Simple 3D Images

• Marker

• Virtual Buttons

Performance

• Simultaneous Object

Recognition

• Optimized for

Mobile

Flexibility

• Unity

• Eclipse

• Xcode

• 3rd Party

Android Development Options• Download Qualcomm Vuforia SDKs http://developer.qualcomm.com/ar

Eclipse Unity 3

Advantages • Free

• Low level APIs provide

enhanced flexibility and

performance

• Fully integrated game

engine reduces

development time and

cost

• Single app supports both

iOS and Android

Availability Now Now

29

Vuforia AR Applications Structure

30

Example

Target Management

Creating Image Targets

Targets are Trackable

DataSets

Camera start

Camera stop

Pixel Format

Conversion

The pixel format converter converts between the camera

format (e.g. YUV12) to a format suitable for OpenGL ES

rendering (e.g. RGB565) and for tracking (e.g. luminance)

Tracker

• The tracker detects and track real

world objects in camera video frames.

• Different algorithms take care of

detecting new targets or markers, and

evaluating virtual buttons.

• The tracker can load multiple datasets,

but only one can be active at a time

• The results are stored in a state object

that is used by the video background

renderer and can be accessed from

application code.

State Object Structure

Dataset Targets“Teapot on target A”

Target A

Target B

3D Assets

The State Object

Member Description Type of Data

Camera Frame Current image

Trackables List of active trackable objects

• Planar Image

• Frame Marker

• Multi Target

Events List of events

• Virtual Button 1

• Virtual Button 2, etc.

RGB

YUV

GRAYSCALE

Name

Button Press

Name

4x4

Matrix

• The video background renderer renders the camera image stored in the state object.

• The performance of the background video rendering is optimized for specific devices.

Video BackgroundRenderer

10/18/2012

Application Code

• Query the state object for newly detected targets, markers or updated states of these elements

• Update the application logic with the new input data

• Render the augmented graphics overlay

Application States

onCreate()

initApplication ()

loadTexture()

InitApplication ()

onResume()

Init Tracker()

initApplicationAR ()

loadTrackerData ()

StartCamera ()

initRenddering()

updaterRendering()

onPause ()

StopCamera ()

onDestroy()

deinitApplication()

destroyTrackerData()

deinitTracker()

Native Java

Native C/C++

Creating an Android AR App

GLRenderer implementsGLSurfaceView.renderer {. . .onDrawFrame(..) {

myNativeRenderer()}

}

myNativeCameraStartup (){. . .QCAR::CameraDevice.init(..);QCAR::CameraDevice.start(..);

}

myNativeTrackerStartup{. . .

QCAR::Tracker.setActive(QCAR::PLANAR_IMAGE);

}

myNativeRenderer{. . .// Render video background// Update app logic using State

Object// Render app assets

}

myApp extends Activity {. . .void onCreate() {

QCAR.init(..);}. . .

void onResume() {. . .setupOpenGLViews();myNativeCameraStartup();myNativeTrackerStartup();

}. . .

void onDestroy() {QCAR.deinit();

}}

Native C/C++ Java

Initialize SDK

Start Trackers

Handle Tracker

Updates

Close SDK

QCAR::ImageTracker

Frame Markers

QCAR::MarkerTracker

Virtual Buttons

Buttons coordinates<?xml version="1.0" encoding="UTF-8"?>

<QCARConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="qcar_config.xsd">

<Tracking> <ImageTarget size="247 173" name="wood">

<VirtualButton name="red" rectangle="-108.68 -53.52 -75.75 -65.87" enabled="true" />

<VirtualButton name="blue" rectangle="-45.28 -53.52 -12.35 -65.87" enabled="true" />

<VirtualButton name="yellow" rectangle="14.82 -53.52 47.75 -65.87" enabled="true" />

<VirtualButton name="green" rectangle="76.57 -53.52 109.50 -65.87" enabled="true" />

</ImageTarget> </Tracking> </QCARConfig>

Simple 3D Objects

• Boxes

• Bottles

• Cans

Enables products to trigger AR experiences, and interactive content to reflect geometry of

product packaging

Creating Simple 3D targets

Multi Image Targets

Swappable Datasets

Swappable Datasets contd..

• The script exposes a list of datasets (that are part of the project) to be loaded at scene startup.

• A single script can be used across different scenes

• Mainly used to load/unload and activate/deactivate datasets at runtime.

• Can ONLY be used at scene startup but doesn’t do anything when its properties are changed later on in the application lifecycle.

Dataset Script

Background Texture Access

Background Texture Access contd..

• Video background access allows you to use the

camera video as a texture through the use of

shaders

• Two shaders are used

– vertex shader --- converts video into grayscale and

inverts the black and white

– fragment shader--- listens for a touch on the screen ad

then distort the video around that touch.

Occlusion Management

Occlusion Management contd..

• Illustrates how shaders(four in this case) can be used to

make objects appear semi-transparent.

• Pointing your camera at the FlakesBox target allows you

see the teapot object inside the partially transparent

looking box.

• Four shaders, one per object, are rendered in correct

order (the video background, the checkerboard-box, the

teapot inside and the occlusion box).

Image overlay

Change code in

teapot.h

Images to

be replaced

Cube Structure• -1.00f, -1.00f, 1.00f, // front

• 1.00f, -1.00f, 1.00f,

• 1.00f, 1.00f, 1.00f,

• -1.00f, 1.00f, 1.00f,

• -1.00f, -1.00f, -1.00f, // back

• 1.00f, -1.00f, -1.00f,

• 1.00f, 1.00f, -1.00f,

• -1.00f, 1.00f, -1.00f,

• -1.00f, -1.00f, -1.00f, // left

• -1.00f, -1.00f, 1.00f,

• -1.00f, 1.00f, 1.00f,

• -1.00f, 1.00f, -1.00f,

• 1.00f, -1.00f, -1.00f, // right

• 1.00f, -1.00f, 1.00f,

• 1.00f, 1.00f, 1.00f,

• 1.00f, 1.00f, -1.00f,

• -1.00f, 1.00f, 1.00f, // top

• 1.00f, 1.00f, 1.00f,

• 1.00f, 1.00f, -1.00f,

• -1.00f, 1.00f, -1.00f,

• -1.00f, -1.00f, 1.00f, // bottom

• 1.00f, -1.00f, 1.00f,

• 1.00f, -1.00f, -1.00f,

• -1.00f, -1.00f, -1.00f

x

y

Top

Bottom

BackFrontRight

Left

z

Set of four vertices

defining a plane

One vertex/corner

of a plane defined

by x,y and z

Image overlay contd..

10/18/2012 61

• Step 1: Replace values of

teapot.h with cube.h (find cube.h in dominos app )

• Step2:Remove all coordinates

expect Top .

• Step3:Set the zero vale of z

coordinate of Top.

• Step4:Replace the images in

assets folder with required Image.

• Top

x

y

z

Top

Bottom

BackFrontRight

Left

Top

Video Playback

Video Playback

• Play video on an image or texture on

compatible devices

• Gracefully fall back to an alternate video-

viewing experience on incompatible devices

(android video play triggered on target)

Java

URL Example

Native C/C++Initialize SDK

Start Trackers

Handle Tracker

Updates

Close SDK

onDrawFrame(..) {public static Handler

mainActivityHandler;

public void displayMessage(String text)

{Message msg = new

Message();msg.obj = text;

mainActivityHandler.sendMessage(msg);} }

renderFrame() {for(int tIdx = 0; tIdx < state.getNumActiveTrackables(); tIdx++){

jstring js = env>NewStringUTF(trackable>getName());

jclass javaClass = env->GetObjectClass(obj);

jmethodID method = env->GetMethodID(javaClass,

“displayMessage", "(Ljava/lang/String;)V");

env->CallObjectMethod(obj, method, js);

}

protected void onResume(){

super.onResume();

ImageTargetsRenderer.mainActivityHandler = new Handler() {

@Overridepublic void

handleMessage(Message msg) {

Intent intent = new Intent(Intent.ACTION_VIEW);

intent.setData(Uri.parse("http://www.qualcomm.com/partials/video/34133"));

startActivity(intent);}

Unity3d



I. Installing the Extension into your unity project

II. Compiling a simple AR app, and

III. Mastering some of the more advanced AR topics.

Installation on Windows



I. Download installer EXE-file, from the download page.

II. Run installer

III. Select a location for the package installation convenient to

your development environment. The extension-only

package will also be copied to the Standard Packages

folder of your Unity installation

Compiling a Simple Project



I. Create Project

II. AR Assets and Prefabs to Scene

III. 3D Objects to Scene and Attach to

Trackables

Create Project



Step

1

Step

2

Step

3



I. Step 1: Click On File-> New Project

II. Step 2: Select Project path and name

III. Step 3: Select Project package name

•

Structure inside Unity





� Editor

• Contains the scripts required to dynamically interact with Trackable data in the Unity editor.

� Plugins

• Contains Java and native binaries that integrate the QCAR SDK with the Unity Android application.

� Qualcomm Augmented Reality

• Contains the prefabs and scripts required to bring augmented reality to your Unity application.

� Streaming Assets

• Contains the swappable target datasets downloaded from the online Target Management System.

AR Assets and Prefabs to Scene



� ARCamera prefab:The ARCamera is responsible for rendering the camera

image in the background and manipulating scene objects to react to tracking

data

� ImageTarget prefab :This prefab represents a single Image Target

Trackable object



Delete

Main

Camera

Adding Prefabs



I. Delete main camera from hierarchy

II. Add ARCamera and ImageTarget in hierachy

from project window

ARCamera prefab



ImageTarget prefab



3D Objects to Scene and Attach to

Trackables





I. Cube object (GameObject > Create Other >

Cube)

II. Lighting (GameObject > Create Other >

Directional Light)



Directional

Light

Animation





I. Select Object from hierarchy window

II. In menu bar select( Windows-> Animation)

•

Animation Window





Step

1

Step

2

Step

3 Step

5

Step

4



I. Step 1:Press the recording button and save animation .

II. Step 2:Press it for creating break points in animation.

III. Step 3:Drage time line according to required length.

IV. Step 4:Allows required type of movement (circular/Hor/Ver etc)

V. Step 5:It give options about to play(once/loop etc)

Android Deployment Process



Platform

Player

Settings

Bundle

Identifier

and API

Level

Android Deployment Process



• Unity provides a number of settings when building for Android devices – you

will need to select from the menu (File > Build Settings… > Player Settings…) to

see the current settings. Also, choose your platform now – Android or iOS.

• Click on Resolution and Presentation to select the required Default Orientation.

Note: the Vuforia AR Extension now supports Auto Rotation.

• Click on Icon to set your application icon.

• Click on Other Settings. Set the Minimum API Level to Android 2.2 'Froyo' (API

level 8) or higher. Set Bundle Identifier to a valid name

(e.g. com.mycompany.firstARapp).

• Next save your scene (File > Save Scene).

• Then open the build menu (File > Build Settings…). Make sure that your scene is

part of Scenes in Build. If this is not the case either use Add Current to add the

currently active scene or drag and drop your saved AR scene from the project

view into the Window.

Scripting



ImageTargetB

ehaviour

ScriptsScript, Data Set

and

Image Target

Virtual Button Case Statements

Loading Default Object w.r.t Trackable

Physics Engine



Physics

Properties

Physics Engine contd..



I. Rigidbodies: These are physically simulated objects.Use Rigidbodies for things that the

player can push around, eg. crates or loose objects, or move Rigidbodies around directly by adding forces to it by scripting.

II. Character Controllers: use to make a humanoid character

III. Collider: GameObject that has a Collider but not a Rigidbody. Static Colliders are used for

level geometry which always stays at the same place and never moves around� Box Collider: primitive shape of a cube

� Sphere Collider : primitive shape of a sphere

� Capsule Collider : primitive shape of a capsule

� Mesh Collider : creates a collider from the object's mesh, cannot collide with another Mesh Collider

� Wheel Collider : specifically for creating cars or other moving vehicles

VirtualButtons Example



Scripts

VirtualButtons Example contd..

10/18/2012Qualcomm Vuforia | Vuforia –

Augmented Reality Platform |93

Buttons

3D Object

Creating and deleting Virtual Buttons

at run-time



I. create a new Virtual Button for a given Image Target at run-time by

calling the CreateVirtualButton member function on the corresponding

instance of your ImageTargetBehaviour.

II. destroy a Virtual Button by calling DestroyVirtualButton, again defined

in ImageTargetBehaviour

Soccerball Example



Scripts

Physics

Engine

• Regional Ecosystem Strategy

• Brigitte Alexander 1/17/12

Thank you!

augmented reality on android with vuforia sdk by qualcomm

Documents