Qualcomm Austria Research Center GmbH Operngasse 17-21, A-1040 Vienna, Austria www.qualcomm.com

QCAR 1.5 Beta1 SDK and Unity Extension for AndroidTransition guide for QCAR 1.5 Beta1 Android developers

QCAR 1.5 Beta1 SDK and Unity Extension for Android brings several changes that improve performance, provides the developer greater flexibility in managing trackables and gives access to the video background.

Date: Version:

December 6, 2011 1.5 Beta1 for Android

Table of Contents1 Introduction to QCAR SDK 1.5 Beta1 for Android .......................................................... 4 1.1 1.2 2 Overview ............................................................................................................... 4 Document usage ................................................................................................... 4

Native SDK for Android .................................................................................................. 5 2.1 2.1.1 2.1.2 2.1.3 2.1.4 2.2 2.2.1 2.2.2 2.3 2.3.1 2.3.2 2.3.3 2.3.4 2.3.5 2.3.6 Getting Started ...................................................................................................... 5 Using the On-Line Documentation ..................................................................... 5 Step 1: Setting up the Development Environment .............................................. 5 Step 2: Installing the QCAR SDK ..................................................................... 13 Step 3: Compiling & Running a QCAR Sample App ......................................... 19 QCAR Native SDK API Changes ......................................................................... 25 New API .......................................................................................................... 25 Modified API .................................................................................................... 30 Migrating your project from QCAR 1.0.6 to QCAR 1.5 Beta1 ............................... 34 AndroidManifest.xml ........................................................................................ 34 Android.mk ...................................................................................................... 34 Eclipse Project Settings ................................................................................... 34 config.xml and qcar-resources.dat ................................................................... 35 Native Application C++ code (e.g. ImageTargets.cpp) ..................................... 35 Native Application Java code (e.g. ImageTargets.java).................................... 36

3

Unity Extension for Android .......................................................................................... 39 3.1 3.1.1 3.1.2 3.1.3 3.2 3.2.1 3.2.2 3.3 Getting Started .................................................................................................... 39 Step 1: Installing the Extension ........................................................................ 39 Step 2: Compiling a Simple Project .................................................................. 40 Mastering Advanced Topics ............................................................................. 48 Unity SDK API Changes ...................................................................................... 59 Default Handlers .............................................................................................. 59 Datasets .......................................................................................................... 59 Upgrading a QCAR 1.0.6 Unity Project to QCAR 1.5 ........................................... 65

4

DevGuide changes ...................................................................................................... 67 4.1 4.1.1 4.1.2 4.1.3 Swap Dataset at Runtime .................................................................................... 67 Config.xml Definitions ...................................................................................... 67 Active Trackable Code Fragment ..................................................................... 67 Loading Dataset Code Fragment ..................................................................... 68

Page 2 of 73

4.2 4.2.1 4.2.2 4.3 4.3.1 4.3.2 4.3.3 5

Frame Markers .................................................................................................... 69 Definition.......................................................................................................... 69 Creation in QCAR SDK 1.5 Beta1 and Code Fragment ................................... 69 Samples .............................................................................................................. 70 Overview.......................................................................................................... 70 ImageTargets .................................................................................................. 70 BackgroundTextureAccess .............................................................................. 71

Appendix QCAR Public API documentation .............................................................. 73

Page 3 of 73

1 Introduction to QCAR SDK 1.5 Beta1 for Android 1.1 OverviewThe QCAR SDK 1.5 Beta1 and the QCAR Unity Extension introduces several changes to the 1.0 API, but in general retains the same functionality to detect and track trackables. New features like loading target datasets at runtime and several performance improvements to tracking robustness, and speed were made. Features of the QCAR SDK 1.5 Beta1 for Android include: Swappable datasets allows a an app to dynamically load a previously created dataset based on some application level trigger Streamlined access to the video background texture to apply custom shaders for special visual effects, Detecting targets at steeper angles Improved robustness during tracking with less likelihood of losing targets Better pose estimate yielding more accurate tracking Less jitter in the pose calculation resulting in a more stable augmentation, Improved detection and tracking of Frame Markers, Faster QCAR system start-up times.

NOTE: This documentation and the QCAR SDK 1.5 Beta1, as well as the QCAR Unity Extension 1.5 Beta1 software packages are for Android only. NOTE: If you have been developing applications with previous versions of the QCAR SDK or the QCAR Unity Extension you must apply several small changes to your existing project to make it compatible with the QCAR SDK 1.5 Beta1 and the QCAR Unity Extension. Please carefully review the corresponding section of this document for the full list of changes.

1.2 Document usageThis document contains four main sections: Native SDK for Android use this if you are developing with the native QCAR SDK Unity Extension for Android use this if you use the QCAR Unity Extension DevGuide changes this section summarizes the modifications to the DevGuide Public API the appendix contains the updated public API documentation off-line

The beginning of the Native SDK and Unity Extension sections contains the updated Getting Started guide. It follows the same structure as published previously on the Developer Portal: https://ar.qualcomm.at/qdevnet/sdk. To ease your transition locating the subtle differences, we have highlighted the changes throughout the Getting Started sections. After the Getting Started guide you will find new and modified functionality explained. Deprecated API is marked with strikethrough. A step-by-step migration guide wraps up each of the first two sections to help transitioning your existing applications from QCAR 1.0 to the QCAR 1.5 Beta1.

Page 4 of 73

2 Native SDK for Android 2.1 Getting StartedIf you are an Android developer and already have the Android SDK and NDK installed, then please go directly to Step 2: Download and install the QCAR SDK. If you are new to Android software development, you will find it very useful to start at Step 1 below. 1. Setting-up the development environment 2. Installing the QCAR SDK 3. Compiling & Running a QCAR sample app If you have used the QCAR SDK in the Beta phase you might want to check the section on application publishing. Note that several changes have been applied to the environment setup and some API features that you should revisit.

2.1.1 Using the On-Line DocumentationThe first step for both newcomers and experienced AR developers is to follow the Getting Started guide above and run a QCAR sample app. After that, depending on your skill and comfort level with AR, you may go directly to one of the following options:

If you are familiar with the concepts of vision based augmented reality, then you can look at the Developer Guide to learn more about QCAR SDK features. If you are a "No text - unless it starts with //" type of person, then the API Reference is the right place for you!

2.1.2 Step 1: Setting up the Development EnvironmentSupported Development Platforms

The QCAR SDK supports Android OS 2.1 and above. The recommended development environment is Microsoft Windows 7 32/64bit or Windows XP. The components to build the actual code (JDK, Eclipse+ADT and gcc) are available across multiple platforms. While building on Linux Ubuntu and Mac OS X environments is possible, we will be unable to support those platforms with detailed documentation. However, we have included hints and notes to platform specific issues that will help you set up your development environment on Linux Ubuntu 11.10 or Mac OS X 10.6 (Snow Leopard). This set-up guide has been written for the Win7 32/64bit platform with special notes for other operating systems. NOTE: If you already have both the Android SDK and the NDK setup, then please go directly to Step 2: Installing the QCAR SDK.

Page 5 of 73

Set-up the Android Development Environment

The QCAR SDK requires both the Android SDK and the NDK. The Android NDK is an extension to the Android SDK that lets Android developers build performance-critical parts of their applications in native code. SDK and NDK communicate over the Java-Native-Interface (JNI). To set-up the development environment please install the following components in the order they appear below: 1. 2. 3. 4. 5. 6. 7. JDK Eclipse IDE Android SDK Downloader Android ADT Android SDK platform support Cygwin Environment Android NDK

We recommend to use the following versions of the above listed tools with the QCAR 1.5 Beta1 SDK: Java SE Development Kit: use version Java SE 7u1 Eclipse: use latest version Android SDK Downloader: use version Android SDK Tools revision 15 Android ADT: choose the latest version that is for SDK tools rev 15 Android SDK Platform Support: use Android SDK Platform-tools revision 9 Cygwin Environment: use latest version 1.7.9-1. Android NDK: use version Android NDK 7

2.1.2.1 JDK Download the Java SE Development Kit (JDK) from the following site: http://www.oracle.com/technetwork/java/javase/downloads/ Click the 'Download' button from the JDK section of the 'Java Platform, Standard Edition' table. Install the JDK environment with default settings. Detailed installation instructions and system requirements can be found at: http://www.oracle.com/technetwork/java/javase/index-137561.html MAC: The JDK is already integrated into the Mac OS X operating system.

Page 6 of 73

2.1.2.2 Eclipse IDE In this step you will install the Eclipse IDE. Download the latest version of Eclipse IDE for Java Developers from: http://www.eclipse.org/downloads/ Unpack the downloaded ZIP package and copy the contents of the archive starting with the subdirectory 'eclipse' to your program directory path in C:\Program Files\eclipse. You may also create a shortcut to eclipse.exe on your desktop or start menu. Start eclipse.exe The very first time that Eclipse is started, the IDE will ask you to assign storage space for your workspace. This directory will contain ONLY IDE specific settings and information. This is not your application development workspace (despite the notice in this window). This type of information is typically stored in the user's home directory, which is the default value here: C:\Users\USERNAME\workspace. Check "Use this as the default and do not ask again".

MAC: For Mac OS X we recommend that you use the 32-bit version of Eclipse. LINUX: Eclipse is not always able to determine the location of the JVM, despite being in the path, To fix this, insert into /eclipse.ini the following line at the top:-vm

For example:-vm /usr/bin/java

2.1.2.3 Android SDK Downloader The Android SDK is distributed through an SDK starter package containing the SDK Tools. Download the starter package from: http://developer.android.com/sdk/index.html Unzip the archive and copy the contents into a directory, Ex. :\Development\Android\android-sdk-windows\. Throughout the Get Started guide we will refer to the base directory of your development environment as= C:\Development\Android

NOTE: Do not use pathnames with spaces.

Page 7 of 73

Add the tools\ directory to your Windows path. Right-click on Computer on the desktop and select Properties. Use the Advanced system settings to open the System Properties window and select Environment Variables on the Advanced tab. Look for variable Path in the System variables window. After pressing Edit, scroll to the end of Variable value: and add the full path to the tools\ directory to the end of the path, separated by a semicolon from the one before. For example:;C:\Development\Android\android-sdk-windows\tools\

NOTE: The last "\" at the end of the Path variable has to be included. Android ADT installation in the next step will use this path to identify the Android development environment.

Troubleshooting

For troubleshooting Android related issues and for more detailed instructions on the Android SDK set-up, please refer to the following:

Quick Start Android SDK set-up Installing the Android SDK

MAC: Update PATH variable to point to the make utility and the Android SDK tools directory in the /etc/rc.common or ~/.bash_profile file:PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec:/System/Library/CoreS ervices:/Developer/usr/bin:~/Development/Android/android-sdkmacosx/tools;export PATH

LINUX: Update your PATH to point to the Android SDK tools directory. If you use bash shell, add the following to ~/.bashrc :export PATH=/opt/android-sdk-linux/tools/:$PATH

2.1.2.4 Android ADT Android Development Tools (ADT) is a powerful extension to Eclipse that connects it with the Android SDK and helps with app development. This package is installed from within Eclipse. Select Help->Install New Software... . Add the url https://dl-ssl.google.com/android/eclipse/ into Work with field. Eclipse will ask you to provide an arbitrary name for the update site. After a short while Developer Tools appears in the field. At the very minimum, please select Android Development Tools and Android DDMS - which adds debugging support from the list, and then click on Next-> .

Page 8 of 73

Eclipse component selection After reviewing this selection and accepting the license terms, the downloader fetches the files and puts them in your Eclipse directory. After accepting the certificate and an automatic restart of Eclipse, the installation of the Android ADT is complete. 2.1.2.5 Android SDK Platform Support To develop for Android, support for the appropriate Android platform must be installed. The Android SDK Manager is used to install additional components and support for different platforms. Select in Eclipse the menu Window->Android SDK Manager. In case the Android SDK location was not setup correctly within Eclipse, go to Window->Preferences>Android and set the SDK location field to the root of your SDK install directory. In the Android SDK Manager window, sort by API level, click Deselect All, and check the following boxes:

Page 9 of 73

From Tools:

Android SDK Platform-tools

From Android 4.0 (API 14):

Documentation for Android SDK

From Android 2.2 (API 8):

Samples for SDK (optional)

From Android 2.1 (API 7):

SDK Platform

From Extras:

Google USB Driver package (not compatible with Linux)

Android SDK component selection To install the selected options you must hit Install 5 packages and accept all licenses on the next window. Use Accept All as a shortcut and hit Install.

Page 10 of 73

Android component license acceptance Finally, add the platform-tools\ directory to your Windows path:;C:\Development\Android\android-sdk-windows\platform-tools\

NOTE: The last "\" at the end of the Path variable has to be included. MAC: Update PATH variable to point to the Android SDK Platform-tools directory in the /etc/rc.common file or ~/.bash_profile:PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec:/System/Library/CoreS ervices:/Developer/usr/bin:~/Development/Android/android-sdkmacosx/tools:~/Development/Android/android-sdk-macosx/platformtools:export PATH

LINUX: Update your PATH to point to the Android SDK Platform-tools directory. If you use bash shell, add the following to ~/.bashrc :export PATH=/opt/android-sdk-linux/tools/:/opt/android-sdklinux/platform-tools/:$PATH

The Eclipse environment is now ready for Android development.

Page 11 of 73

2.1.2.6 Cygwin Environment LINUX: This section is not relevant for Linux users who have GNU make installed and in their path. MAC: This section is not relevant for Mac users who have Apple Developer Tools (XCode) installed. Install XCode if necessary from http://developer.apple.com/xcode/. A GNU compiler is required to compile dynamic applications as shared libraries for the Android NDK. Android make files are designed to run with gcc4. On Windows, a convenient way to have the complete environment prepared for this, is to install Cygwin.Cygwin uses an installer helper to manage the installation process. Go to http://www.cygwin.com/setup.exe and select "Install from Internet" when prompted at "Choose a Download Source" in the installer. We recommend not changing the Root Directory in the next window, and leaving it at "C:\cygwin". The "Local Package Directory" holds the downloaded packages. You may want to keep them with the downloaded Setup.exe in the same directory so as to have a Cygwin installer directory.

Choose a download site with a known fast connection near you. When the package information is downloaded you will see a hierarchical browser to select packages. Select the following package from the hierarchy for download:All -> Devel -> "make: The GNU version of the 'make' utility"

Select the word "skip" to change it to the actual version number, which is currently 3.81-2. Finish the installation by clicking next. Your Cygwin environment is fully set-up to work with the QCAR SDK. If you have other similar environments installed, make sure to set your Windows path variable to point to "C:\cygwin\bin" so that bash uses this version of GNU's make.exe. 2.1.2.7 Android NDK The Android NDK is an extension to the Android SDK that lets Android developers build performance-critical parts of their applications in native code. Download the NDK package from: http://developer.android.com/sdk/ndk/index.html Unzip the archive and copy the contents into a directory. To be consistent with our previous setup we recommend that you put the contents in "C:\Development\Android\androidndk-r7\". Thus Android SDK and Android NDK share the same parent directory. We will later add the QCAR-SDK and your project files here. NDK requires the above directory to be added to the Windows path! To do this right-click on 'My Computer' on the desktop and select properties. On the 'Advanced' tabs select 'Environment Variables' and look for Variable 'Path' in the 'System variables' window. After pressing 'Edit', scroll to the end of 'Variable value:' and add the full path to the directory to the end of the path, separated via semicolon from the one before.

Page 12 of 73

In the above example, you would have to add:;C:\Development\Android\android-ndk-r7\

NOTE: Path has semicolon at the beginning. Do not use pathnames with spaces. Alternatively, you can also set a User variable with the name Path, but this is only valid for the current user. The last "\" at the end of the Path variable has to be included. To test your installation you can compile any of the NDK sample applications. Using a Cygwin bash shell, navigate to the root directory of any demo application (e.g. for the 'sanangeles' sample app with out installation path above):cd /cygdrive/c/Development/Android/android-ndk-r7/samples/san-angeles ndk-build

The compiler should produce a dynamically linked library libsanangeles.so and write it to /libs/armeabi within the application directory. NDK includes support for different architectures so you might find different subdirectories in /libs. Now your development environment is ready to host QCAR SDK related content. MAC: Update PATH variable to point to the Android NDK directory in the /etc/rc.common file or ~/.bash_profile:PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec:/System/Library/CoreS ervices:/Developer/usr/bin:~/Development/Android/android-sdkmacosx/tools:~/Development/Android/android-sdk-macosx/platformtools:~/Development/Android/android-ndk-r7:export PATH

LINUX: Update your PATH to point to the Android NDK Platform directory. If you use bash shell, add the following to ~/.bashrc :export PATH=/opt/android-sdk-linux/tools/:/opt/android-sdklinux/platform-tools/:/opt/android-ndk-r7/:$PATH

2.1.3 Step 2: Installing the QCAR SDKClean Installation

QCAR SDK is distributed using installers for the following platforms: Windows, Mac OS X, and Linux. To start developing with the QCAR SDK you will need to,

Download & Install QCAR-SDK under . Adjust QCAR Environment settings in Eclipse.

Page 13 of 73

After accepting the license agreements the installer creates a directory structure in your Android development environment. This structure will ensure that sample apps can be easily built and deployed using the Android NDK and the Eclipse Java developer environment.Upgrading from a Previous Version

When installing an updated version of the QCAR SDK, we suggest that you install it under a new directory tree, just as the installer recommends. To compile the new set of sample applications it is required to adjust the Eclipse workspace variable setting. Lastly you will need to move your projects under the new (see below).Resulting Directory Structure

To streamline development we have designed a directory structure that keeps the QCARSDK and your applications in separate trees. This will ensure easier updates to the SDK, leaving your application source-trees untouched. In the previous steps we used a starting directory for the SDK and the NDK installation that we called:= C:\Development\Android

The downloaded installer creates a directory structure that will integrate into your Android development environment. Following the recommended installation location the development environment should finally result in the structure below. Here xx-yy-zz and xx.yy.zz stand for the version number of the QCAR SDK.

\ android-ndk-r7\ android-sdk-windows\ qcar-android-xx-yy-zz\ build\ licenses\ samples\ assets\ readme.txt

QUALCOMM Augmented Reality SDK License Agreements Sample applications with full source code Additional assets for the QCAR SDK Starting read-me document

Page 14 of 73

2.1.3.1 Install the QCAR SDKDownload

QCAR SDK is distributed through the Qualcomm Augmented Reality Developer Platform site. Follow the instructions below to access the installer. NOTE: Although we distribute installers for the platforms listed below, support for QCAR SDK development is limited to the Win 7 32/64 bit platform.Windows

Although we recommend that you develop on Windows 7 32/64-bit or Windows XP 32-bit environments, the QCAR SDK has also been successfully run on Windows XP 64-bit. However, we currently do not offer specific support for this platform. 1. Download installer EXE-file from the Downloads page. 2. Run installerMac OS X

Installation has been tested on Mac OS X 10.6. 1. Download archive file from the Downloads page. 2. Unarchive and run installerLinux

Installation has been tested on Ubuntu 11.10. 1. 2. 3. 4. 5. Requires JVM to be installed Download the installer file from the Downloads page. Open terminal window and change directory to the installer file location Change the mode of the installer to be executable Execute the installer file on the command line

Page 15 of 73

2.1.3.2 Set the QCAR Environment Variable Our recommended directory structure allows for SDK upgrades independent of application development. Your future AR projects will be in the recommended structure under . One workspace variable must be set in Eclipse so that is aware of this hierarchy: In Eclipse, go to Window->Preferences. Navigate to Java->Build Path->Classpath Variables using the hierarchy browser. Create a new variable by selecting New.... AddQCAR_SDK_ROOT

into the Name: field and navigate using Folder... to the \qcarandroid-xx-yy-zz directory we defined in the QCAR SDK Setup section. In our example above the variable value isC:\Development\Android\qcar-android-xx-yy-zz

where xx-yy-zz denotes the current QCAR SDK version number. This setting is essential as the classpath settings in the sample files use this reference to include common shared JAR files. MAC: The preferences menu on Mac OS X is under Eclipse->Preferences.

Page 16 of 73

2.1.3.3 Prepare Test Device for DevelopmentDeveloper Settings on the Device

Android devices require special settings for development. In this step we will:

Allow apps to be installed from unknown sources Enable USB debugging

On the device, go to Settings->Applications and choose "Unknown sources" as shown below. Accept the warning shown on the right. This setting allows the direct installation of unsigned APKs from within Eclipse.

Allowing unknown sources for software installation Go to the Development dialog on the above screen and choose the two settings as shown below:

Page 17 of 73

USB debugging is mandatory. The stay awake setting will help with development.Install the USB Driver

Connect your device to the development PC via the USB cable. On initial connection, Windows will recognize the new device and attempt to locate a compatible a driver. The Android SDK already includes some USB drivers - others will have to be obtained directly from the device manufacturer. SDK pre-packaged drivers can be located in the following directory:\android-sdk-windows\extras\google\usb_driver

When the device installation finishes you are ready to use your test device. On some machines the USB driver installation will ask you to reboot the machine. You can skip this step and the device should be accessible without a reboot. NOTE: You should contact the device manufacturer to insure that you are using the correct USB driver.

Page 18 of 73

2.1.4 Step 3: Compiling & Running a QCAR Sample App

We are now going to build a sample application included in the QCAR SDK package. The ImageTargets application is a good place to start learning about the SDK as it shows detection and tracking of natural features using common images. This section will show you how to build the native C++ source files with the NDK package of the Android SDK and then use Eclipse to build the Java sources and create the APK package that can be deployed to the device.

2.1.4.1 Compile the Shared ObjectCompile the Shared Object with Application Code

Android NDK applications are deployed as shared objects with a Java bootloader. In this step we create the binary *.so file that is later packaged through Eclipse. For each sample app you will need to build the application *.so separately.Build ImageTargets shared library

Change directory to ImageTargets in the \qcar-android-xx-yyzz\samples directory and executendk-build Gdbserver Gdbsetup Gdbserver Gdbsetup Compile++ arm : : : : [arm-linux-androideabi-4.4.3] libs/armeabi/gdbserver libs/armeabi/gdb.setup [arm-linux-androideabi-4.4.3] libs/armeabi-v7a/gdbserver libs/armeabi-v7a/gdb.setup : ImageTargets Clean...->Clean All Projects should fix these errors as Eclipse has to build some standard directories. This also creates the APK package for deployment.

2.1.4.3 Run the Sample ApplicationPrint the Image Target

Print all image targets in \qcar-android-xx-yyzz\samples\ImageTargets\media from any of the formats onto a US Letter or A4 sized paper with the page scaling 'none' option, keeping the original aspect ratio of the image intact.

Page 21 of 73

"chips" Image Target

"stones" Image Target

tarmac Image TargetDeploy & Run the Application

With the device connected, select the "ImageTargets" project from the Package Explorer in Eclipse and choose Run->Run As-> Android Application. Eclipse automatically installs it to a connected Android device using ADB, and starts running it on the device. Now when you hold your device up against the printed chips target, you should see the following image:

You have successfully deployed your first application with the QCAR-SDK!

Page 22 of 73

After a splash screen the live camera image is shown. Use your device to look at the printed target. You should see a textured teapot centered on top of the target, registered to the plane. Next, press the menu button, and select Switch to Tarmac dataset, and then hold your device up against the tarmac printed target and you will see the teapot, only now it is orange. The ImageTargets app shows how the SDK can detect and track a single image from a predefined set of images. The app also shows how to switch between datasets without too much effort, as well as how to deal with camera settings, such as autofocus. With a small code change, it can also be compiled to detect and track multiple images simultaneously. Check the Tips & Tricks section in the Developer Guide for more information on this.Troubleshooting

Should you encounter problems on the installation, check the device connection settings in Prepare Test Device for Development. In Eclipse, you can see if the device is connected correctly via the ADB in the DDMS panel (Window->Open Perspective->DDMS). The device must be listed under 'Devices'. Alternatively, open a Cygwin bash shell and executeadb devices

The output should show the attached device:$ adb devices List of devices attached HT012P123456 device

If the device list is empty, or a given device is not listed, kill the ADB server by executingadb kill-server

Executing 'adb devices' again will restart the server and re-detect devices. 2.1.4.4 Publish Your Application Once you have successfully finished developing your own application, you may distribute it through an app store such as the Android Market. NOTE: The QCAR SDK license agreement contains certain responsibilities that you as a developer must follow when distributing apps. You must inform end-users of your application that certain types of anonymous usage data is collected by the QCAR SDK and sent back to Qualcomm servers. For further details, please see the license agreement, which is available at https://ar.qualcomm.at/legal/license and in particular (but without limitation) note the requirements of Section 8 thereof, which specifically requires that you include, in a mandatory end-user license agreement for each of your applications, a clause whereby the end-user unconditionally and irrevocably expressly consents to:

Page 23 of 73

(i) the collection by QUALCOMM and/or its affiliates of Statistics from the Software (including but not limited to: (a) information about the end users' devices such as device unique identifier, make, model, operating system name and version and kernel version, (b) information about our Software used to create your software or augmented reality end user application such as the SDK version and device profile, and (c) information about your software or augmented reality application and its use such as settings (e.g., camera resolution settings, configuration settings), start and stop dates and times, camera on/off events, target image obtained/lost events, and other general usage information (collectively "Statistics")). (ii) the transfer of Statistics to QUALCOMM and/or its affiliates in the United States for QUALCOMM and/or its affiliates to (a) facilitate the provision of new products, updates, enhancements and other services, (b) to improve the Software and our products, services and technologies, and (c) to provide new products, services or technologies to You and/or our customers.

Page 24 of 73

2.2 QCAR Native SDK API ChangesThis section details the changes that were applied to the QCAR Native SDK API. Some new functions were created, and some existing functions were updated or deprecated.

2.2.1 New API2.2.1.1 QCAR::TrackerManager In QCAR SDK 1.0 access to all types of trackables was provided via the Tracker singleton. A more streamlined and decoupled approach is introduced with QCAR SDK 1.5 Beta1, where the TrackerManager class allows initialization and de-initialization of different trackers independently. This change allows trackers to be turned on/off as required by the application and thus saves memory and improves performance. Two independent trackers now track the different trackable types. Image Targets and Multi Targets are handled by the ImageTracker class, the Frame Markers are independently accessible via the MarkerTracker class. It is important to highlight that in QCAR 1.5 Beta 1, target dataset generated using the Target Management System is not loaded by the TrackerManager, but by the ImageTracker directly. NOTE: With this change it is also not possible to load a Frame Marker from the configuration file. These must be created at application runtime using the MarkerTracker API. Relevant APIs The TrackerManager API contains the following member functions: virtual Tracker virtual Tracker virtual bool *initTracker (Tracker::TYPE type)=0 *getTracker (Tracker::TYPE type)=0 deinitTracker (Tracker::TYPE type)=0

2.2.1.2 QCAR::DataSet The Swappable Target Dataset feature is supported in a new class. This new class contains Image Targets and Multi Targets and allows runtime loading, enabling, running and disabling groups of trackables. You can load multiple datasets at runtime (up to the application memory space limit). Note that at any given time only one dataset can be active. This active dataset is used by the ImageTracker to find and track targets in the camera image. Note that loading a dataset may take significant time and therefore it is recommended that datasets be loaded in the background. For recommendations on how to switch between datasets refer to the ImageTracker class documentation.

Page 25 of 73

NOTE: with QCAR SDK 1.5 Beta1 you do not have to bundle the dataset explicitly into your app, you can also load it from the SDCard or download it from the Internet at application runtime. Dataset generation The process to generate QCAR 1.5 Beta 1 compatible target dataset that you will use in your application has not changed. Using the Target Management Service (TMS) you upload your images and download a group of trackables as one trackable assets package. NOTE: QCAR SDK 1.5 Beta1 will not load datasets created and downloaded prior to the publication of the Beta! To use your existing targets, you must log-in to the TMS, recreate your project by uploading your images again to create the targets and download your new target datasets again. NOTE: During the beta phase of QCAR SDK 1.5 the functionality of the TMS remains unchanged to support QCAR SDK 1.0 developers. When you plan to use a downloaded dataset with the QCAR SDK 1.5 Beta1 you have to rename the files after opening the trackable assets package. Instructions on how to rename the files are below. The dataset name that is also used in the load() function is myDataset1: rename rename config.xml qcar-resources.dat myDataset1.xml and myDataset1.dat

In the QCAR SDK 1.5 Beta1 version of the Unity Extension a convenience function to accomplish the name change has been added. Refer to Changes to Unity Extension section for details.

Page 26 of 73

Relevant APIs Target dataset load and trackable reference methods available in the Tracker class in QCAR SDK 1.0 have been moved to the new DataSet class in the QCAR SDK 1.5 Beta1. Load the dataset by specifying the path and storage location: virtual bool enum load (const char *path, STORAGE_TYPE storageType)=0 STORAGE_TYPE { STORAGE_APP, STORAGE_APPRESOURCE, STORAGE_ABSOLUTE } The storage types STORAGE_APP, STORAGE_APPRESOURCE allow you to load dataset assets from specific, platform independent root folders. The relative path to the dataset XML must be passed to this function for all storage locations other than STORAGE_ABSOLUTE. Given an existing ImageTracker to create and load a dataset named myDataset1.xml that is placed in the assets/ you would callQCAR::DataSet* myDataset = 0; myDataset = imageTracker->createDataSet(); if (myDataset == 0) { LOG("Failed to create a new tracking data."); return 0; } if (!myDataset->load("myDataset1.xml", QCAR::DataSet::STORAGE_APPRESOURCE)) { LOG("Failed to load data set."); return 0; }

From the MyApp folder of the SDCard you would load the same dataset using the following piece of codeif (!myDataset->load("/sdcard/MyApp/myDataset1.xml", QCAR::DataSet::STORAGE_ABSOLUTE))

NOTE: Some Android phone models do have different pathnames for the external SDCard storage location, such as /sdcard or /emmc. The QCAR SDK does not abstract the external SDCard location, you must ensure that it is correctly located correctly. Refer to http://developer.android.com/guide/topics/data/data-storage.html for details.

Page 27 of 73

virtual int

getNumTrackables () const =0 getTrackable (int idx)=0

virtual Trackable *

These functions have been moved from the Tracker class to DataSet in QCAR SDK 1.5 Beta1, since you can work with several datasets at a time. The function getNumTrackables() returns the overall number of 3D trackable objects in this dataset. The getTrackable() function returns a pointer to a trackable object in the dataset.

virtual MultiTarget * createMultiTarget (const char *name)=0 virtual bool destroy (MultiTarget *multiTarget)=0

Starting with QCAR SDK 1.5 Beta1 the creation of MultiTargets is performed with the createMultiTarget() function of the DataSet class. This function creates a new MultiTarget and registers it with the dataset. The function destroy() destroys a MultiTarget.

2.2.1.3 QCAR::ImageTracker The ImageTracker class derived from the QCAR::Tracker class tracks Image Targets and Multi Targets contained in a QCAR 1.5 Beta1 compatible target dataset. The ImageTracker class provides methods for creating, activating and deactivating datasets. All dataset operations shall be called only while the ImageTracker is not active. During the execution of the QCAR::UpdateCallback this is guaranteed. While multiple datasets can be loaded as explained in the DataSet class description, only one dataset can be active at any given time. Before using another dataset you must deactivate the currently active dataset, otherwise activateDatSet() returns FALSE. This also happens if the dataset to be activated was already active. NOTE: The methods for swapping the active dataset should not be called while the ImageTracker is working at the same time. Doing so will make these methods block and wait until the tracker has finished. The suggested way of swapping datasets is during the execution of UpdateCallback, which guarantees that the ImageTracker is not working concurrently. Alternatively the ImageTracker can be stopped explicitly. However, this is a very expensive operation.

Page 28 of 73

Relevant APIs The methods of the ImageTracker allow creating, active, deactivating, and destroying datasets. The getActiveDataSet() returns a pointer to the currently active dataset. virtual DataSet * virtual bool virtual bool virtual DataSet * virtual bool createDataSet ()=0 activateDataSet (DataSet *dataset)=0 deactivateDataSet (DataSet *dataset)=0 getActiveDataSet ()=0 destroyDataSet (DataSet *dataset)=0

2.2.1.4 QCAR::MarkerTracker Derived from the QCAR::Tracker class the MarkerTracker provides methods to create and destroy Frame Markers dynamically. NOTE: The methods for creating and destroying Frame Markers should not be called while the MarkerTracker is working at the same time as these methods block and wait until the MarkerTracker has finished. Therefore the suggested way of doing this is to perform create/delete operations during the execution of UpdateCallback, which guarantees that the MarkerTracker is not working concurrently. Alternatively the MarkerTracker can be explicitly stopped.

Relevant APIs Frame Markers can be created, destroyed or referenced using the methods listed below: virtual Marker * createFrameMarker (int markerId, const char *name, const QCAR::Vec2F &size)=0 destroyMarker (Marker *marker)=0 getNumMarkers () const =0 getMarker (int idx) const =0

virtual bool virtual int virtual Marker *

2.2.1.5 VideoBackgroundTexture In QCAR SDK 1.5 Beta1 you can now access the video background that is rendered by QCAR. The QCAR::Renderer provides methods to bind a shader to the rendering of the video background (refer to the modified Renderer API). The new VideoBackgroundTextureInfo struct provides information on the video background.

Page 29 of 73

2.2.2 Modified API2.2.2.1 QCAR::Renderer The way how the QCAR SDK renders the video background has been altered starting with the QCAR SDK 1.5 Beta1 release. In QCAR 1.0 begin() already triggered the video background rendering, see following pseudo-code snippet:renderFrame() { // Clear color and depth buffer glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); // Render video background: QCAR::State state = QCAR::Renderer::getInstance().begin(); // at this point video background is already rendered, // you can add your rendering on top of the video background ...

}

QCAR::Renderer::getInstance().end();

With the QCAR SDK 1.5 Beta1 release begin() locks only the State until end() is called, thus it is guaranteed that drawVideoBackground() between those calls will render the video background in sync with the pose data from the corresponding State. The difference is that drawVideoBackground()can be called now at any point between begin()and end(). Methods of the Renderer class must only be called from the render thread.renderFrame() { // Clear color and depth buffer glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); // Get the state from QCAR and mark the beginning of a rendering section QCAR::State state = QCAR::Renderer::getInstance().begin(); // at this point NO video background is rendered, ... // explicitly render the Video Background QCAR::Renderer::getInstance().drawVideoBackground(); // now you can add rendering after the video background is rendered ...

}

QCAR::Renderer::getInstance().end();

This change allows, for example, rendering the video background multiple times with different transforms for multi-views in games, or rendering elements behind the video polygon. For more sophisticated effects or shaders applied to the video texture you would use the new function bindVideoBackground() method.

Page 30 of 73

Relevant APIs The new method added and described above is drawVideoBackground(), begin() and end()are only added for reference. virtual State virtual bool virtual void begin ()=0 drawVideoBackground ()=0 end ()=0

Full access to the video background rendering is given with the QCAR SDK 1.5 Beta1. virtual bool bindVideoBackground (int unit)=0

The new method bindVideoBackground()allows binding the video background texture to a texture unit and rendering it with your custom shader as shown in following pseudo-code snippet:renderFrame() { // Get the state from QCAR and mark the beginning of a rendering section QCAR::State state = QCAR::Renderer::getInstance().begin(); QCAR::Renderer::getInstance().bindVideoBackground(0); // Load the shader and upload the vertex/texcoord/index data ... // To create vertex shader effects it is recommended // to refine the mesh of the video background polygon ... // Render the video background with the custom shader glUseProgram(vbShaderProgramID); ... // Call rendering glDrawElements(GL_TRIANGLES, vbNumIndices, GL_UNSIGNED_BYTE, vbOrthoQuadIndices); // Wrap up this rendering glUseProgram(0); // Render your AR overlay objects ... // It is always important to tell the QCAR Renderer // that we are finished QCAR::Renderer::getInstance().end();

}

Other methods changed in QCAR SDK 1.5 Beta1 in QCAR::Renderer : virtual bool setVideoBackgroundTextureID (int textureID)=0

Convenience method to configure the layout of the video background (location on the screen and size of polygon).

Page 31 of 73

virtual const VideoBackgroundTextureInfo &getVideoBackgroundTextureInfo ()=0 Returns the texture info associated with the current video background. The VideoBackgroundTextureInfo contains width and height of the video background texture in pixels and the format.

2.2.2.2 QCAR::Tracker Due to the introduction of the new trackers ImageTracker and MarkerTracker, the separate loading and handling of datasets, some methods of QCAR::Tracker have been moved in QCAR SDK 1.5 Beta1. The length of the identifier has been reduced from 64 to 25 for all trackables. The following methods have moved to QCAR::DataSet: virtual int virtual int virtual Trackable * load ()=0 getNumTrackables () const =0 getTrackable (int idx) const =0

The following methods have moved to QCAR::ImageTracker: virtual bool virtual void activate (unsigned int flags)=0 deactivate (unsigned int flags)=0

The following method hasmoved to QCAR::CameraDevice: virtual const CameraCalibration & getCameraCalibration () const =0 The following methods have moved to QCAR::State: virtual int virtual Trackable * getNumActiveTrackables () const =0 getActiveTrackable (int idx) const =0

Relevant APIs Starting and stopping the trackers is still performed with QCAR::Tracker: virtual bool virtual void start ()=0 stop ()=0

In addition a query method returning the type of tracker has been added: virtual TYPE getType ()=0

Page 32 of 73

2.2.2.3 QCAR::CameraDevice The read-only method to retrieve the camera calibration data has been moved from QCAR::Tracker to QCAR::CameraDevice: virtual const CameraCalibration & getCameraCalibration () const =0

2.2.2.4 QCAR::ImageTarget The ImageTarget class now has a method that allows changing the size (width and height) attribute of an ImageTarget in scene units. The dataset that this ImageTarget belongs to must not be active when calling this function or it will fail. Returns true if the size was set successfully, false otherwise. virtual bool setSize (const Vec2F &size)=0

2.2.2.5 QCAR::MultiTarget The create and destroy methods of MultiTarget are deprecated starting with QCAR SDK 1.5 Beta1. QCAR::Dataset provides the corresponding methods to create and destroy the Multi Target with the correct correspondence to the composing Image Targets in a specific DataSet. Following methods have been removed: static MultiTarget * create (const char *name) static bool destroy (MultiTarget *MultiTarget)

Page 33 of 73

2.3 Migrating your project from QCAR 1.0.6 to QCAR 1.5 Beta1This section will give you a thorough description on how to upgrade your existing applications prepared with the QCAR SDK 1.0 to QCAR 1.5 Beta1. The samples provided with the QCAR SDK 1.5 Beta1 do have all these changes applied; instructions below are based on our samples and might vary in detail based on how you implemented your application.

2.3.1 AndroidManifest.xmlWith the QCAR SDK 1.5 Beta1 we do not require OpenGL ES 2.0 if a device has only OpenGL 1.5 Beta1 support (Android 2.1), so you can safely comment out the following line:

The camera must be included specifically, otherwise devices with no camera would be able to install QCAR SDK based applications, which is of no value, so you must include the line:

2.3.2 Android.mkTo conform the makefile to fit the slightly changed directory structure of the QCAR SDK 1.5 Beta1, make the following updates to your Android.mk file: LOCAL_SRC_FILES = ../../../build/lib/$(TARGET_ARCH_ABI)/libQCAR.so LOCAL_SRC_FILES = ../../../bin/Android/$(TARGET_ARCH_ABI)/libQCAR.so and LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/../../../build/include LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/../../../include

2.3.3 Eclipse Project SettingsNOTE: You will have to update QCAR_SDK_ROOT in your Eclipse project settings by setting its value in Window > Preferences > Java > Build Path > Classpath Variables to the new path of your QCAR SDK 1.5 Beta1 installation.

Page 34 of 73

2.3.4 config.xml and qcar-resources.dat

NOTE: QCAR SDK 1.5 Beta1 will not load datasets downloaded prior the publication of the Beta! You must log-in to the TMS, upload your images to create the targets and download your new target datasets. During the beta phase of QCAR SDK 1.5 Beta1 the functionality of the TMS remains unchanged to support QCAR SDK 1.0 developers. When you plan to use a downloaded dataset with the QCAR SDK 1.5 Beta1 you have to manually rename the files after opening the trackable assets package. Below you see the instructions, where the dataset name (also used in the load() function) is myDataset1: rename rename config.xml qcar-resources.dat myDataset1.xml and myDataset1.dat

2.3.5 Native Application C++ code (e.g. ImageTargets.cpp)Based on the ImageTargets.cpp sample application we show what changes have to be applied to get it running using QCAR SDK 1.5 Beta1. In your specific case the instructions may vary based on how you implemented your code. 1. You must add new includes to support the new API:#include #include #include

2. Swappable Target Dataset Functions:Since there are new functions for the Tracker, you must add four new JNI exported functions (copy from ImageTargets.cpp): initTracker, deinitTracker, loadTrackerData, destroyTrackerData.Be sure to edit those four functions to work with your named target datasets (since

you are now allowed many datasets at once), and be sure to edit the method signatures to match your project. 3. The renderFrame() method changes from one line to two. The code was:QCAR::State state = QCAR::Renderer::getInstance().begin();

becomes

Page 35 of 73

QCAR::State state = QCAR::Renderer::getInstance().begin(); QCAR::Renderer::getInstance().drawVideoBackground();

4. Update the configureVideoBackground() method (see ImageTargets.cpp), as it now corrects the rendering background size to handle mismatch between screen and video aspect ratios. 5. The camera start method was:// Start the tracker: QCAR::Tracker::getInstance().start(); // Cache the projection matrix: const QCAR::Tracker& tracker = QCAR::Tracker::getInstance(); const QCAR::CameraCalibration& cameraCalibration = tracker.getCameraCalibration();

becomes// Start the tracker: QCAR::TrackerManager& trackerManager = QCAR::TrackerManager::getInstance(); QCAR::Tracker* imageTracker = trackerManager.getTracker( QCAR::Tracker::IMAGE_TRACKER); if(imageTracker != 0) imageTracker->start(); // Cache the projection matrix: const QCAR::CameraCalibration& cameraCalibration = QCAR::CameraDevice::getInstance().getCameraCalibration();

6. The camera stop used to be a single line:QCAR::Tracker::getInstance().stop();

and now becomes a bit more complex:QCAR::TrackerManager& trackerManager = QCAR::TrackerManager::getInstance(); QCAR::Tracker* imageTracker = trackerManager.getTracker( QCAR::Tracker::IMAGE_TRACKER); if (imageTracker != 0) imageTracker->stop();

2.3.6 Native Application Java code (e.g. ImageTargets.java)Again, based on the ImageTargets.java sample application we show what changes have to be applied to get it running using QCAR SDK 1.5 Beta1. 1. You must add in several new application status constants:// Application status constants: private static final int APPSTATUS_UNINITED private static final int APPSTATUS_INIT_APP private static final int APPSTATUS_INIT_QCAR = -1; = 0; = 1;

Page 36 of 73

private static final private static final private static final private static final private static final private static final

int int int int int int

APPSTATUS_INIT_TRACKER = 2; APPSTATUS_INIT_APP_AR = 3; APPSTATUS_LOAD_TRACKER = 4; APPSTATUS_INITED = 5; APPSTATUS_CAMERA_STOPPED = 6; APPSTATUS_CAMERA_RUNNING = 7;

Add in the new the splash screen member variables:// The handler and runnable for the splash screen time out task. private Handler mSplashScreenHandler; private Runnable mSplashScreenRunnable;

There is also a new shutdown lock member variable that must be added:// An object used for synchronizing QCAR initialization, dataset loading and // the Android onDestroy() life cycle event. If the application is destroyed // while a data set is still being loaded, then we wait for the loading // operation to finish before shutting down QCAR. private Object mShutdownLock = new Object();

2. To handle the new shutdown lock, you must update InitQCARTask (see ImageTargets.java) with the following changes: Use shutdown lock in doInBackground, Switch to APPSTATUS_INIT_TRACKER on success in onPostExecute, Remove INIT_CANNOT_DOWNLOAD_DEVICE_SETTINGS error message.

3. The function LoadTrackerTask needs to be updated to handle the shutdown lock. The changes need to look like this:protected Boolean doInBackground(Void... params) { // Prevent the onDestroy() method to overlap: synchronized (mShutdownLock) { // Load the tracker data set: return (loadTrackerData() > 0); } }

4. Add declarations for the new tracker native methods:/** Native tracker initialization and deinitialization. */ public native int initTracker(); public native void deinitTracker(); /** Native functions to load and destroy tracking data. */

Page 37 of 73

public native int loadTrackerData(); public native void destroyTrackerData();

5. Update the onPause method after stopping the camera, instead of before:if (mAppStatus == APPSTATUS_CAMERA_RUNNING) { updateApplicationStatus(APPSTATUS_CAMERA_STOPPED); } // QCAR-specific pause operation QCAR.onPause();

6. Update onDestroy() method (see ImageTargets.java) so it performs the following tasks: dismisses splash screen timeout handler uses shutdown lock destroys tracker data deinitializes tracker

7. Update updateApplicationStatus() method (see ImageTargets.java) with the following: rename APPSTATUS_INIT_TRACKER case to APPSTATUS_LOAD_TRACKER add APPSTATUS_INIT_TRACKER case:case APPSTATUS_INIT_TRACKER: // Initialize the ImageTracker if (initTracker() > 0) { // Proceed to next application initialization status updateApplicationStatus(APPSTATUS_INIT_APP_AR); } break;

APPSTATUS_INIT_APP_AR goes to APPSTATUS_LOAD_TRACKER instead of APPSTATUS_INIT_TRACKER splashScreenHandler is stored as a member variable.

Page 38 of 73

3 Unity Extension for Android

3.1 Getting StartedThe QCAR Extension for Unity enables vision detection and tracking functionality into the Unity 3 IDE and allows developers to easily create AR applications and games. Developers need to first obtain Unity Android and/or Unity iOS from Unity Technologies in order to use the QCAR extension. The QCAR extension is compatible with both Unity Standard and Unity Pro. However, most developers will find that Unity Standard is sufficient for their needs. Please visit the Unity website http://www.unity3d.com for further information about Unity 3 and how to download it. Developers who have already installed Unity 3 can download the QCAR extension from the Downloads section at the top of this page. This section guides you through the easy steps of 1. Installing the Extension into your unity project, 2. Compiling a simple AR app, and 3. Mastering some of the more advanced AR topics.

3.1.1 Step 1: Installing the Extension3.1.1.1 Download from this site Use the Android download page to get the package for your development platform and proceed to the installation instructions below. Please note that Unity 3.2, 3.3, or 3.4 is required for Android. Check the list of Android devices supported by the QCAR Extension for Unity. NOTE: Unity Android users should check the Unity documentation at http://unity3d.com/support/documentation/Manual/Android-Devices That Have Been Tested.html to make sure that our supported devices are also supported by Unity. Installation on Windows The recommended development environment for Android is Microsoft Windows 7 32/64bit or Windows XP. Please note that Unity 3.2, 3.3, or 3.4 is required. 1. Download installer EXE-file, from the download page. 2. Run installer 3. Select a location for the package installation convenient to your development environment. The extension-only package will also be copied to the Standard Packages folder of your Unity installation.

Page 39 of 73

NOTE: When using Unity on Win7 64 you may get the following message in the editor: Your 64 bit Windows installation is missing an important service pack patch. Please apply patch in http://support.microsoft.com/kb/976038 to ensure stability. NOTE: Should the extension-only package not be listed in your Unity Standard Assets list, copy the package manually to the Standard Packages folder of your Unity installation. Installation on Mac OS X Developer experience has been also successfully tested on Mac OS X 10.6. 1. Download archive file, from the QCAR download page. 2. Unarchive and run installer 3. Select a location for the sample package convenient to your development environment, Extension-only installation is defaulting to your Unity installation. Installation Result Five Unity packages should be installed: For Android: qcar-unity-android-xx-yy-zz-unitypackage: the base QCAR extension qcar-unity-imagetargets-android-xx-yy-zz.unitypackage: a sample project using Image Targets qcar-unity-framemarkers-android-xx-yy-zz.unitypackage: a sample project using Frame Markers qcar-unity-multitargets-android-xx-yy-zz.unitypackage: a sample project using Multi Targets qcar-unity-virtualbuttons-android-xx-yy-zz.unitypackage: a sample project using Virtual Buttons qcar-unity-videobackgroundaccess-android-xx-yy-zz.unitypackage: a sample project using video texture effects

Note that after installation of the QCAR unity package, there will be a new high-level menu item QCAR. This menu item provides easy access to documentation as well as target dataset naming and properties. 3.1.1.2 Download from the Unity AssetStore Alternatively, you can download the extension directly into your Unity project right from the Unity Asset Store. When in Unity 3, select Asset Store from the Window menu. Search for "Qualcomm Augmented Reality" and then download and import the desired package.

3.1.2 Step 2: Compiling a Simple Project3.1.2.1 Create Project After you have installed the package, create a new project in Unity and select qcar-unityandroid-xx-yy-zz.unitypackage from the list of Import the following packages. Alternatively right-click in the Project view of an open project and choose Import Package Browse to the qcar-unity-android-xx-yy-zz.unitypackage you just installed and import it or double-clik on the downloaded package. When creating

Page 40 of 73

the Unity project avoid using spaces in the name because this will cause problems later. Note that by double-clicking on a package, you can selectively import parts of that package. NOTE: If you do not see the qcar unity package in the list, go back to Step 1 and manually install the packages. You will next need to add a target (or several targets) to your project. You can do this in two ways: by creating a target on the Target Management System (TMS) or by using existing targets from other projects. To use the TMS method, see DevGuide -> Target Management System online to create and download a package. Either double-click the downloaded package or right-click on Unity Project for Import Package, then Custom Package then select the downloaded package. Click on Import to import the target dataset. For 1.5Beta, you will receive an error that config.xml is ignored, no corresponding DAT file found. You will need to click on the QCAR menu, then Rename 1.0.x Data set, and choose and appropriate name for your target dataset, for this example, we will use ChipsBorder. If you are copying the dataset files from another project, be sure to also copy any files located in the Editor/QCAR/ImageTargetTextures folder. These will be used to texture the target plane in the Unity editor.

Page 41 of 73

You should now see the following folder structure inside Unity:

Here's a quick rundown of the folder contents: Editor Contains the scripts required to dynamically interact with Trackable data in the Unity editor. Plugins Contains Java and native binaries that integrate the QCAR SDK with the Unity Android application. Qualcomm Augmented Reality Contains the prefabs and scripts required to bring augmented reality to your Unity application. Streaming Assets Contains the swappable target datasets downloaded from the online Target Management System.

Page 42 of 73

3.1.2.2 Add AR Assets and Prefabs to Scene Now that you have imported the QCAR Unity Extension, you can easily adapt your project to use augmented reality. Expand the Qualcomm Augmented Reality folder, and then expand the Prefabs folder. Delete the Main Camera, and then drag an instance of the ARCamera prefab into your scene. The ARCamera is responsible for rendering the camera image in the background and manipulating scene objects to react to tracking data. With the ARCamera in place and the target assets available in the StreamingAssets/QCAR folder, you can run the application on a supported device and see the live video in the background.

Inspector view of the ARCamera

Page 43 of 73

Next, drag an instance of the ImageTarget prefab into your scene. This prefab represents a single Image Target Trackable object. Select the ImageTarget object in your scene, and look at the Inspector. There should be a Trackable Behaviour attached, with a property named Image Target. This property contains a drop-down list of all the available Image Targets for this project. You must select the Data Set and Image Target from your StreamingAssets/QCAR project. In this example, we choose StonesAndChips. (It is automatically populated from the config.xml file that is downloaded from the online Target Management System). The Unity sample apps come with several Image Targets that you can use. To use them copy them over from the ImageTargets sample or create your own at the My Trackables section of this site.

Inspector view of the ImageTarget

NOTE: When you added the Image Target object to your scene, a gray plane object appeared. This object is a placeholder for actual Image Targets. In the inspector view of the Image Target you will find a pop-up list called Image Target. From this list you can choose any Image Target that has been defined in one of the StreamingAssets/QCAR targets. After doing that the Image Target object in your scene adopts the size and shape from the Image Target it represents. It is also textured with the same image that the Image Target was created from.

Page 44 of 73

3.1.2.3 Add 3D Objects to Scene and Attach to Trackables Now you can bind 3D content to your Image Target. As a test, create a simple Cube object (GameObject > Create Other > Cube). Add the cube as a child of the ImageTarget object, by selecting it in the Hierarchy list and drag it onto the ImageTarget item. Move the cube in the scene until it is centered on the Image Target. You may also want to add a Directional Light to the scene to provide lighting (GameObject > Create Other > Directional Light). 3.1.2.4 TrackableEventHandler New to 1.5Beta, a default event handler (DefaultTrackableEventHandler) has been added to the Image Target. This default handler will cause the cube you just drew to automatically appear or disappear an automatic reaction to the appearance of the target in the video. You can override this default behavior if you want to one could also imagine playing a fade-out animation, showing an info screen or playing a sound for example. For a more detailed description of the ITrackableEventHandler interface please see 0 Responding to Tracking Events. 3.1.2.5 Adding Data Set Load to Camera Also new to 1.5Beta is the concept of multiple target datasets. You will need to tell Unity which dataset to load for your application. For this test, since we have borrowed from the ImageTarget app, we will load the StonesAndChips dataset, and make that the active dataset.

Page 45 of 73

3.1.2.6 Deploy Application The next step is to deploy your application to a supported device.

Android Deployment Process Unity provides a number of settings when building for Android devices you will need to select from the menu (File > Build Settings > Player Settings) to see the current settings. Also, choose your platform now Android. Click on Resolution and Presentation to ensure that Default Orientation is not set to Auto Rotation (can use portrait or Landscape Left). The QCAR extension does not support Auto Rotation. Click on Icon to set your application icon.

Page 46 of 73

Click on Other Settings. Set the Minimum API Level to Android 2.1 'Eclair' (API level 7)or higher. Set Bundle Identifier to a valid name (e.g. com.mycompany.firstARapp).

Next save your save (File > Save Scene). First, save your scene (File > Save Scene). Then open the build menu (File > Build Settings...). Make sure that your scene is part of Scenes in Build. If this is not the case either use Add Current to add the currently active scene or drag and drop your saved AR scene from the project view into the Window. You can now build the application. Attach your Android device and then click on Build And Run to initialize the deployment process.

The created AR scene

3.1.2.7 Using the Application You should have a printout of the appropriate Image Target in front of you. If you are working based off a Trackable in a sample app, the images are located at Editor/QCAR/ImageTargetTextures//*.jpg. Otherwise just print out the image that you uploaded to the Target Management System and make sure that the aspect ratio doesnt change. When you look at the target using the device camera, you should see your cube object bound to the target. Congratulations, you have successfully augmented reality! 3.1.2.8 Running in the Editor You can adjust the transform of the ARCamera object to get your entire scene in view, and then run the application in the Unity editor. There is no live camera image or tracking in the editor, instead all Trackables are assumed to be visible. This allows you to test the non-AR components of your application, such as scripts and animations, without having to deploy to the device each time.

Page 47 of 73

3.1.3 Mastering Advanced TopicsThis section will guide you on how to create the various types of Trackables and Virtual Buttons supported by the QCAR SDK and how to use them in Unity. We have also included details on how to use some of the special options supported by the Extension. 3.1.3.1 Using Trackables Using Image Targets To create new target files, login to the online Target Management System: http://ar.qualcomm.at/qdevnet Create a new project, and create new Single Image Trackables by uploading your own images. Give each Trackable a unique name, these will show up in the drop-down AR Targets list in Unity. When you have created all of your Trackables, select the set you are interested in and choose download selected trackables. When prompted, be sure to select the Unity Editor format. A .unitypackage file will be downloaded to your computer. Double-click on this file to import it into the currently open Unity project. Alternatively, you can choose Assets > Import Package > Custom Package within Unity, and browse to the file. Be sure to accept all files in the package. All files will go in the Streaming Assets/QCAR folder, along with scaled versions of your images in the Editor/QCAR/ImageTargetTextures folder. Once you have imported the .unitypackage, the Unity project will automatically detect the new configuration file and update the available Image Targets accordingly. NOTE: for the 1.5Beta, you will need to rename the newly imported dataset using the QCAR menu, then Rename 1.0.x Data Set, and choose an appropriate name for the target dataset. NOTE: If there are existing files that you would overwrite with your import youll be queried if you really want to do that you may want to make a backup of your modified file before importing. NOTE: The Trackable sizes that are initially used in the target dataset might be scaled larger than a typical Unity object. You may want to scale down the Trackable in the inspector before using it in your scene. 3.1.3.2 Using Frame Markers The QCAR Unity Extension also supports Frame Markers. To add a Frame Marker to your scene, simply drag an instance of the FrameMarker prefab into your Scene or Hierarchy view. This prefab behaves similarly to the ImageTarget prefab, except that you specify a Frame Marker by ID rather than name. Make sure that each FrameMarker in your scene has a unique ID. As you drag out new instances of the FrameMarker prefab, they are automatically given the next highest marker ID. NOTE: You will find all Frame Markers as png files with the binary code in the Editor/QCAR/FrameMarkerTextures/frame_markers_transparent.zip

Page 48 of 73

package. To print them, scale the markers with nearest-neighbor option to a convenient size. NOTE: Frame markers are no longer placed in the target dataset file.

Using Multi Targets A Multi Target is a Trackable 3D object, made up of multiple Image Targets arranged in a known configuration (such as a box). You can use the online Target Management System (TMS) to create Multi Targets with simple shapes. You can also edit the config.xml file by hand, to create different configurations. Each "part" of a Multi Target is simply an Image Target that has a defined position and orientation relative to a common center. To change the scale, position or orientation of a part, you must use the TMS to edit the respective item or edit the target dataset file. Once you've added the target dataset file to Unity, you can drag an instance of the MultiTarget prefab into your scene. A gray cube appears. This is a placeholder object. Use the dropdown in the Inspector to choose a Multi Target by name. A new 3D object will replace the placeholder object, with the configuration defined in your target dataset file. Each plane of the Multi Target will be textured with its corresponding Image Target image. Please note that currently you cannot change the configuration of a Multi Target in the Unity editor. Do not change the transform of any of the ChildTargets. You can translate or rotate the target as a whole, but you cannot scale the target in the editor. If you need to scale the target, open the target xml file and scale the individual Image Targets. Multi Targets in Unity behave similarly to Image Targets. You can bind content to the target by adding it as a child. There is one added feature, however, and that is that the Multi Target automatically creates a set of depth masks to properly occlude other virtual objects. Please see the QCAR-MultiTargets sample project for a full example.

Page 49 of 73

Page 50 of 73

3.1.3.3 Using Virtual Buttons Creating Virtual Buttons in the Unity Editor Virtual buttons are "hotspots" on an Image Target that a user can touch in the real world to trigger an action. You add them by dragging an instance of the VirtualButton prefab onto an existing Image Target in the scene. You can then translate and scale the button to match the desired location, and give it a name. Virtual Buttons added this way are automatically written to the config.xml file. Note that a Virtual Button needs to be a child of the Image Target in the scene.

NOTE: Virtual Buttons only work with Image Targets, and not with Frame Markers or Multi Targets. Now that you have placed Virtual Buttons on the target, the next step is to handle events attributed to these buttons. You accomplish this by implementing the IVirtualButtonEventHandler interface. This interface has two methods, OnVirtualButtonPressed and OnVirtualButtonReleased. When the event handler is called, you will be passed an instance of the Virtual Button, including the name of the button. In this way you can respond to different button events. Please see the QCARVirtualButtons sample project for a full example. Creating and deleting Virtual Buttons at run-time You can create a new Virtual Button for a given Image Target at run-time by calling the CreateVirtualButton member function on the corresponding instance of your ImageTargetBehaviour. Note that both the position and size arguments are defined in the local Image Target space. You can in turn destroy a Virtual Button by calling DestroyVirtualButton, again defined in ImageTargetBehaviour.

Page 51 of 73

Note that with the introduction of swappable data sets, the data set must be deactivated before creating a virtual button:Example code: // Deactivate the current data set ImageTracker imageTracker = (ImageTracker) TrackerManager.Instance.GetTracker(Tracker.Type.IMAGE_TRACKER); DataSet dataSet = imageTracker.GetActiveDataSet(); imageTracker.DeactivateDataSet(dataSet); // Create a virtual button ImageTargetBehaviour itb = GetComponent(); VirtualButtonBehaviour vbb = itb.CreateVirtualButton("mybutton", new Vector2(0, 0), new Vector2(0.1f, 0.1f)); // Register an event handler // Here we assume this class extends IVirtualButtonEventHandler vbb.RegisterEventHandler(this); // Reactive the dataset imageTracker.ActivateDataSet(dataSet);

Reconfiguring Virtual Buttons at run-time It is possible to change the following attributes of a Virtual Button at run-time: Sensitivity Position on the Image Target Size of the area

You can set the sensitivity of a Virtual Button by setting the SensitivitySetting property in the VirtualButtonBehaviour instance. The position and the size of a Virtual Button can be changed by respectively changing the position and scale components in the transform of the Virtual Button game object. Tying Virtual Buttons to other game objects One typical use case for reconfiguring a Virtual Button at run-time is when you want a button to move along with another game object. To accommodate this, a Virtual Button does not need to be a direct child of the Image Target but can be attached to another game object which itself has an Image Target as a parent. When the game object moves, the Virtual Button will simply be moved along and the reconfiguration will be taken care of automatically. NOTE: This only works under predefined conditions that are enforced by editor scripts. The Virtual Button must be located at the center of its parent object. It is always axis aligned with the Image Target. Furthermore the parent cannot have a non-uniform scale. If it does the editor script automatically rescales the object and prints a warning.

Page 52 of 73

In the example below a Virtual Button is a child of the soccer ball:

Please see the QCAR-VirtualButtons sample for the full example.

3.1.3.4 Swappable Datasets The DataSetLoadBehaviour is a script that is by default attached to the ARCamera object. The script exposes a list of datasets (that are part of the project) to be loaded at scene startup. It also contains a drop down list to choose a dataset to be activated. You can use different settings in DataSetLoadBehaviour across different scenes. This enables you to implicitly bind datasets to Unity scenes. Please be aware that this script was not created to provide a runtime API. It only loads/activates the datasets at startup but doesnt do anything when its properties are changed later on in the application lifecycle. To change datasets at runtime a different interface has to be used. Runtime Dataset API Mainly three classes are used to load/unload and activate/deactivate datasets at runtime TrackerManager, ImageTracker and DataSet. Loading a dataset: Check if the file at the given storage location exists: DataSet.Exists(string path, DataSet.StorageType storageType); Note that you can also call Exists without specifying a storage type. The assumption then is that your dataset has been bundled with the application in StreamingAssets/QCAR.

Page 53 of 73

Request an ImageTracker instance from TrackerManager: TrackerManager.Instance.GetTracker(Tracker.Type.IMAGE_TRACKER); Create a new DataSet instance: ImageTracker.CreateDataSet(); Load a dataset file from its storage location: DataSet.Load(string path, DataSet.StorageType storageType); o The same storage type rules apply as for Exists.

Activating a dataset: Use the imageTracker instance to activate a dataset (remember that only a single dataset can be active at a time). ImageTracker.ActivateDataSet(DataSet dataset);

Example code:// Load and activate a data set at the given path. private bool LoadDataSet(string dataSetPath, DataSet.StorageType storageType) { // Check if the data set exists at the given path. if (!DataSet.Exists(dataSetPath, storageType)) { Debug.LogError("Data set " + dataSetPath + " does not exist."); return false; } // Request an ImageTracker instance from the TrackerManager. ImageTracker imageTracker = (ImageTracker)TrackerManager.Instance.GetTracker(Tracker.Type.IMAGE_TRACKER); // Create a new empty data set. DataSet dataSet = imageTracker.CreateDataSet(); // Load the data set from the given path. if (!dataSet.Load(dataSetPath, storageType)) { Debug.LogError("Failed to load data set " + dataSetPath + "."); return false; } // (Optional) Activate the data set. imageTracker.ActivateDataSet(dataSet); return true; }

Adding content to a Trackable: Request a DataSetTrackableBehaviour instance from the dataset: DataSet.GetTrackable(int index); Attach a TrackableEventHandler to its game object: GameObject.AddComponent(); You need to either write your own implementation of the ITrackableEventHandler interface or use the DefaultTrackableEventHandler that comes with the QCAR extension. The TrackableEventHandler provides callbacks to handle Trackable lost and found events. Add a child object for augmentation to the Trackable. GameObject.transform.parent = DataSetTrackableBehaviour.transform;

Page 54 of 73

Example Code:// Add Trackable event handler and content (cubes) to the Trackables. private void AttachContentToTrackables(DataSet dataSet) { // Get the number of Trackables defined in the data set. int numTrackables = dataSet.GetNumTrackables(); // Loop over all Trackables. for (int i = 0; i < numTrackables; ++i) { DataSetTrackableBehaviour dstb = dataSet.GetTrackable(i); GameObject go = dstb.gameObject; // Add a Trackable event handler to the Trackable. // This Behaviour handles Trackable lost/found callbacks. go.AddComponent(); // Create a cube object. GameObject cube = GameObject.CreatePrimitive(PrimitiveType.Cube); // Attach the cube to the Trackable and make sure it has a proper size. cube.transform.parent = dstb.transform; cube.transform.localScale = new Vector3(0.7f, 0.7f, 0.7f); cube.transform.localPosition = new Vector3(0.0f, 0.35f, 0.0f); cube.transform.localRotation = Quaternion.identity; cube.active = true; dstb.gameObject.active = true; } }

Destroying a data set: Request an ImageTracker instance as explained above. Make sure that the data set to destroy is not active (otherwise destruction will fail): ImageTracker.Deactivate(DataSet dataset); Destroy the data set: ImageTracker.DestroyDataSet(DataSet dataset, bool destroyTrackableGameObjects);

Example Code:// Destroy a dataset. private bool DestroyDataSet(DataSet dataSet) { // Request an ImageTracker instance from the TrackerManager. ImageTracker imageTracker = (ImageTracker)TrackerManager.Instance.GetTracker(Tracker.Type.IMAGE_TRACKER); // Make sure the data set is not active. imageTracker.DeactivateDataSet(dataSet); // Destroy the data set // (choose true to also destroy the Trackable objects that belong to the data set). bool success = imageTracker.DestroyDataSet(dataSet, true); return success; }

Page 55 of 73

3.1.3.5 Special Options Responding to Tracking Events There are two interfaces you can implement to handle tracking events, the ITrackableEventHandler and the ITrackerEventHandler. The ITrackableEventHandler contains the OnTrackableStateChanged method, which is called each frame after the Trackable's data has been updated. Use this method to detect changes in the tracking state, such as whether the Trackable was just lost or just found. See the DefaultTrackableEventHandler for a reference implementation, located in the Assets/Qualcomm Augmented Reality/Scripts folder. We recommend copying the default event handler before modifying it, to avoid future upgrade conflicts. You can create a single class that implements the ITrackableEventHandler, and attach it to each of the Trackables in your scene. Or, you can create different implementations for different Trackables. Be sure to register your implementation with the TrackableBehaviour via the EventHandler field (see the samples for more details). NOTE: The relative orientation of all Trackables will be updated before the first OnTrackableStateChanged method is called. This means it is safe to do relative transform calculations in this callback. The ITrackerEventHandler contains the OnTrackablesUpdated method, which is called after each Trackable's ITrackableEventHandler has been called (once per frame). This might be a good place for game logic that depends on all Trackables being updated and stable. Initialization Error Handling It is important for your application to detect and handle all of the possible QCAR initialization errors. These include cases where the device is not supported or device settings cannot be downloaded. Each of the sample applications comes with a DefaultInitializationErrorHandler attached to the ARCamera object. Use the sample implementation as a starting point for creating your own error handler. Enabling and Disabling AR Objects Enabling and disabling the various QCAR Behaviours works as expected in Unity. Disabling the TrackerBehaviour will stop the tracker and the camera, freeing up CPU cycles for menus or other non-AR application behavior. Disabling Trackables or VirtualButtons will prevent these objects from receiving updates from the tracker, and re-enabling them will restore these updates. AR Camera Object Options The following options are available on the ARCamera object: Camera Device Mode Setting: Select between default, fast, and high-quality camera modes. Max Simultaneous Image Targets: Set the number of Image Targets the tracker should try to track simultaneously. Multi Frame Enabled: Set to split the detection of untracked Image Targets over multiple frames.

Page 56 of 73

Synchronous Video: Set to synchronize rendering updates with camera updates. Not recommended for scenes with animated content. World Center Mode: o If set to USER, you can set up a single Trackable as the world center. This Trackable will never move in the scene, instead the camera and all other Trackables will move in relation to it, as long as it is visible. o If set to AUTO, the first visible Trackable is selected to be the world center. o If set to NONE, the camera is fixed in place and all Trackables move in relation to it.

Image Target Object Options The following options are available on the Image Target object: Data Set: A drop-down list of available data sets, defined by the sets located in the StreamAssets\QCAR folder. Image Target: A drop-down list of available Image Targets, defined in the data set xml file in the StreamingAssets\QCAR folder. Preserve child size: If checked, any children of this target maintain their size when the target is changed. If unchecked, the child objects will resize to match the new target scale.

Virtual Button Options The following options are available on the Virtual Button object: Name: The name of the Virtual Button, used to uniquely identify the button when handling events. Sensitivity Setting: The sensitivity of press detection. HIGH for fast detection, LOW for robust detection.

Camera Flash and Auto Focus The CameraDeviceBehaviour provides access to two camera features, the flash and auto focus. You can turn the flash LED on with the SetFlashTorchMode method. You can also start and stop auto focus, and choose a focus mode that works with your tracking needs. See the ImageTargets sample project for a sample script that tests these various modes, Scripts/CameraDeviceMenu.cs. Camera Image The CameraDeviceBehaviour also provides access to the per-frame camera image. This can be useful for image processing, render effects, and screenshots. You can register for various image formats (e.g. RGB565, grayscale) using the SetFrameFormat method. Then you can get the camera image for the current frame with the GetCameraImage method. The image is returned as a byte array with the necessary size information. Video Background Access Video background access allows you to use the camera video as a texture. Of course this also means that you can now modify the video with your own custom shaders.

Page 57 of 73

Follow these main steps to use the video background as a texture: On start turn-off automatic video background rendering: QCARRenderer.Instance.DrawVideoBackground = false Get the native texture ID from your Unity texture object: int nativeTextureID = myTexture.GetNativeTextureID() Pass the texture ID over to QCAR: QCARRenderer.Instance.SetVideoBackgroundTextureID(nativeTextureID) In the update callback wait until the camera is active. That is the case once a video background info is available: QCARRenderer.Instance.IsVideoBackgroundInfoAvailable() Request information about the video texture from QCAR and change your update your game objects accordingly (e.g. update UVs): QCARRenderer.VideoTextureInfo texInfo = QCARRenderer.Instance.GetVideoTextu reInfo()

The QCAR-VideoBackgroundAccess sample fills you in on the details. You can use the Scripts->VideoTextureBehaviour.cs script as a starting point.

Page 58 of 73

3.2 Unity SDK API Changes

3.2.1 Default HandlersIn version 1.0 of the QCAR extension it was necessary to implement certain interfaces to achieve some default behavior. The interfaces were ITrackableEventHandler and IQCARErrorHandler. ITrackableEventHandler is used to make augmentations appear and disappear. If not implemented, the behavior was that 3D objects just stayed on screen whenever the Trackable was lost. The IQCARErrorHandler was responsible to catch errors such as unsupported device error. To speed up development of applications which only need minimal functionality we have introduced default implementations of the given interfaces (ITrackableEventHandler and IQCARErrorHandler). These default implementations are attached to the Trackable prefab (DefaultTrackableEventHandler) and the ARCamera prefab (DefaultInitializationErrorHandler) that come with the extension package. To implement your own advanced behavior just replace the Default scripts with your own implementations. Also note that the IQCARErrorHandler interface has been removed. Querying error codes is now done by calling QCAR.CheckInitializationError() at application startup.

3.2.2 DatasetsThe main change in QCAR SDK 1.5 Beta1 is the support for multiple datasets in a single application. You can now load and unload datasets whenever you want. Multiple datasets can be loaded at the same time but only a single dataset can be active at a time. 3.2.2.1 Dataset Import In version 1.0 of the QCAR Unity extension there was only support for a single dataset file (called config.xml). This file was automatically read when put into the StreamingAssets/QCAR folder and its values were automatically applied to objects in the scene. Likewise, changes in the Unity scene were automatically written back to the file whenever they were applied. With 1.5, the mechanism has become less automatic to allow for multiple datasets. What has changed for the dataset import: Datasets are checked for correct naming: This means that there need to b