Challenges and opportunities in ARToolKit development

Contents

Introduction................................................................................................................. 3

How does an ARToolKit function?...................................................................... 4

Marker and Non-marker based techniques for selecting an ARToolKit.................................................................................. 5

Software toolkits available for building AR applications................................................................................. 7

Considerations for building an ARToolKit........................................................ 9

Challenges with AR Development.................................................................. 10

How does the future look for ARToolKits?...................................................... 11

hedgehoglab.com 2

hedgehoglab.com 3

Introduction

Recent developments in the sphere of HCI (human-computer interaction) have opened new frontiers in immersive arenas including Virtual Reality (VR) and Augmented Reality (AR). Both technologies offer users a great interface and have been successfully used in areas ranging from gaming and entertainment to education and even health. Our previous white paper on the subject focused predominantly on the driver’s, growth and challenges for VR. Now, by contrast, we would like to look at the growth of AR, and the ARToolKits that are taking the software development world by storm. Indeed, such tools have been instrumental in building some of the most powerful AR apps on the market, across various platforms, including iOS and Android.

In terms of investment, all looks promising. In 2016, the AR market was valued at US$2.39bn. By 2023, that figure is expected to reach US$61.39bn, growing at a CAGR of 55.71%. And, of course, advancements in computer graphics, hardware, vision and UI (user interface) will all add impetus to this expansion.

AR, as we currently know it, represents a merger of several technologies, which aim to enhance the user’s experience of the ‘real world’. Far from being a novel development, it has existed for more than 35 years, since Ivan Sutherland first introduced the Virtual Reality application.

To understand AR, it is thus essential to distinguish it from VR. The former overlays virtual computer graphic images on top of the real world, supplementing reality as we know it, rather than replacing it. The latter, meanwhile, confines the user to a digitally constructed environment, with which he or she can interact. A sound AR system would therefore be capable of merging computer-generated graphics with the real world so seamlessly that it may even be hard to spot the difference between the two. An ideal AR platform should comprise the following features:

• The ability to combine real and virtual environments

• The potential for real time interaction with the user

• Potentially registering real and virtual objects with one another

A review of scientific literature indicates that AR is currently being used across many fields - extensively when it comes to teaching geometry, spatial relationships and molecular structures. And although the majority of people remain unfamiliar with it, we have no doubt it will spread into the mainstream as more and more AR apps make their way on to the market.

In saying this, developers need the best possible toolkits to build good quality apps, and these would combine a number of elements, such as voice, gesture, animation and persistent data to create a mesmerising impact on users.This technical white paper aims to improve understanding of the ARToolKit, explore the techniques for selecting toolkits, look at the popular toolkits available and consider what needs to be taken into account to build a high quality AR experience.

2023EXPECTED VALUATION

$61.39bn

CAGR of 55.71%

Apple ARKit on iPhone X

To understand

AR, it is thus

essential to

distinguish it from

VR. The former

overlays virtual

computer graphic

images on top

of the real world,

supplementing

reality as we

know it, rather

than replacing it.

2016 ARMARKET VALUE

$2.39bn

hedgehoglab.com 4

ARToolKits are mainly coded software libraries that enable developers to build Augmented Reality apps using languages like C and C++. Such libraries, which were initially available for free, were developed for research and non-commercial purposes. However, with the rise of powerful cellular devices, other digital gadgets and improved applications, newer commercial ARToolKits today assist developers in creating and constructing mind-bending AR apps.

So the pertinent questions are: how does toolkit function in an AR app? What role does the hardware play? What happens to the image when captured? How are markers identified and what roles do markers play in an ARToolKit?

The answers to all of the above lie in five simple steps.

• First, an image is captured through a camera and transmitted to the app

• The image is then fragmented into black and white squares

• Computer vision algorithms calculate the position of the camera, relative to the black squares

• A process of ‘homography’ transforms the image inside the black squares and compares the template with the patterns in the database. This process identifies the marker

• Once the marker is detected and the position of the camera is known, a computer graphics model is drawn from the top of the video.

The next section throws light on two common techniques used for selecting an ARToolKit.

How does an ARToolKit function?

hedgehoglab.com 5

Marker and Non-marker based techniques for selecting an ARToolKit

Camera-based ARToolKits can be selected using two techniques, marker-based and non-marker-based.

Most AR apps with image recognition are marker-based. Digital images are identified with the help of optical square markers, usually comprising black squares within white boxes of predefined sizes. The square markers are identified with the support of a camera or a digital device using an algorithm. The algorithm uses the image of the camera, as well as the decoded ID markers, to augment the virtual object onto the physical world model. Well-known examples of marker-based AR apps include: ScanLife, Popcode and, of course, Google Goggles.

With the advent of powerful HMDs (head mounted devices) and wearables accompanied by the best built-in cameras, AR apps are now capable of running markerless tracking. This medium allows the use of all small parts of the environment as a base, replacing the superimposed virtual objects. Markerless AR is thus assisted by one’s surroundings, rather than the static identified markers. Non-marker-based AR is capable of extracting and storing information about the surroundings for future use. The technology is preferred over marker-based technology, as it is non-invasive, light and can store larger images from the environment for future use. Google’s Tango, for example, uses camera vision to detect and scan the environment in real time and creates virtual images to superimpose on it, without using external signals or the GPS.

The camera-based techniques employed in AR have opened up new frontiers in commercial and entertainment spaces. We envisage that the following areas will be greatly impacted by AR tracking:

Entertainment and GamingMarkerless technology has the potential to take the entertainment sector to previously unimagined places. Imagine walking into a room and being able to weave your own gaming world... just by scanning the information available to you. Hardware such as the Microsoft HoloLens can create incredible immersive experiences in the gaming sphere. The beauty of AR lies in the fact that it does not require a television, console or standard method of control. Markerless AR can convert any plane into a gaming field and we are certain it can change the face of the gaming world.

Product showcasing and visualisation Markerless AR has the potential to impact our buying habits, especially when it comes to assets requiring long-term investments. For instance, imagine you are considering buying an electric vehicle from the likes of Tesla. However, you’re unsure which colour it might look best in, or what it might feel like to drive it in the city. How about taking a prototype experience? Jelmer Verhool released a recent prototype of such an interactive experience from Oslo, using an Apple ARKit to showcase the unreleased Tesla 3. On a similar note, imagine you want to get a new appliance fitted in your kitchen, but are unsure how the kitchen will look after this alteration. Markerless AR can help since it lets you use your mobile to scan the kitchen environment, and potentially place a virtual version of the product within it, taking customer visualisation to the next level. Such tools also allow customers to browse and shop in their own time, free from pushy salespeople.Jelmer Verhoog - Tesla Model3 in Apple ARKit

ARKit-Powered Game ‘The Machines’ on iPhone X

hedgehoglab.com 6

Healthcare education and surgeryMedical tutoring apps are capable of making healthcare education more immersive and interactive. This is especially so with apps made with the recent ARToolkits, as compared to those developed without ARKit features. So, how would ARKit framework help improve learning in medicine?

• Plane detections make it possible for medical students to study virtual 3D models directly on their desks. The learning experience is thereby rendered richer and more interesting, compared to learning through photographs and videos.

• Visual Inertial Odometry supports training medical interns. 3D models mean the anatomy, including organs, can be viewed from any angle and distance.

ARToolkits can even make it easier, safer and faster to carry out surgery, improving a number of processes, from tumour identification and removal to visualising blood flows.

Military applicationIn the army, HMDs with AR apps can provide real time assistance to soldiers in battle zones. The devices can transmit critical information, including the location of enemies, battle tanks or any other element that poses a threat. Similarly, fighter jets can be equipped with ‘heads up’ AR apps featuring transparent displays to relay critical information such as altitude, airspeed, the horizon line and key data relevant to war zones.

AR apps in NavigationEnhanced GPS combined with AR would allow easy movement from one location to another. Using a smartphone’s camera in combination with the GPS, users could view selected routes overlaid on the live view in front of their vehicles.

Repairs and maintenance in manufacturingUsing HMDs, technicians can see superimposed images of machines and engines with detailed information in the actual line of sight. Detailed procedures for fixing the machines could also be displayed in the corner of the devices, with live 3D images showing exactly what has to done to repair broken parts. AR apps can label essential components and help simplify the complex steps involved in major repairs. Similarly, simulations can be used to train technicians, significantly reducing training expenses.

Marker and Non-marker based techniques for selecting an ARToolKit

Limpid Armor AR system for armoured vehicles

HoloLens Mixed Reality Surgery

ARToolkits can even make it easier, safer and faster to carry out surgery, improving a number of processes, from tumour identification and removal to visualising blood flows.

Software toolkits available for building AR applications

AR applications are built using different software toolkits. In this section, we will compare them to help developers make wise choices. Daqri-ARToolKitInitially introduced as an open source code, this toolkit was then acquired by Daqri, which now provides the commercialised version. The original toolkit still exists, though, and hence free libraries are available to all. The toolkit includes work instructions, data visualisation templates, and distinct methods and best practices for creating AR content. It supports the following functions:

• Tracking black square and planar images• 2D image recognition, camera calibration

and optical stereo calibration• Single camera orientation tracking• Optical HMD display support• Mapping additional elements via OpenGL

(graphics library)• Plugins for Unity and OpenSceneGraph• Supports optical head-mounted display • Fast enough for real time AR applications

Supported platforms: Android, iOS, Linux, Windows and Mac OS.

The open source comes with disadvantages of its own — most significantly, the SDK’s documentation is quite weak. And although the libraries contain prototypes for building test apps, the illustrations are fairly mediocre and offer no information on the plans for a framework update.

7hedgehoglab.com

Wikitude SDK animated 3D models

Limpid Armor AR system for armoured vehicles

VuforiaOne of the most popular platforms for AR development, Vuforia - which is supported on Android, iOS, UWP and Unity Editor - boasts the following features:

• recognises the different types of visual objects (a box, cylinder, planes), as well as text and environments

• facilitates scanning of targeted objects using Vuforia Object Scanner

• integrates powerful plugins at various levels• incorporates virtual buttons and maps

additional elements via OpenGL• extends tracking to deliver a continuous

visual experience, even when the target is out of sight

Vuforia’s framework is incomplete, which can complicate matters for developers working with the SDK for the first time. Commercial versions of the software are free from watermark, while the free versions are not particularly flexible to work with and less suited for cloud platforms.

WikitudeWikitude, another powerful library, allows developers to create AR apps with next level functions including: reconstructing places on virtual maps; facilitating AR gaming; and offering real time searching of events, articles and social media posts. The kit can be used in both Android and iOS, and supports Unity. Moreover, the documentation of the library is well structured and elaborate. Developers can avail of the free versions initially. However, premium versions are chargeable on a regular basis. Features of Wikitude include: image recognition and tracking, 2D and 3D tracking technology, GEO data, cloud recognition and HTML augmentation.

Daqri Smart Glasses and Helmet

hedgehoglab.com 8

KudanKudan boasts one of the fastest frameworks of all software toolkits. The SDK assists in mapping multi-polygonal models and facilitates easy import of 3D models from the modelling software packages. The libraries require minimal memory space to store images on devices. However, the toolkit library framework is short and developers would require more detailed information to build AR apps. The library also suffers from limited built-in functionality as it has no direct access to OpenGL. Kudan offers the following capabilities :

• Easy recognition of 2D and 3D images• Compatibility with Unity and the ability to

generate databases • Allows markerless tracking (instead of fiducial

marks, it relies on the use of natural features like edges, corners or textures)

• Maps additional elements through separate component over OpenGL

Supported platforms: Android and iOS.

Apart from the above, two new kits, introduced Apple and Google have made their foray to the world of app development. It’s certainly worth looking into these toolkits offered by the giants, and how they are capable of possibly changing the immersive landscape?

ARKitApple’s ARToolKit, known as the ARKit, has been doing the rounds for quite some time, and was officially acknowledged at the WWDC 2017. It was introduced to help developers integrate AR experiences into apps and games. Working with the iOS and using the phone’s camera, processors and motion sensors, the toolkit aims to enhance immersive experiences.

ARKit uses Visual Inertial Odometry (VIO) to track the space around Apple’s devices, including iPads or iPhones. VIO enables the devices to sense empty spaces while in motion. Further, the kit is capable of analysing room layouts and detecting horizontal planes like tables and floors. This means virtual objects can be placed on those surfaces. Take, for instance, IKEA’s new app, developed using ARKit. This enables customers to visualise the placement and positioning of furniture on planes and empty spaces before making purchases.

ARCoreARCore, Google’s toolkit, is available on Android 7.0 Nougat and all more recent operating systems. Developers can use it for building apps for the Pixel, Galaxy S8 or any other OEM devices. It works with an OpenGL, Unity and Unreal, and boasts three distinct features:

• Motion tracking - Using the phone’s camera to detect an individual’s position in an open space

• Environmental understanding - capable of detecting plane and horizontal surfaces in the surroundings

• Light estimation - allows lighting and shadow of virtual objects to match the surroundings

ARCore supports other 3D tools from Google like Blocks and Tilt Brush, as well as gaming projects from the likes of Epic Games, Niantic (the builders of Pokemon Go) and Wayfair. Given Android’s huge market base - millions of devices are powered by the platform - the toolkit has the potential to become the biggest in the world.

Software toolkits available for building AR applications

Apple ARKit

ARCore

hedgehoglab.com 9

Considerations for building an ARToolKit

As discussed previously, one of the fundamental problems involved in developing AR applications is that of tracking the user’s viewpoint. ARToolKit uses computer vision algorithms to solve this issue by calculating the real camera position and orientation relative to physical markers in real time. It’s not an exact science, but is impressively accurate, resulting in realistic AR experiences. So the key question is: how does one build a robust, high-quality AR experience? What factors need to be considered?

Lighting conditions Tracking becomes challenging and tedious when lighting conditions are poor, since the process involves image analysis. Indeed, tracking quality becomes abysmal when a camera is projected on a blank wall or used in an area that is too dark. Good quality lighting conditions are therefore advisable for generating amazing AR experiences.

Generation of user feedbackTracking assists in analysing a real life image when a device is in motion. ARToolKit helps users understand the scene better, even if the device moves subtly. However, vigorous motion can result in a blurred image. Good quality AR cameras provide tracking state information, which programmers can use for developing UI and to resolve issues related to low-quality tracking scenarios.

Plane detectionPlane detection is an important phenomenon for AR, as it is capable of producing sharp, unequivocal images. ARToolKits refine the position and time extent once a plane is detected.

ARCore plane detection

hedgehoglab.com 10

Challenges with AR Development

Building an AR application isn’t easy and developers face challenges from two ends - firstly on the hardware front and secondly when it comes to software. Hardware related challengesSensor readings are crucial for AR apps. However, accelerometer and compass mechanisms in digital devices are not capable of filtering electric inference from vastly populated surfaces. The problem can possibly be resolved through an exponential smoothing technique, used for smoothing time series data.

Weak pattern recognitionsCameras in current hardware are well equipped to take high quality 2D images. However, they are still quite weak when it comes to identifying 3D images. One possible solution could be to use 2D QR and barcode as Augmented Reality markers.

GPS faultsCurrent digital devices can detect readings up to six metres away, quite enough for digital maps and navigation systems. However, GPS tracking can impact the AR software performance if the AR markers installed in the phones are not big enough.

Integration and security issuesCurrent web application designs restrict AR browsers to integrate their functions into social media platforms. AR solutions are still susceptible to data leaks and software is incapable of preventing hackers from invading the personal information of users. One possible solution, however, would be to use strong AR frameworks in the first place.

Lack of multi-user experienceAR solutions developed for digital devices are incapable of providing a multi-user experience, with apps being limited to single users. Therefore, ARToolkits need to develop libraries that allow multi-user software to be built.

Google AR Experiments - Hidden World. Reveal an animated world at your feet. By Rachel Park Goto & Jane Friedhoff

hedgehoglab.com 11

How does the future look for ARToolKits?

With the current US$ 2bn market expected to grow around 30 times in 5-6 years the future certainly looks bright. Hardware improvements in digital devices combined with rapid strides in computer graphic development and UI design will also boost the growth of the sector. Development of non-invasive markerless technology is certainly set to disrupt the field, taking over marker-based technology and opening new avenues in areas like entertainment and commerce.

The different ARToolKits discussed in this white paper offer various features - and a wide range of possibilities. As toolkits become better, they will provide an even wider range of options for developers looking to create world-class apps. A quantum leap in AR technology is certainly on the anvil, as the ARToolKit libraries present great opportunities for the prominent platforms like iOS and Android.

hedgehog lab AR projects Northumbria Water - read more. Road Respect - read more.

Microsoft and Zero Carbon Futures - read more

October 2017

References http://eprints.utm.my/6013/1/147-nihra.pdfhttps://developer.apple.com/documentation/arkit/understanding_augmented_reality#see-alsohttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.5966&rep=rep1&type=pdfhttp://www.tinmith.net/lca2004/ARToolkit/ARToolKit2.33doc.pdfhttps://developer.apple.com/documentation/arkit/understanding_augmented_realityhttps://www.ideals.illinois.edu/bitstream/handle/2142/27688/AR_Smart_Phone_Note_rev3.pdf

Blog references http://cases.azoft.com/top-5-tools-creating-augmented-reality-apps/https://thinkmobiles.com/blog/best-ar-sdk-review/http://r-stylelab.com/company/blog/it-trends/the-challenges-of-augmented-reality-developmenthttps://thinkmobiles.com/blog/best-ar-sdk-review/https://appreal-vr.com/blog/apple-arkit-healthcare-applications/

About ushedgehog lab is a global software consultancy that specialises in multi-platform software and connected device innovation. We architect, design, develop, and market apps, augmented reality, virtual reality, mixed reality & other digital solutions for phones, tablets, TV, kiosks & other post-PC devices.

We help brands & enterprises prepare for a future dominated by post-PC technologies. We specialise in working with our clients on Innovation & R&D on mobile & emerging platforms.

Founded as a pure play post-PC consultancy in 2010, and with offices in London, Newcastle Upon Tyne, Boston, Austin, India & Copenhagen, hedgehog lab has a team that is growing rapidly by focusing on delivering great post-PC experiences and campaigns to clients across the UK & abroad.

hedgehog lab has architected and developed mobile solutions and software for clients like News International, Channel 4, Financial Times, Booths, Societe Bic, East Coast Trains, AkzoNobel, B&M, Mitsubishi, CBRE, England Rugby, NESN, Northumbrian Water, Camping and Caravanning Club, HULT, Thinking Digital and Grainger plc.

With smartphone sales accelerating at a rapid pace and tablet technologies being spear-headed by the iPad, we have bet our future on a completely post-PC world where mobile devices, smart connected devices and touch screen technologies out-pace fixed desktop PC devices.

We are one of the leading smart phone & tablet application developers in our region, working with cutting-edge mobile platforms like the iOS, tvOS, WatchOS, Android, Augmented Reality and Virtual Reality.

Newcastle, UK Generator Studios,Trafalgar Street,Newcastle upon Tyne,NE1 2LA

t: 0191 249 8039

London, UK 18-21 Corsham Street,London,N1 6DR

t: 020 3198 2470

Boston, USA 15th Floor,50 Milk Street, Boston, MA,02109

t: 617-806-6434

Austin, USA 600 Congress Avenue,14th Floor,Austin, TX,78701

t: 512-348-6566

India Unit 301 Maple Celestia,Plot 49 Jayabheri Enclave,Gachibowli, Hyderabad, Telangana 500032

t: +9140 65888039

Copenhagen, Denmark Hejrevej 30,2400 Nørrebro,Copenhagen,

t: +45 78755970

w: hedgehoglab.com

e: info@hedgehoglab.com

@hedgehoglab