augmented reality basic

8/6/2019 Augmented Reality Basic

1/12

Requirements

Additional required products

External code libraries:

y Flex SDK code libraryy FLARToolkity Papervision3D code library

Optional drawing/rendering programs:

y FLAR Marker Generator[AIR, 322K]y Blender3D

User Level

Beginning

Required products

y Flash Professional (Download trial)Sample files

y augmented_reality_source.zip Augmented reality (AR) is a field of computer science that involves combining the physical world and an

interactive, three-dimensional virtual world. This article takes a look at augmented reality, its current uses,

and its future potential. Then you'll dig in and see how to apply this exciting technology using Adobe Flash

CS4 Professional, Adobe Flash Player 10, and a webcam.

While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined

with decades of computer science development. Virtual reality (VR), AR's more familiar counterpart, is the

replacement of the user's physical reality (particularly that which is experienced through sight and hearing)

with a computer-generated reality. The idea of a virtual experience is excitingcreating entertaining and

educational sensory encounters that do not exist in our everyday lives.

From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR

advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing

reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) andcolor effects (such as night vision or thermal vision) to reality creates a better user experience. The user is

still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive

with today's technology, making it more feasible than VR.

Applications of augmented reality

The video game industry has released major augmented reality products for more than a decade.

The EyeToy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites

the live video feed with CG onto the TV screen. This toy detects the user's body position in front of the

camera as an alternative input method to the typical joystick or gamepad, deepening the user's immersion

into the game world. Detection schemes continue to advance, allowing for more engaging interaction.

There are AR applications outside of console games, including military and consumer products, too. Night-

vision goggles and targeting-assistance technology help marksmen in battle, and children's stories come to

life with AR-enhanced books. The uses are vast.

With AR-enhanced books, each page of a real-world book is outfitted with a markera pattern that is

detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top

of a physical image printed on paper. In this article, you will learn to create this type of AR experience.

Let's dig in!


2/12

Augmented reality with Flash CS4

Let's create a project in which a webcam will capture the user, holding a marker image in an arbitrary

position, and the application will augment the webcam feed with a 3D model overlaid onto the marker's

position in a believable way. You can get this project up and running in less than an hour.

With Flash Player 10, developers have the most robust toolset yet for rich application development. Flash

Player handles the heavy lifting such as video input, pixel-level image manipulation, and the heavy number

crunching afforded with ActionScript 3. All you need is a webcam, a few open source libraries, and Flash

CS4 to get the job done.

New project setup

Before coding a Flash CS4 project, you need to prepare the libraries and assets. Once the project is set up,

most changes will happen within the document class (see Figure 1). To get started:

1. Create a new project in Flash CS4 by choosing File > New > Flash File (ActionScript 3.0), and savethe project asAugmentedReality.fla inside an AugmentedReality folder on your desktop.

2. Create a document class by choosing File > New > ActionScript File, and save itas AugmentedReality.aswithin the AugmentedReality folder.

3. Set this file as the project's document class to the AugmentedReality class file by choosing Window >Property > Class. The class file will be empty at this point.

4. Create a libs folder inside the AugmentedReality folder, and add it to the project's library path bychoosing File > Publish Settings > Flash > ActionScript 3.0 > Library Path > [Folder Icon].

5. Download the three libraries discussed in the next section, and add each SWC to the libs folder.6. Create an Assets folder and add the 3D Collada model file and two AR data files.

Figure 1. Augmented reality project folder after setup


3/12

External library setup

A great benefit of ActionScript3's object-oriented programming model is that developers can

importexisting libraries of code to save ramp-up time and vastly expand thedevelopment power of Flash.

This implementation of AR takes advantageof three such libraries, each with a specific role in the project:

y Flex SDK code library: Processes the Embed metatag needed to import FLAR data files.y SparkProject's Flash Augmented Reality code library (FLARToolkit): Handles marker graphic

detection.

y Papervision3D code library (PV3D): Handles the importing, positioning, and rendering of the 3Dmodel.

Asset preparation

TheFlash Augmented Reality Toolkit (FLARToolkit) library holds much of themagic for this application.

It will introspect the webcam image for aparticular graphic image and judge where in real space to map the

3Dmodel. The FLARCameraParams.dat file, which comes with the toolkit,will be used as-is. It corrects

webcam distortion and helps the toolkitwith its general detection routine. The graphic marker

imageinformation is held within the FLARP

attern.pat file. The file is madeto work with a particular markergraphic image (see Figure 2). If youwant to use a different graphic, create a JPEG image with your

favoritedrawing program, and use the FLAR Marker Generator(AIR, 322K) to generate a new pattern file.

Figure 2. Tower marker graphic

The marker image is a graphic drawn, printed, and shown to the endapplication as it runs. FLAR, with help

from the marker data file andparameters file, will detect this shape via webcam. To design agraphic, fit the

desired shape within a white square that is centeredwithin a larger black square. Keep your shape simple

for best results.

Notethe sun shape in the graphic. This helps the application detect whichway is up for the marker and

assists its angle and rotation detection.Your design doesn't need to have this shape per se, but it should

beasymmetrical in some way for best results.

Also, you'll needa 3D model in the Collada format that Papervision3D requires. Colladamodels can be

created and exported using many popular 3D modelingtools, including the open sourceBlender3D.

The document class


4/12

TheAugmentedReality.as document class is where all of the project's codelives. The following code shows

the completed class. As you can see,much of the complexity of FLAR and Papervision3D for the project

ishidden within the imported libraries and is therefore out of the way:

package {

//--------------------------------------

// Imports

//--------------------------------------

import flash.display.BitmapData;

import flash.display.Sprite;

import flash.events.Event;

import flash.media.Camera;

import flash.media.Video;

import flash.utils.ByteArray;

import org.libspark.flartoolkit.core.FLARCode;

import org.libspark.flartoolkit.core.param.FLARParam;import

org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;

import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;

import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;

import org.libspark.flartoolkit.pv3d.FLARBaseNode;

import org.libspark.flartoolkit.pv3d.FLARCamera3D;

import org.papervision3d.lights.PointLight3D;

import org.papervision3d.materials.shadematerials.FlatShadeMaterial;

import org.papervision3d.materials.utils.MaterialsList;

import org.papervision3d.objects.parsers.DAE;

import org.papervision3d.objects.primitives.Cube;

import org.papervision3d.render.BasicRenderEngine;

import org.papervision3d.scenes.Scene3D;

import org.papervision3d.view.Viewport3D;

//--------------------------------------

// Class Definition

//--------------------------------------

public class AugmentedReality extends Sprite

{

//--------------------------------------

// Class Properties

//--------------------------------------

// 1. WebCam

private var video : Video;

private var webcam : Camera;


5/12

// 2. FLAR Marker Detection

private var flarBaseNode : FLARBaseNode;

private var flarParam : FLARParam;

private var flarCode : FLARCode;

private var flarRgbRaster_BitmapData : FLARRgbRaster_BitmapData;

private var flarSingleMarkerDetector : FLARSingleMarkerDetector;

private var flarCamera3D : FLARCamera3D;

private var flarTransMatResult : FLARTransMatResult;

private var bitmapData : BitmapData;

private var FLAR_CODE_SIZE : uint = 16;

private var MARKER_WIDTH : uint = 80;

[Embed(source="./assets/FLAR/FLARPattern.pat",

mimeType="application/octet -stream")]

private var Pattern : Class;

[Embed(source="./assets/FLAR/FLARCameraParameters.dat",mimeType="application/octet -stream")]

private var Params : Class;

// 3. PaperVision3D

private var basicRenderEngine : BasicRenderEngine;

private var viewport3D : Viewport3D;

private var scene3D : Scene3D;

private var collada3DModel : DAE;

// Fun, Editable Properties

private var VIDEO_WIDTH : Number = 640; //Set 100 to 1000 to set width

of screen

private var VIDEO_HEIGHT : Number = 480; //Set 100 to 1000 to set

height of screen

private var WEB_CAMERA_WIDTH : Number = VIDEO_WIDTH/2; //Smaller than

video runs faster

private var WEB_CAMERA_HEIGHT : Number = VIDEO_HEIGHT/2; //Smaller

than video runs faster

private var VIDEO_FRAME_RATE : Number = 30; //Set 5 to 30. Higher

values = smoother video

private var DETECTION_THRESHOLD : uint = 80; //Set 50 to 100. Set to

detect marker more accurately.

private var DETECTION_CONFIDENCE : Number = 0.5; //Set 0.1 to 1. Setto detect marker more accurately.

private var MODEL_SCALE : Number = 0.8; //Set 0.1 to 5. Set higher to

enlarge model

// Fun, Editable Properties: Load a Different Model

private var COLLADA_3D_MODEL : String =

"./assets/models/licensed/hummer/models/hummer.dae";


6/12

//--------------------------------------

// Constructor

//--------------------------------------

/**

* The constructor is the ideal place

* for project setup since it only runs once.

* Prepare A,B, & C before repeatedly running D.

**/

public function AugmentedReality()

{

//Prepare

prepareWebCam(); //Step A

prepareMarkerDetection(); //Step B

preparePaperVision3D(); //Step C

// Repeatedly call the loop method

// to detect and adjust the 3D model.

addEventListener(Event.ENTER_FRAME, loopToDetectMarkerAndUpdate3D);

//Step D

}

//--------------------------------------

// Methods

//--------------------------------------

/**

* A. Access the user's webcam, wire it

* to a video object, and display the

* video onscreen.

**/

private function prepareWebCam() : void

{

video = new Video(VIDEO_WIDTH, VIDEO_HEIGHT);

webcam = Camera.getCamera();

webcam.setMode(WEB_CAMERA_WIDTH, WEB_CAMERA_HEIGHT, VIDEO_FRAME_RATE);

video.attachCamera(webcam);

addChild(video);}

/**

* B. Prepare the FLAR tools to detect with

* parameters, the marker pattern, and

* a BitmapData object to hold the information


7/12

* of the most recent webcam still-frame.

**/

private function prepareMarkerDetection() : void

{

// The parameters file corrects imperfections

// In the webcam's image. The pattern file

// defines the marker graphic for detection

// by the FLAR tools.

flarParam = new FLARParam();

flarParam.loadARParam(new Params() as ByteArray);

flarCode = new FLARCode(FLAR_CODE_SIZE, FLAR_CODE_SIZE);

flarCode.loadARPatt(new Pattern());

// A BitmapData is Flash's version of a JPG image in

memory.

// FLAR studies this image every frame with its

// marker-detection code.bitmapData = new BitmapData(VIDEO_WIDTH, VIDEO_HEIGHT);

bitmapData.draw(video);

flarRgbRaster_BitmapData = new

FLARRgbRaster_BitmapData(bitmapData);

flarSingleMarkerDetector = new FLARSingleMarkerDetector(flarParam,

flarCode, MARKER_WIDTH);

}

/**

* C. Create PaperVision3D's 3D tools including

* a scene, a base node container to hold the

* 3D Model, and the loaded 3D model itself.

**/

private function preparePaperVision3D() : void

{

// Basics of the empty 3D scene fit for

// FLAR detection inside a 3D render engine.

basicRenderEngine = new BasicRenderEngine();

flarTransMatResult = new FLARTransMatResult();

viewport3D = new Viewport3D();

flarCamera3D = new FLARCamera3D(flarParam);

flarBaseNode = new FLARBaseNode();scene3D = new Scene3D();

scene3D.addChild(flarBaseNode);

// Load, scale, and position the model

// The position and rotation will be

// adjusted later in method D below.

collada3DModel = new DAE();


8/12

collada3DModel.load(COLLADA_3D_MODEL);

collada3DModel.scaleX = collada3DModel.scaleY =

collada3DModel.scaleZ = MODEL_SCALE;

collada3DModel.z = 5; //Moves Model 'Up' a Line

Perpendicular to Marker

collada3DModel.rotationX = -90; //Rotates Model Around 2D

X-Axis of Marker

collada3DModel.rotationY = 180; //Rotates Model Around 2D

Y-Axis of Marker

collada3DModel.rotationZ = 90; //Rotates Model Around a

Line Perpendicular to Marker

// Add the 3D model into the

// FLAR container and add the

// 3D cameras view to the screen

// so the user can view the result

flarBaseNode.addChild(collada3DModel);

addChild(viewport3D);}

/**

* D. Detect the marker in the webcamera.

* If found: move, scale, and rotate the

* 3D model to composite it over the marker

* in the user's physical space.

**/

private function loopToDetectMarkerAndUpdate3D(aEvent : Event) : void

{

// Copy the latest still-frame of the webcam video

// into the BitmapData object for detection

bitmapData.draw(video);

try {

// Detect *IF* the marker is found in the latest

still-frame

if(flarSingleMarkerDetector.detectMarkerLite(flarRgbRaster_BitmapData,

DETECTION_THRESHOLD) &&

flarSingleMarkerDetector.g etConfidence() > DETECTION_CONFIDENCE) {

// Repeatedly Loop and Adjust 3D Model to Match

Marker

flarSingleMarkerDetector.getTransformMatrix(flarTransMatResult);

flarBaseNode.setTransformMatrix(flarTransMatResult);

basicRenderEngine.renderScene(scene3D, flarCamera3D, viewport3D);

}


9/12

} catch(error : Error) {}

}

}

}

Stepping through the document class

ActionScript 3 class basics

ActionScript 3 class files are structured into several sections, and each has a specific role in

this project.

y Imports:Typically your class will take advantage of existing codelibrarieslibraries youmay not have coded yourself. Each importstatement loads in the needed class or package

from an existinglibrary. You need libraries from Flash, FLARToolkit, and

Papervision3Dfor this project.

y Class definition: The bulk of your classes code should be in the class definition section.y Class properties:Declare all public, private, and static variables. Arguments and

localvariables are declared within the constructor body and the methodbodies. For this

project, you need only private variables; no outsideclasses need access.

y Constructor: Thisfunction runs when the class is created. The function's name mustmatchthe class name. It runs only once so it's an ideal place to initializeyour class. In the

following steps, setup code will be placed here. Inthis project, code is placed to set up the

camera, FLARToolkit, andPapervision3D and to prepare a repeating loop to handle

markerdetection.

y Methods: Methods are classfunctions or reusable chunks of code. This project uses onlyfourmethods to handle setup and marker detection.

Imports and properties

Startingat the top of the AugmentedReality.as document class, we see importstatements for the needed

libraries. Within the Properties section,variables are declared. Most of this section is pretty standard,

exceptthe Pattern and Params property declarations from lines 5862. Thissection showing the Flex Embed

metatag within Flash CS4 appears here:

[Embed(source="../assets/FLAR/FLARPattern.pat",


private var Pattern: Class;

[Embed(source="../assets/FLAR/FLARCameraParameters.dat",


private var Params : Class;

Adobe Flex uses a special type of code called the Embed metatag.Embed tags, as they are commonly

called, enable a property to beinitialized with a value loaded from an external file. A unique aspectis that

the byte loading happens at compile time (as the project ispublished) rather than at runtime (while the user

interacts with theapplication). The bytes for the loaded file are baked into the SWFfile. Flash CS4 now

supports this feature as well with help from theflex.SWC, which is included in this project's libs folder.


10/12

Class constructor

Sincethe constructor of a class is called only once, it is an ideal locationfor one-time-only setup code.

Starting on line 93, the preparationmethods are called for each of the following: the webcam, the

markerdetection, and Papervision3D. In addition to the setup methods,

the loopToDetectMarkerAndUpdate3D function is set up to run again and again. By listening to

theENTER_FRAME event built into Flash, this loop will be called repeatedlyas long as the application is

running.

Webcam setup

Flashfeatures great webcam support. As the application runs and a camera isdetected, Flash Player will

automatically prompt the user to enable itas line 96 of the code executes. The Video object is the

windowonscreen where the video will be shown, and the Camera object recordsand feeds its live footage to

that Video object.

FLAR setup

Acornerstone of this project is marker detection. As the marker graphicis detected via webcam, itsposition, rotation, and scale in the user'sphysical space are calculated. This calculation will be used in

thenext steps. Starting with line 97, FLAR will search each still-frame ofwebcam footage for the

predefined marker graphic (Pattern object). Thedetection scheme uses an ActionScript BitmapData object,

which containsonly the latest still-frame of video at any given time (the BitmapDataDraw function).

Papervision3D setup

For the mostpart, the 3D setup at line 98 is typical of any Papervision3D project.A BasicRenderEngine

object will handle the bulk of the vector math,converting 3D geometry to 2D graphics for display on the

screen. TheCollada 3D model is loaded and inserted into the 3D scene, and the viewinto that scene

(ViewPort object) is added to the Stage (see Figure 3).Luckily this is all done for you. In the final project,

this view islayered on top of the 2D webcam Video object shown earlier so thecomposite of 2D and 3D

will look believable.

Figure 3. Collada model loaded and rendered with Papervision3D


11/12

Repeated detect-and-update routine

As mentioned earlier, by using the ENTER_FRAME you ensure

this loopToDetectMarkerAndUpdate3D function on lines 102 will be called 30 times per second.

Each time,the BitmapData object will copy in the latest still-frame from thewebcam footage. FLAR runs

detection on the still-frame. If the markeris detected, FLAR returns updated position, rotation, and scale

data(TransformMatrix object). The container (FLARBaseNode object) holdingthe 3D model is updated to

that position so the model appears to be atthe same position in the user's physical space as the marker.

Theresulting effect is nothing short of amazing for first-time viewers.

Running the completed project

Printout the graphic marker. Show the marker to the running application viathe webcam (see Figure 4).

The red line is for illustration purposes todenote successful initial marker detection. Much like

ArnoldSchwarzenegger's Terminator character, the project can now read in thesurrounding environment,

visually scan and detect relevant objects, andmake calculations based on that information. It works best to

have asimple backdrop of a solid wall in the application because complexscenes may confuse the detection

scheme.

Figure 4. Application detecting the graphic marker via webcam

Next, seeing the marker graphic through the webcam, the applicationsuperimposes an animated 3D model

onto the user's physical space (seeFigure 5). The model now rotates and scales believably as the usermoves

the marker.


12/12

Figure 5. Application replaces the marker with a 3D model

Where to go from here

There you have it. Augmentedreality is within the grasp of all Flash developers. The FLARToolkitcan

detect multiple markers to bring even more 3D into your world. Astechnology matures, you will be able to

detect more complex graphicmarkers within more complex scenes. The choices are endless. What willyou

choose to do with augmented reality: print and show a marker to anonline application as we've done here,

buy and play with a real-worldAR-enabled toy, or detect and destroy like the Terminator?

augmented reality basic

Documents