X3D Extension for

(Mobile) AR Contents

International AR Standards WorkshopSeoul, Korea

Oct 11-12, 2010

Gerard J. Kim (WG 6 AR Standards Study Group Coordinator)

Korea University

Approach

Extensibility to existing frameworks

X3D (Scene graph)

Because AR is implemented as VR!

KML, OpenGIS, …

We need location representation

Generality/Flexibility to accommodate

Different AR platforms (~Platform independence)

Mobile, Desktop, HMD, …

Sensors and devices

Vision based, Marker based, Location based, …

Focused on file format (Scene graph based?)

vs. Contents representation

Machine consumption

Various display types and platforms

<Video See-through> <Optical See-through>

<Desktop>

VideoCombiner

Optical Combiner

DisplayCamera

Display

DisplayCamera

VideoCombiner

[R. Azuma, 1997]

<Mobile>

camera

f

fovVirtual cam

f’ = f

fov’ = fov

Transform T (from tracking system) T = T’

Live camera capturing the real world Synthesized mixed reality world as a virtual space with video placed at the same distance (e.g. f = f’) from virtual camera position but with infinite depth values

AR/MR Implementation

Various sensing

Virtual cam

virtual object

sensed

Live video with depth

Background pixelsrendered at their depth

Virtual cam

virtual object

sensed

Live video with depthVirtual cam

virtual object

sensed

Live video with depth

Background pixelsrendered at their depth

Various sensing

MR/AR Contents

• Context: Condition or situation that triggers an augmentation and mixing of real and virtual objects

• Resource: Raw data or information used for augmentation

• Content: One or more pairings of contexts and Resources + behaviors (that uses the resources)

context

resource

Related work

Jung et al. (InstantReality Suite)Extension of Sensor nodes – Physical contextsExtension of Viewpoint nodes – Specification of camera parametersLayers: One layer served as background videoExtension of X3DLightNode: Lighting effects

SFImageSenosr : X3DDirectSensorNode {SFImage [in/out] value …SFBool [] out FalseSFString [] label…}

DEF frame SFImageSensor { label “Video Frames” }

ROUTE frame.value_changed TO surfaceTex.set_image

Major proposals

Extend “View” node: Resolution between “live” camera and virtual

Define “Live” camera node (G. Lee / ETRI)

– Not necessarily for “AR” contents (e.g. Video textures)

– Parameters set by user

More detailed parameter specification for “View”

Set by user

Routed from “Live” camera node

– With possibility of behavioral manipulation

“Routed” from sensor

– Camera could be tracked separately

Default: same as the world

– Note that view can be relative to anything

LC, VC

LC

VC

LC, VC

Major proposal

Extending movie texture node (for AR background)

Also proposed by G. Lee / Instant Reality

Extend existing virtual “Sensor” nodes

New X3DARNodes for target real object description

ImagePatch, 3DObject, GPSLocation, SingleValue, …

Existing: E.g. Visibility, Proximity, Touch sensor …

New: RangeSensor, UIClickSensor, …

Not included in this proposal

Lighting and Rendering issue

Depth sensing and occlusion effects

Extended point of interest (e.g. path, hierarchical POI)

Platform type specification

e.g. Resolution difference

AR contents(Real/

Physical)

X3D (Virtual

)

WORLD

View(Virtual Camera)

OtherX3D Nodes

AR Node + Sensor

ROUTE*

LiveCamera

Movie Texture*

Virtualized Physical Contexts

Abstraction of MR/AR contents as a collection of context and resources

connected by “Event in”’s and “Event out”’s.

<Scene> <Group> <TouchSensor DEF='TOUCH' description='touch to activate'/> <TimeSensor DEF='TIME' cycleInterval='3'/> <PositionInterpolator DEF='INTERP_POS' key='0 0.25 0.5 0.75 1' keyValue='0 0 0 1 0 0 0 0 0 -1 0 0 0 0 0'/> <Transform DEF='BALL'> <Shape> <Appearance> <Material/> </Appearance> <Sphere/> </Shape> </Transform> </Group><ROUTE fromField='touchTime' fromNode='TOUCH' toField='startTime' toNode='TIME'/><ROUTE fromField='fraction_changed' fromNode='TIME' toField='set_fraction' toNode='INTERP_POS'/><ROUTE fromField='value_changed' fromNode='INTERP_POS' toField='translation' toNode='BALL'/></Scene>

<Scene>

<Group>

<Marker DEF = “HIRO” enable “TRUE” filename=”C:\

hiro.patt”/>

<VisibilitySensor DEF='Visibility' enabled=”TRUE”/>

<Transform DEF='BALL'>

<Shape>

<Appearance>

<Material/>

</Appearance>

<Sphere/>

</Shape>

</Transform>

</Group>

<ROUTE fromNode=’Vsibility’ fromField='visible' toNode=’BALL’

toField=’visible’ />

</Scene>

X3DNode

X3DARNode X3DChildNode

X3DSensorNodeImagePatch

3DObject

GPSLocation

X3DEnvironmentalSensorNode

UIConfigNode

VisibilitySensor

ProximitySensorSingleValue

UIDevice

…

UIClickSensor

UIScrollSensor

…

RangeSensor

…

• Vision based feature recognition and tracking (e.g. fiducials, markers, 3D points)

• Non-vision based env. sensor events and values (e.g. RFID, GPS, distance)

• User interaction devices events and values (e.g. buttons, touch screen, jog dial)

• Context information (e.g. user age)

Real Object X3DARNode Main Attributes Sensor used

Marker ImagePatch ID, Position, Orientation

Visibility

3D point 3DObject ID, Type, Position, Orientation

Visibility

GPS Location GPSLocation ID, Coordinate Range

RFID SingleValue Value (Boolean) Existence

Ultrasonic sensor SingleValue Distance (Integer) Proximity

Button UIDevice Value (Boolean) UIClickSensor

User Age SingleValue Age (Integer) Range

X3DARNode

Placeholders for physical objects within AR/MR world “implementation”

X3DARNode : X3DNode{

SFNode [in, out] metadataSFNode [in, out] parentSFString [in, out] descriptionSFBool [in, out] enabled

}

X3DARNode is the base type for the Marker, Location and General Event, …

ImagePatch (Marker) & VisibilitySensor

ImagePatch : X3DARNode{

SFNode [in, out] metadataSFNode [in, out] parentSFString [in, out] descriptionSFBool [in, out] enabledSFString [in, out] filenameSFVec3f [in, out] positionSFRotation [in, out] orientation

}

VisibilitySensor : X3DEnvironmentalSensorNode <!-- Existing -->{

SFVec3f [in, out] centerSFBool [in, out] enabledSFNode [in, out] metadataSFVec3f [in, out] sizeSFTime [out] enterTimeSFTime [out] exitTimeSFBool [out] isActive

}

Location & RangeSensor GPSLocation : X3DSensorNode{

SFNode [in, out] metadataSFNode [in, out] parentSFString [in, out] descriptionSFBool [in, out] enabledSFInt32 [in, out] device_descriptionSFBool [out] statusMFString [out] values

}

RangeSensor : X3DEnvironmentalSensorNode{

SFVec3f [in, out] centerSFBool [in, out] enabledSFNode [in, out] metadataSFVec3f [in, out] sizeSFTime [out] enterTimeSFTime [out] exitTimeSFBool [out] isActiveSFInt32 [in, out] sequenceSFString [in, out] lBoundSFString [in, out] uBoundSFString [in, out] value

}

Live camera LiveCamera = MR/AR Capture Camera

Within the Scene node Image field is the out valueCamera internal parameter projmat fieldCamera external parameter Set to World but can be tracked

Live Camera { SFString [in, out] label "default“ SFString [out] parent SFImage [out] image SFMatrix4f [out] projmat "1 0 0 … " SFBool [out] on FALSE SFBool [out] tracking FALSE SFVec3f [out] position SFRotation [out] orientation}

Routing from LiveCam

From Live Camera node “image” field

ToBackground (LiveURL field)Shape (MovieTexture field)

Live video Background<Scene> <Background groundAngle='1.309 1.571'

groundColor='0.1 0.1 0 0.4 0.25 0.2 0.6 0.6 0.6' skyAngle='1.309 1.571' skyColor='0 0.2 0.7 0 0.5 1 1 1 1' backUrl='mountns.png' frontUrl='mountns.png' leftUrl='mountns.png' rightUrl='mountns.png'/>

</Scene>

<Scene> <LiveCamera DEF='USBCam1' source='dev#'/> <Background liveSource='USBCam1'/> </Scene>

<Scene> <Background videoUrl='bgvideo.mpg'/> </Scene>

MovieTexture Node

Add MovieTexture to X3DTextureNode hierarchy Used for TextureBackground Fix TextureBackground relative to camera

Allow connection to live camera (not just through streaming server)

MovieTexture Node<Shape> <Appearance> <MovieTexture loop='true'   url=' "wrlpool.mpg" "http://www.web3d.org/x3d/content/examples/Vrml2.0Sourcebook/wrlpool.mpg" '/> </Appearance>

<IndexedFaceSet ccw='false' coordIndex='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16'> <Coordinate point='2.00 0.6 0.00 1.85 0.6 0.67 1.41 0.6 1.41 0.67 0.6 1.85 0.00 0.6 2.00 -0.67 0.6 1.85 -1.41 0.6 1.41 -1.85 0.6 0.67 -2.00 0.6 0.00 -1.85 0.6 -0.67 -1.41 0.6 -1.41 -0.67 0.6 -1.85 0.00 0.6 -2.00 0.67 0.6 -1.85 1.41 0.6 -1.41 1.85 0.6 -0.67 2.00 0.6 0.00'/> </IndexedFaceSet> </Shape>

Live Camera Movie Texture<Scene> <Shape> <Appearance> <MovieTexture loop='true' url='wrlpool.mpg'/> </Appearance> <IndexedFaceSet ccw='false' coordIndex='0 1 2 ... 15 16'> <Coordinate point='2.00 0.6 0.00 ... 2.00 0.6 0.00'/> </IndexedFaceSet> </Shape> </Scene>

<Scene> <LiveCamera DEF='USBCam1' source='dev#'/> <Shape> <Appearance> <MovieTexture liveSource='USBCAM1' keyColor= '0 0 1' /> </Appearance> <IndexedFaceSet ccw='false' coordIndex='0 1 2 ... 15 16'> <Coordinate point='2.00 0.6 0.00 ... 2.00 0.6 0.00'/> </IndexedFaceSet> </Shape> </Scene>

Live Camera and Virtual Camera

Calibrating the virtual camera according to the parameters of live capture camera

Internal parameter = projection matrixExternal parameter = camera pose

Manual Direct specification Routing

From the Live camera From the Sensor

W0

T

Method 1Viewpoint : X3DViewpointNode{ SFMatrix4f [in] projmat SFVec3f [in,out] position SFRotation [in,out]orientation

SFNode [in,out] liveCamera Add distortion parameters here}

<Scene> <LiveCamera DEF='USBCam1' source='dev#'/>

<Viewpoint liveCamera='USBCam1'/>

<Shape> … </Shape></Scene>

Method 2

<Scene> <LiveCamera DEF='USBCam1' source='dev#'/>

<Viewpoint DEF='MRView'/>

<Shape> … </Shape>

<ROUTE fromNode='USBCam1' fromField='projmat' toNode='MRView' toField='projmat'/>

<ROUTE fromNode='Tracker' fromField='position' toNode='MRView' toField='projmat'/>

<ROUTE fromNode='Tracker' fromField='orientation' toNode='MRView' toField='projmat'/> </Scene>

Other Activities Draft document

Teleconferences with Web3D

Implementation: k-MART

Domestic workshop

April, POSTECH, Pohang, Korea

June, KIST, Seoul, Korea

Future

More

Extensions

Examples

Implementations

International consensus