ngmast 2012

From 2D Map to Mobile 3D Mirror World

Yu You, David Murphy Mixed Reality Solutions NRC, Tampere, Finland

A Live Virtual Advertising Use Case

Nokia Research Center

Mobile 3D Mirror World

• Mirror world (MR) is “informationally-enhanced virtual models or ‘reflections’ of the physical world” [2].

• Provides increased precision of an added dimension with more lively and immersive UX

• A nice complement to Augmented Reality (AR), which is commonly used in real-time and on-line situation

• Mobile MR can take advantage of mobile AR where mobile sensors and other positioning and tracking technologies are used


Motivation

• study the mobile advertising scenarios from a narrow subject (street-level mirror world) and try to answer an immediate question: how and where to publish the advertisement content?

• This paper focuses on: • Accurate but simple content automatic placement • Mobile content creation and visualization


• Data set • 360 degree panoramic street-level imagery • 3D building models with global geo. coordinates • Ground-terrain 3D mesh data

• Technology • OpenGL-based client renders the building models and ground

around the chosen panoramic image onto which the panoramic image is projectedA mobile client

• Video 1: Nokia CityScene

Experiment setup


The system in a nutshell

• Web-centric end-to-end setup with RESTful Web APIs for ad. content management and mobile 3D mirror world client

• WGS84 geo. boundary or proximity queries

• KML/KMZ is the main data representation format

• 3D objects are modeled separated in COLLADA files


Scalability consideration

• Unlike ad. content data, which is rather dynamic, any other static data are stored in different servers, i.e. provided by Content Delivery Network (CDN)

•  Typically CDNs does not offer geo. query functionality •  Each 3D data has a geo. coordinate and then its geo-hash is calculated (e.g.

we use quad-key method) at a fixed resolution (zooming) level (i.e. 17, appr. 200m2)

•  Therefore an index is created for all data in that geo. boundary and stored in the file system using a directory naming convention, which is correspondent to its geo-hash value. So does the real data.

•  Client needs to fetch the metadata (index) files at one given geo. coordinate and then all 3D data.


Mobile 2D map-based content planning

• Two types of ad. content representation 1.  Static object like billboards standing on one location 2.  Mobile 3D object along the vehicle driving road

•  2D coordinates are needed only; the altitude is calculated from the ground terrain mesh

• Problems •  Hard to position content accurately to the buildings, not

to mention the façade •  That reflects again the common problem in other

systems where geo-tagged data (e.g. point of interest) are misplaced in the middle of streets; or even worse - on the wrong side of the street

1

2


3D content manual placement

• High-value campaigns requires pixel-level accuracy • Developed a on-device authoring tool for non-

expert users with following few steps •  Choose the appropriate building façade • Drawing the size of the bounding box as the content

placement or drawing canvas •  Saving the drawing or inserted content with

metadata (e.g. the normal to the façade and bounding box data in the 3D)

• The content placement layer can be used to render any media data properly in 3D scene

• However, manual placement is time-consuming and not scalable


3D content heuristic/auto placement

• Auto alignment from only 2D coordinates •  Select the closest building model (the center of mass of the surrounding

models) • Determine the intersected façade by casting a light-ray to the model from a

predefined view point •  External media like banner images are converted to SVG format to scaling of

the scene (e.g. zooming in/out)


Video 2

Moving box for parking info


Findings and open issues

• Alternative ad. placement (e.g. façade vs. ground) •  Placing banners altered the appearance of the building facade, and thus deteriorated

the representational value of the mirror world

•  Lighting, visual improvement •  Illumination enhancement e.g. artificial lighting and content blending with proper

shedder, but lighting blobs on facades and tree shadows are difficult to be segmented

• Proper ad. content selection for moving artifacts •  Fixed geo. location as the context wont always work for mobile artifacts

•  Study more metrics concerning mobile/3D-specific user actions for analytical processing

•  Navigation, messaging etc. •  Zooming and panning

• Various pricing strategies vs. moving speed and frequencies • Mobile Web app. vs. native application

•  One solution for all OS


Summary

• Bringing ad. content to 3D Mirror World allows innovative types of advertising, coming along with new challenges

• We’ve proposed two basic types of ad. approaches but expect more realistic representations to appear