services surround you

Visual Comput (2008) 24: 847–855DOI 10.1007/s00371-008-0266-4 O R I G I N A L A R T I C L E

Niels HenzeRene ReinersXavier RighettiEnrico RukzioSusanne Boll

Services surround you

Physical-virtual linkage with contextual bookmarks

Published online: 5 June 2008© Springer-Verlag 2008

N. Henze (�)OFFIS Institute for InformationTechnology, Oldenburg, [email protected]

R. ReinersFraunhofer Institute for AppliedInformation Technology, [email protected]

X. RighettiVRLab, EPFL, [email protected]

E. RukzioComputing Department, LancasterUniversity, [email protected]

S. BollMedia Informatics, University ofOldenburg, [email protected]

Abstract Our daily life is pervadedby digital information and devices,not least the common mobile phone.However, a seamless connectionbetween our physical world, such asa movie trailer on a screen in the mainstation and its digital counterpartssuch as an online ticket service,remains difficult. In this paper, wepresent contextual bookmarks thatenable users to capture information ofinterest with a mobile camera phone.Depending on the user’s context,the snapshot is mapped to a digitalservice such as ordering tickets fora movie theater close by or a link tothe upcoming movie’s Web page.

Keywords Mobile interaction ·Physical-virtual linkage · Physicalinteraction · Contextual bookmark ·Content analysis

1 Introduction

Mobile devices are a part of our everyday lives as mostpeople rely on mobile phones, PDAs, and multimediaplayers as personal and pervasive information devices.Ubiquitous network access has been made possible by theuse of these devices, while the ever-growing amount ofdigital content provides information about almost every-thing and all kinds of digital services are available frombuying a cinema ticket to renting a car. These digitalcontent and digital services often have physical represen-tations in the real world. Movie trailers are shown onpublic displays and advertisement posters promote cheepcar rentals. However, a seamless connection between our

physical world and related digital services is not possibletoday. Mobile interaction only takes place between users,their mobile devices, and digital services.

Over the last few years, there has been an increasinginterest in extending the interaction between a user andher device to an interaction that takes objects from the realworld into account. Outcome of this development are firstcommercial prototypes, as for example by Nokia [11] andKooaba1. However, linking between the real and the digit-al world today is only possible for specific objects andservices. The concept of contextual bookmarks presentedin this paper goes one step further: it enables users to cre-

1 http://www.kooaba.com/

848 N. Henze et al.

ate, use, and share bookmarks of real world objects. Theintention is to support a wide bandwidth of real world ob-jects and provide its users context sensitive with digitalservices. The example implementation recognizes objectswithout the necessity of visual or electronic markers bytaking contextual information into account. This introduc-tory section will present a scenario which shows the usecase of a mobile contextual bookmark application. Then,the underlying concept of contextual bookmarks will bedescribed.

1.1 Scenario

Mary is on the way to a conference in Berlin and sits inthe metro. The train is equipped with small advertisingscreens; suddenly, the trailer of the latest Tarantino moviepops up on the screen. Tarantino is one of Mary’s favoritemovie writers and she does not want to miss this movie.She points at the screen with her mobile phone and book-marks the trailer as a movie she would like to see. Duringthe conference’s first session a talk about mobile inter-action gets Mary’s attention. Using her phone Mary takesa contextual bookmark of the presenter and sends it toher colleague Jim as he is interested in this topic. Jim re-ceives the bookmark, opens the related webpage and sendsan email to the presenter in order to ask him for a meet-ing.

Later that day in the demo session Mary spots a fascin-ating poster on a novel interaction technique for mobilenavigation. Since the poster’s authors are busy with talk-ing to other participants, Mary uses the poster as a link toaccess a related video and watches it on her mobile.

It is already late afternoon, when the conference fin-ishes and she is on the way back to her hotel. Browsingthrough the bookmarks Mary collected today the systemautomatically checks the cinemas near Mary’s hotel andasks her if she wants to go to the Tarantino movie shebookmarked in the train. Mary remembers that she has anappointment with Jim tomorrow morning and refuses witha sigh. Arriving in the hotel Mary considers watching thebookmarked movie in her hotel room. The bookmark pro-vides Mary with a link to download the trailer. Watchingthe trailer she advises her mobile phone to get a license fordownloading the movie to the hotel’s set-top box. Usingher mobile Mary remotely controls the set-top box to skipthe promotion trailer and watches Tarantino’s great movieuntil she falls asleep.

1.2 Contextual Bookmarks

The system used by Mary allows her to memorize inter-esting information and situations she faces while on themove. It provides her with access to services which arerelated to physical objects. Such services can range fromsimple web pages, over specific services such as buyinga licence for a movie to rather object-related services such

as controlling a set-top box. The basis of the idea behindthe scenario is the concept of the contextual bookmark thatwill be described in the following.

A contextual bookmark is a context sensitive link cre-ated from a physical object, event or person. It comprisesa snapshot of a user’s context for a specific point in time.Context information about the user are her location, thetime, and may also cover information about preferencesand intentions. Besides context, a bookmark contains in-formation about the bookmarked object, event or person.This meta-information includes a name and descriptionbut also links to digital representations of the physicalobject, digital services, and background information. Asdescribed in the scenario persons, movies shown on publicdisplays, and posters can serve as hooks to create con-textual bookmark. However, the concept is not limited tothese entities. Further examples are exhibits in museums,cooking recipes, and places. A contextual bookmark cre-ated from Tarantino’s movie Death Proof would point tothe movie itself (not to the movie’s poster). It consistsof the user’s context at the time the bookmark was cre-ated and meta-information. The context information couldcomprise the user’s location, a photo of the poster, thetime, and the user’s preferences. Meta-information wouldcontain the type of the object, its name, and a thumbnail.Linked services could comprise the trailer of the movie,the movie in the iTunes Store or online cinema ticketsbooking.

An important aspect of contextual bookmarks is thatcontext is not only important to recognize the user’s inten-tion when she creates a bookmark. The context is also ne-cessary to provide the user with digital services she needs.That means that a bookmark of Tarantino’s movie pro-vides the user with a low resolution version of the movie’strailer if the user is an environment with poor network cov-erage and the next cinema is far away. In a city like Berlin,on the other hand, the system could provide a cinema-booking services and the high resolution DRM encryptedmovie itself.

The first prototype of a system implementing our con-cept of Contextual Bookmarks will be described in thefollowing sections. The prototype focuses on specific as-pects of the concept. However, our approach, instead ofproviding a solution for a single scenario, aims to supporta flexible solution that enables the integration of varioussensor data, media types, means of data access, additionalservice, and matching algorithms for different scenarios.

The remainder of this paper is organized as follows:Sect. 2 reviews the related work in the field while Sect. 3introduces the architecture of the contextual bookmarksystem. The single components realizing the contextualbookmarks are elaborated in Sect. 4. We show the flow ofinformation and interaction from capturing to using andsharing a bookmark in Sect. 5. Finally, Sect. 6 presentsthe lessons learned during our experiments and concludeswith an outlook to future work.

Services surround you 849

2 Related work

Mobile devices are more and more used for interactionswith objects in the real world. This can be distinguishedin direct interactions such as touching or pointing andindirect interactions such as using the mobile phone asa remote control [3, 13]. The research presented in thispaper focuses on pointing-based interactions as they allowthe interaction with objects from a distance. Such inter-actions are seen as very intuitive as they correspond to oureveryday interactions. We often point to objects with ourindex finger in order to show the other person somethingof interest or to support a statement.

Fitzmaurice was one of the first who described thevision of using a mobile device for pointing-based inter-actions with remote objects. He described, for instance,an application in which the user was able to point to cer-tain locations on a map in order to get additional infor-mation [5]. Several approaches to implement such inter-actions are used or have been proposed in the last fewyears. There are, in principle, marker-based and imagerecognition-based approaches. Furthermore, there existapproaches in which a laser pointer attached to a mobilephone was used to interact with an object [16] or in whichthe infrared interface was used to establish a link [2]. Thefirst approach has the advantage that the user gets a dir-ect visual feedback because the user sees the dot of thelaser pointer on the object. Unfortunately, there aren’t anymobile phones with a built-in laser pointer available. Thesecond approach has the disadvantage that some mobilephone vendors stopped the integration of IrDA interfacesas they were not accepted or used by users.

The NaviCam prototype, which was presented in 1995,was one of the first systems in which a mobile device withan attached camera was used for interactions with mark-ers attached to objects [12]. Using this system it was, forinstance, possible to get additional information about pic-tures, to interact with a calendar or to open an interactivedoor. Nowadays many mobile phones have a built-in cam-era and applications for the interpretation of visual mark-ers are preinstalled or can be downloaded on demand. InJapan, for instance, QR codes can be found in newspapers,advertisements and posters as corresponding applicationsare installed in almost every camera phone [7]. Anotherpopular example is Semapedia that uses Semacodes at-tached to places, sights or objects that provide links theWikipedia articles about them [14]. Furthermore, Nokiaships their new Nokia N series phones with a preinstalledmarker interpreter that can be used to extract the URL ofmarkers shown next to an advertisement. The most im-portant advantages of these markers are that they are easilyrecognizable and the extraction of the encoded informa-tion can be done on the mobile phone itself without anyinvolvement of a server. Unfortunately visual markers af-fect the visual appearance of the tagged object. In add-ition, the user has to know how to interact with the marker.

It is a problem for some users to focus the marker in sucha way that the whole marker is seen by the camera andthat the resolution is sufficient for the interpretation of themarker.

The advantage of marker-less image recognition ap-proaches is that no marker has to be attached to the picturethat disturbs its appearance. Fritz et al. presented a systemin which a PDA with attached camera was pointed ontosights in order to get more information about them [8]. ThePhoneGuide is a similar system but focuses primarily on theinteraction with exhibits in a museum in order to get moreinformation about them [6]. Most of these systems are im-plemented using a client-server approach in which an imageis sent to a server that then extracts certain features and com-pares them with a corresponding database. This requires anauthoring process which leads to a potentially huge set ofpictures stored in the database which are linked to relevantinformation. This leads then again to a huge demand of pro-cessing power for the comparison of the pictures taken withthe ones stored in the database. To reduce this demand someprojects store also the location and orientation of the user inorder to limit the search space [11, 15].

The advantage of the contextual bookmark systemwhen compared with existing approaches is the possibil-ity to take a bookmark of a diverse variety of objects likea bookmark of a movie trailer shown on a public dis-play, person met at a conference or a poster at a bus stop.Further supporting the proposed system is the manage-ment of these contextual bookmarks that is comparablewith the bookmark functionalities we know from Internetbrowsers. So it is possible to store them, group them or toexchange them which is, to our knowledge, not supportedby other systems. Also, the Contextual Boookmark systemuses bookmarks in combination with other services likebuying a licence for a movie advertised by a poster andinteraction with a TV in order to play the movie.

3 Architecture

Existing approaches focus on the recognition of specificreal world objects. Thus, they are limited to very rare inter-action scenarios. The concept of contextual bookmarks,however, covers a much wider spectrum. One key as-sumption is that users will not accept an interaction thatis limited to certain types of real world objects like ad-vertisement pictures in magazines or movie trailers. Ourvision is to build a system that addresses a wide spectrumof real world objects as well as not limiting to specificuse cases so a flexible concept and architecture is needed.The architecture presented here follows the concept ofcontextual bookmarks previously described. It supportsdifferent recognition techniques and can be extended tocollect more information about the current context wherethe bookmark has been created. As shown on Fig. 1, thebasic architecture can be divided into five components:

850 N. Henze et al.

real world object, personal interaction device, recognition,storage, and playback.

The personal interaction device is the main mobilecomponent that centralizes the interaction between theuser and the system. It is used to manage bookmarks andis also responsible for collecting context information dur-ing the creation of the contextual bookmark, such as forpictures taken by an integrated camera, the user’s location,and the time.

In order to recognize real world objects, a digital repre-sentation of these objects must be modeled. In addition tothese real world objects (advertisement posters, buildingsor even persons), immaterial content such as video con-tent shown on public displays or music played on a soundsystem must also be taken into account.

For the recognition of different kinds of real worldobjects a recognition component that encapsulates mul-tiple recognition techniques suited to the actual object, isneeded. For the implementation of this component, twopotential solutions are to be discussed: the recognitioncomponent could either be located on the mobile deviceitself (distributed approach) or on a backend recognitionserver (centralized approach). Since the recognition of realworld objects in general is a computing-intensive task, es-pecially regarding the recognition of multimedia content,a backend recognition server is necessary. On the otherhand, simple tasks like the recognition of RFID-taggedobjects require no special processing power and thus couldbe performed on the mobile device.

Once the contextual bookmark has been identifiedusing the collected context and the result from the imagerecognition component, it can be used to access any re-lated digital content. But, in order to provide universalaccess to the bookmarks from different devices, a central-ized persistent storage is necessary. This functionality is

Fig. 1. The central components and their interaction with thecontextual bookmark approach

provided by a storage component that mediates the shar-ing of contextual bookmarks between different users.

Besides sharing bookmarks with other users, makinguse of a Contextual Bookmark also means accessing thedigital items connected to it. Different physical objectscan be augmented with different types of digital items.As a result, a bookmark created from a scientific posterat a conference could not only provide publications foundin the references, but also video sequences related to thetopic. Consequently, the associated content displayed onthe user’s mobile device and/or on any nearby devices suchas a public display is handled by the playback component.

Before video sequences associated with the bookmarkare transferred on demand from an online repository tothe device, the content must be adapted to the device’sresolution, bandwidth, processing and storage capabilities.This means that only the references of contextual book-marks are exchanged, while the content itself is obtainedfrom a central repository and adapted to the current de-vice’s capabilities. This not only reduces network trafficneeded for the actual exchange, but it also ensures an opti-mal viewing experience for the current device.

A concrete implementation of the main components,i.e. the personal interaction device as well as the recogni-tion and storage components, is described in more detailin Sect. 4. This implementation is capable of recogniz-ing posters and snapshots taken from movie trailers be-ing shown on a public display. The resulted contextualbookmark then allows retrieving additional information ofdigital content connected to the real world object.

4 Components

Based on the architecture defined in Sect. 3 a concrete in-stantiation of this architecture is implemented. The centralpoint of this implementation is an application for mobilephones that enables users to create, manage, and sharecontextual bookmarks. The recognition component is im-plemented on a dedicated server that can recognize videosand posters. For the presentation of videos, a video playerwas implemented. A web server with a database backendis used for persistent storage and sharing of bookmarks.These components are described in the following.

4.1 Personal interaction device

The developed mobile application to create contextualbookmarks that represents the personal interaction devicecomponent is based on Java ME, the most common plat-form for mobile phones (e.g. the Nokia N95 equipped withan integrated GPS receiver and a high resolution camerashown in the left of Fig. 2). The developed applicationprovides the user with functionalities to create, use, andshare Contextual Bookmarks. The application starts withthe main menu shown in the middle of Fig. 2. Using the


main menu the user can access the different functionalitiesof the application. When the defined bookmark functionis selected the image of the phone’s integrated camera isshown on the display. The user can then point at a videoor poster he or she wants to bookmark. The applicationcreates a snapshot of the current context containing, inparticular, a photo, the time, and the location. The book-mark is sent to a recognition server that recognizes thebookmarked content. According to the recognized con-tent, the phone receives a thumbnail of the content as wellas a name and a link to an according trailer and video.

Created bookmarks are stored on the phone and syn-chronized with a storage component for backup purposesand to exchange bookmarks. Selecting the browse func-tion from the main menu the user can scroll through a listof created bookmarks shown in the right of Fig. 2. Select-ing a bookmark the user can view the received thumbnailand description. In addition, according videos and trailerscan be watched if available. To share bookmarks they canbe sent via Short Messaging Service. Bookmarks can alsobe sent to playback components implementing the neces-sary protocol using an IP connection. The output devicescan then be remotely controlled.

4.2 Real world objects

The implementation supports the recognition of analogueposters as well as videos as a representative for digital ob-jects. The core of the recognition process is the compari-son of a photo taken by the mobile application with thevideo and or poster. Since photos of real world objects aretaken by the user it can be assumed that the characteris-tics of these photos vary. We expect that typical examplesfor photographed videos and posters are shown in Fig. 3.Even if the quality of these examples is high, the poster isdistorted and a shadow darkens the center of the poster. Inaddition, the poster is curled and surrounded by the street

Fig. 2. a Nokia N95 running the contextual bookmark prototype,b Main menu of the mobile application, c Bookmark managementview containing two bookmarks

setting. Furthermore, it can not be ensured that the photo isin landscape or portrait format. The poster might be rotated,partially occluded, and varied by lighting conditions. Theresolution of mobile phone cameras is low and so too is theresolution of the photos. Characteristics of videos shown onpublic displays vary even more because displays differ interms contrast, resolution, and color fidelity.

To show content on public displays an applicationhas been developed that shows videos based on a simpleplaylist. To enable the recognition component to use thetemporal and location context prior to the recognition ofvideos, the public display application provides an inter-face that offers the display’s position and identifiers of theplayed videos. Each public display registers at the recogni-tion server. Thus, the recognition server is informed whatcontent was displayed on which public screen.

4.3 Recognition

Retrieving a video or poster based on an image taken bya mobile phone’s camera is the core of the matching pro-cess. For large collections of physical objects containingthousands of posters and videos, recognition based on thecontent alone is hardly possible. Thus, the number of po-tential media items is reduced using the user’s context, inparticular, his or her position and the time when the book-mark is created. After reducing the number of potentialobjects the photo of the object is compared with descrip-tors of the objects. For this comparison, the scale-invariantfeature transform algorithm developed by Lowe [10] isused. SIFT is invariant or robust against most of the vari-ations described in Sect. 4.2. Based on SIFT, algorithmscan be developed that are robust against partial occlusion.A Best-Bin-First [4] algorithm based on KD-trees can beused to match an image with a small number of objects

Fig. 3. a Photo of a poster used to create a bookmark of the Simp-sons movie, b Photo of a public display used to create a bookmarkof the movie

852 N. Henze et al.

by comparing the SIFT keypoints of the image with SIFTkeypoints of the objects. For the recognition of posterswe implemented standard techniques similar to the ap-proach from Liu et al. [9]. However, the complexity ofthis approach increases linearly with the number of objectsand videos consist of about 1500 frames per minute. Witha naive approach each frame is treated as an individual ob-ject. Thus, we enhance the algorithms as described in thefollowing.

Videos are analyzed offline taking the temporal charac-teristics of videos into account. First, the SIFT keypointsare extracted from each frame. Most frames have similarproceeding and successive frames and thus the accord-ing SIFT keypoints are similar as well. Similar keypointsare merged to a single keypoint reducing the number ofkeypoints drastically. Merged keypoints allow more ro-bust matching, in particular for photos with long exposuretime. After processing all frames a KD-tree for storing themerged keypoints is created that is later used to recog-nize photographed videos. If the recognition componentreceives a photo, SIFT keypoints are extracted and theprocess iterates through the list of videos. For each ofthe photo’s keypoints the keypoint’s two nearest neigh-bor keypoints are computed. According to the distancesbetween the photo’s keypoint and the two neighbors, itis decided if the keypoint is a match. Further heuristics,such as using all of the video’s keypoints only once, in-crease the processes performance and robustness. The pro-cess determines the object with the highest number ofmatches and transmits according links back to the mobilephone.

4.4 Storage

As soon as the requested object is determined by therecognition component, the according contextual book-mark is transmitted to the storage component. Thiscomponent is implemented using a MySQL database withan Apache Web-Server frontend. Since the interface isbased on standard web technologies, accessing the stor-age component from different devices and applicationscan be implemented easily. Each bookmark is stored witha unique identifier inside the database. This identifierserves as a handle that is used by the mobile applicationto exchange bookmarks over low bandwidth connectionssuch as the Short Messaging Service. Applications receiv-ing the handle can request the database to retrieve theaccording contextual bookmark.

4.5 Playback

To use different content types linked by a contextual book-mark, playback of these links is encapsulated in the play-back component. This component is realized as a softwareplayer derived from the Java Media Framework, whichensures platform independence. The same component is

used on the mobile phone as well as on the home the-ater PC. The playback component uses a toolkit for openhuman-computer interaction with ambient services thatabstracts from concrete input devices. The network trans-parent protocol based on XML-RPC enables to use thesame user interface to control playback on the phone aswell as playback on the TV at home.

5 Communication

This section describes the communication between thecomponents described in Sect. 4. With regard to the ini-tially presented scenario, three fundamental interactionswith contextual bookmarks are presented: creation ofa contextual bookmark, exchange of Contextual Book-marks, and accessing content associated with a Context-ual Bookmark. The following subsections illustrate thecommunication between the different components that isestablished for each of the three cases focussing on thecreation and exchange of bookmarks.

5.1 Creation of a contextual bookmark

Figure 4 shows the messages exchanged between the mo-bile phone that demands recognition the user’s contextand the recognition component. The recognition compo-nent only addresses videos and posters presented near theuser’s location.

The result of the analysis is sent back to the mo-bile device that stores the result in the local memoryas a Contextual Bookmark. Concurrently, the contextualbookmark is also stored on the central storage componentfor later access as demanded by the user, for example ifshe needs to transfer the bookmarks to another mobile de-vice or exchange between different mobile devices.

5.2 Exchange of bookmarks

As illustrated by Fig. 5, a contextual bookmark is ex-changed between two mobile devices by sending the con-textual bookmark’s ID via the GSM network. The receiv-ing device queries the central storage component for thetransmitted ID and downloads the contextual bookmarkvia the web server.

5.3 Showing digital content associatedwith a contextual bookmark

When accessing the digital content behind a contextualbookmark, the application checks whether the content isprotected by DRM and if a licence has to be acquiredfirst. If this is the case, an additional step for payment isneeded. The next choice consists in selecting the outputdevice. The content can either be shown on the mobiledevice itself or be sent to a larger display. The mobile


Fig. 4. Sequenze diagram for the creation of contextual bookmarks

Fig. 5. Sequenze diagram for the exchange of contextual book-marks

device then acts as a remote control for this display. Asdescribed in Sect. 3, the content itself is not transferred,but a reference to it. Since the large screen provides differ-ent capabilities with respect to display quality, the contentsent to this device can be adapted to its capabilities. Forthe mobile device, the same counts; the device receivescontent fitting to its capabilities concerning display qualityand resolution as well as memory space.

6 Lessons learned

During the development, demonstration and evaluation ofthe first version of the contextual bookmark prototype,we gathered new insights into the related research ques-tions and were able to analyze implementation questionsin a more detailed way. The most important ones will bediscussed in the following.

6.1 Extensibility

One way of extending the functionality of the contextualbookmark system is through the exploration of recogni-tion algorithms that are also applicable to content differentfrom image data. Context information also includes thelocation at which a contextual bookmark is created aswell as user preferences, which could help to optimize thesearch strategy and the resulting presentation which couldbe oriented at a user model. The fundamentals for theseextensions exist in the current implementation.

An alternative to a centralized service that contains allthe multimedia content is the distribution of the contentamong different devices. This solves the problem of pos-sible bottlenecks in situations where a huge amount ofcontent is requested from and processed by the repositoryserver. This approach assumes that the mobile device isequipped with sufficient storage and adequate data com-munication mechanisms. On the other hand, the exchangeof contextual bookmarks would become much more time-consuming since the users have to wait for the whole datatransfer.

6.2 Network integration and scalability

The current prototype works on the basis of a wirelessnetwork in which the snapshots and information are ex-changed and requests as well as responses are sent. Theonly access to the GSM/UMTS network is establishedwhen a certain bookmark ID is sent to another mobilephone. The current system is not yet scalable to a verylarge system of public displays and screens due to thelimitation of the context information to the time frame inwhich the snapshot is taken by the mobile phone’s cam-era. With that timeframe information, all available videocontents on all public displays must be analyzed. Futuredevelopments will enrich the gathered context informa-tion by localization data such that the public display fromwhich the snapshot was taken can precisely be determined.Future versions will also be capable of connecting to dif-ferent networks such as GSM network or UMTS by mak-ing use of ad-hoc networking mechanisms.

6.3 Usability

The contextual bookmarks showcase provides an excellenthands-on experience because people can feel a real inter-

854 N. Henze et al.

action taking place which is tangible as well as feasible inreal-time. The interaction still has room for improvementregarding the payment system and specific menu entries.The interaction could be enriched by more functionalitysuch as a browser-like interface to access the informa-tion associated with a bookmark in an easier way. At thecurrent stage of development, exactly one query resultis returned by the recognition component, the others arediscarded. The presentation of several results that exceeda certain threshold in the form of a ranking is a conceptto be explored in the future. It has to be clarified if usersreally want to choose among different search results orwhether they preferred one fixed outcome that is directly.

7 Conclusion

In this paper, we presented our approach to contextualbookmarks that allows the capture of objects in the realworld using a camera phone and mapping the photo toa digital concept that is related to the digital object. The

illustrating examples of capturing an interesting poster ora movie trailer from a public display showed how one caneasily bookmark real world entities and receive a mappingto digital content. The poster can be mapped to an on-line ticket store or the movie trailer can be watched on thehome TV.

The implementation of contextual bookmarks will beextended by further recognition techniques to validate ourarchitecture and address more types of real world objects.In particular, location-based searches [1] could be one pil-lar that provides context-sensitive digital services. We alsowork on the integration of a mobile guidance system andwearable interfaces in the contextual bookmark system.Furthermore we will evaluate our system with users inorder to compare it with existing approaches, to find issuesin our current implementation and to get further insight ofthe users’ needs.

Acknowledgement This paper is supported by the European Com-munity in the project InterMedia. We thank Rob Hess for hisimplementation of the SIFT algorithm.

References1. Ahlers, D., Boll, S.: Location-based web

search. In: Scharl, A., Tochtermann, K.(eds.) The Geospatial Web. HowGeo-Browsers, Social Software and theWeb 2.0 are Shaping the Network Society.Springer, London (2007)

2. Ailisto, H., Korhonen, I., Plomp, J.,Pohjanheimo, L., Strömmer, E.: Realisingphysical selection for mobile devices. In:Proceedings of Physical Interaction –Workshop on Real World User Interfacesin Conjunction with Mobile HCI,pp. 38–41. University of Munich, Munich(2003)

3. Ballagas, R., Borchers, J., Rohs, M.,Sheridan, J.: The smart phone: a ubiquitousinput device. IEEE Pervasive Comput. J. 5,70–77 (2006)

4. Beis, J., Lowe, D.: Shape indexing usingapproximate nearest-neighbour search inhigh-dimensional spaces. In: Conference onComputer Vision and Pattern Recognition(Puerto Rico), pp. 1000–1006. IEEEComputer Society (1997)

5. Fitzmaurice, G.: Situated informationspaces and spatially aware palmtopcomputers. Commun. ACM 36(7), 39–49(1993)

6. Föckler, P., Zeidler, T., Brombach, B.,Bruns, E., Bimber, O.: PhoneGuide:museum guidance supported by on-deviceobject recognition on mobile phones. In:

Proceedings of the International Conferenceon Mobile and Ubiquitous Computing,pp. 3–10. ACM, New York, NY (2005)

7. Fowler, G.: QR codes: In Japan, BillboardsTake Code-Crazy Ads to New Heights.Wall Street Journal (10.10.2005).http://online.wsj.com/article/SB112888593628963880.html?mod=telecommunications_primary_hs

8. Fritz, G., Seifert, C., Luley, P., Paletta, L.,Almer, A.: Mobile vision for ambientlearning in urban environments. In:Attewell, J., Savill-Smith, C. (eds.) MobileLearning Anytime Everywhere, pp. 77–80.Learning and Skills Development Agency,London (2004)

9. Liu, Q., McEvoy, P., Lai, C.: Mobilecamera supported document redirection. In:Proceedings of the ACM InternationalConference on Multimedia, pp. 791–792.ACM, New York, NY (2006)

10. Lowe, D.: Distinctive image features fromscale-invariant keypoints. Int. J. Comput.Vis. 60(2), 91–110 (2004)

11. Nokia Corporation: When “now” just isn’tsoon enough. Using camera phones to putmobile Net users in touch with theinformation they want, easily and instantly.Nokia (10/2007). http://www.nokia.com/

NOKIA_COM_1/Press/twwln/press_kit/Point_&_Find_Press_Backgrounder_October_2007.pdf. Accessed 15 April 2008

12. Rekimoto, J., Nagao, K.: The worldthrough the computer: computer augmentedinteraction with real world environments.In: Proceedings of the 8th ACMSymposium on User Interface Software andTechnology, pp. 29–36. ACM, New York,NY (1995)

13. Rukzio, R., Broll, G., Leichtenstern, K.,Schmidt, A.: Mobile interaction with thereal world: an evaluation and comparison ofphysical mobile interaction techniques. In:Proceedings of the European Conferenceon Ambient Intelligence, pp. 1–18.Springer, Berlin Heidelberg (2007)

14. Semapedia.org: Semapedia – Hyperlinkyour World. www.semapedia.org. Accessed15 April 2008

15. Simon, R., Fröhlich, P., Obernberger, G.,Wittowetz, E.: The point to discoverGeoWand. In: Adjunct Proceedings of the9th International Conference on UbiquitousComputing, pp. 3–10. Springer, BerlinHeidelberg (2007)

16. Välkkynen, P., Korhonen, I., Plomp, J.,Tuomisto, T., Cluitmans, L., Ailisto, H.,Seppä, H.: A user interaction paradigm forphysical browsing and near-object controlbased on tags. In: Proceedings of PhysicalInteraction – Workshop on Real WorldUser Interfaces in conjunction withMobileHCI, pp. 31–34. University ofMunich, Munich (2003)


NIELS HENZE is working as a research as-sociate at the Institute OFFIS in Oldenburg,Germany. He is involved in European researchprojects such as ENABLED and InterMedia. Hisresearch interests are multimodal interaction andinteraction with real world objects.

RENE REINERS is working as a research as-sistant at the Fraunhofer Institute for AppliedInformation Technology (FhG FIT) in Sankt Au-gustin, Germany. He is involved in the EU re-search projects InterMedia and HYDRA. His in-terests are the exploration of supportive tools forrapid prototyping and the design of user inter-faces in the field of human-computer interaction.

XAVIER RIGHETTI is currently working as a re-search assistant and pursues his Ph.D. at VRLab,the Virtual Reality Laboratory of the Ecole Poly-technique Federale de Lausanne (EPFL). Hismain research topic in human-computer inter-action includes the development of tangible userinterfaces in the context of wearable computing.

ENRICO RUKZIO is working as an academicfellow and lecturer in the Computing Depart-ment at Lancaster University. Enrico’s researchinterests are physical mobile interactions andapplications as well as context-aware mobileservices. He founded the PERMID (PervasiveMobile Interaction Devices) and MIRW (Mobile

Interaction with the Real Word) workshop seriesand is co-author of more than 40 internationalpublications.

SUSANNE BOLL is professor for media infor-matics and multimedia systems in the depart-ment of computing science at the University ofOldenburg, Germany. She is also a member ofthe scientific board of the OFFIS Institute forInformation Technology, where she heads themultimedia and Internet information services di-vision. Focuses of her research are semantic in-formation retrieval, mobile multimedia systems,and intelligent multimodal user interfaces.

services surround you

Documents