VersaPen: Exploring MultimodalInteractions with a ProgrammableModular Pen

Marc TeyssierLTCI, CNRS, Télécom ParisTechUniversité Paris-SaclayParis, Francemarc.teyssier@telecom-paristech.fr

Gilles BaillySorbonne Universités, UPMCUniv Paris 06, CNRS, ISIRParis, Francegilles.bailly@upmc.fr

Éric LecolinetLTCI, CNRS, Télécom ParisTechUniversité Paris-SaclayParis, Franceeric.lecolinet@telecom-paristech.fr

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).CHI’17 Extended Abstracts, May 06-11, 2017, Denver, CO, USAACM 978-1-4503-4656-6/17/05.http://dx.doi.org/10.1145/3027063.3052964

AbstractWe introduce and demonstrate VersaPen, a modular penfor expanding input capabilities. Users can create their owndigital pens by stacking different input/output modules thatdefine both the look and feel of the customized device. Ver-saPen investigate the benefits of adaptable devices andenriches interaction by providing multimodal capabilities,allows in-place interaction, it reduces hand movements andavoids cluttering the interface with menus and palettes. Thedevice integrates seamlessly thanks to a visual program-ming interface, allowing end users to connect input andoutput sources in other existing software. We present vari-ous applications to demonstrate the power of VersaPen andhow it enables new interaction techniques.

Author KeywordsPen input; Multimodal interaction; Modular input

ACM Classification KeywordsH.5.2 [Information interfaces and presentation]: User Inter-faces

IntroductionUser customization is broadly used in off-the-shelf software.Users can choose the position and the content of palettesor change the correspondence between hotkeys and com-mands. In contrast, physical devices such as keyboards,

mice or stylus remain mostly static. They have few inputand output capabilities that users can seldom customize foradapting these devices to their tasks and applications.

We demonstrate VersaPen (Figure 3), a highly customiz-able device in terms of input and output capabilities. Ver-saPen relies on a set of versatile modules including sensorsor displays (Figure 1) that can be easily combined by endusers by stacking them together while preserving the formfactor of a standard pen (Figure 2).

VersaPen offers multiple advantages: Users creates instru-ments adapted to their tasks and applications by physicallycustomizing a single device.By attaching controls to thephysical device, VersaPen reduces hand movements andcursor round trips between menus, palettes and the objectsof interest. It also favors multimodal interaction: severalsensors close to each other facilitate their use in parallel.

Figure 1: Set of modulescomposing the pen, includingsensors and displays.

Figure 2: A user staking inputmodules to create a customizeddigital pen.

We illustrate the adaptability of VersaPen with several inter-action techniques relying on multimodal interaction, in-placeinteraction and devices augmentation. Our primary contri-butions are the demonstration of an adaptable and modularinteraction device and the presentation of a set of interac-tion techniques, which aim to bring further discussion aboutthe opportunities raised by VersaPen.

Related WorkPen augmentation has been proposed to alleviate the prob-lem of mode switching, such as switching between drawing,annotating and issuing commands, or more generally forselecting modes [1]. To achieve this, various input modal-ities have been considered [2]. Studies also considereddifferent output modalities, such as visual or haptic feed-back [3]. VersaPen not only embeds many of these I/Omodalities, but also allows users to rapidly prototype andcustomize devices by combining them.

Figure 3: VersaPen has the same form factor as traditional pens.

Tangible User Interfaces (TUIs) aim at the customization ofphysical interfaces. Toolkits have been proposed to quicklyprototype physical interfaces [4]. However, the target usersare developers or product designers rather than the endusers who will manipulate these devices. In contrast, MagGetz[5] proposes a set of user-customizable physical controlsbut their form factor (size and shape) is not appropriate forpen-based interaction.

Adaptable mechanisms such as themes, skins, or stylesfor customizing user experience (interface, inputs) havebeen provided in various applications as software features.Other systems have also been proposed for creating moreadaptable interfaces in research [6]. These works focus onsoftware and provide a way of adapting input interactionrather than device customization.

Some modular devices are now commercially availablesuch as modular keyboards (e.g Azio Levetron Keyboard)or the Google Ara smartphone. While users can modify theshape of these devices, the interaction techniques remainessentially the same. The commutable device proposed in

[7] shares some characteristics with VersaPen. However itaims to be a tool for one specific task and does not sharethe same versatility and handleable form factor.

ImplementationOur device consists in a series of modules that can bestacked to build an augmented pen. Once connected to acomputer, the user can map the sensors and actuators ofthe pen to existing applications.

Figure 4: VersaPen flow-basedprogramming interface.

A module consists of two nested 3D-printed cylinders. Theinner cylinder contains the electronic parts and fits intothe outer shell to form the final module (18mm). We builta set of 15 modules, including various input modules suchas a buttons, a mouse wheel, an accelerometer, a touchsensor and a microphone. Output modules provide visualfeedback, through RGB LED modules, or haptic feedbackvia a vibro-motor. The tip of the pen is conductive to sup-port compatibility with a capacitive screen and is sensitiveto pressure. The form factor results from a compromiseto achieve sufficient rigidity while enabling a comfortablegrasp. Each module encloses electronic parts.

A microcontroller small enough (< 9.5mm) to fit inside thepen-sized container, manages the embedded sensor/actuatorand handles the communication between modules. Datais acquired locally and differentiated by the IDs of the at-tached modules and sensors. These IDs allow sending spe-cific instructions to a given output module. Messages fromthe entire stack are passed via a shared bus to an Arduinocontroller connected to the computer. A server translatesincoming messages and streams them via TCP to the soft-ware managing events.

Figure 5: Different input methodsare combined to improve workflow.

Figure 6: VersaPen used as agame controller.

We implemented a simple flow-based programming inter-face in order to let end users to connect the various inputand output sources to computer applications through visual

programming (Figure 4). Users can experience and explorethe demonstration as the customization process is compati-ble with existing applications.

Applications and demonstrationTo showcase the possibilities of VersaPen, we highlight itsmain capabilities and illustrate how it allows creating newtools adapted for specific needs and applications.

In-place interactionSelecting items on menus, toolbar or palettes require cur-sor round-trips between the objects of interests (located onthe center of the screen) and graphical widgets. In contrast,VersaPen favors "in-place" tool switching. Users can rapidlychange tools without moving the pen from the object of in-terest. We used this ability to enhance a 3D world sculptingapplication (Unity3D, Figure 5). The pressure and the slidermodules are respectively used to control the topology ofthe 3D object and the orientation of the camera while thewheel module serves to access the texturing functions. Thetangible controls can easily be reached by the user.

Multimodal & parallel interactionUsers can control multiple continuous inputs at the sametime using VersaPen. For instance, they can simultaneouslycontrol the size (pressure module) and the color (touchmodule) of a digital brush while drawing (moving the pen)with a single hand. One module allows users to interactwith both hands. It consists of two components, one at-tached to the pen and another one held in the other hand,which is connected to the pen by a flexible wire. This al-lows manipulating auxiliary controls with the non-dominanthand, as for instance the brush color while the dominanthand controls the brush position and pressure. Adding avibro-motor and a joystick module transforms VersaPen intoa game controller (Figure 6).

Space-efficient interactionEditing applications typically offer many commands thatare organized in palettes and toolbars that tend to hide alarge part of the workspace. VersaPen solves this problemby letting users assign a subset of useful functionalities toappropriate VersaPen modules. We used this capability in adrawing application featuring a novice and expert mode tosave screen real-estate. In novice mode, turning the wheelof the pen temporary displays a palette and the user cannavigate between commands by turning the wheel. In ex-pert mode, the same action is performed without waiting forthe palette to be displayed on the screen. Moreover, Thehaptic and LED modules can serve to provide feedbackabout the current tool.

Shortcuts and automatic configurationVersaPen buttons can serve to create shortcuts for acti-vating frequent commands. For instance, one button canchange the order of the overlapping windows on the screen.When the user toggles this button, the window under thefrontmost window comes to the front (and so on by clickingagain). This allows to perform drag and drop operations be-tween windows without releasing the pen. More generally,users can set up favorite shortcuts and link them to buttonsor other elements of the pen, with different settings for eachapplication, to improve interaction efficiency.

Figure 7: Additional information isdisplayed on the LED module.

Figure 8: Modules can expandinput capabilities of existingdevices (here a keyboard).

Device augmentationVersaPen can also serve as an auxiliary device of an exist-ing device or application to provide additional feedback. Forinstance, a 2D data visualization application can take ad-vantage of the LED module to convey additional information(Figure 7). VersaPen can also extend the capabilities of an-other device as shon in Figure 8 where VersaPen is used toextend a keyboard. The slider module of the pen allows toscroll content without moving the hands from the keyboard.

DiscussionVersaPen is a step towards modular and multimodal de-vices. It enables explorating various interactions by allowingto easily add/remove modules and assemble them in differ-ent ways. We expect this capability to inspire novel usagesof pen interaction and leverage multi-modal interaction.

REFERENCES1. Yang Li, Ken Hinckley, Zhiwei Guan, and James A

Landay. Experimental analysis of mode switchingtechniques in pen-based user interfaces. In CHI, 2005.

2. Feng Tian, Xiang Ao, Hongan Wang, Vidya Setlur, andGuozhong Dai. The tilt cursor: Enhancingstimulus-response compatibility by providing 3dorientation cue of pen. CHI, 2007.

3. Johnny C Lee, Paul H Dietz, Darren Leigh, William SYerazunis, and Scott E Hudson. Haptic pen: a tactilefeedback stylus for touch screens. In UIST. ACM, 2004.

4. Saul Greenberg and Chester Fitchett. Phidgets: easydevelopment of physical interfaces through physicalwidgets. In UIST ’01. ACM, 2001.

5. Sungjae Hwang, Myungwook Ahn, and Kwang-yunWohn. Maggetz: Customizable passive tangiblecontrollers on and around conventional mobile devices.UIST ’13, New York, NY, USA, 2013. ACM.

6. Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips,and Nicolas Roussel. User interface façades: towardsfully adaptable user interfaces. ACM, 2006.

7. Alec Jacobson, Daniele Panozzo, Oliver Glauser, CédricPradalier, Otmar Hilliges, and Olga Sorkine-Hornung.Tangible and modular input device for characterarticulation. ACM TOG ’14, 2014.