nxttour: an open source robotic system operated over the internet

NXTTour: An Open Source Robotic SystemOperated over the Internet

João Alves, Sancho Oliveira and Anders Christensen

Departamento de Ciências e Tecnologias da Informação

OSDOC’13, July 11, 2013, Lisbon, Portugal

Contents

1 Introduction

2 Related Work

3 System Prototype

4 Preliminary Experiments

5 Conclusion

J. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 2 / 37

Introduction

Contents

1 Introduction

2 Related Work

3 System Prototype


5 Conclusion


Introduction

Introduction

To demonstrate a low-cost robot that can be tele-operated overthe internet using open source software.

Differential drive robot based on the LEGO MindstormsNXT and an Android smartphoneCompatible with the most commonly used modern internetbrowsersCompatible with several versions of Android


Introduction

Introduction

The work presented in this paper relies exclusively on opensource software.

Our system is a generic prototype of a tele-operated robotTele-operated robots have many potential uses since theyallow users to have a presence, to explore, and sometimeseven to perform actions remotelyUses internet as the communication medium, so it can beoperated independently of the location of the user


Related Work

Contents

1 Introduction

2 Related Work

3 System Prototype


5 Conclusion


Related Work

Related WorkThe work presented in this paper is related to autonomous, mobile robots and totele-operated robots.

To simplify the design of robotic controllers, Brooks [2] proposed thesubsumption architectureThe presence of video and mapped information can help in a navigationtasks as Nielsen [6] concludes in his workThe design of a controller even for a simple task such as obstacle avoidancecan be non-trivial as Grechanovsky et al. [4] concludes in is studyThe Stanford Cart was presented by Moravec [5] and uses the images fromthe camera to plan its path and to avoid obstacles identified visuallyThrun et al. [7] built a robot for the Smithsonian Museum that was aninteractive guideA robot that can be controlled over the internet was presented in Bianchiniet al. [1]


System Prototype

Contents

1 Introduction

2 Related Work

3 System PrototypeOverviewTechnologiesCommunicationsThe RobotThe NXTSlave Module

Mode of operation – SELFMode of operation – OPERATORCommunication Protocol – NXT2Android

The NXTController ModuleWeb ServerDynamic Registration

The NXTBrowser ModuleAuthentication ProcessImplementation


5 ConclusionJ. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 8 / 37

System Prototype Overview

Overview

Several modulesDeveloped independentlyInterfaces and simple communication protocols betweenthe modulesExtensible code base


System Prototype Technologies

Technologies

We resorted to a number of technologies in the design anddevelopment of the system prototype:

A Lego NXT Mindstorms kit was used to build the robotThe leJOS1 firmware replacement was used for theNXTSlave moduleThe Android Software Development Kit (SDK) was used forthe NXTController moduleStandard technologies such as HTML, javascript andCascading Style Sheets (CSS) were used for NXTBrowsermodule

1http://lejos.sourceforge.net/


http://lejos.sourceforge.net/

System Prototype Communications

Communications

The three modules communicate with each other throughdifferent means of communication and using different protocols,some standard, others implemented specifically for the systemprototype.


System Prototype The Robot

The Robot

Actuators:rear end working inreverse

Sensors:Touch sensorColor sensorLleft ultrasonic sensorRight ultrasonic sensor

The robot, built with the LEGO Mindstorms NXT, is a differentialdrive robot that uses two motors, each connected to the rearwheels. The robot’s front wheel is a passive caster wheel. Therobot has an adaptable support for an Android smartphone.


System Prototype The NXTSlave Module

The NXTSlave Module

Runs on the LEGO Mindstorms NXT systemTranslates the commands issued by the operator into actualmovementCollects and sends sensory information back to the operatorSelf-preservation mode that can subsume controlImplements autonomous obstacle avoidance and recoverybehaviorsCommunicates with the control module, NXTController,through Bluetooth.written in Java language and uses the leJOS firmware



The NXTSlave Module

The logic to prevent collisions with obstacles and to recoverfrom dangerous situations such as when the robot is about todrive over an edge is implemented in this module.The robot has two modes of operation: ”SELF” and”OPERATOR”. The mode determines how the robot iscontrolled.



Mode of operation – SELF

This mode is the Self-preservation mode. The robot depending onthe readings of sensors, switches to this mode — SELF mode.

When an obstacle is sensed through one of the ultrasonicsensors, the robot stops immediately and checks if the obstacleis temporary or permanent.If the obstacle is permanent, the robot performs an avoidancemaneuver: the robot moves straight backward, backward left,or backward right depending on obstacle’s relative location.If the touch sensor is activated, it means that the third passivewheel no longer touches the ground and that the robot is aboutto drive over an edge. In this situation, the robot performs amovement backward in order to recover and avoid falling.



Mode of operation – OPERATOR

As long as there are no dangerous situations, the robot executesthe commands sent by the NXTController. The NXTSlave has aset of preconfigured actions (move left, move right, moveforward and stop) that can be requested by the NXTController.



Communication Protocol – NXT2Android

Uses BluetoothNXTController receives the status of the sensors fromNXTSlaveMinimize bandwidth usageMaximize the responsiveness of the systemMessages are composed of an header with an ID, followedby a Byte indicating the message type, and then thepayload.

The information exchanged can be regarding the battery voltageof NXT, sensory readings, the current control mode, and so on.Messages are only sent to the NXTController when valueschange.


System Prototype The NXTController Module

The NXTController Module

Runs on an AndroidsmartphoneResponsible for receiving theoperator’s commands andsending them to the NXTSlaveCaptures and transmits videoto the NXTBrowserForwards sensory informationand overall system statusinformationUser interface for theconfiguration of all parts of thesystemEstablishes the connections tothe other modules



Web Server

The NXTController runs a web server based on NANOHTTPd2

enabling the module to communicate using the HyperTextTransfer Protocol (HTTP) protocol with NXTBrowser module.A fixed uniform resource locator (URL) based on a dynamicregistration system was implemented to allow the system to bealways accessible.

2http://elonen.iki.fi/code/nanohttpd/


http://elonen.iki.fi/code/nanohttpd/


Dynamic Registration

Given the volatility of IP addresses that are often dynamicallyassigned, we developed a registration system to facilitate asimple connection process. When the NXTController web serverstarts, it registers its newly assigned IP address on a server witha fixed name (address set in the options as ”addresspublishing-site”). When the user navigates to the URL of theserver with a fixed name (eg. http://jpralves.net/51),the server will automatically redirect the user to the currentaddress of the NXTController. When the web server is disabled,redirection is also disabled.


http://jpralves.net/51

System Prototype The NXTBrowser Module

The NXTBrowser Module

The NXTBrowser module runs in a browser:

User interface exposed to the operatorReceives context information from the robotSends commands to itThe NXTController module runs an embedded web serverto which NXTBrowser connects.



The NXTBrowser ModuleThe NXTBrowser has an interface that provides the operator with sensoryinformation:

(a, b) obstacles proximity (color codes)(c, d) measured distance(e) touch sensor state(f, g) tachometers of motors(c) remote video

(h) information about the NXTSlave(j) information about Android(k) Four buttons of command(l) operation mode(m) video feed



Authentication Process

To prevent unauthorized access to the robot, the operator musttype in the correct password before a session can begin. Theembedded web server will only allow access to control of therobot in the address that is the result of the SHA-1 hash [3] ofthe password concatenated with a random key that is sent by it(SALT).



Implementation

User interface is partly based on jQuery3

Followed a design that would allow the use in all kinds ofdevices with at least a resolution of 1024 × 600 pixelsHas four buttons that allow the operator to interact with therobotReceives JSON messages from the server about stateinformation

3http://www.jquery.com


http://www.jquery.com

Preliminary Experiments

Contents

1 Introduction

2 Related Work

3 System Prototype

4 Preliminary ExperimentsNXTBrowserNXTControllerNXTSlave

NXTSlave - risk of falling down stairsNXTSlave - obstacle detection and avoidance

5 ConclusionJ. Alves, S. Oliveira, A. Christensen (DCTI - ISCTE-IUL) NXTTour Jul-2013 25 / 37



A number of preliminary experiments were done to assess thesystem prototype. A different set of experiments was designedfor each module.


Preliminary Experiments NXTBrowser

NXTBrowser

Since NXTBrowser runs in a browser and since no standard currentlyexists for streaming video that works across all browsers, the videofeed in the system prototype is, in fact, implemented as a sequence ofJPEG images captured by the camera of the Android device onboardthe robot.Tests were performed with four browsers representing the mostimportant rendering engines:

Engine Browser Version Site VideoTrident IE 9 X XWekbit Chrome 22 X XWebkit Safari 6 X XGecko Firefox 16 X X

All the tested browsers successfully presented the HTML-baseduser interface and the video feed.


Preliminary Experiments NXTController

NXTController

Extensive tests were made assess the compatibility of theNXTController with different Android devices and OS versions.Eight different Android handsets were tested and all of themwere able to run the NXTController module successfully. Thefollowing table lists the Android devices tested and theirspecification:

Brand / Model Version CPU RAM Screen WeightGoogle / Nexus S 4.1.2 1 GHz Cortex-A8 512MB 480 × 800 129gSamsung / GT-S6500D 2.3.6 800 MHz Cortex-A5 512MB 320 × 480 105gSamsung / GT-I9070 2.3.6 2× 1 GHz Cortex-A9 768MB 480 × 800 120gHTC / One S 4.0.4 2× 1.5 GHz Krait 1GB 540 × 960 119gTCT / Vodafone Smart II 2.3.7 832 MHz 512MB 320 × 480 120gHuawei / U8510 IDEOS X3 2.3 600 MHz Qualcomm 256MB 320 × 480 104gHuawei / U8815 Ascend G300 4.0.3 1 GHz Cortex-A5 512MB 480 × 800 140gHTC / Desire C 4.0.3 600 MHz Cortex-A5 512MB 320 × 480 100g


Preliminary Experiments NXTSlave

NXTSlave

Preliminary functional experiments were preformed to test if therobot was able to identify dangerous situations, and to test thatthe NXTSlave module activates the proper programmedbehavior in case a dangerous situation is identified.



NXTSlave - risk of falling down stairs

If the robot approaches stairs straight on (with an angle of ≈90◦) it correctly triggers the autonomous reverse behavior toavoid falling down the stairs. If the angle gets more acute, thatis, smaller than 25◦or greater than 155◦, the robot often fallsbecause one of the rear wheels falls over the edge before thestairs are detected.



NXTSlave - obstacle detection and avoidance

The robot approached a wall at different angles – between20◦and 160◦. The robot acted as expected when the wall wasapproached from the left, right or front. During a different set ofexperiments, we tested the performance of the obstacledetection with objects of various sizes and shapes. We foundthat robot did not always detect narrow obstacles, such as theleg of a chair. In particular, when approaching a narrow obstaclestraight on, the robot often did not detect the obstacle becauseof the orientation (at ±45◦ with respect to the front of the robot)and the limited resolution of the ultrasonic sensors.


Conclusion

Contents

1 Introduction

2 Related Work

3 System Prototype


5 ConclusionFuture Works


Conclusion

Conclusion

We have presented a low-cost robot based on a LEGOMindstoms NXT kit and an Android smartphone that can betele-operated from a browser over the internet.

The robot accepts commands from the operator, such asturn left, turn right, and move forward.The onboard software constantly monitors the sensors toassess the current situation.Autonomous behaviors are triggered whenever adangerous situation is identified.A remote operator thus cannot make the robot collide withobjects or damage the robot by driving it over an edge.


Conclusion

Conclusion

The GUI for the operator is web-based and it is compatiblewith commonly used modern browsers.The interface shows a real-time video feed from the robotand the current sensor readings.Our system is simple and straightforward to use withoutprior training.Autonomous behaviors that ensure the safety of the robotand its environment combined with an easy to use interfaceis essential for tele-operated robots to be adopted innon-technological industries such as the real estatebusiness.We have made extensive use of open source technologiesand we have made all source code for the system availableas open source software.


Conclusion

Conclusion

We have created a page on GitHub:

https://github.com/jpralves/tourrobot


https://github.com/jpralves/tourrobot

Conclusion Future Works

Future Works

The code-based for our system prototype is modular andextensible.The robot’s low-level module, NXTSlave, is currentlyimplemented specifically for LEGO Mindstorms NXT, but it couldbe ported to a different robotic platform with no changes in theother modules. The use of other platforms, such as Arduino orRaspberry Pi, would, for instance, allow for the construction ofmore capable robotic systems.In ongoing work, we are adapting the user interface so that thesystem can be operated from a smartphone. We are alsostudying autonomous behaviors that let the robot return to acharging station when its battery level drops below a certainthreshold.



Thanks



D. Bianchini, M. B. Meneses, V. L. d. Marchi, andJ. Dobgenski.Robô móvel controlado remotamente via Web.7o Congresso Nacional de Iniciação Científica -CONIC-SEMESP 2007, (7):102–117, 2007.

R. Brooks.A robust layered control system for a mobile robot.Robotics and Automation, IEEE Journal of, 2(1):14–23,1986.

D. Eastlake and T. Hansen.RFC6234 - US Secure Hash Algorithms (SHA andSHA-based HMAC and HKDF), 2011.

E. Grechanovsky and I. S. Pinsker.An algorithm for moving a computer-controlled manipulatorwhile avoiding obstacles.



Proceedings of the 8th International Joint Conference onArtificial Intelligence, pages 807–813, 1983.

H. P. Moravec.The Stanford cart and the CMU rover.Proceedings of the IEEE, 71(7):872–884, 1983.

C. W. Nielsen and M. A. Goodrich.Comparing the usefulness of video and map information innavigation tasks.pages 95–101, New York, New York, USA, 2006. ACM, ACMPress.

S. Thrun, M. Bennewitz, W. Burgard, A. B. Cremers,F. Dellaert, D. Fox, D. Hahnel, C. Rosenberg, N. Roy,J. Schulte, and D. Schulz.MINERVA: a second-generation museum tour-guide robot.volume 3, pages 1999–2005. IEEE, IEEE, 1999.


nxttour: an open source robotic system operated over the internet

Technology

related work related

introduction introduction

christensen dcti iscte

iul nxttour

related work contents

introduction contents

system prototype contents

sancho oliveira