percept: indoor navigation system for the blind and...

University of Massachusetts AmherstDepartment of Electrical and Computer Engineering

ECE 671 - Computer Networks - Fall 2013Professor Aura Ganz

Percept: Indoor Navigation System for theBlind and Visually Impaired

Project 3

Xue YuThiago A. Teixeira

Group #2

December 9th, 2013




Project 3


Group #2

December 9th, 2013




Project 3


Group #2

December 9th, 2013

Abstract

Vision loss is taking an enormous toll around the world. There are 161 million people worldwidewith some kind of vision impairment. In this project, we develop an application to provide aidfor the person who has visual disability. This application enhance perception of the indoorenvironment using passive RFID tags deployed in the environment, a custom- designedhandheld unit which serves as the PERCEPT client device, and a PERCEPT server that generatesand stores the building and RFID tags information. Since the visually impaired person chooseshis destination by touching the kiosks, the destination will be transmit from the customizedglove to the Android smart phone via Bluetooth. Then the Android client sends a request to thePERCEPT server to get the current location and destination. After, an accurate indoorlocalization algorithm calculates the best path to the destination. The application will provideaudio instructions based on a reference starting point. During the path, if the user touchesanother RFID tag, the application will update the location and update the path. In this work, anew approach is proposed. The PERCEPT system makes use of the available Wi-Fi receivedsignal strength from the nearby access points to provide the visual impaired user a roughposition and the closest RFID tags. The new functionality will help the visually impaired user tolocate points of interest within buildings where the navigation is not easy. The interest inindoor navigation has been growing in the past years, with different services andmethodologies being applied. However, indoor navigation is not an easy task, as theenvironment changes constantly and the signal strength is not constant. Although we describethe proposed work, the new functionality was not completed implemented. For this reason, wefocus our work on describing the basic functionalities of the Percept application, ourmethodology and its implementation details. We develop the system in the Disabilities Servicesperspective, as we do not have access to visually impaired person. In this context, we havedeployed a database to store the users’ information and send help when needed.

Keywords: Visual Impaired, Android, Wi-Fi, RFID, Indoor Localization, Fingerprinting, Database

1. Specific AimsThe goal of this application is to help the visually impaired people to navigate through indoorenvironments more independent. To achieve this goal, we added a passive radio frequencyidentification (RFID) reader on a customized sports glove. The glove hosts an Arduinomicrocontroller with a Bluetooth module that will serve as the communication entity betweenthe RFID reader and the smart phone.

The visually impaired user will then scan the RFID tags that are posted at key locations, such asdesignated drop-off points (explained in section 2), elevators, stairs, doors, restrooms andemergency exits to name a few. Once the smart phone receives the information from themicrocontroller, it forwards to the server that will calculate the path to the destination, sendingit back to the smart phone. Since the user's location and the destination are known, based onthe RFID tag he scanned and the destination he chose at the Kiosk, the system calculates andcreates the appropriate path, giving to the user navigation instructions through audio. It isimportant to notice that as the visually impaired person does not have a reference point (whichfor a normal person is trivial task), the path will always start with a reference point, such as adoor or the Kiosk.

As we do not have access to visual impaired person, we will develop the application in theDisabilities Services perspective. The basic functionalities of the application will be thefollowing:

Track the path of Visually Impaired users travel;

Indicate to Disabilities Service when a Visually Impaired user needs help;

Provide the mechanism for Disabilities Service to write instructions to help VisuallyImpaired users.

In addition to the basic functionalities, we propose a new system functionality to help the usernavigate through the building. The new functionality is to use the available Wi-Fi received signalstrength and the Wi-Fi access points (AP) unique MAC address to create a virtual radiationdensity map, using a method called fingerprint. The fingerprinting is based on two phases, thetraining phase (also called offline) and the tracking phase (also called online). The fingerprintingoffline phase is determining the signal characteristics of a given point and store in a database.The online phase is picking up the signal, compare to the database and determine the location.

Furthermore, we developed a database to store user’s identification, current location anddestination. We believe that having the database and showing it to the operator will help theoperator to have a better visualization, hence, guiding better the visually impaired user to his orher destination.

2. Background and SignificanceVisual impairment is a disease that affects many people in the world, regardless to their age orethnics. In 2011, the National Health Interview Survey (NHIS) Preliminary Report estimated that

more than 10% of all adult Americans have some kind of visual impairment, even when theywear glasses or contact lens [1]. The majority of visually-impaired people struggle to find hisway to a known location without asking for help. They need to memorize a large amount ofinformation and rely on their perception of the environment. The task becomes almostimpossible when the environment is unknown.

To address this situation, some methods were developed along the years, where we canmention a few such as the Braille system, cane, dog guide, enlarged print or audio options. TheUniversity of Massachusetts Amherst has a pick-up and drop-off service for the visually-impaired. Each building within the campus has a designated drop-off point defined by theCampus Disability Services.

To help visually impaired people to be more independent and reach their destination within abuilding, PERCEPT indoor navigation system is proposed [2]. PERCEPT makes use of passive RFIDtags to label important landmarks inside buildings, such as elevator, exits, stairs, meetingrooms, and restrooms. A customized smart phone and a customized glove are used forproviding audio feedback to the user upon reading the RFID tags. Within the building, PERCEPTintroduces the concept of Kiosks. Kiosks are junctions where the users can convey his intentionto the system. Kiosks are located in key points such as the designated drop-off points, buildingentrances, and elevator.

Due to its functionality, low deployment cost of RFID tags and low maintenance, the PERCEPTsystem is competitive in comparison to other systems, raising the awareness of theMassachusetts Bay Transit Authority (MBTA) and some other institutions. MBTA is interesting inpiloting the PERCEPT system in the subway stations of Boston [3]. We believe that in the nearfuture, the smart phones will be used as the RFID readers, leaving the glove as an optionaldevice. Due to the mobile payment growth, by 2015, 50% of the smart phones will have nearfield communication (NFC) chip [4], which also can read RFID tags.

Moreover, not all buildings are friendly to the visually impaired people, as some buildings tendsto look like a maze (e.g. some shopping malls, airports, hospitals) and others are just verycrowded (e.g. train, bus and subway stations). Although Percept improves the indoor navigationtremendously, some environments can be tough even for someone using the system.

At the same time, there have been a lot of efforts put on indoor navigation; as the GPS does notwork on indoors [5]. However, due to the heterogeneity of the wireless network cards and theelectromagnetic signals in the indoor environments, indoor positioning is a very tricky task.Recently, some companies have announced accurate indoor algorithms using fingerprinting(described in section 1).

In our proposed work, we use fingerprinting as secondary information, because it still notaccurate enough for a visually impaired person who requires almost pin pointed coordinates.However, even with the fingerprinting current accuracy in the order of one to two meters, webelieve that it still can help the user navigate through rough environments. The enhancedversion of Percept application can then provide the visually impaired user with his or her roughlocation, as well as the user’s current location to the Disabilities Services operator.

3. Proposed WorkThe PERCEPT indoor navigation system described in [2] works in the following way: First theuser with visual impairment contact the UMass campus disability services for pick up and dropoff on the destination building. At the designated drop off point, the user finds a kiosk. On thisfirst kiosk the user is able to read (using one of the techniques described previously) thedestination RFID tag. Once the destination is identified, the user uses the glove to scan the RFIDtag. The PERCEPT system then creates a path and guides the user to the destination. Along thepath, the user may encounter other RFID tags, where he can identify and update his location.Each tag the user scans yields an updated path to the destination. Once the user reaches thedestination, he has the option to press the Help button on the PERCEPT glove and the systemgenerates the path back to the designated exit.

Using the PERCEPT Android API, we will create an application to reproduce the PERCEPTfunctionalities described above and test it in the Knowles Engineering Building.

Although the PERCEPT system makes the life of the visual impaired person easier by guiding theperson to the destination, the user have to rely on his human senses and localization to gothrough the building and finding his way to the destination. In large buildings the user can easilyget lost, since the floor plan is unknown, a RFID tag may not always be available or the usercannot always find or scan the tags. RFID tags have the advantage of give the exact location tothe user, since it is fixed on the point. To scan the RFID tag, the reader has to be within 5 cm ofthe tag and on a specific orientation, otherwise it will not read.

With the provision of almost ubiquitous deployment of Wi-Fi networks within the commercialand residential buildings premises, we can make use of the Wi-Fi received signal strength (RSS)to add a functionality to the PERCEPT system. Using the method called fingerprinting, we canscan the floor plan for the available Wi-Fi APs, creating a rough location of the user. ThePERCEPT system will then inform the user via an audio message of the nearby RFID tags andbased on the user's rough location, the system will guide him to the nearest RFID tag. Webelieve that having this extra piece of information will help improve the user experience, basedon the suggestions on [6], where the users reported the willing to have the directions given innumber of steps or feet.

The University of Massachusetts Amherst has Wi-Fi APs deployed among the campus. TheseAPs are on fixed location and are always turned on, in contrast to another APs run byindividuals or laboratories, where the user can turn off it at any time. We will make use of thealready deployed infrastructure to create our fingerprinting map. To distinguish between APswith same service set identifier (SSID), we will use the AP unique MAC address. Figure 1 showsan example of how a fingerprint map looks like.

Using a software capable of reading the APs information and store it in a database, we willsurvey the Knowles engineering building and create our own fingerprint map. For instance,when the user arrives at the designated drop off point, his smart phone will read the Wi-Fi RSS.Based on the device readings, it will assist the user on the nearest point of interest (in this casethe location of the Kiosk. If the user wants to go to the second floor, the PERCEPT system will

guide him to the elevator. After the user leaves the elevator, the smart phone will read the Wi-Fi signals, identifying and informing the user is now on the second floor. The system will alsoassist the user on the nearest Kiosk (if available) or RFID tag.

In this work, we assume that the disability services will take care of the first hop (pick up theperson with disability and drop off at the designated point). However, this is not always thecase. To address this issue, we believe that in the future we can develop a hybrid outdoornavigation for people to locate themselves from one building to another, like the GPSnavigators do with cars. We can then extend the work from the university environment to thetown environment, as it is a common practice today that the towns run their own free Wi-Finetworks.

Fig. 1 - Example of a fingerprinting map. (Courtesy: Cisco Systems)

4. Research and Design MethodsAs previously stated, our goal is to design a application for the visually impaired user, capable ofproviding methods to track the path that the visually impaired users travel within a building;indicate help to Disabilities Services when a visually impaired user needs help; and provide themechanism for Disabilities Services to write instructions to help visually impaired users to get totheir destinations. In addition to the basic goals, we proposed the use of fingerprinting.

Our Android application was initially designed to achieve the goals with a simple user interface(UI) containing four buttons to call the functions and communicate with the Percept server. Theextra functionalities were to be added later.

In the following, we describe the concepts of each functionality of the application. On section 5,we describe the details behind implementation, provide screens, our fingerprint database, andour database, as another extension.

The first button to appear on the top left is the Find button. The function of this button is tolocate the user using Google Maps API, and retrieve the building nodes from the server. Whenthe Percept application is open, the main screen is the Google Maps world map. The user needsto hit Find in order to locate the building nodes. Please remember that we are developing anapplication for the Disability Services operator, not the visually impaired user.

The second button on the menu is the Send Help. This button simulates the visually impaireduser scanning a RFID tag. Whenever the visually impaired user reads a RFID tag, the node is sentto the server. If the user is lost, he or she press Send Help, and the application asks for the finaldestination. The user selects his or her final destination and the information is sent to theoperator at the Disabilities Services. Please note that the user is visually impaired, thus we aresimulating a situation where the user asks for help to the Disabilities Services.

The third button on the application menu is the Get Help. On this function, the DisabilitiesServices operator receives the user information (current location and destination). Theoperator can then write instructions to the visually impaired user.

The fourth button is the Track button. As the name states, the track button allows the operatorto receive and view the visually impaired user traversed nodes in order of time.

The figure 2 below represents the flow of our application. Although the Google Maps API is aseparate API, for simplicity we are only stating it on our main activity.

Fig. 2 –Flow of the Percept Android Application

4.1. Extensions

Our test bed was the first floor of the Knowles Engineering Building (KEB), located at theUniversity of Massachusetts Amherst. The KEB is a small building with one hallway, three exits,research laboratories, and administrative offices. There is also one elevator located in one sideof the building. The training phase for fingerprinting is time consuming, as we have to scan eachpoint for a certain amount of time, save the readings to a database, filter the access points MACaddress, and compute the average signal strength. For these reasons, we defined four points ofinterest, one on each exit, and the elevator. The details of the implementation are given in thesection 5.

Main Activity

Start Thread ();onClickListeners();GetBuildingNodes();SendHelp();GetHelp();UpdateHelp();GetTraversedNodes();GoogleMapsAPI

Helps List

Database

Server API

BuildingNode.javaHelp.javaTraverseNode.java

During development, we decided to store the building nodes and the help list in a localdatabase. By doing that user do not need to reload the data from server when they reopen theapplication. Besides, the user also can keep the history and log locally.

Finally, we design a blind user application, which can prove our assistance app works well andgive audio instructions to visually impaired person.

5. Implementation DetailsIn this section, we explain in details how our project was implemented. We start with the mainfunctionalities, moving further to the extension part. We describe functions and codingschemes to better understand the code flow. The main functions will be highlighted in bold.

5.1. User Interface

Fig. 3 – User Interface

We use linear layout to display the buttons. We add a list view on each of the button. Bytouching buttons, listview shows, touching again, listview hides. In each item we set aonItemClickListener and a adapter in order to determine which item we are clicking.

Fig. 4 – User Interface architecture

5.2. Find functionality

Once the user click Find, an onClick Listener triggers the CameraUpdate method to move themap center to the Knowles building. It also calls the function GetBuildingNodes. The functionGetBuildingNodes retrieves a list of building nodes from the Rest Client to an array. We countthe number of building nodes and inform to the user via a Toast. We receive the building nodeinformation indexed by the sequential number (starting on 1 until the end of the array) and thebuilding node ID. As we want to use the building node ID later on to create the trackingfunction, we map the building node and the actual building index (which is different from thearray index) using a HashMap. The GetBuildingNodes function also retrieves the latitude andlongitude of each building node. The application screen is shown on figure 2. The code flow isshown on Fig. 3.

Fig. 5 – Screenshot for the Find function and the retrieved building nodes

onClick_Find() GetBuildingNodes()

Get Building Nodes fromRest Client

Create ListCreate HashMap

ReturnEnd of OnClick_Find()

Fig. 6 – Code flow for the Find function.

5.3. Sent help functionality

As the user navigates through the building, he or she can press Send Help anytime, to callDisabilities Services. The application asks the user to select a destination node, which is furthersent to the server. Every scanned node is sent to the server. We differentiate the current nodeand the destination node by using a flag, called action. If action equals to one, the traversed

nodes are send to the server; if action equal to two, the user selects the destination node. Thisflow is shown on Fig. 7.

onClick_SendHelp() SendHelp()

Set Action = 2Select

DestinationNode

Yes

Send Help tothe Server

Set Action = 1ReturnEnd of onClick_SendHelp()

No

Fig. 7 – Code flow for the Send Help function.

5.4. Give instruction and sent to server

Another function of our application is for the Disabilities Services operator to send messages tothe visually impaired user. This function is called Send Instructions. The send instructionsoption is given to the operator when he or she clicks on the user last visit node. A dialog boxappears, enabling the operator to send text messages as instructions to the server. The diagramof Fig. 9 shows the code flow for this function.

5.5. Track functionality

The button Track retrieves a list of traversed nodes from the server, storing it on an array list.The building nodes coordinates are comes in a string type. We have to slip the two coordinates(latitude and longitude), parse to double to get a list of our traversed nodes that we can nowdraw the line. Please note that we use the values of the HashMap created earlier to identify thecorrect traversed nodes. The application screen is shown on figure 8. This flow is shown on Fig.10

Fig. 8 – Screenshot showing the tracking nodes

OnClick_Track()

Get Traversed Nodes

Split StringConvert to LatLng()

Create List

Build Polyline

Return

Fig. 10 – Code flow for Tracking function.

onMarkerClick()

Click Buttons

Show Help RequestInformation

SendInstructions?

Update Database

Return

Fig. 9 – Code flow that allows the operator to sendinstructions to the visually impaired user.

5.6. Extension5.6.1. Local Database

In this app, the database is implemented using SQLite. SQLite is a relational databasemanagement system contained in a small C programming library. In contrast to other databasemanagement systems, SQLite is not a separate process that is accessed from the clientapplication, but an integral part of it. Every application has its own database and can be easilymanaged by SQLiteOpenHelper.

The SQLiteOpenHelper class makes it easy for ContentProvider implementations to deferopening and upgrading the database until first use, to avoid blocking application startup withlong-running database upgrades. When the SQLiteOpenHelper is first called, it calls its onCreatefunction if no database exists. No matter the database is existed or not, by the functionSQLiteOpenHelper.getWritableDatabase we can obtain the current database, pass it to avariable and start to do the operation. During the backup procedure, the functionSQLiteDatabase.insert can be called to insert a new line of data into the database and thefunction SQLiteDatabase.query can be also used for searching data in the database. Moreover,the function SQLiteDatabase.delete removes a row from the database.

Our application has a database to store the building nodes information, when the user clicks onit (in real life the user scans the reader). The database stores the traversed building nodes ID,the visually impaired user ID, and their destination node. The database is available to thePercept users for table lookups. The matching between the ListView and the help request isdone using a BasedAdapter.

When the Help List is in the foreground, by clicking the different “Go to Help” buttons indifferent views, the application will pass the specific help information in this view. To achievethis function, we use a method in Base Adapter “get(position)” to find the correspondinglocation in the database, since database is a global variable. In Google Maps activity, we can usethe parameter to show the location on the map.

Figures 8 and 9 show the user interface for the Help list function and the Show Building Nodes.This is the view of the operator for the stored helps and the building nodes IDs.

Fig. 2 – A view of the Database Structure.

Fig. 3 – A view of the filled fields in the database

Fig. 4 – Show Help List database Fig. 5 – Get building nodes database

Fig. 6 – Flow of the List View Activity

5.6.2. Blind user APP

We build another app for blind user, which can receive the audio instructions. The application sendrequest to server to get the respond instructions and then use TextToSpeech class covert the text toaudio instruction.

Fig. 7 – Blind user App

5.6.3. Fingerprinting

The fingerprinting extension is divided in two parts: the training phase, and the tracking phase,also known as the offline and online phase, respectively. Based on the application needs, wedefined four points of interest in the Knowles Engineering Building to create our Wi-Fidatabase. The four points are the following: Emergency Door West, Emergency Door East,North Door, and Elevator.

We then scanned the all available Wi-Fi signals, get their MAC addresses, BSSID, operationchannel, mode of operation, and version of the 802.11 protocol. The training phase yields a listdozens of access points.

However, if we want our database to be reliable, we have to rely on the access points that areonline most of the time. In our environment, we filter the access points to use the university’saccess points only. This constrain reduced the number of observed access points dramatically.Another constrain we added, was to not consider the access points with RSSI lower than -80 dB.Yet, we selected the access points where we can encounter on the four points or have a goodRSSI. The result of this exercise is the table 1.

Table 1 - Wi-Fi database for the Knowles Engineering Buiding.

We did not implement the online phase on our application.

6. ConclusionWe believe our application fulfills the requirements for basic functionalities, and we were ableto implement some extra operations, such as the database and the voice command.

Find building nodes; Send help to the server; Get help from the server; Track user’s traversed nodes; Send instructions to the user; Build a database of building nodes and help list; Blind user application with voice command.

7. References[1] http://www.afb.org/section.aspx?SectionID=15 (Last access on 26 Oct 2013).

[2] A.Ganz, S. Gandhi, J.Schafer, T. Singh, E. Puleo, G. Mullett, and C. Wilson. "PERCEPT: Indoornavigation for the blind and visually impaired." Engineering in Medicine and BiologySociety,EMBC, 2011 Annual International Conference of the IEEE.

[3] http://www.umass.edu/researchnext/visionary-eye (Last access on 26 Oct 2013).

[4] http://usatoday30.usatoday.com/tech/news/story/2012-01-08/cnbc-near-fieldcommunication-mobile/52443756/1 (Last access on 26 Oct 2013).

[5] New Indoor Navigation Technologies Work Where GPS Can’t. IEEE Spectrum Online (LastAccess in 09/12/2013).

[6] A. Ganz, J. Schafer, E. Puleo, C. Wilson,. M. Robertson, “Quantitative and QualitativeEvaluation of PERCEPT Indoor Navigation System for Visually Impaired Users”, IEEE Engineeringin Medicine and Biology Society, San Diego, CA, September, 2012.

8. Group ContributionsXue Yu

User Interface Google Maps API Get Building Nodes Send Help Get Help Get Traversed Nodes Select different help request to help and give instructions Build local database with SQLite Blind user APP with audio instruction

Thiago Teixeira

Send Help Track Function Hash Map Traversed Nodes – On Marker Click Listener Wi-Fi Database Reports

percept: indoor navigation system for the blind and...

Documents