Grad Students: Manindra Moharana, Narendran Thangarajan, Soham ShahUndergrads: Cary Cheng, Christine He, Jessica Cho, Joann Kim,

Koa Nies, Luke Pickett

On-Screen T9 Keyboard for Bob

CSE 218

Locked-in Syndrome

Motivation

Trends in Interest on LIS by countryTrends in Interest on LIS by news headlines

Enable people affected by Locked-In Syndrome (LIS) to interact with any touch based display.

Existing Solutions - Tobii Eye Tracker

Pros● Navigate the web● Communicate by words● Skype, emails, music, etc.

Cons● Expensive● Complicated to use ● Bad tech support

Existing Solutions - EyeGaze Edge Talker

Pros● Standard keyboard with horizontal

and vertical eye tracking● Text to Speech● Quick access to frequent phrases

Cons● Difficult to use● Bob’s use case

Image Courtesy : https://www.youtube.com/watch?v=lY22CZ7XP-4, www.eyegaze.com

Vision

1. Easy to use tool for communication

2. Collect data to improve our tool and foster research.

Outline

● Requirements

● Product Description

● Team

● Software Process

● Architecture

● System design and implementation

● Post Mortem

Requirement Methods

● Professor● Articles on LIS

Two types of LIS○ Classical○ Total

We focus on classical LIS patients

Product Description

● Vertically T9 on-screen keyboard

● Activate on fixation

● Blink gestures for switching context (keyboard - predictions)

● Wink gesture to send keystrokes to application

TeamGrads

Undergrads

Weekly team meetings

Facebook group for communication

Google Docs for collaboration

Git(Bitbucket) + Eclipse

Software Process

● SCRUM1. Product Owner2. Scrum Masters3. Development Team

● Sprints1. Prototype (Madness

Demo)2. UX design, bugfixes,

backlog.

Architecture

Hardware● EyeTribe eye tracker● Microsoft Surface Pro 3

Software● Java● HTML/CSS/JS● Python/mongoDB

Microsoft Surface Internals

EyeTribe server

eyeTalk application backend

eyeTalk application

frontend

12

3

System

Four main components :

1. eyeTalk Backend2. eyeTalk UI3. T9 and word prediction4. eyeTalk analytics

1. eyeTalk backend

eyeTalk Backend

Image Courtesy : Eyetribe Docs

1. eyeTalk backend

● Written in Java

● GazeManager and IGazeListener to communicate with EyeTribe server.

● Technical Challenge : Handle saccades using running average filter

● At any point in time, provides the current smoothened value of X,Y

eyeTalk Backend

eyeTalk UI

Get gaze data points in real-time (polling)

2. eyeTalk UI

● Built using Processing graphics library (Java)

● Design focus/constraint - Use only vertical eye movement for control

● HCI design principles applied

● Dwell time (1.5 sec) based button clicks - with progress bar animation

● T9 and Manual input mode supported

2. eyeTalk UI● Modal screen to select from multiple

word predictions

● Integrated TTS (CMU Sphinx)

● On-screen keyboard, works without staying in focus

● Long blink to send keystrokes to foreground application (notepad, email client, etc.)

● Customisable UI

● T9

● Word Completion○ Higher weights for more

frequently used words

● Word Prediction○ Learns commonly used phrases

from corpus

● Preprocessing for quick lookups

3. T9 and word prediction

4. eyeTalk analytics

● Written in Python, JS.DB : mongoDB

● Technical Challenge : Real-time analytics using map-reduce to calculate heatmap on demand.

● Future:○ Streaming API support.○ Real-time rendering using

websockets.eyeTalk Backend

Push data points

map-reduce queries

How they all fit together

eyeTalk Backend

eyeTalk UI

Get gaze data points in real-time (polling)

Get predictionsT9 and word prediction algorithm

eyeTalk analytics

Post-mortem

● Lessons learnt○ Time management with part-time developers.○ Precision while assigning tasks.○ Identifying skillsets and exercising comparative advantage.

● What went right/wrong○ Team split and task allocation.○ Productive meetings.○ Avoiding new tools for project management.

● Sub-teams by expertise vs. interest.● Evaluation

Thanks!