1

The Context Engine -A Device to Augment

Cognition

Perry McDowell

The MOVES Institute

2

Augment Cognition

Add to your thinking, hopefully improving itThat’s been the whole point of computersWe want to make it completely natural and

automatic In some ways, similar to:

An admiral’s aide or efficient secretary A senior NCO or subject matter expert

Want to do this with a wearable computerMove nearer to ubiquitous computing

3

Literary Inspiration for the Military Context Engine

“Our suits give us better eyes, better ears… better intelligence… the beauty of it [is] you don’t have to drive it, fly it, conn it, operate it, you just wear it… This leaves you with your whole mind free to handle your weapons and notice what is going on around you… If you load a mud foot down with a lot of gadgets that he has to watch, someone a lot more simply equipped – say with a stone axe – will sneak up and bash his head in while he is trying to read a vernier.”

- Robert A. Heinlein, Starship Troopers

4

Steve Mann’s Six Attributes of Wearable Computing

1. Unmonopolizing of the user's attention It does not cut you off from the outside world. You can attend to other matters while using the

apparatus. Assumes computing will be a secondary activity. Provides enhanced sensory capabilities.

2. Unrestrictive to the user You can do other things while using it

5

Steve Mann’s Six Attributes of Wearable Computing(cont)

3. Observable by the user It can get your attention continuously if you desire The output medium is constantly perceptible by

the wearer.

4. Controllable by the user5. Attentive to the environment

Environmentally aware, multimodal, multisensory.

6. Communicative to others Can be used as a communications medium.

6

Steve Mann’s Six Attributes of Wearable Computing(cont)

Implied by the above six properties is that it must also be:

Constant: always on, running, and ready. Personal: human and computer are inextricably

intertwined. Prosthetic: you can adapt to it so that it acts as a true

extension of mind and body; after time you forget that you are wearing it.

Assertive: resists, if you wish, prohibition or requests by others for removal.

Private: others can't observe or control it unless you let them.

7

Example of Uses -Finding a Restaurant

You’re downtown and decide you want to eat out

The engine makes a recommendation based upon several factors:

Tonight’s specialsTonight’s specials Friends’ favoritesFriends’ favorites ReviewsReviews Current waitCurrent wait

LocationLocation CostCost Favorite foodsFavorite foods Favorite restaurantsFavorite restaurants

8

Finding a Restaurant

Let’s take it one step further: You can tell it you want to take your wife out for

your anniversary - handle it It knows your favorite romantic restaurants, so it makes a

reservation at one It contacts one of the babysitters you use and arranges

services for that night Orders flowers delivered to your wife’s office Reminds you to buy a present, or orders one from the web

for you Schedules it with your wife’s contact engine

Tells you that everything is ready to go

9

Examples of Uses - Watch Turnover

At watch relief, transfer all pertinent data to your relief List of contacts and their status, current/ pending

orders, equipment status, upcoming events, etc.

The system can’t forget anything, rush so it can hit the pit, misinterpret what it’s told…

The system would also be able to give you advice - “No, you don’t want to turn left…”

10

Example of Uses - Crossing the River

You are the leader of a platoon of infantry on a covert mission, currently moving to conduct a rendezvous across a river The bridge you were expecting to use is

destroyed

What are you going to do now?

11

Crossing the River (cont)

Determining your current options, weighing them, and deciding upon a course of action takes time

Context engine immediately determines options and the best course of action and makes a recommendation It can explain its reasoning fairly quickly to

you

12

Examples of Uses -Summary

In the dinner example, it acts mainly as an assistant - it knows your likes and searches data for the best match

In the river example, it acts as an expert system, giving you a better solution

In the watch example, it combines the two - it stores data for use later, but can have intelligence (either the previous watch officer or from the system) built in

13

Goals

Our goal for this project is to take Commercially Off the Shelf (COTS) hardware and software to create a proof of concept for the Context Engine.

14

Difficult Problems

Determining the “symbolic content” of the user’s environment

Once it is symbolized, interpreting the environment to give information and advice

Presenting the information to the user in an unobtrusive and helpful manner

15

Breakdown of the Context Engine’s Processes

LocationLocation

ActionsActions

PeoplePeople

ObjectsObjects

Outer World Outer World AbstractionAbstraction

Determine Determine Course of Course of

ActionAction

Output Output to Userto User

User’s User’s CommandsCommands

Goal 1Goal 1

Goal 4Goal 4 Goal nGoal n

Goal 2Goal 2

Goal 3Goal 3

PreferencesPreferences

Visual Visual DisplayDisplay

Aural Aural DisplayDisplay

16

Initial Process - Recognize the World

LocationLocation

ActionsActions

PeoplePeople

ObjectsObjects

Outer World Outer World AbstractionAbstraction

To course To course of action of action

17

Symbolic Content

Having the computer determine where it is, who/what is present and what is happening Computer visualization Wireless recognition

Bluetooth Aural clues GPS

18

Current Systems Which Convert Data to Abstraction

Face Recognition Several companies has fairly effective

facial recognition systems. Lau Technology’s daughter company, Viisage,

scanned all the faces at the Super Bowl, looking for terrorists

We anticipate getting an SDK from one of these companies to begin our research

19

Current Systems Which Convert Data to Abstraction

Wearable belt for outdoor sports Determines events which might indicate

the wearer is incapacitated High acceleration Vital Signs

Alerts authorities to

potential injury

20

Determining The Course of Action

Determine Determine Course of Course of

ActionAction

User’s User’s CommandsCommands

Goal 1Goal 1

Goal 4Goal 4 Goal nGoal n

Goal 2Goal 2

Goal 3Goal 3

To Output To Output to User to User

From World From World Recognition Recognition

21

Determining the Course of Action

Once you know what is in the environment, what does that mean? Network solution

Take the dynamic extensibility from NPSNET-V Embedded XML information explains what an

object is and how to interact with it SIMS style smart environment

Objects broadcast what they can do

22

Determining the Course of Action

Get input from the user’s body Affective computing takes body measurements to

determine your mental/ physical state The context engine will be able to indicate when

medical assistance is required Expedite triage also

System knows not to deliver bad news when you’re angry/ tired/ morose/ etc.

System can tell you’re beginning to panic and “slaps you in the face”

23

Affective Computing GearThis affective wearable includes a ProComp sensing system

(upper left corner) surrounded by four sensors, clockwise from top: respiration, galvanic skin response (GSR) (center, left), blood volume pressure (BVP), and electromyogram (EMG). This unit attaches to a PC104 standard based computer (lower right) which receives data from a Twiddler hand-held keyboard (lower left). and displays data with the Private Eye (far right, below respiration sensor.) (J. Healey)

24

Determining the Course of Action

User can input commands which give the computer tasks to determine

The user’s commands are evaluated in light of the current goals Goals are determined either explicitly or

implicitly Explicitly means that the user has previously

defined this as a goal Implicitly means the computer has learned that

this goal is important

25

Determining the Course of Action

Database sharing Gives one user the knowledge and insight

of another user Can be as simple as identifying someone Can be as complex as planning an exercise

Will raise emphasis on training people's machines closer to the level we currently spend on training people May be possible to train both simultaneously

26

Outputting Information to the User

Output Output to Userto User

PreferencesPreferences

Visual Visual DisplayDisplay

Aural Aural DisplayDisplay

From Course From Course of Actionof Action

27

Presenting Information to the User

Interface must be: Light Small Unobtrusive Intuitive Robust Easy to use Cheap

Easier said than done

28

Presenting Information to the User

“Fitting” the system to specific users Everyone wants different

information User must find the given

information useful Don’t want the annoying Don’t want the annoying

paperclip scenariopaperclip scenario

29

30

Conclusion

Context Engine must be easy and natural to use

The problem has three major areas: Recognizing the world Determining the course of action Presenting information to the user

31

NPS/MOVES Work

Construct a virtual world and build a more complete and robust engine

Several options as to the input to the world: Joint SIGINT Avionics Family (JSAF) Data

Plan on using agents: To determine context from the symbolic

representation To determine recommended course of action

32

Potential Thesis Work

You can work in any part of the project discussed in this brief First part will be extremely difficult Second part will be difficult, but you’ll be

working on a smaller part of the problem Third part will be very interesting, and is a

great opportunity for someone interested in human factors work

33

Questions?

Contact Information:

Perry McDowell

Lecturer of Computer Science

plmcdowe@nps.navy.mil

http://www.movesinstitute.org/~mcdowell