drawings for alternative communications patent
TRANSCRIPT
1 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
1 / 4
3
22
FIG. 1
AT&T 3G 8:22 AM
Text Calendar Photos Camera
Stocks Maps WeatherRecorder
Calculator Notes SettingsClock
AAC+
Mail Safari iPodPhone
3
SMS 11
Thursday
73*
+ -
x /
1014
16
20
18
12
AAC+
2 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
2 / 4
3
FIG. 2PRIOR ART
MICROPHONE SPEAKER KEYBOARDDISPLAYCAMERA
USER INTERFACE
10
24 16 26 14 18
12
CPU
28
ACCELEROMETER
34
MEMORY
30
SOFTWARETRANSCEIVER
CIRCUITRY
36
32
WIRED
PORT
40
BATTERY
42
MISCELLANEOUS
PERIPHERALS
44
38
3 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
3 / 4
3
FIG. 3PRIOR ART
PSTN
INTERNET
PLMN
46
50
52
10D
56
10C
10B 10A
48
54
4 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
4 / 4
3
SMS11
73*
RH
10
58e
58d
58b
FIG. 4
58c
58a
62
6058f
64
5 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
5 / 4
3
FIG. 5
CAREGIVER ACQUIRES MOBILE TERMINAL
SOFTWARE LOADED ONTO MOBILE TERMINAL
ESTABLISH A USER PROFILE
STORE A CONTEXTUAL CRITERION FOR
OUTPUTTING A SPEECH ICON
OUTPUT THE SPEECH ICON
100
102
104
106
110
108
IS CRITERION SATISFIED?
YES
NO
6 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
6 / 4
3
AT&T 3G 8:22 AM
Profiles
AAC+BACK
AAC+
10
16
18
General Settings
Profiles and Settings
FIG. 6
AAC+
TO FIG. 8
“Caregiver Mode” Enabled
(tap to change to “End User Mode”)
7 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
7 / 4
3
FIG. 7
USER SELECTS PROFILES COMMAND
USER SELECTS PRE-EXISTING PROFILE OR TEMPLATE
150
152USER CREATES NEW PROFILE
158
LOAD154
GO TO BLOCK 108 OF FIGURE 5
EDIT156
GIVE NAME TO PROFILE160
GO TO BLOCK 104 OF FIGURE 5
8 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
8 / 4
3
FIG. 8
AT&T 3G 8:22 AM
AAC+
Profiles
BACK
Custom profiles
Templates
Nonverbal adolescent with autism
Help with expressing feelings
Adult user with semantic-pragmatic...
Jimmy Smith
Laura’s normal settings
Settings for Laura’s vacation
TO FIG. 10
TO FIG. 9200
202
204
9 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
9 / 4
3
FIG. 9
AT&T 3G 8:22 AM
AAC+
Profile: Jimmy Smith
BACK
208
220
218
EDIT
Icon size: Large
Default voice: Young Male
Assoc. #1: “McD Menu” at McDonalds
LOAD
210
212
206
Assoc. #2: “Help group” after “Help!”
214
Assoc. #3: “Basic conversation” near Mrs.
Miller
216
10 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
10
/ 43
FIG. 10
AT&T 3G 8:22 AM
AAC+
Create / Edit a Profile
BACK
Profile Name:
Enter name here
Speech/Language Impairment Info
General Settings for This Profile
Icons
SAVE PROFILE
To Fig. 11
To Fig. 15
To Fig. 30
User’s Schedule/Calendar
Icon-Context Associations To Fig. 18
Contexts To Fig. 21
11 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
11
/ 43
FIG. 11
=
AT&T 3G 8:22 AM
AAC+BACK
How would you like to provide speech/
language impairment information?
Take a survey
Enter specific impairment info
Perform a user evaluation
TO FIG. 12
TO FIG. 13
TO FIG. 14
Import data from another source
12 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
12
/ 43
FIG. 12
=
AT&T 3G 8:22 AM
Language Development SurveyBACK
In response to the following questions, please
mark “Yes” or “No”.
Does the user know the correct names of
some colors?
YES NO
Does the user say about 50 different words?
YES NO
Does the user say about 100 different words?
YES NO
Does the user say words that tell something
about other words, such as “loud” in the
utterance a “loud noise”?
13 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
13
/ 43
FIG. 13
=
AT&T 3G 8:22 AM
Enter Impairment InformationBACK
Dyslexia
Does user have dyslexia?
Vocabulary
General vocabulary level
Known vocabulary words
Stuttering and Annunciation
Below Avg Avg Above Avg
NO
Does user stutter? YES
Known annunciatory problem words:
Type here (e.g., “bathroom”)
CATEGORIES Colors
Aqua
Blue
Brown
Burgundy
Crimson
Forest Green
Gold
Green
Lavender
14 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
14
/ 43
FIG. 14
=
3G 8:22 AM
Say: “bathroom”BACK
Say the word “bathroom”:
PLAYSTOP
SaveUndo
RECREW 66
66a
14
Say, “bath-room”
please.
15 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
15
/ 43
FIG. 15
=
AT&T 3G 8:22 AM
AAC+BACK
Which general settings would you like to
program for this profile?
Caption font
Caption size
Icon style
Icon size
Language
Voices
Repetitions
Participatory voice output
Icons per screen
TO FIG. 16
TO FIG. 17
Select a default voice:
Young male (synthetic)
Adult female (synthetic)
Rosa
Voice 124153
Jimmy Smith’s voice
Save
68
Volume
16 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
16
/ 43
FIG. 16
=
AT&T 3G 8:22 AM
AAC+BACK
Configure default “Repetitions” settings that
should apply to this profile:
Number of repetitions
Accuracy threshold
Penalties
Rewards
Prompt users to repeat
voice outputs
Number of required repetitions:
Save
ON
0
1
2
3
4
How accurate must the repetition be?
Save
30%
40%
50%
60%
70%
Configure “Repetitions” for a particular speech
icon or group of icons:
Browse categories
Search
Penalties
Save
Enforce a penalty if
user fails to repeat
Disable all voice output
OFF
Disable a speech icon
Restrict game/Web access
Rewards
Save
Give a reward if user
completes repetition
Tokens and points
ON
Messages
Enable game/Web access
TO
FIG. 30
17 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
17
/ 43
FIG. 17
=
AT&T 3G 8:22 AM
AAC+BACK
Configure default “Participatory voice output”
settings that should apply to this profile:
Frequency of participation
Accuracy threshold
Penalties
Rewards
Prompt users to participate
in voice outputs
Frequency of participation:
SaveON
How accurate must the participation be?
Save
30%
40%
50%
60%
70%
TO FIG. 30
Sometimes
Word or phrase length
Word or phrase type
Always
Browse categories
Search
Specific words, phrases, etc:
Type of participation
Save
User finishes
Step by step
Increase MLU over time
User begins
Type of participation:
18 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
18
/ 43
FIG. 18
AT&T 3G 8:22 AM
Icon-Context AssociationsBACK +
“BK Menu” at 32 Main Street Burger King
“Help group” after “Help!”
Rule 1 (11/15/10 4:57 pm)
“Basic conversation” near Mrs. Miller
Rule 2 (11/23/10 6:18 am)
“Recess icons” on Mon-Fri from 12:15 to...
“Nervous emotions” based on heart rate and…
TO FIG. 19
“Swing set” at McNamee Park
19 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
19
/ 43
FIG. 19
=
AT&T 3G 8:22 AM
BACK Create TO FIG. 20
WQ E R T Y U I O P
SA D F G H J K L
XZ C V B N M
space return.?123
Name your Icon-Context Association:
Icon-Context Assoc.
20 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
20
/ 43
FIG. 20
=
AT&T 3G 8:22 AM
Icon-Context AssociationsBACK
How would you like to create an
association between a speech icon and a
particular context?
Start with a context, then select
icon(s) that should be presented
in that context
Start with an icon, then select
context(s) in which that icon should
be presented
TO FIG. 21
TO FIG. 30
21 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
21
/ 43
FIG. 21
=
AT&T 3G 8:22 AM
ContextsBACK
Sounds, noises or speech
Physical movement
Objects or people near this device
TO FIG. 22
Temperature
Lighting
The location of this device
The user’s body
Times, dates and scheduled events
+ TO FIG. 29
Search
22 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
22
/ 43
FIG. 22
=
AT&T 3G 8:22 AM
Icon-Context AssociationsBACK
Specify a location that should trigger
output of a speech icon:
Search/browse by location name
Find a location on a map
Draw a “zone” on a map
How would you like to identify a
location?
70
TO FIG. 23
Browse categories
Search
23 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
23
/ 43
FIG. 23
AT&T 3G 8:22 AM
Results for “McDonalds”BACK
1. 5065 Main Street | Anycity, USA | 2.84 mi
2. 390 Route 105 | Anycity, USA | 4.31 mi
3. 12 Drawley Lane | Nextown, USA | 5.39 mi
4. 101 Elm Street | Nextown, USA | 7.01 mi
5. 65 Far Drive | Farville, USA | 12.4 mi
Show More Results
TO FIG. 24View Selected
24 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
24
/ 43
FIG. 24
AT&T 3G 8:22 AM
View/edit context (McDonald…)BACK
McDonald’s
5065 Main Street | Anycity, USA
General Information
TO FIG. 25
ATM WIFI
View/Edit Icons and Settings for “McDonald…”
Add New Speech Icon(s)
Browse categories
Search
More
info
TO FIG. 27No icons have been saved in
association with this location.
25 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
25
/ 43
FIG. 25
AT&T 3G 8:22 AM
Icon results for “fries”BACK
FRENCH
FRIES (1)
Saved to 13 contexts
Frequency of use: High
Add Selected Icons to Context
FRENCH
FRIES (2)
Saved to 4 contexts
Frequency of use: Medium
FRIED
DOUGH
Saved to 0 contexts
Frequency of use: Low
More
info
More
info
More
info
26 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
26
/ 43
FIG. 26
AT&T 3G 8:22 AM
Add “FRENCH FRIES (1) to…”BACK
FRENCH
FRIES (1)
Add speech icon:
FRENCH FRIES (1)
to “McDonald’s
5065 Main St.”?
TO FIG. 27
McDonald’s
5065 Main
Street
Save Icon to This Location
27 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
27
/ 43
FIG. 27
AT&T 3G 8:22 AM
Icons and settings for “McD…”BACK
BURGER FISH
SANDWICH
FRENCH
FRIES
Caption font
Caption size
Icon style
Icon size
Language
Voices
Repetitions
Participatory voice output
Icons per screen
Icons (6):
Settings for this context:
Icon ranking
Screen
Editor
TO FIG. 28
Volume
28 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
28 / 4
3Screen Editor for “McDonald’s 5056 Main Street”BACK
FIG. 28
Screen Editor for “McDonald’s 5056 Main Street”BACK
BURGER
FISH SANDWICH
FRENCH FRIES
ICE CREAM
PLEASECOLA
72
74
76
78
80
82
SAVE
SAVE
29 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
29
/ 43
FIG. 29
AT&T 3G 8:22 AM
Create a new locationBACK
Name this location:
Drag and drop pins to define boundaries:
SaveUndo
Enter name here
84
30 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
30
/ 43
FIG. 30
AT&T 3G 8:22 AM
IconsBACK +
actions
animals
around the house
art
body parts
clothing
colors
Search TO FIG. 31
31 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
31
/ 43
FIG. 31
AT&T 3G 8:22 AM
Icon results for “burger”BACK
BURGER
Saved to 9 contexts
Frequency of use: HighTO FIG. 32
BURGER
WITH BACON
Saved to 2 contexts
Frequency of use: Low
HAM
BURGER
Saved to 0 contexts
Frequency of use: Low
32 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
32
/ 43
FIG. 32
AT&T 3G 8:22 AM
Icons: BURGERBACK
BURGER
Saved to 9 contexts
Frequency of use: High
Related icons
Caption font (Arial)
Caption size (Normal)
Icon style (Photo)
Icon size (Medium)
Language (English)
Voices (Young male)
Repetitions (OFF)
Participatory voice output (OFF)
Settings for this icon:
Volume (Default)
TO FIG. 21Add “BURGER” to a new context
33 / 43
Inventor(s): TEDESCO et al.Title: METHODS FOR AUGMENTATIVE AND ALTERNATIVE COMMUNICATIONS INFORMED BY ENVIRONMENTAL,
BEHAVIORAL, AND BIOLOGICAL CONTEXTSDocket No.: 2001-011.0001
Filing Date: FILED HEREWITH
33
/ 43
FIG. 33
AT&T 3G 8:22 AM
View/edit context (Low blood…)BACK
Low blood sugar
(beneath 70 mg/dL for more than
15 minutes)
General Information
View/Edit Icons and Settings for “Low blood…”
Add New Speech Icon(s)
Browse categories
Search
More
infoBURGER
70
34 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
34 / 4
3
FIG. 34A
=
8:22 AM
Create a new location
Name this location:
SaveUndo
FIG. 34B FIG. 34C
10
Drag and drop pins to define boundaries:
“Maiden Ridge” Neighborhood
STEPHEN
“Maiden Ridge” Speech Icons (Screen 1)BACK
SALLY
GO
JOE PETER
PASS TURN
“Maiden Ridge” Speech Icons (Screen 2)BACK
“Maiden Ridge” Speech Icons (Screen 3)BACK
ME
WANT
I MRS. SMITH
HOME PLEASE
SKATEBOARD
RAMP
BASKETBALL
FOOTBALL
COOL
FUN
COOL
FUN
86a
86b
86c
88a
88b
90
88c
92
16
35 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
35 / 4
3
FIG. 35A
Set maximum levels for:
FIG. 35B FIG. 35C
130 BPM
Heart Rate
Med-High
Electrodermal Activity
Heart R
ate (BP
M):
Levels Exceeded!
Transmitting message...
GS
R:
8:22 AM
AAC+
10
58b
58d
-5000
0
5000
10000
15000
=
8:22 AM
11
7
3
*
MedLow Med-Low
54
AAC+ -- Caregiver Workstation
Profile: Karen Settings History Progress
Save
Changes
Undo
Now editing Icon-
Context Association:
“Nervous emotions
based on heart rate
and electrodermal...”
AAC+
Are you feeling
NERVOUS or
SCARED?
“Nervous Emotions...” Speech Icons (Screen 1)BACK
NOW
94b
94a
I FEEL
NERVOUS EXCITED SCARED FINE
Active Voice OutputBACK
NOW”“I FEEL NERVOUS
14
94c
SPEAK
96
10
56
36 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
36 / 4
3
58e
FIG. 36A
=
8:22 AM
“MILLER!” (voiceprint)
Record audio for “MILLER!”:
PLAYSTOP
SaveUndo
RECREW
62
Cur
rent
aud
io:
MATCH!
Opening “Basic Conv…”
“MIL
LER
!”:
8:22 AM
AAC+
FIG. 36B FIG. 36C
“Basic Conversation near Mrs. Miller” Speech IconsBACK
MILLERHI MRS.
14
Repetition Exercise!BACK
MILLERHI MRS.
JIMMY - NOW YOU SAY:
Repetition Exercise!BACK
MILLERHI MRS.
Listening...
98a
98b 98c
37 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
37 / 4
3
58e
FIG. 36D FIG. 36E FIG. 36F
Repetition Exercise!BACK
MILLERHELLO MRS.
14
Repetition Exercise!BACK
JIMMY - NOW YOU SAY:
Repetition Exercise!BACK
98i
98j 98k
Repetition Exercise!BACK
MILLERHI MRS.
Listening...
98d
Repetition Exercise!BACK
MILLERHI MRS.
Listening...
98e
Repetition Exercise!BACK
MILLERHI MRS.
Repetition Complete!
98f
Repetition Exercise!BACK
Congratulations!
98g
14
MILLERHI MRS.
You earned 10 points
toward your goal.
Goal: Trip to moviesBACK
98h
90%
Your goal:
Progress:TRIP TO MOVIES
THERE
MILLERHELLO MRS.THERE
Congratulations!
You earned:
MILLERHELLO MRS.THERE
100%
TRIP TO MOVIES
38 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
38 / 4
3
LOADED AS CURRENT PROFILE?: YES
FIG. 37
I-001
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
HOT HOT.TXT NONE DEFAULT 0 OFF HOT.WAV
I-002 FOOD FOOD.GIF NONE MEDIUM 0 OFF (TTS)
I-003 LOST LOST.MOV C-001 LARGE 0 OFF LOST.MOV
NONE
NONE
“EMERGENCY”
I-004 REFRIGERATOR FRIDGE.JPG C-003 DEFAULT 1 OFF FRIDGE.MP3
I-00N ETC. ETC. ETC. ETC. ETC. ETC. ETC.
“IN THE
KITCHEN”
ETC.
ICON
IDICON LABEL ICON FILE
ASSOCIATED
CONTEXT IDICON SIZE REPETITIONS PARTICPATION
ASSOCIATED
VOICE
ASSOCIATED
CONTEXT
LABEL
SPEECH ICON DATABASE
39 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
39 / 4
3
LOADED AS CURRENT PROFILE?: YES
FIG. 38
C-001
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
EMERGENCYI-003, I-138,
I-284, I-019...LARGE
DEFAULT TO
PROFILE
BODY TEMP >
100* F
PRESSURE
SENSOR
DETECTS...
BRIGHTNESS
“VERY HIGH”
OR “VERY L…”
C-002BULLY
NEARBYI-675, I-029,
I-444, I-572LARGE
DEFAULT TO
PROFILE
MT-ID-001
WITHIN 10
METERS
LOCATION IS
“ELM SCHOOL”ETC.
C-003IN THE
KITCHEN
I-002, I-004,
I-887,I-373...
DEFAULT TO
ICON
DEFAULT TO
PROFILE
WORD
“KITCHEN” IS
UTTERED
CAMERA
DETECTS... ETC.
0
DEFAULT TO
ICON
DEFAULT TO
ICON
C-004NERVOUS
EMOTIONS BA...
I-432, I-071,
I-273, I884...
DEFAULT TO
PROFILEON
HEART RATE >
130 BPM
GSR DATA
INDICATES...ETC.
I-00N ETC. ETC. ETC. ETC. ETC. ETC. ETC.
DEFAULT TO
ICON
ETC.
CONTEXT
ID
CONTEXT
LABEL
ASSOCIATED
ICON ID(S)ICON SIZE PARTICPATION TRIGGER 1 TRIGGER 2 TRIGGER NREPETITIONS
SAVED CONTEXT DATABASE
40 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
40 / 4
3
FIG. 39
WEDNESDAY
FEBRUARY 13:13 PM
PLAYDATE
WITH SALLY30- 40 DB
“HELP,” CAR
HORN, ...JIMMY, SALLY
MCNAMEE
PARK
DATE TIMEEVENT/
SCHEDULEAMBIENT
VOLUME
LAST 5
SOUNDS
PEOPLE
PRESENTLOCATION
SLIDE MEDIUM-HIGH 77* F 99* F 82 MG/DL NORMALNORMAL
DAYLIGHT
OBJECTS
PRESENT
MOTION
ACTIVITY
OUTSIDE
TEMPERATUREBODY
TEMPERATURE
BLOOD
GLUCOSE
LEVEL
BLOOD
PRESSUREBRIGHTNESS
LOADED AS CURRENT PROFILE?: YES
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
CURRENT CONTEXT DATABASE
41 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
41 / 4
3
LOADED AS CURRENT PROFILE?: YES
I-039
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
SHAREACTIONS,
SOCIAL
WEDNESDAY,
2/1, 1:13 PM
VOICEPRINT
OF SALLY
SLIDE, BALL,
SALLY, ...NONE 97*
I-447 BULLYPEOPLE,
SCHOOL, ...
THURSDAY,
2/9, 12:17 PM
60 DB+
(UNINTEL...)UNKNOWN MT-ID-001 97*
I-602 NONE DRINKSSUNDAY,
2/12, 11:15 AM(NONE) SUN NONE 100*
PLAYGROUND
SCHOOL
BACKYARD
ICON
IDICON LABEL
ICON
CATEGORY/
CATEGORIES
DATE / TIMEDETECTED BY
MICROPHONE
DETECTED BY
CAMERA
OTHER
MOBILE
TERMINALS
NEARBY
BODY
TEMPERATURELOCATION
USAGE HISTORY DATABASE
FIG. 40
I-00N ETC. ETC. ETC. ETC. ETC. ETC. ETC.ETC.
42 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
42 / 4
3
LOADED AS CURRENT PROFILE?: YES
I-039
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
SHARE (21)ACTIONS (61),
SOCIAL (17)
WEDNESDAY
(11)
PLAYGROUND
(13)
SALLY
(10)
BALL
(10)
MOM’S PHONE
(4)
TUESDAY
(5)
HOME
(6)
“BALL”
(5)
OTHER
(11)
NONE
(17)
SATURDAY
(5)
OTHER
(2)
OTHER
(6)
12:00 – 1:00
PM (8)
3:00 – 4:00 PM
(7)
2:00 – 3:00 PM
(6)
ICON
IDICON LABEL
ICON
CATEGORY/
CATEGORIES
DATE LOCATIONDETECTED BY
MICROPHONE
DETECTED BY
CAMERA
OTHER
MOBILE
TERMINALS
NEARBY
TIME
USAGE TRENDS DATABASE
(INCIDENCE IN PARENTHESIS)
FIG. 41
43 / 43
Inventor(s): TE
DE
SC
O et al.
Title: M
ET
HO
DS
FO
R A
UG
ME
NT
AT
IVE
AN
D A
LTE
RN
AT
IVE
CO
MM
UN
ICA
TIO
NS
INF
OR
ME
D B
Y E
NV
IRO
NM
EN
TA
L, B
EH
AV
IOR
AL, A
ND
BIO
LOG
ICA
L CO
NT
EX
TS
Docket N
o.: 2001-011.0001F
iling Date: F
ILED
HE
RE
WIT
H
43 / 4
3
LOADED AS CURRENT PROFILE?: YES
FIG. 42
PROFILE ID: P-0001
PROFILE NAME: JIMMY SMITH
LANGUAGE
DEVELOPMENT
SURVEY
QUESTION 1
YES
LANGUAGE
DEVELOPMENT
SURVEY
QUESTION 2
YES
GENERAL
VOCABULARY
LEVEL
40TH
PERCENTILE
KNOWN
VOCABULARY:
COLORS
BLUE,
YELLOW
KNOWN
VOCABULARY:
PEOPLE
ME, DAD,
MOM, SARAH...
KNOWN
VOCABULARY:
TRANSPORT
BIKE, CAR,
TRAIN...
DYSLEXIA
NO
LANGUAGE
DEVELOPMENT
SURVEY
QUESTION 3
NO
LARYNX
TRAUMA
NO
LANGUAGE
REGRESSION
TRIGGER 1
LOUD AREAS
LANGUAGE
REGRESSION
TRIGGER 2
NEW PEOPLE
STUTTERING
YES
(MODERATE)
STUTTERING
PROBLEM
WORD 1
BATHROOM
STUTTERING
PROBLEM
WORD 2
BARKING
STUTTERING
PROBLEM
WORD 3
CHICKEN
STUTTERING
PROBLEM
WORD 4
CHURCH
USER IMPAIRMENT DATABASE