How may I help you?

User: I need to do some surveillance looking for missile launchers.

I need to do some surveillance looking for missile launchers

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Text: I need to do some surveillance…Start Time: XDuration: Y

Ø

[mission-type = ?] & [ “surveillance” => mission-type = Recon] => [mission-type = Recon]

Learned from previous interactions

How may I help you?

System: Do you want to start a new mission or use parameters from an old one.

Do you want to start a new mission or use parameters from an old one.

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Do you want to start…Start Time: XDuration: Y

Ø

There are two options when synthesizing a new system:1. Start with parameters from an old mission2. Start with a clean slate

How may I help you?

User: I want to use the same parameters as last week’s mission.

I need to do some surveillance looking for missile launchers

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Text: I need to do some surveillance…Start Time: XDuration: Y

Ø

Look up last week’s mission

Mission DB

System: Which mission?

Mission Type Start Date End Date Target

Recon 7/27/2009 13:45 7/28/2009 13:45 34°31 59″N 69°09 58″E′ ′

Recon 7/29/2009 14:00 7/30/2009 14:00 35°35 60″N 68°55 01″E′ ′

Recon 7/29/2009 6:00 8/1/2009 18:00 34°31 59″N 69°09 58″E′ ′

Which mission

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Which missionStart Time: XDuration: Y

Ø

There was more than one mission last week

Mission DB

User: (This one)red

Mission Type Start Date End Date Target

Recon 7/27/2009 13:45 7/28/2009 13:45 34°31 59″N 69°09 58″E′ ′

Recon 7/29/2009 14:00 7/30/2009 14:00 35°35 60″N 68°55 01″E′ ′

Recon 7/29/2009 6:00 8/1/2009 18:00 34°31 59″N 69°09 58″E′ ′

This (one)1

Stroke 1 = Red arrow with (X,Y,L)

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Points: {X, Y, Time}, …Color: Red

Text: This oneStart Time: XDuration: Y

Look up the parameters for the selected mission

Mission DB

Mission Type: Recon

Mission Signal: Video

Video Signal Resolution: 1024 x 768

Video Signal Disposition: Streamed

Mission Target Mission Egress RouteMission Ingress Route

System: I have loaded the following parameters, would you like to make any changes?

I have loaded the following parameters, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I have loaded…Start Time: XDuration: Y

Ø

Mission DB

[mission-target = (X,Y)] => [mission-target-terrain = plains] => [LOS = true][mission-target = (X,Y)]

[∆ ingress/egress > twistiness-limit] => [mission-capability: Precision Guidance][mission-ingress-route = {X,Y}…][mission-egress-route = {X,Y}…]

[mission-signal-type = Video][video-signal-resolution = 1024 x 768] => [mission-capability: Video Recon]

[mission-signal-disposition = streamed][mission-signal-disposition = streamed] & [LOS = true] => [mission-capability: Retransmission]

User: I’d like to change the target.

Mission Type: Recon

Mission Signal: Video

Video Signal Resolution: 1024 x 768

Video Signal Disposition: Streamed

Mission Target Mission Egress RouteMission Ingress Route

I’d like to change the target

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: I’d like to change…Start Time: XDuration: Y

[mission-ingress-route = ? ][mission-egress-route = ?][mission-target = ?]

[mission-target-terrain = ?][LOS = ?]

[mission-signal-type = Video][video-signal-resolution = 1024 x 768] => [mission-capability: Video Recon]

[mission-signal-disposition = streamed]

System: Please indicate on the map the new target area.

Please indicate on the map the new target area

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Please indicate…Start Time: XDuration: Y

Ø

[mission-target = ?]

User: ()red

Stroke 1 = Red circle with (X,Y,R)

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Points: {X, Y, Time}, …Color: Red

Ø

[mission-target = (X,Y)] => [mission-target-terrain = mountain] => [LOS = false][mission-target = (X,Y)]

[mission-signal-disposition = streamed] & [LOS = false] => [mission-capability: Non-LOS Retransmission]

System: Please indicate the desired routes to and from the target area.

Click and drag points on the routes to change them

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Click and drag…Start Time: XDuration: Y

Ø

[mission-ingress-route = ? ]

[mission-egress-route = ?]

User: (This way to the target)teal and (this way back)blue.

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Ø

[mission-ingress-route = {X,Y}….]

[mission-egress-route = {X,Y}….]

Ø

Ø

[∆ ingress/egress > twistiness-limit] => [mission-capability: Precision Guidance]

Mission Type: Recon

Mission Signal: Video

Video Signal Resolution: 1024 x 768

Video Signal Disposition: Streamed

System: I have loaded the following parameters, would you like to make any changes?

Mission Target Mission Egress RouteMission Ingress Route

I have loaded the following parameters, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I have loaded…Start Time: XDuration: Y

Ø

[mission-target = (X,Y)] => [mission-target-terrain = mountain] => [LOS = false][mission-target = (X,Y)]

[∆ ingress/egress > twistiness-limit] => [mission-capability: Precision Guidance][mission-ingress-route = {X,Y}…][mission-egress-route = {X,Y}…]

[mission-signal-type = Video][video-signal-resolution = 1024 x 768] => [mission-capability: Video Recon]

[mission-signal-disposition = streamed][mission-signal-disposition = streamed] & [LOS = false] => [mission-capability: Non-LOS Retransmission]

User: No

Mission Type: Recon

Mission Signal: Video

Video Signal Resolution: 1024 x 768

Video Signal Disposition: Streamed

Mission Target Mission Egress RouteMission Ingress Route

No

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: NoStart Time: XDuration: Y

System stores the parameters

System: Please indicate when you want the platform on target.

August 1July 26 July 27 July 28 July 29 July 30 July 31

-each tick is 3 hours-Precipitation

Heavy Precipitation

Cloudy

Clear

Please indicate when you want the platform on target

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Please indicate when you want…Start Time: XDuration: Y

Ø

[mission-type = Recon] requires [mission-time-on-target = ?]

User: ()red

-each tick is 3 hours-

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

Heavy Precipitation

Cloudy

Clear

Stroke 1 = Red line with (X,Y,L)

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

Points: {X, Y, Time}, …Color: Red

Ø

[mission-time-on-target = 7/30/09 0300 - 8/1/09 0300] & [mission-target = (X,Y)]

=> [mission-target-weather = Precipitation]

[mission-time-on-target = 7/30/09 0300 - 8/1/09 0300]

System: Both the Reaper and the Predator are acceptable platforms for the currentmission. Which platform would your prefer to use?

Predator

Reaper

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Both the Reaper and the Predator are acceptable platforms for the current mission. Which platform would your prefer to use?

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Both the Reaper and the ….Start Time: XDuration: Y

Ø

[mission-capabilities = Video Recon, Precision Guidance,

Non-LOS Retransmission

=> Weight = X, Power = Y, Memory = Z

[mission-platform = ?] &[Predator: Weight > X, Power > Y, Memory > Z] &[Reaper: Weight > X, Power > Y, Memory > Z] &[Scan Eagle: Weight < X, Power < Y, Memory < Z] &[Avenger: Weight < X, Power < Y, Memory < Z]

User: Let’s use the Predator

Predator

Reaper

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Let’s use the Predator

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: Let’s use the PredatorStart Time: XDuration: Y

[mission-platform = Predator]

System: Why did you choose the Predator?

Predator

Reaper

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Why did you choose the Predator

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: Why did you…Start Time: XDuration: Y

Ø

System wants to capture rationale behind platform selection

User: The Predator is more reliable in precipitation.

Predator

Reaper

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

The Predator is more reliable in precipitation

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: The Predator…Start Time: XDuration: Y

If [mission-target-weather = Precipitation] => Predator is the preferred platform

System: In order to fulfill the given mission specifications, your platform will need these capabilities.

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

In order to fulfill the given mission specifications, your platform will need these capabilities

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: In order to fulfill the …Start Time: XDuration: Y

Ø

[mission-capabilities: Precision Guidance, Video Recon, Non-LOS Retransmission]

System: I can synthesize a software system that supplies these capabilities and I have provided asimulation of each collected signal, would you like to make any changes?

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I can synthesize…Start Time: XDuration: Y

Ø

Precision Guidance, Video Recon,

Non-LOS Retransmission

PAMS

Valid configuration for: Precision Guidance, Video Recon,

Non-LOS Retransmission

User: Tell me why the video is so grainy.

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

Tell me why the video is so grainy

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: Tell me…Start Time: XDuration: Y

System looks up how it determined the quality of the video signal

System: The weather forecast is for precipitation and the Video Recon capabilitydoes not perform well in those conditions.

Recon Mission

Video Signal

Signal Transmission

PrecipitationPrecipitation

The weather forecast is for precipitation and the Video Recon capabilitydoes not perform well in those conditions

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: The weather forecast…Start Time: XDuration: Y

Ø

(Video Recon => Low Resolution Camera Resource ) &[mission-weather = precipitation]

=> 25% effective/quality

User: Can I improve the video quality?

Recon Mission

Video Signal

Signal Transmission

PrecipitationPrecipitation

Can I improve the video quality

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: Can I…Start Time: XDuration: Y

System looks up a way to get a video signal that is of a higher quality

System: You can improve the video quality by substituting HD Video Recon for Video Recon.

HD Video ReconVideo ReconRecon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

You can improve the video quality by substituting HD Video Recon for Video Recon in your AV Recon capability

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: You can…Start Time: XDuration: Y

Ø

(Video Recon => Low Resolution Camera Resource )& [mission-weather = precipitation]

=> 25% effective/quality

(HD Video Recon => High Resolution Camera Resource )& [mission-weather = precipitation]

=> 50% effective/quality

User: Let’s use HD Video Recon

HD Video ReconVideo ReconRecon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

Let’s use HD Video Recon

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: Let’s…Start Time: XDuration: Y

[mission-capabilities: Precision Guidance, , Non-LOS Retransmission] HD Video Recon

System: The configuration that you have selected can support only an one day mission. Which is more important, having a two day mission or high quality video?

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

August 1July 26 July 27 July 28 July 29 July 30 July 31

>?<?

High Video Resolution

Precipitation

The configuration that you have selected can support only an one day mission. Which is more important, having a two day mission or high quality video

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: The configuration…Start Time: XDuration: Y

Ø

Power = Y=>

[time-on-target = 0400 on 7/30/09 - 0400 on 8/1/09 ] &[Low Output Battery Pack: Can’t provide Y at all] &[High Output Battery Pack: Can provide Y for only 24 hr]

[mission-capabilities: Precision Guidance,HD Video Recon,Non-LOS Retransmission]

User: Having a two day mission is more important

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

August 1July 26 July 27 July 28 July 29 July 30 July 31

>?<?

High Video Resolution

Precipitation

Two day mission is more important

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: High quality…Start Time: XDuration: Y

HD Video Recon Mission Capability << [time-on-target = 0400 on 7/30/09 - 0400 on 8/1/09 ]

System: I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes?

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I can synthesize…Start Time: XDuration: Y

Ø

Precision Guidance, Video Recon,

Non-LOS Retransmission

PAMS

Valid configuration for: Precision Guidance, Video Recon,

Non-LOS Retransmission

User: First tell me why I need non-LOS retransmission.

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

First tell me why I need non-LOS retransmission instead of normal retransmission.

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: First tell me…Start Time: XDuration: Y

Looks at the inferences that led to adding the non-LOS retransmission capability

System: You chose to stream your data back to base, but the mountainous terrain at the target area means that there will be no line of site back to base. This means that you require the non-LOS retransmission capability. Would you like to make any changes?

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

You chose to stream your data back to base, but the mountainous terrain at the target area means that there will be no line of site back to base. This means that you require the non-LOS retransmission capability. Would you like to make any changes?

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: You chose to …Start Time: XDuration: Y

Ø

[mission-target = (X,Y)] => [mission-target-terrain = mountain] => [LOS = false][mission-target = (X,Y)][mission-signal-disposition = streamed]

User: I want to substitute AV Recon for Video Recon

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

I want to substitute AV Recon for Video Recon

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: I want to …Start Time: XDuration: Y

System searches for a capability matching “AV Recon”

System: What is AV Recon?

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

What is AV Recon

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: What is AV ReconStart Time: XDuration: Y

[“AV Recon”] => ?

Ø

User: AV Recon combines Video Recon and Audio Recon.

Video Recon

Precision Guidance

Non-LOS Retransmission

Recon Mission

Video Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

AV Recon combines Video Recon and Audio Recon

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: AV Recon…Start Time: XDuration: Y

Video Recon Mission Capability

Audio Recon Mission Capability=> AV Recon Mission Capability

System: I have added AV Recon.I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes?

AV Recon

Precision Guidance

Non-LOS Retransmission

Audio Signal

Video Signal

AV Recon Signal

Recon Mission

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Precipitation

I have added AV Recon. I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I have added…Start Time: XDuration: Y

Ø

Precision Guidance,Video Recon, Audio Recon,Non-LOS Retransmission

PAMS

Valid configuration for: Precision Guidance, Video Recon, Audio Recon,

Non-LOS Retransmission

AV Recon

User: Yes, add the Image Filtering capability

AV Recon

Precision Guidance

Non-LOS Retransmission

Audio Signal

Video Signal

AV Recon Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Recon Mission

Precipitation

Yes, add the Image Filtering capability

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: Yes, add the Image Filtering…Start Time: XDuration: Y

[mission-capabilities: Precision Guidance, AV Recon,

Non-LOS Retransmission, ]Image Filtering

System: This list of capabilities exceeds the maximum useful memory capacity of 48 MB for the Predator. Please indicate your preference by dragging them onto the line below.

AV Recon Non-LOS RetransmissionImage Filtering Precision Guidance

RequiredOptional

This list of capabilities exceeds the maximum useful memory capacity of 48 MB for the Predator. Change your capability set to use 16 fewer MB of memory.

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: This list of capabilities has an…Start Time: XDuration: Y

Ø

[mission-platform = Predator]

Memory = 60 MB=>

=> Memory = 48 MB

[mission-capabilities: Precision Guidance, AV Recon,

Non-LOS Retransmission, Image Filtering]

User: -user drags icons-

AV Recon

Non-LOS Retransmission

Image Filtering

Precision Guidance

RequiredOptional

ø

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØØ

Image Filtering << Precision Guidance << Non-LOS Retransmission << AV Recon

System: I had to remove the Image Filtering capability.I can synthesize a software system that supplies the following capabilities and I have provided a simulation of each collected signal, would you like to make any changes?

AV Recon

Precision Guidance

Non-LOS Retransmission

Audio Signal

Video Signal

AV Recon Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Recon Mission

Precipitation

I had to remove the Image Filtering capability. I can synthesize a software system that supplies these capabilities and I have provided a simulation of each collected signal, would you like to make any changes

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: I had to remove…Start Time: XDuration: Y

Ø

PAMS

Valid configuration for: Precision Guidance, Video Recon, Audio Recon,

Non-LOS Retransmission

Image Filtering << Precision Guidance << Non-LOS Retransmission << (Video Recon & Audio Recon)

Precision Guidance,Video Recon, Audio Recon,Non-LOS Retransmission,Image Filtering

AV Recon

User: No

AV Recon

Precision Guidance

Non-LOS Retransmission

Audio Signal

Video Signal

AV Recon Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Recon Mission

Precipitation

No

Application

MIDOS Processor

Speech Recognizer Stroke Recognizer

ØText: NoStart Time: XDuration: Y

System: The software system has been synthesized and the platform will be deployedat 0300 on 7/30/09

AV Recon

Precision Guidance

Non-LOS Retransmission

Audio Signal

Video Signal

AV Recon Signal

Signal Transmission

August 1July 26 July 27 July 28 July 29 July 30 July 31

Recon Mission

Precipitation

The software system has been synthesized and the platform will be deployedat 0300 on 7/30/09

Application

MIDOS Processor

Speech Synthesizer Stroke Synthesizer

Text: The software system…Start Time: XDuration: Y

Ø