multi interpreter architecture overview. of fill-in multi-interpreter architecture overview 2 kb...

Multi Interpreter Architecture Overview

of FILL-IN

Multi-Interpreter Architecture Overview

2

KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames (EF)

Inference• Comparisons• Manipulations• Logical/Computational

Interpretation• Token-symbol• Symbol-type role• Type role-EF role

Current EF Management• Merge from world• Merge from KB

World Inputs/Outputs• Text , Databases , Imagery

of FILL-IN


3

KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames





KB ManagementEvent frame dimensionsISA relationshipsContainment relationshipsAttributesEvent rolesEvent models

of FILL-IN


4




Current EFManagement• Merge from world• Merge from KB

World Inputs/Outputs• Text , Databases , ImageryInterpretation

Token – symbolSymbol - type rolesType roles-phys repsEvent frame - boundariesEvent frame builder

of FILL-IN


5





World Inputs/Outputs• Text , Databases , ImageryInference

Event role classifiersSpatial constraintsAttribute constraintsEvent role processing

of FILL-IN


6



Interpretation• Token-symbol• Symbol-type role• Type role-Event Frame

role

Current Event Frame Management• Merge from world• Merge from KB


Current Event Frame Management

Co-referenceBelief managementSpatio-temporal orderingEvent linking

MI Interpretation subsystem overview

Interpretation Subsystem Overview

Types and Schema

KnowledgeBase

PreprocessingChain

SymbolData

Event FrameBuilder

ResolvedEvent frames

Knowledge Base

Event frame Builder

LC SlotManipulator

SymbolData LC Slot

FillerPhysical Rep

Matcher

Best Physical Representation Fit

ResolvedEvent frames

InputDocuments

Objective: Extract semantics from sequences of word/tokens into typed expressions/event frames.

• Single phase of syntactic and semantic processing

• Very High Level language-coupled ontology

• Novel Language-Independent Logical Form, using LC-Types as the basic building blocks for both event frame classes and

instances.

NLP Pipeline Processing Chain

Sentence

Detector

Doc

umen

t Tex

t

Tokenizer

Sen

tenc

es

Multi-Token

Aggregator

Toke

ns

Annotation Result

Mul

ti-To

kens

POS Tagger

Pen

n Tr

eeba

nk

PO

S T

ags

POS Mapper

Phy

sica

l Rep

P

OS

Tag

sTempora

l Resoluti

onN

atty

Res

ults

Regex Stage(s

)In

fo U

pdat

es

NLPInterpre

t

Par

se

Res

ults

Event frame

Building

Eve

nt fr

ame

Stru

ctur

es

Knowledge Base

Preprocessing

Knowledge Base

LC Event Frame Builder Approach

Goal: Achieve understanding of what is being asserted and the associated actors.

Approach: Use semantic cues and dynamic composability of LC logical schema to resolve relationships between event frames and derive object roles within them.

Semantic cues provide functional relationship between event framesand individual components within them.

Dynamic composability of event frames allows:

• Associations of these relationships with the appropriate components

• Tracking of temporal associations within multiple event frames

Symbol Data

S1 S2 … SN

Event Frame Builder Component Description

LC Slot Manipulator Physical Representation Matcher

LC Slot Filler

InterpretationGraph

Physical RepresentationMappings

KB Terms

KnowledgeBase

KB Term

LC-Type Rules• Fit symbol into L/C

slot of existing or new Event frame

Template Matching• Use grammatical

cues to switch between event frames

Finite State Machine Processing

012

01

Phys Rep 1

Phys Rep 2

S1

S2

S3

S4

KeysElement Symbol

Example of MI processing a sentence from one of the

motivating scenario sentences

Event Frame Builder Example(Initial Event Frame Boundaries)

“021715zmay14 imagery shows one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w.”

Found three LC-type expressions (i.e., event frames):

1) First-Order Expression: “0211715zmay 14 imagery shows”

• Captures an assertion about the “image”, (i.e., “image” is the Locator, “shows” is the associated Process captured as Content)

2) Second-Order Expression: “one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w”

• Captures association between two expressions, (i.e., a message described by a second-order expression is associated with the first-order expression). This composability enables further inferencing: What did imagery show?

3) Nested Expression: “loaded”

• Captures the content describing the locator “missiles”. This composability enables inferencing on weapon readiness.


Event Frame Builder Example(Physical Representation)

Each expression (i.e., event frame) is described by a physical representation Within each expression, semantic cues, each described by its own physical representation, provide functional relationship between

expressions and individual components within expressions.• “021715zmay14” is a temporal component indicating the time of the assertion• “shows” is a second-order marker signifying functional relationship between two expressions• “with” and “ivo” are semantic cues indicating functional connection between individual components within the same

expression.

First-Order Event frame Second-Order Event frame Nested Event frame


Event Frame Builder Example

An event frame Structure EF is graph representation of NLP results enabling reasoning and inferencing over data in time and space.• Logical Schemas associated with physical representations, provided by the NLP results, are used to populate the EF• Dynamic composability of LC logical schema allows tracking of temporal associations within multiple expressions.• Events are represented by a number of slots enabling tracking of associated attributes, time and space per expression basis.

The start and end of theevent is the same

The space of theevent is unknown No other ACTED_UPON

at completion of event

ACTOR Object or Event filling the role of ACTOR in an assertion. S Space in which event occurred

ACTION Action identifying the event S0, S1 Space at the beginning and end of event

ACTED_UPON Object or Event filling the role of ACTED_UPON in an assertion T Time during which event occurred

ACTED_UPON_T1 Object or Event filling the role of ACTED_UPON at completion of event T0, T1 Time at the beginning and end of event


Event Frame Builder Example continued

“show” is a transformation event and a semantic cue for a second-order expression. The assertion fills in the “Message” slot for the “show” event.

The geolocation is realized into latitude/longitude and captured as space in the assertion.

“with” is a semantic cue for containment. There is a containment relationship between “SA-6 tel” and “three missiles”.

Example of ontological ISA information stored

in the KB

Inference using the knowledge base

18

An SA-6 is an inherited type from ground-to-air

threat

Example of user display

ReCOURSE planning alert interface

20

Detailed information

Selected JIPTL target/mission

Alerts generated against JIPTL

Drill down into alert shows source documents

21

Examples of uniformity of representations using abstract

dimensional underpinnings

of 110

Dimension labels vs. Values vs. operators vs. meta-operators

Color.green = Get_color(Object.ball) Integer.9 = Sum(Int.7, Int.2) Profit{$}.1000 = Avg((Obj.Shoes, Profit) , (Obj.Socks, Profit)) Ops_assess_score.0.9 = Assess(Plan_id.72, Phase.post_exe) Author_name.Scott = Get_author_name(Book.Waverly) Space.SW = Get_Area_of_Operations(Plan_spec.Alpha) Relevance metric.Village Sentiment =

Get_Worst performance(Relevance Metric(Plan_spec.Alpha))

Anatomy of a dimensional expression

23

Examples of logical markers

of 110

1. Primary locator (e.g., argument) markers1. Grammatical2. Locations 3. Objects 4. Attributes5. Processes

2. Primary content (e.g., predicate) markers1. Auxiliary verbs2. Processes

3. Nested expression markers4. Object role in process markers5. Function role component markers6. Second order expression markers7. Inter-expression expression markers

Expression Slot Manipulation Markers

25

26

Markerso The, A, An when found at the beginning of a sentence, or at the beginning of a clause

whose prior clause ended with a semi colon, indicate that the following substantive types are locators -and include the primary locator- until a new marker is found. Note that content markers are shown in green. Underlined text represents what is assigned the locator role. Purple text represents nested expression markers

The ball is green The old man who lives by the sea went diving A strange smell began wafting over the arena A human skull believed to be over 6000 years old was found frozen under the

ice An apple a day keeps the doctor away An apparition, like none he’d seen before was right there staring in his eyes.

of 110

Nested Expression Markers

27

Nested expression markers: the nested expression serves as a locator o Who, that, when found following one or more substantive types that followed a locator

marker and before the appearance of a content marker indicate that the following substantive types are secondary locators (i.e. they restrict the range of or serve as contents relative to the primary locator found before the marker) until the next schema role marker is found

The old man who lives by the sea is wise The person who stole my pen is in big trouble Those kids who were throwing rocks at the police were later arrested The city that never sleeps is hosting this year’s conference for persons that can’t

stay awake. The fighter that would never quit finally did Those clandestine organizations that support hacking are finally coming under

scrutiny The old car that rattled was last seen speeding down the interstate Bob, that rascal, did it again

Bungee jumping, that crazy activity, is finally gaining in popularity

Visual examples of composite event

frames

2 objects, 1 process: Asymmetric, non-transitive, non-touch to touch,

Event frame – pre execution

Event frame post execution

Event frame during execution

Obj1 Obj2

Obj1 Obj2

Obj1 Obj2

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre kissed maryPierre called maryThe tree fell on the carThe book fell on the floorPierre ran to the waterPierre stood on the ladder

Process.n

Process.n

Process.n

2 objects, 1 process: Symmetric, non-transitive, non-touch to touch,

event frame – pre execution

Event framepost execution

Event Frameduring execution

Obj1 Obj2

Obj1 Obj2

Obj1 Obj2

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre danced with MaryPierre and Mary met at the cafe

Process.n

Process.n

Process.n

3 objects, 1 Process: Symmetric, non-transitive, non-touch to touch,

Event frame– pre execution

event frame post execution

event frame during execution

Obj1 Obj2

Obj1 Obj2

Obj1 Obj2

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre and Mary shared the last cookie Pierre and Mary had a conversation

about mathPierre and Mary bought a house togetherPierre and Mary were having tea

in the treehousePierre and Mary shared the only blanket

Process.n

Process.n

Process.n

Obj3

Obj3

Obj3

3 objects where the active object 1.1 is a part of object 1, AND object 1.1 remains a part of object one during the course of executing the event frame

1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = argument and the rest = predicate




Obj3

Obj3

Obj3

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre kissed mary with his lipsPierre slapped the officer with his handPierre pounded the door with his fistThe tree trunk fell on the roof The book hit the floor with its spinePierre put his foot in the waterPierre gripped the ladder with his toes

Process.n

Process.n

Process.n

Obj1Obj1.1

Obj1Obj1.1

Obj1Obj1.1

3 objects where the active object 1.1 is associated with object 1 pre-execution, AND object 1.1 disassociates from object one during the course of executing the event frame

1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = locator and the rest = content




Obj3

Obj3

Obj3

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre gave his shirt to the victimPierre used his shirt to trap the mouseThe huge branch broke off [the tree] and fell on the roof The book jacket blew off and hit the floor The plane took off from the carrier to look for survivorsPierre’s hat fell off and hit the ground.

Process.n

Process.n

Process.n

Obj1Obj1.1

Obj1Obj1.1

Obj1Obj1.1

3 objects where object 1 is touching an independent object 2 in such a way that object 2 then touches object 3.





Obj3

Obj3

Obj3

T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre surrendered his gun to the officer Pierre gave the book to maryPierre handed the papers to the undercover copPierre sold the car to his neighborPierre gave the sandwich to his brother

Process.n

Process.n

Process.n

Obj1 Obj2

Obj1Obj2

Obj1Obj2

4 objects where object 1 is touching an independent object 2 in such a way that object 2 then touches a part (object 3.1) of object 3.





T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre put the gun in the officer’s hand Pierre placed the book on mary’s headPierre landed his plane on the deck of the carrierPierre threw the peanuts in to the elephant’s mouth

Process.n

Process.n

Process.n

Obj1 Obj3Obj3.1

Obj1Obj2

Obj1Obj2

Obj3Obj3.1

Obj3Obj3.1

Obj2

Obj2

5 objects where obj 1.1, a part of object 1, is touching an independent object 2 in such a way that object 2 then touches a part (object 3.1) of object 3.





T.Pre , S.n

T.during , S.n

T.post , S.n

Pierre used his elbows to put the gun in the officer’s hand Using only his pinkies, Pierre placed the book on mary’s headPierre landed feet first on the deck of the carrierPierre used his good hand to throw

peanuts in to the elephant’s mouth

Process.n

Process.n

Process.n

Obj1 Obj3Obj3.1

Obj1Obj2

Obj1Obj2

Obj3Obj3.1

Obj3Obj3.1

Obj1.1

Obj1.1

Obj1.1