situation based analysis and control for supporting event-web applications

Situation based analysis and control for supporting Event-web applicationsVivek Singh Advisor: Professor Ramesh Jain

Outline

A. BackgroundB. E2E project

▫ Project overview▫ Situation based control▫ Current status/ example▫ Research challenges

C. Situ-itter▫ Overview▫ Research challenges

D. Research Plan

Event-web

•Events and objects as basic organization and linking mechanism▫Multimodal▫Closer to real world

•Users gain insights and experiences

Events everywhere– (near future)•Events are all around us.

▫Ubiquitous sensors▫Excellent signal processing

techniques▫Wide-spread information broadcast▫Excellent data management

techniques•Large volumes of event data,

streaming in real time.•How can we use it? – machines

don’t understand them.

Motivation: From events to situations…•Given a plethora of event data. How can

we:▫Disambiguate relevant and irrelevant

events?▫Combine events into meaningful

representations ?▫Allow inference and cascading effects▫Support different interpretations based on

application domain▫Support Control & decision making

Event based reasoning

Symbolic inference

Domain semantic

sControl

Situation based control: Motivations

1. Inherent support for event-based (temporal) reasoning

2. The ability of the controller to reason based on symbols (rather than just signals)

3. Explicit inclusion of domain semantics (to support multiple applications)

Related WorkArea Sample

workEvent-based

Symbolic inferenc

e

Explicit Domain

semantics inclusion

Decision

making

Focus area

Situation Awareness

Endsley98 X X Defense/ Tactical

Situation Modeling

Yan06 * X Databases

Situation Management

Jakobson07 * X X Defense/ Tactical

Situational Control

Pospelov86 X Semiotics/ Linguistics

Event detection

Jain03 X Vision/ Multimedia

Knowledge based systems

Sullivan86 X X X Intelligent systems

Discrete Event Control

Ho89 X X Control theory

Situation Calculus

McCarthy69

X X * Logic

Situation based control

This work X X X X Symbolic Control

Applications

•Energy efficient buildings:▫When to switch off air-conditioner?

•Telepresence:▫Which camera feed to send out?

•Business analysis:▫What should be the correct price for iPhone?

•Earthquake rescue effort:▫Where to send out the next fire-fighter

engine?

Outline


▫ Project overview▫ Situation based control ▫ Current status/ example▫ Research challenges


D. Research Plan

Outline




D. Research Plan

E2E communication: Project Overview

Sentient Information

System

Sentient Information

System

Environment 1

Environment 2

Web

Device to Device communication

Towards Environment to Environment (E2E) multimedia communication systems, in Multimedia Tools and Applications Journal, Springer Netherlands, 2009.

Also in: ACM Workshop on Semantic Ambient Media Experiences (SAME), ACM Multimedia workshop, 2008.

Env. 1

Env. 4

JSM 2

JSM 1

Env. 5Env. 3

Env. 2

Joint SM

Shared Visualization Spaces for Environment to Environment Communication , in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.

E2

E

Com

mu

nic

ati

on

Natural interactio

n

Semantic interactio

n

Seamlessinteractio

n

Bi-directional connectivit

y

Not depend on physical similarity

Handle privacy

Event-based architectur

e

Scalable architectur

e

Sensor abstraction

Multimodal information

No fixed application

Live and archived modes

Network/

Transmission

Environment Model

Environment Server

Situation based

controller

Actuator / Presentatio

n Model

MMDB

Sensors

Actuators / Presentation Devices

Physical Environment

EventBase

Environment: Node Architecture

Outline




D. Research Plan

Situation Calculus: Quick overview▫ enter(P1), startWork(P1)▫ enter(P1), exit(P1), enter(P1), startWork(P1),

stopWork(P1), startWork(P1)

- isInRoom(P1, s(k))- isWorking(P1, s(k))

Events

Fluents

isInRoom(P1, s) ˄ ~isWorking(P1, s) → IncreaseMusicVolume() Control

isInRoom(P1, s) 0

isWorking(P1, s)

01

1 Situation

Situation = Not events , nor sequence of events, but their assimilated descriptor

Situation calculus: Basics (1/3)

•Logic formalism designed for representing and reasoning about dynamical domains.

•It builds upon traditional predicate, 1st and 2nd order calculus, but is different because it allows for truth values to change over time.

•Situation:▫“The set of necessary and sufficient world

state descriptors for undertaking control decision”.

Situation Calculus: Basics (2/3)•Ω = {A, S, O, F}

▫Actions (A) for actions i.e. those which change the 'state’ of the world. A= Aex U Asys

▫Situation (S) for `history of events' ,▫Objects (O) as the default sort for everything

else,▫Fluents (F) are predicates reified with situations.

(value assignments which change with time). Relational (give True/False answers) or Functional (return any value as computed)

•Do(action, situation): A X S → S

Situation Calculus: Basics (3/3)•D = Dfnd U Duna U ε U Dap U Dss U D0

▫Dap is a set of action precondition axioms, one per action symbol A.

▫Dss is a set of successor state axioms (SSAs), one for each fluent symbol f, which characterizes all the ways the value of a particular fluent can be changed.Poss(a, s) → [F(x, do(a, s)) ↔ γ+

F(x, a, s) ˅ ((F(x, s)˄ γ-

F (x, a, s))]

▫D0 is a set of axioms describing the initial situation S0.

Control theoretic problem formulation

• Aex(k) = f1(U(k))

• Inp(k) = f2(Aex(k) , Asys(k-1) )

• S(k) = f3(Inp(k) , S(k-1) )

•Asys(k) = f4(S(k), G)•Asys(k) = f4(f3(Inp(k) , S(k-1) ), G)•Y(k) = f5(Asys(k)

Implementing the controller

•Do(A, S) → S’.•S(k) = Do(Inp(k), S(k-1))

•Φ1(X1) →α1 (X1);• ...•ΦN(XN) →αN(XN);

•SGoal |= (~Φ1(X1) ˅α1(X1)) ˄ ... ˄(~ΦN(XN) ˅αN(XN))

• ᴲAsys(k) : Do(Asys(k), S(k)) → SGoal

S(k) = f3(Inp(k) , S(k-1) )Asys(k) = f4(S(k), G)

Implementing the controller

D’ = D U Dca

D’ = Dfnd U Duna U ε U Dap U Dss U D0 U Dca

Situation Based Controller

A. InferenceEngine

C. System Goal

B. Knowledge Base

Situation modeling

1. Identify the relevant Objects (O) , Actions (A) and Fluents (F)

2. Identify the preconditions for each action (Dap)

3. Identify the after-effects of each action (Dss)

4. Describe the initial situation (D0)

5. Identify the goal state using action-condition constraints (Dca)

Outline




D. Research Plan

Loc 1: Desk Loc2: Whiteboard

Loc 3: Engineering

Model

Actions possible:1. Work on PC2. Work on Table

Conditions Actions

Move to location

Activity Selected

Cam

Desired Volume

Desk WorkOnPC

1 1

Desk WorkOnTable

2 2

Whiteboard

- 3 3

Model - 4 4User

Situation modeling: E2E application

Situation based control for cyber physical environments, Accepted: IEEE workshop on situation management, MILCOM, 2009

Step 1: Identify the relevant Objects, Actions and Fluents.

Step 2: Identify the preconditions for each action

Step 3: Identify the after-effects of each action

Step 4: Describe the initial situation

Step 5: Identify the goal-state using the action-condition constraints

Finding the optimal control action

Sample executions

DecreaseVolume, DecreaseVolume, DecreaseVolume, S0

•Exogenous action: MoveToLoc(`Model’) at the end of second cycle

IncreaseVolume, IncreaseVolume, SelectCam(4) MoveToLoc(`Model’), DecreaseVolume, DecreaseVolume, S0

Outline




D. Research Plan

Research Challenge 1: Generic adaptability•Tools to allow system designers to

undertake their domain’s situation modeling •Necessary and sufficient details for handling

application•Discrete, hybrid or continuous•Current status:

▫Dap U Dss U D0 U Dca

•To Do▫Providing easy tools for users to inscribe such

domain knowledge

Research Challenge 2: Enhanced sensing based on feedback •Top down+ bottom up sensing

▫Sensing = F(current_state)•Detect and discard noisy event data.

▫Only allow valid sequences of input events▫Invalid(Seq) ↔(KB U S0 |= ¬Seq)

▫Discard (WearSocks >(T) WearShoes)

•Anomaly detection using these techniques▫Event based (semantic) level not signal

level

Research Challenge 3: Reasoning and analysis•Minimal representation: Find the minimal

set of events Emin which lead the situation changing from S0 to SGoal.

•Handling un-observable systems:▫Can we find the unknown state S0, by looking

at patterns of events and the changes in the system state (fluents) [e.g. in Chess]

•Approach:▫Using planning and projection operators of

situation Calculus

Research Challenge 4: Using Predictive Analysis for control action•Using estimates of future exogenous

actions for better control •Signal based data

▫Kalman Filter▫Model Predictive Control

•Symbolic data▫Semantic Kalman filter?

“Coopetitive multi-camera surveillance using Model Predictive Control”, Machine Vision and Applications Journal, 2008.

Outline




D. Research Plan

Situ-itter: Looking beyond rooms…

•Can an entire city or country be considered a cyber physical system.

•Humans as sensors:▫Everywhere !▫Perception, Censors, Rumors, Delays

•Applications▫Should iPhone price be increased/decreased?▫Detect swine flu in Mexico ->> Issue pork-

import health warnings in Alaska▫DEMO

Research Challenge 5: Scalability of situation based control •Number of Events and conditions to be

considered▫Hierarchical approach

•Supporting multiple applications with different complexity levels▫Creating models for different applications

•Approaches:▫Allow users to define models ▫Learn patterns ▫Use public knowledge/ Ontologies

Outline




D. Summary and Plan ahead

Current status: Systems • E2E project

▫Working prototypes DBH2059, CalIT2

▫Skype based lite-version▫Collaborative nodes

National university of Singapore (Observation System) INRIA, France (emotion enhanced E2E)

• Situ-itter▫Proof-of-concept

• Multimodal observation systems, ACM Multimedia 2008. • ObSys: A Generic Sensing Architecture for Multimodal Observation Systems, Submitted

to TOMCCAP: ACM Transactions on Multimedia Computing, Communications and Applications

• Toward Environment-to-Environment (E2E) Affective Sensitive Communication Systems, submitted to: MTDL workshop, ACM-MM, 2009.

Future work: Systems

•Robust bi-directional E2E communication between UCI, and Singapore

•Implementing situation controller into physical sensors

•Building Twitter crawler/ real-time analysis tool

Area Challenges Status Type of contribution (expected)

Approach

Overall Framework

Temporal + Symbolic reasoning

Prelim. Tools Situation Calculus

Use domain semantics

Prelim. Tools Situation Modeling

Generic & Scalable

Support Multiple applications

Prelim. /Plan

Tools -User tools-Learning -Ontologies

Large number of events

Plan Tools Hierarchical Control

Reasoning and Analysis

Minimal event set Plan Logic-based Min (Seq) : Do(Seq, S0) -> Sgoal

Partial Observability Plan Logic-based S0: Do(Seq, S0) -> Sgoal

Feedback enhanced sensing

Noisy event data , anomalies

Plan Logic-based Invalid (Seq)<-> KB U S0 |= ¬Seq

Top-down + bottom up sensing

Plan Optimality Sensing =F(S_curr)

Predictive Control

Sensor/ device selection

Plan Optimality Symbolic Kalman Filter+ Model Predictive Control

Research Plan

•In progressing order of importance for my work

•Year 3 --Tools▫Finalize overall framework ▫Make it generic and scalable

•Year 4 – Logic based approaches▫Use inference, reasoning and analysis▫Feedback enhanced sensing

•Year 5 – Optimality based contributions▫Predictive Control

Publications• E2E

1. {VKS, HP, IR, RJ}: Towards Environment to Environment (E2E) multimedia communication systems, in Multimedia Tools and Applications Journal, Springer Netherlands, 2009.

2. {VKS, HP, IR, RJ}: Also in: ACM Workshop on Semantic Ambient Media Experiences (SAME), ACM Multimedia workshop, 2008.

3. {VKS, IR, RJ}:User availability detection in E2E systems, in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.

4. {HP, VKS, AM, RJ}: Shared Visualization Spaces for Environment to Environment Communication , in Workshop on Media, Arts, Science and Technology (MAST 09), 2009.

5. {IR, VKS, HP, RJ}: Environment to Environment (E2E) communication systems for collaborative work, Poster in Computer Supported Cooperative Work (CSCW) 2008.

VKS=Vivek Singh, HP=Hamed Pirsiavash, IR=Ish Rishabh, AM=Aditi Majumder, RJ=Ramesh Jain

Publications• Situation based control

1. {VKS, RJ}: Situation based control for cyber physical environments, Accepted: IEEE workshop on situation management, MILCOM, 2009

• With external collaborators1. {MS,VKS, RJ, MK}: Multimodal observation systems, ACM

Multimedia 2008. 2. {MP,VKS, BH,RJ}:“Toward Environment-to-Environment (E2E)

Affective Sensitive Communication Systems”, MTDL workshop, ACM-MM, 2009.

3. {MS,VKS, RJ, MK}: ObSys: A Generic Sensing Architecture for Multimodal Observation Systems, Submitted to TOMCCAP: ACM Transactions on Multimedia Computing, Communications and Applications

4. {VKS, RJ, MK}: Motivating contributors in Social media networks, submitted to: ACM MM workshop on Social media.VKS=Vivek Singh, RJ=Ramesh Jain, MS=Mukesh Saini,

MK=Mohan Kankanhalli, MP=Marco Paleari, BH=Benoit Huet

Publications• Prior work: Master’s thesis

1. “Coopetitive multi-camera surveillance using Model Predictive Control”. Journal of Machine Vision and Applications, 2009.

2. Adversary aware surveillance systems, IEEE TIFS, Trans. Info. Forensics and Security, 2009.

3. “Coopetitive Multimedia Surveillance”, International Conference on Multimedia Modeling (MMM'2007).

4. "Towards adversary aware surveillance systems", IEEE International Conference on Multimedia and Expo, (ICME-2007).

5. A Design Methodology for Selection and Placement of Sensors in Multimedia Surveillance Systems”, ACM Multimedia Workshop on Video Surveillance and Sensor Networks (ACM MM, workshop-VS SN 06)

6. “Coopetitive Visual Surveillance using Model Predictive Control”, (ACM-Multimedia, workshop-VSSN 05)

• Journals (3 accepted, 1 submitted), • Conferences (4), • ACM-MM workshops (5),• Other venues (3)

situation based analysis and control for supporting event-web applications

Technology

project overview environment

situation calculus

research plan

environment communication

aex u asys situation

eventweb events

symboliccontrol control

history of events