situation recognition from multimodal data tutorial (icme2016)
TRANSCRIPT
125
SITUATION RECOGNITION FROM MULTIMODAL DATA
Vivek K Singh1 Siripen Pongpaichet2 and Ramesh Jain2
1Rutgers University 2University of California Irvine
125
Todayrsquos slides
2
httpwwwspringercomusbook9783319305356Or email us for a softcopy
httpbitly29JL30M
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Todayrsquos slides
2
httpwwwspringercomusbook9783319305356Or email us for a softcopy
httpbitly29JL30M
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Course Outline1) Concept recognition from Multimedia data (20 mins Ramesh Jain)bull Trends bull Why situation recognition is different from object event scene recognition etc2) Situation recognition across multiple research domains (20 mins Vivek Singh)bull Situation Algebra Situation Calculus Robotics hellip3) Situation recognition (45 mins Vivek Singh) bull Situation modeling bull Situation operators 4) Designing situation based applications (30 mins Siripen Pongpaichet)bull Motivation and essential requirementsbull Application scenarios Thailand flood hurricane Sandy city sensing asthma relief5) Future trends and open problems (20 mins Ramesh Jain)bull Future trends bull Open problems for Multimedia research 6) Question and Answers (15 mins)
3
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
CONCEPT RECOGNITION FROM MULTIMEDIA DATA(20 mins Ramesh Jain)
4
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Introduction
bull Object event scene recognition bull Trends bull Situation recognition ( is different from object event
scene recognition etc)
5
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
1256
Data Information Knowledge Wisdom
Data is Essential But we are really interested in products
Information Knowledge and Wisdom
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
1257
What is Important in lsquoBig Datarsquo
Multimedia
Realtime Uncertainty
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
1258
The Grand Challenge
Sense making from multimodal massive geo-social data-
streams
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Fundamental Problem
Connecting People to Resources effectively efficiently and promptly
in given situations
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12510
What is Cyber Space
Who invented it
Theory of Control and Communication in
Animals
Machines
Societies
Published first in 1942
Back to Future
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12511
Cybernetics 101bullDesired state (Goal)bullSystem model and Control Signal (Actions)
bullCurrent State (using multimedia data)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12512
In Smart Systems Feedback is the Key
InputComputed
using System Model
FeedbackOutput compared with
desired goal
Actual System
OutputObserved
Continuously
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Social Networks
Connecting People
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12505032023 14
Connecting People And
Resources
Social Life Networks
Aggregation and
Composition
Situation Detection
Alerts
Queries
Information
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12515
Traditional Social SystemsbullModels of Systems were difficult to form
bullCurrent State of the system could not be determined
bullReal time lsquoactionsrsquo could not be implemented
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12516
Emerging Social Systems
bullSocial models can be determined using warehouses of Big Data
bullSocial observations are now possible with little latency
bullActions could be targeted to precise sources
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12517 EventShop Global Situation Detection
Predictive Situation
Recognition
Evolving Global Situation
Predictive Personal Situation
Recognition
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
PersonaDatabase
Resources
Needs
Data Ingestio
n
Wearable Sensors
Calendar
Locationhellip
Dat
a So
urce
s
hellip
Data Ingestion
and aggregatio
n
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12518
Concept Recognition Last Century
Environments
Real world Objects
Situations
Activities
Single M
edia
SPACETIME
ScenesLocation aware
Visual Objects
Trajectories
Visual Events
Location unaware
Static Dynamic
Location aware
Location unaware
Static Dynamic
Data = Text or Images or Video
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Visual Concept Recognition First research papers
bull 1963 Object Recognition [Lawrence + Roberts]bull 1967 Scene Analysis [Guzman]bull 1984 Trajectory detection [Ed Chang+ Kurz]bull 1986 Event Recognition [Haynes + Jain]bull 1988 Situation Recognition [Dickmanns]
1960 1970 1980 1990 2000 2010
Object SceneTrajectory
Event
Situation
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12520
Concept Recognition This Century
Environments
Real world Objects
Situations
Activities
SPACETIME
REAL-
WORLD
Location aware
Location unaware
Static Dynamic
Heterogeneous M
edia
Location aware
Location unaware
Static Dynamic
Data is just DataMedium and sources do not matter
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12521
Concept recognition from multimedia data
SPACETIME
REAL-
WORLD
ScenesLocation aware
Visual Objects
Situations
Visual Events
Location unaware
Static Dynamic
Heterogeneous M
edia
Single M
edia360 K 114K
34 KLocation aware
Location unaware
Static Dynamic
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Situation Recognition Next Frontierbull Data Abundance
bull Big Opportunity for Multimediabull Timebull Spacebull Multiple diverse data streams
bull Need new frameworks
22
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS
(20 mins Vivek Singh)
23
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12524
Related Work Data to SituationsArea Combine
hetero data
Human sensors
Data analytics
Define situations
Location aware
Real-time streams
Toolkits
Situation Awareness
X X o o X
Situation Calculus
X
Web data mining
o X X o X
Social media mining
o X X o X
Multimedia Event detection
X o o o
Complex event processingActive DB
X X o X
GIS X o X X o
Mashup toolkits(Y pipes ifttt)
X X o X X
X
X X
X
X
X X
X
X
X
X
XThis work X X X X X X X
XX
o = partial support
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Defining Situations Situation Calculus
bull A situation s is the complete state of the universe at an instant of time ndashMcCarthy 1968
bull Snapshot of the world at a given time- Reiter 1991bull The set of necessary and sufficient world state descriptors
for undertaking control decision ndash Singh amp Jain 2009
25
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Situation Calculus Quick overviewbull enter(P1) startWork(P1)bull enter(P1) exit(P1) enter(P1) startWork(P1)
stopWork(P1) startWork(P1)- isInRoom(P1 s(k))- isWorking(P1 s(k))
EventsFluents
isInRoom(P1 s) ˄ ~isWorking(P1 s) rarr IncreaseMusicVolume() Control
isInRoom(P1 s) 0
isWorking(P1 s) 01
1 Situation
Situation = Not events nor sequence of events but their assimilated descriptor
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Problems with this approachbull Scalability
bull Listing all the rules bull Frame problem - Specifying the non-effects
bull Assumes 100 confidence in events detectedbull Space is not a first class citizenbull Does not deal well with heterogeneous data
27
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Situationsbull Multiple definitions
bull Situation awarenessbull Situation modelingbull Situation detection bull Situation calculusbull Context based computing
ldquothe perception of elements in the environment within a volume of time and space the comprehension of their meaning and the projection of their status in the near future (Endsley 1988)rdquo
ldquoknowing what is going on so you can figure out what to dordquo (Adam 1993)rdquoldquothe complete state of the universe at an instant of timerdquo (McCarthy 1969)
ldquoa set of past contexts andor actions of individual devices relevant to future device actionsrdquo rdquo (Wang2004)rdquo
ldquohellipextensive information about the environment to be collected from all sensors independent of their interface technology Data is transformed into abstract symbols A combination of symbols leads to representation of current situationshellipwhich can be detectedrdquo(Dietrich 2003)
ldquoA situation is a set of contexts in the application over a period of time thataffects future system behaviorrdquo (Yau 2006)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12529
Situations commonalitiesbull Goal Basedbull Space-Timebull Future Actions
bull Abstractionbull Computationally
Grounded
Work Goal Based Space-Time Future Actions Abstraction Computationally
GroundedMcCarthy 1968 X Barwise 1971 X X Endsley 1988 X X X XSarter 1991 o X Adam 1993 X X Dominguez1994 X X X XSmith 1995 X o X X Steinberg 1999 X X X oJeannot 2003 X Moray 2004 o X Dietrich 2004 X XYau 2006 X X XDostal 2007 o X Singh 2009 X X XMerriam-Webster (accessed 2012) o
This work (aim) X X X X Xo = Partial support
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12530
Situation Definition
bull Situation An actionable abstraction of observed spatio-temporal characteristicsbull eg flu epidemic severe asthma threat road
congestion wildfire flash-mob
Goal Based Space-Time Future Actions Abstraction Computationally
Grounded
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Overall Framework Motivating example
31
STT data
TweetlsquoUrrghhellip sinusrsquo
Loc NYCDate 3rd Jun 2011
Theme Allergy
Situation Detection User-Feedback
lsquoPlease visit nearest CDC center at 4th St
immediatelyrsquo
Date 3rd Jun 2011
Aggregation
1) Classification2) Control action
Operations
Alert level = High
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12532
Applicationsbull Healthcare
bull Alert me if there is a flu epidemic in my areabull Telepresence
bull Which camera feed to send outbull Business analysis
bull Where is the most suitable place to open a new lsquoiphonersquo store bull Weather
bull Alert me when the fall colors blossom in New England bull Daily living
bull Which place (and at what time) is conducive for exercisingbull Weather climate politics traffic hellip
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
SITUATION RECOGNITION FRAMEWORK
(45 mins Vivek Singh)
33
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
A) Situation Modeling
B) Situation Recognition
C) Visualization Personalization and Alerts
hellip hellip
STT Stream
Emage
Situation
hellipC1
v2 v3
v5 v6
prod
Δ
i) Visualization
ii) Personalization
+
+Available resources
iii) Alerts
Personal context
Personalized
situation
Overall framework 34
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12535
A) Situation Modelingbull Help domain experts externalize their internal models of situations of interest eg epidemic
bull Building blocks bull Operators bull Operands
bull Wizard bull A prescriptive approach for modeling situations using
the operators and operands
Singh Gao Jain Situation recognition An evolving problem for heterogeneous dynamic big multimedia data ACM Multimedia lsquo12
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12536
Growth rate (Flu reports) Feature
Thresholds (0 50)
Data source
Meta-data
-Emage (Reports)
Representation level
Twitter-Flu
Building Blocks Operandsbull Knowledge or data driven building blocks
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Building Blocks Operators
Δ Transform hellipSpatio-temporal
window
37
Aggregate +
Classification Classification method
Characterization Growth Rate = 125
Property required
Pattern Matching
72+
Pattern
prod Select +Mask
Φ Learn Learning method
Features
Situation
f f
1) Data into right representation
2) Analyze data to derive features
3) Use features to evaluate situations
Supporting parameter(s)
Data OutputOperator Type
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12538
Situation ModelingGet_components (Situation v)1) Identify output state space2) Identify S-T bounds3) Define component features
v=f(v1 hellip vk)bull If (type = imprecise)
bull identify learning data source method
4) ForEach (feature vi) If (atomic)
bull Identify Data source bull Type URL ST bounds
bull Identify highest Rep level reqdbull Identify operations
Else Get_components(vi)
vf1
v4
v2 v3
D1
Emage
Δ
D2
prod
Emage
Δ
D3
Δ
Emage D2
prod
Emage
Δ
f2
v5 v6
ltUSA 5 mins001x 001gt
ϵ Low Mid High
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Epidemic Outbreaks
Unusual Activity Growth Rate
Current activity level
Historical activity level
Emage (reports ILI)
Δ
Twitter-Flu
TwittercomltUSA 5 mins
001x 001gt
Emage (Historical avg)
Δ
Twitter-Avg
DB ltUSA 5 mins
001x 001gt
Δ
Twitter-Flu
Emage (reports ILI)
TwittercomltUSA 5 mins
001x 001gt
ϵ Low mid highltUSA 5 mins 001x
001gt
Growing Unusual activity
1)Model
Emage (reports ILI)
Δ
Twitter-Flu
Emage (population)
Δ
CSV-Population
π
TwittercomltUSA 5 mins
001x 001gt
Censusgov ltUSA 5 mins
001x 001gt
2) Revise
Subtract
Subtract
Multiply
Classification Thresh (3070)
Normalize[0100]
3) Instantiate39
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12540
Level 1 Unified representation
(STT Data)
Level 3 Symbolic rep (Situations)
Properties
Properties
Properties
Level 0 Raw data streams eg tweets cameras traffic weather hellip
Level 2 Aggregation
(Emage)
hellip hellip hellip
STT Stream
Emage
Situation
B) Situation evaluation Workflow
Operations
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Data Representationbull E-mage
bull Visualizationbull Spatio temporal data representationbull Data analysis using media processing operators
(eg segmentation background subtraction convolution)
41
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Data Modelbull Spatio-temporal element
bull stel = [s-t-coord theme(s) value(s) pointer(s)]bull E-mage
bull g = (x (tm v(x))|xϵ X = R2 tm ϵ θ and v(x) ϵ V = N)bull Temporal E-mage Set
bull TES= (t1 g1) (tn gn)bull Temporal Pixel Set
bull TPS = (t1 p1) (tn pn)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12543
Situation Recognition Algebra
Singh Gao Jain Social Pixels Genesis and Evaluation ACM Multimedia lsquo10
S No Operator Input Output
1 Filter prod Temporal E-mage Stream Temporal E-mage Stream
2 Aggregation KTemporal E-mage Stream Temporal E-mage Stream
3 Classification Temporal E-mage Stream Temporal E-mage Stream
4 Characterization Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
5 Pattern Matching Spatial Temporal
Temporal E-mage Stream Temporal Pixel Stream
Temporal Pixel Stream Temporal Pixel Stream
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Media processing engine
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Implementation and resultsbull Twitter feeds
bull Geo-coding user home locationbull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo stream (since Jun 2009)
and the higher rate lsquoGardenhosersquo stream since Nov 2009bull Flickr feeds
bull API bull Tags RGB values from gt800K images
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Testing Data Representation + Algebra
bull Applications bull Business analytics bull Political event analyticsbull Seasonal characteristics
bull Data bull Twitter feeds archive
bull Loops of location based queries for different termsbull Over 100 million tweets using lsquoSpritzerrsquo lsquoGardenhosersquo APIs
bull Flickr feedsbull API Tags RGB values from gt800K images
bull Implementation bull Matlab + Java + Python
46
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12547
Sample Queriesbull Select E-mages of USA for theme lsquoObamarsquo
bull prodspatial(region=[24-125][24-65]) (TEStheme=Obama)
bull Identify three clusters for each E-mage abovebull kmeans(3) (prodspatial(region=[24-125][24-65])(TEStheme=Obama))
bull Show me the cluster with most interest in lsquoObamarsquobull prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65]) (TEStheme=Obama)))
bull Show me the speed for high interest cluster in lsquoKatrinarsquo emagesbull speed(epicenter(prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina)))))
bull How similar is pattern above to lsquoexponential increasersquobull exp-increase(speed(epicenter (prodvalue(v=1) (kmeans(n=3) (prodspatial(region=[24-125][24-65])
(TEStheme=Katrina))))
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12548 ATampT retail locations
ATampT total catchment area
iPhone theme based e-mageJun 2
Aggregate interest
Under-served interest areas
-Subtract
DecisionBest Location is at
Geocode [39 -122] just north of
Bay Area CA
SpatialMax
ltgeonamegtltnamegtCollege Cityltnamegtltlatgt390057303ltlatgtltlnggt-1220094129ltlnggtltgeonameIdgt5338600ltgeonameIdgtltcountryCodegtUSltcountryCodegtltcountryNamegtUnited StatesltcountryNamegtltfclgtPltfclgtltfcodegtPPLltfcodegtltfclNamegtcity villageltfclNamegtltfcodeNamegtpopulated placeltfcodeNamegtltpopulationgtltdistancegt10332ltdistancegtltgeonamegt
+ Add
to Jun 15 2009
Convolution
Store
catchment area
Convolution
Store catchment
area
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Flickr Social Emagesbull Jan ndash Dec 2009
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12550
Seasonal characteristics analysisbull Fall colors in New England
bull Show me the difference between red and green colors for New England region as it varies throughout the year
bull subtract(spatial(sum)(πspatial(R=[(40-76) (44-71)]) (TEStheme=Red)) spatial(sum)
(πspatial(R=[(40-76) (44-71)])(TEStheme=Green)))
Jan
0
Dec
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Year average Peak of green
At [35 -84] at the junction of Chattahoochee National Forest Nantahala National Forest Cherokee National Forest and Great Smoky Mountains National Park
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
OTHER APPROACHES
52
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics
Marco Balduini Emanuele Della ValleISWC 2015 ndash Data Sets and Ontologies
Slides adapted from httpwwwslidesharenetMarcoBalduinifrappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Proposed Approach54
03052023
Re-use consolidated concepts from bull geo-spatial vocabulariesbull time related vocabulariesbull provenance vocabularies
Model visual analytics concepts bull pixelsbull frame
Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users
Singh VK Gao M Jain R Social pixels genesis and evaluation (ACM MM 2010)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually55
03052023
Time
Reality
Capture
Frame
Digital Reflex
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually56
03052023
Grid
Cell
Time
Frame
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually57
03052023
Pixel Frame 1
Time
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually58
03052023
Place A
Event A
Time
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually59
03052023
Event A
Time
Frame 1
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
FraPPE visually60
03052023
Event B
Place B
Time
Frame 2
Frame 1
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12561
City Sensing listens to the pulse of Milano Design Week on April 9th 2014
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12562
Tweeting Cameras
Slides courtesy Yuhui Wang Francesco Gelli and Mohan Kankanhalli
Adapted from ldquoTweeting Cameras for Event Detectionrdquo in Proc WWW Conference 2013
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Physical amp Social Sensors Fusion For Situation Awareness
PhysicalSensors
SocialSensors
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Real-world Events
Hispanic ParadeCBGB Musical Festival
Columbus Day Parade
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
twtw
tw tw
tw tw
Historic TweetsRecent Tweets
Event time and location
Retrieve recent tweets Retrieve historic tweets
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
12565
Probabilistic Spatio-Temporal Databull Definition (PST Probabilistic Spatio-Temporal Data) The
fundamental building block for low-level concept representation is the probabilistic spatio-temporal element ldquopstrdquo
bull pst = [loc temp label prob pointer] (1)wherebull loc = [lat lon] represents the geo-location ndash latitude and longitude ndash of the camera location bull temp stores the time information of captured databull label represents semantic concept such as car humancrowd parade etc detected in the stream Generally these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiersbull prob is the confidence value in [01] representing the output of a concept detector as a probability value
bull pointer points to actual raw data stream
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Physical Sensors (Concept ldquoCrowdrdquo)
125
Social Sensors (MillionMarchNYC BlackLivesMatter)
125
Fused Information
125
CMage1
09
08
07
06
05
04
03
02
01
0
Gaussian Process based Prediction
Sensor Image Patch
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
Fusing Sensor Cmage with Social Cmage
Sensor Cmage(concept ldquopeople marchingrdquo)
Social Cmage(concept ldquoMillionsMarchNYCrdquo)
Fused Cmage
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
DESIGNING SITUATION BASED APPLICATIONS(30 mins Siripen Pongpaichet)
71
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
Outlinebull EventShop System Requirementsbull EventShop System Architecturebull Demobull Building Applications using EventShopbull Conclusion
72
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
EventShop Requirement
73
Granulari-ties
Heterogeneous
Model Prediction
UsersOpen-Source
Storage
Generic
Streams
Support fast data flow
Handle heterogeneous types of data streamsEfficiently aggregate data at different granularities
Provide storage system to archive both data input and system outputCreate situation model and provide actionable information
Generic computational platform for situation recognitionOpen-source softwareUser friendly and interactive interface
Contain predictive component
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
EventShop Architecture74
AlertOutputData Ingestor
Data Source Parser
Data Adapter
Emage Generator
(+resolution mapper)
Query Processing
EventShop Storage
Query Parser
Query Rewriter
Emage Stream Processing
Action Parser
Situation
Emage
Visualization (Dashboard)
Actuator Communication
Event Property amp Other Information
(eg spatio-temporal pattern)
ᴨ
ᴨmicro
Data Access Manager
Live StreamArchived Stream
Situation Stream
Physical Data Source (eg sensor
streams geo-image streams)
Logical Data Source (eg preprocessing data streams social
media streams)
Raw Event
REST API ServicesData Source Query Alerts STT-Emage
EventShop UI External AppsVisual Analytics
Inte
rface
AP
IP
roce
ssin
g La
yer
Sto
rage
Lay
er
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
EventShop UI
75
Save Query Reset Query
Create Query
Pollen
Tweets_Asthma
Available
Available
350
AQI Available 357
361
Grouping Stopped 35
Asthma_ Risk
Stopped 36
Asthma_ Interpolate
Stopped 37
Asthma_ Interpolate
Stopped 37
Asthma_ Stopped 37
Query Graph
Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char
Redraw
ds361
ds350
ds357
F1 Q1
F2
F3
Q2
Q3
A1 Q4 G1 Q5
httpeventshopicsuciedu8080eventshoplinux
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
Demo
76
httpswwwyoutubecomwatchv=E5unHXZmSr8
125
Demo
77
httpswwwyoutubecomwatchv=IwJYEZd8Bbg
125
Building applications using EventShopSNo Application Data Used Application
deployed Scale Data modalities Operators used
1 Wildfire detection in California Real Yes Macro Satellite data
Google insights F A Ch
2 Hurricane monitoring Simulated No Macro na F A Ch P
3 Flu epidemic surveillance Real No Macro Twitter Census F A C
4 Allergy recommendation Real Yes Macro Twitter Air Quality Pollen Count F A C
5 Asthma management Real Yes Macro Personalized alerts
In situ sensors Satellite data
Asthma Tracking F I Pr
6 Thailand flood mitigation Real Yes Macro Personalized alerts KML F A C
7 Photos as Micro-Reports Real Yes Macro Flickr F Cl
8 Trash management Real amp Simulated In progress Macro Trash sensors
micro-reports F A Pr
LegendF = Filter A = AggregateC = Classification Ch = CharacterizationI = Interpolation Cl = ClusterP = Pattern MatchingPr = Prediction
78
125
Asthma ManagementApplication
79
125
Asthma Management Application
80
(1) Macro
Situation
Macro Data Streams
(3) Situation-
Action Rules
Sensor streams
Social media
Geo-temporal data
Personal Data Streams
(2) Personal Situation
Behavioral streams
Profile + Preferences
125
Asthma Risk Estimation
81
Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration
visualize data on Feb 12th 2008
Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015
Spectral Spatial Gaussian Process (SSGP)
125
Experiment Results
82
Data Model PMSE MAPE
SingleData Source
CMAQ - 10619 272873CMAQ LR 09586 271077
Stations Kriging 09077 229672CMAQ SSGP 03468 142727
MultipleData Sources
ALL SGP 03006 135109ALL SSGP 02858 131087
CMAQ Kriging SSGP
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Asthma ManagementApplication
79
125
Asthma Management Application
80
(1) Macro
Situation
Macro Data Streams
(3) Situation-
Action Rules
Sensor streams
Social media
Geo-temporal data
Personal Data Streams
(2) Personal Situation
Behavioral streams
Profile + Preferences
125
Asthma Risk Estimation
81
Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration
visualize data on Feb 12th 2008
Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015
Spectral Spatial Gaussian Process (SSGP)
125
Experiment Results
82
Data Model PMSE MAPE
SingleData Source
CMAQ - 10619 272873CMAQ LR 09586 271077
Stations Kriging 09077 229672CMAQ SSGP 03468 142727
MultipleData Sources
ALL SGP 03006 135109ALL SSGP 02858 131087
CMAQ Kriging SSGP
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Asthma Management Application
80
(1) Macro
Situation
Macro Data Streams
(3) Situation-
Action Rules
Sensor streams
Social media
Geo-temporal data
Personal Data Streams
(2) Personal Situation
Behavioral streams
Profile + Preferences
125
Asthma Risk Estimation
81
Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration
visualize data on Feb 12th 2008
Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015
Spectral Spatial Gaussian Process (SSGP)
125
Experiment Results
82
Data Model PMSE MAPE
SingleData Source
CMAQ - 10619 272873CMAQ LR 09586 271077
Stations Kriging 09077 229672CMAQ SSGP 03468 142727
MultipleData Sources
ALL SGP 03006 135109ALL SSGP 02858 131087
CMAQ Kriging SSGP
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Asthma Risk Estimation
81
Traffic Flow Aerosol Concentration PM25 CMAQ Model PM25 Concentration
visualize data on Feb 12th 2008
Mengfan Tang Pranav Agrawal Siripen Pongpaichet Ramesh Jain Geospatial interpolation analytics for data streams in EventShop ICME 2015
Spectral Spatial Gaussian Process (SSGP)
125
Experiment Results
82
Data Model PMSE MAPE
SingleData Source
CMAQ - 10619 272873CMAQ LR 09586 271077
Stations Kriging 09077 229672CMAQ SSGP 03468 142727
MultipleData Sources
ALL SGP 03006 135109ALL SSGP 02858 131087
CMAQ Kriging SSGP
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Experiment Results
82
Data Model PMSE MAPE
SingleData Source
CMAQ - 10619 272873CMAQ LR 09586 271077
Stations Kriging 09077 229672CMAQ SSGP 03468 142727
MultipleData Sources
ALL SGP 03006 135109ALL SSGP 02858 131087
CMAQ Kriging SSGP
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Asthma Risk Estimator Model and Result
83
Asthma HospitalizationGround Truth
FILTERLOC=CA
FILTERLOC=CA
AGGFUNC=AVG
GROUPTHRESHOLD
Asthma Risk Area without Interpolation
GROUPTHRESHOLD
Asthma Risk Area with interpolation
AGGFUNC=AVG
PM25 ConcentrationFrom Stations
InterpolatedPM25 using SSGP
Pollen
OzoneAQI
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK
Mengfan Tang1 Pranav Agrawal1 Feiping Nie2 Siripen Pongpaichet1 Ramesh Jain1
1University of California Irvine USA2Northwestern Polytechnical University China
Tuesday July 12th 2016 at 5PM Room Cascade I Special Session Multimedia Cloud Computing and Big Data
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Detecting Situations from Micro-Reports
85
Photos
Reports Events
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
PHOTOS as Kodak Moments
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Disruption PHOTOS as Information
Smartphone camera captures
EVENTS
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
bull Truthfulness bull Accuracy bull Objectivity bull Fairness and Public accountability
Reports of Events from Journalists
Seek Truth and Report it as Fully as Possible
125
Reports of Events from Citizens
125
Reports of Events from Citizens
125
FASTSubjective
EASY Noisy
LATESTAmbiguous
were SO yesterdayMicro-Blogs
Multimedia Micro-Reports
125
Compelling Universal
Objective Spontaneous
Multimedia Micro-Reports (MMRs) are now and future
125
Capturing and Reporting events using multimedia such as photos videos sensors and texts
Converting multimedia data to multimedia micro-reports using MediaJSON
Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics
Emerging opportunities for numerous apps smart city public health emergency rescue
What are the challenges
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
What ObjectsWho PeopleWhen EventsWhere Location
Why IntentEmotions
How Photo and audio
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Compelling Universal
Objective Spontaneous
Multimedia Micro-Reports (MMRs) are now and future
125
Capturing and Reporting events using multimedia such as photos videos sensors and texts
Converting multimedia data to multimedia micro-reports using MediaJSON
Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics
Emerging opportunities for numerous apps smart city public health emergency rescue
What are the challenges
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
What ObjectsWho PeopleWhen EventsWhere Location
Why IntentEmotions
How Photo and audio
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Capturing and Reporting events using multimedia such as photos videos sensors and texts
Converting multimedia data to multimedia micro-reports using MediaJSON
Integrating multimedia micro-reports with other data sources for situation recognition trend analysis and culture analytics
Emerging opportunities for numerous apps smart city public health emergency rescue
What are the challenges
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
What ObjectsWho PeopleWhen EventsWhere Location
Why IntentEmotions
How Photo and audio
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
What ObjectsWho PeopleWhen EventsWhere Location
Why IntentEmotions
How Photo and audio
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Capturing and Reporting Events with Krumbs SDK
httpskrumbsnet
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Real-time MMR Dashboard
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Converting Multimedia Data into MMR
micro_reports[ where
geo_location latitude3290233332316081 longitude-1172441166718801
whenstart_timeJun 14 2009 112519
AMend_timeJun 14 2009 112519 AMtime_zoneAmericaLos_Angeles
what[concept_namepeopleconfidence09836078882217407visual_concept_providerCLARIFAIhellip concept_namefoodconfidence08526291847229004
visual_concept_providerCLARIFAI] tagrdquoniceday summer sourcedefault_srchttpshellipjpg sub_event[] why[]hellip]
Photo
What
Where
When
Who
Why
Sound
MediaJSON
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
MediaJSONData
Wrapper
Data Wrapper
Data Wrapper
Data Wrapper
IoT(Event-driven operation)
Converting Multimedia Data into MMR
MediaJSONData
Wrapper
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Number of photos in London per day
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Evolving Photo Concepts in London
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
bull Year of interest year 2008 and 2012bull Training locations Beijing (China) bull Testing location London (UK) bull Create ldquoOlympic Gamesrdquo Event Model from ldquoBAG of Visual Conceptsrdquo
Detecting Olympic Games
Olympic Games = basketball court game gymnastics people sport stadium swim tennis
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Evolving Photo Concepts in BeijingEvent model of ldquoOlympic Gamerdquo
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Detecting London Olympic GamesSummer Olympic Game
in July and August
Paralympic Games in September
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
bull Temporal range 1 year from July 2011 to June 2012
bull Location Thailand
Detecting Emergency Situations
City flood = outdoor water road car
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Photos from City Flood Cluster
November 5 2011
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Smart City Project in DC
105
US Presidential Inauguration in DC
Earth Day Concert
Cherry Blossom Festival
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Integrating MMR with other data sources for Situation Recognition (In progress)
h t t p s m a r t c i t i e s i n n o v a t i o n c o m
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125107
Trash Fill Level Situation in DC
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
730 800 830 900 930 1000 10300
20
40
60
80
100
0
3550
90
Real-Time Fill Level Situations at a given location of an event
Prediction based on Events HistoryEvents Data
Real-Time Trash Fill Level Situation
730 800 830 900 930 1000 10300
20
40
60
80
100
120
1020
40
70
90100
20
0
35
50
90
Fill Level Situations at a given EventFill Level from Event History Real-Time Fill Level
20 42
Now
Predicted Trash Fill Level in 30 minutes at a given location
78 99
30 minutes
730 800 830 900 930 1000 10300
20406080
100120
1030
40
7090
100
20
Projected Trash Fill Level at a given location based on Event History
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Conclusionbull EventShop Architecturebull Situation-based Applications using EventShopbull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial
videos presentations and publications please visit my home page httpwwwicsuciedu~spongpai
Siripen Pongpaichet (spongpaiuciedu)
109
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
FUTURE TRENDS AND OPEN PROBLEMS(20 mins Ramesh Jain)
110
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Future Trends
bull Future trends bull Open problems for Multimedia research
111
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
This century is different from the last
Should we think differently
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125In 20th century we tolerated
photos in our textual documents
In 21st century you create visual documents that tolerate text
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125Major Disruption in Photos From Memories to Information Sources
Photos are the most compelling source of information
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125115
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125116
We are immersed in Big Data
Multimedia
Realtime Uncertainty
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125117
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Data as a Platformbull Multi-modal
bull Multimedia has to become multimodalbull Data Streamsbull Important things ndash Situation recognitionbull Real time action for
118
Connecting People to Resources effectively efficiently and promptly
in given situations
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Contact Informationbull Vivek Singh Rutgers Universitybull VivekksinghRutgersedubull
bull Siripen Pongpaichetbull spongpaiicsuciedu
bull Ramesh Jainbull jainicsuciedu
119
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-
125
Useful linksbull Copies of publications
bull httpwpcomminforutgerseduvsinghpublicationbull Todayrsquos Slides
bull httpsdldropboxusercontentcomu5887580Tutorial_SituationRecogpdf
bull EventShop bull Online Service httpslnicsuciedu8085eventshoplinux bull Open Source httpdabuntugithubioes bull For more information about EventShop including tutorial videos
presentations and publications httpwwwicsuciedu~spongpai bull Related Projects
bull Tweeting Cameras httpssitesgooglecomsitefredyuhuiwangbull Frappe http
wwwstreamreasoningorglivefestivalcomunicazione2014
120
- situation recognition from multimodal Data
- Todayrsquos slides
- Course Outline
- Concept recognition from Multimedia data
- Introduction
- Data Information Knowledge Wisdom
- What is Important in lsquoBig Datarsquo
- The Grand Challenge
- Fundamental Problem
- Theory of Control and Communication in
- Cybernetics 101
- In Smart Systems Feedback is the Key
- Social Networks
- Social Life Networks
- Traditional Social Systems
- Emerging Social Systems
- Slide 17
- Concept Recognition Last Century
- Visual Concept Recognition First research papers
- Concept Recognition This Century
- Concept recognition from multimedia data
- Situation Recognition Next Frontier
- Situation recognition across multiple research domains
- Related Work Data to Situations
- Defining Situations Situation Calculus
- Situation Calculus Quick overview
- Problems with this approach
- Situations
- Situations commonalities
- Situation Definition
- Overall Framework Motivating example
- Applications
- Situation recognition FRamework
- Slide 34
- A) Situation Modeling
- Building Blocks Operands
- Building Blocks Operators
- Situation Modeling
- Slide 39
- B) Situation evaluation Workflow
- Data Representation
- Data Model
- Situation Recognition Algebra
- Media processing engine
- Implementation and results
- Testing Data Representation + Algebra
- Sample Queries
- Slide 48
- Flickr Social Emages
- Seasonal characteristics analysis
- Year average Peak of green
- Other approaches
- FraPPE a vocabulary to represent heterogeneous spatio-temporal
- Proposed Approach
- FraPPE visually
- FraPPE visually (2)
- FraPPE visually (3)
- FraPPE visually (4)
- FraPPE visually (5)
- FraPPE visually (6)
- City Sensing listens to the pulse of Milano Design Week on Apri
- Tweeting Cameras
- Physical amp Social Sensors Fusion For Situation Awareness
- Real-world Events
- Probabilistic Spatio-Temporal Data
- Physical Sensors (Concept ldquoCrowdrdquo)
- Social Sensors (MillionMarchNYC BlackLivesMatter)
- Fused Information
- CMage
- Fusing Sensor Cmage with Social Cmage
- DESIGNING SITUATION BASED APPLICATIONS
- Outline
- EventShop Requirement
- EventShop Architecture
- EventShop UI
- Demo
- Demo (2)
- Building applications using EventShop
- Asthma Management Application
- Asthma Management Application
- Asthma Risk Estimation
- Experiment Results
- Asthma Risk Estimator Model and Result
- A Graph Based Multimodal Geopatial Interpolation Framework
- Detecting Situations from Micro-Reports
- PHOTOS as Kodak Moments
- Disruption PHOTOS as Information
- Reports of Events from Journalists
- Reports of Events from Citizens
- Micro-Blogs
- Multimedia Micro-Reports (MMRs) are now and future
- What are the challenges
- Capturing and Reporting Events with Krumbs SDK
- Capturing and Reporting Events with Krumbs SDK (2)
- Real-time MMR Dashboard
- Converting Multimedia Data into MMR
- Converting Multimedia Data into MMR (2)
- Number of photos in London per day
- Evolving Photo Concepts in London
- Detecting Olympic Games
- Evolving Photo Concepts in Beijing
- Detecting London Olympic Games
- Detecting Emergency Situations
- Photos from City Flood Cluster
- Smart City Project in DC
- Integrating MMR with other data sources for Situation Recogniti
- Trash Fill Level Situation in DC
- Prediction based on Events History
- Conclusion
- Future trends and open problems
- Future Trends
- This century is different from the last
- Slide 113
- Slide 114
- Slide 115
- We are immersed in Big Data
- Slide 117
- Data as a Platform
- Contact Information
- Useful links
-