Download - Multimedia big data 140619
• Marshal McLuhan in 1964. – A medium itself, not the content it carries, should be the focus of study.
– A medium affects the society in which it plays a role not only by the content delivered over the medium, but also by the characteris@cs of the medium itself.
BAMMF June 20, 2014 2
Media
Mul+
BAMMF June 20, 2014 3
Media
+ +
Digital technology allowed combining and using media leading to a major disrupAon.
Photos are the killer content type on mobile. Quick to consume like text, but easier to produce on a
phone.
BAMMF June 20, 2014 4
From Mary Meeker’s presenta@on
MulAmedia and Big Data are both the Latest Elephant.
BAMMF June 20, 2014 12
Lots of people excited about these but with narrow perspecAves.
Two Major Human AcAviAes
• CommunicaAons
• ComputaAons
BAMMF June 20, 2014 14
Both require MulAmedia AND Big Data.
Storytelling: The Oldest and most powerful Art.
• A story is a chain of events that begins at one place and ends at another.
• A story is a selecAve batch of informaAon. It selects details, arranges them, perhaps embellishes them.
BAMMF June 20, 2014 18
h6p://janefriedman.com/2011/09/27/what-‐is-‐a-‐story/
Where do Events happen?
• In Real World.
• Images, video, audio, all other sensors are just a means to detect and capture some aspect of the event.
BAMMF June 20, 2014 19
Events are ‘Connectors’ Events create ‘Context’
Event contains Information and Experiences
Spatial Causal
Experiential Informational
Temporal Structural
So What is Digital Storytelling?
• Prepare a database of all events. – Collect all experienAal and informaAonal data about each event.
• Considering audience and the message, select important events.
• Render Events – For each event decide what informaAon and experienAal data should be used.
– Render all events and related data in the best format for highest quality of experience.
BAMMF June 20, 2014 21
Do you edit your videos?
• Why is video ediAng so difficult? – Even a_er so many products?
• How can we make it usable by consumers? • Is Video ediAng a real problem? • Is it possible to reinvent video ediAng given all the ‘big data’?
BAMMF June 20, 2014 26
Virage Video Indexing Lyn Russell
00:08:26:12 00:08:34:29 00:08:40:00
00:08:26:12 00:08:34:29 00:08:40:00
“Clinton spoke with reporters” “When She arrived” “Older citizens
who have”
Clinton running for senate
Speaking at rally
Improving Medicare
Lyn Russell
Hillary Clinton Face ID
Speaker ID
User Annotation
Speech to Text
Time Code
Keyframes
Video
Time
Hillary Clinton
Different Data Tracks
Ac+vity
Loca+on
Manual
Visual
Audio
OTHER
Walk
Walk
Walk
Walk
Walk
Walk
Exercise Drive Home Drive Mee+ng Work Exercise Home Drive Mee+ng Work Home Mee+ng Work Exercise
Event Track
Fundamental Opportunity in MulAmedia CommunicaAon and CompuAng: UAlize complementary and correlated informaAon from mulAple sources to form holisAc picture of the world.
BAMMF June 20, 2014 31
ComputaAon: at least 50,000 years
• CounAng was primarily used to keep track of social and economic data such as number of group members, prey animals, property, or debts.
• CompuAng was invented for Resource management.
BAMMF June 20, 2014 32
Big data is the Rorschach blot of our Ames – an incomprehensible shape on to which we project our dreams and
nightmares, hopes and fears.
BAMMF June 20, 2014 34
Rorschach blot
The most Fundamental Problem in Society:
Connecting People to Resources Effectively, Efficiently, and Promptly
in given Situations.
36 BAMMF June 20, 2014
What differenAates:
• Advanced SocieAes from Primi6ve SocieAes • Advanced countries from a Developing countries
38 BAMMF June 20, 2014
How People’s needs are connected to Resources.
Where do most Data Scientists apply their Magic?
We live in a targeted world. AdverAsers need efficient ways to si_ through the volume of data.
BAMMF June 20, 2014 43
• Desired state (Goal) • System model and Control Signal (AcAons)
• Current State (using mulAmedia data)
EventShop : E. Situation Detection
Predic+ve Situa+on
Recogni+on
Evolving Global Situation
Predic+ve Personal Situa+on
Recogni+on
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation Engine
Persona Database
Resources
Needs
Data Inges+on
Wearable Sensors
Calendar
Location ….
Dat
a So
urce
s ….
Data Inges+on
and aggrega+on
Database Systems
Satellite
Environmental Sensor Devices
Social Network
Internet of Things
Actionable Information
BAMMF June 20, 2014 48
CriAcal Requirements:
• Environmental SituaAon RecogniAon. – Detects geo-‐spaAal situaAons. – Space is a key dimension. – Helps in idenAfying resources and connecAng to them.
• Personal SituaAon RecogniAon. – Centered around an individual. – Uses environmental situaAon. – Helps in modeling individuals and idenAfying their needs.
BAMMF June 20, 2014 49
Data to Situa+ons
Situa+ons
Event Streams
Mul+modal Observa+ons
Low-‐level Analysis
Aggrega+on and Classifica+on
BAMMF June 20, 2014 50
A Grand Challenge for data has always been:
Sense making from mulAmodal massive geo-‐social data-‐streams.
51 BAMMF June 20, 2014
A REAL Challenge
• SituaAon Modeling • SituaAon RecogniAon
(Object recogniAon, event recogniAon, speech recogniAon, … dominated last century.)
BAMMF June 20, 2014 52
Challenge: Unifying MulAmodal Big Data
• Spa+o-‐temporal-‐thema+c (STT) real-‐@me streams • E-‐mage as a unifying GRID representa+on
53
(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)
d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)
BAMMF June 20, 2014
54
Billions of data sources. Environment for
SelecAng, and Combining
appropriate sources to detect situaAons. PredicAon for Pro-‐acAve acAons InteracAons with different types of Users
BAMMF June 20, 2014
The Creative Destruction of Medicine
CompuAng Power, Data Universe
Social Networking
Internet
Mobile connecAvity, Bandwidth
InformaAon Systems
Imaging
Wireless Sensors
Genomics
New Medicine
CreaAv
e De
strucAon
BAMMF June 20, 2014 58 Old Medicine
Asthma Triggers
Indoor Allergens
Outdoor Allergens
Smoke
Air PolluAon
Chemical Irritants
Asthma/Allergies
BAMMF June 20, 2014 60
Everyone’s Respiratory Health is Different
Disaster Situa+on
Assimila+on and Control
Environmental
Resources and Historic Data
Governmental Agencies
Internet of Things
Social Sources
Experts
Users
All Users are not EQUAL.
BAMMF June 20, 2014 62
AcAviAes
CommunicaAons
Media
Persona: Turning Disassociated Data into Meaningful InformaAon
AGGREGATED PERSONAL ANALYTICS
Sensors
63
Persona
BAMMF June 20, 2014
Wearable 2013: Data Data Everywhere, nor any Insight to Use.
h6p://www.medhelp.org/ BAMMF June 20, 2014 64
HEALTH PERSONA
Logical Sensor Physiological Sensors Fitness Tracking Sensors
Life Event
Kine+c Event
Physiological Event
Food Event
Personicle
(Health) Persona Framework
Calendar Ac+vity
-‐level
Heart rate
Home Drive Mee+ng Work Exercise Walk
Walk
BAMMF June 20, 2014 65
Time
Foursquare
Personal Calendar
Moves
Drive
Walk
Mee+ng in DBH room 3233
Walk
Drive
Mee+ng with trainer in gym
Ac+vity Level
Personicle
Personicle
Home Drive Walk
Mee+ng Work Walk
Drive Exercise Work
Laleh at University Club
Ea+ng dinner
BAMMF June 20, 2014 66
Insight: Exercise Triggers My Asthma
• Red: “Severe asthma attack”. • Orange: High activity level.
• Intense Exercise triggers asthma! Personicle
Ac+vity Stream
t1 t2 t3 t4 t5
BAMMF June 20, 2014 67
UCI Allergy/Asthma App
Triggers Recommendation
Stop Jogging!
Symptoms SHORTNESS OF BREATH
CHEST TIGHTNESS OR PAIN
COUGHING
WHEEZING
How Do You Feel?
BAMMF June 20, 2014 68
From Persona to Societal Health
Health Persona of different people
Environmental factors
Disease outbreak
Disease Modeling
Geographical loca+on specifica+ons
BAMMF June 20, 2014 70
Use ObjecAve Massive Volume of Personal Data for Building Beker Disease Models.
Persona and Societal Health
Use Disease Model and Personal Data for Beker Quality of Life.
71 BAMMF June 20, 2014
Live EventShop and CollaboraAon • Live EventShop Demo
– h6p://auge.ics.uci.edu/eventshop/ • Current Collaborators
– Cyber-‐Physical Cloud Compu+ng Project • NICT, NIST
– SLN4MOP Project • Sri Lanka Farmers; Prof. Ginige in Sydney leading
– Open Source EventShop. • HCL
– Other partners BAMMF June 20, 2014 72
MulAmedia and Big Data
• MulAmedia was always Big Data. • Many Major Challenges in Big Data are the same as in MulAmedia.
• Time for MulAmedia community to Accept and Act.
BAMMF June 20, 2014 73