september 29, 2002ubicomp 021 nist meeting data collection jean scholtz national institute of...
TRANSCRIPT
September 29, 2002 Ubicomp 02 1
NIST Meeting Data CollectionNIST Meeting Data Collection
Jean Scholtz
National Institute of Standards and Technology
Gaithersburg, MD 20899-8940
USA
September 29, 2002 Ubicomp 02 2
Goals / ApproachGoals / ApproachGoals• (Intermediate) Support evaluation efforts for speech
community• (Future) Support evaluation efforts for
context-aware/ubicomp communityApproach• Provide rich/diverse pool of audio video corpora for
advanced recognition research • Provide infrastructure for integration and evaluation• Provide flexible architecture for incorporating and
annotating activities in sensor –rich environment
September 29, 2002 Ubicomp 02 3
NIST Data Collection LaboratoryNIST Data Collection Laboratory• Typical meeting space equipped with standard meeting
equipment• Instrumented with
– 200 microphones, 5 cameras – all synchronized with NIST SmartFlow system across 13 processors
– several disk arrays handle huge storage requirements
• Monitor/Review workstation in adjoining room– operator can start/stop data streams, select video views,
audio channels, and manipulate cameras– participants can review meeting recordings and deselect
excerpts from public distribution
• Relational database log with user interface
September 29, 2002 Ubicomp 02 4
EntranceRamp
Whiteboard
Modular ConferenceTable 13’ x 5’
NIST ‘SmartFlow’ synchronization
NIST Meeting Data Collection LaboratoryNIST Meeting Data Collection Laboratory
7’ x
7’ s
cree
n
CeilingProjector
E-White
board
mic micmic
mic
mic
mic mic mic
omnimic
quadmic
omnimic
COTS MicData Collection
System
Wireless Mic
Receivers
omnimic
3 Mic ArrayData Collection
Systems
Mic
Arr
ay
Mic Array
Mic Array
5 VideoData Collection
Systems
September 29, 2002 Ubicomp 02 5
Meeting Review/Monitor Meeting Review/Monitor WorkstationWorkstation •Used to both collect
and review recordings
•LINUX-based, interfaces with Smartflow live or archived data
•User can select video views and audio channels
•User can control camera view/movement with joystick
September 29, 2002 Ubicomp 02 6
Scenario OntologyScenario Ontology
Scenario Type
Scenario Example
DimensionsTasksTime Participants
Dur
atio
n (m
inut
es)
Loc
al v
s. R
emot
e
Sam
e pe
ople
Dif
fere
nt p
eopl
e
Org
. pre
vs.
adh
oc
#of
part
icip
ants
Dom
ain
expe
rtis
e
Mod
erat
or
plan
ning
Bra
inst
orm
ing
Neg
otia
tion
Dec
isio
n m
akin
gC
ompe
titi
ve
perf
orm
ance
Dis
sem
inat
ion
of I
nfor
mat
ion
Different
Focus group discussion
Place
Local 60 5-8 none adhoc
Round Table discussion: May include expert to present info and moderate discussion
Staff meeting Local 45 5-8 adhocParticipants discuss real technical issues, brainstorm ideas and make decisions.
Planning Local 45 5-8 none adhocEvent planning: Plan the different tasks that each member will carry out, timelines and coordination.
Interactive game playing Local 60 3-5none pre
Information exchange and decision making Information gathering and decision making
Local
Local
45
45
5-8
3-5 adhoc
prenone
none
Office Furnishing: An expert will help participants to choose office furniture, carpet, etc. for an office.
Participants engage in a storytelling games, e.g. Once upon a time, Settlers of catan.
Shop on line: Participants search the internet and collaborate to purchase computer hardware.
September 29, 2002 Ubicomp 02 7
Pilot Corpus Design• 20 hours of meetings are being collected
– ~60 GB per hour uncompressed data rate– data stored/distributed on large hard disks
• Varied forums– focus groups, game playing, interacting with experts, real
working group meetings, event planning
• Varied meeting lengths– 15 minutes to 1 hour
• Varied number of participants– 3 to 8 participants
September 29, 2002 Ubicomp 02 8
Sample of Meetings CollectedSample of Meetings CollectedTopic Meeting Type Date Length
(min)
Native
M / F
Non-native
M / F
Participants
IAD New Year Party Planning
Planning 12/12/01 60 0 / 2 0 / 1 3
IAD Open House Planning 3/5/02 56 1 / 4 1 / 1 7
ITL Party Planning Planning 1/11/02 28 3 / 2 0 / 1 6
Bioterrorism Focus group 11/15/01 120 3 / 1 0 / 0 4
Office Furniture Layout
Interaction with expert
2/14/02 55 1 / 3 0 / 2 6
Settlers of Catan Game playing 1/25/02 60 0 / 0 3 / 1 4
Once Upon a Time Game playing 3/4/02 58 1 / 1 2 / 0 4
Real Working Group Staff meeting 2/13/02 69 3 / 2 1 / 0 6
Shop On-line Collaborative design
3/19/02 60 1 / 0 3 / 0 4
September 29, 2002 Ubicomp 02 9
Use of Architecture
Streaming Sensor Input:video, audio, images, location tracks, application interactions
Distributed Services: Smartflow, Device management sw, Distributed data management, Archival
Storage, Service/ device discovery
MetaData Architecture: ATLAS, Annotation/review stations, Annotation ontology, Visualization,
Browsing, Annotation generation, Annotation tools, Logging tools for COTS products, sensor data annotation
Context Architecture: Annotation integration and abstraction, Realtime annotation maintenance,
Context entities (people, activities, artifacts)
Component Technologies: Context abstraction components, simulations for context-aware apps
Corpora
Component test bed
Context-Aware apps
Non-aware apps
Componenttest bed
Test bed
Test bed
Architecture
Component Technologies: Recognition components (ASR, Speaker ID, Face Recognition), Location
recognition, Visualization of sensor data
Architecture
Architecture
Can incorporate component technologies, log interactions, evaluate
Can incorporate context widgets, log interactions, evaluate
September 29, 2002 Ubicomp 02 10
Future PlansFuture Plans•Extend to distributed meetings/activities
•Provide test bed capabilities to allow developers of applications to incorporate new component technologies and capture user interactions for user-centered evaluations.
• Develop annotation ontology for user interactions to facilitate user-centered evaluations of meeting room applications.
• Extend this capability to context-aware applications allowing developers access to sensor inputs without having to install those sensors in their environment.
• Create community of researchers interested in using this technology