computer science engineering lee sang seon. introduction basic notions for temporal video...
TRANSCRIPT
Temporal Video Bound-aries
Computer Science EngineeringLee Sang Seon
WhyTemporal Video Boundaries
Techniqueis useful in the
Video content analysis?
Index
Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries Conclusion Q & A
Introduction
Brief definition of Temporal Video Boundary technique
→ Examine the temporal boundary problem at different levels of video content structure analysis
Why we need Temporal Video Boundary technique?
Show example
Example : Oscar awards
Insufficient metadata
opening
ending
Example : Oscar awards
Detailed metadata
opening
ending
actor
winners
awards
ending
Basic notions - modali-ties Video contains three types of modalities (i) Visual (ii) Audio (iii) Textual
Each modality has three levels(i) low-level (ii) mid -level (iii) high-level→ levels describe the amount of details described in each modality in terms of granularity and ab-straction
Basic notions - modali-ties For each modality and for each level there if
a set of attributes. These can be formalized as vectors:
Basic notions - modali-ties Adding to this, given a set of vectors
→ their average value denote the vector
Basic notions - method Local method→ the difference is computed between con-
secutive frames
Global method→ the difference if computed over a series of
frames
Micro-Boundaries
Definition Boundaries associated to the smallest video
units for which a given attribute is constant or slowly varying
The attribute can be any feature in the visual, audio, or text domain
Example
Make family histogram
Data structure that represents the color in-formation of a family of frames.
Set of frames that exhibits uniform features
= Frame histogram
Histogram difference measures Histogram difference using L1 metrics
Bin-wise histogram intersection
Total number of color bins used
Histogram of previous frame
Histogram of current frame
Merging of family his-tograms
Multiple ways to compare and merge families - contiguity & memory
1. Contiguous with zero memory → A new frame histogram is compared with
previous frame histogram
2. Contiguous with limited memory→ A new frame histogram is compared with
previous family histogram
Multiple ways to compare and merge families - contiguity & memory
3. Non contiguous with unlimited memory → A new frame histogram is compared with all
previous family histograms within the same video.
4. Hybrid→ First a new frame histogram is compared using
the contiguous frames and then generated family histograms are merged using non con-tiguous case.
Compare different Histogram difference measures
Macro-Boundaries
Definition Boundaries between collections of video micro-
segments that are clearly identifiable organic parts of an event defining a structural (action) or thematic (story) unit
Video : collection of stories that may or may not be interconnected
→ Macro-Boundaries detection= Segmenting stories
textual cues
audio cuesvisual cues
Two types of uniform segment detection
Unimodal segment detection A video segment exhibits same characteristic
over a period of time
Multimodal segment detection A video segment exhibits a certain characteris-
tic taking into account attributes from different modalities
Single Modality Segmen-taion
Partition a continuous bit-stream of audio data into non-
overlapping segments
Classification
Seven mid-level audio cate-gories
Using low-level audio features
Audio segmen-tation & classifi-
cationText transcript
Extracted from either the closed captions or speech-to-
text conversion
Segmented and categorized with respect to a predefined
topic list
Frequency-of-word-occurrence metric is used
Multimodal Segmentaion
Pre-merging Steps
Uniform seg-ment detection
Intra-modal segment clus-
tering
Attribute tem-plate determi-
nation
Dominant at-tribute deter-
mination
Template ap-plication
Descent Meth-ods
Goal :Create macro-bound-
aries that are more ac-curate than the bound-aries produced by indi-
vidual modalities.
Descent MethodsText seg-
ment
Audio segment
Video segment
Single descent Method
Single descent with intersecting
union
Single descent with intersec-
tion
Single descent with secondary
voting attributes
Single descent with conditional
union
Mega-Boundaries
Definition Boundaries between collections of macro-seg-
ments that exhibit different structural and fea-ture consistency (e.g. different genres)
Example Commercial detection method
Trigger & Verifiers Model
Features that can aid in determining the location of the commercial break
Triggers
Features that can determine the boundaries of the commercial break
Verifiers
Black frames
Time interval be-tween detected
black frames as trig-gers
Used as verifiers
Letterbox change
High cut rate(= low cut distance)
Bayesian Belief Network Modelstart
Genetic Algorithms
Conclusion
Type of bound-aries Methods Example
Micro-boundaries Frame & Family histogram comparing and merging
Visual scene segmenta-tion
Macro-boundaries Single modality segmenta-tion
&Multimodal segmentation
Multimodal story segmen-tation
Mega-boundaries Trigger & Verifier Commercial detection
Whenever metadata is availableor unavailable,
we can segment video by using this technique that
categorized three types
&Thank you!
Q & A