computer science engineering lee sang seon. introduction basic notions for temporal video...

Temporal Video Bound-aries

Computer Science EngineeringLee Sang Seon

WhyTemporal Video Boundaries

Techniqueis useful in the

Video content analysis?

Index

Introduction Basic notions for temporal video boundaries Micro-Boundaries Macro-Boundaries Mega-Boundaries Conclusion Q & A

Introduction

Brief definition of Temporal Video Boundary technique

→ Examine the temporal boundary problem at different levels of video content structure analysis

Why we need Temporal Video Boundary technique?

Show example

Example : Oscar awards

Insufficient metadata

opening

ending

Example : Oscar awards

Detailed metadata

opening

ending

actor

winners

awards

ending

Basic notions - modali-ties Video contains three types of modalities (i) Visual (ii) Audio (iii) Textual

Each modality has three levels(i) low-level (ii) mid -level (iii) high-level→ levels describe the amount of details described in each modality in terms of granularity and ab-straction

Basic notions - modali-ties For each modality and for each level there if

a set of attributes. These can be formalized as vectors:

Basic notions - modali-ties Adding to this, given a set of vectors

→ their average value denote the vector

Basic notions - method Local method→ the difference is computed between con-

secutive frames

Global method→ the difference if computed over a series of

frames

Micro-Boundaries

Definition Boundaries associated to the smallest video

units for which a given attribute is constant or slowly varying

The attribute can be any feature in the visual, audio, or text domain

Example

Make family histogram

Data structure that represents the color in-formation of a family of frames.

Set of frames that exhibits uniform features

= Frame histogram

Histogram difference measures Histogram difference using L1 metrics

Bin-wise histogram intersection

Total number of color bins used

Histogram of previous frame

Histogram of current frame

Merging of family his-tograms

Multiple ways to compare and merge families - contiguity & memory

1. Contiguous with zero memory → A new frame histogram is compared with

previous frame histogram

2. Contiguous with limited memory→ A new frame histogram is compared with

previous family histogram

Multiple ways to compare and merge families - contiguity & memory

3. Non contiguous with unlimited memory → A new frame histogram is compared with all

previous family histograms within the same video.

4. Hybrid→ First a new frame histogram is compared using

the contiguous frames and then generated family histograms are merged using non con-tiguous case.

Compare different Histogram difference measures

Macro-Boundaries

Definition Boundaries between collections of video micro-

segments that are clearly identifiable organic parts of an event defining a structural (action) or thematic (story) unit

Video : collection of stories that may or may not be interconnected

→ Macro-Boundaries detection= Segmenting stories

textual cues

audio cuesvisual cues

Two types of uniform segment detection

Unimodal segment detection A video segment exhibits same characteristic

over a period of time

Multimodal segment detection A video segment exhibits a certain characteris-

tic taking into account attributes from different modalities

Single Modality Segmen-taion

Partition a continuous bit-stream of audio data into non-

overlapping segments

Classification

Seven mid-level audio cate-gories

Using low-level audio features

Audio segmen-tation & classifi-

cationText transcript

Extracted from either the closed captions or speech-to-

text conversion

Segmented and categorized with respect to a predefined

topic list

Frequency-of-word-occurrence metric is used

Multimodal Segmentaion

Pre-merging Steps

Uniform seg-ment detection

Intra-modal segment clus-

tering

Attribute tem-plate determi-

nation

Dominant at-tribute deter-

mination

Template ap-plication

Descent Meth-ods

Goal :Create macro-bound-

aries that are more ac-curate than the bound-aries produced by indi-

vidual modalities.

Descent MethodsText seg-

ment

Audio segment

Video segment

Single descent Method

Single descent with intersecting

union

Single descent with intersec-

tion

Single descent with secondary

voting attributes

Single descent with conditional

union

Mega-Boundaries

Definition Boundaries between collections of macro-seg-

ments that exhibit different structural and fea-ture consistency (e.g. different genres)

Example Commercial detection method

Trigger & Verifiers Model

Features that can aid in determining the location of the commercial break

Triggers

Features that can determine the boundaries of the commercial break

Verifiers

Black frames

Time interval be-tween detected

black frames as trig-gers

Used as verifiers

Letterbox change

High cut rate(= low cut distance)

Bayesian Belief Network Modelstart

Genetic Algorithms

Conclusion

Type of bound-aries Methods Example

Micro-boundaries Frame & Family histogram comparing and merging

Visual scene segmenta-tion

Macro-boundaries Single modality segmenta-tion

&Multimodal segmentation

Multimodal story segmen-tation

Mega-boundaries Trigger & Verifier Commercial detection

Whenever metadata is availableor unavailable,

we can segment video by using this technique that

categorized three types

&Thank you!

Q & A