analysis, indexing and visualization of presentation videosmmerler/poster_dsp176-merler.pdf ·...
TRANSCRIPT
Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Analysis, Indexing and Visualization of Presentation Videos
Mic
he
le M
erl
er
em
ail
: m
me
rle
r@cs
.co
lum
bia
.ed
u
C
om
pu
ter
Sci
en
ce D
ep
art
me
nt,
Co
lum
bia
Un
ive
rsit
y
Mo
tiv
ati
on
& D
om
ain
De
scri
pti
on
Mic
he
le M
erl
er
em
ail
: m
me
rle
r@cs
.co
lum
bia
.ed
u
C
om
pu
ter
Sci
en
ce D
ep
art
me
nt,
Co
lum
bia
Un
ive
rsit
y
Mo
tiv
ati
on
& D
om
ain
De
scri
pti
on
Do
ma
in c
ha
lle
ng
es:
“W
ILD
” !
Ma
ny
v
ide
os
are
alr
ea
dy
arc
hiv
ed
Low
qu
ali
tyLa
ck o
f S
tru
ctu
reA
qu
ick
ly i
ncr
ea
sin
g q
ua
nti
ty o
f p
rese
nta
tio
n v
ide
os
is
GO
AL
: H
elp
use
rs e
ffic
ien
tly
M
oti
va
tio
n &
Do
ma
in D
esc
rip
tio
nD
om
ain
ch
all
en
ge
s: “
WIL
D”
! •
Lack
of
ad
dit
ion
al
•N
ot
reco
rde
d b
y
•U
nco
nst
rain
ed
ca
me
ra
alr
ea
dy
arc
hiv
ed
Low
qu
ali
tyLa
ck o
f S
tru
ctu
re
Vid
eo
s o
f p
rese
nta
tio
ns
are
to
ols
no
wa
da
ys
A q
uic
kly
in
cre
asi
ng
qu
an
tity
of
pre
sen
tati
on
vid
eo
s is
pu
bli
cly a
vail
ab
le a
nd
re
trie
vab
le o
n t
he
we
bG
OA
L :
He
lp u
sers
eff
icie
ntl
y
an
d e
ffe
ctiv
ely
acc
ess
•
Lack
of
ad
dit
ion
al
sou
rce
s o
f
•N
ot
reco
rde
d b
y
pro
fess
ion
al
•U
nco
nst
rain
ed
ca
me
ra
mo
vem
en
ts
Vid
eo
s o
f p
rese
nta
tio
ns
are
to
ols
no
wa
da
ys
em
plo
yed
in a
la
rge
va
rie
ty o
f sy
ste
ms
an
d e
ffe
ctiv
ely
acc
ess
(ed
uca
tio
na
l) in
form
ati
on
sou
rce
s o
f
info
rma
tio
n
(e.g
. e
lect
ron
ic
pro
fess
ion
al
cam
era
me
n
mo
vem
en
ts
•S
lid
es
Tru
nca
tio
n�
Dis
tan
ce o
r E
-le
arn
ing
�C
on
fere
nce
pro
cee
din
gs
(ed
uca
tio
na
l) in
form
ati
on
(e.g
. e
lect
ron
ic
cop
ies
of
slid
es)
•Li
gh
t ca
nn
ot
be
use
d a
s cl
ue
•S
lid
es
Tru
nca
tio
n
•C
om
pre
ssio
n�
Co
nfe
ren
ce p
roce
ed
ing
s
�S
tud
en
t p
rese
nta
tio
ns
resu
lts
1-2
0 o
f 1
,16
0
use
d a
s cl
ue
•N
ot
ed
ite
d
•C
om
pre
ssio
n
•S
tan
da
rd p
roce
ssin
g
�S
tud
en
t p
rese
nta
tio
ns
�C
orp
ora
te t
alk
s6
59
eve
nts
9
K a
uth
ors
12
K le
ctu
res
1
4K
vid
eo
sre
sult
s 1
-20
of
1,1
60
•N
ot
ed
ite
d•
Sta
nd
ard
pro
cess
ing
do
es
no
t a
pp
ly
�C
orp
ora
te t
alk
s1
2K
lect
ure
s 1
4K
vid
eo
s
3.
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
GO
AL
: E
nsu
re e
nd
use
rs s
ati
sfa
ctio
nw
ith
ho
w t
he
1
. U
ser
Pre
ferr
ed
Fa
ce I
nd
exe
s3
. G
rap
hic
s In
de
x G
en
era
tio
nG
OA
L:
En
sure
en
du
sers
sa
tisf
act
ion
wit
h h
ow
th
e
info
rma
tio
n e
xtra
cte
d f
rom
th
e v
ide
os
is p
rese
nte
d
Re
sult
s
1.
Use
r P
refe
rre
d F
ace
In
de
xes
3.
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
Ex
pe
rim
en
tal
Se
tup
Re
sult
s
�1
57
5 A
ma
zon
Me
cha
nic
al
Turk
HIT
s
(15
sp
ea
kers
x 3
ord
eri
ng
x 3
5 u
niq
ue
wo
rke
rs)
�M
ost
pe
op
le p
refe
r H
ea
d &
Sh
ou
lde
r F
RO
NTA
Lv
iew
�3
5%
of
vote
s w
en
t to
Le
ft a
nd
Rig
ht
¾ H
ea
d &
Sh
ou
lde
r!P
rop
ose
d S
olu
tio
n(1
5 s
pe
ake
rs x
3 o
rde
rin
g x
35
un
iqu
e w
ork
ers
)�
35
% o
f vo
tes
we
nt
to L
eft
an
d R
igh
t ¾
He
ad
& S
ho
uld
er!
Co
nfi
rms
resu
lts
of
psy
cho
log
ica
l st
ud
ies
on
in
fere
nce
of
Pro
po
sed
So
luti
on
�LB
P H
isto
gra
m +
Co
lor
His
tog
ram
Co
nfi
rms
resu
lts
of
psy
cho
log
ica
l st
ud
ies
on
in
fere
nce
of
he
ad
3D
in
form
ati
on
fro
m 3
/4 v
iew
of
face
[B
urk
e V
R0
7]
Ind
ex
pre
sen
tati
on
vid
eo
s b
ase
d o
n f
ou
r m
ajo
r cu
es:
�LB
P H
isto
gra
m +
Co
lor
His
tog
ram
�O
nli
ne
Clu
ste
rin
g (
vis
ua
l + t
em
po
ral)
wit
h a
vg.
Lin
kag
e
Ind
ex
pre
sen
tati
on
vid
eo
s b
ase
d o
n f
ou
r m
ajo
r cu
es:
�Te
xt (
+a
ud
io t
ran
scri
pts
)�
Gra
ph
ics
�O
nli
ne
Clu
ste
rin
g (
vis
ua
l + t
em
po
ral)
wit
h a
vg.
Lin
kag
e
�N
orm
. C
ross
Co
rre
lati
on
fo
r Te
mp
late
Ma
tch
ing
�
Text
(+
au
dio
tra
nsc
rip
ts)
�S
pe
ake
r fa
ces
�G
rap
hic
s
�M
osa
ics
5.0
),
(
11
),
(1
2
C k
ji
cx
CC
xS
j
+=
∑ =χ
ixregion
It h
as
be
tte
r il
lum
ina
tio
n
It h
as
be
tte
r re
solu
tio
n
I ca
n s
ee
/te
ll m
ore
ab
ou
t th
e w
ho
le a
pp
ea
ran
ce o
f th
e p
ers
on
I ca
n s
ee
be
tte
r th
e e
ye
s a
nd
exp
ress
ion
of
the
pe
rso
n
�M
osa
ics
()
() )
(4.0
)(
4.0
),
(
5.0
),
(1
2
jj
ji
kjk
ij
CT
tC
SC
xT
cx
C
−+
−+
=
+∑ =
βα
χj
Ccluster
I ca
n s
ee
be
tte
r th
e e
ye
s a
nd
exp
ress
ion
of
the
pe
rso
n
I p
refe
r th
is p
ose
of
a p
ers
on
in
ge
ne
ral
I p
ick
ed
th
e b
est
ou
t o
f a
bu
nch
of
ba
d p
ictu
res
No
ne
of
the
ab
ov
e(p
lea
se e
xpla
in y
ou
r re
aso
n w
ith
a f
ew
wo
rds
in t
he
bo
x b
elo
w)
BA
CK
-EN
D
()
()
jj
ji
),
(
),
(j
ij
iC
xT
Cx
S<>
BA
CK
-EN
D)
,(
),
(j
ij
iC
xT
Cx
S<>
2.
Au
tom
ati
c G
en
era
tio
n o
f S
pe
ak
ers
Fa
ce I
nd
exe
sU
ser
Pre
ferr
ed
Te
xtu
al
Ind
ex
Gra
ph
ics
Ind
ex
4.
Tex
tua
l In
de
x G
en
era
tio
n2
. A
uto
ma
tic
Ge
ne
rati
on
of
Sp
ea
ke
rs F
ace
In
de
xes
Use
r P
refe
rre
d
Face
In
de
xes
Text
ua
l In
de
x
Ge
ne
rati
on
Gra
ph
ics
Ind
ex
Ge
ne
rati
on
4.
Tex
tua
l In
de
x G
en
era
tio
n1
34
Ed
ge
s C
on
ne
cte
d
Ge
om
etr
ic +
Ed
ge
Lo
cal A
da
pti
ve O
tsu
Fa
ce I
nd
exe
sG
en
era
tio
nG
en
era
tio
n
Se
lect
ion
ba
sed
on
3 q
ua
lity
me
asu
res
�V
iola
Jo
ne
s d
ete
cto
rFa
ce
LoG
ed
ge
sE
dg
es
Co
nn
ect
ed
Co
mp
on
en
ts
Ge
om
etr
ic +
Ed
ge
De
nsi
ty C
on
stra
ints
Loca
l Ad
ap
tive
Ots
u
(LA
O)
Bin
ari
zati
on
Tess
era
ctO
CR
Sp
ea
ker
Face
S
em
an
tic
Sh
ot
25
1.
Re
solu
tio
n
Se
lect
ion
ba
sed
on
3 q
ua
lity
me
asu
res
�V
iola
Jo
ne
s d
ete
cto
r
�C
olo
r sk
in f
ilte
rFa
ce
De
tect
ion
Co
mp
on
en
tsD
en
sity
Co
nst
rain
ts(L
AO
) B
ina
riza
tio
nTe
sse
ract
OC
R
Co
mp
lete
d T
ask
sS
pe
ake
r Fa
ce
Ind
ex
Ge
ne
rati
on
Se
ma
nti
c S
ho
t
Re
pre
sen
tati
on
25
hw
×1
. R
eso
luti
on
Siz
e o
f th
e f
ace
re
gio
n
�C
olo
r sk
in f
ilte
rD
ete
ctio
nR
csca
rch
Inte
rvie
w w
ith
Cli
en
t
{ i
site
d P
roje
ct S
pa
ce
Co
mp
lete
d T
ask
s
Ind
ex
Ge
ne
rati
on
Re
pre
sen
tati
on
hw
×S
ize
of
the
fa
ce r
eg
ion
Face
Se
ed
s
Qia
nt
Ho
use
Re
sid
en
t A
sso
cia
tio
n M
ee
tin
g
�1
ho
ur
an
d 4
5 m
inu
tes
of
vid
eo
, 8
stu
de
nt
pre
sen
tati
on
s
Face
Se
ed
s
Co
mp
lete
d T
ask
s
�1
ho
ur
an
d 4
5 m
inu
tes
of
vid
eo
, 8
stu
de
nt
pre
sen
tati
on
s
�1
3 s
lid
es
pe
r p
rese
nta
tio
n (
ave
rag
e)
O tf
�M
ILTr
ack
(pre
dic
tio
n):
Face
Re
sea
rch
Inte
rvie
w
Cli
en
tA
fte
r vo
cab
ula
ry
corr
ect
ion
LAO + Tesseract
�1
3 s
lid
es
pe
r p
rese
nta
tio
n (
ave
rag
e)
VA
ST
MM
Bro
wse
r [1
]6
tf
P tf
�M
ILTr
ack
(pre
dic
tio
n):
�V
iola
Jo
ne
s d
ete
cto
r(o
bse
rva
tio
n):
Face
In
terv
iew
C
lie
nt
Pro
ject
Sp
ace
Ho
use
Re
sid
en
t A
sso
cia
tio
n M
ee
tin
g
corr
ect
ion
Tesseract
8000
Recognized
LAO + Tesseract
VA
ST
MM
Bro
wse
r [1
]6
tf
2.
Po
se
�V
iola
Jo
ne
s d
ete
cto
r(o
bse
rva
tio
n):
�S
imp
lifi
ed
Ka
lma
nfi
lte
r:Tra
ckin
gH
ou
se R
esi
de
nt
Ass
oci
ati
on
Me
eti
ng
Tesseract
6000
Recognized Characters
FR
ON
T-E
ND
O t
P tt
ff
f)
1(α
α−
+←
�Le
ft a
nd
rig
ht
¾ p
ose
cla
ssif
iers
Ed
ge
his
tog
ram
de
scri
pto
r
2.
Po
se�
Sim
pli
fie
d K
alm
an
filt
er:
4000
6000
Number Recognized Characters
FR
ON
T-E
ND
tt
tf
ff
)1(
αα
−+
←
15
38
3
15
38
5
15
38
7
15
37
1
15
40
9
15
35
5
�E
dg
e h
isto
gra
m d
esc
rip
tor
�S
VM
R
BF
ke
rne
lFa
ce T
rack
s
2000
4000
Number Characters
6.
Fin
al
Bro
wse
r in
terf
ace
15
38
3
15
38
5
15
38
7
15
37
1
15
40
9
15
35
5
�S
VM
R
BF
ke
rne
l
�Fa
ceTr
ace
rd
ata
set
Face
Tra
cks
2000
Number
6.
Fin
al
Bro
wse
r in
terf
ace
Tra
inin
g S
et
(le
ft ¾
, fr
on
t,
rig
ht
¾)
~1
0K
im
ag
es
Home
Search
Explore Collections
Visual Search
Login
or sign up
0
Recognition Method
Tra
inin
g S
et
(le
ft ¾
, fr
on
t,
rig
ht
¾)
~1
0K
im
ag
es
Test
Se
t ~
12
K i
ma
ge
s
Av
era
ge
Te
st A
ccu
racy
81
.5%
�S
ele
ctio
n o
f fa
ces
to m
atc
h
Se
arc
h
Home
Search
Explore Collections
Visual Search
Login
or sign up
Nu
mb
er
N
um
be
r P
reci
sio
nR
eca
ll
Recognition Method
Av
era
ge
Te
st A
ccu
racy
81
.5%
�S
ele
ctio
n o
f fa
ces
to m
atc
h
�LB
P d
esc
rip
tor
+ S
q.
L2
dis
tan
ceTra
cks
Se
arc
h
People Index
Graphics Index
Search Tips
Phone + P05 + G08
Tag
Nu
mb
er
GT
Wo
rds
Nu
mb
er
Re
c. W
ord
sP
reci
sio
nR
eca
ll3
Sk
in R
ati
oM
atc
hin
gPhone + P05 + G08
Tag
People Index
Graphics Index
22
76
11
26
0.4
95
0.6
65
Un
iqu
e S
pe
ake
rs
area
skinPixels
skinRatio
#=
>
185
.1
RU
niq
ue
Sp
ea
kers
Face
Tra
cks
5.
Se
ma
nti
c S
ho
t R
ep
rese
nta
tio
nE
nh
an
ced
Fe
atu
re B
ase
d M
osa
ic
area
skinRatio
=
>
>
⇔=
107
.0
185
.1
skin
Pixel
RBGR
Face
Tra
cks
5.
Se
ma
nti
c S
ho
t R
ep
rese
nta
tio
nE
nh
an
ced
Fe
atu
re B
ase
d M
osa
ic
>>+
+⇔
=
112
.0
107
.0
)(
skin
Pixel
2
RG
BG
R
RB
Face
In
de
x
Resolution 10 secs
>
++
112
.0
)(
2B
GR
RG
�S
ele
ct“b
est
fa
ces”
to
pre
sen
t t
o e
nd
use
r
Face
In
de
x
Ge
ne
rati
on
�P
TZ
Est
ima
tio
n
�S
IFT
+ R
AN
SA
C o
n
min max
Resolution 10 secs
Click on an icon to find the graphic in the video
pre
sen
t t
o e
nd
use
rG
en
era
tio
n�
SIF
T +
R
AN
SA
C o
n
key
fra
me
sskinRatio
wresolution
wpose
wQ
⋅+
⋅+
⋅=
32
1
35
0O
verl
ay R
eco
gn
ize
d T
ext
min max
Tagline
Click on an icon to find the graphic in the video
skinRatio
wresolution
wpose
wQ
⋅+
⋅+
⋅=
32
1
Av
era
ge
Tra
ck M
atc
hin
g T
ime
(se
cs)
33
5
30
0
35
0O
verl
ay R
eco
gn
ize
d T
ext
Tagline Frames
Test
on
3
33
5T
rack
Ma
tch
ing
Fa
ce S
ele
ctio
n
Left
/rig
ht3
4 E
xtra
ctio
n
20
0
25
0P
rob
lem
Sta
tem
en
t
•E
colo
gic
al
Imp
act
Te
xt1
Te
xt3
Te
xt4
Te
xt5
Te
xt7
Te
xt8
Te
xt1
0
Frames
51
ou
t o
f 5
8 w
ith
He
ad
& s
ho
uld
er,
¾ p
rofi
le v
iew
Test
on
3
stu
de
nt
Left
/rig
ht3
4 E
xtra
ctio
n
Sk
in-R
es
Ext
ract
ion
K-M
ea
ns
Co
mp
uta
tio
n
15
0
20
0•
Wa
ste
go
es
to L
an
dfi
lls
•E
ne
rgy
So
urc
e
•C
ost
Eff
icie
ncy
•W
ast
e D
isp
osa
l B
ill
Te
xt1
Te
xt3
Te
xt4
Te
xt5
Te
xt7
Te
xt8
Te
xt1
0
Text
Problem
Phone
51
ou
t o
f 5
8 w
ith
He
ad
& s
ho
uld
er,
¾ p
rofi
le v
iew
stu
de
nt
pre
sen
ta-
K-M
ea
ns
Co
mp
uta
tio
n
50
10
0
15
0•
Ele
ctri
cal
Bil
l
•M
s W
ilso
n i
s lo
ok
ing
fo
r a
n e
co-f
rie
nd
ly,
cost
eff
icie
nt,
an
d e
asy
to
use
pro
du
ct t
ha
t
wil
l co
nve
rt h
er
soli
d w
ast
e i
nto
usa
ble
en
erg
y
�S
eg
me
nt
vid
eo
in
to s
em
an
tica
lly
dis
tin
ct s
ho
ts b
ase
d
on
sli
de
s
People
pre
sen
ta-
tio
nv
ide
os,
0
50
en
erg
y
En
ha
nce
Gra
ph
ics
on
sli
de
s
�C
ha
ng
es
in t
ext
use
d t
o a
sse
ss s
lid
e c
ha
ng
es
Graphics
45
min
ute
s
ea
ch
20
1
9
K-M
ea
ns(
10
0)
sele
ct (
10
0)
min
-min
0
12
3E
nh
an
ce G
rap
hic
s�
Ch
an
ge
s in
te
xt u
sed
to
ass
ess
sli
de
ch
an
ge
s
Graphics
ea
chK
-Me
an
s(1
00
)se
lect
(1
00
)m
in-m
in