OCWC Global meeting 2011OCWC Global meeting 2011

Semi-automatic and easy creation ofSemi-automatic and easy creation of Clearning friendly OCW video contentlearning friendly OCW video content

Satoshi SHIMADA1 TadashiNAKANISHI1 Akira KOJIMA1Satoshi SHIMADA1, TadashiNAKANISHI1, Akira KOJIMA1

d Y hi i FUKUHARA2and Yoshimi FUKUHARA2

1NTT Cyber Solutions Laboratories 2Meiji Universityy j y

MotivationMotivationCreating a learning-friendly edited video, for video sharing g g y , gon the web, is painful and time-consuming.

Take timeTake time

Good skills requiredGood skills required

Costs moneyCosts money

How to make this process easy?How to make this process easy?

At the same time how to make edited video learningAt the same time, how to make edited video learning friendly ?friendly ?

The goal is to automatically extract information from aThe goal is to automatically extract information from a lecture video and use them to produce a better version of lecture video and use them to produce a better version ofthe video.

ProblemProblemWe need to take a video and edit it.

The cost of video capture can be reduced by fix HD camera.

Wide angle HD camera can record whole of lecture.

However, HD video is not suitable for sharing on the Web., g

Because,Because,

(1)File size of HD video is too large(1)File size of HD video is too large

(2)Slide in simple downsizing video(2)Slide in simple downsizing video

i d blis unreadable

( )(3)The entire scene is boring Video captured by

Video editing manually takes much timefixed camera

Video editing manually takes much time.

Proposed methodProposed methodProposed workflow for the easy and better creation

Cature the entire scene to a video bird's-eye view Cature the entire scene to a videoby Full HD cameraby Full HD camera

Video file is moved to Laptop

Automatic detection of importantAutomatic detection of important Information Semi-automatic Information

Screen/Speaker/Slide change transformationS /Sp a /S a g

Post production (Mixing)Post-production (Mixing)Screen + Speaker +Screen + Speaker +Slide change informationSlide change information

produced video

PrinciplePrinciple1920 i l

- speaker1920 pixels

- Screen or black board - atmosphere of hallare assembled based on 1080 pixels

editing template

j ti di t ti f-projection distortion of screenbrightness of speaker Editi t l t B-brightness of speaker

are correctedEditing template A Editing template B

are corrected.

720 pixels720 pixels

400 i l400 pixels

Video processingVideo processing(1)Speaker detection: find faces in the image(1)Speaker detection find faces in the image

linear Interpolation is conducted for no face imagelinear Interpolation is conducted for no face image

(2) S d t ti fi d t l d i t(2) Screen detection: find a rectangle under appropriate diti i d t lconditions, size and vertex angle

B th b iti d l t hBecause the camera can be positioned almost everywhere in the lecture room a homography is used to map thein the lecture room, a homography is used to map the screen coordinates to a flat rectangle as if it was seenscreen coordinates to a flat rectangle as if it was seen from the front of itfrom the front of it.

(3) Chaptering: Based on a subtraction method upon N equidistant frames to determine if a ‘Slide change’ has

Soccurred or not in ‘Screen region’

Usability evaluationUsability evaluation

15 Participants without prior knowledge were asked how 15 Participants without prior knowledge were asked howthey reviewed the post-processed video in comparison y p p pawith the original one.gVideos used: 4(=20min×2, 1hour×2) ,720×408 pixels

C i f M O i i SComparison of Mean Opinion Score

Proposed methodOriginal video

Original video (simple down conversion) Screen visibility(simple down conversion) S y

Speaker visibility

Not boring

Speaker visibility

gPresence

P t d id 1 2 3 4 5

Overall impression for learning usePost processed video

by the proposed method1 2 3 4 5

goodbadfor learning use

Other exampleOther exampleO i i l idOriginal video

Facial search area Screen area set by manuallyFacial search area by manually

Post processed video by the proposed method

h b i f l h iddl f lat the begin of lecture at the middle of lecture

Video sharing on the WebVideo sharing on the WebEnhance the video sharing function by using SceneKnowledge, which provides a user-friendly web interface to view videos, annotate them and post comment.

P t d dPost-producedlecture video

Annotationlecture video

Annotation within aChapter is displayed within a certain

Chapter is displayed

chapter of the Comment input form videoComment input form

SceneKnowledge: A Video scene-based video sharing andcomment posting system

How to use our softwareHow to use our softwareSceneEditor is a client software implementing the proposed

method.

Minimum operation is as followsMinimum operation is as follows1 determine the editing duration1. determine the editing duration2. Set facial search area2. Set facial search area3. Set clipping area of speaker3. Set clipping area of speaker4. detect screen area . a a5. select or set an editing template g p6. comand video procressing

--- auto processing ---

Time required for automated post-production is;Time required for automated post production is;Roughly correspond to the time of the video itselfRoughly correspond to the time of the video itself

Empowering communities with ICT Innovation

SceneKnowledge: A Video-based Knowledge Sharing SystemKnow how extraction by scene based video sharingKnow-how extraction by scene-based video sharing

A wide variety of knowledge and know-how is being lost due to changes in the structureof households coupled with Japan's low birth rate and ageing population. Although

OverviewUtilizing video: SceneKnowledge Collective knowledge

stored by participants writing comments andVid t tof households coupled with Japan s low birth rate and ageing population. Although

efforts are being made to preserve this heritage in video records, people do not obtainknowledge just by watching videos. Our proposed system splits video content intomanageable units so that the corresponding knowledge and know-how can be

Comments are linked to videos allowing

writing comments and Q&A responsesVideo content

makes movements and situations easy

to understandmanageable units so that the corresponding knowledge and know how can bediscovered and shared more easily. This system can be accessed remotely on PCs andmobile terminals, allowing information to be shared widely in everyday situations andproviding a useful tool for lifelong learning and for invigorating communities

to videos, allowing users to quickly access

other scenes of interest to themPoints reinforcedproviding a useful tool for lifelong learning and for invigorating communities.

■ By watching in scene units it is possible to collect useful on topic comments

Features New findings are encouraged by

i ith

Points reinforced by visual

annotation

■ By watching in scene units, it is possible to collect useful on-topic comments.■ Users can search the comments to quickly discover other scenes of interest to them.■ New discoveries can be made by comparing comments, attached videos etc..

comparing with appended videos

■ Can also be accessed on mobile phones and touch-screen smartphones.■ The optional Scene Editor tool incorporates lecture video material from universities

and the like to produce polished results

Can be accessed anywhere by mobile

phones or smartphones

Lecture video slides and lecturer extracted

■ To support e learning aimed at improving sports skills or technical ability*¹

and the like to produce polished results.

Application scenarios

Video analysis: Vk video handling library

smartphones

Can also be used for scene listings

and neatly arranged

■ To support e-learning aimed at improving sports skills or technical ability*¹■ As a forum for the sharing and exchange of knowledge within communities*²■ As a forum for review and lifelong learning tied in with the delivery of university

Video analysis: Vk video handling library

ーーーーーー

ーーーーL id

gor for digest

playback

g g y ylectures*³

■ As an in-house sharing site for corporate training and technology transfer■ As a system for archiving important video content and summarized video clips

ーーーーLong videos cover

a mixture of many different topics Indexes are added by video

analysis to make the content■ As a system for archiving important video content and summarized video clips*1 Tests conducted jointly with NTT Knowledge Square “N-Academy”*2 Tests conducted jointly with Sakuho Town “Farming community support technology”*3 Joint study with Keio University “JOCW”

analysis to make the content easier to grasp

Copyright © 2011 NTT. All Rights Reserved.Contact: cc-forum@lab.ntt.co.jp

3 Joint study with Keio University JOCW