TAU-DANCE:

Tel-Aviv University Multiview and Omnidirectional Video Dance

Database

Leah Bar Shay Rochel Nahum Kiryati School of Electrical Engineering

Tel Aviv University Tel Aviv 69978, Israel

nk@eng.tau.ac.il

January 2005

User’s Guide

© Copyright 2005 by Leah Bar, Shay Rochel and Nahum Kiryati. All rights reserved.

1. Overview “TAU-DANCE: Tel-Aviv University Multiview and Omnidirectional Video Dance Database” has been created to promote scientific research in image and video processing, computer vision, multimedia understanding and other fields. It is potentially useful for research related to motion analysis, video segmentation, multi-camera studies, omnidirectional image analysis and other scientific problems. TAU-DANCE is a collection of video clips showing a man and a woman dance. The database currently contain ten different dances, each filmed from five different views. TAU-DANCE was recorded in VIA, the Vision and Image Analysis Laboratory in the School of Electrical Engineering of Tel Aviv University. The laboratory room is an open-space working environment; the dancers used areas that were not occupied by furniture and by the audience. TAU-DANCE video clips of the same dance are roughly synchronized to a level of about three frames. A few seconds of the ambient sound, as recorded by the cameras, are provided to allow more accurate synchronization. Copyright considerations dictate that TAU-DANCE is provided without the music. Additional information about TAU-DANCE is provided in the rest of this document.

2. The Filming Setup and Views The video clips in TAU-DANCE were filmed using cameras of various types, from five different views. All cameras were set to zoom-out and manual focus. The TAU-DANCE setup is illustrated in Figure 1. Detailed information on the hardware and processing of each view follows.

Video 5Video 3 Video 2 Video 4

Video 1

Sony Handycam VisionTRV98E (Analog PAL High8)

Axis 2420 (Analog PAL) camerawith RemoteReality NetVision-A

Omnidirectional lens.Recorded with

Sony miniDV Handycam VisionDCR-TRV900E PAL

Sony miniDV DigitalHandycam DCR-TRV22E PAL

Sony miniDV Digital HandycamDCR-TRV8E PAL

Canon VC-C4 PTZcamera (Analog PAL).Recorded with a VCR

5.80 m x 9.20 m

Room

Figure 1: The TAU-DANCE setup. TAU-DANCE was recorded in VIA: the Vision and Image Analysis laboratory, School of Electrical Engineering, Tel Aviv University.

View 1 As shown in Figure 1, this view was filmed from the northeastern corner of the room. The camera was mounted on a tripod (Figure 2) and pointed roughly towards the door. Hardware: Camera - Sony miniDV Digital Handycam DCR-TRV8E PAL Audio – captured by the camera microphone. Video – recorded by the camera to MiniDV tape. Processing: The filmed material was transferred to a computer by firewire, edited and coded using the Pinnacle Studio 8 program.

Figure 2: Camera 1 Figure 3: Camera 1 - close-up

Figure 4: Camera 1 - close-up

View 2 This view was filmed from the center of the southern wall of the room (see Figure 1). The camera was attached to a pole connected to the southern wall (Figures 5 and 6). The camera was positioned about 1.5 meters above the floor and pointed towards the center of the northern wall. Hardware: Camera – Canon VC-C4 PTZ Analog PAL Audio – No audio recorded for this view. Video – recorded by a VCR. Processing: The filmed material (VHS VCR tape) was digitized by a Sony miniDV Handycam Vision DCR-TRV900E PAL camera, then captured to a computer (by firewire), edited and coded using the Pinnacle Studio 8 program. The movies created from this view have the same resolution as the other movies, but due to the analog recording of the film, have a somewhat smaller active frame. Figure 6: Camera 2 – close-up Figure 5: Camera 2 – connected to the lower part of the pole

View 3 The camera was positioned by the southern wall of the room, about 1m from the southwestern corner (see Figure 1), at a height of about 1.5 meters above the floor (Figures 7 and 8). The camera pointed towards the northwestern part of the room. Hardware: Camera - Sony Handycam Vision TRV98E Analog PAL Audio – captured by the camera microphone. Video – recorded by the camera to an analog Hi-8 tape. Processing: The filmed material (Hi-8 tape) was digitized by a Sony miniDV Handycam Vision DCR-TRV900E PAL camera. It was then transferred to a computer by firewire, edited and coded using the Pinnacle Studio 8 program. The movies created from this view have the same resolution as the other movies, but due to the analog recording of the film, have a somewhat smaller active frame. Figure 7: Camera 3 Figure 8: Camera 3 – close-up

View 4 As shown in Figure 1, this view was filmed from the southeastern corner of the room. The camera was mounted on a tripod (Figures 9, 10 and 11) and pointed towards the door of the room. Hardware: Camera - Sony miniDV Digital Handycam DCR-TRV22E PAL Audio – captured by the camera microphone. Video – recorded by the camera to MiniDV tape. Processing: The filmed material was transferred to a computer by firewire, edited and coded using the Pinnacle Studio 8 program. Figure 9: Camera 4 – corner Figure 10: Camera 4

Figure 11: Camera 4 – close-up

View 5 This is the omnidirectional (panoramic) view. It was taken from the center of the southern wall of the room (see Figure 1). The camera with the omnidirectional lens was attached to the top part of a pole connected to the southern wall (Figures 12 and 13). The camera was facing down and viewing the whole room (360°) from the height of the lens downwards, except for a blind spot under the camera. Hardware: Camera – Axis 2420 Analog PAL. Lens – RemoteReality NetVision-A omnidirectional lens. Audio – captured with an external microphone connected to a Sony miniDV Handycam Vision DCR-TRV900E PAL. Video – digitized and recorded on the fly with the same Sony miniDV Handycam Vision DCR-TRV900E PAL to MiniDV tape. Processing: The filmed material was transferred to a computer by firewire, edited and coded using the Pinnacle Studio 8 program. Figure 12: Camera 5 – connected Figure 13: Camera 5 – close-up to the upper part of the pole Figure 14: TRV900E – recording Figure 15: External microphone device for camera 5 for camera 5

3. Illumination The illumination of the scene during the filming process was the combination of several different light sources:

• The normal fluorescent lighting of the room, consisting of numerous fluorescent lamps installed all over the ceiling.

• Two 500-Watt halogen lamps mounted on a tripod near the center of the room facing up and northwest (see Figure 16).

• Two 500-Watt halogen lamps mounted on a tripod located at the southern wall near camera 3 (see Figure 17).

• One 150-Watt halogen flash lamp mounted on a tripod located at the southwestern corner of the room, near camera 3 (Figure 17).

Figure 17: Southern wall lighting Figure 16: Center room lighting 4. The Dances TAU-DANCE currently contains 10 dances, of different types, rhythms and lengths. The dances are listed below, with their assigned numbers and lengths (min:sec):

1. English waltz - 03:56 2. Tango - 03:35 3. Rumba - 04:16 4. Cha-Cha-Cha- 06:31 5. Pasodoble (1) - 02:17 6. Pasodoble (2) - 02:10 7. Rock & Roll - 02:31 8. Rumba with Cha-Cha-Cha - 05:00 9. Mambo - 03:20 10. Salsa - 04:20

5. Video Formats Each clip of TAU-DANCE is available at two resolutions. The database thus contains two sets of clips: low and high resolution. The two sets were coded using different codecs. All the clips originated as PAL type were therefore interlaced with a frame rate of 25 fps. Set One – Low Resolution: The movies in this set were coded at a low bit rate and a small frame size. Only one field from the interlaced frame was coded. The video parameters:

• Codec - MPEG-1 (VCD) • Frame resolution - 352x288 • Frame rate - 25 fps • Bit rate - 1150 Kbits/sec

Set two – High Resolution: The movies in this set were coded at a high bit rate and the largest frame size possible. The movies are interlaced. The video parameters:

• Codec - Microsoft MPEG-4 Video Codec V1 • Frame resolution - 768x576 • Frame rate - 25 fps • Bit rate - 6000 Kbits/sec

Movie Structure: Each clip in TAU-DANCE begins with a title slide, shown for 20 seconds. The dance itself is then shown. The clip ends with about half a second (13 frames) of the title slide. 6. Audio The original movies included audio tracks with the music that the dancers had danced to. Due to the music copyright restrictions, the sound tracks had to be removed from the database. However, a few seconds of the ambient sound, as recorded by the cameras, are provided to facilitate accurate synchronization between different views of the same dance.

7. Database Organization and File Names As described in section 5, TAU-DANCE contains two sets of clips, organized in two resolution (top-level) folders. Each top-level folder contains 10 mid-level folders, one for each dance. Each dance folder contains 5 movie files, one for each view of the dance. The database also includes this readme file and a MultiDisplay application (see section 8). Top-level folder names

• High resolution – “TAU_DANCE_mpeg4v1” • Low resolution – “TAU_DANCE_mpeg1”

Mid-level (dance) folder names • “Dance1-English_waltz” • “Dance2-Tango” • “Dance3-Rumba” • “Dance4-Cha-Cha-Cha” • “Dance5-Pasodoble1” • “Dance6-Pasodoble2” • “Dance7-Rock_and_Roll” • “Dance8-Rumba_with_Cha-Cha-Cha” • “Dance9-Mambo” • “Dance10-Salsa”

Movie file names “Dance<#>_View<#>_<codec type>_<resolution>.<mpg/avi> Examples of high resolution (mpeg4) movie file names: Dance1_View1_mpeg4v1_768x576.avi Dance1_View2_mpeg4v1_768x576.avi . . . . . . Dance1_View5_mpeg4v1_768x576.avi . . . . . . Dance5_View1_mpeg4v1_768x576.avi . . . . . . Dance5_View5_mpeg4v1_768x576.avi . . . . . . Examples of low resolution (mpeg1) movie file names: Dance1_View1_mpeg1_352x288.mpg . . . . . . Dance1_View5_mpeg1_352x288.mpg . . . . . . Dance10_View1_mpeg1_352x288.mpg . . . . . . Dance10_View5_mpeg1_352x288.mpg

8. MultiDisplay Player Application We include with the database an application (EXE file) that allows to display five MPEG-1 movies simultaneously on the same screen. This allows watching the same dance from the five views at the same time. The player application uses five Microsoft Media Player objects. If you use Media Player version 9 (or above) you will be able to use the Media player’s controls (buttons, etc.). You need to browse and choose the five movies you wish to view. The application allows you to pause, stop and restart all five movies together. The application is intended for use with Microsoft Windows 98 and above. The recommended configuration is:

• Screen resolution – 1024x768 (works in other resolutions) • Microsoft Media Player 9 and above (works with versions 7, 8)

9. Obtaining and Using TAU-DANCE

• TAU-DANCE is copyrighted by Leah Bar, Shay Rochel and Nahum Kiryati. All rights reserved.

• To obtain a copy of TAU-DANCE, and to obtain permission to use parts of TAU-DANCE in publications and presentations, please contact Prof. Nahum Kiryati: nk@eng.tau.ac.il.

• TAU-DANCE is a scientific project. It comes “as is”, without any warranty.