visualization for multimedia contents
TRANSCRIPT
![Page 1: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/1.jpg)
Visualization for Multimedia Contents
Takayuki ItohOchanomizu University, Japan
Shonan NII Meeting 2014/3/10
Itoh Laboratory, Ochanomizu University
![Page 2: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/2.jpg)
This talk introduces our works on…
4
Itoh Laboratory, Ochanomizu University
Music browsingPhotograph browsing
[AVI 10, ACM SAC 11 etc][ACM SAC 11, IV 11 etc]
![Page 3: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/3.jpg)
5
Itoh Laboratory, Ochanomizu University
Part I:Photograph browsing
![Page 4: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/4.jpg)
Photograph browsing tools
• To browse large photo collection intuitively
6
Itoh Laboratory, Ochanomizu University
PhotoMesa Design Gallery
![Page 5: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/5.jpg)
Photograph browsing tools
• To browse large photo collection intuitively• Personal or professional• Structured (tree, graph, time-sequence…) or scattered• Image processing algorithms
– Meta-data based: (time, place, person …)– Low-level feature based: (color, texture, shape …)– High-level recognition based: (human face, object, scene …)
7
Itoh Laboratory, Ochanomizu University
PhotoMesa Design Gallery
![Page 6: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/6.jpg)
Our zooming interface for photo exploration
• Overview the photo groups first, then zoom in– [Zoom-out] display representative photos of groups– [Zoom-in] display each photos in the zoomed groups
8
Itoh Laboratory, Ochanomizu University
[IV08]
Zoom Out
Zoom In
![Page 7: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/7.jpg)
Tree visualization for photo browsing
• Hierarchy representation by nested rectangles• Mapping photos onto icons (= leaves) (for zoom in)
• Mapping photos into rectangles (= branches) (for zoom out)
9
[TVCG04, CG&A06]
Itoh Laboratory, Ochanomizu University
Photo on icon
Photo in rectangle
Hierarchy representation
![Page 8: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/8.jpg)
Photo collection as “life log”
• Many photos by travel, events, ...• Often we reflect our life from “When, Where, Who”
↓• Enjoyable life reflection by looking at photos
from the viewpoints of “When, Where, Who”
10
Visit to Kyoto every year
Hokkaido with family in 2007
Hawaii with friends in 2009
High school travel to Okinawa
Itoh Laboratory, Ochanomizu University
![Page 9: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/9.jpg)
Itoh Laboratory, Ochanomizu University
Photo browser by “When, Where, Who”
• Photo display based on time and place
• Smooth interaction(Rotation, zoom)
• All-in-one display of faces in the photos
• Photo discovery by interaction among time, place, and face
11
Extracted facesY: Latitude
X: Longitude
Z: Time
Photo display
Rotation
Zoom in
Zoom in
[AVI 10, ACM SAC 11 etc]
![Page 10: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/10.jpg)
Life log of a student (1/2)
12
Europe with family
Guam with friends
Domestic trip in lab.
Europe with family
Itoh Laboratory, Ochanomizu University
Lab.
Friends
FamilyFamily
Photograph owner
[AVI 10, ACM SAC 11 etc]
![Page 11: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/11.jpg)
Life log of a student (2/2)
13
Mid. America with family
Japan & Korea trips in lab.
Europe with friends USA trip
in lab.Japan with
friends
Australia
Itoh Laboratory, Ochanomizu University
Friends
FamilyFamily
Lab.
[AVI 10, ACM SAC 11 etc]
![Page 12: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/12.jpg)
14
Itoh Laboratory, Ochanomizu University
Part II:Music browsing
![Page 13: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/13.jpg)
Music browsing tools in my lab
15
Itoh Laboratory, Ochanomizu University
Overview & summarization
Recommendation & preference analysis
Visual user interface
![Page 14: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/14.jpg)
Music browsing tools
• Quick finding & understanding before listening• Goal & purpose
– [Daily use] search, recommendation, …– [Expert use] education, practice, performance, …
• Input data– Audio (WAV, MP3 …) or Note (MIDI …)
• Musical genre– Pops or Classics– With vocals or Without vocals
16
Itoh Laboratory, Ochanomizu University
![Page 15: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/15.jpg)
Music browsing tools in my lab
17
Itoh Laboratory, Ochanomizu University
Overview & summarization
Recommendation & preference analysis
Visual user interface
![Page 16: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/16.jpg)
Visualizing structure of orchestra music
• Summarization of classical music score which has tens of note tracks
• Coloring of note blocks based on melodies
18
Woodwind
Brasswind
Strings
Former Latter
Itoh Laboratory, Ochanomizu University
[IV 11]
![Page 17: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/17.jpg)
Example: “Valse des fleurs”
1919
Melody Harmony Bass Others
Remark to Purple melody
Itoh Laboratory, Ochanomizu University
[IV 11]
![Page 18: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/18.jpg)
Condensation (= Vertical compression)
• Reduce the number of tracks while preserving important melodies
20
Itoh Laboratory, Ochanomizu University
[IV 11]
![Page 19: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/19.jpg)
Condensation (= Vertical compression)
• Reduce the number of tracks while preserving important melodies
21
Itoh Laboratory, Ochanomizu University
[IV 11]
![Page 20: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/20.jpg)
Visual Music Recommendation
• Input evaluation→ Learn & recommend preferable tunes
22
Itoh Laboratory, Ochanomizu University
Plot = tune
Automaticallygenerated
playlist
Not yet Playing Negative Positive
Buttons (play, stop,positive, negative)
[VINCI11]
![Page 21: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/21.jpg)
Recommendation patterns
• Users A&B have similar preferences– Analog sounds– Simple arrange
• User C has much different preference– Low inharmonicity
23
Itoh Laboratory, Ochanomizu University
Loudness Loudness Loudness
High/low ratio High/low ratio Inharmonicity
User A User B User C
[VINCI11]
![Page 22: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/22.jpg)
Conclusion
• Photograph browsing– Zooming interface [Tree vis.]– Meta-data based structuring
• Music browsing– Summarizing scores [Time-varying vis.]– Recommendation with scatterplots [Multi-dimensional vis.]
24
Itoh Laboratory, Ochanomizu University
![Page 23: Visualization for Multimedia Contents](https://reader031.vdocuments.us/reader031/viewer/2022011723/61d2c758d4f8cf04b06896d0/html5/thumbnails/23.jpg)
References
Hierarchical Data Visualization Using a Fast Rectangle-Packing Algorithm, TVCG, 10(3), 302-313, 2004.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE13.pdfHierarchical Visualization of Network Intrusion Detection Data in the IP Address Space, CG&A, 26(2), 40-47, 2006.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE14.pdfCAT: A Hierarchical Image Browser Using a Rectangle Packing Technique, International Conference on Information Visualisation (IV08), 82-87, 2008.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE37.pdfA Personal Photograph Browser for Life Log Analysis based on Location, Time, and Person, ACM Symposium on Applied Computing, 1250-1257, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE58.pdfColorscore - Visualization and Condensation of Structure of Classical Music, International Conference on Information Visualisation (IV2011), 420-425, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE65.pdfMusiCube: A Visual Music Recommendation System featuring Interactive Evolutionary Computing, Visual Information Communication - Information Symposium (VINCI'11), 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE67.pdf
25
Itoh Laboratory, Ochanomizu University