Multimedia Information System

Yezhou YangYezhou Yang

(Include slides from Ze-nian Liu and Tamara Berg)

Course Info

Instructor: Yezhou Yang; yz.yang@asu.eduOffice: 562 Brickyard

Lectures: Tues/Thurs 6:00-7:15pm, Tempe BYAC 190

Office Hours: Tues/Thurs 2:00-3:00pm and by appt

Course Webpage: http://www.umiacs.umd.edu/~yzyang/course/CSE408(Temporary)

Reference Books

Yezhou Yang

How to pronounce?"Ye" the same as the beginning of "Yes", "Zhou" follows "Drow", and "Yang" is almost the same as "Young".

Assistant Professor, CIDSE, ASU

My main interests lie in Computer Vision, Robot Vision, especially exploring visual primitives in human action understanding from visual input, grounding them by natural language as well as high-level reasoning over the primitives for intelligent agents, aka. Robots.

Assistant Prof. 2016-

Active Perception Group

Pre-Requisites

- CSE 310: Data structures and Algorithms.- You should be comfortable with some programming language (e.g. Matlab, Python, C++…).- I will provide most necessary background for the course as we go.

-Come talk to me if you have any questions!

What is Multimedia?

What is Multimedia Information?

What is Multimedia Information System?

What is Multimedia?

-Multimedia is content that uses a combination of different content forms such as text, audio, images, animation, video and interactive content. Multimedia contrasts with media that use only rudimentary computer displays such as text-only or traditional forms of printed or hand-produced material. (Wiki)

-Digital media – usually recorded, displayed, or accessed using electronic devices, but can also be part of a live performance.

Li, Drew, and Liu16

1.1 What is Multimedia?

• When different people mention the term multimedia, they often have quite different, or even opposing, viewpoints.

‐ A consumer entertainment vendor: interactive TV with hundreds of digital channels available, or a cable TV-like service delivered over a high-speed Internet connection; a smartphone.

‐ A Computer Science (CS) student: applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity.

• Multimedia and Computer Science:‐ Graphics, HCI, visualization, computer vision, data

compression, graph theory, networking, database systems, Natural Language Processing, Robotics...

16

Li, Drew, and Liu17

Components of Multimedia

• Multimedia involves multiple modalities of text, audio, images, drawings, animation, and video.

Examples of how these modalities are put to use:1. Geographically-based, realtime augmented-reality, massively

multiplayer online video games.2. Shapeshifting TV, where viewers vote on the plot path.3. Tele-medicine.4. A camera that suggests what would be be best type of next

shot.5. A web-based video editor that lets anyone create a new video

by editing, annotating, and remixing editable professional videos on the cloud.

6. Cooperative education environments that allow schoolchildren to share a single educational game using two mice at once.

17

Li, Drew, and Liu18

7. Searching (very) large video and image databases for target visual objects, using semantics of objects.

8. Compositing of artificial and natural video into hybrid scenes.

9. Visual cues of video-conference participants, taking into account gaze direction and attention.

10. Making multimedia components editable — allowing the user side to decide what components, video, graphics, and so on are actually viewed = making components distributed.

11. Building “inverse-Hollywood” applications that can recreate the process by which a video was made.

18

Li, Drew, and Liu19

Multimedia Research Topics and Projects

• To the computer science researcher, multimedia consists of a wide variety of topics:

1. Multimedia processing and coding: multimedia content analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc.

2. Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases.

3. Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems.

4. Multi-modal interaction and integration: “ubiquity” — web-everywhere devices, multimedia education including Computer Supported Collaborative Learning, and design and applications of virtual environments.

19

Li, Drew, and Liu20

Current Multimedia Projects

• Many exciting research projects are currently underway. Here are a few of them:

1. Camera-based object tracking technology: tracking of the control objects provides user control of the process.

2. 3D motion capture: used for multiple actor capture so that multiple real actors in a virtual studio can be used to automatically produce realistic animated models with natural movement.

3. Multiple views: allowing photo-realistic (video-quality) synthesis of virtual actors from several cameras or from a single camera under differing lighting.

4. 3D sentiment- and speech-capture technology: allow synthesis of highly realistic facial animation from speech.

20

Li, Drew, and Liu21

5. Specific multimedia applications: aimed at handicapped persons with low vision capability and the elderly — a rich field of endeavor.

6. Digital fashion: aims to develop smart clothing that can communicate with other such enhanced clothing using wireless communication, so as to artificially enhance human interaction in a social setting.

7. Distributed medical care: an initiative for providing interactive health monitoring services to patients in their homes

8. Augmented Interaction applications: used to develop interfaces between real and virtual humans for tasks such as augmented storytelling.

21

What is Multimedia information?

We will especially focus on multimedia information as accessed via modern devices (such as computer, smartphone, personal robot etc.)

including algorithms for storing, organizing, retrieving, and manipulating various forms of media.

We will discuss the fundamentals of each type of digital media plus a selection of special topics such as retrieval, modeling, and manipulation.

What is Multimedia system?

Traditional Multimedia Systems

Natural Language Text

Sound

Image

Video

Etc.

Intelligent Multimedia Systems

Natural Language Text

Sound

Image

Video

Etc.

Customer Segments and Value Proposition of Multimedia Systems• Who is its customer?• What is the key technology?• Why will they buy it?

End Users?Decision Makers?Payers?

What Customer Problem are these Multimedia Applications helping to solve?

What Customer needs are these Multimedia Applications satisfying?

Image Retrieval System (Image Search)

Realtime video-chat augmenting system

Augmented Reality App.

Amazon Echo voice assistant system

Autonomous Driving Car

What is Multimedia?

What is Multimedia Information?

What is Multimedia Information System?

Course contents

● Visual media (images and videos)

● Textual media (texts )

● Acoustic media (audio )

● Other media (tactile etc.)

● Media features

● Media based similarity, recognition, and indexing

● Media compression, storage and basic operations (ffmpeg)

● Media information visualization

● Interactive and active multimedia systems

● Multimedia applications design and value analysis

Text

Text Basics – words and more, intro to NLP/text processingTons of text on the web. How can we access it effectively?

Web document retrieval (including the algorithm that started Google!)

Sound

• Sound – What is sound?– How do we record sound in a digital device?– How can we filter/manipulate sound?– Create your own composition through direct

wave manipulation/combination

Images/Video

• Image formation & the camera• Popular file formats & compression• Consumer photo sharing

Image from Alyosha Efros

Image by katiew – Flickr.com

Image Manipulation

Image blending and compositing

1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)

Composite by David Dewey, Slide from Alyosha Efros

2. Blend them into the composite (in the right order)

Image Manipulation

Combined Media

Various types of media often appear together

Web pages with text & images/videoimage search can be implemented as text searchon nearby words, or as text + image analysis

Movies/games – combination of images, video, sound

Education – can use a combination of media for effective teaching

Medicine – virtual surgery for training and tele-surgery

Other kinds of Digital Media

Social MediaTagging & AnnotationLocation InformationInteractionTactile SensingRecommendation Systems

Workload

Assignments:There will be 3 programming assignments and a couple homeworks, covering the core types of digital media we will be studying.

All assignments should be submitted by email to:

ASUCSE408@gmail.com

1. Assignments/Quizzes: 10%2. Group projects: 45%3. Midterm: 15%4. Final: 30%

Decisions on borderline grades will be influenced by class attendance and participation.

Discussing and exchanging ideas within group members are encouraged. However, except if specifically allowed by the instructor, copying or rephrasing from any outside sources (e.g., fellow students, Internet, etc.) on any material to be graded is not permitted, and will be considered plagiarism. Any kind of plagiarism or cheating attempt will be severely dealt with, which would normally lead to an F in the class. When using third party software, please make it clear how it is used in your project and how much it contributes overall to the project. More details on the Academic Integrity Policy can be found under the following link: https://provost.asu.edu/academic-integrity/policy.

CSE 408

NLP

Signal Processing

ComputerVision

AI Robotics

The Magic Door!