IST 402: Multimedia Big DataJames Z. Wang

College of Information Sciences and TechnologyThe Pennsylvania State University

wang.ist.psu.edu

Why Big Data is Indeed a Big Deal?

• Forbes 10/19/2014, “84% Of Enterprises See Big Data Analytics Changing Their Industries' Competitive Landscapes In The Next Year”• http://www.forbes.com/sites/louiscolumbus/2014/10/19/84-of-enterprises-see-

big-data-analytics-changing-their-industries-competitive-landscapes-in-the-next-year/• Big Data Analytics + The Internet of Everything• Industrial Internet Insights Report for 2015, by GE and Accenture

Strong Investment in Big Data

Multimedia Big Data is in Every Field

Sofia thanks Saudi for her new citizenship

How to Teach IST Students about this Important Topic?

Participation (30%) – Presentation (30%) – Project (40%)

• Hand-on labs• Big data computing platform: LAMP: LINUX/APACHE/MySQL/PHP• Image processing: MATLAB• Fuzzy systems: MATLAB• Computer Vision: OpenCV• Machine learning: WEKA• Deep learning: planned but were unable to carry out

• Most students reported having no programming experience• Prerequisites: IST 210 (databases) and IST 220 (computer networking)

• Recently published research articles are assigned as reading assignments• 10 graded quizzes to test their comprehension

Topics Covered in Lectures

• Why multimedia big data• Image processing basics• Machine learning basics• Fuzzy systems and applications in meteorology• Image retrieval and applications in social media, Web information retrieval,• Image annotation and applications in robotics, autonomous vehicles, smart

cities, forensics, biomedicine• Visual arts and art history (analysis of van Gogh’s brushstrokes)• Inferring aesthetics and emotions

Participation (30%) – Presentation (30%) – Project (40%)

• Each student prepares and delivers one presentation and lead the discussions on a subject• The instructor provides a few articles to start their study• These articles are to be used as seeds for finding other articles related to the

topic assigned

Topics Covered by these Presentations

• Speech Recognition• Human Pose Estimation• Image Recognition and Video Analysis• Natural Language Processing, Speech Recognition• Local Feature Extraction, Salient Object Detection, and Edge Detection• Semantic Segmentation• Visual-Textual Sentiment Analysis, and Social Media• Image Retrieval• Image Synthesis, Style Transfer

Participation (30%) – Presentation (30%) – Project (40%)

• Teams of up to four students• Open-ended topics: retrieval, analysis, visualization, understanding, mining,

…• In consultation with the instructor, each team select a problem • The problem should attempt to exploit digital content’s relationship to space,

to time, to authorship, and/or to other media types or context• The project can leverage the interactive and networked nature of some media

types• End of the semester: a peer-reviewed talk/demo and a written report• Students are expected to learn a good amount of information or knowledge,

with the guidance of the instructor, outside of the class

11 Team Projects

• Use machine learning to guess the filter applied to a photo on social media• Use deep learning to separate the vocal and background instrument parts of a song• Use face recognition to identify faces from twitter• Use deep learning to recognize objects (e.g. pizza) in social media photos • Use deep learning to recognize player numbers and identify players from jerseys in

the video• A mobile-based social media app that can automatically link people with uploaded

group images using face recognition• Use deep learning to classify dangerous sounds from smart cities’ microphone

networks and alert the human user (police)• Use deep learning to identify different types of dogs from images• Use the IBM Watson for real-time social media sentiment analysis• Generating music playlist according to the automatically-sensed mood in an album• Automatic traffic monitoring and classification for smart cities

“This class was phenomenal and I learned so much about something I never thought I could or would have the opportunity to understand. We need to cover more cutting edge topics like this in IST with consistency!”

--- Anonymous student comment about the course, left on the course evaluation form