Multimedia Data Mining using Deep Learning

Presented by Bhagyashree V. Barde

Contents

IntroductionData MiningMultimedia mining•Text Mining•Image MiningShallow artificial Neural Networks modelsDeep artificial Neural Network modelsConclusion

Introduction

• There is lot of multi-media data such as image, text, audio, video, etc.

• To extract meaningful information from that data we use various techniques.

• One of them is Deep learning.• Deep learning is a new area of machine

learning

Data Mining• Data: It is a collection of numbers, words, measurements, observations or even just descriptions of things.• Information: Organized data that has meaning.• Knowledge: Information can be converted into knowledge about

historical patterns and future trends.• Data Mining: The task of discovering interesting patterns from Large

amounts of data

Contd..

Fig. knowledge discovery process

Contd..1.Data cleaning: Noise data and irrelevant data are removed from the collection.2.Data integration: Multiple data sources, often heterogeneous, may be combined in a common source.3.Data selection: The data relevant to the analysis is decided on and retrieved from the data collection. 4.Data transformation: The selected data is transformed into forms appropriate for the mining procedure.5.Data mining: It is the crucial step in which clever techniques are applied to extract patterns potentially useful.6.Pattern evaluation: Strictly interesting patterns representing knowledge are identified based on given measures.7.Knowledge representation: Discovered knowledge is visually represented to the user.

Data Mining Techniques

• Characterization• Discrimination• Classification and Prediction• Cluster analysis• Outlier analysis• Association analysis• Evolution analysis

Multimedia Mining

Fig. Categories of Multimedia Data Mining

Multimedia data mining

Static media Dynamic media

Image mining

Audio Mining

Video Mining

Text Mining

Contd..

• Text mining• Image mining• Video Mining• Audio mining It is a technique by which content of an audio

signal can be searched, analyzed, etc.

Converting Un-structured data to structured data

• Data resides in fixed field within a record or file is called structured data.

• Unstructured data means pixel representation for an image, audio, video and character representation for text

Data Mining tool

Structured data

Unstructured data

Multimedia data mining process

Raw data

Training set

Model

Data collection

Data Pre-processing

1.Data cleaning2.Feature extraction

Machine learning

Architecture for multimedia data mining

Input Multimedia contents

Text Image Audio Video

Spatiotemporal segmentation

Feature extraction

Evaluation of resultFinding the similar patterns

Contd..• Input• Multimedia content : It is selection stage which require user to

select the databases or subset of fields .• Spatiotemporal segmentation : It is useful for object

segmentation. It is nothing but moving objects in image sequences in the video.

• Feature extraction : It is preprocessing step.• Finding similar patterns : It include some approaches of

finding similar pattern contain classification, clustering, etc.

Text Mining

• Text Mining is to process unstructured information, extract meaningful information from the text.

Fig. Text mining process

Contd..• Text preprocessing Tokenization Stop word removal Stemming • Text transformation • Feature selection • Text mining methods • Evaluation

Techniques used in textmining

• Information Extraction : Analyze unstructured text and then finding relationships within text.

• Categorization : Assign one or more category to text document.

• Clustering : It find groups of documents with similar content

• Visualization : It improve and simplify the discovery of relevant information

• Summarization : It reduce the length and detail of a document.

Image Mining

• Image Mining is an extended branch of data mining that is concerned with the process of knowledge discovery concerning images.

• Image Mining deals with the extraction of image patterns from a large collection of images.

Contd..

Query Image

Feature Extraction

QueryImage

Features

Image collection

Similarity Matching

Query Image Features

Feature Extraction

Feature Databases

Fig. Content Based Image Retrieval System Architecture

Shallow artificial neural network

• Many learning schemes use shallow artificial neural network.

• Shallow artificial neural network has only one hidden layer.

Fig. General symbol of neuron

Contd..

• If ANN becomes complicated then it tends to be slow and are prone to over fitting.

• It starts to capture noise instead of relationships between image.

• They are often incapable to extract meaningful patterns from high-dimensional input .

What is Deep learning?• Deep learning is a machine learning technique .

• Deep learners are a type of artificial neural networks with multiple layers.

• Multiple layers learn representations by increasing the level of abstraction from one layer to another.

Contd..

• Multimedia data mining has been used for image tagging.

• Tagging has become a standard mechanism on the Internet for annotating multimedia data and search engines rely on tags to retrieve multimedia data.

• Image caption generation is the process of generating a descriptive sentence of an image.

• For that we are using Deep learning.

Deep artificial neural network

• It consists of multiple hidden layer.• It works for decision making.• They take an array of numbers that can

represent pixels and run a series of functions on that array .

• It gives one or more number as output.• The outputs are usually a prediction of some

object that you are trying to guess from input.

Contd..• In deep neural network there are multiple hidden

layer with lowest layer takes the raw data like images, text, sound, etc.

• Then each neurons stores some information about the data they encounter.

• Each neuron in the layer sends information up to the next layers of neurons .

• So the higher you go up, the more abstract features you learn.

Convolutional neural network

• It is a type of feed forward artificial neural network.

• Variations of multilayer perceptron which are designed to use minimal amounts of preprocessing.

• It take fixed size input and generate fixed size output.

Recurrent neural networks

• Recurrent networks has connections that feed back from the output to the input layer and also input layer feed back to themselves.

• It allows loops.• It use their internal memory to process arbitrary

sequences of inputs.• It can handle arbitrary input, output length.• It required much more data to give better result.• It is more complex model.

Conclusion

• Deep learning has proven to be suitable for problems where shallow learners didn’t provide satisfactory results.

• The combination of convolutional network and RNN has yielded very promising results in many domains.

• one drawback is that these methods mostly used supervised approaches.

Thank you