ghent university and gugc-k: overview of teaching and research activities

Intro MMLab, iMinds & UvdT

Ghent University and GUGC-K:Overview of Teaching and Research ActivitiesResearch SeminarKAIST, 18 August 2015

Wesley De Neve@wmdeneve

Ghent University iMinds & KAIST

ELIS Multimedia Lab

#ELIS Multimedia Lab

1

Teaching activitiesGhent University Global CampusGhent University Home Campus

Research activitiesGhent University Home CampusGhent University Global CampusOutline


2




3

WHO?


Ghent University, BelgiumRector: Prof. Anne De PaepeVice-rector: Prof. Freddy MortierGhent University Global Campus, KoreaCampus President: Prof. Jozef VercruysseCampus Vice-president: Dr. Thomas Buerman


5

WHERE?



Incheon Global Campus (IGC)University of UtahGeorge Mason UniversityGhent UniversitySUNY at StonybrookUniversity of Nevada


Bachelor

Master

PhD

Environmental Technology

Molecular Biotechnology

Food Technology


Molecular Biotechnology

Food Technology

Bachelor

Master

PhDDouble AccreditationResident and Flying FacultyGhent University DegreeQuality ControlGhent University AppointmentIntegrated Research PlanEnvironmental Technology


Research-focused programPractical excersises in laboratoriesGraduation projectDouble accreditationNVAO January - August 2013MoE March November 2013Ghent University degree

Company internshipsOne semester in Belgium



#ELIS Multimedia LabEnglishBiologyMathematicsInorganic chemistryOrganic chemistryInformaticsPhysicsBiochemistryMolecular biologyGeneticsStatisticsEconomicsMarketingModelingSimulationProcess engineeringLegislationProcess technologyEntrepreneurshipIntellectual propertyProject management

Process control

#ELIS Multimedia LabTeaching Activities

Informatics 1(Fall term 5 credits)Informatics 2(Spring term 5 credits)


15



#ELIS Multimedia LabCourse contentmanagement, analysis, and visualization of large-scale datasets

Lecture on the art of (deep) machine learning

Hands-on sessionword2vec for natural language processing (NLP)

Apache SparkTeaching Activities

Big Data Science(Spring term)

#ELIS Multimedia LabTeaching activitiesGhent University Global CampusGhent University Home Campus


#ELIS Multimedia LabTerrain classification for Hyperspectral imagesViktor Slavkovikj

#ELIS Multimedia LabHyperspectral imageseach pixel contains hundreds of measurements of the electromagnetic spectrumoften captured through remote sensing e.g., through a camera mounted on an airplane

Problem: how to do terrain classification?e.g., corn, wheat, and woodsProblem Statement

#ELIS Multimedia LabArchitecture Convolutional Neural Networkinput layerconvolutional layerconvolutional layerconvolutional layerfully connected layerfully connected layeroutput layeroutput: one out of16 terrain classes800 hidden units(hyperbolic tangent)800 hidden units(hyperbolic tangent)filter size: 9x16filter size: 1x16filter size: 1x16input: 9 pixels andtheir spectral bandsimplementation: by means of Python and Lasagne, a lightweight library to quicklybuild and train neural networks in Theano

#ELIS Multimedia LabDebugging the CNN

#ELIS Multimedia LabData augmentation through the addition of Gaussian noiseminor impactsimilar observation for max-pooling, ReLUs, and DropOut

Classification results on par with the state-of-the-artoverall accuracy between 80% and 95%

Experimental ResultsIndian PinesTest results5%training data10%training data20%training dataNon-augmentedOverallaccuracy (%)85.46 1.7392.76 0.9396.54 0.47AugmentedOverall accuracy (%)86.54 0.3092.70 1.0096.58 0.55

#ELIS Multimedia LabVideo content understandingBaptist Vandersmissen

#ELIS Multimedia LabGoals

Representation Learning using Neural NetworksSpatial & Temporal Feature ConstructionGeneration of Fine-grained DescriptionsFocus on Video Content Understandingobjects, actions,& scenes

#ELIS Multimedia LabTechniques

Main focus is on neural network techniques that are able to capture temporal behaviour3-D Convolutional Neural NetworksRecurrent Long Short-Term Memory NetworksConvolve over spatial (2D) and/or temporal domain (3D) to acquire knowledge of inputProcess sequence of inputs and acquire knowledge based on memory cellsRecurrent Reservoir Computing NetworksRandomly assigned weights in the reservoir, combined with a readout layer using linear regressionbaseline video features: IDTF, AlexNet (ImageNet), C3D (FAIR)implementation: Theano, Caffe, and Lasagne


Data

Focus on

Action recognition datasetCrawled Vine videosRealistic action videosSocial and mobile contentWell-known and widely usedNoisy and short-form data

UCF101

#ELIS Multimedia LabFirst Exemplary ApproachConvolutional Neural NetworkLong Short-Term Memory Networkf1fnf2

videoRepresentation f2Representation f1Representation fnVideo RepresentationClassification

#ELIS Multimedia LabSecond Exemplary ApproachConvolutional Neural NetworkClassificationConvolutional Neural Networkf1fnf2m1 mkm2raw framesmotion flowsFusionVideo Representation

#ELIS Multimedia LabReservoir computing for video Event detectionAzarakhsh Jalalvand

#ELIS Multimedia LabGoaldetect the status of a door: open, closed, half-openuse of a simple, efficient, and effective system

Approachuse of a fixed low-resolution camera (3030 pixels)privacy reasons: people are not recognizablelow bandwidth needed to stream the datause of Reservoir Computing Networks (RCNs)good in modeling temporal information (cf. speech)good in dealing with noisy data

Video Event Detection (1/2)

#ELIS Multimedia LabImplemented solution: small neural network of 200 nodesfast trainingreservoir: random assignment of connection weightsreadout layer: gradient descent for linear regressionreal-time responserobust against noiselow light conditions & people occurring

Video Event Detection (2/2)

Reservoir

#ELIS Multimedia LabDemo


Reservoir computing for visual content analysis

handwritten digit recognition (MNIST)

house number detection and recognitionNext Steps

#ELIS Multimedia LabTwitter micropost modelingFrederic Godin

#ELIS Multimedia LabProblem statementCurrent Natural Language Processing (NLP) research focuses on clean text: news articles, Wikipedia articles

What about noisy, short-form, and unstructured microposts?

Lack of correct spelling, a lot of slang

Lack of context

Lack of consistent grammar rules (~structure)

#ELIS Multimedia LabA simple, general but effective neural network architecture (1)

Use Googles word2vec (=simplified neural network) to generate good feature representations for words (=unsupervised learning)

Feed word representations to another neural network (NN) for any classification task (=supervised learning) TweetFeature representationMachine learning:classificationLabelLearn word2vec word representations once in advanceTrain a new NNfor any NLP task

#ELIS Multimedia LabA simple, general but effective neural network architecture (2)

W(t-1)W(t)W(t+1)Look

upN-dimN-dimN-dimFeed forward neural networkLabel(W(t))TweetFeature representationMachine learning:classificationLabelConcatenate (3N-dim)Window = 3fromSeoultoIm going from Seoul to Daejeon. #KTX

#ELIS Multimedia LabWord2vec: automatically learning good features

Model trained on 400 million tweets having 5 billion words2-D projection of a 400-D space of the top 1000 words used on Twitter

#ELIS Multimedia LabPart-of-Speech tagging: is it a verb, noun or article?

ImgoingfromLook

up400D400D400DFFNN:400 hidden nodesVerbslang

NIPS Workshop on Modern Machine Learning Methods and Natural Language Processing

#ELIS Multimedia LabNamed Entity Recognition: is it a location, company or TV show (1)?

fromSeoultoLook

up400D400D400DFFNN:400 hidden nodesLocationThe same word representationsThe same network, but with different weights

#ELIS Multimedia LabNamed Entity Recognition: is it a location, company or TV show (2)?

Used both standard features as word representationsOnly using word representationsACL 2015 Workshop on Noisy User-generated Text

#ELIS Multimedia LabNext Steps

Replace word2vec word representations with character representations

Use Convolution Neural Networks as pattern filters, to prevent ahuge increase in vocabulary size (e.g., a convolutional filter should beable to map the" and "da" onto the same pattern)

Combine character representations to form word representations that can be classified

#ELIS Multimedia Labhumor detection on twitterAbhineshwar Tomar

#ELIS Multimedia LabObservationlots of humor on Twitter

Questioncan we automatically detecthumorous tweets?

Motivationhumor is engaging (ads!)creation of intelligent agentswith social & emotional skills

Humor Detection on Twitter


Different kinds of humorsarcastic humorblack humorself-deprecating humorsatireparodyPersonal contextMultimodal tweetsLanguage usageWhy Humor Detection on Twitter Is Challenging

#ELIS Multimedia LabBinary classification problem: humorous or non-humorous

Collection of tweets in Englishtweets containing #lol, #rofl, #lmao, #funny, #hilarious, dataset of 373,498 tweets50/50 humorous and non-humorous

Featuresword2vec

Classification techniquefeed-forward neural network with ReLUsApproach (1/2)

#ELIS Multimedia LabApproach (2/2)300-D tweet vectorGooglesword2vecHumorous/non-humorousFeed-forward neural network300-D input layer400-D hidden ReLU layer350-D hidden ReLU layer200-D hidden ReLU layer2-D output layerTweetPlease kill Jar Jar Binks please


Classification accuracy: 81.07%Preliminary ResultsHumorous TweetsYou know you're at a Croatian jam whn your uncle forces you to take shots .....I've finally learned how to play spadesWatermelon inside of a watermelon!! My fav vine!Some boys will wear dark sunglasses in Church, then be blaming God later when they end up as WeldersIt's so weird to thing that over in the other side of the country there are people going to sleep while I'm getting upGot a new TV set for downstairs and my dad said "I bet I can do this in 15 minutes" and almost 1 hour later it's nearly finished#RapLikeLilWayne I walk while I sleep. Call that Sleep walkin!!!! #whaddup


Collect more training data by making use of Reddit

Experimentation with recurrent neural network techniques

Multimodal word/concept vector representations, integrating both textual and visual information

Next Steps

#ELIS Multimedia Labmultimodal Condition monitoring for Wind turbinesOlivier Janssens


Healthy wind turbineBroken wind turbineMulti-sensor monitoring of bearings to detect faults early oninfrared imaging, vibration data, and temperature dataClassificationwhite box models: random decision forests and SVMsblack box models: CNNs

Condition Monitoring: Failure Prevention


52

Condition Monitoring: Smearing Fault Detection


53

Infrared imaging analysishandcrafted features + SVM: accuracy of 88.25%

Vibration data analysishandcrafted features + RDF: accuracy of 87.25%CNN: accuracy of 91.77%

Ongoing research: ensemblingcreation of a multimodal system using early and/or late fusionSome Observations

#ELIS Multimedia Labgenomic data compressionTom Paridaens


Challenge: data handlingDNA sequencing is outrunningDNA storage, transmission, andanalysis

Research questionhow about compressing DNA by making use of video coding tools in order to alleviate storage, transmission, and analysis problems?

Problem Statement


56

Modular and extensiblethanks to the use of the pipes and filters design patternBlock-based compressionallows selecting the best coding tool per block (adaptivity)enables random access, streaming, and parallel processing

Codec Architecture (1/2)Input filterEncoding filterPipeOutput filterPipePipePipeStatistics

#ELIS Multimedia LabCodec Architecture (2/2)

EfficiencyFunctionalityEffectiveness

Proposed solution

SOTAallowing for a flexible trade-off betweenefficiency, effectiveness, and functionalityhas always been a major design goal


58

Effectiveness: compression of the human Y chromosome

Efficiency< 3 minutes: 4.30 MB10 minutes: 4.21 MB7 hours: 3.75 MBExperimental ResultsFormatFile size (MB)No compression (FASTA)18.70Binary7.01Huffman5.16Proposed framework (December 2014)4.26Proposed framework with CABAC (August 2015)3.75

#ELIS Multimedia LabCompressionsupport for the protein alphabetperformance optimizations (I/O, GPU)

Privacy protection and streamingencryption

Compressed-domain manipulationonly download and decode that part of the compressed genome that belongs to a particular gene (region-of-interest)

DCC + MPEG standardizationFuture Activities

PastFuture


60



#ELIS Multimedia LabDeep Learning for Biotech DataDeep machine learningMultimedia dataBiotechdataSongdoGhentimportant: unique (specialized) use cases and corresponding data sets, given the current speed of change in the field of deep learning

#ELIS Multimedia LabUse Case 1: Quantification of Parasite Movement

#ELIS Multimedia LabWhat?commonly used in chemistry to create a fingerprint by which molecules can be identified

Applicationsmedical diagnosis and food analysis (a/o)

Use Case 2: Raman Spectroscopy (1/2)

#ELIS Multimedia LabChallenges?data: vast amounts of datadevice: different devices,different characteristicsnoise: environment, side effectscomposite materials:overlapping signals

Goalnoise-robust automatic Raman spectrum identification using signal processing and machine learning techniques

Use Case 2: Raman Spectroscopy (2/2)

#ELIS Multimedia LabiMinds & ETRI

R&D collaboration in the field of IoT,Big Data, and network communication (5G)joint international research labs(in Songdo?)

#ELIS Multimedia LabMemoranda of Understanding (MoU)

Joint research projects (legal status GUGC-K?)

Joint doctoral degrees

Visits of masters and Ph.D. students in Spring 2016?GPU cluster in Songdo (for deep CNNs, a/o)4 Xeon CPUs8 Titan Black GPUs with 96 GB of memory128 GB of system memory2 TB SSD + 16 TB of storage capacity3200 Watt of power consumption

Further Ideas


ELIS Multimedia Lab
