cse 252c: advanced computer visioncseweb.ucsd.edu/~mkchandraker/classes/cse252c/...cse 252c, sp20:...

Lecture0:Introduction

CSE252C:AdvancedComputerVision

ManmohanChandraker

CSE252C,SP20:ManmohanChandraker

Virtualclassrooms

• VirtuallecturesonZoom

– Onlyhostsharesthescreen

– Keepvideoturnedoffincaseofbandwidthissues

– Microphonesmutedunlessspeakingtoreducenoise

– Butpleasedospeakup(remembertounmute!)

– Slidestobeuploadedonwebpagejustbeforeclass

– WhiteboardthroughcameraandiPadtowritethings

• LecturesrecordedanduploadonKaltura

– Availableunder“MyMedia”onCanvas

CSE252C,SP20:ManmohanChandraker

Virtualclassrooms

• VirtualinteractionsonZoom

– Askandanswerplentyofquestions

– Trytohavein-classdiscussions

– “Raisehand”featureonZoomwhenyouwishtospeak

– InstructororTAscanunmute

– Postquestionsonchatwindow

– TAwillhelpkeeptrackofraisedhandsandchatwindow

– Ifinstructorcannotnotice,TAwillinterruptatlogicalpause

• Happytotryothersuggestions

CSE252C,SP20:ManmohanChandraker

Enrollmentlogistics

• Waitlist

– Thereisalongwaitlistfortheclass

– Departmenthasnotagreedtoaddmorestudents

– Youarewelcometoattendlecturesevenifonwaitlist

– TolimitTAworkload,wecangradeonlyenrolledstudents

• Canvas

– Allenrolledandwaitlistedstudentsshouldhaveaccess

• AllannouncementswillbepostedonPiazza

– SendemailtoTAs(CCinstructor)ifdidnotgetPiazzainvite

CSE252C,SP20:ManmohanChandraker

ComputerVision

CSE252C,SP20:ManmohanChandraker

Definingcomputervision

Wall-E: Fact and Fiction (Minh Do, Princeton University)CSE252C,SP20:ManmohanChandraker

Studyingcomputervision

Source:2Pmedia

• Visionisafundamentalinterfacetotheworld

CSE252C,SP20:ManmohanChandraker

Studyingcomputervision

Source:2Pmedia

• Visionisafundamentalinterfacetotheworld

CSE252C,SP20:ManmohanChandraker

Humans

Machines

Virtual

(Cyborgs!)

• Visionisafundamentalinterfacetotheworld

• Weareallusersofcomputervision

Studyingcomputervision

Transportation Design

Communication E-CommerceCSE252C,SP20:ManmohanChandraker

Studyingcomputervision• Visionisafundamentalinterfacetotheworld

• Weareallusersofcomputervision

• Deepandattractivescientificproblems• Howdowerecognizeobjects?

• Whydonewbornbabiesrespondtoface-likeshapes?

[Farroni etal.,2005]

CSE252C,SP20:ManmohanChandraker

Definingcomputervision

CSE252C,SP20:ManmohanChandraker

Definingcomputervision

CSE252C,SP20:ManmohanChandraker

Ourproblemsbecamemorecomplex

Definingcomputervision

CSE252C,SP20:ManmohanChandraker

Ourambitionsbecamehigher

Definingcomputervision

CSE252C,SP20:ManmohanChandraker

Ourconceptschangedovertime

Definingcomputervision

CSE252C,SP20:ManmohanChandraker

Ourtoolsadvancedovertime

[Mahony etal.,2019]

Gainingperspectiveoncomputervision

CSE252C,SP20:ManmohanChandraker

ImportantforAutopilot:Donothitapolice car!

Gainingperspectiveoncomputervision

CSE252C,SP20:ManmohanChandraker

ImportantforAutopilot:Donothitapolice car!

ImportantforRobots:Stayclearofhumans!

[Wired.com]

Gainingperspectiveoncomputervision

CSE252C,SP20:ManmohanChandraker

ImportantforAutopilot:Donothitapolice car!

ImportantforRobots:Stayclearofhumans!

[Wired.com]

Autonomousdriving

Source:WiredCSE252C,SP20:ManmohanChandraker

AutonomousdrivingWhereisourcar?

StructurefromMotion

VisualSLAM

Whereareotheragents?

Objectdetection

3Dlocalization

Wherearesceneelements?

Semanticsegmentation

Whatisasafepath?

Behaviorprediction

Pathplanning

CSE252C,SP20:ManmohanChandraker

Afewtopicsinthiscourse

• StructureandMotion

• FacesandHumans

• ObjectsandStuff

• MaterialandLighting

• BiasandPrivacy

CSE252C,SP20:ManmohanChandraker

StructureandMotion

CSE252C,SP20:ManmohanChandraker [Agarwaletal.,ICCV2009]

StructureandMotion

CSE252C,SP20:ManmohanChandraker[SongandChandraker,CVPR2015]

FacesandHumans

CSE252C,SP20:ManmohanChandraker

[Blanz andVetter]

[Microsoftblog, 2018]

FacesandHumans

CSE252C,SP20:ManmohanChandraker [Kanazawaetal.,CVPR2019]

ObjectsandStuff

CSE252C,SP20:ManmohanChandraker [Choi,ICCV2015]

ObjectsandStuff

CSE252C,SP20:ManmohanChandraker

[Wangetal.,CVPR2019]

[Leeetal.,ICCV2017]

MaterialandLightingInputimage Augmented image

CSE252C,SP20:ManmohanChandraker [Lietal.,CVPR2020]

MaterialandLighting

CSE252C,SP20:ManmohanChandraker [Li,Yeh,etal.,CVPR2020]

BiasandPrivacy

CSE252C,SP20:ManmohanChandraker

Caucasian African-American East-Asian

Proportion in datasets: 80% Proportion in datasets: 10% Proportion in datasets: 5%

High accuracy Low accuracy Low accuracyTraining on biased data without domain adaptation

High accuracy High accuracy High accuracyUsing domain adaptation to address dataset bias

BiasandPrivacy

CSE252C,SP20:ManmohanChandraker

Noadaptation

Withadaptation

Labelsrequire1.5hours perimage!

Sourcedomain:goodweather,with labels

Targetdomain:rainyweather,no labels

Trainonsource,apply ontarget

Trainonsource,adapt totarget

[Tsaietal.,ICCV2019]

BiasandPrivacy

CSE252C,SP20:ManmohanChandraker

America

Collect

data

Label

data

Train

modelDeployExisting

solution

Newmarket

Newmarket

....

....

....

....

Asia

Collect

data

1week

Label

data

Adapt

model

1week

Deploy

Africa

Collect

data

Label

data

Adapt

modelDeploy

1week 1year 1week

1week 1week

BiasandPrivacy

CSE252C,SP20:ManmohanChandraker [Xiaoetal.,AAAI2020]

Overallgoalsforthecourse

• Introducethestate-of-the-artincomputervision

• Studyprinciplesthatmakethempossible

• Getunderstandingoftoolsthatdrivecomputervision

• Enableoneorallofseveralsuchoutcomes

– Pursuehigherstudiesincomputervision

– Joinindustrytodocutting-edgeworkinAI

– GainanappreciationofmodernAItechnologies

CSE252C,SP20:ManmohanChandraker

CourseDetails

CSE252C,SP20:ManmohanChandraker

Coursedetails• Eachclasswillcovertopicsincomputervision

• Examplesoftopics

• Correspondence

• Stereo

• Opticalflow

• Structurefrommotion

• Facerecognition

• Humanposeestimation

• Materialandlighting

• Semanticsegmentation

• Objectdetection

• Tracking

• Actionrecognition

• Domainadaptation

• Privacyandfairness

CSE252C,SP20:ManmohanChandraker

Coursedetails

• Topicsstructuredintoafewmodules

– Background

– StructureandMotion

– FacesandHumans

– ObjectsandStuff

– MaterialandLighting

– AdaptationandBias

CSE252C,SP20:ManmohanChandraker

Coursedetails

• “Lightning”presentations

– Fourstudentstopresentinoneclass

– Timelimit:5minutes

– Paperstobeassignedbyinstructor

– Orderofpresentation:alphabetic

• Sendpresentation1daybeforeclass

– Well-practicedandfluentpresentation

– Includenarrationifasynchronous

– Askandanswerquestionsafterpresentation

CSE252C,SP20:ManmohanChandraker

Coursedetails

• Presentationformat(1slideforeach):

1.Motivationandproblemdescription

2.Priorwork

3.Methodoverview

4.Methodanalysis

5.Experiments

6.Futureworkanddiscussion

CSE252C,SP20:ManmohanChandraker

Coursedetails• Classwebpage:

– http://cseweb.ucsd.edu/~mkchandraker/classes/CSE252C/Spring2020/

• Instructoremail:

– mkchandraker@eng.ucsd.edu

• TAs:Zhengqin LiandYou-YiJau

– Emails:zhl378@eng.ucsd.edu andyjau@eng.ucsd.edu

• Grading

– 10%presentation

– 60%assignments

– 30%finalexam

• Aimistolearntogether,discussandhavefun!

CSE252C,SP20:ManmohanChandraker

MyInterests

CSE252C,SP20:ManmohanChandraker

Sceneunderstandingforself-drivingLearningtosimulate

Large-scale,real-time,monocularSFM.

Accuracycomparabletostereosystems.

Teacher

CNN

Student

CNN

ImageHint

loss

Student

Detector

Teacher

Detector

Distillation

loss

GT

loss

Distillationnetworksforfastandaccurateobjectdetection

DistillationforcompressedCNN(student) tomimicuncompressed

CNN(teacher),toachievegreateraccuracyatthesamespeed.

MonocularSFMLearningtopredictuncertainfuturebehavior

DESIRE:DeepStochasticIOCRNNEncoder-Decoder

• DeepCVAE(autoencoder) togeneratediversehypotheses.• RNNtorankpredictions basedonmotion,sceneandinteractions.• Deepinversereinforcementlearningforlong-termfuturerewards.

Reinforcementlearningforsimulations.

Ensuringdiversity andcoverage.

NeurIPS 2017

CVPR2017CVPR2015

ICLR2019

CSE252C,SP20:ManmohanChandraker

Geometricandsemantic3Dreconstruction

WeaklysupervisedsemanticreconstructionMetriclearningforcorrespondence

OcclusionreasoningandlargetransformationsDeepsupervisionforocclusion-reasonedparts

CVPR2016

CVPR2017,PAMI2018

NeurIPS 2016,ECCV2018

ECCV2018,CVPR2019

Input

Occlusion-reasoned

top-viewlayout

CSE252C,SP20:ManmohanChandraker

Physically-basedlearningforshapeandmaterial

Materialsand

global

illumination

Refractive

interfaces

Shapeand

motion

WarpNet ColorNet

SIGGRAPHAsia2018

WACV2018,2019

ICCV2017ICCV2017CSE252C,SP20:ManmohanChandraker

UnsupervisedadaptationtonewdomainsReconstructionwithunaligneddataFacerecognitionforprofileinputs

FromrainytogoodweatherCarrecognitionacrosscameraandlightingconditions

CVPR2019

3DV2017

ICCV2017

CVPR2018CSE252C,SP20:ManmohanChandraker

Take-homemessage

• ComputervisionisakeybranchofAI

• Enablesseveralmodernapplicationsaroundus

• Alotofhighlyvisibleandhigh-impactactivity

• Hugeindustryinterest

• Thisisagreattimetostudycomputervision!

CSE252C,SP20:ManmohanChandraker