cs 4763 fundamentals of multimedia systems -introduction to image
Post on 03-Feb-2022
7 Views
Preview:
TRANSCRIPT
CS 4763 Fundamentals of Multimedia Systems - Introduction to Image Processing
Qi Tian
Computer Science Department University of Texas at San Antonio
qitian@cs.utsa.eduhttp://www.cs.utsa.edu/~qitian/
Image Processing
Manipulation of multidimensional signals− image (photo)− video− CT, MRI− Fluid flow
),( yxf),,( tyxf),,,( tzyxf),,,( tzyxv
A Typical Image Processing System
object observe digitize store process Refresh/store
Record
output
DisplayImaging systems
Sample and quantize
Digital storage (disk)
Digital computer
On-line buffer
X-ary, radar imaging, infrared imaging, ultrasound imaging, medical imaging, geophysical imaging
A/D
Fundamentals of Image Processing
Representation– acquisition, digitization, and display to mathematical
characterization of images for subsequent processing– a prerequisite for an efficient processing techniques such as
enhancement, filtering, and restoration.
Processing Techniques– Image compression, image restoration, and image reconstruction– Statistical image processing techniques
Communications
Multimedia Processing Techniques
– Coding/compressionStorage and communications
JPEG, JPEG2000MPEG-1 (CD, mp3), MPEG-2 (HDTV, DVD) H.261, H.263
– Enhancement, restoration, reconstructionfeature extraction for image analysis and visual information displayremoval of degradation in an image, LS, ML, Max entropy, MAP2D -> 3D image MRI, CT, Radon transform
– Analysis, detection, recognition, understandingquantitative measurements from an image to produce a description on it
– Visualization
Advanced Processing Techniques
Statistical processing techniques– Hidden Markov model (HMM)
– Probabilistic graphical models
– Bayesian networks
– Markov random field
Many applications to speech recognition, pattern classification, data
compression, and channel coding, etc.
History of Image/Video Coding
1950
1960
1970
1980
1990
2000+
Math PR, CV, CG
Fractal 3-D Model based coding
Signal ProcessingBased
PCM DPCM
Transform CodingVQ
Subband Coding Wavelets
Reference:– F. Nebeker, Signal Processing: The Emergency of a Discipline,
1948-1998– IEEE History Center, 1998
Broadband TV (NTSC)500 × 500 × 8 × 3 × 30 bits/sec≈100 Mb/sec (compression is necessary!)Modem: 56Kb/sec
Picture Element– Pixel West coast people in USC– Pel East people in MIT
Image/Video Compression
Signal-Processing Based:Encoder
),( yxfH
),( yxgSignal Proc.
Representation ),( yxg
Decoder1H− ),(ˆ yxf),( yxg
Image/Video Compression
3D Model-Based:Encoder
Representation P
Decoder
),( yxfH
Analysis Model Parameter P
Model
),(ˆ yxfP 3D
Model
Image/Video Compression
Fractal-Based:Encoder
Representation S
Decoder
),( yxf System S ),( yxf
Find S for which is an Attractor.),( yxf
SAny signal
),(ˆ yxf
Iteration
Image/Video Compression Standard
Facsimile: Fax Group 1, 2, 3, 4JBIG (Joint Bi-level Image Expert Group)
Images: JPEG (http://www.jpeg.org/)
JPEG2000
Video: H.261, H.263 P × 64 Kb/s (P =1 ~ 30)MPEG 1 1.2 Mb/s Video, CD, MP3MPEG 2 1.2 – 20 Mb/s, sports, HDTV, DVDMPEG 4 1 kb/s → 1Mb/s, very low speed video
coding, MultimediaMPEG 7 Multimedia description, audio/video
MPEG 21 Multimedia framework
Based on Wavelet Transform
A de facto image for the past three decade for its rich texture
Lena
What are Challenging Problems in Multimedia Processing?
Multimedia Processing is taken in a broad sense, including:Image/Video compression, enhancement, restoration, reconstruction, analysis, recognition, understanding, visualization, and synthesis/animation.
Examples
Face modeling, detection, and recognitionEmotion recognitionGesture recognitionGender/age/ethnicity recognitionAudio-visual speech recognitionImage/video superresolutionImage/video browsing, indexing, and retrievalBiometrics
Face Related Research
Face modelingFace detectionFace recognitionFacial expression recognition
Generic Face Model
Texture mapping
Face model morphing
Generic Face Model
The generic face model is generated from a MRI data set
Customize A Genetic Face on An Individual
Polygon Mesh: 2240 Vertices + 3946 Triangles.Polygon Mesh: 2240 Vertices + 3946 Triangles.NonNon--Uniform Rational BUniform Rational B--Splines (NURBS): 63 control points.Splines (NURBS): 63 control points.
The iFACE system in a distributed collaborative environment. (a) Avatar in the head mounted display, (b) avatar in the desk screen of
MIC3E, (c) avatar in the main screen of MIC3E
Avatar – talking head
University of Illinois at Urbana-Champaign
Text-Driven Face Animation
“We strive to make the meter on animation production, and are always looking for new technology that will enable faster, more appealing character creation,”
said Joel Kransove, Digital Director of Nickelodeon. (Source: Digital Producer)
Speech-Driven Face Animation
“Game characters have become synthetic actors and dialogue is an essential element of the effect we create. The quality of the lip-synching can make or break the sense of reality,”
said Scott Cronce, vice president and CTO at Electronic Art (Source: Gamepro)
Video-Driven Face Animation
Emotion Recognition
Emotion Recognition
Emotion Recognition
Face Detection Techniques
Face Detection Techniques
Face Recognition: Why it is easy?
Face Recognition: Why it is hard?
Beauty Check
What Are the Causes and Consequences of Human Facial Attractiveness?
Babyfaceness
Symmetry
Social perception
Universities of Regensburg, Germany
Which is more attractive?
Universities of Regensburg, Germany
BabyfacenessLarge head
Large curved forehead
Facial elements (eyes,
nose, mouth) located
relatively low
Large, round eyes
Small, short nose
Round cheeks
Small chinKate Moss4-year old girl
Include mature female features: high, prominent cheek bones and concave cheeks
Which one is cuter?
Miss Germany (2002)
A selection of the 22 contestants of the final round of the contest
Real vs. Virtual Miss Germany
Image Analysis
Texture synthesis and transferImage Super-resolutionImage RepairsIllumination/Lighting changes and transfer
Texture Synthesis and Transfer
+
SIGGRAPH’01 Effros & Freeman, MIT, 2001
synthesis
transfer
Texture Synthesis and Transfer
Image Superresolution
True Sub-sampled
Intelligent guess about details of texture
Image Superresolution
Gaussian filter Bicubic interpolation
Image Superresolution
Median filter Wiener filter
Image Superresolution
Dynamic resolution enhancement Amos Storkey
True
Image Repairs
Image Repairs
Original Image
Result
Segmentation
Image synthesis based on Tensor Voting
Curve connection
Image Repairs
Illumination Effects on Images
Relighting – Basic Algorithm
Step 2: Approximate radiance environment map
Step 3: Synthesize novel appearance by adjusting the 9 spherical harmonic coefficients
Step 1: Align image with generic 3D face model
Lighting Transfer
input target results
Image/Video Retrieval
Image database
CBIR based on color, texture, shape/structure
MARS: Multimedia Analysis and Retrieval System
metadata
User Interface
Similarity ranking
memory
Feature weighting
Visual C++
Feature Extraction
C/C++ Color
Texture
structure
State-of-the-artCBIR Systems
QBIC (IBM), PhotoBook (Media Lab), Netra (UCSB), VisualSeek (Columbia), PicHunter (NEC-NJ), Amore (NEC-CA), EI Niňo (Praja), MARS (UIUC), Virage (Virage Inc.), CORE, PictoSeek, Piction, InfoScope …
Research CommunitiesComputer Vision, Image/Video Processing, Library and Information Science, Database and Management Systems
Leading Journals & StandardPAMI, ACM Multimedia, IJCV, CVIU
MPEG-7
MARS using global features
Biometrics
Security Threats:We now live in a global society of increasing desperate and dangerous people whom we can no longer trust based on identification documentswhich may have been compromised.
A challenging Pattern Recognition ProblemEnabling technology to make our society safer, reduce fraud and offer user convenience.
Too many passwords to remember
Identification Problems
Identity Theft: Identity thieves steal PIN (e.g., date of birth) to open credit card account, withdraw money from accounts and take out loans
3.3 million identity thefts in U.S. in 2002; 6.7 million victims of credit card fraud
Surrogate representations of identity such as password and ID cards no longer suffice
Biometrics
Automatic recognition of people on their distinctive anatomical (e.g., face, fingerprint, iris, retina, hand geometry) and behavioral (e.g., signature, gait) characteristics.
Person identification is now an integral part of the infrastructure needed for diverse business sectors such as banking, border control, law enforcement…
Biometric Applications
Biometric Applications
There are ~500 million border crossing/year (each way) in the US
Want to charge it?
Biometric Characteristics
Biometric Market Growth
International Biometric Group
“State-of-the-art” Error Rate
False accept rate (FAR):Proportion of imposters accepted
False reject rate (FRR):Proportions of genuine users rejected
Multibiometrics
Soft Biometrics
Privacy Concerns
Tracking
top related