chapter 4 hardware & software that enables multimedia
TRANSCRIPT
Chapter 4
Hardware & Software
That
Enables Multimedia
Hardware That Enables
Multimedia
Hardware needs to be considered when you are
specifying, designing, creating, and
maintaining a multimedia presentation
Why?
Display Adapters
Resolution: pixel by pixel size that defines the size of the display area limited by hardware
E.g., my laptop: max resolution 1600 x 1200 pixels
Video memory
Used to hold RGB (or CYM) pixel intensity information
Motion video
Frame rate indicates how many times per second video memory must be updated
Amount of memory updated depends on
Size of video image
Color resolution
Frame rate
Image and Text Capture: OCR,
Digital Cameras, Scanners
Real image, digital image, still image, motion image
OCR: Optical Character Recognition
A method to capture printed text, as characters
Depends on
OCR software
e.g., range of fonts recognized
Quality of page being scanned
Likely will require proof reading
Still image capture: digital camera, scanner
Issues: Optics quality, precision of photocells (e.g., CCDs), dpi: dots per inch resolution
Speakers/Sound
Sound formats
.WAV, .MP3
MIDI (musical instrument digital interface)
Microphone
Microphone
Generates analog signal for sound
Can digitize with an A/D converter (sound input board)
Video
Video Cameras, frame grabbers
Analog video:
Generates analog video signal (e.g., NTSC, PAL)
Need a frame grabber (video input card)
Converts an analog video signal to a digital image frame
Digital video:
Directly Digitizes camera image at some frame rate
E.g., digital camcorder with firewire interface
Graphics Card and Graphical Processing
Unit (GPU)
Positional Information
Mouse, joystick, digitizing tablets, track pads, touch screens
Positional information is analog
E.g., rotations of the wheels on a mouse device
E.g., coordinate information on a digitizing tablet; stylus pressure
This information is digitized
Change information is often important
Future developments
Eye tracking, eye gaze
In combination with mouse or other devices
Mass Storage
DVD, CD-R, CD-RW, Mini disc (MD)
Direct access devices
Capacity
CD-ROM: about 720MB
MD: 74 minutes audio @ 44.1 kHz
DVD: from 4.4 GB to 16GB (8 hrs video)
Speed: Write speed, read speed, rewrite speed
Write once (CD-R), write multiple times (CD-RW; MD)
Laser disks: Analog format only?
60 minutes of 30 fps video
Virtual Reality (VR)
Goal: “to provide the user with a computer-controlled environment that is as close to physical reality as possible”
VR system equipped with sophisticated output devices
And specialized input devices
Allow the user to react in a similar to “natural” manner
Getting the computer system to adapt (in part) to the human
As opposed to the human adapting to the computer system
Emphasis similar to that in interactive multimedia
A great deal of output from the computer to the user
Less input from the user to the computer
Networks:
Modems & Network Interfaces
Modems Connection to phone, cable, ADSL, or ISDN
Network Computers connected via some physical
networking (e.g., LAN, Internet) hardware
Need network interface hardware (e.g., ethernet)
Required to interface to a network
Network bandwidth Modems: Lower bandwidth
Cable modem, ADSL, LAN connection: Higher bandwidth
Character encoding ASCII, EBCDIC, Unicode (16 bit), ISO/IEC
standards
Software That Enables Multimedia
Operating systems
Development software
Delivery software
Operating System Software
Device drivers
Multi-tasking, multi-threading
Memory management
Disk space management
Processor (CPU) management & scheduling
Window management
Servers (e.g., printing, email, web, telnet, ssh)
User level software
Includes programs to access and deal with lower levels of operating system (e.g., format a disk, partition a disk, change the priority of a task or process)
end user level applications to get work done
E.g., Macromedia Director, Power Point, Web browser
Device Drivers
E.g., Display adapter driver, Firewire card driver, Network card driver, Video Camera Driver, Microphone driver
Devices have specific hardware
Drivers are specific to the hardware
Enable the operating system to use that specific hardware
Device drivers depend
on operating system
on the specific device
Device drivers enable programs
To be less dependent on hardware; more generic
Codec’s
Compression and Decompression software for video and audio
“drivers” for the particular format used for the files
Depend on format, not on hardware
Within a particular general format (e.g., AVI or Quicktime)
E.g., with AVI, a Cinepak or Indeo codec can be used
Sorensen is available with Quicktime
Many codec’s can be used
Codec must be on development system and on the destination system
Networking - 1
Networking functions are incorporated
into operating systems
Remote disk mounting
Network device (e.g., printer) access
Lower levels of network data transmission
(e.g., physical, data link layers)
Multimedia software often depends on
networks
Dynamic loading of codec & other components
Access to web-based presentations
Networking - 2
Communication protocols
Physical layer protocols (ISO layer 1)
E.g., Ethernet, token ring, ATM
Higher protocol layers
TCP/IP (layers 2 & 3)
SMTP – simple mail transfer protocol (layer 6)
HTTP – hypertext transfer protocol (layer 6)
Performance objectives
Accuracy, transmission speed, error correction, user-perceived delays, overall throughput
Client/Server Architecture - 1
Server
A process that provides access to particular hardware or software resources
Process: A program that runs independently on a computer
Often used in the context of networks
Used within software design generally, as well
“Server” may designate a software process
Or may designate a computer (hardware)
Depends on the kind of service being provided
Client/Server Architecture - 2
E.g., database server
Computer will have database software
May also have additional hardware resources (e.g., more disk space)
E.g., a software license server (“metering”)
Enables some maximum number of people to be using a software resource at a given time.
Server receives requests
Remote requests from a client process on another computer
Local requests from a client process on same computer
Requests are sometimes for data sometimes for programs
Operating System &
Multimedia Requirements
Development of multimedia
Running multimedia
Some operating systems may be more suited for running multimedia
Issues include:
Memory (RAM) required
Operating system availability
Application availability
Speed
Networking
Multimedia Development
Software
Software needed to develop multimedia
And software needed to deliver
E.g., Shockwave playback in a browser has different requirements than running Macromedia Director
Development
Preproduction, production, postproduction
Types of development software
2D image capture, scanning & processing; Digital video capture & processing
Animation; 3D modeling, rendering; Sound capture & processing; Text; Web development; Authoring
Vector vs. Bitmap Graphics - 1
Representation of objects in images E.g., a rectangle with a black outline and a gray
interior
Bitmap (raster) graphics Specify value for each pixel
Application data file (e.g., Photoshop) stores collection of pixels making up rectangle
E.g., 300 x 600 = 180,000 pixels of rectangle
Requires 176KB to 527KB of data (uncompressed)
Vector graphics Use rectangle drawing tool & color-fill
Application program generates a data file that represents rectangle as a command and parameters
E.g.,
Rectangle, 0, 0, 300, 600, black, gray (32)
Requires about 100 bytes of information
Vector vs. Bitmap Graphics - 2
Space Tradeoff
Bitmap graphics (uncompressed) take more space
Vector graphics take less space
Speed Tradeoff
Bitmap graphics (uncompressed) are drawn faster
Vector graphics need to be decoded and then the bitmap (pixels) generated
Will be slower than bitmap representation
An issue on slower computers or if a very large number of objects need to be displayed
Expressiveness
Some objects can’t be easily drawn in vector fashion
Contemporary drawing programs may provide both vector and bitmap graphics
Video - 1
Video is captured in discrete units called frames Frame: Similar to a still picture
Temporal and spatial compression Spatial: within a frame
Temporal: across frames
Key-frame concept in temporal compression Some compression methods use this form of coding
Store information on one frame at particular intervals
This is a “key frame”
Then store changes from this point
Uses frame differencing
Makes use of common background often shared between multiple frames
Video - 2
Time-dependent medium
Illusion of motion depends on frame rate
Don’t perceive motion below about 15 fps
Also: animation and sound
Time-dependent media place high computational demands on a computer
E.g., with 1024 x 768 image resolution, and 24 bit color depth @ 30 fps, what is the (apparent) data transfer rate in KB/second?
Video - 3
1024 * 768 = 786, 432 pixels/frame
786, 432 pixels/frame* 3 bytes/pixel =
2,359,296 bytes/frame
2,359,296 bytes/frame * 30 frame/sec =
70,778,880 bytes/sec
(70,778,880 bytes/sec) / (1024 bytes/KB) =
69,120 KB/sec
(Apparent: because this is without any data compression)
Animation
Similar to video
Each frame created via software
not from a camera source
History
traditionally done by hand, frame by frame
Model-based animation
Online (real-time) versus offline
Model-based: software representation of world
Online: Frames are generated on-the-fly, in response to dynamic changes of the model
Offline: Frames generated in response to model changes
Related Techniques
Biological animation
Animal gait (مشية الحيوان), bipedal motion (حركة القدمين)
Morphing (تتحول) Gradual changes in one image to transform
it to a second image
Applying changes gradually
E.g., taking a few seconds @ 30 fps
Can be used for animation purposes
VRML
Virtual reality modeling language
"an open standard for 3D multimedia and shared virtual worlds on the Internet."
MPEG, Quicktime, AVI
Standard formats for video and audio
information
Includes techniques/standards for
compression, decompression
MPEG
MPEG Audio
ISO/IEC standardization
Open standard
Specification is available to everyone (for a fee)
No single company “owns” the standard
Quicktime, AVI
Quicktime
Primarily Macintosh/Apple
http://www.quicktime.com
AVI
Primarily Windows/Microsoft
AVI: “Audio Video Interleave”
E.g., see http://www.jmcgowan.com/avi.html
MPEG Standards
MPEG-1:
Applications: MP3, Video CD
MPEG-2
MP3, Digital Television set top boxes, Digital Versatile Discs (DVD).
MPEG-4
Proposed fixed & mobile web standard
MPEG-7: Multimedia Content Description
MPEG-21: Multimedia Framework
MPEG: Video
MPEG-1
Roughly VHS quality at 1,15 Mbit/s
MPEG-2
“MPEG-2 Video was at least good, and in many cases even
better than standards or specifications developed for high
bitrate or studio applications”
http://mpeg.telecomitalialab.com/standards/
mpeg-2/mpeg-2.htm
MPEG: Audio
MPEG standards have layers
Layer: Operating modes with increasing complexity and performance
MPEG-1 provides
Mono and stereo coding at 32, 44.1, and 48 kHz sampling rate
Layer I, II, III: varying bit rates from 32 to 448 kbit/s
Optimized for 128 kbits/s stero
MPEG-2 BC provides
Backwards compatible multichannel extension to MPEG-1
Up to 5 channels plus a 'low frequent enhancement' channel
Bit rate range is extended up to about 1 Mbit/s;
an extension of MPEG-1 towards lower sampling rates 16, 22.05, and 24 kHz for bitrates from 32 to 256 kbit/s (Layer I) and from 8 to 160 kbit/s (Layer II & Layer III).
http://mpeg.telecomitalialab.com/faq/audio.htm#5
MPEG-2 & MPEG-4: Audio
MPEG-2 Advanced Audio Coding (AAC)
High-quality audio coding standard for 1 to 48 channels
Sampling rates of 8 to 96 kHz
Multichannel, multilingual, and multiprogram capabilities
Bit rates from 8 kbit/s to more than 160 kbit/s/channel
Multiple encode/decode cycles
Three profiles (layers) of AAC provide varying levels of complexity and scalability.
MPEG-4 will provide
coding and composition of natural and synthetic audio objects at a very wide range of bit rates.
Emphasis on new functionalities
MPEG-4
Goal: Higher levels of interaction
1. represent units of natural, visual or audiovisual content, called "media objects". These media objects can be of natural or synthetic origin; this means they could be recorded with a camera or microphone, or generated with a computer;
2. describe the composition of these objects to create compound media objects that form audiovisual scenes;
3. multiplex and synchronize the data associated with media objects, so that they can be transported over network channels providing a QoS appropriate for the nature of the specific media objects; and
4. interact with the audiovisual scene generated at the receiver’s end.
http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm
MP3
MPEG-1/2 Audio Layer-3
MPEG-1 Audio Layer-3
MPEG-2 Audio Layer-3 (successor)
MP3 and AAC Explained
http://citeseer.nj.nec.com/421409.html
MP3 MPEG-3!
Sound Representation
On a sample basis E.g., about 44 kHz for CD quality audio
Each sample is a number
gives the amplitude of the analog sound wave at that point
On a feature basis E.g., frequency, duration, musical note
Also possible: for speech, phonemes
Phoneme is a unit of sound in a particular language
Phonemes may sound differently dependent on context
E.g. /p/ (personal, paternal, computer, rapid), /b/, and /s/
Difference is analogous to the difference between bitmapped (raster) & vector graphics
Tradeoffs– Similar to those between bitmap & vector graphics
Example 1
With a sampling rate of 44.1 kHz, and 16 bitsound resolution, how many KB does it take to represent 3 minutes of sound? (Without compression)
Example 1
44,100 samples/sec * 2 bytes/sample =88, 200 bytes/sec
88, 200 bytes/sec * 60 sec/minute =
5,292,000 bytes/min
5, 292, 000 bytes/min * 3 min =
15,876,000 bytes
15,876,000 bytes/(1024 bytes/KB) =
15, 503.90625 KB
(Or about 15 MB)
Example 2
If we are representing speech phonemes, and we want to represent individual words, how much data does it take to encode the English word “Hello”?
Assume that English has 39 phonemes
Assume that we have one phoneme per character
Problem is to represent sequences of phonemes
Each phoneme can take on 39 possible values
Need at least 6 bits of information to represent 64 different values , So, need at least 6 bits per phoneme
Word “Hello” has 5 characters
Assuming also 5 phonemes 5 * 6 = 30 bits
30 bits/(8 bits/byte) = 3.75 bytes Need 4 bytes
Sound Filtering
A sound can be filtered to remove a certain set of frequencies
Some types of filtering
Low pass: Allows only low frequencies through
High pass: Allows only high through
Band pass: Allows only a band range of frequencies through
Other types of filtering or processing
Trimming, splicing, volume
Web browsers
Process various file types
HTML
Media formats: images, sounds, movies
Some types of data are processed directly
Built into the browser
Other types
Require additional programs or “plugin-ins”
E.g., a plugin to display PDF or Adobe Portable Document format files or a plugin to display Shockwave files
Some plugins are called “players” or “viewers”
E.g., a viewer to display MS Word documents
Multimedia platform
requirements:
Multimedia platform should
Support
the development of a wide variety of multimedia applications
applications in a distributed system
Be able to
establish and control streams of continuous media
to choose the quality of service for a data stream
to express relations between several media streams
Multimedia Delivery Software
Two general means of delivery Compiled or Interpreted
Compiled Machine code generated for a computer CPU
(e.g., Intel-based) and operating system (O/S)
Faster; limited to computer type & O/S; stand-alone
E.g., Macromedia Director “Projector”
Interpreted Instructions created that can be interpreted by
a program; E.g., a plugin
Slower; more portable; need interpreter (e.g., player) software
E.g., Macromedia Shockwave
In both cases, media files are embedded and/or retained as separate files