8/8/2019 7133238-Jmf

1/59

8/8/2019 7133238-Jmf

2/59

JMF Media Processing Model

8/8/2019 7133238-Jmf

3/59

Media Streams

Often contain multiple channels Tracks

Example MPEG-1 file usually 2 tracks

Audio track

Video track

Demultiplexing

Multiplexing

8/8/2019 7133238-Jmf

4/59

Example

Process an mpeg-1 a/v media stream Transcode video track to H.263

Transcode audio track to GSM

Steps Demultiplex to obtain tracks

Decompress video track

Compress using H.263

Decompress audio track

Compress using GSM Multiplex two tracks

Save to file

8/8/2019 7133238-Jmf

5/59

JMF Design Goals

Enable input, processing and output of time-based

media

Provides common cross platform API for

accessing underlying media frameworks

Extensible

support additional content types and formats,optimize handling of supported format, create new

presentation mechanisms

8/8/2019 7133238-Jmf

6/59

Supported Content Types

Supported types Not always both decode and encode

Differences between platform independent and dependent

versions

Audio

WAV, GSM, MIDI, etc

Image

JPEG, etc

Video

H.261,H.263, MPEG-1, Quicktime, AVI, etc

8/8/2019 7133238-Jmf

7/59

Recording, processing, and

presenting time-based media

8/8/2019 7133238-Jmf

8/59

High-level Architecture

8/8/2019 7133238-Jmf

9/59

Some JMF Base Interfaces

Clock

Controller

Control

8/8/2019 7133238-Jmf

10/59

Time Model

The Clock interface Defines basic timing and synchronization operations

Contains a Timebase

Based on the system clock

time-base time

Simply provides current time

Clock marks time for a particular media stream

media time Current position within a media stream

8/8/2019 7133238-Jmf

11/59

Time Model

8/8/2019 7133238-Jmf

12/59

Clock

Playback rate How fast the Clock is running in relation to itsTimeBase

Examples:

rate of 1.0 represents normal running time

rate of2.0 means presentation will run at twice thenormal rate

ClockT

ransform Media-time =media start-time+ Rate(time-base time

time-base start-time)

8/8/2019 7133238-Jmf

13/59

Example

Example: Have a 20 sec MPEG video stream MediaStartTime= 10 secs,

TimeBaseTime= 3 secs

TimeBaseStartTime= 0 secs, TimeBaseTimeTimeBaseStartTime= 3 secs

Media-time= media start-time+ Rate(time-base time time-base start-time)

So if Rate

=1.0

, MediaTime=

?? Alternatively, if rate = -2.0, MediaTime= ??

8/8/2019 7133238-Jmf

14/59

Achieving Synchronization

Example Want to force a video renderer to sync to the

timebase of an audio renderer X = audio_renderer.getTimeBase()

Video_renderer.setTimeBase(X)

Both objects would now use the same source of

time.

8/8/2019 7133238-Jmf

15/59

ControllerInterface

Controller Interface Defines basic state and control mechanism for an

object that controls, presents or captures time-based media

Two types of Controller: Players and Processors

(considered later)

8/8/2019 7133238-Jmf

16/59

Controllers

8/8/2019 7133238-Jmf

17/59

Controller lifecycle

8/8/2019 7133238-Jmf

18/59

Controller Events JMF objects can post a MediaEvent

Events posted by a Controller: TransitionEvents

Posted when a controller changes state

Change notification events

e.g. RateChangeEvent

ControllerClosedEvents

Posted when Controller shuts down

Corresponding listener interface for each type of JMFobject that can post MediaEvents

8/8/2019 7133238-Jmf

19/59

JMF Event Model

8/8/2019 7133238-Jmf

20/59

Controls

Mechanism for setting and querying attributes ofan object

Certain objects expose Controls e.g. often used by PlugIns to provide access to their

Control objects

Examples

FrameRateControl

GainControl Can associate listener for when volume changes

8/8/2019 7133238-Jmf

21/59

Key objects in JMF

Managers

DataSources

Players

Processors

DataSinks

8/8/2019 7133238-Jmf

22/59

General Managers

Intermediary objects Enables new implementations of key interfaces

4 main types Manager

PlugInManager

PackageManager

CaptureDeviceManager

8/8/2019 7133238-Jmf

23/59

The Manager object

Manager object used for instantiating: DataSources,

used to deliver time-based multimedia data

Players,

used to control and render multimedia data

Processors,

used to process data and output the processed data

D

ataSinks, takes a DataSource as input and renders the output to a specified

destination

8/8/2019 7133238-Jmf

24/59

The Manager Object

8/8/2019 7133238-Jmf

25/59

Data Model in JMF

Data Sources Media players use DataSources to manage the

transfer of media-content

DataSource encapsulates location of media and the

protocol used to deliver the media

Typically Identified by a:

URL, MediaLocator

8/8/2019 7133238-Jmf

26/59

Capture

Capture devices represented as DataSources e.g. microphone, video capture board, etc

Devices can deliver multiple data streams e.g. audio and video from a camera

e.g. multiple audio tracks from a recording session

You may then wish a single DataSource to contain

multiple SourceStreams

Manager.createMergingDataSource(SourceStreams)

8/8/2019 7133238-Jmf

27/59

Push and Pull Data Sources

Data sources can be categorized according to how

data transfer is initiated

Pull Data-Source Client initiates the data transfer e.g. HTTP and FILE

Push Data-Source Server initiates the data transfer e.g. broadcast and multicast media

8/8/2019 7133238-Jmf

28/59

Players

Processes an input stream and renders it at a

precise time

Does not provide any control over the processing

that it performs or how it renders the media data

8/8/2019 7133238-Jmf

29/59

Players

Player extends the Controller interface. Has a lifecycle

Sends media events

Player as a MediaHandler player = Manager.createPlayer(myDataSource);player = Manager.createPlayer(myMediaLocator);

player = Manager.createPlayer(myUrl);

8/8/2019 7133238-Jmf

30/59

Players

8/8/2019 7133238-Jmf

31/59

UI Components

Players provides access to UI Components Player (or Processor) can provide two UI

components

Visual component

Control-panel component

Can retrieve these components using methods:

getVisualComponent()

getControlPanelComponent()

8/8/2019 7133238-Jmf

32/59

Player States Continued

Players post transitional events as they move

between states

ControllerListener Is the Player in an appropriate state?

Only certain methods make sense in certain states

e.g. calling getTimeBase method on an unrealized

player gives an error

8/8/2019 7133238-Jmf

33/59

Processors

Can also be used to present media data

Specialized type of Player that provides control

over processing performed on the input media

stream

8/8/2019 7133238-Jmf

34/59

Processing

8/8/2019 7133238-Jmf

35/59

Processor Stages

8/8/2019 7133238-Jmf

36/59

Additional Processor States

Two additional stand-by states: Configuring

Configured can use TrackControls

8/8/2019 7133238-Jmf

37/59

Processing Controls

For a given track, can control processing

operations performed by the Processor by using

the TrackControlfor that track.TrackControl[] = processor.getTrackControls()

Can explicitly select: Effect, Codec and Renderer plug-ins to use

TrackControl[1].setCodecChain( array_of_codecs )

8/8/2019 7133238-Jmf

38/59

Configuring the Processor

Consider using a processor to transcode an

(audio+video) QuickTime movie changing

mpeg video track to h.263p = Manager.createProcessor(dataSource)

p.configure()

p.setContentDescriptor.QUICKTIME

tcs[]

=p.getTrackContro

ls()

Returns an array, e.g.2 track controls

8/8/2019 7133238-Jmf

39/59

8/8/2019 7133238-Jmf

40/59

Processor Summary

A Processor does not have to output data as a

DataSource, such a processor (i.e. one that

renders the data) is effectively a configurableplayer.

8/8/2019 7133238-Jmf

41/59

Data Storage and

Transmission

DataSink Used to read data from a DataSource and render the

media to an output destination

Typical actions

Write data to a file, across a network etc

8/8/2019 7133238-Jmf

42/59

Using theD

ataSinkMediaLocator dest = new

MediaLocator(file://newfile.wav);

dsink = Manager.createDataSink(ds, dest);

dsink.addDataSinkListener(this);dsink.open();

p.start();

dsink.start();

Wait forEndOfStream event

Close DataSink and remove Listener dsink.close()

8/8/2019 7133238-Jmf

43/59

Example

Applet Movie Player

8/8/2019 7133238-Jmf

44/59

Applet Movie Player

Simple Java Applet that demonstrates how to

create a simple media player with a media event

listener. It will play the media clip right away andcontinuously loop.

8/8/2019 7133238-Jmf

45/59

Basic Steps

Initialisation Retrieve applets FILE parameter

Use this to locate media file and build

URL

Create Player using the Manager object

Register applet as a ControllerListener

8/8/2019 7133238-Jmf

46/59

Steps 1 & 2: Resolving a URL

for the media stream// The applet tag should contain the path to the

// source media file, relative to the htmlpage.

if ((mediaFile = getParameter("FILE")) == null)

Fatal("Invalid media file parameter");try {

// Create a url from the file name and the url

// to the document containing this applet.

if ((url = new URL(codeBase, mediaFile)) == null)

Fatal("Can't build URL for " + media

File);

8/8/2019 7133238-Jmf

47/59

Step 3: Using Manager to

Create a Player// Create an instance of a player for this media

try {

player = Manager.createPlayer(url);

}catch (NoPlayerException e) {

System.out.println(e);

Fatal("Could not create player for " + url);

}

8/8/2019 7133238-Jmf

48/59

Step 4: Register applet as a

ControllerListener// Add ourselves as listener for player's events

player.addControllerListener(this);

} catch (MalformedURLException e) {

Fatal("Invalid media file URL!");

} catch (IOException e) {

Fatal("IO exception creating player for " +

url);

8/8/2019 7133238-Jmf

49/59

Controlling the Player Starting the Playerpublic void start() {

if (player != null)

player.realize(); Stopping the Playerpublic void stop() {

if (player != null) {

player.stop();

player.deallocate();

}

}

8/8/2019 7133238-Jmf

50/59

Responding to media events

Need to Implement ControllerListener

When the Player is realized

Posts a RealizeCompleteEvent Get the Visual component

if (( visualComponent =

player.getVisualComponent())!= null) {

cPanel.add(visualComponent);

Get the Control Panel component

When the media has reached the end Posts an EndOfMediaEvent

Rewind and start over

player.setMediaTime(new Time(0));

8/8/2019 7133238-Jmf

51/59

Extensibility Can extend JMF functionality in two ways:

Using Plug-ins

Effectively implementing custom processing components that can

be interchanged with standard components used by a Processor

By direct implementation

i.e implementing directly the Controller, Player , Processor,

DataSource, or DataSink interfaces

e.g. implementing a player to utilise a h/w MPEG decoder

e.g. integrating existing media engines

8/8/2019 7133238-Jmf

52/59

Interface Plugin

The base interface for JMF plug-ins.

A PlugIn is a media processing unit that accepts

data in a particular format and processes orpresents the data. Registered through the PlugInManager.

Methods

Open(), Close(), getName(), reset() Sub-interfaces

Codec, effect, demultiplexer , multiplexer, etc.

8/8/2019 7133238-Jmf

53/59

Codecs

One input and one output

Methods setInputFormat()

setOutputFormat()

getSupportedInputFormats()

getSupportedOutputFormats()

process()

input buffer, output buffer

8/8/2019 7133238-Jmf

54/59

Example 2

Accessing individual frames

8/8/2019 7133238-Jmf

55/59

FrameAccess

Problem

How to access individual decoded video frames

from a Processor while processing the media.This could be used for scanning the decoded data;

computing statistics for each video frame, etc.

8/8/2019 7133238-Jmf

56/59

FrameAccess Solution

use a pass-through plugin codec as a callback whenindividual frames are being processed.

Steps: 1) Build the pass-through codec.

2) Create a processor from the input file. This processorwill be used as a player to playback the media.

3) Get the TrackControls from the processor.

4) Set your codec on the video track:

TrackControl.setCodecChain(your_codec[])

8/8/2019 7133238-Jmf

57/59

Basic code// Get Video track as a track control

TrackControlvideoTrack = null;

for (int i = 0; i < tc.length; i++) {

if (tc[i].getFormat() instanceof VideoFormat) {videoTrack = tc[i]; break;

}

}

// Instantiate & set frame access codec to data flow

path.Codec codec[] = { new PreAccessCodec(),

new PostAccessCodec() };

8/8/2019 7133238-Jmf

58/59

PreAccessCodecpublic class PreAccessCodec implements Codec {

void accessFrame(Buffer frame) {

long t = (long)(frame.getTimeStamp()/10000000f);

System.err.println("Pre: frame #: " +

frame.getSequenceNumber () +

", time: " + ((float)t)/100f +

", len: " + frame.getLength());

}

Other methods, e.g. getSupportedInputFormats etc.

8/8/2019 7133238-Jmf

59/59

The Codecs Process method

public int process(Buffer in, Buffer out) {

// This is the "Callback" to access individual frames.

accessFrame(in);

// Swap the data between the input & output.Object data = in.getData ();

in.setData(out.getData ());

out.setData(data );

// Copy the input attributes to the output

out.setFormat(in.getFormat());out.setLength(in.getLength());

out.setOffset(in.getOffset());