mp3: multi-view surveillance system
DESCRIPTION
MP3: Multi-view Surveillance System. Instructor: Klara Nahrstedt April 20, 2012 CS414. MP3: Multi-view Surveillance System. Window for server 1. Window for server 2. Server Capture Audio and Video at a fixed rate Video: 30 fps, Audio: 8000Hz Client Requests for Video and/or Audio - PowerPoint PPT PresentationTRANSCRIPT
MP3: Multi-view Surveillance System
Instructor: Klara NahrstedtApril 20, 2012
CS414
MP3: Multi-view Surveillance System
Window for server 1
Window for server 2
• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz
• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and Passive Mode
• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480
• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240
Client Behavior• Client can request active mode from Server 1
and passive mode from Server 2• Client can request active mode from Server 2
and passive mode from Server 1• Client can request active mode from both
Server 1 and Server 2• Client can request passive mode from both
Server 1 and Server 2
Client User Interface: Example
SWITCH SWITCH
Video window for Server 1
Connect Edit
Video window for Server 2
Computation window for Server 2Computation window for Server 1
Resource Admission at Client
• User defines streaming MODE: active or passive• Client performs resource admission
Server 1 Client
Control Channel
MODE
User
Resource Admission at Client
• Client– Available Application Bandwidth ABN
– Application Frame Size, MN = ?
– Application Frame Rate, RN = ?– Audio Bandwidth = 8000 * 16– Request Bandwidth• Active Mode: BN = (MN * RN) + 8000 * 16
• Passive Mode: BN = (MN * RN) = (MN * 10)
resource.txt
Optimistic Allocation
8000Hz Audio Signal
How to define RN for active mode?
fps: 10
[fps:15-25]
Resource Admission at Client
• Client– Available Application Bandwidth ABN
– Application Frame Size, MN = ?
– Application Frame Rate, RN = ?– Request Bandwidth
• Active Mode: BN = (MN * RN) + Audio Bandwidth
resource.txt
Show your computation on the Screen
ABN = (MN*RN) + Audio Bandwidthif RN > 25, RN = 25if RN < 15, REJECT requestelse RN
Resource Negotiation with Server
• Information sent to server includes– Requested Frame Rate (FPS) for video, – Video Resolution, and – Media type (audio, video)
Server 1 Client
FPS, ResolutionMedia types
MODE
Resource Admission at Server
• Server– Available Application Bandwidth ABN – Application Frame Size, MN = ?– Application Frame Rate, RN = [received from client]– Audio Bandwidth = 8000 * 16– Request Bandwidth
• Active Mode: BN = (MN * RN) + 8000 * 16• Passive Mode: BN = (MN * RN) = (MN * 10)
– Admitted if BN <= ABN
– Else Rejected
resource.txt
Optimistic Allocation
Video Data Flow Architecture
RTP
Pack
et
Rate Control
Network
Video Source Video Filter Media Encoder Video Muxer App Sink
RTP
Pack
et App SourceDecode Bin2Video Sink
Synchronization
Payload
Media Type Timestamp RTP headerID
UDP Packet Payload
Segment
Merge
Server-Side
Client-Side
Rate Control: QoS Enforcement
• You can build leaky bucket• You can implement token bucket• You can use gstreamer rate control• You can implement your own
method
Sleep (100) => 10 fps
You can simply read using a loopWhile(true){ SendPacket(); Thread.sleep(100);}
Resource Reservation: Client
– Available Application Bandwidth ABN
– Used Bandwidth for Server 1 B1
– Available Bandwidth ABN = ABN –B1
– Admission is successful for B2 if B2 <= ABN
resource.txt
Server 1 60 Kbps
Server 2 35 KBps
Resource Table
Video Data Flow Architecture using RtpBin
RTP
Pack
et
Rate Control
Network
Video Source Video Filter Media Encoder Video Muxer App Sink
RTP
Pack
et
App SourceDecode Bin2Video Sink
Synchronization
Segment
Merge
Server-Side
Client-Side
RtpBinUdpSinkUdpSrc
Implementation Using RtpBin @ Server
Video Source Video FilterTee Video Scale Scale Filter
Video Rate Rate Filter Video Encoder RTP Payload
RTP Bin
UDP Sink
Queue
UDP SinkUDP Source
RTCP
RTCP
RtpBin combines •RtpSession•RtpSsrcDemux•RtpPtDesmux•RtpJitterBuffer
Implementation Using RtpBin @ Client
Video SinkVideo DecoderRTP DePayload
RTP Bin
UDP Source
UDP SourceUDP Sink
RTCP
RTCP
rtpBin.connect(new Element.PAD_ADDED() {public void padAdded(Element element, Pad pad) {
if (pad.getCaps().toString().contains("video")) {//LINK TO VIDEO SINK via VIDEO MONITOR
} else if (pad.getCaps().toString().contains("audio")) {
// LINK TO AUDIO SINK via AUDIO MONITOR}}
});
Internals of RtpBin
http://gstreamer.freedesktop.org/documentation/rtp.html
Evaluation Process for MP3• Case I: – Server 1 is requested to send stream in ACTIVE mode– Server 2 is requested to send stream in PASSIVE mode– Client has resource to support 25 fps in ACTIVE mode
• Case II: – Server 1 is requested to send stream in ACTIVE mode– Server 2 is requested to send stream in PASSIVE mode– SWITCH streaming mode of Server 1 and Server 2– Client has resource to support 25 fps in ACTIVE mode
Evaluation Process for MP3• Case III: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– Modify resource.txt so that ACTIVE mode use 15fps– Low interference in streaming
• Case IV: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– SWITCH mode of Server 1 and Server 2– Client has resources to support 15fps in ACTIVE mode
Evaluation Process for MP3• Case V: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in PASSIVE mode– Modify resource.txt so that ACTIVE mode use 25fps– Low interference in streaming
• Case VI: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in PASSIVE mode– Update resource.txt so that client has enough available
bandwidth to support two ACTIVE modes– SWITCH mode of Server 2 into ACTIVE mode– By default, no audio playback from Server 2; however
client should receive audio streams from Server 2
Evaluation Process for MP3• Case VII: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in ACTIVE mode– MUTE Server 1 audio playback and UNMUTE Server 2
audio playback– No interference in streaming– Client has enough resource to support two ACTIVE mode
• Case VIII: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in ACTIVE mode– SWITCH mode of Server 1 into PASSIVE mode
Evaluation Process for MP3• Case IX: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– Modify resource.txt so that admission for ACTIVE mode is
not successful– Streaming from Server 2 turns off, no interference in
Server 1 streaming
• Case X: – Server 1 is requested to streams send in ACTIVE mode– Modify resource.txt so that Server 2 can be requested in
ACTIVE mode with 15 fps– SWITCH between modes of Server 1 and Server 2
Evaluation Process of MP3• For All Cases:
– Cases are designed to run continuously one after another process without restarting the program or starting from the beginning
– Maintain audio-visual synchronization in ACTIVE mode– Show the computation of resource admission– Show the current frame rate, bandwidth usage, and
network jitter
Demonstration and Interview
• You must use at least 3 PCs/ laptops for the demo• If you want to use wireless/ wired with only your
laptops, please setup the demo in SC3401• If you want to use wired demo with lab PCs, please
setup your demo at SC0216• Demo date: April 30th • Demo time:– SC3401 @ 5pm – SC0216 @ 6pm
Additional Features for MP3?• It is highly recommended that you build an
additional feature in MP3 to SURPRISE US!• Bonus Point: 20• Bonus point will be awarded based on the
amount of surprise!