application and desktop sharing - columbia...

Application and Desktop Sharing

Omer BoyaciNovember 1, 2007

Overview

• Introduction• Demo• Architecture• Challenges• Features• Conclusion

Application Sharing Models

• Application specific+ Efficient- Participants need application- Application has to be modified

• Generic- Inefficient (sometimes)

+ Participants don't need application

+ All applications are supported

Overview


Application Sharing

Sharing an application with multiple users

There is only one copy of the application

Participants do not need application itself

Briefly, participantsreceive screen updatessend keyboard and mouse events

Desktop sharing is also supported.

Screenshot

Screenshot (2)

Screenshot (Overlapped Windows)

1

23

4

Screenshot (Multicast App. Tool)

1

23

4

4

Screenshot (Ultra VNC)

1

2 3

4

4

Screenshot (App & Desk Sharing)

1

2

4

Supported Platforms/OS

Server Client*Windows + +*nix +Mac OS X - +

-+

*Client is Java based.

Overview


Demo

Demo

Overview


System Architecture

• Client/Server Software Architecture

System Architecture


Screen Updates

System Architecture


Keyboard Mouse Events

Client (Viewer) Architecture

• Client can– Connect to server– Wait for incoming connections

• Client supports– TCP– UDP (+Multicast)

Client (Viewer) Architecture

• Client receives these commands – Open new window– Window size changed– Pixel update– Close window

• Client sends– BFCP (Binary Floor Control Protocol) commands– Keyboard and mouse events

Windows Server Architecture

Kernel space

User space

Mirror Driver

0X 300 ,500Y 200 ,600

Server Main Thread

Window coordinates


X 320 ,340Y 100 ,200Kernel space

User space X 320 ,340Y 100 ,200

X 320 ,340Y 100 ,200

X 320 ,340Y 100 ,200

Mirror Driver

2X 300 ,500Y 200 ,600

Server Main Thread

Window coordinates


Kernel space

User space X 320 ,340Y 100 ,200

X 320 ,340Y 100 ,200

X 320 ,340Y 100 ,200

Mirror Driver

2X 300 ,500Y 200 ,600

Server Main Thread

Window coordinates


K e r n e l s p a c e

U s e r s p a c e X 3 2 0 , 3 4 0Y 4 5 0 , 5 5 0

X 4 2 0 ,4 4 0Y 2 0 0 ,6 0 0

X 3 2 0 , 3 4 0Y 4 0 0 , 5 0 0 2

S e r v e r M a i n T h r e a d

W i n d o w c o o r d i n a t e s

C l i e n t T h r e a d - 1



X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

Overview


Challenges

• Different client bandwidths/speeds• Late Joiner• The effects of packet loss• Reliable multicast

Multimedia Support (Movies)


• Our system uses PNG to compress and transmit the region updates

• PNG is lossless and effective for computer generated images but ineffective for real world captures like pictures or movies

• JPG is more suitable for photographic images• However, JPG is lossy and not effective for

computer generated images (text, line, shapes,...)

• Our system should use both


• Composite image comparing JPEG and PNG: notice artifacts in JPEG versus solid PNG background.

Multimedia Support (PNG vs JPG)

Size x Size 4x0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

150

Image Size

PNG

JPG

Size x360x150162K

Size 4x720x300648K

1:301:20

1:51:4


Ethernet (60Mb/s)

Size x Size 4x0

2.5

5

7.5

10

12.5

15

17.5

20

22.5

FPS

PNG

JPG

Size x Size 4x0

100

200

300

400

500

600

700

800

900

1000

1100

KB/s

PNG

JPG


Size x Size 4x0

2.5

5

7.5

10

12.5

15

17.5

20

22.5

FPS

PNG

JPG

Size x Size 4x0

50

100

150

200

250

300

350

400

KB/s

PNG

JPG

Wireless (4Mb/s)

PNG/JPG Detection Algorithm

Region>40,000px

?

YES

New Region

?

NO

Use DetectedFormat

YES

-1,0,1

coordinates

PNG Size

Time Stamp

counter

Region recordCreate a record&

Start Checking

Detected?

Continue Checking

NO YES

Sharing a Movie (Media Player)

Sharing a Movie File

• Capturing from the Frame Buffer is expensive.• Instead

– Transcode the movie on the fly to JPGs

• FFmpeg (subproject of mplayer)● FFmpeg is a complete solution to record, convert

and stream audio and video. It includes libavcodec, the leading audio/video codec library. FFmpeg is developed under Linux, but it can compiled under most operating systems, including Windows.

• FFmpeg-Java (subproject of FMJ)– Java wrapper (JNA: Java Native Access)

Sharing a Movie (Our Method)

Sharing a Movie File

Movie File (wmv,mpg,divx)Movie File (wmv,mpg,divx)

JPG images

FFmpeg-Java

Java Streaming

Server

Java Client

Java Client

UDP/Multicast

decodeNextFrame()encodeFrameToJPG()

Demo

DemoMultimedia Support

Challenges


Different Client Bandwidths/Speeds





X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

1 M b p s

2 M b p s

3 M b p s


• Possible Solutions– Slowest one

– Average speed

– Fastest one



• Problem: Penalize everybody except the slowest

– Average speed

– Fastest one




– Average speed• Possible solution

– Fastest one




– Average speed• Possible solution (Can we do better?)

– Fastest one




– Average speed• Possible solution (Can we do better?)

– Fastest one• The best solution• Client bandwidths are fully utilized





X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

X 320,340

Y 400,500

timestam

pP

NG

1 M b p s

2 M b p s

3 M b p s



Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 3

20,340

Y 4

00,500

timesta

mp

PN

G

X 32

0,340

Y 40

0,500

timestam

pP

NG

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG

1Mbps

2Mbps

3Mbps

Challenges


Late Joiner

• Force server to generate full screen update

Late Joiner

• Force server to generate full screen update– Problems

• Misbehaving clients can degrade performance• If Join/Leave rate is high, too much burden on server

– Solution• Generate full screen updates if really necessary• Otherwise start the new client from last full screen update


Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG

1Mbps

2Mbps

3Mbps

Client Thread -4

8Mbps


Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG

1Mbps

2Mbps

3Mbps

Client Thread -4

8Mbps

FIR


Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

timesta

mp

PN

G

X 32

0,340

Y 40

0,500

timestam

pP

NG

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG

1Mbps

2Mbps

3Mbps

Client Thread -4

8Mbps


Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

timesta

mp

PN

G

X 32

0,340

Y 40

0,500

timestam

pP

NG

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG

1Mbps

2Mbps

3Mbps

Client Thread -4

8Mbps


Server Main Thread

Client Thread -1

Client Thread -2

Client Thread -3

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

timesta

mp

PN

G

X 32

0,340

Y 40

0,500

timestam

pP

NG

X 32

0,34

0Y

400,5

00

time

stam

pP

NG

X 3

20,34

0Y

400

,500

timesta

mp

PN

G

X 3

20,340

Y 4

00,500

time

stamp

PN

G

X 32

0,34

0Y

400,5

00

timestam

pP

NG1Mbps

2Mbps

3Mbps

Client Thread -4

8Mbps

Client Thread -5

2Mbps

Challenges


The effects of Packet Loss

• This problem applies to – Multicast– UDP

• The PNG images can be large– Regular desktop can be ~900KB– ~600 Ethernet packets– One packet loss wastes all PNG image


• Solution– Small PNG images

• Around ~1500 bytes• Consist of a few scanlines

– Disadvantages• Increased CPU usage (client&server)• Lower compression ratio (%20 lower)

– Advantages• One packet loss = no update for a few scanlines

Challenges


Reliable Multicast

• RTP Library stores last N rtp packets

• Clients send NACK for lost packets

• RTP Library resend the requested packets

Overview


Recording

• Clients can record the whole/part session• Anybody can play these files locally• These files can be streamed to receivers via

streaming server• Streaming server supports multiple receivers

– Also late joiners

Listening Client

• Client waits for incoming connections• It can display windows from multiple user• Can be used for RGB cable replacement

Overview


Conclusion

• Application sharing allows users to share a single application with multiple participants.

• Participants don't need the application.

• It is not specific to a single application.

• Extra features like recording is added.

Thanks

• Questions?

application and desktop sharing - columbia...

Documents