effect of message size and number of clients on ws frameworks for cis*6650.01 service oriented...

Effect Of Message Effect Of Message Size and Number Size and Number of Clients on WS of Clients on WS

FrameworksFrameworksFor CIS*6650.01 Service Oriented For CIS*6650.01 Service Oriented ComputingComputing

Dariusz GrabkaDariusz GrabkaGerett CommefordGerett Commeford

Jack ColeJack Cole

April 3, 2007 2Effects on WS Frameworks

BackgroundBackground

Web Service Frameworks provide Web Service Frameworks provide functionality to applications that they functionality to applications that they host, including:host, including: SOAP message handlingSOAP message handling XML parsingXML parsing data marshalling into and out of application data marshalling into and out of application

spacespace Performance doing these mundane but Performance doing these mundane but

intensive chores can greatly affect the intensive chores can greatly affect the overall throughput of an applicationoverall throughput of an application


GoalsGoals

The purpose of this experiment is to The purpose of this experiment is to compare major Web Service frameworks compare major Web Service frameworks to better under-stand their performance to better under-stand their performance under conditions of:under conditions of: increasing number of clientsincreasing number of clients increasing XML message lengthincreasing XML message length

We were particularly interested in We were particularly interested in conditions leading to unacceptable conditions leading to unacceptable service response (a denial-of-service service response (a denial-of-service situation)situation)


WSTestWSTest

WSTest is a Web Service benchmark WSTest is a Web Service benchmark developed by Sun and augmented by developed by Sun and augmented by MicrosoftMicrosoft

Results published by both sides in 2004Results published by both sides in 2004 Each showed results that demonstrated superior Each showed results that demonstrated superior

performance for their own platformperformance for their own platform http://msdn2.microsoft.com/en-us/vstudio/aa700840.aspxhttp://msdn2.microsoft.com/en-us/vstudio/aa700840.aspx

Tests results were published for just a couple Tests results were published for just a couple of casesof cases

Tests were performed on a single client, and Tests were performed on a single client, and on a network of 50 clients, on a network of 50 clients, with similar resultswith similar results


WSTestWSTest

Four testsFour tests EchoVoid – bi-directional empty messagesEchoVoid – bi-directional empty messages EchoStruct – echo an array of structures EchoStruct – echo an array of structures

containing an int, float, and stringcontaining an int, float, and string EchoList – echo a linked list of structuresEchoList – echo a linked list of structures GetOrder – receive a simulated purchase GetOrder – receive a simulated purchase

orderorder We have implemented EchoStruct as it We have implemented EchoStruct as it

provides a suitable range of testing provides a suitable range of testing optionsoptions


Test EnvironmentsTest Environments

Hardware:Hardware: 3 identical 2.4 GHz Pentium 4 systems3 identical 2.4 GHz Pentium 4 systems 512 Mb RAM512 Mb RAM 1 Gigabit throughput switch1 Gigabit throughput switch

Software (running under Windows Server SE Software (running under Windows Server SE 2003):2003):

Each platform hosts a different frameworkEach platform hosts a different framework IBM WebSphere Application Server Express 6.1IBM WebSphere Application Server Express 6.1 Microsoft .NET with IIS 6.0Microsoft .NET with IIS 6.0 Sun JAVA Application Server PE 9Sun JAVA Application Server PE 9


Test ConditionsTest Conditions

One server servicing one or two One server servicing one or two clients; no additional programs clients; no additional programs activeactive

All systems were removed from All systems were removed from external networksexternal networks Benchmarked at 11MBps total Benchmarked at 11MBps total

throughput, 98% efficiency between throughput, 98% efficiency between two machinestwo machines


Test ApplicationsTest Applications The EchoStruct service was implementedThe EchoStruct service was implemented Servers were developed in the native Servers were developed in the native

language for the frameworklanguage for the framework Java for WebSphere and Sun ASJava for WebSphere and Sun AS C# for .NETC# for .NET A framework-free gSoap server was also A framework-free gSoap server was also

developeddeveloped Two types of clients were developed Two types of clients were developed

C# (using .NET), creating client threadsC# (using .NET), creating client threads standard C (using gSoap), creating client standard C (using gSoap), creating client

processesprocesses Document-literal style was used Document-literal style was used


Test ApplicationsTest Applications Timing performed using the Windows Multi-media Timing performed using the Windows Multi-media

library with 1 mS resolutionlibrary with 1 mS resolution Two metrics: Two metrics:

Response time (start to end)Response time (start to end) User's view of service responseUser's view of service response

Latency (round-trip time)Latency (round-trip time) Removes client-side overhead from the measurementRemoves client-side overhead from the measurement

Care was taken to launch multiple clients Care was taken to launch multiple clients simultaneouslysimultaneously

Multiple transactions were used to ensure we Multiple transactions were used to ensure we were measuring service time and not client were measuring service time and not client deploymentdeployment


Test ApplicationsTest Applications

Detailed experiments of:Detailed experiments of: Variable number of clients from 1 to 1200Variable number of clients from 1 to 1200 Variable number of array structures (and Variable number of array structures (and

therefore message length) from 1 to 1000therefore message length) from 1 to 1000 Variable number of iterations from 1 to Variable number of iterations from 1 to

20002000 Single experiments to provide insight Single experiments to provide insight

into possible future directionsinto possible future directions These will be discussed at the end of this These will be discussed at the end of this

reportreport


Assumptions Assumptions

All hardware platforms and underlying All hardware platforms and underlying software perform essentially the samesoftware perform essentially the same Benchmarks shows some small variation in Benchmarks shows some small variation in

hard-drive write speeds (not considered)hard-drive write speeds (not considered) Windows client scheduling algorithms Windows client scheduling algorithms

are reasonableare reasonable Frameworks are reasonably configured Frameworks are reasonably configured

"out of the box""out of the box"


Array Tests & Results Array Tests & Results

EchoStruct Array TestEchoStruct Array Test Each structure contains an integer, a Each structure contains an integer, a

float, and a string (which we set to 30 float, and a string (which we set to 30 characters)characters) Serializes to approximately 120 bytesSerializes to approximately 120 bytes

Tests were run for array sizes of n=5, 50, Tests were run for array sizes of n=5, 50, 100, 200, 500, and 1000.100, 200, 500, and 1000.

In each trial 5 simultaneous clients In each trial 5 simultaneous clients submitted 30 repetitions of the array echo submitted 30 repetitions of the array echo test => 150*n echoed transactionstest => 150*n echoed transactions

30 trials were performed and times 30 trials were performed and times gathered from each individual clientgathered from each individual client


Array Test – Total Client Array Test – Total Client TimeTime


Array Test – Latency Array Test – Latency TimeTime


Client Tests & Results Client Tests & Results

EchoStruct Client TestEchoStruct Client Test Same structure as other testsSame structure as other tests Tests were run for n simultaneous Tests were run for n simultaneous

clients, n = 1, 5, 10, 20, 40, 60clients, n = 1, 5, 10, 20, 40, 60 In each trial, clients submitted 30 In each trial, clients submitted 30

repetitions of the 50 element array echo repetitions of the 50 element array echo test => 1500*n echoed transactionstest => 1500*n echoed transactions

Multiple trials were performed and Multiple trials were performed and times gathered from each individual times gathered from each individual clientclient


Client TestsClient Tests


Client Test – 600 ClientsClient Test – 600 Clients


Client Test – 1200 Client Test – 1200 ClientsClients


Iteration Tests & Results Iteration Tests & Results

EchoStruct Iteration TestEchoStruct Iteration Test Same structure as other testsSame structure as other tests Tests were run for n repetitions by each Tests were run for n repetitions by each

client, n = 5, 30, 100, 200, 500, 1000, client, n = 5, 30, 100, 200, 500, 1000, 20002000

In each trial, 5 simultaneous clients In each trial, 5 simultaneous clients submitted n repetitions of the 5 element submitted n repetitions of the 5 element array echo test => 25*n echoed array echo test => 25*n echoed transactionstransactions

Multiple trials were performed and times Multiple trials were performed and times gathered from each individual clientgathered from each individual client


Iteration TestIteration TestIteration Tests (Simultaneous C Clients)

0

5

10

15

20

25

0 500 1000 1500 2000

Iterations

To

tal

Lat

ency

(S

)

WebSphere Sun AS9 .NET


ConclusionsConclusions Performance of the 3 frameworks is similarPerformance of the 3 frameworks is similar Contrary to expectations, service times Contrary to expectations, service times

increase ~ linearly with both number of increase ~ linearly with both number of clients and size of transaction clients and size of transaction within limits within limits of our testingof our testing

As expected, number of iterations also As expected, number of iterations also produces a linear effectproduces a linear effect

Some results show interesting anomaliesSome results show interesting anomalies We believe that there is still much to be We believe that there is still much to be

discovered by pushing the frameworks discovered by pushing the frameworks harderharder


Further InvestigationFurther Investigation

1. Comparison to non-Framework-1. Comparison to non-Framework-based Web servicesbased Web services By using a gSoap C/C++ server, we can By using a gSoap C/C++ server, we can

perform the same tests with minimal perform the same tests with minimal overheadoverhead

This will give a near best-case This will give a near best-case performance metricperformance metric Preliminary testing shows more than 50% Preliminary testing shows more than 50%

improvementimprovement



2. Distribute the clients over multiple platforms2. Distribute the clients over multiple platforms Running multiple clients on a single platform will Running multiple clients on a single platform will

eventually lead to saturation of the client, and eventually lead to saturation of the client, and results will no longer be reflecting only the server results will no longer be reflecting only the server loadload

We postponed this effect by developing very fast We postponed this effect by developing very fast clients. Better to use multiple client platforms. clients. Better to use multiple client platforms.

In a single test (WebSphere, 50 clients, array size In a single test (WebSphere, 50 clients, array size 50) we found that time was 5% 50) we found that time was 5% worseworse when 2 when 2 platforms were usedplatforms were used

Implies some optimization for single clientsImplies some optimization for single clients Implies non-saturation to that point in our testingImplies non-saturation to that point in our testing



3. Comparison with client and server 3. Comparison with client and server on same platformon same platform Some network transfer time can be Some network transfer time can be

removed from the experiment and then removed from the experiment and then used to characterize other resultsused to characterize other results

Preliminary testing shows that this Preliminary testing shows that this introduces interesting scheduling introduces interesting scheduling complicationscomplications



4. Comparison of different message 4. Comparison of different message stylesstyles RPC-literal and Document literal styles RPC-literal and Document literal styles

involve different activities from the involve different activities from the frameworksframeworks

Additional clients and servers are Additional clients and servers are requiredrequired Versions have been created using gSoap Versions have been created using gSoap Operation has been verified but no Operation has been verified but no

comparisons have been performedcomparisons have been performed



5. Comparison of message wrapping5. Comparison of message wrapping Even though the services are Even though the services are

functionally identical, the wrappings functionally identical, the wrappings may be slightly differentmay be slightly different

We have used EtherReal to snoop the We have used EtherReal to snoop the SOAP messages, but have not pursued SOAP messages, but have not pursued this investigation yetthis investigation yet

effect of message size and number of clients on ws frameworks for cis*6650.01 service oriented...

Documents

ws frameworkswstestwstest

service time

service situationeffects

variable number of clients

ws frameworksfor cis

ws frameworksgoalsthe

unacceptable service

web service benchmark