ASM Performance Test Report

Conducted By: Murray Wardle & Srimanta MukherjeeAccess TestingDate: 19/10/2010

1. Executive Summary...........................................................................31.1. Objectives....................................................................................31.2. Analysis Session Summary...........................................................31.3. Conclusions..................................................................................5

2. Scenario Configuration.......................................................................62.1. Files..............................................................................................62.2. Scheduler Information..................................................................62.3. Scripts..........................................................................................62.4. Run Time Settings........................................................................7

Test steps.................................................................................................................73. Running Vusers..................................................................................84. Average Transaction Response Time.................................................95. Errors per Second.............................................................................116. Throughput.......................................................................................137. Hits per Second................................................................................148. Connections......................................................................................159. Terminology.....................................................................................16

9.1. LoadRunner Objects...................................................................169.2. Graph Information......................................................................16

2

1. Executive Summary

1.1. Objectives

Determine whether the system is capable of supporting the target transaction rate while retaining its functional stability and responding within pre-defined guidelines.

Identify any bottlenecks preventing the achievement of performance targets and assist in their resolution.

1.2. Analysis Session Summary

Analysis Summary Period: 18-Oct-2010 03:47:05 PM - 18-Oct-2010 06:01:05 PM

Project Name: ASMTest Name: 01_LoadTest1_100%Test Description: ASM Load test at 100% peak load.Run Time: 18-Oct-2010 03:46:43 PMDuration: 2 hours and 14 minutes.User Notes: none

Statistics Summary

Maximum Running Vusers: 17

Total Throughput (bytes): 148,921,442

Average Throughput (bytes/second): 18,520

Total Hits: 16,804

Average Hits per Second: 2.09   

Total Errors: 17

3

 Transaction Summary 

Transactions: Total Passed: 1,377

Total Failed: 34

Total Stopped: 0

         Average Response Time

Transaction NameSLA Status

Minimum

Average Maximum

Std. Deviation

90 Percent

Pass

Fail

Stop

VU01_00_ALL 160.31 180.978 200.919 9.357 194.715 57 0 0

VU01_01_LandingPage 0.062 0.18 2.531 0.449 0.141 57 0 0

VU01_02_ClkImageASMqvw 1.048 1.121 1.484 0.061 1.172 57 0 0

VU01_03_ClickTabMnthlyRep 0.249 0.315 1.969 0.222 0.313 57 0 0

VU01_04_ClickTabHeldOfferDetails 0.156 0.18 0.203 0.01 0.187 57 0 0

VU01_05_ClickDropDownMonthFirstHeld 0.25 0.265 0.297 0.008 0.266 57 0 0

VU01_06_SelectMonth 0.203 0.216 0.719 0.068 0.219 57 0 0

VU01_07_SelectRegion 0.125 0.344 0.781 0.25 0.734 57 0 0

VU01_08_OrderByBusinessManager 0.109 0.325 0.906 0.241 0.687 57 0 0

VU01_09_FilterByProductName 0.016 0.018 0.031 0.006 0.031 57 0 0

VU01_10_BusinessVehicleLoan 0.156 0.386 2.312 0.362 0.766 57 0 0

VU02_00_ALL 97.796 121.62 136.811 7.744 131.607 56 0 0

VU02_01_LandingPage 0.062 0.273 2.453 0.602 0.219 56 0 0

VU02_02_ClickImageASMqvw 1.063 1.129 1.5 0.068 1.187 56 0 0

VU02_03_CurrentHeldOffersPage 0.187 0.202 0.266 0.013 0.219 56 0 0

VU02_04_CurrentHeldOffersRegion 0 0.001 0.062 0.008 0 56 0 0

VU02_05_BusinessManagerSummarisation 0.062 0.068 0.203 0.025 0.063 56 0 0

VU02_06_FiltersRegion 0.093 0.122 0.468 0.05 0.126 56 0 0

VU02_07_OrderByDealership 0.031 0.054 0.063 0.011 0.063 56 0 0

VU03_00_ALL 211.542 234.986 247.495 8.045 242.949 14 17 0

VU03_01_LandingPage 0.062 0.171 2.437 0.415 0.156 31 0 0

VU03_02_ClickImageASMqvw 1.031 1.107 1.235 0.041 1.156 31 0 0

VU03_03_ClickTabTimeAnalytics 0.265 0.29 0.343 0.015 0.297 30 1 0

VU03_04_Select_StartDate 0.125 0.128 0.188 0.012 0.125 28 2 0

VU03_05_Select_EndDate 0.125 0.127 0.156 0.006 0.125 27 1 0

VU03_06_SelectFirstStatus 0.204 0.219 0.234 0.004 0.219 24 3 0

VU03_07_SelectSecondStatus 0.296 0.3 0.312 0.006 0.312 23 1 0

VU03_08_SelectGranularity 0.515 0.525 0.594 0.02 0.531 23 0 0

VU03_09_FilterByReferredItems 0.484 0.496 0.547 0.013 0.501 22 1 0

VU03_10_FilterByRegion 0.484 0.512 0.562 0.025 0.562 19 3 0

VU03_11_OrderByAverageTime 0.406 0.43 0.578 0.039 0.437 16 3 0

VU03_12_ClearFiltersWorkAround 0.14 0.157 0.203 0.014 0.156 14 2 0

1.3. Conclusions

4

Five noticeable spikes in page response times occurred for he “QlikView Access Point” page, the impact of these spikes to the end user is minimal. All page response times were acceptable during the test.

Seventeen errors occurred during the test, all of these errors resulted in a “Failed to connect” message being sent to the client browser. Only page in the “Time Analytics Tab” were affected. Error details and timestamps for each occurrence of these errors have been provided to the project for further investigation.

5

2. Scenario Configuration

2.1. Files

Name: d:\Apps\HP\Performance Center\orchidtmp\scenarios\scn38.tmp

Results in Session:d:\Apps\HP\Performance Center\orchidtmp\Results\5\5.lrr

Session Name: D:\TEMP\MercuryOrchid_85829274\Sess_85835068\ASM\01_LoadTest1_100_\18_Oct_2010 03_46_43 PM\Results\Results.lra

2.2. Scheduler Information

Started On: 10/18/2010 15:47:05Ended On: 10/18/2010 18:01:06Duration: Until CompletionLoad Behavior: Load all Vusers simultaneously

2.3. Scripts

Script Type File

VU02_HeldOffersTab_v0_1_1Multi+QTWeb

\\aupperfdb01\LRFS\2\Scripts\VU02_HeldOffersTab_v0_1\VU02_HeldOffersTab_v0_1.usr

VU01_MonthlyHeldOfferDetailsRep_v0_1_1

Multi+QTWeb

\\aupperfdb01\LRFS\2\Scripts\VU01_MonthlyHeldOfferDetailsRep_v0_1\VU01_MonthlyHeldOfferDetailsRep_v0_1.usr

VU03_TimeAnalyticsTab_v0_1_1Multi+QTWeb

\\aupperfdb01\LRFS\2\Scripts\VU03_TimeAnalyticsTab_v0_1\VU03_TimeAnalyticsTab_v0_1.usr

6

2.4. Run Time Settings

Target load level: 100% Current peak load

Cache settings: Each iteration of each VU will emulate a new user with an empty browser cache

Groups: Each script will form a single group

Load Injectors: Load from each group will be spread evenly across all injectors

Test steps Ramp up: 15 minutes

Sustain: 90 minutes

Ramp down: 15 minutes

Process Total VUsThink Time Settings* Pacing**

Trans. Per Hour

VU01_HeldOffersReport 6 75% - 125% 576 - 864 30

VU02_HeldOffersTab 6 75% - 125% 576 - 864 30

VU03_TimeAnalytics 5 75% - 125% 960 - 1440 15

Total 17 75

*The range of random deviation from the recorded think time.**The range of start-to-start pacing, in seconds, selected as random.

7

3. Running Vusers

Displays the number of Vusers that executed Vuser scripts, and their status, during each second of a load test. This graph is useful for determining the Vuser load on your server at any given moment.

Title: Running VusersCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: Vuser Status = (Run)Group By:Granularity: 30 Seconds

Color Scale Measurement

Graph Min.

Graph Ave.

Graph Max.

Graph Median

Graph SD

1 Run 0.0 8.094 17 8 5.162

8

4. Average Transaction Response Time

Displays the average time taken to perform transactions during each second of the load test. This graph helps you determine whether the performance of the server is within acceptable minimum and maximum transaction performance time ranges defined for your system.

Title: Average Transaction Response TimeCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: Transaction End Status = (Pass), (Include Think Ti

me)Group By:Granularity: 120 Seconds

Color Scale Measurement Min. Ave. Max. SD

1 VU01_01_LandingPage 0.062 0.18 2.531 0.449

1 VU01_02_ClkImageASMqvw 1.048 1.121 1.484 0.061

1 VU01_03_ClickTabMnthlyRep 0.249 0.315 1.969 0.222

1 VU01_04_ClickTabHeldOfferDetails 0.156 0.18 0.203 0.01

1 VU01_05_ClickDropDownMonthFirstHeld

0.25 0.265 0.297 0.008

1 VU01_06_SelectMonth 0.203 0.216 0.719 0.068

1 VU01_07_SelectRegion 0.125 0.344 0.781 0.25

1 VU01_08_OrderByBusinessManager 0.109 0.325 0.906 0.241

1 VU01_09_FilterByProductName 0.016 0.018 0.031 0.006

9

1 VU01_10_BusinessVehicleLoan 0.156 0.386 2.312 0.362

1 VU02_01_LandingPage 0.062 0.273 2.453 0.602

1 VU02_02_ClickImageASMqvw 1.063 1.129 1.5 0.068

1 VU02_03_CurrentHeldOffersPage 0.187 0.202 0.266 0.013

1 VU02_04_CurrentHeldOffersRegion 0.0 0.001 0.062 0.008

1 VU02_05_BusinessManagerSummarisation

0.062 0.068 0.203 0.025

1 VU02_06_FiltersRegion 0.093 0.122 0.468 0.05

1 VU02_07_OrderByDealership 0.031 0.054 0.063 0.011

1 VU03_01_LandingPage 0.062 0.171 2.437 0.415

1 VU03_02_ClickImageASMqvw 1.031 1.107 1.235 0.041

1 VU03_03_ClickTabTimeAnalytics 0.265 0.29 0.343 0.015

1 VU03_04_Select_StartDate 0.125 0.128 0.188 0.012

1 VU03_05_Select_EndDate 0.125 0.127 0.156 0.006

1 VU03_06_SelectFirstStatus 0.204 0.219 0.234 0.004

1 VU03_07_SelectSecondStatus 0.296 0.3 0.312 0.006

1 VU03_08_SelectGranularity 0.515 0.525 0.594 0.02

1 VU03_09_FilterByReferredItems 0.484 0.496 0.547 0.013

1 VU03_11_OrderByAverageTime 0.406 0.43 0.578 0.039

1 VU03_12_ClearFiltersWorkAround 0.14 0.157 0.203 0.014

Most Page response times were under 0.8 seconds and were stable and consistent throughout the entire test.

Five noticeable spikes in page response times occurred for the “QlikView Access Point” page (http://aupqlik01/qlikview/ - Measurement timers VU01_01, VU02_01 & VU03_01). The worst of these spikes was 2.5 seconds.

The initial ASM page (Measurement timers VU01_02, VU02_02 & VU03_02) was the slowest page which typically took 1.2 seconds. Again, response times for this page were stable and consistent throughout the entire test.

Overall page response times were acceptable during the test.

10

5. Errors per Second

Displays the average number of errors that occurred during each second of the scenario run, grouped by error code.

Title: Errors per SecondCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: NoneGroup By:Granularity: 1 Second

Color Scale Measurement Graph Min.

Ave. Graph Max.

Graph Median

Graph SD

1Failed to connect | Error Description: Failed to connect | Error Number: -2147352567

0.0 0.002 1 0.0 0.046

The error “Failed to connect | Error Description: Failed to connect | Error Number: -2147352567” occurred 17 times during the test on the following pages.

Time Analytics Tabo Click Tab Time Analytics (x1)o Select Start Date (x2)o Select End Date (x1)o Select First Status (x3)

11

o Select Second Status (x1)o Filter By Referred Items (x1)o Filter By Region (x3)o Order By Average Time (x3)o Clear Filters (x2)

Only functionality in the ‘Time Analytics Tab’ area was affected by this issue. No issues were encountered in the ‘Held Offer Detail Report’ or ‘Held Offers Tab’ areas of ASM.

12

6. Throughput

Displays the amount of throughput (in bytes) on the Web server during the load test. Throughput represents the amount of data that the Vusers received from the server at any given second. This graph helps you to evaluate the amount of load Vusers generate, in terms of server throughput.

Title: ThroughputCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: NoneGroup By:Granularity: 120 Seconds

Color Scale Measurement

Graph Min.

Ave. Graph Max.

Graph Median

Graph SD

1 Throughput 0.0 18520.264

48645.475 17153.775 11771.508

13

7. Hits per Second

Displays the number of hits made on the Web server by Vusers during each second of the load test. This graph helps you evaluate the amount of load Vusers generate, in terms of the number of hits.

Title: Hits per SecondCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: NoneGroup By:Granularity: 120 Seconds

Color Scale Measurement

Graph Min.

Ave. Graph Max.

Graph Median

Graph SD

1 Hits 0.0 2.09 5.45 2.008 1.342

14

8. Connections

Displays the number of Connections

Title: ConnectionsCurrent Results:d:\Apps\HP\Performance Center\orchidtmp\Results\5\

5.lrrFilters: NoneGroup By:Granularity: 120 Seconds

Color Scale Measurement

Graph's Min.

Graph's Ave.

Graph's Max.

Graph's Median

Graph's SD

1 Connections 0.0 25.833 34 32 11.441

15

9. Terminology

9.1. LoadRunner Objects

Term Definition

Vuser ScriptsA Vuser script describes the actions that a Vuser performs during the scenario. Each Vuser executes a Vuser script during a scenario run. The Vuser scripts include functions that measure and record the performance of your application�s components.

Load TestTests a system's ability to handle a heavy workload. A load test simulates multiple transactions or users interacting with the computer at the same time and provides reports on response times and system behavior.

Run-Time Settings

Run-Time settings allow you to customize the way a Vuser script is executed. You configure the run-time settings from the Controller or VuGen before running a scenario. You can view information about the Vuser groups and scripts that were run in each scenario, as well as the run-time settings for each script in a scenario, in the Scenario Run-Time Settings dialog box.

ScenarioA scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions that they perform, and the machines on which they run their emulations.

Scheduler

The Schedule Builder allows you to set the time that the scenario will start running, the duration time of the scenario or of the Vuser groups within the scenario, and to gradually run and stop the Vusers within the scenario or within a Vuser group. It also allows you to set the load behavior of Vusers in a scenario.

Session

When you work with the Analysis utility, you work within a session. An Analysis session contains at least one set of scenario results (lrr file). The Analysis utility processes the scenario result information and generates graphs and reports. The Analysis stores the display information and layout settings for the active graphs in a file with an .lra extension. Each session has a session name, result file name, database name, directory path, and type.

TransactionsA transaction represents an action or a set of actions used to measure the performance of the server. You define transactions within your Vuser script by enclosing the appropriate sections of the script with start and end transaction statement.

VusersVusers or virtual users are used by LoadRunner as a replacement for human users. When you run a scenario, Vusers emulate the actions of human users working with your application. A scenario can contain tens, hundreds, or even thousands of Vusers running concurrently on a single workstation.

9.2. Graph Information

Term DefinitionAverage Average value of the graph measurement's.Hits The number of HTTP requests made by Vusers to the Web server.Maximum Maximum value of the graph measurement's.Measurement This is the type of resource being monitoredMedian Middle value of the graph measurement's.Minimum Minimum value of the graph measurement's.

Network Delay The time it takes for a packet of data sent across the network to go to the requested node and return.

Network Path The Network Path is the route data travels between the source machine and the destination machine.

Response time The time taken to perform a transaction.

Scale (or granularity)

In order to display all the measurements on a single graph, thus making the graphs easier to read and analyze, you can change the scale or (granularity) of the x-axis. You can either set measurement scales manually, view measurement trends for all measurements in the graph, or let Analysis scale them automatically. The Legend tab indicates the scale factor for each resource.

Standard Deviation (SD)

The square root of the arithmetic mean value of the squares of the deviations from the arithmetic mean.

Throughput Throughput is measured in bytes and represents the amount of data that the Vusers received from the server.

Vuser Load When you run a scenario, the Vusers generate load or stress on the server. LoadRunner monitors the effect of this load on the performance of your application.

16

17